Sai Nikhil T

WORK HISTORY

ABOUT SAI NIKHIL T

Senior Data Engineer with 10+ years in (3 Years Intership) of experience in designing, developing, and deploying scalable Spark-based ETL pipelines across enterprise environments. Specialized in PySpark, Python, and Apache Spark (batch & streaming) for large-scale data transformation, integration, and analytics. Proven ability to develop and optimize ETL workflows handling multi-terabyte datasets across AWS EMR, Lambda, Glue, and S3 environments. Strong expertise in working with relational databases including Oracle, PostgreSQL, SQL Server, and MySQL for efficient data extraction and transformation. Skilled in implementing validation logic, schema enforcement, and data quality checks as part of robust ETL workflows. Designed and optimized Spark DataFrame-based transformations, improving performance, scalability, and cost efficiency on cloud platforms. Hands-on experience in AWS cloud data stack: Glue, Redshift, EMR, Lambda, Athena, RDS, Step Functions, CloudWatch. Experienced in Azure Databricks, Synapse, and Data Factory for data integration and workflow orchestration. Strong background in data modeling, partitioning strategies, query tuning, and performance optimization in Spark and SQL. Experienced in building event-driven architectures leveraging Lambda, SNS, and SQS for automated pipeline orchestration. Proficient in orchestrating workflows using Apache Airflow, Step Functions, and Databricks Jobs for end-to-end data pipeline automation. Built real-time ingestion and streaming pipelines using Spark Structured Streaming and Kafka for operational and analytics use cases. Strong track record of collaborating with data architects, analysts, and business stakeholders to deliver reliable and actionable data solutions. Experienced in migrating legacy ETL frameworks (Informatica, Hive) into modern PySpark and AWS Glue-based solutions. Adept at implementing DevOps practices with CI/CD pipelines (Jenkins, GitHub Actions, Azure DevOps) for PySpark code deployments. Built data validation and profiling frameworks in Python and Pandas to support testing and monitoring of production ETL jobs. Worked extensively with Snowflake, Redshift, and Aurora for analytical and transactional workloads. Integrated Spark with graph databases (Neo4j) and real-time monitoring tools like Splunk for advanced analytics. Recognized for mentoring teams, conducting code reviews, writing documentation, and enabling best practices in ETL design. Strong communicator skilled in translating complex data engineering solutions into business insights that drive decision-making.