Sunil P.

About Sunil P.

I am a Data Engineer with over 7 years of experience in building and optimizing data solutions that empower businesses to make smarter decisions. My work focuses on creating scalable pipelines, managing large datasets, and leveraging technologies like Apache Spark, Kafka, AWS, Azure, and Snowflake to deliver efficient and reliable data workflows. I have a strong track record of designing and implementing data lakes, warehouses, and ETL pipelines that simplify complex data integration and transformation processes. From migrating systems to the cloud to developing real-time solutions, I take pride in solving data challenges and driving impactful outcomes for businesses. With expertise in tools like PySpark, Kafka, and SQL, I enjoy collaborating with teams to turn raw data into actionable insights. I’m passionate about using advanced cloud platforms like AWS Glue and Azure Data Factory to deliver seamless, high-performance solutions. Whether it’s designing a new architecture or optimizing existing workflows, I bring a hands-on, results-oriented approach to every project. I’m always looking for ways to push the boundaries of what’s possible with data. If you’d like to connect or discuss how I can help drive your data initiatives forward, feel free to reach out!

Experience

1. Senior Data Engineer

   Centene Corporation

   Aug 2022 — Present · St. Louis, MO, US

   Collaborated with business stakeholders to gather requirements and designed, analyzed, and implemented big data applications to drive informed decision-making, optimizing customer acquisition and business processes. Developed and maintained scalable ETL/ELT pipelines using Azure Data Factory, T-SQL, Spark SQL, and U-SQL to extract, transform, and load data into Azure Data Storage, including Data Lake and SQL. Deployed Docker containers and virtual machines with Kubernetes for processing data pipelines and enabling seamless operations. Built real-time Kafka streaming pipelines using Scala to ingest and transform data from multiple sources. Utilized AWS Data Migration Services, Schema Conversion Tool, and Matillion ETL for efficient data migration and management, including the transition of Teradata objects into Snowflake. Designed on-demand table creation workflows using AWS Lambda, Glue, and PySpark for efficient data processing on S3. Developed machine learning algorithms for personalized recommendations, clustering, and customer behavior analysis. Architected end-to-end data pipelines on GCP for data ingestion and transformation, coordinating team responsibilities. Implemented robust monitoring solutions with Ansible, Terraform, Docker, and Jenkins for reliable data pipeline management. Built and optimized Spark applications with PySpark and Spark SQL for advanced transformations and aggregations, uncovering actionable customer insights. Prepared and blended data using Alteryx and SQL to create Tableau-ready datasets and published them on Tableau Server for business intelligence. Automated job execution on AWS EC2 using Oozie workflows and Hive for efficient data processing. Enhanced the performance and efficiency of ETL processes through optimization strategies applied to EMR clusters. Processed and imported data from diverse sources into Spark RDD for seamless analysis and advanced workflows.

Education

• University of Denver

  Master of Science - MS