Jayanth y

About Jayanth y

Data engineer with 9+ years of experience in designing and implementing scalable data pipelines, ETL frameworks, and data warehousing solutions across cloud platforms like AWS, GCP, and Azure. Proficient in Big Data technologies including Hadoop, Spark, Kafka, and Hive, and skilled in Python, Scala, Java, and SQL.Experienced in building and maintaining data lakes, real-time streaming pipelines, and data warehouses, with expertise in tools such as Databricks and Informatica. Strong understanding of machine learning techniques and their integration into data pipelines using Spark MLLib, TensorFlow, and Scikit-learn. Experienced in DevOps practices, including CI/CD pipelines and containerization using Docker and Kubernetes, ensuring the deployment of scalable and reliable solutions.Demonstrated ability to lead data engineering projects, collaborate with cross-functional teams, and deliver high-quality data solutions in Agile environments.

Experience

1. Senior Data Engineer

   Birlasoft

   Mar 2022 — Present · St Louis, MO, US

   Designed and implemented a cloud-based data processing platform using AWS services (S3, EC2, EMR, Redshift, Lambda), reducing processing time by 40%.• Led the development of ETL pipelines using Databricks and Spark, integrating data from Hadoop, Kafka, and RDBMS into Snowflake for real-time analytics.• Managed data models (star and snowflake schemas), improving query performance and cutting data retrieval time by 50%.• Developed custom data transformation scripts in Python and Scala, automating data ingestion and processing tasks.• Migrated on-prem data warehouses to AWS Redshift and Snowflake, reducing operational costs by 30%.• Ensured compliance with data governance policies using AWS Glue Data Catalog.• Deployed machine learning models via Spark MLLib and TensorFlow for predictive analytics.• Automated infrastructure provisioning with Terraform and CloudFormation.• Established CI/CD pipelines using Jenkins, Docker, and Kubernetes, streamlining deployments.• Created Tableau and Power BI dashboards, delivering actionable insights to stakeholders.• Integrated real-time data into data warehouses using Kafka and Spark Streaming.• Led a team of data engineers, ensuring code quality and best practices.• Optimized Spark jobs and Hive queries, improving processing efficiency by 35%.• Implemented security protocols using AWS IAM, securing data access and encryption.• Delivered data projects within Agile sprints, collaborating with cross-functional teams.

Education

• Jawaharlal Nehru Technological University

  Bachelor of Technology - BTech, Computer Science