Madhav Goel

About Madhav Goel

Data Engineer having more than 4.5 years of experience in big data technologies and data processing tools.Experience in big data components - Hadoop,Hive, Hdfs, Spark, DBT, Databricks, Snowflake, Talend, Sqoop and many others.Hands-on experience crafting data pipelines using Pyspark on AWS, proficient in configuring AWS services like ECS, S3, Glue and Athena, and adept at optimizing PySpark code for distributed processing on AWS clusters.Familiar with ETL processes and have in-depth knowledge of data processing with ETL tools i.e Abinitio,Talend.Experience in creating end to end data pipelines to process big data with Pyspark SQL and dataframe APIs.Have in-depth knowledge of PySpark,Hadoop, SQL,Unix,Python,Hive, etc.Experience of working in Agile methodology and adaptive to challenges.

Experience

1. Data Engineer

   EPAM Systems

   Jun 2024 — Present · Gurugram, IN

   Customer: Clarivate Analytics• Part of the Data Ingestion team, responsible for designing and building scalable big data pipelines within the Life Sciences & Healthcare (LS & H) domain. Leveraged AWS services such as ECS, S3, Secrets Manager, Glue, and CleanRoom, with Snowflake as the data lakehouse platform, to enable secure and efficient data management.• Key contributor to the Data Delivery team, managing both scheduled and ad-hoc data shipments to customers. Utilized Talend for seamless data integration and Snowflake as the cloud data platform, ensuring accurate, efficient, and secure data delivery while maintaining high client satisfaction and operational reliability.• Currently leading the end-to-end development of a DBT framework from scratch to migrate Talend jobs. Implementing Apache Airflow for workflow orchestration and automation, enhancing data processing efficiency and operational scalability.

Education

• KIET Group of Institutions

  Bachelor of Technology - BTech, ECE

  2017 — 2021