Rahul Dupati

Senior Bi Data Engineer @CVS Health

NY, US

Design, develop, and maintain scalable and efficient ETL/ELT pipelines using Informatica PowerCenter and Azure Data Factory to ingest, transform, and load data from diverse sources into Snowflake data warehouse environments.• Implement and optimize data ingestion workflows leveraging Apache Kafka for real-time streaming data integration and Oracle GoldenGate for change data capture (CDC) to ensure near real-time data availability.• Develop and manage data transformation processes using dbt (data build tool) integrated with Snowflake to build modular, tested, and version-controlled data models.• Utilize PySpark and Snowflake Fabric for large-scale data processing and transformation tasks, ensuring high performance and data quality.• Orchestrate complex data workflows and pipeline scheduling using Apache Airflow and Azure Data Factory, ensuring reliable execution and monitoring of data jobs.• Manage and optimize data storage and lakehouse solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and Apache Iceberg for efficient data versioning and schema evolution.• Implement metadata pipeline management to maintain data lineage, governance, and quality standards across the data ecosystem.• Collaborate with data analysts, data scientists, and business stakeholders to gather requirements and deliver actionable insights through well-structured data pipelines.• Write reusable, maintainable, and efficient code in Python for automation, data validation, and pipeline enhancements.• Monitor and troubleshoot ETL processes, ensuring data accuracy, consistency, and timely delivery.• Participate in data quality initiatives by implementing validation rules and data profiling techniques to ensure high data integrity.• Integrate Snowflake with orchestration and transformation tools, leveraging Snowflake Python Connector, Snowpark, and IDMC (Informatica Data Management Cloud) for seamless data operations.