Abhishek Reddy Gogi Reddy

Data Engineer @Northern Trust

Atlanta, GA, US

Worked closely with business users to define the business requirements.• Designed and implemented end-to-end data pipelines using Azure Data Factory to orchestrate data movement from on-premises and cloud sources.• Developed transformation logic in Azure Databricks using PySpark, handling large-scale data from multiple sources (Parquet,CSV). Optimized PySpark applications on Databricks, which yielded a significant amount of costreduction.• Developed Databricks notebooks using PySpark and Spark-SQL for data extraction,transformation and aggregation from multiple systems and stored on Azure Data Lake Storageusing Azure Databricks notebooks• Created ADF Pipelines to build the Type 1 and Type 2 dimension.• Integrated ADF pipelines with Azure Key Vault and Linked Services for secure credential management.• Scheduled and monitored batch workloads using ADF triggers and monitored job performance using Azure Monitor and Log Analytics.• Have good experience working with Azure Data Lake storage and loading data into Azure SQL Synapse analytics (Datawarehouse).• Worked with CDPF Frame work to generate ETL Pipelines.• Designed, developed, and optimized data models (staging, intermediate, and marts) in DBT, ensuring scalability and maintainability.• Implemented modular SQL transformations using dbt models, CTEs, and Jinja templating.• Contributed to data documentation and lineage tracking using dbt docs and dbt lineage graphs.• Implemented dimensions and fact table through the Python Frame work.• Generated Airflow Dag’s to automate the ETL Pipelines.• Worked on auto dag generated code to automate the Airflow dags.• Implemented SQL queries to extract and analyze the data and improving the query performance.

JNTUH College of Engineering Hyderabad

Bachelor of Technology - BTech

Rivier University

Master's degree, Computer Science