Avinash Reddy
Sr. Data Engineer | Azure Data Factory • Databricks • Synapse • Data Lake | Python • PySpark • SQL
- Role
- Sr Data Engineer at Discover
- Location
- San Diego, CA, US
- LinkedIn followers
- 500 followers
Experience
Sr Data Engineer
Aug 2024 — Present
As a Data Engineer, I designed and maintained scalable Azure-based data pipelines using Azure Data Factory and Azure Databricks to process large volumes of structured and semi-structured media and user engagement data from multiple source systems. I built modular, reusable, and cost-optimized workflows that ensured high performance, scalability, and maintainability across enterprise environments.I implemented complex PySpark transformation logic in Databricks, including deduplication, schema validation, null handling, data standardization, and quality enforcement to maintain enterprise data standards. I also developed Slowly Changing Dimension (SCD Type 2) processes to accurately track historical data changes and enable time-based analytical reporting.I designed and optimized Azure Synapse Analytics data warehouse models using star and snowflake schemas, improving query efficiency and supporting downstream BI reporting. Through advanced query tuning, indexing, and partition optimization, I enhanced performance for large-scale analytical workloads.I orchestrated end-to-end ETL/ELT workflows using ADF triggers and Apache Airflow, managing dependencies, failure handling, retries, and SLA-driven executions. I proactively monitored pipeline health, implemented alerting mechanisms, and resolved production issues to ensure reliable daily processing.Additionally, I collaborated closely with analytics, BI, and business stakeholders to translate reporting requirements into analytics-ready datasets. I delivered curated data layers that served as trusted sources for Power BI dashboards, enabling actionable insights into user engagement and media performance. I also contributed to platform best practices, including Git-based version control, documentation standards, and reusable framework development to improve team productivity and maintainability.
Education
KAKATIYA INSTITUTE OF TECHNOLOGY AND SCIENCE
Bachelor of Technology - BTech
2019 — 2023
University of North Texas G. Brint Ryan College of Business
Business Analytics
2023 — 2025
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.