Vamc Reddy

About Vamc Reddy

Computer System Validation (CSV) Engineer with over 5 years of experience in validating…

Experience

1. Senior Data Engineer

   Verizon

   Jan 2024 — Present

   Designed and developed scalable data pipelines using Azure Data Factory and Azure Databricks (PySpark/Scala) to ingest, transform, and load large volumes of structured and semi-structured data into ADLS Gen2 and Synapse Analytics.Developed reusable and modular ETL/ELT frameworks in Databricks, integrating Python, SQL, and Spark for transformation logic and leveraging Delta Lake for versioned, reliable data storage.• Applied industry best practices for metadata management and data quality using Azure Purview, ADF parameterization, and dynamic pipelines.• Collaborated with business and product owners in Agile teams to define data solutions and deliver new features, while mentoring junior engineers in Azure and Databricks usage.Built CICD pipelines using Azure DevOps and YAML to automate deployments of notebooks, datasets, linked services, and other infrastructure-as-code components.Integrated Azure services such as Key Vault, Event Hub, Blob Storage, and Synapse within Databricks workflows to support secure, scalable, and event-driven data operations.Utilized Databricks REST APIs and CLI for operational automation, including notebook deployment, job scheduling, and cluster lifecycle management.Performed detailed root cause analysis and resolution of data issues in Databricks notebooks, improving data accuracy and reducing recurring pipeline failures.Created reusable notebook templates in Databricks to standardize ingestion, transformation, and validation logic across multiple pipelines and teams.Monitored and tuned cluster performance in Databricks, adjusting autoscaling, worker types, and job parallelism to balance performance and cost.Configured Git integration with Azure Repos and GitHub within Databricks, enabling version control, collaboration, and seamless CI/CD with Azure DevOps pipelines.