Krishnavamsi Rangu

Azure Data Engineer @Wipro

IN

An Azure Data Engineer specializing in data extraction, ingestion and pipeline management. Ensure correctness and seamless data flow across various layers in Azure Databricks.*Key Responsibilities**Data Extraction and Ingestion- Extracts data from diverse sources using Azure Data Factory- Loads data into the Acquisition Layer(Landing Layer) in Azure Delta Lake.*Data Layer Management- Manages data progression from bronze layer to silver layer(cleaned & Enriched data) and to gold layer for analytics- Ensures data correctness and integrity throughout the layers- Ensures efficient and reliable data processing.*Pipeline Development and Management- Build and manage data pipelines using Azure Data Factory- Utilized Azure Devops for CI/CD. 1.Streamlined ETL pipelines to improve data ingestion and transformation process, reducing latency and enhancing reliability for downstream analytics.2.Developed and deployed Pyspark scripts to automate data quality checks, incorporating robust validation logic and alert mechanism to identify anomalies proactively.3.Optimized query performance and reduce runtime by implementing efficient indexing, partitioning and caching strategies for large datasets.4.Monitored and ensured high availability of data pipelines by setting up logging, monitoring and automated recovery processes.5.Collaborated with cross-functional teams to define and implement best practices and deliver scalable solutions that meet business requirements.Mastered the performance Tuning initiative that attained the queries to yield a 40% improvement in data processing speed.