Pavani P

Data Engineer @SC Johnson

US

Designed and implemented a scalable ETL pipeline for Business Intelligence using Apache Kafka for real-time data streaming and Databricks (PySpark) for distributed data processing, efficiently handling 2 million records. Optimized complex SQL and SparkSQL queries, reducing report generation time by 40% and enhancing query performance for large-scale datasets- Built a centralized data lake on Azure Data Lake and integrated it with Azure Synapse Analytics, managing more than 5 TB of structured and unstructured data, improving accessibility and enabling more efficient reporting for stakeholders- Developed and deployed real-time, interactive dashboards using Power BI, driving a 30% increase in data-driven decision-making across business teams- Reduced data latency by 35% by leveraging Apache Kafka for high-throughput data ingestion and Databricks (PySpark) for scalable distributed processing- Utilized Azure Functions and Azure SQL Database for serverless computing and cost-efficient querying, cutting infrastructure costs by 25%- Implemented robust data governance practices, including data quality checks and data lineage tracking, ensuring data reliability and accuracy across all pipelines- Collaborated with cross-functional teams to adopt Azure DevOps for continuous integration and deployment of data solutions.

MADANAPALLE INSTITUTE OF TECHNOLOGY & SCIENCE

Bachelor of Technology - BTech

Kennesaw State University

Master of Science in Computer Science