Sahithi Palumari

Sr Data Engineer @UnityPoint Health

IA, US

Utilizing Azure Data Factory, data ingestion pipelines were designed and put into place, enabling the smooth integration of various data sources into Azure Blob Storage, including SQL databases, Oracle databases, CSV files, and REST APIs.• Developed data processing workflows with Spark for distributed data processing and transformation tasks using Azure Databricks.• Created data processing workflows with Azure Databricks, utilizing Spark for tasks involving distributed data transformation and processing.• Azure Data Factory and Databricks were used to implement data quality checks and data cleansing procedures to guarantee the accuracy and integrity of the data throughout the pipeline.• Designed end-to-end ETL data pipelines, guaranteeing their scalability and seamless operation. Using linked services to link on-premises and cloud data sources, this involved making extensive use of copy activity for data movement and lookup activity for data validation.• Utilizing SQL queries and ADLS Gen2 for scalable storage solutions, optimized query and indexing strategies were put into place to improve the efficiency of data fetching.• Snowflake and Azure cloud services were integrated to create a safe and effective data warehousing system that allowed stakeholders to produce informative reports for strategic marketing analysis.• Kafka and Spark Streaming were used to design and implement real-time data processing solutions that allowed for the ingestion, transformation, and analysis of large amounts of streaming data.• PySpark was integrated with Azure Blob Storage and Azure Data Factory to process and ingest data in a seamless manner, facilitating interoperability and data integration throughout the Azure ecosystem.• Azure Data Factory pipelines and Azure Logic Apps were used to automate data discovery and classification tasks in Azure Purview, decreasing manual labor and boosting productivity.

Narsimha Reddy Engineering College

Bachelor's degree