Sandeep K

Azure Data Engineer @Cigna Healthcare

Dallas, TX, US

Configured event-driven triggers and scheduling mechanisms to automate data pipelines and workflows.• Implemented solutions for data lineage and metadata management to track and monitor data flow and transformations.• Identified and resolved performance bottlenecks in data processing and storage layers, optimizing query execution and reducing data latency.• Implemented strategies for partitioning, indexing, and caching in Snowflake and Azure services to improve query performance and reduce processing time.• Conducted performance tuning exercises and capacity planning to ensure scalability and efficiency of the data infrastructure.• Developed a CI/CD framework for data pipelines using the Jenkins tool.• Collaborated with DevOps engineers to build automated CI/CD and test-driven development pipelines using Azure, meeting client requirements.• Hands-on experience in programming with scripting languages like Python and Scala.• Executed Hive scripts through Hive on Spark and SparkSQL, contributing to ETL tasks, data integrity maintenance, and pipeline stability verification.• Utilized Kafka and Spark Streaming for processing streaming data in specific use cases.• Designed and implemented a data pipeline using Kafka, Spark, and Hive for ingestion, transformation, and data analysis.• Developed Spark core and Spark SQL scripts using Scala to optimize data processing speed.• Utilized JIRA for project reporting, creating sub-tasks for development, QA, and partner validation.• Experienced in Agile methodologies, participating in daily stand-ups and internationally coordinated PI Planning.• Designed and implemented scalable data ingestion pipelines using Azure Data Factory, extracting data from various sources such as SQL databases, CSV files, and REST APIs.• Developed data processing workflows using Azure Databricks, leveraging the distributed data processing and transformation capabilities of Spark.