Salim Khan

Senior Data Engineer @CVS Health

IL, US

Participated in the analysis, design, and development phase of the Software Development Lifecycle (SDLC) and worked in an agile environment with sprint planning, scrum calls, and retrospective meetings. Utilized JIRA for project management and GitHub for version control. Led a comprehensive data integration project utilizing Azure Data Factory (ADF), Databricks, and Snowflake to streamline data processing and analytics. Managed diverse data sources, including SQL databases, APIs, and on-premise databases, ensuring seamless data ingestion and integration. Developed and maintained non-production and production transformations in the Azure environment and created data pipelines using PySpark Programming, Spark, Hive, Pig, Python, Impala, and HBase. Created pipelines in ADF using Linked Services, Datasets, and Pipelines to extract, transform, and load data from sources like Azure SQL, Blob Storage, and Azure SQL Data Warehouse. Spearheaded the creation of PySpark scripts for data transformation and utilized Databricks for advanced data transformation, ensuring data quality and consistency. Implemented Spark Streaming to divide streaming data into batches as an input to the Spark engine for batch processing. Developed and deployed ETL packages using SQL Server Integration Services (SSIS) to support data migration and transformation processes. Created and managed complex SSIS packages to automate data extraction, transformation, and loading (ETL) from various data sources into data warehouses. Configured Snowflake to store metadata and configuration details for each data source, enhancing data management and governance. Implemented a multi-layered data architecture by creating silver and gold tables in Databricks and designed external tables in Snowflake for easy access to transformed data.