Sai Koushik

Azure Data Engineer @HealthStream

Collaborated on requirements gathering and played a key role in designing and developing data pipelines. • Led time-sensitive data migration of multi-state datasets from SQL Server to Snowflake using Python and Snow SQL. • Built ETL data pipelines in Azure Data Factory to ingest data from diverse sources, including SharePoint, SQL Server, GCS Buckets, DB2, and Azure Data Lake Storage (ADLS). • Utilized PySpark scripts in Databricks notebooks for efficient data ingestion and transformation in Azure Data Lake Storage. • Developed and maintained CI/CD pipelines for Big Data solutions using Azure DevOps, ensuring seamless deployment. • Created pipelines and triggers in Azure Data Factory for extracting and ingesting data into Blob Storage and Azure Data Lake Storage Gen2. • Deployed Kubernetes clusters to orchestrate containerized applications, ensuring high availability and scalability. • Automated deployments to Kubernetes clusters by developing CI/CD pipelines with Jenkins and GitLab. • Designed and deployed data pipelines in Talend on Azure, optimizing data processing performance. • Conducted performance tuning of T-SQL and Snowflake queries for enhanced data processing efficiency. • Developed custom ETL solutions using PySpark and Shell scripting for efficient data movement in and out of the Hadoop cluster. • Built Azure Databricks notebooks to extract data from Data Lake and Blob Storage, loading it into on-premises SQL Server databases• Performed statistical analysis using SQL, Python, and Excel, leveraging Excel VBA Macros and Microsoft Access Forms. • Built ETL solutions in Databricks, executing code in notebooks against data stored in Data Lake and Delta Lake• Used Azure Data Factory to orchestrate Databricks data preparation and load processed data into SQL Data Warehouse.