e Likhitha

About e Likhitha

I have overall 7+ years of experience in IT Industry and in Data Engineering, I…

Experience

1. Sr Data Engineer

   Mayo Clinic

   Mar 2024 — Present · Rochester, MN, US

   Responsibilities-Designed and implemented complex ETL workflows on Azure Data Factory (ADF) for data ingestion, transformation, and loading from diverse sources within on-premises and cloud storages including EHRs, claims, and PROs utilizing PySpark clusters for scalability-Integrated Apache Kafka publish-subscribe messaging system as a source for streaming data from various sources and used Databricks jobs and workflows to schedule and automate ETL pipeline execution-Utilized Spark tools within Hadoop for complex data transformations, cleansing, and feature engineering needed for advanced analytics-Configured and utilized Apache Sqoop to load data with diverse data formats including CSV, JSON, Parquet, and Avro from Hadoop HDFS to Snowflake clusters for effective performance and scalability-Developed secured (using OAuth, API Keys) Restful APIs using Python to query data from the Snowflake warehouses-Designed and implemented customized IAM policies within Azure Active Directory (AAD) for role-based access control (RBAC) to data pipelines and resources-Packaged individual data processing tasks as Docker containers for scalability and managed container images and versions in Azure Container Registry (ACR) for efficient deployments-Utilized Jenkins as an alternative CI/CD server for building and deploying data pipelines and configured Jenkins plugins.