Rahul Patel

About Rahul Patel

Seasoned Sr. Azure Data Engineer with 10+ years of expertise in designing and implementing efficient, accurate, and scalable data-driven solutions. Proficient in utilizing Azure Cloud services such as Azure Data Factory, Databricks, Azure Logic Apps, Azure Functions, Snowflake, Cosmos DB and Azure Synapse to streamline data processing and integration. Skilled in building and optimizing ETL pipelines using Azure Data Factory, PySpark, Spark SQL, and Polybase. Demonstrated experience in managing data migration across Azure Data Lake and ADLS Gen2 components to support robust and reliable analytics platforms. Looking for New Data Engineer Opportunities and can relocate anywhere in USA

Experience

1. Azure Data Engineer

   Wells Fargo

   Nov 2023 — Present · Charlotte, NC, US

   Enhanced Spark performance by fine-tuning partitioning, caching, and optimizing SparkSQL queries, reducing job completion time by 35%- Designed and implemented data integration solutions using Apache Kafka, Apache NiFi, and Azure Data Factory for seamless data ingestion from various sources- Orchestrated data ingestion into Azure Data Lake and SQL Database using Data Factory, centralizing data storage for analytics- Leveraged Microsoft Azure services including HDInsight, BLOB, Data Factory, Logic Apps, and conducted POCs on Azure Databricks- Conducted ETL processes using Azure Databricks, migrating Oracle ETL workflows to Azure Synapse Analytics for better scalability- Spearheaded migration of SQL databases to Azure Data Lake, SQL Database, and Synapse, ensuring smooth data transfer and integration- Integrated on-prem systems to Azure cloud storage (Blob, SQL Database) using Data Factory and Databricks for real-time analytics- Used PolyBase for efficient data loading into Azure Synapse Analytics, reducing load times by 50%- Leveraged Azure SQL Database Import/Export Service for seamless data integration- Utilized Azure Key Vault to manage secrets, referenced in Data Factory and Databricks- Engineered Databricks ETL pipelines using notebooks, Spark DataFrames, SQL, and Python- Established Databricks Job workflows to extract data from SQL servers and upload files to SFTP using PySpark and Python- Collaborated with DevOps to implement CI/CD pipelines, improving pipeline stability and reducing manual intervention- Processed schema-oriented and non-schema-oriented data using Scala and Spark- Implemented advanced Spark scripts with Scala and SQL for accelerated data processing using Hive tables- Leveraged Spark Streaming to partition data into batches for optimized processing- Utilized JIRA for issue/project management and Git for version control.