Rishi Reddy

Senior Data Engineer @Evolent

Arlington, VA, US

Involved in building a data warehouse on Azure using Data Factory, Databricks, and SQL Serverless• Designed and developed ETL pipelines using Data Factory to ingest data from multiple sources into Azure Data Lake• Built dynamic data pipelines to process multiple tables and files in the same pipeline using Watermark Table, parameters, variables, and various activities with Azure Data Factory• Implemented Slowly Changing Dimensions (Type-2) in delta lake using PySpark.• Worked on performance optimization to increase Databricks performance by using proper joins, avoiding shuffles, salting, partitioning, bucketing, and using Z-Order clustering.• Involved in processing and flattening JSON data using Mapping Data Flows, and Pyspark.• Actively involved in monitoring and support of Data Factory pipelines and Databricks notebooks.• Designed and developed SSIS Packages to extract data from various data sources such as Access database, Excel spreadsheet and flat files into SQL server for further Data Analysis and Reporting by using multiple transformations provided by SSIS such as Data Conversion, Conditional Split, merge, union all and lookup transformation and send mail task. • Worked as a BI developer with following responsibilities• Maintained and optimized existing databases, monitored database performances and growth, performance tuning,• Monitoring nightly jobs and investigate if there is any failure and fix the problem before the next run time.• Daily Digest Email Project Process, to send automatically notifications about a job failures and successes• Writing complex queries and Stored Procedures• Creating a working plan to migrate to the cloud environment: Azure • Creating complex SSIS packages to dynamically send required data elements to multiple Research Partners• Creating complex SSIS packages, including script tasks (C#) to dynamically import thousands of files to the database

The University of Texas at Arlington

Master's degree