Pravalika Gonegari

Senior Data Engineer @American Express

Phoenix, AZ, US

Design, develop and maintain non-production and production transformations in Azure environment and create the data pipelines using PySpark Programming, using Spark, Hive, Pig, python, Impala, and HBase to ingest customer DB Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing. Wrote Spark applications for data validation, cleansing, transformation, and custom aggregation and used Spark engine, Spark SQL for data analysis and provided to the data scientists for further analysis. Developed rest API\'s using python with flask and Django framework and done the integration of various data sources including Java, JDBC, RDBMS, Shell Scripting, Spreadsheets, and Text files. Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks, Delta Lake and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory. Worked on data migration to Hadoop and hive query optimization. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services -(Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. Used MongoDB to stored data in JSON format and developed and tested many features of dashboard using Python, Bootstrap, CSS, and JavaScript Install and configure Apache Airflow for Snowflake data warehouse and created dags to run the Airflow. Worked with PowerShell and UNIX scripts for file transfer, emailing and other file related tasks. Created a new data model that embeds NoSQL submodules within a relational data model by applying Hybrid data modelling concepts.

Jawaharlal Nehru Technological University

Bachelors, Information Technology