Sai Kiran
Senior Data Engineer @Gap
Signup · Get unlimited contacts
WORK HISTORY
Senior Data Engineer @Gap
Participated in the analysis, design, and development phase of the Software Development Lifecycle (SDLC). Experience in the agile environment, used to have sprint planning meeting, scrum calls and retro meetings for every sprint, Used JIRA for the project management and GitHub for the version control. Designed and implemented real-time data processing pipelines using Apache Kafka, Apache Spark, and Spark Streaming to process and analyze large volumes of data. Developed and maintained HBase clusters on Hortonworks and Cloudera distributed platforms for storing large scale unstructured data. Worked on various Azure services such as Azure Data Lake, Data Factory, Azure Databricks, and Azure SQL Database to store and process data. Utilized NoSQL databases such as MongoDB and HBase for data storage and retrieval. Designed and deployed data pipelines using Azure cloud platform (HDInsight, Data Lake, Databricks, Blob Storage, Data Factory, Synapse, SQL, SQL DB, DWH, and Data Storage Explorer). Built SCD Type 2 Dimensions and Facts using Delta Lake and Databricks. Involved in Data mapping specifications to create and execute detailed system test plans. The data mapping specifies what data will be extracted from an internal data warehouse, transformed, and sent to an external entity. Responsible for analyzing various data sources such as flat files, ASCII Data, EBCDIC Data, Relational Data (Oracle, DB2 UDB, MS SQL Server) from various heterogeneous data sources. Developed custom-built ETL solution, batch processing, and real-time data ingestion pipeline to move data in and out of the Hadoop cluster using PySpark and Shell Scripting. Created Azure Databricks (Spark) notebook to extract the data from Data Lake storage accounts and Blob storage accounts to load on premises SQL server database. Performed statistical analysis using SQL, Python, R Programming and Excel. Worked extensively with Excel VBA Macros, Microsoft Access Forms
EDUCATION
Jawaharlal Nehru Technological University
Bachelor’s computer science and engineering, Computer Science
ABOUT SAI KIRAN
Motivated Data Engineer having 8+ years of professional experience in Data Engineering,…
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.