Kiran D

About Kiran D

Data Engineering professional with 15+ years of experience designing and building…

Experience

1. Senior Data Engineer

   Prime Video & Amazon MGM Studios

   Jun 2024 — Present · Culver City, CA, US

   Design, develop, and maintain scalable and robust data pipelines, ETL/ELT processes to collect, process, and store large volumes of data using Snowflake, AWS, Informatica Cloud (IICS/IDMC), and Python.• Leverage AWS data services such as EC2, Lambda, Dynamo DB, SNS, SQS, RDS, DMS, S3, PostgreSQL, Glue, and Athena to manage and optimize data storage, processing, and retrieval.• Develop and maintain data pipelines, data storage solutions, data processing and data integration using AWS Glue, DBT and Snowflake. • Create and Test DBT models and transformations.• Develop data pipelines to handle and load real-time streaming data from Kafka, and batch processing data from RDBMS to Snowflake data warehouse.• Build ETL processes using Snowflake\'s features such as SnowPipes, SnowSQL, Streams, Procedures, Tasks, Dynamic Tables, Views, Zero Copy Cloning and Time-Travel.• Create Snowflake external stages for AWS S3 buckets and file formats for handling semi-structured data for CSV, JSON, and PARQUET files.• Build auto data ingestion processes using SnowPipes, SnowSQL (COPY INTO) to load files from external stages such as AWS S3 buckets into Snowflake Cloud.• Create Snowflake streams to capture the changes to implement the SCD Type 2 and automate it using Tasks.• Write Python and Shell scripts to connect, fetch the data from Oracle, and transfer it to AWS S3 for Snowflake consumption.• Develop Informatica Cloud (IDMC/IICS) task flows and mappings to integrate data from on-prem and cloud.• Write SQL queries using Joins, CTEs, and Windows or Analytical Functions to improve query performance.• Deploy scripts using CI/CD pipelines and DevOps tools such as Jenkins, Git, and Bitbucket.• Optimize the performance of the Snowflake data warehouse with Query Profiler, caching, and scaling.• Optimize ETL processes and SQL queries to retrieve, analyze, and transform data efficiently.• Migrate 2 TB of legacy data to the Snowflake Data Platform, ensuring 99.9% data accuracy.