Srikanth Konne

About Srikanth Konne

Data Engineer | Data Analyst Grad I\'m Srikanth Konne, and I\'ve recently…

Experience

1. Data Engineer

   Marriott International

   Jul 2023 — Present · US

   Pioneered data pipelines for data ingestion from multiple sources (Workday, Oracle, Salesforce, third-party APIs) and loaded data into Snowflake data warehouse using Spark, Python, AWS (S3, EC2, Lambda, Glue), Snowpipe, and containerized the workflow using Docker fo r consistent deployment.• Designed AWS Cloud Formation templates setting up Auto Scaling for EC2 instances and involved in the automated provisioning of resources.• Involved in rigorous data pipeline unit testing, CI/CD, and orchestration of workflows utilizing AWS Step Functions and AROW, Airflow.• Built a serverless AWS Lambda solution to track hourly files landing on S3 and to trigger the data load of these files into inhouse DataLake when an appropriate threshold is met, by establishing connectivity to the Data Catalog API for schema validation and metadata management resulting in a 80% increase in data loading speed.• Engineered Python/Pyspark scripts to transform and load data of multiple formats (json, csv, dat, txt, xlsx, etc.) from S3 to Snowflake base tables.• Created monitoring alerts on null checks on important columns and data quality checks to cut down on redundant data using SnowSQL for each base entity. Migrated on-prem PostgreSQL data to Snowflake via AWS Glue in an incremental format leveraging Spark SQL API.• Led a successful Proof of Concept processing Google protobuf data streams using Spark Structured Streaming integrated with AWS Kinesis, Kinesis Firehose, and S3 ensuring low-latency analytics for dynamic datasets.