Sai Kumar

Senior Data Engineer @TD

Toronto, ON, CA

Responsible for the execution of big data analytics, predictive analytics, and machine learning initiatives. Developed Scala scripts, UDF are using both data frames/SQL and RDD in Spark for data aggregation, queries and writing back into S3 bucket. Wrote, compiled, and executed programs as necessary using Apache Spark in Scala to perform ETL jobs with ingested data. Worked on Big data on AWS cloud services i.e, EC2, S3, EMR and DynamoDB Implemented End-to-end solution for hosting the web application on AWS cloud with integration to S3 buckets. Created complex ETL Azure Data Factory pipelines using mapping data flows with multiple Input/output transformations Worked on Azure BLOB and Data Lake storage and loading data into Azure SQ. Worked with Azure SQL Database Import and Export Service. Used Azure Key vault as central repository for maintaining secrets and referenced the secrets in Azure Data Factory and also in Databricks notebooks. Built a common SFTP download or upload framework using Azure Data Factory and Databricks. Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing. Involved in designing & deploying multi-tier applications using all AWS services (EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, IAM) focusing on high-availability, fault tolerance, and auto-scaling in AWS Cloud Formation Wrote Spark applications for data validation, cleansing, transformation, and custom aggregation and used Spark engine, Spark SQL for data analysis and provided to the data scientists for further analysis. Developed rest API\'s using python with flask and Django framework. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services -(Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks