Sri Lekha
Azure Data Engineer @TELUS
Signup · Get unlimited contacts
WORK HISTORY
Azure Data Engineer @TELUS
Toronto, ON, CA
Involved in understanding the requirements of the End Users/Business Analysts and Developed Strategies for ETL processes. • Performed ETL on data from different source systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services -(Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks. • Performed Data Aggregation, Validation, and Azure HDInsight using Hive Scripts.• Created partitioned tables in Hive, also designed a data warehouse using Hive external tables, and also created hive queries for analysis.• Implemented data ingestion from various source systems using Azure Data factory, Sqoop, and Spark.• Hands-on experience implementing Spark and Hive jobs performance tuning.• Performed monitoring and management of the Hadoop cluster by using Azure HDInsight.• Involved in extraction, transformation and loading of data directly from different source systems (flat files/Excel/Oracle/SQL) using SAS/SQL, SAS/macros. • Generated PL/SQL scripts for data manipulation, validation, and materialized views for remote instances. • Created and modified several database objects such as Tables, Views, Indexes, Constraints, Stored procedures, Packages, Functions and Triggers using SQL and PL/SQL. • Created large datasets by combining individual datasets using various inner and outer joins in Spark SQL and dataset sorting and merging techniques. • Wrote Pyspark scripts to parse XML, and JSON files and load the data in the database. • Performed File system management and monitoring on Hadoop log files.• Used Spark API over Hadoop YARN to perform analytics on data in Hive.• Used Spark SQL to process a huge amount of structured data to aid in better analysis for our business teams.
EDUCATION
Saveetha School of Engineering
Bachelor of Technology - BTech
Sri Chaitanya College of Education
INTERMEDIATE
Dr KKR’S Gowtham International School
10
ABOUT SRI LEKHA
5+ years of IT experience in Analysis, Design, and Development in Big Data technologies like Spark, MapReduce, Hive, Yarn, HDFS, and Azure cloud services including programming languages like Python, and Pyspark with 3+ years of experience in Azure Data engineer. • In-depth knowledge in working with Distributed Computing Systems and parallel processing techniques to efficiently deal with Big Data. • Experience with Python, and SQL on Azure cloud platform, a better understanding of Data Warehouses like Azure Synapse and Azure Data-bricks platform, etc. • Understanding of Hadoop architecture and various components including HDFS, Yarn, MapReduce, Hive, Pig, HBase, Kafka, Oozie, etc, • Strong experience building Spark applications using Python as a programming language. • Good experience troubleshooting and fine-tuning long-running spark applications. • Strong experience using Spark RDD API, Spark Data frame/Dataset API, Spark-SQL, and Spark ML frameworks for building end-to-end data pipelines. • Good experience working with real-time streaming pipelines using Kafka and Spark-Streaming. • Strong experience working with Hive for performing various data analyses. • Good experience in automating end-to-end data pipelines using the Oozie workflow orchestrator. • Worked on Docker-based containers for using Airflow. • Expertise in configuring and installation of PostgreSQL, Postgresplus advanced Server on OLTP to OLAP systems from high-end to low-end environments. • Detailed exposure to Azure tools such as Azure Data Lake, Azure Data Bricks, Azure Data Factory, HDInsight, Azure SQL Server,zz
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.