Sumathi M

About Sumathi M

Highly skilled and dedicated data engineer with experience in designing, implementing, and optimizing data infrastructure and solutions, particularly on the Azure platform. Proven track record of consistently contributing to the successful development and deployment of complex data-driven applications. I excel in leveraging various technologies and tools to extract valuable insights from vast and diverse datasets, driving data-informed decision-making processes within organizations. I work on building and deploying scalable and secure Big Data solutions using Azure, Hadoop, Spark, and other technologies. I have experience in developing and implementing streaming applications in Scala and Python, integrating Kafka, Spark Streaming, and HDFS/HBase, to ingest and process data from various sources. I possess excellent communication and interpersonal skills, enabling me to effectively convey complex technical concepts to both technical and non-technical stakeholders. In summary, I am a results-oriented data engineer with a proven ability to drive the development and optimization of data infrastructure. With my comprehensive skill set, industry experience, and commitment to excellence, I am poised to make valuable contributions to any organization\'s data engineering initiatives.

Experience

1. Senior Data Engineer

   Tiger Analytics

   Jan 2021 — Present · Dallas-Fort Worth, TX, US

   Worked on Delta Lake to read and write the large sets of data which is on top of Azure Data Lake Store (ADLS Zen2)• Built the frameworks and modules to process large data sets using the concepts of spark SQL, data frames and load them into use case extraction layer, develop the complex queries including aggregation grouping logic and generate the final data sets which are consumed by the data scientists.• Performed the data analysis and profile the data using SQL, Spark SQL and other cloud technologies to resolve the business-related issues for product owners and data scientists. The data consumed for this analysis is vast, in order to meet the performance requirements, the code is developed in Spark and analysis needs to be performed at multiple hierarchy levels.• Worked on segmentation and cleaning of various data sources related to the Model for better data analysis and to increase the final output performance of the model.• Analyzed the data in ADLS and DB2 to come up with the missing data sources in ADLS and helped solution architect in gathering the requirements.• Worked on Visualizing the final model results in Power BI by connecting the Power BI to ADLS and import the model output tables.• Worked on Power BI with custom visuals like ESRI Map visualization to show the model output and use DAX code to calculate the additional measures in Power BI as when required.• Designed and built ETL pipelines to automate ingestion of structured and unstructured data. • Migrated the existing data from Mainframes/Teradata/SQL server and perform ETL operations on it.• Developed Spark applications for ETL transformations using python. • Used Azure Databricks to run Spark applications and Azure Data Factory to perform ETL as part of pre-processing.• Created new jobs in Control-M on Windows. Passing parameters to Control-M job, creating job dependencies and alerts as per application team and business requirement.

Education

• Sree Vidyanikethan Degree College, A.Rangampet, Tirupati.

  Bachelor of Technology - BTech

  2014 — 2018

• Eastern Illinois University

  Master of Science - MS

  2019 — 2020