Lohith Chowdary

About Lohith Chowdary

Big Data engineer

Experience

1. Data Engineer

   SEPHORA

   Feb 2021 — Present · San Francisco, CA, US

   1) Created Source to Target Mappings (STM) for the required tables by understanding the businessrequirements for the reports.2)Developed PySpark and SparkSQL code to process the data in Apache Spark3) Written multiple MapReduce programs in Python for data extraction, transformation andaggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.4) Used Python extensively for in built functions and Multi-Threading in Pyspark scripts.5) Creating email alerts to notify the stake holders for different sources when average thresholdreached with PySpark.6) Used Spark for Parallel data processing and better performances.7) Working with Data Science team for data model analysis and design.8) Monitoring the Partitions and logs in Production on daily basis and back filling the missingpartitions by running the jobs manually.9) Monitoring the Explore and Final Hadoop clusters part of GCP support.10) Meet with business and engineering teams on a regular basis to keep the requirements in sync anddeliver on the requirements.10) Used Jira as an agile tool to keep track of the stories that were worked on using the Agilemethodology.11) Attending Scrum meetings with the team on weekly basis.12) Used HiveQL UDF’s to analysis semi-structure data.13) Worked on JSON scripts for integration pipeline.14) Worked on GCP call support.15) Debugged and fixed the tables issue in explore node