Deepak Sharma

About Deepak Sharma

Team-oriented Senior Cloud Data Engineer with technical hands-on and lead experience. Good understanding of designing large data warehouse systems, including relational and dimensional data models. Successful in implementing Cloud (AWS) and On-Premise (Cloudera) technologies, including data-pipeline and data-lake. Key Activities/Highlights : Implemented on-premise Cloudera data-lake and secure data-pipelines in AWS utilizing Terraform for infrastructure as code- AWS • Collaborated in the design of secure data-lake solution, and led the implementation using a staged multi-zone approach from multiple source systems. • Leveraged AWS services for the pipelines including Glue ETL/Crawler/DataCatalog, Lambda, Step-functions, Lake formation, CloudWatch, Athena, Redshift and QuickSight • Developed the infrastructure using Terraform, Glue, Lambda, PySpark and DBT (Data build tool)- Cloudera • Built on-premise Cloudera (CDP) clusters including OS and cluster software installation on multiple Linux servers. Designed and implemented the data-lake using a secure multi-zone architecture • Enabled authentication (Kerberos) for securing the cluster for valid access. Enabled security for data in motion using trusted certificates implementation (TLS/SSL) • Designed data-lake zones, to support raw, trusted and refined data, enforcing security for data at rest (HDFS encryption) by applications and zones • Utilized Apache Atlas to provide the metadata layer for the ingested data, with an option to provide business taxonomy for schema attributes and provide visual lineage of loaded data • Implemented policies through Apache Ranger for data access at Hive and HDFS level including auditing for policy violations. Managed the encryption keys for data-lake zones through Ranger - Enterprise Data Warehouse • Orchestrated and successfully transitioned platform migration of 100TB+ EDW to lower-cost compute/storage platform within 24-hour production outage window, resulting in $M\'s maintenance and support savings • Instrumental in designing processes to bulk-load huge amount of data leveraging ETL and database tools, resulting in 70% better run time, 80% reduction in CPU, and 60% data compression • Instituted RMAN Image backup method for 100TB+ database, enabling restores within SLA and avoiding $Ms in server/storage spend for disaster recovery • Organized EDW storage migration; restored Image backup for first time and tested DR capabilities; saved $K\'s in purchase of 3rd-party data migration tools and related labor costs

Experience

1. Senior Data Engineer

   RVO Health

   Sep 2023 — Present · US

Education

• Harvard Business School

  Program for Leadership Development, Human Resources Management/Personnel Administration, General

• Institute of Engineering & Technology

  Bachelor of Technology - B. Tech., Computer Science

  1984 — 1988

• St. George's College, Mussoorie, INDIA

  High School, Science

  1976 — 1983