Get your Emails warmed up and ready to send
Build sender reputation and reach more inboxes with automated email warmup.
Warm up your emails
Starting at $49/month
Mike D.
Lead Aws Data Engineer @Kaiser Permanente
Signup · Get unlimited contacts
WORK HISTORY
Lead Aws Data Engineer @Kaiser Permanente
Los Angeles, CA, US
Lead multiple data engineering teams in designing data platform.• Collaborated with the Product Owners in understanding the requirement and design a data flow system. • Built out a Data Lake on Databricks using Databricks Delta tables stored in AWS S3. • For real-time processing used Kafka and Spark Structured Streaming. • Used AWS Lambda for event driven processing. • For batch-processing used Spark/Scala based jobs with Airflow Orchestration • Used DataFrame, Dataset & Spark SQL APIs with Spark & Scala for data transformation. • Optimized Spark based jobs by partitioning, bucketing, z-ordering the delta tables. • Optimized AWS spend by using EC2 spot instances for processing and S3 life cycle for data storage. • Spark jobs merged data to Snowflake and Elasticsearch. • Used Elasticsearch for search capability of the marketplace portal. • Used Snowflake Marketplace to share datasets with external clients. • Used Datadog and Cloud Watch for monitoring and logging. • Used SAML, AWS IAM role and Databricks SSO feature to manage governance and access control to Data Lake. • Used GIT for version control • Used Azure DevOps for automation/CICD pipeline
EDUCATION
Stanford University
Bachelor of Technology - BTech, Information Technology
ABOUT MIKE D.
Technology Stack : Front End: JavaScript, HTML, CSS, Python and Shell…
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.