Andrew P. S.

About Andrew P. S.

Efficiency/reliability expert. If it’s crunch time, I’m your go to. I’ll help optimize…

Experience

1. Senior Data Engineer

   Meta

   Feb 2019 — Present

   Migrated 2.5 million cpu-days and 440 PB worth of daily data warehouse workload from Hive from Data Warehouse to Spark and/or Presto. Saved $20mil/year in efficiency gains.* Ascended to Efficiency Ninja, helping manage a program of data efficiency across the company. Held regular office hours unblocking almost 100 teammate’s projects helping find ways out of stifling performance issues. Helped automate the detection of, and helped automatically or assist in fixing thousands of inefficient pipelines, saving $53mil/year in 2021 * Trusted to run a long-term project focused on operating and improving the single-most heavily used table in Meta\'s data warehouse, a privacy-related user mapping table critical for all user queries. Improved its landing time by two hours a day, saving pipeline-hours per day across the entire warehouse This led to improvements in job orchestration, saving $37 million/year in resource savings from fewer failed jobs across the whole warehouse.* Owned the data infra or “di” namespace, and virtually eliminated administrative cost due to over-quota issues, optimizing or removing the biggest reasons for instability.* Leading project to migrate all of Meta to use only industry standard ANSI-SQL across all batch jobs, in hopes to save many thousands of employee-hours, removing the need to understand subtle differences between HiveQL and regular SQL.