Kanishk Karanawat

About Kanishk Karanawat

Strong educational background in Software engineering with Master of Science (M.S.)…

Experience

1. Staff Software Engineer, Data Platform

   Twitter

   Jan 2022 — Present

   Technical lead for the Data processing team, under Data Platform, responsible for building and managing big data processing frameworks at Twitter, for both batch and stream processing use-cases. Team is responsible for providing a paved path to data and machine learning engineers that allows them to seamlessly provision, author and orchestrate their data pipelines both on-premise (on HDFS) as well as cloud (Google Cloud Platform). Data processing stack includes Apache Beam that provides unified distributed data processing capabilities for both batch and stream processing. Dataflow runner, on Google Cloud Platform, is used to deploy the data pipeline jobs. Driving initiatives: 1. Data Access layer: Build abstraction to easily access datasets stored across different storage sinks (HDFS, GCS, BigQuery) and different serialization/compression disk formats. ElephantBird Format Deprecation: EBLZO format is one of the most common data formats, in Twitter, to store log information in HDFS/GCS. Seamlessly migrate to a more efficient container format, by switching to Avro + Zstandard format. This initiative saves around ~$10M in both OpEx and CapEx costs by reducing both storage as well as network bandwidth costs by 50%. This work impacts the majority of datasets across 50 different teams in the company and requires cross team collaboration with different services like log ingestion, replication and scrubbing services, to ensure codec changes are compatible. Reduce Pub/Sub related costs: Google Pub/Sub is used at Twitter, for stream processing pipelines, in GCP. Pub/Sub pricing model is based on the number of bytes ingested/consumed by producers/consumers. Enabled Zstandard based compression for messages published/consumed via client library that yields up to 50%-70% cost savings. Collaborated with the stream replication team that replicates data from Kafka to Pub/Sub to ensure format compatibility.

Education

• Harvard Business School

  Core: Credential of Readiness

  2014 — 2014

• National University of Singapore

  Bachelor's Degree, Electrical and Electronics Engineering

  2008 — 2012

• Brightlands School

  High School

  2006 — 2008

• Carnegie Mellon University

  Master’s Degree, Software Engineering

  2015 — 2016