Shalin Amin

About Shalin Amin

I have 5+ years of experience and expertise in the areas of Big Data and Software…

Experience

1. Technical Team Lead - Senior Sde (Client - Compass Real-estate)

   Tavant

   Oct 2020 — Present

   Leading Data Engineering and Listings Infrastructure team which is responsible for: • Working for Compass Real Estate as a technical team lead, on real estate data transformation using an in-house Python ETL proprietary framework. • Responsible for pulling data from RETS (Querying 100+ MLSs) and other Web API resources and performing data ingestion & transformation using Kafka and Spark streaming that handles over 1M messages per minute. • Possess hands-on experience in handling SQL and no-SQL databases like MongoDB, PostgreSQL for querying and validating JSONs. • Created Airflow DAGs to run on the web-server which speeded up the process of triggering and monitoring ingestion jobs without the need to manage them from EC2 instance using CLI. • Utilized various AWS services such as MSK, Lambda, S3, Parameter Store, Secret Manager and CloudWatch that are parts of data ingestion mechanism as well as tools such as Datadog to monitor the health of said ingestion and transformation mechanisms. • Deployed Python executable builds to development, staging and production instances using K8s, Docker and AWS Elastic Container Service. • Optimized efficiency of ingestion process by automating basic ETL operations using Python frameworks such as Git Python, Fabric and bash scripts as required.