Sanath Pabba
Senior Data Engineer @CPP Investments | Investissements RPC
Signup · Get unlimited contacts
WORK HISTORY
Senior Data Engineer @CPP Investments | Investissements RPC
Toronto, ON, CA
Major contribution on successful migration of two projects: market data in PySpark from on-premise system to cloud (Databricks & AWS) and investment framework python app to AWS cloud. For market data project, set up various checks for the batch process that were key to Production sign off and go-live. Following are some of the key components- Wrote code to perform sanity checks on data generated that included a statistical model which was the core component to ensure data quality - Developed file watcher code to monitor and track arrival of critical files in AWS S3 bucket from upstream system - Set up other critical notifications and PagerDuty to support daily batch runs and maintain SLA apart from regular functional tasks- Key contributor for building a standalone scheduler for scheduling hundreds of databricks jobs in cloud using AWS lambda, step function, SNS and SQS deployed using Terraform code (Infrastructure as Code). TPIF app migration to cloud using a variety of AWS services for data migration, lambda, SQS, SNS, S3, aurora RDS, ECS, fargate and ALB, infrastructures are created using Terraform code- Key contributor to the success of an enterprise wide project, to create a top down return and exposure attribution for the total fund- Set up Jenkins Pipeline to build and deploy database queries to Amazon Aurora using liquidbase between environments in different AWS accounts- Working with Total Portfolio Management on Portfolio Investment Framework and Balancing process- Exposed to other technologies like Python (Flask, SQLAlchemy, Pandas), Database - Oracle, PostgreSQL, Versioning - Github, Scheduling tools - Tidal and Jams, Project management tools - Jira, ServiceNow, Reporting tool - Tableau- Have great experience in handling and onboarding various vendors data like S&P, Bloomberg, Barclays, Haver, rimes, murexTools: AWS S3, Athena, Glue, lambda, step function, cloud watch, airflow, databricks, EMR, EC2,JIRA, Confluence
EDUCATION
Loyalist College
Postgraduate Degree, Project Management
Birla Institute of Technology and Science, Pilani
Master of Business Administration - MBA, Business Analytics
Centre for Development of Advanced Computing (C-DAC)
Postgraduate Degree, Bigdata Analytics
International Institute of Information Technology Hyderabad (IIITH)
Artificial Intelligence and Machine learning
Jawaharlal Nehru Technological University
Bachelor of Technology - BTech, Electronics & Computer science
ABOUT SANATH PABBA
Senior Big Data Engineer | AI Enthusiast | Project Management ProfessionalWith over 10 years of expertise in development, maintenance, and production support in data management, I bring a proven track record of handling and analyzing large-scale datasets, including 2+ petabytes, to derive actionable insights and drive system efficiency.Technical ProficiencyI specialize in Big Data tools within the Hadoop Ecosystem, including Pig, Hive, HDFS, MapReduce, Sqoop, Storm, Spark, Scala, Kafka, Yarn, Presto, Oozie, and Zookeeper. I have extensive experience with ETL tools like Informatica, Snowflake, and Databricks, as well as strong knowledge of NoSQL databases, including Cassandra, HBase, and MongoDB.Proficient in cloud platforms, including:AWS: S3, EC2, EMR, Redshift, Athena, Glacier, GlueAzure: Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure Blob Storage, Azure Event Hubs, Azure Functions, and Azure Machine LearningGoogle Cloud Platform, leveraging these for scalable and efficient data solutions.My data visualization skills include MS Excel, Alteryx, Tableau, and Power BI, enabling me to deliver meaningful insights from complex datasets.AI and Advanced AnalyticsI integrate AI-driven models for predictive analytics and intelligent decision-making, leveraging machine learning frameworks such as TensorFlow and PyTorch. My expertise includes Natural Language Processing (NLP) for analyzing unstructured data and building chatbots, as well as applying deep learning techniques for forecasting and classification.Core SkillsI excel in Python programming, data scripting, and workflows involving AI integration. My orchestration skills include Dockers, Kubernetes, and Amazon EKS for streamlined deployments, and I have extensive experience with Flume and Kafka for real-time stream processing. With a deep understanding of ETL, OLAP/OLTP, and Hadoop distributions like Cloudera, MapR, HortonWorks, and Mesos, I design robust data pipelines and distributed systems.Problem Solving & LeadershipI am recognized for designing and developing Big Data applications in Spark using Scala, improving performance compared to Hive and SQL/Oracle. With a focus on quick and efficient solutions, I have led critical projects while collaborating directly with clients to meet complex requirements. My ability to thrive in Agile environments and manage cross-functional teams ensures consistent delivery of high-quality outcomes.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.