Surya Prakash
Senior Data Engineer at BNY | Hive | Python | Azure | Pyspark | Spark SQL | Azure Databrick| Hadoop | Snow flake| ETL | SQL | Airflow | Agile | Actively looking for new opportunities on C2C/C2H
- Role
- Senior Data Engineer at BNY
- Location
- Cleveland, OH, US
- LinkedIn followers
- 500 followers
About Surya Prakash
Data Engineer with over 5 years of experience in Hybrid Cloud environments (AWS…
Experience
Senior Data Engineer
Nov 2023 — Present · NY, US
Developed cloud migration strategy using AWS services like Database Migration Service and Server Migration Service.• Set up AWS infrastructure with VPC, EC2, S3, DynamoDB, IAM, EBS, Route53, SNS, SES, SQS, CloudWatch, CloudTrail, Auto Scaling, and RDS using CloudFormation templates.• Backed up AWS PostgreSQL to S3 via daily EMR jobs using DataFrames.• Implemented Kubernetes with Docker for auto-scaling and CI, deploying services through Kubernetes.• Used AWS Lambda to run servers without managing them, triggering code via S3 and SNS.• Migrated data to AWS Redshift, RDS, ELB, EMR, DynamoDB, and S3.• Created AWS infrastructure scripts with BASH and Python, including Lambda functions for S3 and DynamoDB.• Deployed and monitored scalable AWS infrastructure with Ansible, migrating data from the Data Center to AWS. Improved Python module to denormalize RDBMS data to JSON, saving 35 hours.• Automated metadata and lineage extraction with Python scripts, saving 70+ hours of manual effort.• Developed Python programs to handle PL/SQL functions unsupported by Snowflake.• Created job chains with Jenkins Job Builder, Parameterized Triggers, and target host deployments.• Orchestrated CI/CD processes using CloudFormation, Terraform, Packer, Docker, and OpenShift.• Created dashboards on the Snowflake cost model and usage in QlikView.• Enhanced Spark performance and optimization in Hadoop using Spark Context, Spark SQL, and Spark Yarn.• Integrated real-time monitoring for data ingestion by migrating logs to AWS CloudWatch, and configuring Alarms and Alerts.• Performed data ingestion into Snowflake warehouse, developing views for entry-level deduplication.• Configured Airflow connection to AWS EMR with secure VPCs and Subnets, developing bash scripts for EMR cluster initialization.• Defined and deployed Star Schema, Snowflake Schema, and Dimensional Data Modeling on an EDW.
Education
Bharath Institute of Higher Education and Research
Bachelor's degree, Computer Science
2016 — 2020
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.