Arief Ahmed Shaik
Data Engineer @M&T Bank
Signup · Get unlimited contacts
WORK HISTORY
Data Engineer @M&T Bank
Architected a cross-cloud PySpark pipeline in Databricks to transform data from Azure Delta Lake and load it into AWS S3. Re-engineered complex Axiom logic in Databricks using PySpark, reducing execution time by 66% and improving maintainability. Boosted Databricks pipeline efficiency by 60% using broadcast joins, caching, coalesce, and repartitioning optimizations. Automated Axiom pipelines using Automic, APIs, and scripts, reducing manual effort and enhancing operational efficiency. Implemented Medallion architecture on ADLS Gen2 using Delta Lake to streamline data flow across bronze, silver, and gold layers. Leveraged Delta Lake features like Time Travel and Optimize to support historical audits and accelerate query execution. Utilized Delta Lake’s Time Travel and Optimize features to enable historical data audits and accelerate query performance by 20%. Created and managed Delta tables in Databricks for batch and streaming workloads, enabling near real-time analytics.
EDUCATION
University college of Engineering, Osmania University
Bachelor of Engineering (B.E), Computer Science
University of Oklahoma
Masters in Computer Science
University of Oklahoma
Master of Science, Computer Science
SKILLS
ABOUT ARIEF AHMED SHAIK
Experienced Data Engineer with 12+ years in designing and optimizing scalable data pipelines, specializing in migrating legacy ETL processes to modern cloud platforms like Databricks and Azure. Strong background in the financial domain, focusing on regulatory reporting using Axiom (Adenza) tools. Skilled in PySpark, Delta Lake, Azure Data Factory, and cross-cloud data integration, adept at translating complex SQL and Axiom logic into efficient Spark-based solutions, driving performance improvements and enabling real-time analytics.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.