Vivek K

About Vivek K

I am a Data Engineer with 8+ years of experience designing, building, and optimizing scalable data platforms and distributed data processing systems to support enterprise analytics, reporting, and machine learning workloads. I specialize in developing high-performance batch and real-time data pipelines that enable reliable ingestion, transformation, and processing of large-scale structured and unstructured datasets. My expertise includes big data technologies such as Apache Spark, PySpark, and the Hadoop ecosystem (HDFS, Hive, YARN), along with modern ETL/ELT frameworks like Apache Airflow, Azure Data Factory, Cloud Composer, and Oozie for orchestrating complex workflows. I have strong hands-on experience across multi-cloud environments including AWS, Azure, and Google Cloud Platform, where I have built scalable cloud-native data solutions using services such as S3, EMR, Lambda, DynamoDB, and Kinesis (AWS); Data Factory, Databricks, Synapse, and ADLS Gen2 (Azure); and BigQuery, Dataflow, Pub/Sub, Dataproc, and Cloud Storage (GCP). I am experienced in building real-time and streaming data pipelines using Apache Kafka, Spark Streaming, Pub/Sub, and Event Hubs to enable event-driven architectures and near real-time analytics. I have designed and implemented enterprise data lake and lakehouse architectures using Delta Lake and modern cloud storage platforms, as well as high-performance data warehouses using BigQuery, Redshift, and Azure Synapse. My core strengths include data modeling (star and snowflake schemas), Spark and SQL performance optimization, data ingestion frameworks (APIs, Sqoop, Flume, and streaming systems), working with formats such as Parquet, ORC, Avro, and JSON, implementing data quality and governance frameworks, enforcing security practices (IAM, RBAC, encryption), and applying CI/CD and DevOps practices using Terraform, Azure DevOps, and Git. I collaborate closely with data scientists, analysts, and business stakeholders to deliver scalable, reliable, and business-aligned data solutions that drive effective decision-making and analytics

Experience

1. Senior Aws Data Engineer

   BNY

   Apr 2025 — Present · Pittsburgh, PA, US

   Partnered with business stakeholders and product teams to define data requirements and translate them into scalable cloud data architecture solutions.Designed and built end-to-end data pipelines to ingest, transform, and deliver high-quality, analytics-ready data into enterprise data lakes and warehouses.Onboarded new data sources by performing data profiling, performance tuning, scalability validation, and security assessments.Developed and maintained serverless and event-driven architectures on AWS (Lambda, S3, Glue, Redshift, EMR, ECS, DynamoDB, SNS, CloudWatch) for scalable and cost-efficient data processing.Built robust ETL workflows using Informatica PowerCenter, Informatica Developer, and AWS Glue for enterprise-grade data integration.Optimized Oracle and SQL Server databases using indexing, partitioning, and query tuning to improve performance and efficiency.Automated deployment processes by implementing CI/CD pipelines using Jenkins and UrbanCode (UCD/UCR).Developed large-scale data processing solutions using Python, SQL, Apache Spark, and PySpark for distributed data transformation.Created interactive dashboards and reporting solutions using Tableau, OBIEE, and QlikView to enable self-service analytics and business insights.Conducted advanced data analysis and pattern discovery using Oracle Data Mining and Informatica tools to support data-driven decision-making.Designed logical and physical data models using ERwin and Visio aligned with enterprise data governance standards.Led Agile ceremonies, performed code reviews, and mentored junior engineers to drive team productivity and best practices.