Niharika Thiruveedula

Aws Data Engineer @Citi Bank

TX, US

Installed and configured multi-node AWS EC2 clusters and managed services using AWS CloudWatch and CloudTrail.Stored logs and sensitive data in versioned AWS S3 buckets; automated snapshot creation with Python.Integrated AWS DynamoDB and Lambda to capture and backup stream data.Designed data warehouses on AWS Redshift, Azure SQL Data Warehouse, and other scalable platforms.Installed and managed Apache Airflow, creating DAGs for automated workflows with S3 integration.Automated data ingestion using PySpark and Scala from APIs, S3, Teradata, and Redshift.Developed and deployed ETL/ELT pipelines using PySpark and Terraform.Wrote Spark SQL and RDD-based PySpark scripts for data transformation and analysis; stored outputs in S3.Built and maintained ETL workflows in IBM DataStage for efficient data pipeline operations.Designed Denodo virtual views to unify access across Snowflake, NoSQL, relational databases, and APIs.Developed a RAG (retrieve-and-generate) NLP system for auto text summarization with tuned language models.Leveraged DataStage parallelism to centralize data from varied sources including APIs and flat files.Integrated real-time Snowflake data access via Denodo, avoiding redundant ETL processes.Improved Hive performance using sequence files, map-side joins, bucketing, and partitioning.Enabled automated infrastructure provisioning using Terraform with CI/CD.Wrote Spark applications in Scala for ETL, batch processing, and real-time data streaming.Built ROLAP and MOLAP cubes with SSAS and Oracle Analytic Workspaces for OLAP analysis.Managed job orchestration in DataStage including sequences, scheduling, and monitoring.Created Hive and Pig UDFs using MapReduce and Python for custom logic implementation.Performed structured and unstructured data profiling; established data quality checks using Hive, Spark, and Python.Designed and implemented scalable Scala-based Spark transformations using DataFrames and RDDs.