Vaishnavi J

Data Engineer @Fidelity Investments

Durham, NC, US

Designed and implemented batch data pipelines using SnapLogic + Control-M to hydrate Snowflake with enterprise data.Developed and optimized stored procedures and advanced SQL queries (e.g, LATERAL FLATTEN) for large-scale transformations.Built and maintained real-time pipelines integrating Kafka with Snowflake Streams, Tasks, and Snowpipe for crypto transactions and pricing.Coordinated and executed Disaster Recovery (DR) activities for SnapLogic and Snowflake, ensuring resilience and compliance.Coded AWS Lambda functions in Python orchestrated with Step Functions to load/validate data from S3 to Snowflake.Leveraged PySpark for transformation testing and validation to ensure accuracy and performance.Configured SNS notifications and EventBridge rules for job monitoring and automated triggers.Designed Step Function pipelines for Oracle-to-Snowflake migrations and daily crypto pricing ingestion.Utilized SQS queues for high-volume Kafka stream ingestion and EC2 for compute-intensive workloads.Built ad-hoc APIs to trigger Lambda/Step Functions and dynamically control Snowflake tasks.Designed a SnapLogic-based data quality framework for source-to-target reconciliation and reporting accuracy.Collaborated with architects, analysts, and engineers to deliver secure, scalable crypto data solutions aligned with business reporting needs.Supported Databricks and PySpark integration within enterprise pipelines, ensuring compatibility with Snowflake transformations.Partnered with business stakeholders to translate crypto holdings, transactions, and pricing requirements into reliable technical workflows.Environment: Snowflake, SnapLogic, Kafka, Snowpipe, Control-M, AWS (Step Functions, Lambda [Python], EC2, S3, EventBridge, SNS, SQS), SQL, MySQL, Python, PySpark, Databricks