Surya T
Senior Big Data Engineer @ Fannie Mae| Hadoop | HIVE | Spark| SQL | AWS | Azure | Data Warehouse | Data Mining | Data Visualization | Data Analyst | ETL | Database |
- Role
- Sr Big Data Engineer at Fannie Mae
- Location
- Charlotte, NC, US
- LinkedIn followers
- 500 followers
Experience
Sr Big Data Engineer
May 2022 — Present · TX, US
As a Senior Big Data Engineer at Fannie Mae, I specialize in leveraging cutting-edge technologies to drive data-driven insights and solutions. With expertise across the Hadoop ecosystem and various Big Data analytics tools, I play a pivotal role in transforming raw data into actionable intelligence. Responsibilities include designing and implementing robust data pipelines for ETL processes using Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics, with data processing in Azure Databricks. Developed JSON scripts for pipeline deployment in Azure Data Factory, streamlining workflows. Applied Spark APIs in Scala for real-time analysis and querying optimization. Designed Oozie workflows for Hadoop jobs, integrating Sqoop, Hive, and Pig. Utilized Hive for data analysis, creating external tables, loading data, and executing complex queries. Managed version control with Azure DevOps. Optimized HiveQL queries for performance and conducted performance tests with the Cassandra-stress tool. Ensured interoperability and integration between Pig, Hive, and relational databases using Sqoop and HCatalog. Configured and maintained Storm cluster topologies for high availability. Contributed to building scalable distributed data solutions on Hadoop, including Hive, Pig, Oozie, and Sqoop installation and configuration. Led a successful POC on Spark and Scala parallel processing, demonstrating real-time streaming capabilities with Spark and Kafka. Utilized PySpark for developing and optimizing Big Data flows. Actively pursued continuous learning and innovation to stay updated on emerging technologies and best practices, fostering a culture of excellence within the team.Environments: Hadoop, Spark, HDFS, Hive, Pig, HBase, Big Data, Apache Storm, Oozie, Sqoop, Kafka, Flume, Zookeeper, MapReduce, Cassandra, Scala, Linux, NoSQL, MySQL Workbench, Java, Eclipse, SQL Server.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.