Shikher Sahu

About Shikher Sahu

1) Almost 3 years of experience in delivering business solutions centered around analytics, utilizing the Hadoop Ecosystem (including Hadoop, Cloudera, Impala, Hortonworks) and cloud environments like Amazon Web Services (AWS) and MS Azure Cloud. 2) Proficiently developed robust Spark RDD-based data processing workflows using Scala, Java, and Python programming languages. 3) Demonstrated expertise in optimizing Spark RDD performance by meticulously tuning various configuration settings, including memory allocation, caching strategies, and serialization methods. 4) Leveraged deep expertise in utilizing Spark RDD transformations and actions to effectively process expansive structured and unstructured datasets. This includes adeptly handling filtering, mapping, reducing, grouping, and aggregating data. 5) Successfully developed and deployed Spark jobs on AWS EMR clusters using both Scala and Python programming languages, ensuring efficient and scalable data processing capabilities. 6) Implemented intricate Spark transformations tailored specifically for large-scale healthcare datasets, facilitating insightful analysis and data-driven decision-making. 7) Achieved superior Spark job performance by finely tuning configurations and meticulously allocating resources to optimize overall processing efficiency. 8) Automated seamless data ingestion from AWS S3 into Spark jobs, enabling real-time processing of incoming data streams with high reliability and efficiency. 9) Integrated Spark seamlessly with AWS Glue to establish automated ETL pipelines, ensuring smooth data flow and adaptive schema evolution. 10) Effectively integrated Hive tables with other prominent big data technologies such as Hadoop, HBase, and Impala, ensuring seamless data interoperability and integration. 11) Managed the intricacies of Hive metastore to effectively oversee table metadata management and facilitate seamless schema evolution processes. 12) Expertly utilized Hive table formats including ORC, Parquet, and Avro, strategically selecting each based on their unique advantages and optimal fit for diverse use cases. 13) Developed comprehensive Hive table partitioning strategies including range, hash, and list partitioning, meticulously balancing query performance optimization with optimal data distribution. 14) Demonstrated adept troubleshooting skills in resolving common Hive table issues such as data skew, table corruption, and performance optimization challenges.

Experience

1. Big Data Developer

   ValueSoft Info Services

   Mar 2024 — Present · IN

   1) Proficiently utilize Spark RDD persistency and caching techniques to streamline data processing and enhance query speed.2) Apply knowledge of Spark RDD lineage and fault tolerance mechanisms to ensure robustness and efficiency in data processing workflows.3) Design and execute intricate Spark SQL queries for comprehensive data aggregation and reporting purposes.4) Manage dependencies and packages within Spark projects using Maven and SBT to ensure smooth execution and integration.5) Leverage the Spark DataFrame API for efficient data manipulation and transformation tasks.6) Develop and implement partitioning and caching strategies to improve execution times of Spark jobs.7) Create custom Spark UDFs (User Defined Functions) to extend functionality and enhance data transformation capabilities.8) Orchestrate Spark job workflows using Apache Airflow for efficient scheduling and monitoring.9) Implement Spark RDD optimization strategies, including data partitioning, shuffle tuning, and pipelining, to optimize resource utilization and query performance.

Education

• Babu Banarsi Das University, Lucknow

  Bachelor's degree

  2014 — 2017