Danfer Yang
Lead Data Platform Engineer @Citi
Signup · Get unlimited contacts
WORK HISTORY
Lead Data Platform Engineer @Citi
CA
Led and developed a cloud-based data reconciliation product running on Kubernetes, streamlining data validation between source and target databases. Utilized technologies including Java, Kubernetes, Spark, AWS, Hadoop, Impala, Snowflake, Scala, and Python.• Designed and implemented a Solr-Spark loader for HBase indexing, solving HBase non-rowkey query challenges using Cloudera Spark, Solr, Kafka, and HBase.• Developed a Solr Microservice API on OpenShift with Java and Spring Boot, enhancing scalability, reliability, and availability for client queries.• Improved HBase availability with a zero-downtime solution by enhancing the HBase Microservice API with MCC library.• Built Spark loader dashboards with Prometheus, Grafana, and Kibana, significantly improving issue detection and resolution for production support.• Led Cloudera CDP 7 upgrade, conducting PoC and testing, and identified common upgrade challenges for developers.
EDUCATION
Beijing Institute of Technology
Master, Dynamic Mechanism and Engineering
Beijing Institute of Technology
Bachelor, Dynamic Mechanism and Engineering
SKILLS
ABOUT DANFER YANG
Highly skilled software developer with over 20 years of experience in designing, developing, and optimizing enterprise-grade solutions. Extensive expertise in Java, Scala, Python, C++, and Shell scripting.Hands-on experience in Big Data ecosystems (Cloudera Spark, Hadoop) and container orchestration platforms (Kubernetes/Docker). Proficient in cloud technologies (AWS). Rich experience in system performance tuning. Proven track record of leadership, architecture design, and project management in diverse and complex environments.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.