Mohammed Khan

Senior Data Engineer Gen Ai @Morgan Stanley

NY, US

As a Senior Data Engineer specializing in GenAI systems, I’ve designed and implemented scalable ETL pipelines using Apache Spark, Airflow, and dbt to ingest and transform structured and unstructured data for AI model training. My work includes building real-time streaming ingestion with Kafka and Spark Structured Streaming and optimizing Delta Lake-based cloud storage solutions on Azure Data Lake Gen2 to support scalable, secure data lakes. These pipelines enabled downstream GenAI applications such as recommendation engines, summarization tools, and real-time analytics.I’ve integrated GPT-based models into enterprise systems using REST APIs and prompt templating strategies that ensure contextual, secure, and business-rule-aware outputs. My contributions include building vector embedding pipelines with OpenAI and HuggingFace models and storing them in Pinecone and Chroma to power Retrieval-Augmented Generation (RAG) systems. Additionally, I engineered prompt assembly logic based on cosine similarity from vector databases, improving the precision and relevance of LLM outputs across use cases like Q&A, document synthesis, and content generation.Beyond engineering, I led enterprise GenAI governance efforts—documenting model usage guidelines, bias risks, and versioned prompt libraries to support ethical and auditable AI deployment. I also managed cloud infrastructure with Docker, Kubernetes, and Azure Machine Learning to deploy containerized inference systems, and implemented robust monitoring and feedback loops for token usage, hallucination detection, and latency. Mentoring junior engineers, leading GenAI CoE initiatives, and ensuring end-to-end data and model traceability through MLflow and metadata tracking were key parts of my leadership in operationalizing GenAI at scale.