Databricks GenAI Engineer · Application Development

A Generative AI Engineer is building a RAG chatbot on Databricks. The pipeline must retrieve context from a Databricks Vector Search index, format it into a prompt template, and call a Foundation Model API endpoint. The engineer wants to compose these steps declaratively so each component can be swapped independently and the chain supports streaming output token by token. Which approach should the engineer use?

• AWrap all three steps inside a single custom Python class that hardcodes the retriever, prompt, and model calls in sequence
• BUse LangChain's legacy LLMChain class, which supports memory injection but does not natively support streaming
• CUse LlamaIndex's SimpleDirectoryReader to load the vector index and query it directly, bypassing prompt templating
• DCompose the retriever, prompt template, and model using LangChain Expression Language (LCEL) with the pipe (|) operator