Intelligent Customer Service Agent for Enterprise and Open-World Data with Explanations and Citations
The Challenge
A leading consultancy required the development of a general QA agent capable of handling both enterprise and open-world data. The agent needed to understand the nature of each query, determine the appropriate steps to generate a reasonable response, and provide explanations and citations. The enterprise data involved millions of documents, and the agent also leveraged custom tools to integrate and analyze open-world data.
Hypothesis
- The data retrieval pipeline must be optimized for efficiency, ensuring quick access to relevant information. Additionally, the language model is required to accurately interpret the query type and reason through the information to generate appropriate responses.
- The model will respond to user queries both within and beyond the enterprise knowledge base, providing not only answers but also supporting citations and explanations to ensure transparency and credibility.
Execution
- Developed an evaluation framework for large language models (LLMs) to accurately diagnose the root cause of incorrect answers and assign a model accuracy score.
- Optimized the data pipeline to ensure faster and more efficient information retrieval.
- Created a query intent routing algorithm to classify query types and direct them to the appropriate sources for accurate responses.
- Refined prompt engineering techniques to enhance the quality and relevance of model-generated answers.
Outcomes
- A QA agent that can answer queries with explanation and proof both from open world and enterprise data.
- A LLM evaluation system that can be used to select a model and benchmark the results.
Project Highlights
90% +
of queries delivered with
appropriate citations
appropriate citations