AI Engineer Take-Home Task: Development of a Document-Enriched AI Chatbot

Objective: Develop a console-based AI chatbot that leverages generative AI techniques to

provide accurate, informative responses by mitigating hallucinations. The chatbot should
utilize external PDF documents and vector database searches to verify information and
enhance the reliability of its responses.

Requirements:

1. Chatbot Framework:
- Implement a basic chatbot interface in the console that can accept user queries and return
responses.
- Use a pre-trained language model (e.g., GPT-4 or similar) as the foundation for generating
responses.

2. Document Handling:
- Integrate a PDF parser to extract text from a collection of provided PDF documents. These
documents serve as a trusted knowledge base for the chatbot.
- Implement functionality to search these documents based on user queries to find relevant
information.

3. Vector Database Integration:

- Create or integrate a vector database to store and retrieve information efficiently. Use this
database to index the contents of the PDF documents.
- Employ vector search techniques to find the most relevant document sections in response to
user queries, aiming to support or validate the chatbot's generated responses.

4. Hallucination Mitigation:
- Develop a mechanism to detect potential hallucinations or inaccuracies in the chatbot's
responses.
- Cross-reference the generated response with the knowledge base (PDF documents) and
vector database to verify facts before presenting the final response to the user.

5. Testing and Evaluation:

- Provide a set of test queries and document the chatbot's performance, specifically focusing
on accuracy, response time, and its ability to mitigate hallucinations.
 - Include a brief evaluation on the effectiveness of the document and vector database
integration in improving response reliability.

Deliverables:

- Source code for the chatbot, including all dependencies and setup instructions.
- A small collection of PDF documents to serve as the initial knowledge base.
- Documentation detailing the system architecture, how to run the bot, and an overview of
the hallucination mitigation strategy.
- A report summarizing the testing methodology, results, and any insights or challenges
encountered during development.

Evaluation Criteria:

- Functionality: Does the chatbot operate as intended, handling user queries and integrating
external data sources effectively?
- Innovation: Are there any novel approaches or technologies used to solve the task?
- Code Quality: Is the code well-organized, commented, and adhering to best practices?
- Performance: How well does the chatbot mitigate hallucinations, and what is the overall
response quality?
- Documentation: Clarity and completeness of the provided documentation and reports.

3 Days