Collect the knowledge documents in PDF format that you want to teach your model.

• For example, we use the bnpp-fin-statement.pdf in a previous POC.
• Optionally, you can use the preprocess.py script to preprocess your document using docling.

You need to prepare a qna.yaml file to describe the types knowledge you want to teach your model.

• The qna.yaml provided in the repo is the one we used together with bnpp-fin-statement.pdf.
• This repository provides a good guide for beginners to work with qna.yaml files.

You can use the InstructLab Service on IBM Cloud to train your model.

---with your serving method

You need to deploy the trained model, either on the cloud or downloading it and serving it locally.

---with your favorite RAG system

You can follow this notebook where has a simple RAG with LLM judge evaluation.

• You don't have to use the RAG system in the notebook---you can use your own RAG system and continue the evaluation with from the section called "Grade responses using Judge Model".
• Note, it's always important to evaluate the responses manually to make sure the model response and the LLM judge are working as expected.