Evaluation Datasets for AI Applications: What, Why and How?

The What

An evaluation dataset is a curated collection of realistic test cases used to measure whether an LLM application or AI agent behaves as intended: correctly, safely, and consistently. The dataset typically includes an input, the expected outcome, and clear grading criteria.

The Why

LLM applications are probabilistic: the same input may produce different answers. AI system components also change frequently as models, prompts, retrieval systems, tools, and workflows evolve. At Tech 42, we are technology experts, but we understand that no one understands your business as much as you do, which is why we ask our customers to build their evaluation datasets. As we build the systems, this evaluation dataset acts as our north star, helping us select the right model for the use case, craft effective prompts, and tune AI tooling. Evaluation datasets provide a reliable way to measure how well an AI system is performing.

In its simplest form, an AI system consists of an LLM, a set of inputs and an output. In order to evaluate the performance of different models for your system, the input and expected output are required. The input will vary based on your use case. At its base, the complete input will contain a system prompt and a user query but can also contain extra context from RAG systems, tool calls, or other context augmentation systems. To evaluate performance, the input is passed through the LLM and the output is compared to the expected output you provide to measure concordance.

For more complex systems, evaluations can happen at 2 levels:

End-to-end evaluation: Tests whether the complete application produces the correct final answer or action.

Node-level evaluation: Tests individual components to identify where failures occur. This can include evaluation of routing, retrieval, LLM generation, tool use, access control, guardrails, and more.

Ideally, evaluation datasets are provided for both final and node-level elements.

The How

Build a set of 20-40 input/output pairs for the final and node-level tests that our team can use to assess system performance.

For each input/question, provide the associated context and expected answer(s).

The inputs/questions should be diverse and test different aspects of the system.

The file that includes these pairs can be in a variety of formats, like csv, json, etc.

If you need help building the evaluation dataset, feel free to reach out to our team.

Framework

1. Describe your system & your task objective. Summarize the task you would like the system to accomplish and the main components of the system.
2. Collect representative cases. Include normal cases, edge cases, and ambiguous or adversarial inputs, important constraints. As the business owner, you know the intended output best, and that knowledge should shape the cases.
3. Provide the System Prompt.
4. Create Q&A pairs. This should be a set of 20-40 question-answer pairs for each part of the system.
   
5. Provide the Context. Provide us with your system prompt and any necessary context if it applies to your use case.
6. Use a clear grading rubric. For answers that have ranges or are subjective, provide us with a clear marking rubric.

Framework Example

Assume I have an application that helps unicorn.co employees get answers without needing human assistance. This virtual agent has two main external data sources (Node level):

1 - A knowledge base of company policies

2 - A database of employee metrics

Let’s create a simple evaluation dataset that the Tech 42 team can use to polish this virtual agent. Because the final end-to-end evaluation is the same as the node-level evaluation in this case, only the node level datasets are necessary.

1. Describe your system & your task objective

The system is an HR Agent that answers employee questions about company policies and employee metrics. It should know that it represents unicorn.co, not unicorn.com. It should be professional and friendly. It should NOT disclose PII information.

The system is made of an Langgraph agent with Sonnet 4.6 and has two external data sources: a knowledge base of company policies and a postgres database of employee metrics.

2. Collect representative cases

Source 1: Company Policies: 

Normal Cases: Asks about PTO, Benefits, Procedures, HR related information

Edge Cases: Maternity, Benefits for 2020 hires, Finance hires are special

Ambiguous inputs: Several company ids due to acquisition. Ask for UNIC_ID

Source 2: Employee Metrics: 

Normal Cases: Asks about annual performance, annual PTO taken, etc.

Edge Cases: Year 2020 was a half year

Important Constraints: Department-based access 

3. Provide System Prompt

You are a helpful HR agent that answers questions about company policy and employee performance. You represent unicicorn.co. You have access to two systems: a knowledge base of company policies and a database of employee metrics. Answer professionally and be friendly. Do not disclose PII information.

4. Create Q&A pairs with context

Ideally this section should be a separate file (json, csv, etc.) with an easily parseable structure. Put each question, context and answer on new lines or structure them in JSON.

For agentic applications, there are more system parts to create tests for. In light of this, we need an expected behavior for each part.

Source 1: Company Policies: 

Q1: How many PTO days do unicorn.co employees get?
C1: Contents of document unicorn_benefits.pdf
A1: PTO days depend on the length of employment:

• Less than 2 years: 15 days
• 2-5 years: 20 days
• Over 5 years: 25 days

Q2: What are the benefits at unicorn.co?
C2: Contents of document unicorn_benefits.pdf
A2: Benefits include training expenses up to $1000/year, gym access, and free parking passes.

… … …

Q30: How do I apply for travel reimbursement?
C30: Contents of document unicorn_travel.pdf
A30: Contact travels@unicorn.co with a filled T765 form. Include all your receipts.

Source 2: Employee Metrics: 

Q1: What was the AI team’s performance last year?
C1: AI team, 230 projects, on budget 90%
A1: The AI team performed 230 projects last year with 90% on budget.

Q2: Which department was the least efficient last year in budget spending?
C2: Guardrail activated.
A2: I cannot report on other departments. I can only report on the AI team.

… … …

Q30: What is the overall travel spending for the AI team in 2020?
C30: AI spending: 55,436 total; 7,654 events; 23,432.12 travel
A30: The team spent $23,432.12 on travel.

4. Use a clear grading rubric

For Q&A that are subjective or very complex, add a grading rubric. For example:

PTO days differ by department, tenure, and employee grade:

• If answer takes all 3 into account: grade at 5/5
• If answer only takes 2 into account: grade at ⅗
• If it only uses a number without any reference to variables: 1/5

Creating a validation dataset will differ by client and by use case. Your Tech 42 engineer can help guide you through this process.

Example datasets

Company Policies Eval Dataset

Employee Metrics Dataset‍