# Don't get carried away by leaderboards. What matters is how a model holds up in your company.

Jaro Rais · 7 min read · 2026-10-16 · AI and language models
Source: https://itashu.co/en/know-how/how-to-test-an-ai-model

You deploy the model that won the leaderboard. Customers wait a long time for answers. Some answers are off. The bill is higher than you planned. **The leaderboard tested one thing. Your operations test everything else.** That's exactly why many AI projects end at the pilot stage.

![A triangle of accuracy, speed and cost with a dot where a particular solution stands](https://itashu.co/img/art-12-hero-1280.webp)

## You always pick two of three: accuracy, speed, cost

You can draw it as a triangle. Accuracy at the top, speed and cost in the corners. **You can get two properties at once, the third suffers.** A large, accurate model serving hundreds of people at once will be expensive. A cheap, fast model will be less accurate.

Testing is how you find out where your solution really sits in that triangle. And find out before your customers do.

- **With a person:** A candidate has excellent grades from university. Whether they can handle your toughest client at four on a Friday, you only find out during the probation period.
- **With AI:** The leaderboard is the grades. Your data, your customers and your busiest hours are the probation period. Nobody can do it for you.

## 01 First question: does it answer correctly in your field?

Leaderboards work like a standardized test. The model gets questions on chemistry, history or programming, and the answers are compared with the correct ones. **But most business questions don't have a single correct answer.** There are a hundred good ways to answer a customer's question.

That's why open-ended answers are evaluated with a method called **LLM as a judge**: another strong model rates the answers against criteria set in advance. Is the tone appropriate? Did the answer help? Did AI make anything up about your products?

One condition applies. **The judging model can't be trusted blindly.** First, an expert from the company rates a sample of answers. The judging model is calibrated to that, and only then rates hundreds more.

> **In practice:** “An online shop's customer line. The head of support rates 50 real AI answers: good, bad, and why. Based on her ratings, the judging model then checks hundreds of new answers every week and flags the problematic ones. She only reviews what was flagged.”

![A person grades a sample of answers and an evaluator model checks the rest on a conveyor](https://itashu.co/img/art-12-judge-1280.webp)

## 02 Second question: can it cope when many people ask at once?

A model can be accurate and still unusable. That's why four things are measured, which we translate into operational terms:

- **How long a person waits for the first word of the answer.** In a chat, that's the difference between a smooth conversation and a closed window.
- How fast the rest of the answer arrives.
- How long the whole answer takes.
- How many people the system serves at once.

It's important to test with the kinds of questions that really come in. A chat with a short question and a long answer loads the system completely differently from a long document that needs a short summary. **If you test the wrong type of questions, you get completely different numbers from what you'll see in production.**

The result of the test is a breaking point. Up to a certain number of simultaneous users, response time is stable, then it rises sharply. That point is the number you use to plan capacity and budget.

- **With a person:** Reception copes with a quiet morning. Whether it copes with Monday morning, when twenty people call at once, you only find out by trying.
- **With AI:** We test load the same way. Not an average day, but the busiest hour, when the most people ask at once.

## With an agent, you test every step, not just the result

An agent isn't one answer. It's a chain of decisions: did it understand what the person wants? Did it pick the right tool? Did it find the right material in the documents? **Every link in the chain is a place where something can go wrong,** so each one has its own test.

Testing can be shown as a pyramid. Many teams start at the top, with expert tests. But the pyramid stands on its foundations:

| LEVEL | WHAT WE TEST |
| --- | --- |
| 1. Operations | Is the system fast enough and can it handle the number of users? |
| 2. Format | Does the answer arrive in the shape the next system or person expects? |
| 3. Security | Is the answer free of personal data that shouldn't be there? |
| 4. Facts | Is the answer based on your data, not a guess? |
| 5. Expertise | Does it answer like an experienced person in your field? |

![A testing pyramid from operations at the base to expertise at the top](https://itashu.co/img/art-12-pyramid-1280.webp)

**Ask yourself:** If your AI started answering worse today, would you notice, or would your customers?

## How we test so AI still works a month later

Testing doesn't end at launch. Models get updated, documents change, customers ask differently. That's why, for every solution, we build a set of check questions from your practice and run it regularly. **When results get worse, we know before you do.**

![A monitor with a dipping chart and an alert bell](https://itashu.co/img/art-12-monitor-1280.webp)

Before launch, we also measure speed and cost under your expected load. The model decision then rests not on a leaderboard, but on numbers from your company.

A leaderboard is a good start. **The real test happens on your data, with your number of users and** your measure of a good answer.

---

Tell us what your AI needs to handle. We'll show you how to test it before your customers do. https://itashu.co/en/consultation