When correct AI answers fail our tests

Anyone who develops AI assistants at AIC Group knows the moments when you look at a red test result and think: “Why does the AI still not get it?”

This time I am not sharing an insight of my own, but one from our software development team. It is the first article to come from there, and I hope it will not be the last, because AI agents are playing an ever larger role in software development in particular. The team has written down two of those moments and what it learned from them.

We wanted our AI assistant to select people aged 18 and over. It suggested “older than 17”. Our test reported an error.

With an age stored in whole years, both conditions match exactly the same people. At first, the red test told us more about our expectations of the answer than about the model’s understanding.

On closer inspection, some answers were correct in substance. Our check, however, demanded a particular form. Elsewhere, the model was asked to judge a fact that our software could already check unambiguously. Two different causes, both of which initially looked like the AI failing to understand.

The idea behind both: when designing an agent prompt, we should decide deliberately how much freedom of decision the model actually needs. Solutions that are equivalent in substance may be given room. Facts that are already settled, on the other hand, do not need to be put up for decision again.

1. Check the desired result

“At least 18” and “greater than 17” are equivalent for whole years of age. If you store only the first form as the model answer, you reject the second, even though it fulfils the task.

Should the AI deliver exactly our model answer, or should it achieve the desired result in substance?

The limit is part of the story: with decimal numbers, the equivalence does not hold. 17.5 is greater than 17, but not yet at least 18. And for unknown ages, we have to define what should apply. In our example, they do not count towards the desired group. Whether the software in use actually carries this out has to be checked separately.

The practical consequence: before the test, define the goal, the valid data range and the permitted variants. Then check counter-examples. A different notation is not automatically an error in substance; conversely, a similar answer is far from being a correct solution.

2. Determine the freedom of decision that is actually needed

An agent prompt also determines what the model is supposed to decide. That is why, for every piece of information requested and every action offered, it is worth asking:

Do we really need the model to decide here, or is the answer already settled by data, rules or the current state of the work?

An extra sentence such as “Always decide correctly” does not remove an unnecessary choice. Where a precondition can be checked unambiguously, the application can check it in advance and align the options it offers accordingly. The model keeps the room it needs to understand and solve the actual task.

A user refers to their “newsletter address”. The model finds the right field and reproduces the user’s words correctly. In an additional output field, however, it claims that the user named the technical field path verbatim. Our check therefore rejects the answer.

There are two different tasks hidden in this:

  • Which field does the user mean? That requires an understanding of language, field descriptions and the context at hand. Where there is ambiguity, a follow-up question is needed.
  • Is the full technical field path in the text? The software can check that unambiguously. It does not need a language model for that.

The design direction we derived from this: if the technical path is missing from the user’s text, “path named verbatim” should not be available as an option for this field at all. Which field “newsletter address” refers to, on the other hand, remains a task of interpretation.

This boundary belongs both in the prompt and in the actions and output formats actually on offer. Reinterpreting a wrong answer afterwards to make it fit would be something else, and it would hide the actual error. Nor should the application quietly make a choice when there is genuine ambiguity.

Where things stand: we have tested the new age check internally. Restricting the path options in advance is the next design change, and we are still checking its effect. We have not solved every model error with this.

blog-wenn-richtige-ki-antworten-durch-unsere-tests-fallen-zwischenbild

The rule of thumb

Read known facts from the responsible source. Calculate unambiguous derivations. Interpret meaning. Clarify open decisions.

To take away: five questions before the next round of prompt or model changes

  1. How do we recognise, in substance, that the task has been fulfilled?
  2. Which different solutions may achieve this goal?
  3. Which details are already fixed by reliable sources or unambiguous calculation?
  4. How much freedom of decision does the model need, and which choices are unnecessary?
  5. Where do genuine interpretation, ambiguity or a follow-up question to the user remain?

The two insights complement each other: keep room for solutions that are correct in substance, and take avoidable decisions out of the model’s brief. The quality of an AI assistant also includes the quality of the tasks and checks we build around it.

Have you ever found a supposed AI error that was actually down to your own yardstick?

Write to us. We’d be happy to share our approach.

Portr

Marc Nawroth

Post from November 3, 2026

More articles…

Jörg Heydorn neben einem Dreieck aus Audience Building, Workflows und Contentsteuerung

Why we rebuilt our MAS

Customers’ decisions are more individual than ever: when, what, through which channel. Why the new MAS is our answer to this, and why everything starts with a central data base.

Read more »

*Articles are partly written with the help of AI.