What labs mean when they say 'evaluation'

Evaluation is the single most important word in frontier AI work. It is also the word professionals entering the field understand least. When a lab says a role is an "evaluation" role, they do not mean quality assurance, they do not mean testing in the software sense, and they do not mean grading student work. They mean something specific and technical, and understanding it changes how you approach the work.

The short version

Evaluation is the process of measuring how good a model actually is at something. Not how good it looks in a demo. Not how good it feels in a chat. How good it actually is, measured against a rubric a working professional would defend, at a specificity high enough to compare to the last version of the model and the next one.

The reason this matters is that a frontier lab does not know how to improve a model until it knows exactly where the current model fails. And it does not know exactly where the current model fails until working professionals in every relevant field have looked at its outputs, scored them, and written down the reasoning.

Evaluation is what closes that loop. Which is why it is one of the largest single categories of expert work at every lab, and why it is paid the way it is.

What evaluation actually looks like

Concretely, an evaluation task usually has this shape.

You are shown a prompt in your field. The prompt is realistic. It might be a clinical vignette, a contract clause, a math problem, a translation input, an engineering specification. It is not a toy.

You are shown one or more model outputs in response to that prompt. Sometimes one output, which you grade on an absolute scale. Sometimes two or more, which you rank against each other.

You are given a rubric. The rubric is the set of criteria the lab is using to judge whether an output is good. You use the rubric to score the outputs. When you disagree with the rubric, you flag that too. That flag can be more valuable than the score, because it tells the lab their measuring stick is wrong.

You write down the reasoning behind your score. Not a paragraph. Not an essay. A few sentences that a reviewer can read and understand your logic. If the model got something wrong that would fail a real practitioner's read, you name the specific failure.

You submit and move to the next one.

Sessions run for a couple of hours at a stretch. Ten to twenty items in a session is typical, depending on the complexity of each prompt. You do this on your own time, on your own schedule.

Why it is priced the way it is

Evaluation looks simple from the outside. Someone reads a model output and scores it. In practice it is one of the most cognitively demanding activities a domain expert can be paid for.

The reason is that you are not just checking correctness. You are pattern-matching against thousands of hours of professional experience to catch failures that would slip past anyone with less training. A model can produce a legal argument that sounds fluent, cites real cases, applies the right statute, and is still confidently wrong in a specific way only a working attorney would spot. The value of an attorney's evaluation is exactly that: the ability to catch the wrong that looks right.

Rates are set accordingly. Evaluation work pays at, or often above, hourly consulting rates in the same field. For evaluators in emerging markets, the rate against local salaries in the same discipline is usually a large multiple.

What labs actually look for in evaluators

Depth in the field, above all. Recent, active practice. Written reasoning that is clear at the first pass and does not require a follow-up. A track record of being right when you disagree.

Not necessarily seniority. Not necessarily prestige. A working general practitioner in a busy clinic can be a better clinical evaluator than a former head of medicine who has not seen patients in five years. A trial lawyer with a hundred cases is a better legal evaluator than a corporate partner who has not seen a courtroom in a decade.

Written English at a professional standard, at the level a peer-reviewed article in your field would expect. Not native fluency. Just the ability to explain your reasoning clearly.

How to enter

If any of the above sounds like the kind of work you want to do, sign up, upload a resume, and see what evaluation roles are currently open against your field. There are usually several per major discipline, in various sub-fields and target languages.

The work is real. The rates are real. The point of getting good at evaluation is that it is a skill that will remain valuable for as long as frontier labs exist. Understanding what the word actually means is the first step.