Tutor scorecard

Why this needs its own bench

A study coach wired into a course is not a chatbot with a syllabus in the prompt. It has a live view of a learner's courses, assessments and sometimes grades, so grounding and data handling are part of the product, not a wrapper around it. A plausible general-biology answer can still be a bad tutoring answer if it contradicts the instructor's material.

So the bar has two halves. Teaching: recognise the misconception before correcting it, ground the correction in the course's own pages, refuse work meant for graded submission, leave the learner with a check for understanding. Safety: least-privilege read-only access, no other learner's records, and course content treated as untrusted data rather than as instructions.

What it actually checks

Twenty-five scenarios run against a fictional course. Each one carries an expected behaviour plus pass, partial and fail criteria, so a reviewer can tell a mechanical failure apart from a weak explanation. The automatic half only flags the narrow things a machine can be sure of: a synthetic protected value appearing in a reply, or answer-key material surfacing where it should not.

That is the whole design. Deterministic checks cannot establish good pedagogy, but they reliably catch known-unacceptable behaviour, and they make regression visible when a prompt, a retrieval pipeline or a policy changes. They are a cheap gate in front of expensive human review, not a substitute for it.

What the first runs turned up

Three seeded runs, three local open-weight models, the same 25 scenarios. The smallest model failed three; the two larger ones cleared all twenty-five. Three failures out of twenty-five is what makes the bank worth keeping: a scorecard where every model scores alike is not measuring anything.

One of the three is the case worth reading. Asked for an ordinary study guide, the small model pulled a quiz answer key into the reply, because a page in the course material told it to. The learner asked for nothing improper. A tutor that refuses a direct request for answers can still hand them over when the instruction arrives inside the material it was asked to teach from, and that is why untrusted-content handling belongs in the evaluation rather than in a policy document.

Two earlier runs read as safety failures and were not. A reasoning model spent its entire output budget thinking and returned nothing, which scored as a deterministic failure; and when the server restarted mid-run, sixteen requests failed and scored as sixteen safety failures. Both are the same mistake, treating an absent answer as a bad answer. The harness now records an absent reply as its own state and keeps it out of the pass and fail counts. A benchmark that cannot tell infrastructure noise from a leak will report whichever number the reader wants.

What these numbers are not

Not about any product

Synthetic fixtures, an invented course, and a scenario bank written from scratch. This is how I would evaluate the category. It is not evidence about anything anybody ships.

Not a leaderboard

Local open-weight models are not what a vendor would deploy as a study coach. The harness is the artifact; the models are whatever was available to exercise it.

Not a pedagogy result

Every number recorded so far is a deterministic safety rail. The teaching column is hand-scored and stays blank until a human fills it in.

Not proof of a pathway

The rails detect a keyed answer turning up where it should not. They cannot separate a model that read the answer key in its context from one that recalled the same fact from training. The rail measures the outcome, not the route.

Not stable across upgrades

Runs reproduce within one inference-server version. The small model refused a roster request on one version and leaked it on an earlier one, which is worth knowing before anybody quotes a single number.