Copilot Studio Custom Graders: Write the Rubric Once, Reuse It in Every Evaluation

Agent evaluations in Copilot Studio are only as good as the rule that decides pass or fail. Until now that rule tended to be re-created for every test set, which means it drifts: one agent is judged on tone, a sibling agent on something slightly different, and nobody can say which standard is current.

Message Center notice MC1531932, published on October 30, 2026, announces the fix: a library of graders you create once and reuse across evaluations, alongside ready-to-use graders from Microsoft. General availability is listed as November 15, 2026. Microsoft marks the notice as awareness only, with no action required.

What is actually changing

From the notice and the earlier preview notice (MC1472547, which started rolling out in public preview on September 15):

• A library for graders. You create, save, manage, find and reuse custom graders instead of rebuilding them inside each test set.
• Ready-to-use Microsoft graders. You can start from graders Microsoft provides and add your own for rules specific to your business.
• GA on November 15, 2026, a week after this post.

It builds on agent evaluation, which Microsoft announced as generally available earlier. A grader is the piece that turns an agent's response into a verdict, so a shared library is a shared definition of "good".

Why a shared library matters more than it sounds

Think about what usually goes wrong in evaluation suites:

1. Criteria copied and edited. Three agents start with the same rule, then each team tweaks its copy. Results stop being comparable.
2. Rules that live in someone's head. "The answer must not promise a delivery date" is obvious to the author and invisible to the next maker.
3. No change history. When pass rates move, you cannot tell whether the agent changed or the rule did.

A library fixes the first two directly. For the third you need discipline of your own, described below.

Writing a grader that earns its place

A good custom grader checks one thing and says why it failed. Some examples that suit a customer-facing agent:

• No unsupported commitments. The answer must not state a price, date or refund unless it appears in the retrieved source.
• Required disclosure. Any answer about an account change must include the verification step.
• Scope. The agent declines questions outside the product area instead of improvising.
• Escalation. When the user states a complaint, the answer offers a human handoff.

Keep each rule narrow. A grader that tries to judge "helpfulness, tone and accuracy" at once produces a score you cannot act on. Split it, and the failing grader names the problem.

Treat the library like code

Because I could not find a built-in version history for graders, I would add my own:

• Name graders with a version, for example no-unsupported-commitments v2, and retire old ones deliberately.
• Keep the rubric text in your repository next to the agent's documentation, so a change goes through review.
• Record which grader versions a release was tested with. If your release process already calls the evaluation API, store the list with the build. My earlier post on gating releases with the Evaluation API shows where that fits.
• Calibrate against humans. Take twenty real answers, mark them yourself, and check the grader agrees. A grader that disagrees with you is a bug in the grader.

What to do this week

You have until November 15 before it is generally available, and the preview has been out since mid-September, so this is a good time to try it on one agent:

1. Pick the three rules you check by hand today.
2. Write them as custom graders and run them on an existing test set.
3. Compare the verdicts with your own on the same answers.
4. Only then attach them to the evaluation you use to approve releases.

What I could not verify

I found the notice text through Message Center mirrors, not a Microsoft Learn page, so details can change before GA. I did not find limits on how many graders a library can hold or how sharing across environments works. Check the documentation once it is published.

Sources

• MC1531932: Copilot Studio, custom graders library and ready-to-use Microsoft graders (published October 30, 2026; GA November 15, 2026)
• MC1472547: Copilot Studio, custom grader library (public preview notice)
• Agent evaluation in Microsoft Copilot Studio is now generally available
• Custom graders in Copilot Studio: setting high standards for agent evals