Gate Copilot Studio Releases with the Evaluation API

Run Copilot Studio agent evaluations from a pipeline with the Power Platform API, then block a release on regressions. A tested gate script, the auth caveats, and the pitfalls.

By Ajith Joseph · Thu Oct 01 2026 · Updated Thu Oct 01 2026 · 20 min read · intermediate

#copilot-studio #power-platform #testing #ci-cd #ai-agents

Copilot Studio can already tell you whether an agent got worse. What it cannot do by itself is stop you shipping when it did, because someone has to click Evaluate, read the result and decide.

The Power Platform API closes that gap. You can start an evaluation run for a test set, poll it, and read the per-case results from a script, which means a pipeline can fail a release the same way a failing unit test does. This post walks through the documented API, builds a small gate on top of it, and is upfront about what I could verify and what you must check in your own tenant.

Everything about the API below comes from Microsoft Learn's page on automating evaluations (last updated August 2026) and the Power Platform REST reference. The script was tested against a local mock server built from those documented shapes. I did not run it against a live tenant, so treat the first real run as part of your setup.

What You Can Automate

The page describes four operations under one base path, all pinned to api-version=2024-10-01:

Operation Request
List test sets GET /copilotstudio/environments/{EnvironmentId}/bots/{BotId}/api/makerevaluation/testsets
Start a run POST .../makerevaluation/testsets/{TestSetId}/run
List runs GET .../makerevaluation/testruns
Run details GET .../makerevaluation/testruns/{TestRunId}

The host is https://api.powerplatform.com. A started evaluation runs asynchronously, so the flow is always start, poll, read.

Three scope notes matter before you invest time:

  • It is documented for the standard harness. The page says it describes agents and agent flows powered by the standard harness. Check the current documentation before assuming it covers GitHub Copilot harness agents.
  • It was listed as preview in Microsoft's April 2026 What's new entry, "Run automated agent evaluations from a REST API." Check the status when you adopt it.
  • There is a no-code alternative. The Copilot Studio connector for Power Automate can trigger evaluations too, which is simpler if your release process already lives in Power Automate.

The Request That Catches People Out

Starting a run takes a small JSON body:

Field Meaning
evaluationRunName A label for the run
mcsConnectionId The Copilot Studio connection to run as, or empty for an anonymous run
runOnPublishedBot Whether to test the published agent
toolsConnections Per-tool user connections for tools that need authentication

runOnPublishedBot defaults to false. Read the schema: when it is false the run goes against the draft agent, not the published one.

That is the right default for gating a change before publishing, and the wrong one for "did the release we just published still work." Neither is a bug. The danger is a gate that does not say which one it tested, so a green check means something different from what the team thinks. The script below forces you to be explicit and prints which agent it tested.

If your tests need an authenticated user, mcsConnectionId selects the user profile. You find it in Power Automate: open Connections, select the Microsoft Copilot Studio connection, and copy the id from the URL, which ends in .../connections/shared_microsoftcopilotstudio/{mcsConnectionId}/details.

Results and States

The run response includes runId, state, executionState, totalTestCases, testCasesProcessed and a callbackUri that the docs describe as the address to poll for results.

The run state is one of: Abandoned, Cancelled, Completed, Deleted, Failed, InProgress, Queued or Unknown. The first five end the run. The details endpoint then returns testCasesResults, and each case has metricsResults, one entry per test method, with a type, a status and, when something went wrong, an errorReason and an aiResultReason explaining the grader's verdict.

Per-case statuses in the portal are Pass, Fail, Invalid or Error. That detail shapes the gate: only an explicit Pass is a pass. An Error is not a failure of the agent's answer but it is also not evidence the release is safe, so a gate should not wave it through.

Authentication, Honestly

This is the part most likely to cost you an afternoon, so here is exactly what the documentation says.

The evaluation page lists a prerequisite of "a user access token issued by Microsoft Entra ID (OAuth 2.0)" from an app registration that has been granted the appropriate scope under the Power Platform API. The general authentication page adds that the Power Platform API "uses delegated permissions only at this time," and that for service principals you should not use application permissions. Instead you create the app registration, add a certificate or secret, request a token with the https://api.powerplatform.com/.default scope and assign the service principal an RBAC role such as Contributor or Reader.

What I could not find is a statement of whether the evaluation operations accept a service principal token. The evaluation page only documents a user token. So you have two paths, and you should find out which one works before building a pipeline around it:

  • Try the service principal path first. It is the right shape for CI. Request a client-credentials token, give the principal an RBAC role, and call the list-test-sets endpoint. A 200 tells you it works. A 401 or 403 tells you it does not.
  • Fall back to a dedicated user account. The authentication page warns that the username and password flow "doesn't work for accounts that have multifactor authentication enabled," and conditional access policies can block token acquisition for the app too. Plan with your Entra administrator rather than discovering this in a failed pipeline.

Either way the script takes the token from an environment variable and never prints it.

The Gate

The script has no dependencies and runs on any recent Node. It finds the test set by its display name, refuses to guess if the name is missing or ambiguous, starts the run, polls until the run reaches an end state, and then applies three rules:

  1. The run must end as Completed
  2. The pass rate must meet your threshold
  3. No case that passed in the baseline may stop passing

The third rule is the one worth having. A pass rate can stay flat while the agent gets worse, because one case improves while another breaks. If you only watch the percentage, that swap looks like nothing happened. Comparing case by case does not have that blind spot, and the baseline file is just a map of case id to passed.

// Release gate for a Copilot Studio agent, built on the Power Platform evaluation API.
//
//   POWER_PLATFORM_TOKEN=<bearer token> node eval-gate.mjs \
//     --env <environmentId> --bot <botId> --set "<test set display name>" \
//     [--min-pass-rate 0.9] [--baseline baseline.json] [--save-baseline baseline.json] \
//     [--published] [--timeout-min 20]
//
// Exit codes: 0 gate passed, 1 gate failed, 2 the run could not be completed (API error, timeout, bad input).
// Endpoints, api-version and field names are from the Microsoft Learn page "Automate evaluations with
// the Power Platform API" and the Power Platform REST reference (api-version 2024-10-01).
import { readFileSync, writeFileSync } from 'node:fs';

const API_VERSION = '2024-10-01';
const DEFAULT_BASE = 'https://api.powerplatform.com';
// EvaluationRunState values that mean the run will not change any more.
const TERMINAL_STATES = new Set(['Completed', 'Failed', 'Cancelled', 'Abandoned', 'Deleted']);

export class GateError extends Error {}

export function makeClient({ base = DEFAULT_BASE, token, environmentId, botId, fetchImpl = fetch }) {
  if (!token) throw new GateError('No access token. Set POWER_PLATFORM_TOKEN.');
  const root = `${base}/copilotstudio/environments/${encodeURIComponent(environmentId)}/bots/${encodeURIComponent(botId)}/api/makerevaluation`;

  async function call(method, path, body) {
    const url = `${root}/${path}?api-version=${API_VERSION}`;
    const response = await fetchImpl(url, {
      method,
      headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },
      body: body === undefined ? undefined : JSON.stringify(body),
    });
    const text = await response.text();
    if (!response.ok) {
      // The token is never included in an error message.
      throw new GateError(`${method} ${path} failed with HTTP ${response.status}: ${text.slice(0, 300)}`);
    }
    return text ? JSON.parse(text) : {};
  }
  return { call };
}

/** Finds an active test set by its display name; refuses to guess when the name is missing or ambiguous. */
export async function findTestSet(client, displayName) {
  const { value = [] } = await client.call('GET', 'testsets');
  const matches = value.filter((s) => s.displayName?.toLowerCase() === displayName.toLowerCase() && s.state === 'Active');
  if (matches.length === 0) {
    const known = value.map((s) => `"${s.displayName}" (${s.state})`).join(', ') || 'none';
    throw new GateError(`No active test set named "${displayName}". Found: ${known}`);
  }
  if (matches.length > 1) throw new GateError(`More than one active test set is named "${displayName}". Rename one.`);
  return matches[0];
}

/** runOnPublishedBot defaults to false in the API, which tests the DRAFT. Be explicit about which one a gate means. */
export async function startRun(client, testSetId, { name, published, mcsConnectionId }) {
  const body = { evaluationRunName: name, runOnPublishedBot: Boolean(published) };
  if (mcsConnectionId) body.mcsConnectionId = mcsConnectionId;
  return client.call('POST', `testsets/${encodeURIComponent(testSetId)}/run`, body);
}

export async function waitForRun(client, runId, { timeoutMs = 20 * 60_000, pollMs = 15_000, sleep = (ms) => new Promise((r) => setTimeout(r, ms)) } = {}) {
  const deadline = Date.now() + timeoutMs;
  for (;;) {
    const details = await client.call('GET', `testruns/${encodeURIComponent(runId)}`);
    if (TERMINAL_STATES.has(details.state)) return details;
    if (Date.now() >= deadline) throw new GateError(`Timed out waiting for run ${runId}; last state was ${details.state}.`);
    await sleep(pollMs);
  }
}

/** A case passes only if it produced results and every test method on it reports Pass. Anything else is not a pass. */
export function summarize(details) {
  const cases = details.testCasesResults ?? [];
  const byCase = {};
  const failed = [];
  for (const c of cases) {
    const metrics = c.metricsResults ?? [];
    const statuses = metrics.map((m) => String(m.status ?? '').toLowerCase());
    const passed = metrics.length > 0 && statuses.every((s) => s === 'pass');
    byCase[c.testCaseId] = passed;
    if (!passed) {
      failed.push({
        testCaseId: c.testCaseId,
        statuses,
        reason: metrics.map((m) => m.aiResultReason || m.errorReason).filter(Boolean).join(' | ').slice(0, 240),
      });
    }
  }
  const total = cases.length;
  const passedCount = total - failed.length;
  return { total, passed: passedCount, failed, passRate: total === 0 ? 0 : passedCount / total, byCase };
}

export function evaluateGate(details, summary, { minPassRate = 1, baseline = {} } = {}) {
  const reasons = [];
  if (details.state !== 'Completed') reasons.push(`run ended as ${details.state}, not Completed`);
  if (summary.total === 0) reasons.push('the run produced no test case results');
  if (summary.passRate < minPassRate) {
    reasons.push(`pass rate ${(summary.passRate * 100).toFixed(1)}% is below the required ${(minPassRate * 100).toFixed(1)}%`);
  }
  // A flat pass rate can hide one case getting worse while another improves, so compare case by case.
  const regressions = Object.entries(baseline)
    .filter(([id, wasPassing]) => wasPassing && summary.byCase[id] === false)
    .map(([id]) => id);
  if (regressions.length > 0) reasons.push(`${regressions.length} case(s) passed in the baseline and no longer pass`);
  return { ok: reasons.length === 0, reasons, regressions };
}

function parseArgs(argv) {
  const out = { minPassRate: 1, timeoutMin: 20 };
  for (let i = 0; i < argv.length; i++) {
    const a = argv[i];
    const next = () => argv[++i];
    if (a === '--env') out.env = next();
    else if (a === '--bot') out.bot = next();
    else if (a === '--set') out.set = next();
    else if (a === '--min-pass-rate') out.minPassRate = Number(next());
    else if (a === '--baseline') out.baseline = next();
    else if (a === '--save-baseline') out.saveBaseline = next();
    else if (a === '--timeout-min') out.timeoutMin = Number(next());
    else if (a === '--connection') out.connection = next();
    else if (a === '--published') out.published = true;
    else throw new GateError(`Unknown argument ${a}`);
  }
  if (!out.env || !out.bot || !out.set) throw new GateError('Required: --env, --bot and --set.');
  if (!(out.minPassRate >= 0 && out.minPassRate <= 1)) throw new GateError('--min-pass-rate must be between 0 and 1.');
  return out;
}

export async function main(argv, { env = process.env, fetchImpl = fetch, sleep, log = console.log } = {}) {
  try {
    const args = parseArgs(argv);
    const client = makeClient({
      base: env.PP_API_BASE || DEFAULT_BASE,
      token: env.POWER_PLATFORM_TOKEN,
      environmentId: args.env,
      botId: args.bot,
      fetchImpl,
    });

    const testSet = await findTestSet(client, args.set);
    log(`Test set "${testSet.displayName}" (${testSet.totalTestCases} cases) on the ${args.published ? 'published' : 'DRAFT'} agent`);
    const started = await startRun(client, testSet.id, {
      name: `gate ${new Date().toISOString()}`,
      published: args.published,
      mcsConnectionId: args.connection,
    });
    const details = await waitForRun(client, started.runId, { timeoutMs: args.timeoutMin * 60_000, sleep, pollMs: sleep ? 0 : 15_000 });

    const summary = summarize(details);
    const baseline = args.baseline ? JSON.parse(readFileSync(args.baseline, 'utf8')) : {};
    const verdict = evaluateGate(details, summary, { minPassRate: args.minPassRate, baseline });

    log(`${summary.passed}/${summary.total} cases passed (${(summary.passRate * 100).toFixed(1)}%)`);
    for (const f of summary.failed) log(`  NOT PASSING ${f.testCaseId} [${f.statuses.join(', ') || 'no result'}] ${f.reason}`);
    for (const id of verdict.regressions) log(`  REGRESSED   ${id}`);
    if (verdict.ok) {
      if (args.saveBaseline) writeFileSync(args.saveBaseline, JSON.stringify(summary.byCase, null, 2));
      log('GATE PASSED');
      return 0;
    }
    for (const r of verdict.reasons) log(`GATE FAILED: ${r}`);
    return 1;
  } catch (err) {
    log(`ERROR: ${err.message}`);
    return 2;
  }
}

if (process.argv[1] && process.argv[1].endsWith('eval-gate.mjs')) {
  process.exit(await main(process.argv.slice(2)));
}

Usage:

POWER_PLATFORM_TOKEN="$TOKEN" node eval-gate.mjs \
  --env "$ENVIRONMENT_ID" --bot "$BOT_ID" \
  --set "Order desk regression" \
  --published \
  --min-pass-rate 0.95 \
  --baseline baseline.json --save-baseline baseline.json

Exit code 0 means the gate passed, 1 means it failed, and 2 means the run could not be completed (an API error, a timeout, bad input). Keeping 1 and 2 apart matters in a pipeline: a regression and a broken pipeline are different problems with different owners.

--save-baseline only writes the file after a passing gate. If it wrote after every run, one bad release would quietly become the new normal.

What the Tests Cover

The test file starts a local HTTP server that implements the documented endpoints, then drives the real main function against it:

// Runs eval-gate.mjs against a local mock of the documented evaluation API, so the polling,
// pass/fail logic and exit codes are tested without a tenant.
import assert from 'node:assert/strict';
import http from 'node:http';
import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { main } from './eval-gate.mjs';

const ENV = '11111111-1111-1111-1111-111111111111';
const BOT = '22222222-2222-2222-2222-222222222222';
const SET_ID = '33333333-3333-3333-3333-333333333333';
const TOKEN = 'secret-token-value';
const PREFIX = `/copilotstudio/environments/${ENV}/bots/${BOT}/api/makerevaluation`;

const caseResult = (id, status, reason = '') => ({
  testCaseId: id,
  state: 'Completed',
  metricsResults: [{ type: 'GeneralQuality', status, aiResultReason: reason, result: { data: {} } }],
});

// scenario = { states: ['Queued', 'InProgress', 'Completed'], cases: [...], testSets?: [...], status? }
function startMock(scenario) {
  const seen = [];
  let polls = 0;
  const server = http.createServer((req, res) => {
    let body = '';
    req.on('data', (c) => (body += c));
    req.on('end', () => {
      const url = new URL(req.url, 'http://x');
      seen.push({ method: req.method, path: url.pathname, version: url.searchParams.get('api-version'), auth: req.headers.authorization, body: body ? JSON.parse(body) : null });
      const send = (code, obj) => { res.writeHead(code, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };
      if (scenario.httpStatus) return send(scenario.httpStatus, { error: { message: 'nope' } });
      if (req.method === 'GET' && url.pathname === `${PREFIX}/testsets`) {
        return send(200, { value: scenario.testSets ?? [{ id: SET_ID, displayName: 'Order desk regression', state: 'Active', totalTestCases: scenario.cases.length }] });
      }
      if (req.method === 'POST' && url.pathname === `${PREFIX}/testsets/${SET_ID}/run`) {
        return send(200, { runId: 'run-1', state: 'Queued', executionState: 'Queued', totalTestCases: scenario.cases.length, testCasesProcessed: 0 });
      }
      if (req.method === 'GET' && url.pathname === `${PREFIX}/testruns/run-1`) {
        const state = scenario.states[Math.min(polls++, scenario.states.length - 1)];
        return send(200, { id: 'run-1', state, testSetId: SET_ID, testCasesResults: state === 'Completed' ? scenario.cases : [] });
      }
      return send(404, { error: 'unexpected ' + req.method + ' ' + url.pathname });
    });
  });
  return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ server, seen, base: `http://127.0.0.1:${server.address().port}` })));
}

async function run(scenario, args, extraEnv = {}) {
  const { server, seen, base } = await startMock(scenario);
  const lines = [];
  const code = await main(
    ['--env', ENV, '--bot', BOT, '--set', 'Order desk regression', ...args],
    { env: { POWER_PLATFORM_TOKEN: TOKEN, PP_API_BASE: base, ...extraEnv }, sleep: async () => {}, log: (l) => lines.push(l) },
  );
  server.close();
  return { code, seen, out: lines.join('\n') };
}

const allPass = [caseResult('a', 'Pass'), caseResult('b', 'Pass'), caseResult('c', 'Pass')];

// 1. All pass: polls through Queued/InProgress, exits 0, and the requests match the documented API.
let r = await run({ states: ['Queued', 'InProgress', 'Completed'], cases: allPass }, ['--published']);
assert.equal(r.code, 0, r.out);
assert.match(r.out, /3\/3 cases passed/);
assert.ok(r.seen.every((q) => q.version === '2024-10-01'), 'every call carries api-version 2024-10-01');
assert.ok(r.seen.every((q) => q.auth === `Bearer ${TOKEN}`), 'every call is authorised');
const start = r.seen.find((q) => q.method === 'POST');
assert.equal(start.body.runOnPublishedBot, true, '--published must set runOnPublishedBot');
assert.ok(typeof start.body.evaluationRunName === 'string');
assert.equal(r.seen.filter((q) => q.path.endsWith('/testruns/run-1')).length, 3, 'polled until Completed');

// 2. Default is the DRAFT agent, because that is the API default and must never be implied.
r = await run({ states: ['Completed'], cases: allPass }, []);
assert.equal(r.seen.find((q) => q.method === 'POST').body.runOnPublishedBot, false);
assert.match(r.out, /DRAFT/);

// 3. One failing case with a pass-rate threshold below it still passes, but the failure is reported.
r = await run({ states: ['Completed'], cases: [caseResult('a', 'Pass'), caseResult('b', 'Pass'), caseResult('c', 'Fail', 'Gave a refund')] }, ['--min-pass-rate', '0.6']);
assert.equal(r.code, 0);
assert.match(r.out, /NOT PASSING c.*Gave a refund/);

// 4. The same run fails the default threshold (100%).
r = await run({ states: ['Completed'], cases: [caseResult('a', 'Pass'), caseResult('b', 'Fail')] }, []);
assert.equal(r.code, 1);
assert.match(r.out, /below the required 100.0%/);

// 5. Flat pass rate, one swap: case a regresses while case c is fixed. Only the baseline comparison catches it.
const dir = mkdtempSync(join(tmpdir(), 'gate-'));
const baselineFile = join(dir, 'baseline.json');
writeFileSync(baselineFile, JSON.stringify({ a: true, b: true, c: false }));
r = await run({ states: ['Completed'], cases: [caseResult('a', 'Fail'), caseResult('b', 'Pass'), caseResult('c', 'Pass')] }, ['--min-pass-rate', '0.6', '--baseline', baselineFile]);
assert.equal(r.code, 1, r.out);
assert.match(r.out, /REGRESSED\s+a/);
assert.match(r.out, /passed in the baseline and no longer pass/);

// 6. --save-baseline writes only after a passing gate.
const saved = join(dir, 'saved.json');
r = await run({ states: ['Completed'], cases: allPass }, ['--save-baseline', saved]);
assert.deepEqual(JSON.parse(readFileSync(saved, 'utf8')), { a: true, b: true, c: true });

// 7. A case with no result, an Error status or an unknown status is not a pass.
const odd = [{ testCaseId: 'x', state: 'Completed', metricsResults: [] }, caseResult('y', 'Error'), caseResult('z', 'Mystery')];
r = await run({ states: ['Completed'], cases: odd }, ['--min-pass-rate', '0']);
assert.match(r.out, /0\/3 cases passed/);

// 8. Terminal failure states fail the gate instead of reading empty results as success.
r = await run({ states: ['Queued', 'Failed'], cases: [] }, []);
assert.equal(r.code, 1);
assert.match(r.out, /run ended as Failed/);

// 9. A run that never finishes is an error (exit 2), not a pass or a hang.
r = await run({ states: ['InProgress'], cases: allPass }, ['--timeout-min', '0']);
assert.equal(r.code, 2);
assert.match(r.out, /Timed out/);

// 10. API errors exit 2 and never print the token.
r = await run({ states: ['Completed'], cases: allPass, httpStatus: 401 }, []);
assert.equal(r.code, 2);
assert.match(r.out, /HTTP 401/);
assert.ok(!r.out.includes(TOKEN), 'token leaked into output');

// 11. Unknown or ambiguous test set names are refused.
r = await run({ states: ['Completed'], cases: allPass, testSets: [{ id: 'z', displayName: 'Other', state: 'Active' }] }, []);
assert.equal(r.code, 2);
assert.match(r.out, /No active test set named/);
r = await run({ states: ['Completed'], cases: allPass, testSets: [{ id: 'z1', displayName: 'Order desk regression', state: 'Active' }, { id: 'z2', displayName: 'Order desk regression', state: 'Active' }] }, []);
assert.match(r.out, /More than one active test set/);

// 12. Missing token and bad arguments are reported before any network call.
const noToken = await main(['--env', ENV, '--bot', BOT, '--set', 'x'], { env: {}, log: () => {} });
assert.equal(noToken, 2);

console.log('eval-gate: all checks passed');

Twelve scenarios run in a couple of seconds. They check that every request carries api-version=2024-10-01 and the bearer token, that --published sets runOnPublishedBot, that the default really is the draft, that polling continues through Queued and InProgress, that a flat pass rate with one swapped case still fails on the baseline rule, that Error, an empty result and an unknown status do not count as passes, that a terminal Failed run fails the gate, that a timeout exits 2, that an API error never prints the token, and that an unknown or duplicate test set name is refused.

To check the tests could fail, I broke the code four ways: ignoring regressions, treating only Completed as terminal, hard-coding runOnPublishedBot, and counting empty results as passes. All four made the tests fail. One of them (a missing terminal state) made the polling loop spin until its 20-minute timeout instead of failing fast, a reminder that the timeout is doing real work.

What the mock cannot tell you is whether your tenant's responses match the documentation. That is the first thing to confirm.

Wiring It Into a Pipeline

A minimal GitHub Actions job, shown as a sketch rather than something I ran:

name: agent-release-gate
on:
  workflow_dispatch:
  pull_request:
    paths: ['agents/order-desk/**']

# One evaluation at a time: the portal runs a single evaluation test set at a time,
# and the API's own limits are not documented, so serialize runs.
concurrency:
  group: copilot-studio-evaluation
  cancel-in-progress: false

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - name: Get a Power Platform API token
        id: token
        run: |
          RESPONSE=$(curl -s -X POST "https://login.microsoftonline.com/${{ secrets.TENANT_ID }}/oauth2/v2.0/token" \
            -d "client_id=${{ secrets.CLIENT_ID }}" \
            -d "client_secret=${{ secrets.CLIENT_SECRET }}" \
            -d "scope=https://api.powerplatform.com/.default" \
            -d "grant_type=client_credentials")
          echo "::add-mask::$(echo "$RESPONSE" | jq -r .access_token)"
          echo "token=$(echo "$RESPONSE" | jq -r .access_token)" >> "$GITHUB_OUTPUT"
      - name: Evaluate
        env:
          POWER_PLATFORM_TOKEN: ${{ steps.token.outputs.token }}
        run: node eval-gate.mjs --env "${{ vars.ENVIRONMENT_ID }}" --bot "${{ vars.BOT_ID }}" --set "Order desk regression" --min-pass-rate 0.95 --baseline baseline.json

The token request is the documented client-credentials form. Whether a service principal is accepted by the evaluation operations is exactly the open question from the authentication section, so run the list-test-sets call by hand first.

Pitfalls

Evaluations are not free. The standard harness documentation I read does not give an evaluation-specific price, so check how your licensing treats them. For agents on the GitHub Copilot harness, Microsoft's billing overview is explicit that generating and creating agent evaluations consume Copilot Credits, along with previewing and testing. A gate that runs on every pull request can add up, so run the full set on a schedule and a small must-pass subset on each change. The companion post on credits covers the arithmetic.

Results expire. Results stay in Copilot Studio for 89 days. Your baseline file in the repository is the durable record, not the portal.

A test set is a contract. If someone edits it, the pass rate changes without the agent changing. Treat edits like code changes, and consider recording the test set's totalTestCases in the gate output so a shrinking set is visible.

Test the same thing you ship. If your pipeline publishes and then evaluates, use --published. If it evaluates before publishing, leave it off. Do not mix them in one gate.

A Rollout Order That Works

  1. Build the test set from real questions, including the refusals. The earlier post on model changes covers how.
  2. Run it by hand in the portal until the result is stable
  3. Call the list-test-sets endpoint with your pipeline identity to prove the auth path
  4. Run the gate once with a low threshold and --save-baseline to record where you are
  5. Raise the threshold and turn on the baseline rule
  6. Make the job required for merges to the agent's branch

The point is not a clever script. It is that a quality bar written down and enforced by a machine stays in place when the team is busy, which a promise to check the evaluation page does not.

Sources

  • Automate evaluations with the Power Platform API, Microsoft Learn, updated 3 August 2026
  • Bots: Run Maker Evaluation Test Set, Power Platform REST API reference
  • Programmability and extensibility: Authentication, Microsoft Learn
  • Run evaluations and view results, Microsoft Learn
  • Overview of usage-based billing (GitHub Copilot harness), Microsoft Learn, updated 28 September 2026
  • What's new in Copilot Studio, Microsoft Learn
  1. AJ's Tech Notes
  2. Gate Copilot Studio Releases with the Evaluation API