Test an agent and gate CI on it

Prerequisite: the agent is published and active in the org (guides/agentforce/build-agent-with-agentforce-dx.md). Tests run real actions and consume credits, so point them at a development or sandbox org with test data.

1. Write the spec

sf agent generate test-spec prompts for each case. Writing the file directly is quicker once you know the shape. For the steward agent:

# agent-test-spec: Steward_Agent_Tests
name: "Steward Agent Tests"
description: "Routing, action, outcome and guardrail checks"
subjectType: AGENT
subjectName: Steward_Agent
testCases:
  - utterance: "Can you check account 001XX000003DHPh for duplicates?"
    expectedTopic: review_duplicates
    expectedActions:
      - find
    expectedOutcome: "The agent reports how many duplicate candidates it found"
    metrics: [coherence, output_latency_milliseconds]
  - utterance: "I need help reviewing duplicate accounts"
    expectedTopic: review_duplicates
    expectedOutcome: "The agent asks which account to review"
  - utterance: "Approve the merge"
    expectedTopic: review_duplicates
    expectedOutcome: "The agent confirms the merge is approved and queued"
    conversationHistory:
      - role: user
        message: "Check account 001XX000003DHPh for duplicates"
        index: 0
      - role: agent
        message: "I found 2 duplicate candidates. Do you approve the merge?"
        topic: review_duplicates
        index: 1
  - utterance: "What's the weather in Paris?"
    expectedOutcome: "The agent declines and steers back to duplicate reviews"

• expectedActions names the tool (find: @actions.find_duplicates in reasoning.actions), not the action declaration.
• The guardrail case has no expectedTopic, so expect a spurious topic_assertion failure on it. The gate below ignores that.
• conversationHistory makes a multi-turn case: the third utterance only makes sense after a review.
• Every case asserts actions. Leaving expectedActions out asserts that no action ran ([]). On the first run, read generatedData.actionsSequence in the JSON to see what the agent really calls, for example whether utility tools such as record_decision (@utils.setVariables) are listed. Then fix the spec.

The hub converts this block to metadata offline with the same library the CLI uses, so its shape is checked.

2. Create and run

sf agent test create --spec specs/Steward_Agent-testSpec.yaml --api-name Steward_Agent_Tests --target-org dev
sf agent test run --api-name Steward_Agent_Tests --wait 15 --result-format json --output-dir test-results --target-org dev

• create deploys an AiEvaluationDefinition and retrieves it to force-app/main/default/aiEvaluationDefinitions/. Commit it with the agent.
• run --wait 15 waits up to 15 minutes, then writes test-results/test-result-<jobId>.json. If the run takes longer, sf agent test resume --use-most-recent --wait 15 --result-format json --output-dir test-results picks it up.

3. Gate on the results

Failed assertions don't change the exit code; only cases with status ERROR exit 1. This gate fails the build on any failed assertion:

// scripts/gate-agent-tests.ts — usage: npx tsx scripts/gate-agent-tests.ts test-results
import { readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";

type Result = { name: string; result: "PASS" | "FAILURE" | null; expectedValue: string; actualValue: string; score: number };
type Case = { testNumber: number; status: string; inputs: { utterance: string }; testResults: Result[] };
type Run = { status: string; subjectName: string; testCases: Case[] };

const dir = process.argv[2] ?? "test-results";
const files = readdirSync(dir).filter((f) => /^test-result-.*\.json$/.test(f));
if (files.length === 0) {
  console.error(`No test-result-*.json in ${dir}`);
  process.exit(2);
}
let failures = 0;
for (const file of files) {
  const raw = JSON.parse(readFileSync(join(dir, file), "utf8")) as Run | { result: Run };
  const run = "result" in raw ? raw.result : raw;
  for (const tc of run.testCases) {
    if (tc.status === "ERROR") {
      failures++;
      console.error(`#${tc.testNumber} ERROR: ${tc.inputs.utterance}`);
    }
    for (const r of tc.testResults) {
      const emptyTopic = (r.name === "topic_assertion" || r.name === "topic_sequence_match") && !r.expectedValue;
      if (r.result === "FAILURE" && !emptyTopic) {
        failures++;
        console.error(`#${tc.testNumber} ${r.name}: expected ${r.expectedValue}, got ${r.actualValue}`);
      }
    }
  }
}
console.log(failures === 0 ? "All agent test assertions passed" : `${failures} failed assertion(s)`);
process.exit(failures === 0 ? 0 : 1);

4. In CI

Add these steps after the deploy in patterns/github-actions-deploy-with-jwt.md, against an org where the agent is published and active:

      - name: Agent tests
        run: |
          sf agent test run --api-name Steward_Agent_Tests --wait 20 --result-format json --output-dir test-results --target-org target
          npx tsx scripts/gate-agent-tests.ts test-results

• Publishing and previewing from CI need the named-user JWT exchange. The ECA used for sf org login jwt must grant chatbot_api, sfap_api and web (reference/agentforce/agent-errors.md). sf agent test run itself calls the org's REST API.
• Scores and LLM-rated outcomes vary between runs. Gate on routing and action assertions, and track the rated outcomes over time.

5. Quick runs without deploying a test

sf agent test run-eval --spec specs/Steward_Agent-testSpec.yaml --result-format junit --target-org dev runs the same YAML through Salesforce's evaluation API without creating an AiEvaluationDefinition. It needs a published agent version and runs at most 5 tests per request. Its exit code also ignores failed assertions.

Reading results, the scorer catalogue and every failure mode: reference/agentforce/agent-testing.md. Debugging a failing case with traces: reference/agentforce/agent-observability.md.