---
cuppy_hub: salesforce
cuppy_path: guides/agentforce/test-an-agent.md
cuppy_section: docs
cuppy_doc: SAKH-ZEV3
cuppy_doc_uuid: b3a9089b-076c-420b-9960-794537e2ffa4
cuppy_permalink: https://cuppy.io/salesforce/d/SAKH-ZEV3
cuppy_version: 1
cuppy_updated: 2026-10-23T16:16:52.575Z
cuppy_canonical: https://cuppy.io/salesforce/docs/guides/agentforce/test-an-agent
cuppy_mcp: https://cuppy.io/mcp
cuppy_body_sha256: ba596cc0ac92810bb87a25673f1b2777e8b7a52e46257de96dc569cdcad4bc88
# cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks.
---


# Test an agent and gate CI on it

Prerequisite: the agent is published and **active** in the org (`guides/agentforce/build-agent-with-agentforce-dx.md`). Tests run real actions and consume credits, so point them at a development or sandbox org with test data.

## 1. Write the spec

`sf agent generate test-spec` prompts for each case. Writing the file directly is quicker once you know the shape. For the steward agent:

```yaml
# agent-test-spec: Steward_Agent_Tests
name: "Steward Agent Tests"
description: "Routing, action, outcome and guardrail checks"
subjectType: AGENT
subjectName: Steward_Agent
testCases:
  - utterance: "Can you check account 001XX000003DHPh for duplicates?"
    expectedTopic: review_duplicates
    expectedActions:
      - find
    expectedOutcome: "The agent reports how many duplicate candidates it found"
    metrics: [coherence, output_latency_milliseconds]
  - utterance: "I need help reviewing duplicate accounts"
    expectedTopic: review_duplicates
    expectedOutcome: "The agent asks which account to review"
  - utterance: "Approve the merge"
    expectedTopic: review_duplicates
    expectedOutcome: "The agent confirms the merge is approved and queued"
    conversationHistory:
      - role: user
        message: "Check account 001XX000003DHPh for duplicates"
        index: 0
      - role: agent
        message: "I found 2 duplicate candidates. Do you approve the merge?"
        topic: review_duplicates
        index: 1
  - utterance: "What's the weather in Paris?"
    expectedOutcome: "The agent declines and steers back to duplicate reviews"
```

- `expectedActions` names the **tool** (`find: @actions.find_duplicates` in `reasoning.actions`), not the action declaration.
- The guardrail case has no `expectedTopic`, so expect a spurious `topic_assertion` failure on it. The gate below ignores that.
- `conversationHistory` makes a multi-turn case: the third utterance only makes sense after a review.
- **Every case asserts actions.** Leaving `expectedActions` out asserts that **no** action ran (`[]`). On the first run, read `generatedData.actionsSequence` in the JSON to see what the agent really calls, for example whether utility tools such as `record_decision` (`@utils.setVariables`) are listed. Then fix the spec.

The hub converts this block to metadata offline with the same library the CLI uses, so its shape is checked.

## 2. Create and run

```bash
sf agent test create --spec specs/Steward_Agent-testSpec.yaml --api-name Steward_Agent_Tests --target-org dev
sf agent test run --api-name Steward_Agent_Tests --wait 15 --result-format json --output-dir test-results --target-org dev
```

- `create` deploys an `AiEvaluationDefinition` and retrieves it to `force-app/main/default/aiEvaluationDefinitions/`. Commit it with the agent.
- `run --wait 15` waits up to 15 minutes, then writes `test-results/test-result-<jobId>.json`. If the run takes longer, `sf agent test resume --use-most-recent --wait 15 --result-format json --output-dir test-results` picks it up.

## 3. Gate on the results

Failed assertions **don't** change the exit code; only cases with status `ERROR` exit 1. This gate fails the build on any failed assertion:

```ts
// scripts/gate-agent-tests.ts — usage: npx tsx scripts/gate-agent-tests.ts test-results
import { readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";

type Result = { name: string; result: "PASS" | "FAILURE" | null; expectedValue: string; actualValue: string; score: number };
type Case = { testNumber: number; status: string; inputs: { utterance: string }; testResults: Result[] };
type Run = { status: string; subjectName: string; testCases: Case[] };

const dir = process.argv[2] ?? "test-results";
const files = readdirSync(dir).filter((f) => /^test-result-.*\.json$/.test(f));
if (files.length === 0) {
  console.error(`No test-result-*.json in ${dir}`);
  process.exit(2);
}
let failures = 0;
for (const file of files) {
  const raw = JSON.parse(readFileSync(join(dir, file), "utf8")) as Run | { result: Run };
  const run = "result" in raw ? raw.result : raw;
  for (const tc of run.testCases) {
    if (tc.status === "ERROR") {
      failures++;
      console.error(`#${tc.testNumber} ERROR: ${tc.inputs.utterance}`);
    }
    for (const r of tc.testResults) {
      const emptyTopic = (r.name === "topic_assertion" || r.name === "topic_sequence_match") && !r.expectedValue;
      if (r.result === "FAILURE" && !emptyTopic) {
        failures++;
        console.error(`#${tc.testNumber} ${r.name}: expected ${r.expectedValue}, got ${r.actualValue}`);
      }
    }
  }
}
console.log(failures === 0 ? "All agent test assertions passed" : `${failures} failed assertion(s)`);
process.exit(failures === 0 ? 0 : 1);
```

## 4. In CI

Add these steps after the deploy in `patterns/github-actions-deploy-with-jwt.md`, against an org where the agent is published and active:

```yaml
      - name: Agent tests
        run: |
          sf agent test run --api-name Steward_Agent_Tests --wait 20 --result-format json --output-dir test-results --target-org target
          npx tsx scripts/gate-agent-tests.ts test-results
```

- **Publishing and previewing from CI** need the named-user JWT exchange. The ECA used for `sf org login jwt` must grant `chatbot_api`, `sfap_api` and `web` (`reference/agentforce/agent-errors.md`). `sf agent test run` itself calls the org's REST API.
- Scores and LLM-rated outcomes vary between runs. Gate on routing and action assertions, and track the rated outcomes over time.

## 5. Quick runs without deploying a test

`sf agent test run-eval --spec specs/Steward_Agent-testSpec.yaml --result-format junit --target-org dev` runs the same YAML through Salesforce's evaluation API without creating an `AiEvaluationDefinition`. It needs a published agent version and runs at most 5 tests per request. Its exit code also ignores failed assertions.

Reading results, the scorer catalogue and every failure mode: `reference/agentforce/agent-testing.md`. Debugging a failing case with traces: `reference/agentforce/agent-observability.md`.


---
<!-- cuppy -->
You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 376 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/salesforce/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.
<!-- /cuppy -->