--- cuppy_hub: salesforce cuppy_path: reference/agentforce/agent-testing.md cuppy_section: docs cuppy_doc: 801F-SHW7 cuppy_doc_uuid: 0d6e0160-81a7-46f5-94cb-7d57ec72c44d cuppy_permalink: https://cuppy.io/salesforce/d/801F-SHW7 cuppy_version: 1 cuppy_updated: 2026-10-23T16:16:52.575Z cuppy_canonical: https://cuppy.io/salesforce/docs/reference/agentforce/agent-testing cuppy_mcp: https://cuppy.io/mcp cuppy_body_sha256: bdb45f3c768b276735ffa0fbce7ec62c826be078a3e9a98ca5b85cdb380deb64 # cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks. --- # Agent testing A test is metadata in the org: a named set of utterances, each with expectations, run against a **published, active** agent. You write a YAML spec, `sf agent test create` turns it into metadata and deploys it, and `sf agent test run` runs it. The end-to-end workflow: `guides/agentforce/test-an-agent.md`. Quick checks while authoring use preview instead (`reference/agentforce/agent-observability.md`). ## Two runners | `--test-runner` | Metadata | Spec shape | Status values | |---|---|---|---| | `testing-center` (default for `create`) | `AiEvaluationDefinition` (`aiEvaluationDefinitions/`) | `testCases[]` with `utterance`, `expectedTopic`, `expectedActions`, `expectedOutcome`, `metrics` | `NEW`, `IN_PROGRESS`, `COMPLETED`, `ERROR`, `TERMINATED` | | `agentforce-studio` (beta) | `AiTestingDefinition` (`aiTestingDefinitions/`) | `testCases[]` with `inputs[]` and `scorers[]` | `CREATED`, `IN_PROGRESS`, `SUCCESS`, `FAILED`, `TERMINATED` | Salesforce's DX testing pages document only the Testing Center process and say the Agentforce Studio runner's documentation is coming. `run`, `resume` and `results` detect the runner from the org; `--test-runner` overrides it. ## Testing Center spec → metadata | Spec field | Becomes the expectation | Notes | |---|---|---| | `expectedTopic` | `topic_sequence_match` (always added, empty when omitted) | The **subagent** name. Promoted standard topics have runtime names with a hash suffix that changes on republish | | `expectedActions: [a, b]` | `action_sequence_match` (stored as `['a','b']`) | Use the **tool** name from `reasoning.actions`, not the action's declaration name. **Always added:** without `expectedActions` the case asserts `[]`, meaning no action ran | | `expectedOutcome` | `bot_response_rating` | Rated by an LLM | | `metrics` | `coherence`, `completeness`, `conciseness`, `output_latency_milliseconds` | Scores, not pass or fail | | `contextVariables: [{name, value}]` | `inputs.contextVariable` | For example `RoutableId`, `CaseId` | | `conversationHistory` | `inputs.conversationHistory` | Turns before the utterance: `role: user` or `role: agent` with a `topic` | | `customEvaluations` | Custom expectations | Compare a JSONPath into the response (`isReference: true`) with an operator | The spec also needs `name`, `subjectType: AGENT` and `subjectName` (the agent's API name), and optionally `subjectVersion` such as `v1`. **Agentforce Studio scorers:** | Grade | Scorers | |---|---| | Pass or fail (need `expected`) | `topic_sequence_match`, `action_sequence_match`, `agent_handoff_match` | | LLM pass or fail (need `expected`) | `bot_response_rating`, `response_match` | | LLM score 0–100 | `coherence`, `conciseness`, `factuality`, `completeness` | | LLM score 0–5 | `task_resolution`, which needs `conversationHistory` | | Numeric | `output_latency_milliseconds` | Studio tests can also target a prompt template (`subjectType: PROMPT`). ## Commands | Command | Does | |---|---| | `sf agent generate test-spec --output-file specs/X.yaml` | Interactive spec from the agent metadata in your project (no org). Use `--from-definition ` to convert an existing definition back to YAML | | `sf agent test create --spec specs/X.yaml --api-name X_Tests --target-org dev` | Converts, deploys and retrieves the metadata. `--preview` writes the XML only; `--force-overwrite` replaces a test with the same name | | `sf agent test run --api-name X_Tests --wait 10 --result-format junit --output-dir test-results --target-org dev` | Starts a run. Without `--wait` it prints the job ID and returns | | `sf agent test resume --job-id ` (or `--use-most-recent`) / `sf agent test results --job-id ` | Waits for a run / reads a finished one. `results` has **no** `--use-most-recent`, although its help text mentions it | | `sf agent test list` | Tests in the org, with runner type | | `sf agent test run-eval --spec specs/X.yaml --target-org dev` | Runs the spec through the newer evaluation API (`api.salesforce.com/einstein/evaluation/v1/tests`) without deploying a test. It takes the same YAML or a JSON payload, runs at most 5 tests per request (`--batch-size`), and needs a published agent version | `--result-format` is `human`, `json`, `junit` or `tap`. With `--output-dir`, the file is `test-result-.`. ## Limits and rules - 1,000 test cases per `AiEvaluationDefinition`; up to **10 runs `IN-PROGRESS`** at a time. - Runs consume Einstein Requests (and possibly Data 360) credits. Actions run **for real**, so they can change data. - Salesforce's Testing API overview says testing is "available only in sandboxes", but its DX pages run tests in a development org. - Scores can drift: Salesforce warns results may change as the service improves. - The Testing Connect API, which `sf agent test` calls: - `POST /services/data/v66.0/einstein/ai-evaluations/runs` with `{"aiEvaluationDefinitionName": "X_Tests"}` starts a run (Salesforce's examples use v63.0); - `GET …/runs/{runId}` reads its status, and `GET …/runs/{runId}/results` its results; - the CLI's library also calls an undocumented `POST …/runs/{runId}/cancel`. ## Failures | Symptom | Cause → fix | |---|---| | CI passes although tests failed | **Failed assertions exit 0.** `run --wait`, `resume` and `run-eval` exit 1 only when a test case has status `ERROR`. Gate on the result file (`guides/agentforce/test-an-agent.md`). Salesforce's sf-skills troubleshooting table says exit 1 means failed tests; the CLI's source disagrees | | `action_sequence_match` fails, but the action ran | You expected the declaration name (`find_duplicates`) instead of the tool name (`find`) | | `topic_sequence_match` fails after a republish | A runtime topic name with a hash suffix changed. Read `generatedData.topic` from the JSON results and update the spec | | Off-topic (guardrail) cases fail `topic_assertion` | With no `expectedTopic`, the generated metadata still asserts an empty topic. Ignore that row for guardrail cases | | Actions never fire in a single-turn case | The agent asks for input first. Add `conversationHistory`, or assert `expectedOutcome` only | | `conciseness` is always 0 | A known bug per Salesforce's sf-skills. Use `coherence` | | Results API crashes with `isReference: true` custom evaluations | A known bug per sf-skills. Use `expectedOutcome` | | "No published agents ('GenAiPlannerBundle', 'GenAiPlugin', 'Bot') found in …" (`generate test-spec`) | Publish the agent and retrieve its metadata into the project first | | "No published version found for agent '…'" (`run-eval`) | Publish at least once (`guides/agentforce/build-agent-with-agentforce-dx.md`) | | Run never starts, or `ERROR` | The agent isn't active: `sf agent activate` | | "Job ID '…' not found or invalid." | A wrong ID or the wrong org | --- You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 376 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/salesforce/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.