Agent testing

A test is metadata in the org: a named set of utterances, each with expectations, run against a published, active agent. You write a YAML spec, sf agent test create turns it into metadata and deploys it, and sf agent test run runs it. The end-to-end workflow: guides/agentforce/test-an-agent.md. Quick checks while authoring use preview instead (reference/agentforce/agent-observability.md).

Two runners

`--test-runner`	Metadata	Spec shape	Status values
`testing-center` (default for `create`)	`AiEvaluationDefinition` (`aiEvaluationDefinitions/`)	`testCases[]` with `utterance`, `expectedTopic`, `expectedActions`, `expectedOutcome`, `metrics`	`NEW`, `IN_PROGRESS`, `COMPLETED`, `ERROR`, `TERMINATED`
`agentforce-studio` (beta)	`AiTestingDefinition` (`aiTestingDefinitions/`)	`testCases[]` with `inputs[]` and `scorers[]`	`CREATED`, `IN_PROGRESS`, `SUCCESS`, `FAILED`, `TERMINATED`

Salesforce's DX testing pages document only the Testing Center process and say the Agentforce Studio runner's documentation is coming. run, resume and results detect the runner from the org; --test-runner overrides it.

Testing Center spec → metadata

Spec field	Becomes the expectation	Notes
`expectedTopic`	`topic_sequence_match` (always added, empty when omitted)	The subagent name. Promoted standard topics have runtime names with a hash suffix that changes on republish
`expectedActions: [a, b]`	`action_sequence_match` (stored as `['a','b']`)	Use the tool name from `reasoning.actions`, not the action's declaration name. Always added: without `expectedActions` the case asserts `[]`, meaning no action ran
`expectedOutcome`	`bot_response_rating`	Rated by an LLM
`metrics`	`coherence`, `completeness`, `conciseness`, `output_latency_milliseconds`	Scores, not pass or fail
`contextVariables: [{name, value}]`	`inputs.contextVariable`	For example `RoutableId`, `CaseId`
`conversationHistory`	`inputs.conversationHistory`	Turns before the utterance: `role: user` or `role: agent` with a `topic`
`customEvaluations`	Custom expectations	Compare a JSONPath into the response (`isReference: true`) with an operator

The spec also needs name, subjectType: AGENT and subjectName (the agent's API name), and optionally subjectVersion such as v1.

Agentforce Studio scorers:

Grade	Scorers
Pass or fail (need `expected`)	`topic_sequence_match`, `action_sequence_match`, `agent_handoff_match`
LLM pass or fail (need `expected`)	`bot_response_rating`, `response_match`
LLM score 0–100	`coherence`, `conciseness`, `factuality`, `completeness`
LLM score 0–5	`task_resolution`, which needs `conversationHistory`
Numeric	`output_latency_milliseconds`

Studio tests can also target a prompt template (subjectType: PROMPT).

Commands

Command	Does
`sf agent generate test-spec --output-file specs/X.yaml`	Interactive spec from the agent metadata in your project (no org). Use `--from-definition <file>` to convert an existing definition back to YAML
`sf agent test create --spec specs/X.yaml --api-name X_Tests --target-org dev`	Converts, deploys and retrieves the metadata. `--preview` writes the XML only; `--force-overwrite` replaces a test with the same name
`sf agent test run --api-name X_Tests --wait 10 --result-format junit --output-dir test-results --target-org dev`	Starts a run. Without `--wait` it prints the job ID and returns
`sf agent test resume --job-id <id>` (or `--use-most-recent`) / `sf agent test results --job-id <id>`	Waits for a run / reads a finished one. `results` has no `--use-most-recent`, although its help text mentions it
`sf agent test list`	Tests in the org, with runner type
`sf agent test run-eval --spec specs/X.yaml --target-org dev`	Runs the spec through the newer evaluation API (`api.salesforce.com/einstein/evaluation/v1/tests`) without deploying a test. It takes the same YAML or a JSON payload, runs at most 5 tests per request (`--batch-size`), and needs a published agent version

--result-format is human, json, junit or tap. With --output-dir, the file is test-result-<jobId>.<ext>.

Limits and rules

1,000 test cases per AiEvaluationDefinition; up to 10 runs IN-PROGRESS at a time.
Runs consume Einstein Requests (and possibly Data 360) credits. Actions run for real, so they can change data.
Salesforce's Testing API overview says testing is "available only in sandboxes", but its DX pages run tests in a development org.
Scores can drift: Salesforce warns results may change as the service improves.
The Testing Connect API, which sf agent test calls:
POST /services/data/v66.0/einstein/ai-evaluations/runs with {"aiEvaluationDefinitionName": "X_Tests"} starts a run (Salesforce's examples use v63.0);
GET …/runs/{runId} reads its status, and GET …/runs/{runId}/results its results;
the CLI's library also calls an undocumented POST …/runs/{runId}/cancel.

Failures

Symptom	Cause → fix
CI passes although tests failed	Failed assertions exit 0. `run --wait`, `resume` and `run-eval` exit 1 only when a test case has status `ERROR`. Gate on the result file (`guides/agentforce/test-an-agent.md`). Salesforce's sf-skills troubleshooting table says exit 1 means failed tests; the CLI's source disagrees
`action_sequence_match` fails, but the action ran	You expected the declaration name (`find_duplicates`) instead of the tool name (`find`)
`topic_sequence_match` fails after a republish	A runtime topic name with a hash suffix changed. Read `generatedData.topic` from the JSON results and update the spec
Off-topic (guardrail) cases fail `topic_assertion`	With no `expectedTopic`, the generated metadata still asserts an empty topic. Ignore that row for guardrail cases
Actions never fire in a single-turn case	The agent asks for input first. Add `conversationHistory`, or assert `expectedOutcome` only
`conciseness` is always 0	A known bug per Salesforce's sf-skills. Use `coherence`
Results API crashes with `isReference: true` custom evaluations	A known bug per sf-skills. Use `expectedOutcome`
"No published agents ('GenAiPlannerBundle', 'GenAiPlugin', 'Bot') found in …" (`generate test-spec`)	Publish the agent and retrieve its metadata into the project first
"No published version found for agent '…'" (`run-eval`)	Publish at least once (`guides/agentforce/build-agent-with-agentforce-dx.md`)
Run never starts, or `ERROR`	The agent isn't active: `sf agent activate`
"Job ID '…' not found or invalid."	A wrong ID or the wrong org