23 Test the outcome as well as implementation

The download test passes. The expected bytes exist. The checksum matches. A user selects the resource and receives “not available.” Every preceding observation can be true. The missing link is between what the implementation checked and what the user was promised.

Testing an outcome begins with the action that should become possible. In the running example, a consumer must open the selected replacement through the supported interface after publication. Before that boundary, the old version must remain usable. A test that inspects only a staging directory cannot establish either claim.

This does not make lower-level tests unimportant. They locate defects and exercise cases that would be expensive through the whole application. The challenge is to build an evidence chain whose links answer different questions, rather than accumulate many assertions about the same intermediate event.

Name the oracle

A test oracle is the basis for deciding whether an observed result is correct. Sometimes it is a simple expected value. Sometimes it is a domain invariant, a known implementation, or a person's judgment about behavior. Whatever its form, it can be wrong.

For acquisition, “the returned status equals complete” is a weak oracle if the implementation itself decides when to return that status. A stronger acceptance observation asks the real consumer to open the requested identity and checks the content or behavior it receives. The status can then be checked against that independent consequence.

Independence is practical rather than absolute. The consumer might share the same faulty catalog code. A checksum might come from the same untrusted source as the bytes. A second implementation might preserve a defect in the first. State what each oracle depends on, especially where agreement could reflect a shared mistake.

Test both sides of a boundary

A useful publication test is a sequence, not just an eventual success assertion. In the illustrative application:

1. Install a known old version and prove the consumer can use it
2. Begin preparing a valid replacement, then hold execution immediately before publication
3. Confirm that a new consumer request still resolves to the old version and that status does not claim the replacement is available
4. Release publication and confirm that a new request resolves to the selected replacement
5. Repeat with an invalid candidate and confirm that the old version remains usable after the failure

The controlled hold makes a normally short interval observable. Without it, a fast machine may race past the defect so that a polling test never sees it. The barrier must be released during teardown even when an assertion fails; otherwise the test can strand the worker it was meant to inspect.

Pumas provides a bounded real counterpart. At draft head 42a198bf, a public HF status test holds the importer at a metadata-write barrier. The input files already exist, but the test requires nonterminal status and no completion receipt. After release, it checks completion alongside the receipt, model/index presence, adopted acquisition state, and queue release. The repository reports a focused pass; the test was not rerun for this book. It exercises a controlled Linux public-core/DTO path, not JSON-RPC, Electron, or every supported platform. Public-boundary test, recorded execution and limits.

The transferable technique is to make the consequential interval testable and inspect what consumers can observe there. The particular internal state names belong to that repository.

Include the actual entry point

A function-level test can establish a sound core while a wrapper mishandles units, errors, or completion. At least one acceptance path should therefore use the interface the relevant consumer actually calls. For a command-line tool, invoke the command. For an API, exercise the protocol boundary. For a desktop interaction, inspect the interface and its behavior under the intended conditions.

Simon Willison's account of agentic manual testing advocates this complementary practice: start the system, invoke its interfaces, inspect observed output, and convert useful discoveries into regressions. His Showboat approach records commands and output rather than relying only on an agent's narrative. Such a demonstration covers the behavior exercised; it is not a proof of unvisited paths. Agentic manual testing.

Exploration and repeatable tests serve different purposes. Exploration can reveal a confusing status transition nobody thought to specify. A regression can make that discovered requirement durable. Automating every exploratory observation immediately may be wasteful; leaving a consequential repeated failure dependent on memory is wasteful in another way.

Make a passing suite vulnerable to a meaningful fault

Ask what deliberate defect should cause the suite to fail. Move the availability signal before publication. Replace the requested identity with a different valid resource. Delete the old active version before checking the candidate. If a suite still passes, it does not establish the corresponding claim, however large its test count.

This is a useful local mutation exercise, not a demand to construct an exhaustive mutation-testing framework. Choose a fault tied to a material promise. The result often exposes that several tests copied the implementation's assumptions rather than checking an external consequence.

Acceptance can also require human observation. A technically accurate progress display might be misleading because the user reasonably interprets a finished bar as permission to close the application. Describe that interpretation and inspect the actual flow. Human judgment should address the unresolved experience, not replace checks that the machine can perform precisely.

Build one complete evidence chain

Take a change whose tests already pass. Write the promised consumer action, the observation that proves it became possible, and one plausible defect the current tests could miss. Add or perform the smallest check that distinguishes the defect from the desired behavior.

Record the exercised revision and environment, then state one important limit. The observable outcome is an acceptance claim supported by a relevant observation, including a failure case. A larger passing count is optional; a clearer connection to the intended result is the point.