Skip to content

fix(structured): return a failure exit code so serve reports 503 - #1131

Merged
kgaughan merged 1 commit into
goss-org:masterfrom
VXNCXNX:fix/structured-failure-exit-code
Sep 13, 2026
Merged

kgaughan merged 1 commit into
goss-org:masterfrom
VXNCXNX:fix/structured-failure-exit-code

Conversation

@VXNCXNX

@VXNCXNX VXNCXNX commented Sep 13, 2026

Copy link
Copy Markdown
Contributor
Checklist
  • make test-all (UNIX) passes. CI will also test this
    • there is no test-all target in the Makefile, so I left this unticked rather than tick it for something else. make test-short-all (fmt, lint, vet, test) passes in full, as does go test ./... under -race -shuffle=on. golangci-lint run ./outputs/... reports 0 issues. The test-int-* targets need a Docker daemon and were not run; nothing under integration-tests/ or extras/ uses the structured format.
  • unit and/or integration tests are included (if applicable)
  • documentation is changed or added (if applicable)
    • docs/cli.md already says serve returns the results in the requested format and "an http status of 200 or 503", and that validate "Exits with status 0 on success, non-0 otherwise". This makes the code match both, rather than changing a documented contract.

Description of change

goss serve -f structured answers 200 on a failing suite, so a healthcheck built on it never fails.

Output in outputs/structured.go ends in an unconditional return 0, and serve.go maps a non-zero outputer code to 503 and zero to 200. Eight of the nine formats already return non-zero on failure. Summary.Failed is counted a few lines above, so the information was already there.

format /healthz on a failing suite validate exit
json, junit, tap, documentation, rspecish, silent, prometheus 503 1
nagios 503 2
structured 200 0

This is issue #992 a second time. That report was "the prometheus output does not report 503 at /healthz when a test fails", and the same gap was left in structured. Draft #1119 found it independently and pinned the wrong value rather than change it, on the grounds that a behaviour change did not belong inside a logging PR.

Because it has now happened twice, the test pins the exit code of all nine outputers rather than only this one, and fails if a tenth is registered without a deliberate choice.

What changes for users

Three things, all intended, but worth naming.

Anyone using -f structured as a healthcheck or a CI gate goes from always-passing to actually reporting failures. That is the point, and it is what #992 asked for.

util/config.go defaults to structured as "most appropriate for package usage", so a Go program embedding goss and calling Validate with a default config now gets a non-zero code on failure where it got 0 before. That is the same correction, but it reaches library callers and not just the CLI.

validate.go short-circuits retries on exitCode == 0, so -f structured -r <timeout> was previously a no-op. It now retries, which means stdout can carry more than one JSON document. -f json -r already behaves exactly this way, so this makes structured consistent rather than introducing something new. Worth knowing if GOSS_RETRY_TIMEOUT is set globally in a container.

The 503 still carries the full structured body, so consumers that read the payload are unaffected.

Verification

Ran the built binary before and after. On a failing gossfile, validate -f structured goes from exit 0 to exit 1 while the other eight formats are byte-identical, and serve --format structured goes from HTTP 200 to 503. A passing gossfile still gives exit 0 and 200, and a skip-only suite stays 0.

Dropping or weakening the guard fails the new test five different ways: reverting to return 0, using >= 0, keying on TestCount instead of Failed, an off-by-one that lets a single failure through, and returning nagios's 2 instead of 1.

One thing the test needed. The prometheus outputer accumulates into package-level counters, so exercising it here made TestPrometheusOutput fail intermittently. Each case now calls defer resetMetrics(), the way prometheus_test.go already does, and 20 consecutive runs plus 8 -race -shuffle=on runs are clean. The fixtures use the existing makeResults helper so this adds no new package-level symbol.

Separately, and not touched here: structured reports "successful": false for passing tests, because it marshals TestResult.Successful, which nothing ever assigns, while json.go derives the value instead. Different bug, happy to send it on its own.

AI disclosure: written with Claude Code. I ran the binary, the healthcheck probes and the mutation checks myself.

The structured outputer ended Output() with an unconditional return 0,
so a failing suite exited 0 and serve.go, which maps a non-zero code to
503, answered 200. A healthcheck on goss serve -f structured could never
fail. Summary.Failed was already counted a few lines above.

This is issue goss-org#992 for a second format: the same gap was reported for
prometheus and fixed in goss-org#1109, and structured was left behind. Because
it has now happened twice, the test pins the exit code of all nine
outputers and fails if a tenth is registered without a choice.

The prometheus outputer accumulates into package-level counters, so each
case resets them the way prometheus_test.go already does.
@kgaughan
kgaughan merged commit f7ebedc into goss-org:master Sep 13, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants