Skip to content

reporting.md

Latest commit

 

History

History
133 lines (88 loc) · 6.04 KB

reporting.md

File metadata and controls

133 lines (88 loc) · 6.04 KB
Raw

Test Reporting

Explorbot generates test reports with @testomatio/reporter. Reports include test steps, screenshots, and result messages for every run. The API tester reports through the same reporter, so the HTML and Testomat.io settings below apply to API runs too.

The Testomat.io cloud reporter is the recommended way to review runs, especially on CI: pair it with the Analyst and screencasts and each run opens with a written overview you can drill from down to a video of any single test. The HTML and markdown reports are the local alternatives. For the full CI setup, see Continuous Integration.

Session analysis

After /explore and /freesail runs, the Analyst agent writes a summary that clusters findings by root cause.

The same markdown is printed to the console, written to disk, and set as the run description on Testomat.io when the cloud reporter is enabled.

On disk: output/reports/<mode>-<sessionName>.md — for example explore-WiseFox42.md or freesail-CleverOwl91.md. Each run gets a unique name, so nothing is overwritten.

On Testomat.io: the markdown becomes the run description, so the analysis sits next to the test list in the cloud dashboard with no extra setup.

See Analyst Agent for the report format and configuration options.

Screencasts

Explorbot can record a video of every test instead of leaving you a trail of screenshots. Enable it on the Historian agent:

export default {
  ai: {
    agents: {
      historian: {
        screencast: true,  // or { size: { width: 1280, height: 720 }, quality: 95 }
      },
    },
  },
};

Each test gets one video in output/screencasts/, with the executed actions overlaid in the corner and the current scenario step shown as a chapter title — the video narrates itself. The file is attached to the test as an artifact, so with artifact storage configured it appears on the test in Testomat.io, ready to watch next to the failure message.

HTML report (local)

With html: true under reporter in explorbot.config.js, Explorbot creates an HTML report after each run in output/reports/<mode>-<sessionName>.html, for example explore-WiseFox42.html. Each session gets its own file, so nothing is overwritten. Open it in a browser to review results. Configs generated by explorbot init enable it, so new projects get HTML reports out of the box.

HTML report

For cloud reporting with history and team features, see Testomat.io cloud report.

Markdown report (local)

Explorbot can write a markdown report next to the HTML one. It's plain text, so you can paste it into a PR description, a chat thread, or a CI summary.

Opt in via explorbot.config.js:

export default {
  reporter: {
    enabled: true,
    html: true,
    markdown: true,
  },
};

Output: output/reports/<mode>-<sessionName>-tests.md, for example explore-WiseFox42-tests.md. Like the HTML report, the filename is session-scoped, so successive runs don't overwrite each other. The -tests suffix keeps it distinct from the Analyst report, which writes <mode>-<sessionName>.md in the same folder.

The markdown report isn't generated unless markdown: true is set. Configs generated by explorbot init enable it as well.

Testomat.io cloud report

Testomat.io provides a cloud dashboard with test history, analytics, and team collaboration. It's free with unlimited test runs.

Testomat.io cloud report

Setup

  1. Register at app.testomat.io.
  2. Create an empty project.
  3. Copy the project API key.

Run with the key

TESTOMATIO=tstmt_your_key_here npx explorbot explore /

Set the key in your shell profile or CI environment so it's always active.

Options

Variable Description
TESTOMATIO Project API key (required)
TESTOMATIO_TITLE Custom name for the test run
TESTOMATIO_ENV Environment label (e.g. staging, production)
TESTOMATIO_SHARED_RUN Merge parallel executions into one run
TESTOMATIO_RUNGROUP_TITLE Group successive runs under one heading (overrides reporter.runGroup)

See the @testomatio/reporter docs for the full list.

Run group

Set reporter.runGroup to file successive runs under one heading on Testomat.io. There is no built-in default, but configs generated by explorbot init set it to today's date (YYYY-MM-DD), so all sessions from one day appear together in the dashboard.

Change or remove it via explorbot.config.js:

export default {
  reporter: {
    enabled: true,
    runGroup: 'Smoke Suite',  // any string; remove to disable grouping
  },
};

If set, TESTOMATIO_RUNGROUP_TITLE from the environment takes precedence over the config.

Artifacts in cloud reports

Explorbot attaches screenshots to test steps and, when screencasts are enabled, a video to every test. To see them in Testomat.io, configure an S3-compatible storage provider under Settings > Artifacts in Testomat.io. Without it, screenshots and screencasts won't appear in cloud reports.

Set these environment variables:

S3_ACCESS_KEY_ID=your_access_key
S3_SECRET_ACCESS_KEY=your_secret_key
S3_BUCKET=your-bucket-name
S3_REGION=us-east-1

For non-AWS providers (DigitalOcean Spaces, Cloudflare R2, Google Cloud Storage, Minio), also set:

S3_ENDPOINT=https://your-provider-endpoint.com

See the @testomatio/reporter artifacts docs for provider-specific examples.