Reporting automated tests
Your Playwright tests already know whether they passed. This is how they tell BulbQA — so an automated verdict lands on the same case a person would have judged by hand, in the same run, next to the ones a person did.
There is nothing to install into your repository, and no change to your playwright.config.ts.
2. Tag each test with the case it covers
Every case in BulbQA has an automation id — an @T followed by eight hex characters. Put it in the test's title and the result lands on that case. Each case shows its own id, ready to copy, in the case editor.
test("signs in with a valid password @T1a2b3c4d", async ({ page }) => {
await page.goto("/login");
await page.getByLabel("Email").fill("sam@example.com");
await page.getByLabel("Password").fill(process.env.TEST_PASSWORD!);
await page.getByRole("button", { name: "Sign in" }).click();
await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible();
});The id is matched exactly, so the form matters: capital T, lower-case hex. A test with no id, or one whose case is not in the run, comes back unmatched and changes nothing — which is the ordinary case, because you report every test you ran and a run holds only the cases someone put in it.
The failure that looks like success
Both of the mistakes above — a command without the reporter, and tests carrying no automation id — produce a suite that runs, passes, and leaves no verdict anywhere. There is nothing to see, which is exactly the problem. The Automation page in your project checks both and says which one is wrong, rather than leaving you to notice an absence.
3. What arrives
| Reported | Detail |
|---|---|
| The verdict | Passed, failed or skipped, per test, on the case it covers. |
| How long it took | Per test, so a suite that is getting slower shows it. |
| The step tree | Every test.step and expect, flattened and indented as it ran. |
| The error | Message and stack, with the terminal colour codes stripped out. |
| Screenshots and video | Rendered in the run, from a bucket you own. |
| The trace | Opened in the Playwright trace viewer, not offered as a zip. |
A reporting integration is never worth your test suite, so none of it can fail your run. If BulbQA is unreachable, or a key has expired, or a screenshot cannot be read, the reporter says so on stdout and the exit code stays the tests'.
4. Artifacts stay in your bucket
Screenshots, videos and traces upload straight from the machine running the tests to a bucket you own. BulbQA stores a URL and never holds the bytes.
BULBQA_S3_BUCKET: qa-artifacts.acme.com
BULBQA_S3_REGION: eu-west-2
# Credentials come from the standard AWS chain — a role, a profile, or keys.Traces and video are the bulkiest thing a suite produces, and a team that turns video: 'on' on should be spending their own storage — and should be able to walk away from BulbQA still holding every artifact they ever produced. Set no bucket and reporting still works; only the screenshots are missing.
5. From CI
A pipeline has no signed-in person, so it carries a CI key instead — minted per project on the CI keys page, and refused against every other project. It names a plan rather than a run:
# .github/workflows/qa.yml
env:
BULBQA_URL: https://app.bulbqa.com
BULBQA_API_KEY: ${{ secrets.BULBQA_API_KEY }}
BULBQA_PROJECT_ID: ${{ vars.BULBQA_PROJECT_ID }}
BULBQA_PLAN_ID: ${{ vars.BULBQA_PLAN_ID }}
BULBQA_COMMIT_SHA: ${{ github.sha }}A run id would be right on the first execution and stale on every one after it. A plan is a living template, so the reporter starts a run from it each time, labels that run with the commit, and reports into it. What you get is one run per pipeline execution, in the same history as the runs your team does by hand.
One caveat, stated plainly
The desktop app supplies the reporter to the tests it launches. A CI job has no desktop app, so it has to resolve the reporter itself — and how we distribute it for that case is not settled yet. Treat the snippet above as the shape of the setup rather than a recipe that runs today, and talk to us if you want CI reporting now.
6. What it does not change
- A run is still a run. An automated verdict sits beside a manual one, and a case the suite did not cover is still waiting for a person. There is no separate "automated run" that has to be reconciled with the real one later.
- Your tests stay yours. They live in your repository, run on your machines, and mean the same thing without BulbQA as with it. The tag in the title is the only trace of us.
- Nothing secret is committed. Every setting is an environment variable, read from the environment and nowhere else — never from a file in your repository, because BulbQA does not keep one there.