Writing a script BulbQA can run

2. One glob, read two ways

A source's glob is used by both halves of sync, and they do different things to the same files:

• Rescan repo reads each file as text and finds its test(…) and it(…) titles, matching any @T id annotated in one to a case. Nothing is imported or executed.describe blocks are deliberately not proposed, and test.skip counts — a test that exists and is currently off is a thing worth seeing in coverage.
• Scan scripts imports each matched file and reads the declarations it exports. That is the only moment your code runs, and it runs because someone pressed the button — never because a QA opened a case.

So a Playwright spec is discovered as a test without exporting anything at all, and that is the whole of its route in. A spec cannot also carry a declaration. Reading one means importing the file, and Playwright's test() refuses to be called outside playwright test — so a spec that exports a form is not an action with a form, it is a file the scan reports as unreadable, and its tests stop being discovered too. A button a QA fills in a form for belongs in a plain script the runner can import.

Discovery is how BulbQA learns a test exists. Reporting is how it learns what the test decided, and that is a separate thing to set up — reporting automated tests covers it.

3. A script declares its own form

A declaration is a plain exported object with a zod schema on it. The form and the code it feeds sit in one object in your file, so they cannot drift — there is nothing to keep in step.

// qa/fixtures/booking.ts
import { z } from "zod";

export const seedBooking = {
  title: "Seed a booking",
  description: "Creates a paid booking on the chosen journey.",
  schema: z.object({
    env: z.enum(["staging", "preprod"]).default("staging").describe("Environment"),
    journeyId: z.string().describe("Journey ID"),
    riders: z.number().min(1).max(200).default(4).describe("Riders"),
    adminPassword: z.string().describe("Admin password"),
  }),
  secrets: ["adminPassword"],
  run: async ({ journeyId, riders }) => {
    const bookingRef = await seed(journeyId, riders);
    return { bookingRef, artifacts: ["shots/booking.png"] };
  },
};

export const cancelBooking = {
  title: "Cancel a booking",
  schema: z.object({ bookingRef: z.string().describe("Booking reference") }),
  run: async ({ bookingRef }) => {
    await cancel(bookingRef);
  },
};

// Two actions: booking.ts#seedBooking and booking.ts#cancelBooking

Every exported declaration in a file is its own action, keyed path#export. One file can hold as many as it likes, and a step binds to the one it means rather than to the file that happens to contain it.

Two kinds, decided by run

• With a run — BulbQA imports the module and calls that function with the answers. No command is involved, and none is shown.
• Without one — the whole file is run through its source's command. Only an export named form may be run-less. Three run-less declarations in one file would be three actions issuing the identical command against the identical file, so anything else without a run is reported as a scan problem naming the fix.

// qa/fixtures/reset-staging.ts — no `run`, so BulbQA runs the file
export const form = {
  title: "Reset staging",
  schema: z.object({
    confirm: z.boolean().default(false).describe("Yes, really"),
  }),
};

// The whole file goes through the source's command:
//   sst shell --stage staging -- pnpm tsx qa/fixtures/reset-staging.ts

What the schema becomes

No new syntax, and no second description of your script to maintain. Without a .describe() the field name is humanised (journeyId becomes "Journey ID"). Anything BulbQA does not recognise degrades to a plain box rather than refusing to render the form.

Masking is a declaration, never a guess. A field is masked because you named it in secrets — inferring it from the name masks the wrong fields and, worse, leaves the right ones in plain sight. Secret values are never recorded on a result.

Repeating rows

An array of objects is a repeating group the QA can add rows to. An array of plain scalars has no row to draw, so it falls back to a single box.

schema: z.object({
  riders: z
    .array(
      z.object({
        name: z.string().describe("Name"),
        bornOn: z.iso.date().describe("Date of birth"),
      }),
    )
    .describe("Riders"),
})

Dropdowns that have to be looked up

"Which driver" cannot be a fixed list, and "which vehicle" depends on the driver just picked. An options function per field is called with whatever is filled in so far.

export const seedTrip = {
  title: "Seed a trip",
  schema: z.object({
    driverId: z.string().describe("Driver"),
    vehicleId: z.string().describe("Vehicle"),
  }),
  options: {
    driverId: async () => listDrivers(),
    // Whatever the QA has filled in so far is the argument, so this one
    // depends on the driver they just picked.
    vehicleId: async ({ driverId }) => listVehicles(driverId),
  },
  run: async (values) => {
    /* … */
  },
};

Return strings, or objects carrying a value/idand a label/name. Unlike reading a form, this runs while the QA is looking at the form, so it gets ten seconds and is refetched whenever any other value changes — nothing says which ones your function reads. A failure lands on the field rather than breaking the form.

4. Reading a form imports your module

A schema is code, not data, so the scan actually imports the file. That cost is contained rather than pretended away: the import happens in a forked child, never in the app; the child is killed as a process group after five seconds; only plain JSON comes back; and zod is resolved from your repository, so whatever version you pin is the one that converts your schema.

• One broken file cannot fail the scan. Thirty fixtures with one that throws on import means twenty-nine imported and one reported by name, with the reason.
• A matched file that exports nothing recognisable is not a problem — most files a glob matches are helpers. An export whose schema will not convert (a database table, a router) is skipped rather than half-read.
• A source whose glob matches nothing is reported as such, usually because a folder moved. Worth fixing promptly: everything that source used to provide detaches.
• Avoid two sources whose globs overlap. A file matched by both yields one action, and the command that wins is the later source's.
• export * re-exports produce a second action for the same declaration under a second path. Harmless, and confusing if you were not expecting it.

5. What your script receives, and hands back

A function action

• run is called with the answers, already parsed through your own schema — your defaults and coercions apply, a number arrives as a number, and a mismatch names the field instead of surfacing somewhere inside your code.
• The answers cross the fork's IPC channel and go nowhere else. They are never on a command line, so nothing is visible in ps.
• Return { artifacts: ["shots/login.png"] } to attach files to the result. Paths resolve inside the repository and anything escaping it is dropped; a path that does not exist is dropped too, rather than turning a passing case into an error. Screenshots, JUnit XML, logs and JSON are recognised by extension.
• Anything else you return is recorded, and a later step in the same case can reference it as ${steps.2.bookingRef} — a whole value, not spliced into a larger string.
• A run gets ten minutes before it is stopped.

A whole-file action

Answers arrive as environment variables — each one twice, bare and prefixed, so existing scripts keep working — plus the entire set as JSON, which is the only way to read a repeating group without everyone reinventing the same parsing.

# Every answer arrives twice, plus the whole set as JSON
JOURNEYID=jr_8812
BULBQA_JOURNEYID=jr_8812
RIDERS=4
BULBQA_RIDERS=4
BULBQA_PARAMS={"journeyId":"jr_8812","riders":4,"env":"staging"}

To hand a value back, print it on a marked line. The last marked line wins, so a script that reports progress and then finishes is read correctly, and an unparseable line is ignored rather than failing a run that did its work.

// A whole-file script hands a value back on a marked line.
console.log(`::bulbqa:output::${JSON.stringify({ bookingRef })}`);

6. Detached is not deleted

When a scan does not find a script it saw before, that action is marked detached — kept, with everything a QA attached to it intact. The step stays bound, the run view still offers it, and it says plainly that the script was not in the repository at the last sync. "This had a script until Tuesday" is a question people ask, and a binding is not ours to throw away.

An action's identity is its path and its export. Three ordinary edits therefore detach one and add another:

• renaming or moving the file;
• renaming the export;
• splitting export const form into named declarations — the old #form action detaches and each new export arrives as its own action.

Nothing outside your repository can tell a rename from a delete plus a create, and pretending otherwise would silently move a QA's binding to a script nobody chose. The Sync page sorts by path and shows the export name, so "this file lost formand gained seedRider" reads as one event. Put the same path and export back and the next scan reattaches it, binding and all.

Scanning is also never wider than what you scanned: a scan reports which sources it looked at, and one source's scan cannot detach another's scripts.