Documentation

Writing Tests

The DSL, goal writing, and the two ways to shape a spec.

A Blop test is a TypeScript file that ends in .blop.ts (or .blop.tsx) and exports one or more agent tests. The runtime is intentionally small: three exports cover almost every case.

The two shapes

Function form: describe + agentTest

Group related tests and write goals inline:

import { agentTest, describe } from "@blopai/cli";

describe("onboarding", () => {
  agentTest("user can create a project", async ({ agent }) => {
    await agent.goto("/");
    await agent.goal(`
      Sign in as the seeded user.
      Create a project called "Checkout QA".
      Verify the project appears in the dashboard.
    `);
  });
});

Each describe extends the test name with >; nest as deeply as you like.

Object form: defineAgentTest

For programmatically generated specs or array exports:

import { defineAgentTest } from "@blopai/cli";

export default defineAgentTest({
  name: "smoke",
  goal: "Open the homepage and verify the primary CTA renders.",
});

export const tests = [
  defineAgentTest({
    name: "checks page title",
    goal: "Go to /, get the page title, and verify it contains 'Blop'.",
  }),
];

Each spec file may export default (a single test) and/or tests (an array).

What agent actually does

agent.goto(url) and agent.goal(text) do not run anything immediately. They build up a numbered goal list. When the runner reaches the test, the agent receives that list along with its browser tools and tries to fulfil it in order.

agentTest("checkout flow", async ({ agent }) => {
  await agent.goto("/cart");                // step 1
  await agent.goal("Add the first item.");  // step 2
  await agent.goto("/checkout");            // step 3
  await agent.goal("Pay with the test card and verify success.");
});

That ordering matters: the agent treats the list as a sequence.

Writing a goal the agent can act on

The single biggest factor in test reliability is the wording of goal. Treat it like a clear ticket for a careful junior engineer.

Good goals

  • State outcomes, not selectors. "Verify the dashboard heading reads My projects and the New project button is visible in the top-right."
  • Demand explicit verification. "Submit the form. Verify the URL becomes /thanks and the heading Order confirmed is on the page."
  • Force a strict pass condition. "Finish the test as passed only if all of the above checks succeed. Otherwise finish as failed and explain why."

Antipatterns

  • "Test the homepage." Too vague; the agent will optimistically pass.
  • "Make sure the button works." Here "works" is undefined.
  • "Sign up, then post a comment, then delete the account." Split multi-flow goals into separate tests.

Naming and organization

  • One describe per user-visible feature (onboarding, billing, auth).
  • Test names finish the sentence: agentTest("can create a project", ...).
  • Co-locate specs with the feature, or keep them in a top-level tests/ directory. Blop discovers either.

Per-test overrides

defineAgentTest accepts baseUrl and timeoutMs for a single test:

defineAgentTest({
  name: "external integration smoke",
  goal: "Hit the staging API health endpoint and verify it returns 200.",
  baseUrl: "https://staging.example.com",
  timeoutMs: 60_000,
});

For function-form tests, use blop.config.ts or the CLI flags instead.

Testing your specs without a real model

To unit-test the runner or your goal templates, pass a mock agentStream to runBlopTests (no API keys, no network calls):

import { runBlopTests, type BlopAgentStreamRunner } from "@blopai/cli";

const mockStream: BlopAgentStreamRunner = async function* ({ nativeTools }) {
  const goto = nativeTools.find((t) => t.name === "browser_goto");
  const finish = nativeTools.find((t) => t.name === "finish_test");

  yield { event_type: "step_start", metadata: {} };
  await goto!.execute({ url: "https://example.com" });
  yield { event_type: "step_finish", metadata: {} };

  await finish!.execute({ status: "passed", reason: "page loaded" });
};

const result = await runBlopTests({
  specFiles: ["tests/example.blop.ts"],
  agentStream: mockStream,
});

Where to go next