yao/agent/test

Agent Test Framework

A comprehensive testing framework for Yao AI agents with support for standard testing, dynamic (simulator-driven) testing, agent-driven assertions, and CI integration.

Quick Start

Standard Tests

# Test with direct message (auto-detect agent from current directory)
cd assistants/keyword
yao agent test -i "Extract keywords from: AI and machine learning"

# Test with direct message (specify agent explicitly)
yao agent test -i "Hello world" -n workers.system.keyword

# Test with JSONL file (auto-detect agent from path)
yao agent test -i assistants/keyword/tests/inputs.jsonl

# Generate HTML report
yao agent test -i tests/inputs.jsonl -o report.html

# Stability analysis (run each test 5 times)
yao agent test -i tests/inputs.jsonl --runs 5

Agent-Driven Input

# Generate test cases using an agent
yao agent test -i "agents:tests.generator-agent?count=10" -n assistants.expense

# Preview generated tests without running (dry-run)
yao agent test -i "agents:tests.generator-agent?count=5" -n assistants.expense --dry-run

Dynamic Mode (Simulator)

# Run dynamic tests with simulator
yao agent test -i tests/dynamic.jsonl --simulator tests.simulator-agent

# See detailed turn-by-turn output
yao agent test -i tests/dynamic.jsonl -v

Script Tests

# Test agent handler scripts (hooks, tools, setup functions)
yao agent test -i scripts.expense.setup -v

# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady" -v

# Run with custom context (authorization, metadata)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v

Input Modes

The -i flag supports multiple input modes:

1. JSONL File Mode

Load test cases from a file:

yao agent test -i tests/inputs.jsonl

Agent is auto-detected by traversing up from the input file to find package.yao.

2. Direct Message Mode

Test with a single message:

# Auto-detect agent from current working directory
cd assistants/keyword
yao agent test -i "Extract keywords from this text"

# Or specify agent explicitly
yao agent test -i "Hello" -n workers.system.keyword

3. Agent-Driven Input Mode

Generate test cases using a generator agent:

# Basic usage (-n specifies the target agent to test)
yao agent test -i "agents:tests.generator-agent" -n assistants.expense

# With parameters
yao agent test -i "agents:tests.generator-agent?count=10&focus=edge-cases" -n assistants.expense

# Dry-run to preview generated tests
yao agent test -i "agents:tests.generator-agent?count=5" -n assistants.expense --dry-run

Note: The -n flag is required for agent-driven input mode to specify which agent to test. The generator agent creates test cases for the target agent.

4. Script Test Mode

Test agent handler scripts:

yao agent test -i scripts.expense.setup -v

Script test input format: scripts.<assistant>.<module> (e.g., scripts.expense.setup → assistants/expense/src/setup_test.ts).

5. Script-Generated Input Mode

Generate test cases using a script:

yao agent test -i "scripts:tests.gen.Generate" -n assistants.expense

Note: scripts.xxx (with dot) runs script tests, while scripts:xxx (with colon) generates test cases from a script.

Test Modes

Standard Mode

Single call to agent with optional message history. Each test is independent and stateless.

{
  "id": "T001",
  "input": "Hello",
  "assert": {
    "type": "contains",
    "value": "Hi"
  }
}

Dynamic Mode

Simulator-driven testing with checkpoint validation. A simulator agent generates user messages while checkpoints verify agent behavior.

{
  "id": "T001",
  "input": "I want to order coffee",
  "simulator": {
    "use": "tests.simulator-agent",
    "options": {
      "metadata": {
        "persona": "Customer",
        "goal": "Order a latte"
      }
    }
  },
  "checkpoints": [
    {
      "id": "greeting",
      "assert": {
        "type": "regex",
        "value": "(?i)hello"
      }
    },
    {
      "id": "ask_size",
      "after": [
        "greeting"
      ],
      "assert": {
        "type": "regex",
        "value": "(?i)size"
      }
    }
  ],
  "max_turns": 10
}

Command Line Options

Flag	Description	Default
-i	Input: JSONL file, message, agents:xxx, or scripts:x	(required)
-o	Output file path	output-{timestamp}.jsonl
-n	Agent ID (optional, auto-detected)	auto-detect
-a	Application directory	auto-detect
-e	Environment file	-
-c	Override connector	agent default
-u	Test user ID	test-user
-t	Test team ID	test-team
-r	Reporter agent ID for custom report	built-in
-v	Verbose output	false
--ctx	Path to context JSON file for custom authorization	-
--simulator	Default simulator agent ID for dynamic mode	-
--before	Global BeforeAll hook (e.g., env_test.BeforeAll)	-
--after	Global AfterAll hook (e.g., env_test.AfterAll)	-
--runs	Runs per test (stability analysis)	1
--run	Regex pattern to filter which tests to run	-
--timeout	Timeout per test	5m
--parallel	Parallel test cases	1
--fail-fast	Stop on first failure	false
--dry-run	Generate test cases without running them	false

Input Format (JSONL)

Each line is a JSON object:

{"id": "T001", "input": "Simple text"}
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
{"id": "T005", "input": "Skip this", "skip": true}

Standard Mode Fields

Field	Type	Required	Description
id	string	Yes	Test case ID
input	string | Message | []Message	Yes	Test input
assert	Assertion | []Assertion	No	Assertion rules
expected	any	No	Expected output (exact match)
user	string	No	Override user ID for this test
team	string	No	Override team ID for this test
metadata	map	No	Additional metadata for hooks
options	Options	No	Context options
timeout	string	No	Override timeout (e.g., "30s")
skip	bool	No	Skip this test
before	string	No	Before hook (e.g., env_test.Before)
after	string	No	After hook (e.g., env_test.After)

Dynamic Mode Fields

Field	Type	Required	Description
id	string	Yes	Test case ID
input	string	Yes	Initial user message
simulator	object	Yes	Simulator configuration
simulator.use	string	Yes	Simulator agent ID (no prefix)
simulator.options	object	No	Simulator options
simulator.options.metadata	map	No	Metadata (persona, goal, etc.)
simulator.options.connector	string	No	Override simulator connector
checkpoints	array	Yes	Checkpoints to verify
checkpoints[].id	string	Yes	Checkpoint identifier
checkpoints[].description	string	No	Human-readable description
checkpoints[].assert	object	Yes	Assertion to validate
checkpoints[].after	array	No	Checkpoint IDs that must occur first
checkpoints[].required	bool	No	Is checkpoint required (default: true)
max_turns	int	No	Maximum turns (default: 20)
timeout	string	No	Override timeout (e.g., "2m")

Options

The options field allows per-test-case configuration:

Field	Type	Description
connector	string	Override connector (e.g., "deepseek.v3")
mode	string	Agent mode (default: "chat")
search	bool	Enable/disable search mode
disable_global_prompts	bool	Temporarily disable global prompts
metadata	map	Custom data passed to hooks
skip	object	Skip configuration (see below)

Options.skip

Field	Type	Description
history	bool	Skip history loading
trace	bool	Skip trace logging
output	bool	Skip output to client
keyword	bool	Skip keyword extraction
search	bool	Skip auto search

Input Types

Type	Description	Example
string	Simple text	"Hello world"
Message	Single message	{"role": "user", "content": "..."}
[]Message	Conversation history	[{"role": "user", ...}, {"role": "assistant", ...}]

Assertions

Use assert for flexible validation. If assert is defined, it takes precedence over expected.

Static Assertions

Type	Description	Example
equals	Exact match	{"type": "equals", "value": {"key": "val"}}
contains	Output contains value	{"type": "contains", "value": "keyword"}
not_contains	Output does not contain value	{"type": "not_contains", "value": "error"}
json_path	Extract JSON path and compare	{"type": "json_path", "path": "$.field", "value": true}
regex	Match regex pattern	{"type": "regex", "value": "\\d+"}
type	Check output type	{"type": "type", "value": "object"}

Assertion Fields

Field	Type	Description
type	string	Assertion type (required)
value	any	Expected value or pattern
path	string	JSON path for json_path type
script	string	Script name for script type
use	string	Agent/script ID for agent type (with agents: prefix)
options	object	Options for agent assertions
message	string	Custom failure message
negate	bool	Invert the assertion result

Agent-Driven Assertions

For semantic or fuzzy validation using an LLM:

{
  "id": "T001",
  "input": "Hello",
  "assert": {
    "type": "agent",
    "use": "agents:tests.validator-agent",
    "value": "Response should be friendly and helpful"
  }
}

The validator agent receives the output and criteria, then returns {"passed": true/false, "reason": "..."}.

Script Assertions

For custom validation logic:

{
  "id": "T001",
  "input": "Test",
  "assert": {
    "type": "script",
    "script": "scripts.test.Validate"
  }
}

Multiple Assertions

All assertions must pass:

{
  "id": "T001",
  "input": "Hello",
  "assert": [
    {
      "type": "contains",
      "value": "Hi"
    },
    {
      "type": "not_contains",
      "value": "error"
    },
    {
      "type": "json_path",
      "path": "status",
      "value": "ok"
    }
  ]
}

File Attachments

Test inputs support file attachments using the file:// protocol:

{
  "id": "T001",
  "input": {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "Analyze this image"
      },
      {
        "type": "image",
        "source": "file://fixtures/receipt.jpg"
      }
    ]
  }
}

Supported types: images (jpg, png, gif, webp), audio (wav, mp3), documents (pdf, doc, txt).

Before/After Hooks

Per-Test Hooks

Defined in JSONL, scripts located in agent's src/ directory:

{
  "id": "T001",
  "input": "Test",
  "before": "env_test.Before",
  "after": "env_test.After"
}

Global Hooks

Via CLI flags:

yao agent test -i tests/inputs.jsonl --before env_test.BeforeAll --after env_test.AfterAll

Hook Script Example

// assistants/expense/src/env_test.ts

export function Before(ctx: Context, testCase: TestCase): any {
  // Setup: create test data
  const userId = Process("models.user.Create", { name: "Test User" });
  return { userId }; // Passed to After
}

export function After(
  ctx: Context,
  testCase: TestCase,
  result: TestResult,
  beforeData: any
) {
  // Cleanup: delete test data
  if (beforeData?.userId) {
    Process("models.user.Delete", beforeData.userId);
  }
}

export function BeforeAll(ctx: Context, testCases: TestCase[]): any {
  Process("models.migrate");
  return { initialized: true };
}

export function AfterAll(ctx: Context, results: TestResult[], beforeData: any) {
  Process("models.cleanup");
}

Script Testing

Test agent handler scripts with the t.assert API:

// assistants/expense/src/setup_test.ts
import { SystemReady } from "./setup";

export function TestSystemReady(t: TestingT, ctx: Context) {
  const result = SystemReady(ctx);

  t.assert.True(result.success, "Should succeed");
  t.assert.Equal(result.status, "ready", "Status should be ready");
  t.assert.NotNil(result.data, "Data should not be nil");
}

export function TestWithAgentAssertion(t: TestingT, ctx: Context) {
  const response = Process("agents.expense.Stream", ctx, messages);

  // Static assertion
  t.assert.Contains(response.content, "confirm");

  // Agent-driven assertion
  t.assert.Agent(response.content, "tests.validator-agent", {
    criteria: "Response should ask for confirmation",
  });
}

Available Assertions

Method	Description
t.assert.True(value, msg)	Assert value is true
t.assert.False(value, msg)	Assert value is false
t.assert.Equal(a, b, msg)	Assert a equals b
t.assert.NotEqual(a, b, msg)	Assert a not equals b
t.assert.Nil(value, msg)	Assert value is null/undefined
t.assert.NotNil(value, msg)	Assert value is not nil
t.assert.Contains(s, sub, msg)	Assert string contains substr
t.assert.Len(arr, n, msg)	Assert array/string length
t.assert.Agent(resp, id, opts)	Agent-driven assertion

Dynamic Mode

For testing complex conversation flows where the path is unpredictable:

{
  "id": "coffee-order",
  "input": "I want to order coffee",
  "simulator": {
    "use": "tests.simulator-agent",
    "options": {
      "metadata": {
        "persona": "Customer ordering a latte",
        "goal": "Complete the coffee order"
      }
    }
  },
  "checkpoints": [
    {
      "id": "greeting",
      "description": "Agent greets customer",
      "assert": {
        "type": "regex",
        "value": "(?i)(hello|hi|help)"
      }
    },
    {
      "id": "ask_size",
      "description": "Agent asks for size",
      "after": [
        "greeting"
      ],
      "assert": {
        "type": "regex",
        "value": "(?i)size"
      }
    },
    {
      "id": "confirm",
      "description": "Agent confirms order",
      "after": [
        "ask_size"
      ],
      "assert": {
        "type": "regex",
        "value": "(?i)confirm"
      }
    }
  ],
  "max_turns": 10
}

Console Output (Dynamic Mode)

► [coffee-order] (dynamic, 3 checkpoints)
ℹ Dynamic test: coffee-order (max 10 turns)
ℹ   Turn 1: User: I want to order coffee
ℹ   Turn 1: Agent: Hello! What can I get for you?
ℹ     ✓ checkpoint: greeting
ℹ   Turn 2: User: A medium latte please
ℹ   Turn 2: Agent: What size would you like?
ℹ     ✓ checkpoint: ask_size
ℹ   Turn 3: User: Medium
ℹ   Turn 3: Agent: Let me confirm: Medium latte. Correct?
ℹ     ✓ checkpoint: confirm
  └─ PASSED (3 turns, 3 checkpoints, 8.5s)

Output Formats

Determined by -o file extension:

Extension	Format	Description
.jsonl	JSONL	Streaming (default)
.json	JSON	Complete structured
.md	Markdown	Human-readable
.html	HTML	Interactive web report

Stability Analysis

Run each test multiple times to measure consistency:

yao agent test -i tests/inputs.jsonl --runs 5 -o stability.json

Pass Rate	Classification
100%	Stable
80-99%	Mostly Stable
50-79%	Unstable
< 50%	Highly Unstable

CI Integration

# Exit code: 0 = all passed, 1 = failures
yao agent test -i tests/inputs.jsonl --fail-fast

# Run with parallel execution
yao agent test -i tests/inputs.jsonl --parallel 4

GitHub Actions Example

- name: Run Agent Tests
  run: |
    yao agent test -i assistants/expense/tests/inputs.jsonl \
      -u ci-user -t ci-team \
      --runs 3 \
      -o report.json

- name: Run Dynamic Tests
  run: |
    yao agent test -i assistants/expense/tests/dynamic.jsonl \
      --simulator tests.simulator-agent \
      -v

- name: Run Script Tests
  run: |
    yao agent test -i scripts.expense.setup -v

Format Rules Reference

Context	Format	Example
-i agents:xxx (CLI)	Colon prefix	agents:tests.generator
-i scripts:xxx (CLI)	Colon prefix	scripts:tests.gen.Generate
-i scripts.xxx (CLI)	Dot prefix (test mode)	scripts.expense.setup
JSONL assertion use	Prefix required	"use": "agents:tests.validator"
JSONL simulator.use	No prefix (agent only)	"use": "tests.simulator-agent"
--simulator flag	No prefix (agent only)	--simulator tests.simulator-agent
t.assert.Agent()	No prefix (method-bound)	t.assert.Agent(resp, "tests.validator")
JSONL before/after	No prefix (in src/)	"before": "env_test.Before"
--before/--after	No prefix (in src/)	--before env_test.BeforeAll

Script input modes:

• scripts.xxx (dot) - Run script tests (*_test.ts functions)
• scripts:xxx (colon) - Generate test cases from a script

Exit Codes

Code	Description
0	All tests passed
1	Tests failed, configuration error, or runtime error