opencode/yao
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Implement Script Testing Framework and Enhance Test Context Management

Browse source 
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code.
- Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed.
- Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests.
- Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data.
- Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
This commit is contained in:
Max 2025-12-21 10:44:59 +08:00
parent aacb81fb52
commit 5e64c78435
15 changed files with 3048 additions and 119 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
agent/assistant/scripts.go
View file

@ -75,6 +75,11 @@ func LoadScripts(srcDir string) (*hook.Script, map[string]*Script, error) {
// Get relative path for determining if it's index // Get relative path for determining if it's index
relPath := strings.TrimPrefix(file, root+"/") relPath := strings.TrimPrefix(file, root+"/")
// Skip test files (*_test.ts, *_test.js)
if strings.HasSuffix(relPath, "_test.ts") || strings.HasSuffix(relPath, "_test.js") {
return nil
}
// Check if it's the root index.ts/js (hook script) // Check if it's the root index.ts/js (hook script)
// Only src/index.ts is the hook script, not src/foo/index.ts // Only src/index.ts is the hook script, not src/foo/index.ts
isRootIndex := relPath == "index.ts" || relPath == "index.js" isRootIndex := relPath == "index.ts" || relPath == "index.js"

736
agent/test/DESIGN.md
View file

@ -5,6 +5,8 @@
Agent Test Package provides a framework for testing AI agents with structured test cases. Agent Test Package provides a framework for testing AI agents with structured test cases.
It supports batch testing, report generation, stability analysis, and CI integration. It supports batch testing, report generation, stability analysis, and CI integration.
Additionally, it supports **Script Testing** for testing Agent handler scripts (hooks, tools, etc.) with a Go-like testing interface.
### Quick Start ### Quick Start
```bash ```bash
@ -23,6 +25,12 @@ yao agent test -i assistants/keyword/tests/inputs.jsonl --runs 5
# Generate HTML report # Generate HTML report
yao agent test -i assistants/keyword/tests/inputs.jsonl -r report.html -o report.html yao agent test -i assistants/keyword/tests/inputs.jsonl -r report.html -o report.html
# Run script tests (test agent handler scripts)
yao agent test -i scripts.expense.setup -v
# Run script tests with specific user/team context
yao agent test -i scripts.expense.tools -u admin -t ops-team -v
``` ```
## Usage ## Usage
@ -77,7 +85,7 @@ yao agent test -i tests/inputs.jsonl \
### Input Modes ### Input Modes
The `-i` flag supports two input modes: The `-i` flag supports three input modes:
**1. JSONL File Mode** - Load test cases from a file: **1. JSONL File Mode** - Load test cases from a file:
@ -104,6 +112,33 @@ When using direct message mode:
- Output is printed to stdout (or saved to `-o` if specified) - Output is printed to stdout (or saved to `-o` if specified)
- Useful for quick testing and debugging - Useful for quick testing and debugging
**3. Script Test Mode** - Test agent handler scripts:
```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v
# Run with specific user/team context
yao agent test -i scripts.expense.tools -u admin -t ops-team
# Run with timeout
yao agent test -i scripts.expense.setup --timeout 30s
# Run specific tests by pattern (like go test -run)
yao agent test -i scripts.expense.setup -run TestSystemReady
# Run tests matching a regex pattern
yao agent test -i scripts.expense.setup -run "TestSystem.*"
```
When using script test mode:
- Input starts with `scripts.` prefix to indicate script testing
- Maps to the script file (e.g., `scripts.expense.setup` → `expense/src/setup_test.ts`)
- Automatically discovers and runs all `Test*` functions in the script
- Uses Go-like testing interface with assertions
- See [Script Testing](#script-testing) section for details
### Default Output Path ### Default Output Path
When `-o` is not specified and using JSONL file mode, the output file is automatically generated in the same directory as the input file: When `-o` is not specified and using JSONL file mode, the output file is automatically generated in the same directory as the input file:
@ -131,8 +166,10 @@ When using direct message mode without `-o`, output is printed to stdout.
| `-c` | `--connector` | Override connector | agent default | `-c openai.gpt4` | | `-c` | `--connector` | Override connector | agent default | `-c openai.gpt4` |
| `-u` | `--user` | Test user ID (global override) | "test-user" | `-u admin` | | `-u` | `--user` | Test user ID (global override) | "test-user" | `-u admin` |
| `-t` | `--team` | Test team ID (global override) | "test-team" | `-t ops-team` | | `-t` | `--team` | Test team ID (global override) | "test-team" | `-t ops-team` |
| | `--ctx` | Path to context JSON file | - | `--ctx tests/context.json` |
| `-r` | `--reporter` | Custom reporter agent ID | - (use built-in) | `-r report.beautiful` | | `-r` | `--reporter` | Custom reporter agent ID | - (use built-in) | `-r report.beautiful` |
| | `--runs` | Number of runs for stability analysis | 1 | `--runs 5` | | | `--runs` | Number of runs for stability analysis | 1 | `--runs 5` |
| | `--run` | Regex pattern to filter tests | - | `--run "TestSystem.*"` |
| | `--timeout` | Default timeout per test case | 5m | `--timeout 10m` | | | `--timeout` | Default timeout per test case | 5m | `--timeout 10m` |
| | `--parallel` | Number of parallel test cases | 1 | `--parallel 4` | | | `--parallel` | Number of parallel test cases | 1 | `--parallel 4` |
| `-v` | `--verbose` | Verbose output | false | `-v` | | `-v` | `--verbose` | Verbose output | false | `-v` |
@ -330,6 +367,437 @@ Create a custom reporter agent at `assistants/reporters/my-reporter/`:
- Make it printable - Make it printable
``` ```
## Script Testing
Script Testing allows you to test Agent handler scripts (hooks, tools, setup functions, etc.) using a Go-like testing interface. This is useful for unit testing individual functions in your agent's TypeScript/JavaScript code.
### Quick Start
```bash
# Run script tests
yao agent test -i scripts.expense.setup -v
# With user/team context
yao agent test -i scripts.expense.setup -u admin -t ops-team -v
# With timeout
yao agent test -i scripts.expense.setup --timeout 30s -v
```
### Script Resolution
The `scripts.` prefix indicates script test mode. The script is resolved as follows:
| Input | Script Path | Test File |
| ----------------------- | ---------------------- | --------------------------- |
| `scripts.expense.setup` | `expense/src/setup.ts` | `expense/src/setup_test.ts` |
| `scripts.expense.tools` | `expense/src/tools.ts` | `expense/src/tools_test.ts` |
| `scripts.keyword.index` | `keyword/src/index.ts` | `keyword/src/index_test.ts` |
The test file naming convention is `{module}_test.ts` (similar to Go's `_test.go` convention).
### Test Function Signature
Test functions must follow this signature:
```typescript
function TestFunctionName(t: testing.T, ctx: agent.Context) {
// Test logic here
}
```
**Requirements:**
- Function name must start with `Test` (case-sensitive)
- First parameter `t` is the testing object with assertions
- Second parameter `ctx` is the agent context (same as used in hooks/tools)
- Functions not starting with `Test` are ignored (can be used as helpers)
### Example Test File
```typescript
// setup_test.ts
// @ts-nocheck
// Test the SystemReady function
function TestSystemReady(t: testing.T, ctx: agent.Context) {
const { assert } = t;
// Call the function being tested
const result = SystemReady(ctx);
// Assert the result
assert.True(result, "SystemReady should return true");
}
// Test error case
function TestSystemReadyWithInvalidContext(t: testing.T, ctx: agent.Context) {
const { assert } = t;
// Modify context to simulate error condition
ctx.User = null;
const result = SystemReady(ctx);
assert.False(result, "SystemReady should return false when user is null");
}
// Helper function (not a test - doesn't start with "Test")
function createMockData() {
return { id: 1, name: "test" };
}
// Test with helper
function TestSetupWithMockData(t: testing.T, ctx: agent.Context) {
const { assert } = t;
const mockData = createMockData();
const result = Setup(ctx, mockData);
assert.NotNil(result, "Setup should return a result");
assert.Equal(result.id, 1, "Result ID should match");
}
```
### Testing Object (`t`)
The `t` parameter provides the testing interface:
```typescript
interface testing.T {
// Assertions object
assert: testing.Assert;
// Test metadata
name: string; // Current test function name
failed: boolean; // Whether the test has failed
// Logging (output appears in test report)
log(...args: any[]): void; // Log info message
error(...args: any[]): void; // Log error message
// Control flow
skip(reason?: string): void; // Skip this test
fail(reason?: string): void; // Mark test as failed
fatal(reason?: string): void; // Mark as failed and stop execution
}
```
### Assertions (`t.assert`)
The `assert` object provides assertion methods:
| Method | Description |
| ----------------------------------- | ---------------------------------- |
| `True(value, message?)` | Assert value is true |
| `False(value, message?)` | Assert value is false |
| `Equal(actual, expected, message?)` | Assert deep equality |
| `NotEqual(actual, expected, msg?)` | Assert not equal |
| `Nil(value, message?)` | Assert value is null/undefined |
| `NotNil(value, message?)` | Assert value is not null/undefined |
| `Contains(str, substr, message?)` | Assert string contains substring |
| `NotContains(str, substr, msg?)` | Assert string does not contain |
| `Len(value, length, message?)` | Assert array/string length |
| `Greater(a, b, message?)` | Assert a > b |
| `GreaterOrEqual(a, b, message?)` | Assert a >= b |
| `Less(a, b, message?)` | Assert a < b |
| `LessOrEqual(a, b, message?)` | Assert a <= b |
| `Error(err, message?)` | Assert err is an error |
| `NoError(err, message?)` | Assert err is null/undefined |
| `Panic(fn, message?)` | Assert function throws |
| `NoPanic(fn, message?)` | Assert function does not throw |
| `Match(value, pattern, message?)` | Assert value matches regex |
| `NotMatch(value, pattern, msg?)` | Assert value does not match regex |
| `JSONPath(obj, path, expected, m?)` | Assert JSON path value |
| `Type(value, typeName, message?)` | Assert value type |
### Agent Context (`ctx`)
The `ctx` parameter is the same `agent.Context` used in agent hooks and tools:
```typescript
interface agent.Context {
// User information (from -u flag or default)
User: {
ID: string;
Name?: string;
};
// Team information (from -t flag or default)
Team: {
ID: string;
Name?: string;
};
// Locale (default: "en-us")
Locale: string;
// Client information
Client: {
Type: string; // "test"
IP: string; // "127.0.0.1"
};
// Metadata (can be set via test case)
Metadata: Record<string, any>;
// Chat/Session ID
ChatID: string;
// Assistant ID (resolved from script path)
AssistantID: string;
}
```
### Script Test Output
Script test results are reported in the same format as agent tests:
```
═══════════════════════════════════════════════════════════════════════════════
Script Test: scripts.expense.setup
═══════════════════════════════════════════════════════════════════════════════
Script: expense/src/setup_test.ts
Tests: 3 functions
User: test-user
Team: test-team
───────────────────────────────────────────────────────────────────────────────
Running Tests
───────────────────────────────────────────────────────────────────────────────
► [TestSystemReady] ...
✓ PASSED (12ms)
► [TestSystemReadyWithInvalidContext] ...
✓ PASSED (8ms)
► [TestSetupWithMockData] ...
✗ FAILED (15ms)
└─ assertion failed: Result ID should match
expected: 1
actual: 2
═══════════════════════════════════════════════════════════════════════════════
Summary: 2 passed, 1 failed, 0 skipped (35ms)
═══════════════════════════════════════════════════════════════════════════════
```
### Script Test Options
Script tests support the following command line options:
| Flag | Description | Default | Example |
| ------------- | -------------------------------- | ----------- | -------------------- |
| `-u` | User ID for context | "test-user" | `-u admin` |
| `-t` | Team ID for context | "test-team" | `-t ops-team` |
| `--ctx` | Path to context JSON file | - | `--ctx context.json` |
| `-v` | Verbose output | false | `-v` |
| `--run` | Regex to filter tests | - | `--run "TestSystem"` |
| `--timeout` | Timeout per test function | 30s | `--timeout 1m` |
| `--fail-fast` | Stop on first failure | false | `--fail-fast` |
| `-o` | Output file for report | stdout | `-o report.json` |
| `-r` | Reporter agent for custom report | - | `-r report.html` |
The `--run` flag accepts a Go-style regex pattern to filter which tests to run:
```bash
# Run only TestSystemReady
yao agent test -i scripts.expense.setup --run TestSystemReady
# Run all tests starting with "TestSystem"
yao agent test -i scripts.expense.setup --run "TestSystem.*"
# Run tests containing "Error"
yao agent test -i scripts.expense.setup --run ".*Error.*"
```
### Custom Context Configuration
The `--ctx` flag allows you to provide a JSON file with custom context configuration, giving full control over authorization data, metadata, and client information:
```bash
# Use custom context file
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```
**Context JSON Format:**
```json
{
"authorized": {
"sub": "user-12345",
"client_id": "my-app",
"scope": "read write",
"session_id": "sess-abc123",
"user_id": "admin",
"team_id": "team-001",
"tenant_id": "acme-corp",
"remember_me": false,
"constraints": {
"owner_only": false,
"creator_only": false,
"editor_only": false,
"team_only": true,
"extra": {
"department": "engineering",
"region": "us-west"
}
}
},
"metadata": {
"request_id": "req-123",
"trace_id": "trace-456",
"custom_field": "custom_value"
},
"client": {
"type": "web",
"user_agent": "Mozilla/5.0",
"ip": "192.168.1.100"
},
"locale": "zh-cn",
"referer": "https://example.com/dashboard"
}
```
**Field Descriptions:**
| Field | Description |
| -------------------------- | --------------------------------------------------- |
| `authorized.sub` | Subject identifier (JWT sub claim) |
| `authorized.client_id` | OAuth client ID |
| `authorized.scope` | Access scope |
| `authorized.session_id` | Session identifier |
| `authorized.user_id` | User identifier (overrides -u flag) |
| `authorized.team_id` | Team identifier (overrides -t flag) |
| `authorized.tenant_id` | Tenant identifier |
| `authorized.remember_me` | Remember me flag |
| `authorized.constraints` | Data access constraints (set by ACL enforcement) |
| `constraints.owner_only` | Only access owner's data |
| `constraints.creator_only` | Only access creator's data |
| `constraints.editor_only` | Only access editor's data |
| `constraints.team_only` | Only access team's data (filter by team_id) |
| `constraints.extra` | User-defined constraints (department, region, etc.) |
| `metadata` | Custom metadata passed to context |
| `client.type` | Client type (web, mobile, test, etc.) |
| `client.user_agent` | Client user agent string |
| `client.ip` | Client IP address |
| `locale` | Locale setting (e.g., "en-us", "zh-cn") |
| `referer` | Request referer URL |
**Priority:** When both `-u`/`-t` flags and `--ctx` file are provided, the context file values take precedence.
### Script Test Report Format
When using `-o` to save results:
```json
{
"type": "script_test",
"script": "scripts.expense.setup",
"script_path": "expense/src/setup_test.ts",
"summary": {
"total": 3,
"passed": 2,
"failed": 1,
"skipped": 0,
"duration_ms": 35
},
"environment": {
"user_id": "test-user",
"team_id": "test-team",
"locale": "en-us"
},
"results": [
{
"name": "TestSystemReady",
"status": "passed",
"duration_ms": 12,
"logs": []
},
{
"name": "TestSystemReadyWithInvalidContext",
"status": "passed",
"duration_ms": 8,
"logs": []
},
{
"name": "TestSetupWithMockData",
"status": "failed",
"duration_ms": 15,
"error": "assertion failed: Result ID should match",
"assertion": {
"type": "Equal",
"expected": 1,
"actual": 2,
"message": "Result ID should match"
},
"logs": []
}
],
"metadata": {
"started_at": "2024-12-17T10:00:00Z",
"completed_at": "2024-12-17T10:00:00Z",
"version": "0.10.5"
}
}
```
### Best Practices
1. **Naming Convention**: Use descriptive test names that explain what's being tested
- Good: `TestSystemReadyWithValidUser`, `TestSetupReturnsErrorOnMissingConfig`
- Bad: `Test1`, `TestIt`
2. **One Assertion Per Concept**: Each test should verify one behavior
```typescript
// Good: Focused tests
function TestSetupCreatesDatabase(t, ctx) { ... }
function TestSetupInitializesCache(t, ctx) { ... }
// Bad: Testing too many things
function TestSetup(t, ctx) {
// tests database, cache, config, etc.
}
```
3. **Use Helper Functions**: Extract common setup logic
```typescript
function setupTestContext(ctx) {
ctx.Metadata.testMode = true;
return ctx;
}
function TestFeatureA(t, ctx) {
ctx = setupTestContext(ctx);
// ...
}
```
4. **Test Error Cases**: Don't just test happy paths
```typescript
function TestSetupWithMissingConfig(t, ctx) {
const { assert } = t;
ctx.Metadata.config = null;
const result = Setup(ctx);
assert.Error(result.error, "Should return error for missing config");
}
```
5. **Clean Up**: If your test modifies global state, clean up after
```typescript
function TestWithGlobalState(t, ctx) {
const originalValue = GlobalConfig.value;
try {
GlobalConfig.value = "test";
// ... test logic
} finally {
GlobalConfig.value = originalValue;
}
}
```
## Input Format (JSONL) ## Input Format (JSONL)
Each line in the input file is a JSON object with the following structure: Each line in the input file is a JSON object with the following structure:
@ -632,8 +1100,13 @@ agent/test/
├── runner.go # Test runner implementation ├── runner.go # Test runner implementation
├── loader.go # Test case loader ├── loader.go # Test case loader
├── resolver.go # Agent resolver ├── resolver.go # Agent resolver
├── environment.go # Test environment setup ├── context.go # Test context creation
├── stability.go # Stability analysis ├── assert.go # Assertion implementation
├── input.go # Input parsing
├── output.go # Output formatting
├── script.go # Script test runner (NEW)
├── script_types.go # Script test types (NEW)
├── script_assert.go # Script assertion bindings (NEW)
└── reporter/ └── reporter/
├── json.go # JSON reporter ├── json.go # JSON reporter
├── html.go # HTML reporter ├── html.go # HTML reporter
@ -665,7 +1138,74 @@ Executes test cases against an agent:
- Executes each test case (optionally multiple runs) - Executes each test case (optionally multiple runs)
- Collects results and stability metrics - Collects results and stability metrics
### 5. Reporter ### 5. ScriptRunner (NEW)
Executes script tests for agent handler scripts:
- Resolves script path from `scripts.` prefix
- Discovers `Test*` functions in the script
- Creates test context with environment
- Executes each test function with testing object and context
- Collects results and generates report
### 6. ScriptTestCase (NEW)
Represents a single script test function:
```go
type ScriptTestCase struct {
Name string // Function name (e.g., "TestSystemReady")
Function string // Full function reference
}
```
### 7. ScriptTestResult (NEW)
Represents the result of running a script test function:
```go
type ScriptTestResult struct {
Name string `json:"name"`
Status Status `json:"status"`
DurationMs int64 `json:"duration_ms"`
Error string `json:"error,omitempty"`
Assertion *AssertionInfo `json:"assertion,omitempty"`
Logs []string `json:"logs,omitempty"`
}
type AssertionInfo struct {
Type string `json:"type"`
Expected interface{} `json:"expected,omitempty"`
Actual interface{} `json:"actual,omitempty"`
Message string `json:"message,omitempty"`
}
```
### 8. ScriptTestReport (NEW)
Represents the complete script test report:
```go
type ScriptTestReport struct {
Type string `json:"type"` // "script_test"
Script string `json:"script"`
ScriptPath string `json:"script_path"`
Summary *ScriptTestSummary `json:"summary"`
Environment *Environment `json:"environment"`
Results []*ScriptTestResult `json:"results"`
Metadata *ReportMetadata `json:"metadata"`
}
type ScriptTestSummary struct {
Total int `json:"total"`
Passed int `json:"passed"`
Failed int `json:"failed"`
Skipped int `json:"skipped"`
DurationMs int64 `json:"duration_ms"`
}
```
### 9. Reporter
Generates reports in various formats. The format is determined by the `-o` file extension: Generates reports in various formats. The format is determined by the `-o` file extension:
@ -713,7 +1253,8 @@ This allows for fully customizable report generation using AI agents
```go ```go
type Options struct { type Options struct {
// Input/Output // Input/Output
InputFile string // Path to inputs.jsonl Input string // Input source: file path, message, or scripts.xxx
InputMode InputMode // Auto-detected: file, message, or script
OutputFile string // Path to output report OutputFile string // Path to output report
// Agent Selection // Agent Selection
@ -737,6 +1278,186 @@ type Options struct {
Verbose bool // Verbose output Verbose bool // Verbose output
FailFast bool // Stop on first failure FailFast bool // Stop on first failure
} }
// InputMode represents the input mode for test cases
type InputMode string
const (
InputModeFile InputMode = "file" // JSONL file input
InputModeMessage InputMode = "message" // Direct message input
InputModeScript InputMode = "script" // Script test mode (NEW)
)
```
### Input Mode Detection
The input mode is automatically detected based on the input value:
| Input Pattern | Mode | Description |
| ----------------- | --------- | -------------------------- |
| `scripts.xxx.yyy` | `script` | Script test mode |
| `*.jsonl` | `file` | JSONL file mode |
| `path/to/file` | `file` | File path (if file exists) |
| `"any text"` | `message` | Direct message mode |
```go
func DetectInputMode(input string) InputMode {
// Check for script test prefix
if strings.HasPrefix(input, "scripts.") {
return InputModeScript
}
// Check if it's a file path
if strings.HasSuffix(input, ".jsonl") || fileExists(input) {
return InputModeFile
}
// Default to message mode
return InputModeMessage
}
```
## Script Testing Implementation
### Script Resolution
```go
// ResolveScript resolves the script path from scripts.xxx.yyy format
func ResolveScript(input string) (*ScriptInfo, error) {
// Remove "scripts." prefix
path := strings.TrimPrefix(input, "scripts.")
// Split into parts: "expense.setup" -> ["expense", "setup"]
parts := strings.Split(path, ".")
if len(parts) < 2 {
return nil, fmt.Errorf("invalid script path: %s", input)
}
// Build paths
// assistantDir: expense
// moduleName: setup
// scriptPath: expense/src/setup.ts
// testPath: expense/src/setup_test.ts
assistantDir := parts[0]
moduleName := parts[1]
return &ScriptInfo{
ID: input,
Assistant: assistantDir,
Module: moduleName,
ScriptPath: filepath.Join(assistantDir, "src", moduleName+".ts"),
TestPath: filepath.Join(assistantDir, "src", moduleName+"_test.ts"),
}, nil
}
```
### Test Function Discovery
Test functions are discovered by scanning the script for functions starting with `Test`:
```go
// DiscoverTests finds all Test* functions in the script
func DiscoverTests(scriptPath string) ([]*ScriptTestCase, error) {
// Use the JavaScript runtime to list exported functions
// Filter for functions starting with "Test"
// Return list of test cases
}
```
### Testing Object Binding
The `testing.T` object is provided to test functions via JavaScript runtime binding:
```go
// TestingT represents the testing object passed to test functions
type TestingT struct {
name string
failed bool
skipped bool
logs []string
assert *AssertObject
}
// AssertObject provides assertion methods
type AssertObject struct {
t *TestingT
}
func (a *AssertObject) True(value bool, message ...string) {
if !value {
a.t.fail(formatMessage("expected true, got false", message))
}
}
func (a *AssertObject) Equal(actual, expected interface{}, message ...string) {
if !reflect.DeepEqual(actual, expected) {
a.t.fail(formatMessage(
fmt.Sprintf("expected %v, got %v", expected, actual),
message,
))
}
}
// ... other assertion methods
```
### Script Execution Flow
```
1. Parse input: "scripts.expense.setup"
2. Resolve script info:
- TestPath: expense/src/setup_test.ts
- ScriptPath: expense/src/setup.ts
3. Discover test functions: [TestSystemReady, TestSetupWithMockData, ...]
4. For each test function:
a. Create testing.T object
b. Create agent.Context with environment
c. Execute: TestFunction(t, ctx)
d. Collect result (passed/failed/skipped)
5. Generate report
```
### Integration with Existing Runner
```go
func (r *Executor) Run() (*Report, error) {
switch r.opts.InputMode {
case InputModeScript:
return r.RunScriptTests()
case InputModeMessage:
return r.RunDirect()
default:
return r.RunTests()
}
}
func (r *Executor) RunScriptTests() (*Report, error) {
// 1. Resolve script
scriptInfo, err := ResolveScript(r.opts.Input)
if err != nil {
return nil, err
}
// 2. Discover tests
tests, err := DiscoverTests(scriptInfo.TestPath)
if err != nil {
return nil, err
}
// 3. Run each test
results := make([]*ScriptTestResult, 0, len(tests))
for _, tc := range tests {
result := r.runScriptTest(tc, scriptInfo)
results = append(results, result)
if r.opts.FailFast && result.Status == StatusFailed {
break
}
}
// 4. Generate report
return r.buildScriptReport(scriptInfo, results), nil
}
``` ```
## Exit Codes ## Exit Codes
@ -785,3 +1506,8 @@ The command exits with code 1 if any tests fail, making it easy to integrate wit
5. **Diff Reports**: Compare results between runs 5. **Diff Reports**: Compare results between runs
6. **Flaky Test Detection**: Automatic identification of unstable tests 6. **Flaky Test Detection**: Automatic identification of unstable tests
7. **Test Prioritization**: Run most important/failing tests first 7. **Test Prioritization**: Run most important/failing tests first
8. **Script Test Enhancements**:
- Parallel script test execution
- Setup/Teardown hooks (`TestMain`, `BeforeEach`, `AfterEach`)
- Mocking utilities for external dependencies
- Code coverage for TypeScript/JavaScript scripts

223
agent/test/README.md
View file

@ -4,6 +4,8 @@ A testing framework for Yao AI agents with support for assertions, stability ana
## Quick Start ## Quick Start
### Agent Tests
```bash ```bash
# Test with direct message (auto-detect agent from current directory) # Test with direct message (auto-detect agent from current directory)
cd assistants/keyword cd assistants/keyword
@ -22,9 +24,22 @@ yao agent test -i tests/inputs.jsonl -o report.html
yao agent test -i tests/inputs.jsonl --runs 5 yao agent test -i tests/inputs.jsonl --runs 5
``` ```
### Script Tests
```bash
# Test agent handler scripts (hooks, tools, setup functions)
yao agent test -i scripts.expense.setup -v
# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady" -v
# Run with custom context (authorization, metadata)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```
## Input Modes ## Input Modes
The `-i` flag supports two input modes: The `-i` flag supports three input modes:
### 1. JSONL File Mode ### 1. JSONL File Mode
@ -51,22 +66,101 @@ yao agent test -i "Hello" -n workers.system.keyword
Output is printed to stdout (or saved to `-o` if specified). Output is printed to stdout (or saved to `-o` if specified).
### 3. Script Test Mode
Test agent handler scripts (hooks, tools, setup functions):
```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v
# Run specific tests with filtering
yao agent test -i scripts.expense.setup --run "TestSystemReady"
# Run with custom context
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
```
Script test input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup` → `assistants/expense/src/setup_test.ts`).
**Writing Test Scripts:**
Test scripts should be placed alongside the source files with `_test.ts` or `_test.js` suffix:
```
assistants/expense/src/
├── setup.ts # Source file
├── setup_test.ts # Test file
├── tools.ts
└── tools_test.ts
```
Test functions must follow the naming convention `Test*` and accept `(t: testing.T, ctx: agent.Context)`:
```typescript
// assistants/expense/src/setup_test.ts
import { SystemReady } from "./setup";
// Test function signature: function Test*(t: testing.T, ctx: agent.Context)
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
const result = SystemReady(ctx);
// Use t.assert for assertions
t.assert.True(result.success, "SystemReady should succeed");
t.assert.Equal(result.status, "ready", "Status should be ready");
t.assert.NotNil(result.data, "Data should not be nil");
}
export function TestSystemReadyError(t: testing.T, ctx: agent.Context) {
// Access context properties
console.log("Testing with user:", ctx.authorized.user_id);
// Test error handling
const result = SystemReady(ctx);
t.assert.False(result.error, "Should not have error");
}
```
**Available Assertions:**
| Method | Description |
| -------------------------------- | ------------------------------ |
| `t.assert.True(value, msg)` | Assert value is true |
| `t.assert.False(value, msg)` | Assert value is false |
| `t.assert.Equal(a, b, msg)` | Assert a equals b |
| `t.assert.NotEqual(a, b, msg)` | Assert a not equals b |
| `t.assert.Nil(value, msg)` | Assert value is null/undefined |
| `t.assert.NotNil(value, msg)` | Assert value is not nil |
| `t.assert.Contains(s, sub, msg)` | Assert string contains substr |
| `t.assert.Len(arr, n, msg)` | Assert array/string length |
**Test Control:**
| Method | Description |
| -------------- | ---------------------------- |
| `t.Log(msg)` | Log a message |
| `t.Error(msg)` | Mark test as failed with msg |
| `t.Fatal(msg)` | Mark failed and stop test |
| `t.Skip(msg)` | Skip this test |
## Command Line Options ## Command Line Options
| Flag | Description | Default | | Flag | Description | Default |
| ------------- | ---------------------------------------- | -------------------------- | | ------------- | -------------------------------------------------- | -------------------------- |
| `-i` | Input: JSONL file path or direct message | (required) | | `-i` | Input: JSONL file path, message, or script ID | (required) |
| `-o` | Output file path | `output-{timestamp}.jsonl` | | `-o` | Output file path | `output-{timestamp}.jsonl` |
| `-n` | Agent ID (optional, auto-detected) | auto-detect | | `-n` | Agent ID (optional, auto-detected) | auto-detect |
| `-c` | Override connector | agent default | | `-c` | Override connector | agent default |
| `-u` | Test user ID | `test-user` | | `-u` | Test user ID | `test-user` |
| `-t` | Test team ID | `test-team` | | `-t` | Test team ID | `test-team` |
| `-r` | Reporter agent ID | built-in | | `--ctx` | Path to context JSON file for custom authorization | - |
| `--runs` | Runs per test (stability analysis) | 1 | | `-r` | Reporter agent ID | built-in |
| `--timeout` | Timeout per test | 5m | | `--runs` | Runs per test (stability analysis) | 1 |
| `--parallel` | Parallel test cases | 1 | | `--run` | Regex pattern to filter which tests to run | - |
| `-v` | Verbose output | false | | `--timeout` | Timeout per test | 5m |
| `--fail-fast` | Stop on first failure | false | | `--parallel` | Parallel test cases | 1 |
| `-v` | Verbose output | false |
| `--fail-fast` | Stop on first failure | false |
## Agent Resolution ## Agent Resolution
@ -119,24 +213,24 @@ Each line is a JSON object:
The `options` field allows per-test-case configuration that maps to `context.Options`: The `options` field allows per-test-case configuration that maps to `context.Options`:
| Field | Type | Description | | Field | Type | Description |
| ------------------------ | ------- | -------------------------------------------------- | | ------------------------ | ------ | -------------------------------------------- |
| `connector` | string | Override connector (e.g., `"deepseek.v3"`) | | `connector` | string | Override connector (e.g., `"deepseek.v3"`) |
| `mode` | string | Agent mode (default: `"chat"`) | | `mode` | string | Agent mode (default: `"chat"`) |
| `search` | bool | Enable/disable search mode (default: `true`) | | `search` | bool | Enable/disable search mode (default: `true`) |
| `disable_global_prompts` | bool | Temporarily disable global prompts | | `disable_global_prompts` | bool | Temporarily disable global prompts |
| `metadata` | map | Custom data passed to hooks (e.g., scenario) | | `metadata` | map | Custom data passed to hooks (e.g., scenario) |
| `skip` | object | Skip configuration (see below) | | `skip` | object | Skip configuration (see below) |
#### Options.skip #### Options.skip
| Field | Type | Description | | Field | Type | Description |
| --------- | ---- | ------------------------ | | --------- | ---- | ----------------------- |
| `history` | bool | Skip history loading | | `history` | bool | Skip history loading |
| `trace` | bool | Skip trace logging | | `trace` | bool | Skip trace logging |
| `output` | bool | Skip output to client | | `output` | bool | Skip output to client |
| `keyword` | bool | Skip keyword extraction | | `keyword` | bool | Skip keyword extraction |
| `search` | bool | Skip auto search | | `search` | bool | Skip auto search |
**Example with options:** **Example with options:**
@ -146,10 +240,18 @@ The `options` field allows per-test-case configuration that maps to `context.Opt
"input": "Query users with status active", "input": "Query users with status active",
"options": { "options": {
"connector": "deepseek.v3", "connector": "deepseek.v3",
"metadata": {"scenario": "filter"}, "metadata": {
"skip": {"trace": true} "scenario": "filter"
},
"skip": {
"trace": true
}
}, },
"assert": {"type": "json_path", "path": "from", "value": "users"} "assert": {
"type": "json_path",
"path": "from",
"value": "users"
}
} }
``` ```
@ -290,7 +392,17 @@ return { pass: true, message: "Validation passed" };
**Multiple expected values (OR logic):** **Multiple expected values (OR logic):**
```jsonl ```jsonl
{"id": "T005", "assert": {"type": "json_path", "path": "error", "value": ["missing_schema", "missing_query"]}} {
"id": "T005",
"assert": {
"type": "json_path",
"path": "error",
"value": [
"missing_schema",
"missing_query"
]
}
}
``` ```
This passes if `error` equals either `"missing_schema"` or `"missing_query"`. This passes if `error` equals either `"missing_schema"` or `"missing_query"`.
@ -382,6 +494,9 @@ yao agent test -i tests/inputs.jsonl -o results.jsonl --fail-fast
# Parse JSONL results # Parse JSONL results
cat results.jsonl | jq 'select(.type == "summary")' cat results.jsonl | jq 'select(.type == "summary")'
# Run script tests
yao agent test -i scripts.expense.setup --fail-fast
``` ```
### GitHub Actions Example ### GitHub Actions Example
@ -394,6 +509,18 @@ cat results.jsonl | jq 'select(.type == "summary")'
--runs 3 \ --runs 3 \
-o report.json -o report.json
- name: Run Script Tests
run: |
yao agent test -i scripts.expense.setup -v
yao agent test -i scripts.expense.tools -v
- name: Run Script Tests with Custom Context
run: |
yao agent test -i scripts.expense.setup \
--ctx tests/context.json \
--run "TestSystem.*" \
-v
- name: Check Stability - name: Check Stability
run: | run: |
jq -e '.results | all(.pass_rate >= 80)' report.json jq -e '.results | all(.pass_rate >= 80)' report.json
@ -401,6 +528,8 @@ cat results.jsonl | jq 'select(.type == "summary")'
## Examples ## Examples
### Agent Tests
```bash ```bash
# Quick development test (auto-detect agent) # Quick development test (auto-detect agent)
cd assistants/keyword cd assistants/keyword
@ -440,6 +569,32 @@ yao agent test -i tests/inputs.jsonl \
-o report.html -o report.html
``` ```
### Script Tests
```bash
# Run all tests in a script module
yao agent test -i scripts.expense.setup -v
# Run specific tests with regex filter
yao agent test -i scripts.expense.setup --run "TestSystemReady"
# Run tests matching a pattern
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v
# Run with custom context (authorization, metadata, etc.)
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
# Run with specific user/team
yao agent test -i scripts.expense.setup -u admin -t ops-team -v
# Combine options
yao agent test -i scripts.expense.setup \
--ctx tests/context.json \
--run "TestSystem.*" \
--timeout 30s \
-v
```
## Exit Codes ## Exit Codes
| Code | Description | | Code | Description |

26
agent/test/assert.go
View file

@ -255,13 +255,27 @@ func (a *Asserter) assertJSONPath(assertion *Assertion, output interface{}) *Ass
result.Actual = actual result.Actual = actual
// Compare expected value with actual value // Compare expected value with actual value
// First, try direct comparison (handles both primitive values and arrays) // If expected is an array, check if actual matches ANY element (IN semantics)
if validateOutput(actual, assertion.Value) { if expectedArr, ok := assertion.Value.([]interface{}); ok {
result.Passed = true // Check if actual is one of the expected values
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path) for _, expectedItem := range expectedArr {
} else { if validateOutput(actual, expectedItem) {
result.Passed = true
result.Message = fmt.Sprintf("path '%s' equals one of expected values", assertion.Path)
return result
}
}
result.Passed = false result.Passed = false
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual) result.Message = fmt.Sprintf("path '%s': expected one of %v, got %v", assertion.Path, assertion.Value, actual)
} else {
// Direct comparison for non-array expected values
if validateOutput(actual, assertion.Value) {
result.Passed = true
result.Message = fmt.Sprintf("path '%s' equals expected value", assertion.Path)
} else {
result.Passed = false
result.Message = fmt.Sprintf("path '%s': expected %v, got %v", assertion.Path, assertion.Value, actual)
}
} }
return result return result

63
agent/test/context.go
View file

@ -13,11 +13,7 @@ import (
// but configurable via Environment // but configurable via Environment
func NewTestContext(chatID, assistantID string, env *Environment) *context.Context { func NewTestContext(chatID, assistantID string, env *Environment) *context.Context {
// Build authorized info from environment // Build authorized info from environment
authorized := &types.AuthorizedInfo{ authorized := buildAuthorizedInfo(env)
Subject: env.UserID,
UserID: env.UserID,
TenantID: env.TeamID,
}
// Create context with standard initialization // Create context with standard initialization
ctx := context.New(stdContext.Background(), authorized, chatID) ctx := context.New(stdContext.Background(), authorized, chatID)
@ -34,6 +30,13 @@ func NewTestContext(chatID, assistantID string, env *Environment) *context.Conte
ctx.IDGenerator = message.NewIDGenerator() ctx.IDGenerator = message.NewIDGenerator()
ctx.Metadata = make(map[string]interface{}) ctx.Metadata = make(map[string]interface{})
// Apply metadata from context config if available
if env.ContextConfig != nil && env.ContextConfig.Metadata != nil {
for k, v := range env.ContextConfig.Metadata {
ctx.Metadata[k] = v
}
}
// Initialize interrupt controller // Initialize interrupt controller
ctx.Interrupt = context.NewInterruptController() ctx.Interrupt = context.NewInterruptController()
@ -47,6 +50,56 @@ func NewTestContext(chatID, assistantID string, env *Environment) *context.Conte
return ctx return ctx
} }
// buildAuthorizedInfo builds AuthorizedInfo from Environment
func buildAuthorizedInfo(env *Environment) *types.AuthorizedInfo {
authorized := &types.AuthorizedInfo{
Subject: env.UserID,
UserID: env.UserID,
TenantID: env.TeamID,
}
// Apply custom authorized config if available
if env.ContextConfig != nil && env.ContextConfig.Authorized != nil {
authCfg := env.ContextConfig.Authorized
if authCfg.Sub != "" {
authorized.Subject = authCfg.Sub
}
if authCfg.ClientID != "" {
authorized.ClientID = authCfg.ClientID
}
if authCfg.Scope != "" {
authorized.Scope = authCfg.Scope
}
if authCfg.SessionID != "" {
authorized.SessionID = authCfg.SessionID
}
if authCfg.UserID != "" {
authorized.UserID = authCfg.UserID
}
if authCfg.TeamID != "" {
authorized.TeamID = authCfg.TeamID
}
if authCfg.TenantID != "" {
authorized.TenantID = authCfg.TenantID
}
authorized.RememberMe = authCfg.RememberMe
// Apply constraints
if authCfg.Constraints != nil {
authorized.Constraints = types.DataConstraints{
OwnerOnly: authCfg.Constraints.OwnerOnly,
CreatorOnly: authCfg.Constraints.CreatorOnly,
EditorOnly: authCfg.Constraints.EditorOnly,
TeamOnly: authCfg.Constraints.TeamOnly,
Extra: authCfg.Constraints.Extra,
}
}
}
return authorized
}
// NewTestContextFromOptions creates a test context from test options and test case // NewTestContextFromOptions creates a test context from test options and test case
func NewTestContextFromOptions(chatID, assistantID string, opts *Options, tc *Case) *context.Context { func NewTestContextFromOptions(chatID, assistantID string, opts *Options, tc *Case) *context.Context {
// Get environment from test case (with options override) // Get environment from test case (with options override)

46
agent/test/output.go
View file

@ -257,6 +257,52 @@ func (w *OutputWriter) DirectOutput(output interface{}) {
} }
} }
// ScriptTestSummary prints the script test summary
func (w *OutputWriter) ScriptTestSummary(summary *ScriptTestSummary, duration time.Duration) {
w.SubHeader("Summary")
// Results
color.New(color.FgWhite).Printf(" Total: ")
fmt.Printf("%d\n", summary.Total)
color.New(color.FgWhite).Printf(" Passed: ")
if summary.Passed > 0 {
color.New(color.FgGreen).Printf("%d\n", summary.Passed)
} else {
fmt.Printf("%d\n", summary.Passed)
}
color.New(color.FgWhite).Printf(" Failed: ")
if summary.Failed > 0 {
color.New(color.FgRed).Printf("%d\n", summary.Failed)
} else {
fmt.Printf("%d\n", summary.Failed)
}
if summary.Skipped > 0 {
color.New(color.FgWhite).Printf(" Skipped: ")
color.New(color.FgYellow).Printf("%d\n", summary.Skipped)
}
// Pass rate
passRate := float64(0)
if summary.Total > 0 {
passRate = float64(summary.Passed) / float64(summary.Total) * 100
}
color.New(color.FgWhite).Printf(" Pass Rate: ")
if passRate == 100 {
color.New(color.FgGreen, color.Bold).Printf("%.1f%%\n", passRate)
} else if passRate >= 80 {
color.New(color.FgYellow).Printf("%.1f%%\n", passRate)
} else {
color.New(color.FgRed).Printf("%.1f%%\n", passRate)
}
// Duration
color.New(color.FgWhite).Printf(" Duration: ")
fmt.Printf("%s\n", formatDuration(duration))
}
// StabilityResult prints stability analysis result for a test case // StabilityResult prints stability analysis result for a test case
func (w *OutputWriter) StabilityResult(sr *StabilityResult) { func (w *OutputWriter) StabilityResult(sr *StabilityResult) {
color.New(color.FgWhite).Printf(" [%s] ", sr.ID) color.New(color.FgWhite).Printf(" [%s] ", sr.ID)

16
agent/test/resolver.go
View file

@ -213,8 +213,16 @@ func DefaultOptions() *Options {
} }
// DetectInputMode detects the input mode from the input string // DetectInputMode detects the input mode from the input string
// Returns InputModeFile if input looks like a file path, InputModeMessage otherwise // Returns:
// - InputModeScript: if input starts with "scripts."
// - InputModeFile: if input ends with ".jsonl" or is an existing file
// - InputModeMessage: otherwise (direct message mode)
func DetectInputMode(input string) InputMode { func DetectInputMode(input string) InputMode {
// Check for script test prefix
if strings.HasPrefix(input, "scripts.") {
return InputModeScript
}
// If input ends with .jsonl or .json, treat as file // If input ends with .jsonl or .json, treat as file
if strings.HasSuffix(input, ".jsonl") || strings.HasSuffix(input, ".json") { if strings.HasSuffix(input, ".jsonl") || strings.HasSuffix(input, ".json") {
return InputModeFile return InputModeFile
@ -269,6 +277,12 @@ func MergeOptions(opts *Options, defaults *Options) *Options {
if opts.ReporterID != "" { if opts.ReporterID != "" {
result.ReporterID = opts.ReporterID result.ReporterID = opts.ReporterID
} }
if opts.ContextFile != "" {
result.ContextFile = opts.ContextFile
}
if opts.Run != "" {
result.Run = opts.Run
}
if opts.Verbose { if opts.Verbose {
result.Verbose = opts.Verbose result.Verbose = opts.Verbose
} }

32
agent/test/runner.go
View file

@ -33,6 +33,11 @@ func NewRunner(opts *Options) *Executor {
// Run executes all test cases and returns a report // Run executes all test cases and returns a report
func (r *Executor) Run() (*Report, error) { func (r *Executor) Run() (*Report, error) {
// For script test mode, use script runner
if r.opts.InputMode == InputModeScript {
return r.RunScriptTests()
}
// For direct message mode, use simplified output (development mode) // For direct message mode, use simplified output (development mode)
if r.opts.InputMode == InputModeMessage { if r.opts.InputMode == InputModeMessage {
return r.RunDirect() return r.RunDirect()
@ -41,6 +46,33 @@ func (r *Executor) Run() (*Report, error) {
return r.RunTests() return r.RunTests()
} }
// RunScriptTests executes script tests and returns a report
func (r *Executor) RunScriptTests() (*Report, error) {
scriptRunner := NewScriptRunner(r.opts)
scriptReport, err := scriptRunner.Run()
if err != nil {
return nil, err
}
// Convert to standard report for unified output handling
report := scriptReport.ToReport()
// Write output if specified
if r.opts.OutputFile != "" {
err = r.writeOutput(report)
if err != nil {
r.output.Error("Failed to write output: %s", err.Error())
} else {
r.output.OutputFile(r.opts.OutputFile)
}
}
// Print final result
r.output.FinalResult(!report.HasFailures())
return report, nil
}
// RunDirect executes a single direct message and outputs the result directly // RunDirect executes a single direct message and outputs the result directly
// This is optimized for development/debugging scenarios // This is optimized for development/debugging scenarios
func (r *Executor) RunDirect() (*Report, error) { func (r *Executor) RunDirect() (*Report, error) {

482
agent/test/script.go Normal file
View file

@ -0,0 +1,482 @@
package test
import (
"fmt"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/yaoapp/gou/application"
v8 "github.com/yaoapp/gou/runtime/v8"
"github.com/yaoapp/gou/runtime/v8/bridge"
"github.com/yaoapp/yao/agent/context"
"rogchap.com/v8go"
)
// ScriptRunner executes script tests
type ScriptRunner struct {
opts *Options
output *OutputWriter
}
// NewScriptRunner creates a new script test runner
func NewScriptRunner(opts *Options) *ScriptRunner {
return &ScriptRunner{
opts: opts,
output: NewOutputWriter(opts.Verbose),
}
}
// ResolveScript resolves the script path from scripts.xxx.yyy format
func ResolveScript(input string) (*ScriptInfo, error) {
// Remove "scripts." prefix
path := strings.TrimPrefix(input, "scripts.")
// Split into parts: "expense.setup" -> ["expense", "setup"]
parts := strings.Split(path, ".")
if len(parts) < 2 {
return nil, fmt.Errorf("invalid script path: %s (expected format: scripts.assistant.module)", input)
}
// Build paths
// assistantDir: expense
// moduleName: setup
// scriptPath: expense/src/setup.ts (or assistants/expense/src/setup.ts)
// testPath: expense/src/setup_test.ts
assistantDir := parts[0]
moduleName := parts[1]
// Try different path patterns
basePaths := []string{
filepath.Join("assistants", assistantDir, "src"),
filepath.Join(assistantDir, "src"),
}
var scriptPath, testPath string
for _, basePath := range basePaths {
// Check for TypeScript files first, then JavaScript
for _, ext := range []string{".ts", ".js"} {
candidateScript := filepath.Join(basePath, moduleName+ext)
candidateTest := filepath.Join(basePath, moduleName+"_test"+ext)
// Check if test file exists
exists, err := application.App.Exists(candidateTest)
if err == nil && exists {
scriptPath = candidateScript
testPath = candidateTest
break
}
}
if testPath != "" {
break
}
}
if testPath == "" {
return nil, fmt.Errorf("test file not found for %s (tried: %s)", input, strings.Join(basePaths, ", "))
}
return &ScriptInfo{
ID: input,
Assistant: assistantDir,
Module: moduleName,
ScriptPath: scriptPath,
TestPath: testPath,
}, nil
}
// DiscoverTests finds all Test* functions in the script
func DiscoverTests(scriptPath string) ([]*ScriptTestCase, error) {
// Read the script file
content, err := application.App.Read(scriptPath)
if err != nil {
return nil, fmt.Errorf("failed to read script: %w", err)
}
// Parse the script to find Test* functions
// We use a simple regex-like approach to find function declarations
tests := make([]*ScriptTestCase, 0)
lines := strings.Split(string(content), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
// Match function declarations: function TestXxx( or export function TestXxx(
if strings.Contains(line, "function Test") {
// Extract function name
name := extractFunctionName(line)
if name != "" && strings.HasPrefix(name, "Test") {
tests = append(tests, &ScriptTestCase{
Name: name,
Function: name,
})
}
}
}
return tests, nil
}
// extractFunctionName extracts the function name from a line
func extractFunctionName(line string) string {
// Remove "export" prefix if present
line = strings.TrimPrefix(line, "export ")
line = strings.TrimSpace(line)
// Match "function Name("
if !strings.HasPrefix(line, "function ") {
return ""
}
line = strings.TrimPrefix(line, "function ")
// Find the opening parenthesis
idx := strings.Index(line, "(")
if idx == -1 {
return ""
}
return strings.TrimSpace(line[:idx])
}
// filterTests filters test cases by a regex pattern (similar to go test -run)
func (r *ScriptRunner) filterTests(tests []*ScriptTestCase, pattern string) ([]*ScriptTestCase, error) {
re, err := regexp.Compile(pattern)
if err != nil {
return nil, err
}
filtered := make([]*ScriptTestCase, 0)
for _, tc := range tests {
if re.MatchString(tc.Name) {
filtered = append(filtered, tc)
}
}
return filtered, nil
}
// Run executes all script tests and returns a report
func (r *ScriptRunner) Run() (*ScriptTestReport, error) {
startTime := time.Now()
// Resolve script
scriptInfo, err := ResolveScript(r.opts.Input)
if err != nil {
return nil, err
}
// Print header
r.output.Header("Script Test")
r.output.Info("Script: %s", scriptInfo.TestPath)
// Discover tests
tests, err := DiscoverTests(scriptInfo.TestPath)
if err != nil {
return nil, err
}
// Filter tests by -run pattern if specified
if r.opts.Run != "" {
tests, err = r.filterTests(tests, r.opts.Run)
if err != nil {
return nil, fmt.Errorf("invalid -run pattern: %w", err)
}
r.output.Info("Tests: %d functions (filtered by: %s)", len(tests), r.opts.Run)
} else {
r.output.Info("Tests: %d functions", len(tests))
}
if len(tests) == 0 {
r.output.Warning("No tests to run")
}
// Load context config if specified
var ctxConfig *ContextConfig
if r.opts.ContextFile != "" {
var err error
ctxConfig, err = LoadContextConfig(r.opts.ContextFile)
if err != nil {
return nil, fmt.Errorf("failed to load context file: %w", err)
}
r.output.Info("Context: %s", r.opts.ContextFile)
}
// Create environment with optional context config
var env *Environment
if ctxConfig != nil {
env = NewEnvironmentWithContext(r.opts.UserID, r.opts.TeamID, ctxConfig)
} else {
env = NewEnvironment(r.opts.UserID, r.opts.TeamID)
}
r.output.Info("User: %s", env.UserID)
r.output.Info("Team: %s", env.TeamID)
// Load all scripts from src directory (including the test file)
// This ensures imports can be resolved properly
srcDir := filepath.Dir(scriptInfo.TestPath)
loadedCount, err := r.loadAllScripts(srcDir)
if err != nil {
return nil, fmt.Errorf("failed to load scripts: %w", err)
}
r.output.Info("Loaded: %d scripts", loadedCount)
// Create report
report := &ScriptTestReport{
Type: "script_test",
Script: scriptInfo.ID,
ScriptPath: scriptInfo.TestPath,
Summary: &ScriptTestSummary{Total: len(tests)},
Environment: env,
Results: make([]*ScriptTestResult, 0, len(tests)),
Metadata: &ScriptTestMetadata{
StartedAt: startTime,
},
}
// Run tests
r.output.SubHeader("Running Tests")
for _, tc := range tests {
result := r.runScriptTest(tc, scriptInfo, env)
report.Results = append(report.Results, result)
// Update summary
switch result.Status {
case StatusPassed:
report.Summary.Passed++
case StatusFailed:
report.Summary.Failed++
case StatusSkipped:
report.Summary.Skipped++
}
// Check fail-fast
if r.opts.FailFast && result.Status == StatusFailed {
r.output.Warning("Stopping due to --fail-fast")
break
}
}
// Complete report
report.Summary.DurationMs = time.Since(startTime).Milliseconds()
report.Metadata.CompletedAt = time.Now()
// Print summary
r.output.ScriptTestSummary(report.Summary, time.Since(startTime))
return report, nil
}
// runScriptTest runs a single script test function
func (r *ScriptRunner) runScriptTest(tc *ScriptTestCase, scriptInfo *ScriptInfo, env *Environment) *ScriptTestResult {
r.output.TestStart(tc.Name, "", 1)
startTime := time.Now()
result := &ScriptTestResult{
Name: tc.Name,
Status: StatusPassed,
}
// Create testing.T object
testingT := NewTestingT(tc.Name)
// Create agent context
chatID := fmt.Sprintf("script-test-%s", tc.Name)
agentCtx := NewTestContext(chatID, scriptInfo.Assistant, env)
defer agentCtx.Release()
// Execute the test function
err := r.executeTestFunction(tc, scriptInfo, testingT, agentCtx)
duration := time.Since(startTime)
result.DurationMs = duration.Milliseconds()
result.Logs = testingT.Logs()
if err != nil {
result.Status = StatusError
result.Error = err.Error()
r.output.TestResult(result.Status, duration)
r.output.TestError(result.Error)
return result
}
if testingT.Skipped() {
result.Status = StatusSkipped
r.output.TestResult(result.Status, duration)
return result
}
if testingT.Failed() {
result.Status = StatusFailed
errors := testingT.Errors()
if len(errors) > 0 {
result.Error = errors[0]
}
result.Assertion = testingT.AssertionInfo()
r.output.TestResult(result.Status, duration)
r.output.TestError(result.Error)
return result
}
r.output.TestResult(result.Status, duration)
return result
}
// loadAllScripts loads all scripts from the src directory
// This ensures that imports can be resolved properly
func (r *ScriptRunner) loadAllScripts(srcDir string) (int, error) {
count := 0
// Check if src directory exists
exists, err := application.App.Exists(srcDir)
if err != nil {
return 0, err
}
if !exists {
return 0, fmt.Errorf("src directory not found: %s", srcDir)
}
// Walk through src directory to find all script files
exts := []string{"*.ts", "*.js"}
err = application.App.Walk(srcDir, func(root, file string, isdir bool) error {
if isdir {
return nil
}
// Get relative path
relPath := strings.TrimPrefix(file, root+"/")
// Generate script ID from file path
scriptID := generateTestScriptID(file, root)
// Load the script
_, err := v8.Load(file, scriptID)
if err != nil {
// Log warning but continue loading other scripts
if r.opts.Verbose {
r.output.Warning("Failed to load %s: %v", relPath, err)
}
return nil
}
count++
if r.opts.Verbose {
r.output.Verbose("Loaded: %s", relPath)
}
return nil
}, exts...)
if err != nil {
return count, fmt.Errorf("failed to walk src directory: %w", err)
}
return count, nil
}
// generateTestScriptID generates a script ID from file path for testing
func generateTestScriptID(filePath string, srcDir string) string {
// Normalize path separators
filePath = filepath.ToSlash(filePath)
srcDir = filepath.ToSlash(srcDir)
// Remove src directory prefix
relPath := strings.TrimPrefix(filePath, srcDir+"/")
relPath = strings.TrimPrefix(relPath, "/")
// Remove file extension
relPath = strings.TrimSuffix(relPath, filepath.Ext(relPath))
// Replace path separators with dots and add test prefix
scriptID := "test." + strings.ReplaceAll(relPath, "/", ".")
return scriptID
}
// executeTestFunction executes a single test function using V8
func (r *ScriptRunner) executeTestFunction(tc *ScriptTestCase, scriptInfo *ScriptInfo, testingT *TestingT, agentCtx *context.Context) error {
// Get the test script (already loaded by loadAllScripts)
testScriptID := generateTestScriptID(scriptInfo.TestPath, filepath.Dir(scriptInfo.TestPath))
script, ok := v8.Scripts[testScriptID]
if !ok {
return fmt.Errorf("test script not found: %s (id: %s)", scriptInfo.TestPath, testScriptID)
}
// Create a new script context
scriptCtx, err := script.NewContext("", nil)
if err != nil {
return fmt.Errorf("failed to create script context: %w", err)
}
defer scriptCtx.Close()
// Get the V8 context
v8ctx := scriptCtx.Context
// Create testing.T JavaScript object
testingTObj, err := NewTestingTObject(v8ctx, testingT)
if err != nil {
return fmt.Errorf("failed to create testing.T object: %w", err)
}
// Create agent context JavaScript object
agentCtxObj, err := agentCtx.JsValue(v8ctx)
if err != nil {
return fmt.Errorf("failed to create agent context object: %w", err)
}
// Get the test function
global := v8ctx.Global()
fnValue, err := global.Get(tc.Function)
if err != nil {
return fmt.Errorf("failed to get test function %s: %w", tc.Function, err)
}
if !fnValue.IsFunction() {
return fmt.Errorf("test function %s is not a function", tc.Function)
}
fn, err := fnValue.AsFunction()
if err != nil {
return fmt.Errorf("failed to convert to function: %w", err)
}
// Call the test function with (t, ctx)
_, err = fn.Call(global, testingTObj, agentCtxObj)
if err != nil {
// Check if this is an assertion failure or a real error
if testingT.Failed() {
// Assertion failure - already recorded
return nil
}
return fmt.Errorf("test function error: %w", err)
}
return nil
}
// RegisterTestingGlobals registers testing-related global functions for V8
// This is called once during initialization
func RegisterTestingGlobals() {
v8.RegisterFunction("__testing_log", testingLogEmbed)
}
// testingLogEmbed provides a console.log-like function for tests
func testingLogEmbed(iso *v8go.Isolate) *v8go.FunctionTemplate {
return v8go.NewFunctionTemplate(iso, func(info *v8go.FunctionCallbackInfo) *v8go.Value {
args := info.Args()
parts := make([]string, len(args))
for i, arg := range args {
goVal, err := bridge.GoValue(arg, info.Context())
if err != nil {
parts[i] = arg.String()
} else {
parts[i] = fmt.Sprintf("%v", goVal)
}
}
fmt.Println(strings.Join(parts, " "))
return v8go.Undefined(iso)
})
}

1000
agent/test/script_assert.go Normal file
View file

File diff suppressed because it is too large Load diff

170
agent/test/script_types.go Normal file
View file

@ -0,0 +1,170 @@
package test
import "time"
// ScriptInfo contains information about the script being tested
type ScriptInfo struct {
// ID is the script identifier (e.g., "scripts.expense.setup")
ID string `json:"id"`
// Assistant is the assistant directory name (e.g., "expense")
Assistant string `json:"assistant"`
// Module is the module name (e.g., "setup")
Module string `json:"module"`
// ScriptPath is the path to the main script file (e.g., "expense/src/setup.ts")
ScriptPath string `json:"script_path"`
// TestPath is the path to the test script file (e.g., "expense/src/setup_test.ts")
TestPath string `json:"test_path"`
}
// ScriptTestCase represents a single script test function
type ScriptTestCase struct {
// Name is the test function name (e.g., "TestSystemReady")
Name string `json:"name"`
// Function is the full function reference
Function string `json:"function"`
}
// ScriptTestResult represents the result of running a script test function
type ScriptTestResult struct {
// Name is the test function name
Name string `json:"name"`
// Status is the test execution status
Status Status `json:"status"`
// DurationMs is the execution duration in milliseconds
DurationMs int64 `json:"duration_ms"`
// Error contains the error message if the test failed
Error string `json:"error,omitempty"`
// Assertion contains assertion failure details
Assertion *ScriptAssertionInfo `json:"assertion,omitempty"`
// Logs contains log messages from the test
Logs []string `json:"logs,omitempty"`
}
// ScriptAssertionInfo contains details about an assertion failure
type ScriptAssertionInfo struct {
// Type is the assertion type (e.g., "Equal", "True")
Type string `json:"type"`
// Expected is the expected value
Expected interface{} `json:"expected,omitempty"`
// Actual is the actual value
Actual interface{} `json:"actual,omitempty"`
// Message is the custom failure message
Message string `json:"message,omitempty"`
}
// ScriptTestSummary contains aggregated statistics for script tests
type ScriptTestSummary struct {
// Total number of test functions
Total int `json:"total"`
// Passed number of test functions that passed
Passed int `json:"passed"`
// Failed number of test functions that failed
Failed int `json:"failed"`
// Skipped number of test functions that were skipped
Skipped int `json:"skipped"`
// DurationMs is the total execution duration in milliseconds
DurationMs int64 `json:"duration_ms"`
}
// ScriptTestReport represents the complete script test report
type ScriptTestReport struct {
// Type indicates this is a script test report
Type string `json:"type"` // "script_test"
// Script is the script identifier (e.g., "scripts.expense.setup")
Script string `json:"script"`
// ScriptPath is the path to the test script file
ScriptPath string `json:"script_path"`
// Summary contains aggregated statistics
Summary *ScriptTestSummary `json:"summary"`
// Environment contains the test environment configuration
Environment *Environment `json:"environment"`
// Results contains individual test results
Results []*ScriptTestResult `json:"results"`
// Metadata contains additional report metadata
Metadata *ScriptTestMetadata `json:"metadata"`
}
// ScriptTestMetadata contains metadata about the script test report
type ScriptTestMetadata struct {
// StartedAt is when the test run started
StartedAt time.Time `json:"started_at"`
// CompletedAt is when the test run completed
CompletedAt time.Time `json:"completed_at"`
// Version is the Yao version
Version string `json:"version"`
}
// HasFailures returns true if there are any failed tests
func (r *ScriptTestReport) HasFailures() bool {
return r.Summary.Failed > 0
}
// PassRate returns the pass rate as a percentage (0-100)
func (r *ScriptTestReport) PassRate() float64 {
if r.Summary.Total == 0 {
return 0
}
return float64(r.Summary.Passed) / float64(r.Summary.Total) * 100
}
// ToReport converts ScriptTestReport to a standard Report for unified reporting
func (r *ScriptTestReport) ToReport() *Report {
return &Report{
Summary: &Summary{
Total: r.Summary.Total,
Passed: r.Summary.Passed,
Failed: r.Summary.Failed,
Skipped: r.Summary.Skipped,
DurationMs: r.Summary.DurationMs,
AgentID: r.Script,
AgentPath: r.ScriptPath,
},
Environment: r.Environment,
Results: r.toResults(),
Metadata: &ReportMetadata{
StartedAt: r.Metadata.StartedAt,
CompletedAt: r.Metadata.CompletedAt,
Version: r.Metadata.Version,
},
}
}
// toResults converts script test results to standard results
func (r *ScriptTestReport) toResults() []*Result {
results := make([]*Result, len(r.Results))
for i, sr := range r.Results {
results[i] = &Result{
ID: sr.Name,
Status: sr.Status,
Input: sr.Name,
DurationMs: sr.DurationMs,
Error: sr.Error,
}
}
return results
}

157
agent/test/types.go
View file

@ -1,7 +1,10 @@
package test package test
import ( import (
"encoding/json"
"fmt"
"math" "math"
"os"
"time" "time"
"github.com/yaoapp/yao/agent/context" "github.com/yaoapp/yao/agent/context"
@ -57,6 +60,8 @@ const (
InputModeFile InputMode = "file" InputModeFile InputMode = "file"
// InputModeMessage indicates input from a direct message string // InputModeMessage indicates input from a direct message string
InputModeMessage InputMode = "message" InputModeMessage InputMode = "message"
// InputModeScript indicates script test mode (testing agent handler scripts)
InputModeScript InputMode = "script"
) )
// Options represents the configuration options for running tests // Options represents the configuration options for running tests
@ -96,6 +101,14 @@ type Options struct {
// Locale is the locale for the test context (default: "en-us") // Locale is the locale for the test context (default: "en-us")
Locale string `json:"locale,omitempty"` Locale string `json:"locale,omitempty"`
// ContextFile is the path to a JSON file containing custom context data (-ctx flag)
// This allows full customization of authorized info, metadata, etc.
ContextFile string `json:"context_file,omitempty"`
// ContextData is the parsed context data from ContextFile
// This is populated internally after loading the file
ContextData *ContextConfig `json:"-"`
// Execution // Execution
// =============================== // ===============================
@ -126,6 +139,92 @@ type Options struct {
// FailFast stops execution on first failure // FailFast stops execution on first failure
FailFast bool `json:"fail_fast,omitempty"` FailFast bool `json:"fail_fast,omitempty"`
// Run is a regex pattern to filter which tests to run (similar to go test -run)
// Only tests matching the pattern will be executed
// Example: "TestSystem" matches TestSystemReady, TestSystemError, etc.
Run string `json:"run,omitempty"`
}
// ContextConfig represents custom context configuration from JSON file
// This allows full customization of the test context including authorized info
type ContextConfig struct {
// Authorized contains custom authorization data
Authorized *AuthorizedConfig `json:"authorized,omitempty"`
// Metadata contains custom metadata to pass to the context
Metadata map[string]interface{} `json:"metadata,omitempty"`
// Client contains custom client information
Client *ClientConfig `json:"client,omitempty"`
// Locale overrides the locale setting
Locale string `json:"locale,omitempty"`
// Referer overrides the referer setting
Referer string `json:"referer,omitempty"`
}
// AuthorizedConfig represents custom authorization configuration
// Matches the structure of types.AuthorizedInfo from openapi/oauth/types
type AuthorizedConfig struct {
// Sub is the subject identifier (JWT sub claim)
Sub string `json:"sub,omitempty"`
// ClientID is the OAuth client ID
ClientID string `json:"client_id,omitempty"`
// Scope is the access scope
Scope string `json:"scope,omitempty"`
// SessionID is the session identifier
SessionID string `json:"session_id,omitempty"`
// UserID is the user identifier
UserID string `json:"user_id,omitempty"`
// TeamID is the team identifier
TeamID string `json:"team_id,omitempty"`
// TenantID is the tenant identifier
TenantID string `json:"tenant_id,omitempty"`
// RememberMe is the remember me flag
RememberMe bool `json:"remember_me,omitempty"`
// Constraints contains data access constraints (set by ACL enforcement)
Constraints *DataConstraintsConfig `json:"constraints,omitempty"`
}
// DataConstraintsConfig represents data access constraints
// Matches the structure of types.DataConstraints from openapi/oauth/types
type DataConstraintsConfig struct {
// OwnerOnly - only access owner's data
OwnerOnly bool `json:"owner_only,omitempty"`
// CreatorOnly - only access creator's data
CreatorOnly bool `json:"creator_only,omitempty"`
// EditorOnly - only access editor's data
EditorOnly bool `json:"editor_only,omitempty"`
// TeamOnly - only access team's data (filter by team_id)
TeamOnly bool `json:"team_only,omitempty"`
// Extra contains user-defined constraints (department, region, etc.)
Extra map[string]interface{} `json:"extra,omitempty"`
}
// ClientConfig represents custom client configuration
type ClientConfig struct {
// Type is the client type (e.g., "web", "mobile", "test")
Type string `json:"type,omitempty"`
// UserAgent is the client user agent string
UserAgent string `json:"user_agent,omitempty"`
// IP is the client IP address
IP string `json:"ip,omitempty"`
} }
// Environment configures the test execution context // Environment configures the test execution context
@ -150,6 +249,9 @@ type Environment struct {
// Accept is the accept format (default: "standard") // Accept is the accept format (default: "standard")
Accept string `json:"accept"` Accept string `json:"accept"`
// ContextConfig contains custom context configuration (from -ctx flag)
ContextConfig *ContextConfig `json:"-"`
} }
// NewEnvironment creates a new test environment with defaults // NewEnvironment creates a new test environment with defaults
@ -175,6 +277,61 @@ func NewEnvironment(userID, teamID string) *Environment {
return env return env
} }
// NewEnvironmentWithContext creates a new test environment with custom context config
func NewEnvironmentWithContext(userID, teamID string, ctxConfig *ContextConfig) *Environment {
env := NewEnvironment(userID, teamID)
if ctxConfig == nil {
return env
}
env.ContextConfig = ctxConfig
// Override with context config values
if ctxConfig.Locale != "" {
env.Locale = ctxConfig.Locale
}
if ctxConfig.Referer != "" {
env.Referer = ctxConfig.Referer
}
if ctxConfig.Client != nil {
if ctxConfig.Client.Type != "" {
env.ClientType = ctxConfig.Client.Type
}
if ctxConfig.Client.IP != "" {
env.ClientIP = ctxConfig.Client.IP
}
}
if ctxConfig.Authorized != nil {
if ctxConfig.Authorized.UserID != "" {
env.UserID = ctxConfig.Authorized.UserID
}
// TeamID takes precedence over TenantID for team override
if ctxConfig.Authorized.TeamID != "" {
env.TeamID = ctxConfig.Authorized.TeamID
} else if ctxConfig.Authorized.TenantID != "" {
env.TeamID = ctxConfig.Authorized.TenantID
}
}
return env
}
// LoadContextConfig loads context configuration from a JSON file
func LoadContextConfig(filePath string) (*ContextConfig, error) {
data, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("failed to read context file: %w", err)
}
var config ContextConfig
if err := json.Unmarshal(data, &config); err != nil {
return nil, fmt.Errorf("failed to parse context file: %w", err)
}
return &config, nil
}
// Case represents a single test case loaded from JSONL // Case represents a single test case loaded from JSONL
type Case struct { type Case struct {
// ID is the unique identifier for this test case (e.g., "T001") // ID is the unique identifier for this test case (e.g., "T001")

177
cmd/README.md
View file

@ -14,17 +14,17 @@ go install github.com/yaoapp/yao@latest
## Global Flags ## Global Flags
| Flag | Short | Description | | Flag | Short | Description |
|------|-------|-------------| | -------- | ----- | ------------------------------- |
| `--app` | `-a` | Application directory path | | `--app` | `-a` | Application directory path |
| `--file` | `-f` | Application package file (.yaz) | | `--file` | `-f` | Application package file (.yaz) |
| `--key` | `-k` | Application license key | | `--key` | `-k` | Application license key |
## Environment Variables ## Environment Variables
| Variable | Description | | Variable | Description |
|----------|-------------| | ---------- | -------------------------------------------- |
| `YAO_ROOT` | Application root directory | | `YAO_ROOT` | Application root directory |
| `YAO_LANG` | Language setting (e.g., `zh-CN` for Chinese) | | `YAO_LANG` | Language setting (e.g., `zh-CN` for Chinese) |
## Commands ## Commands
@ -46,10 +46,10 @@ yao start --debug
**Flags:** **Flags:**
| Flag | Description | | Flag | Description |
|------|-------------| | -------------------- | ----------------------------- |
| `--debug` | Enable development/debug mode | | `--debug` | Enable development/debug mode |
| `--disable-watching` | Disable file watching | | `--disable-watching` | Disable file watching |
--- ---
@ -70,9 +70,9 @@ yao run -s models.user.Find 1
**Flags:** **Flags:**
| Flag | Short | Description | | Flag | Short | Description |
|------|-------|-------------| | ---------- | ----- | ---------------------------------------- |
| `--silent` | `-s` | Silent mode - output result as JSON only | | `--silent` | `-s` | Silent mode - output result as JSON only |
**Argument Syntax:** **Argument Syntax:**
@ -102,11 +102,11 @@ yao migrate --reset
**Flags:** **Flags:**
| Flag | Short | Description | | Flag | Short | Description |
|------|-------|-------------| | --------- | ----- | -------------------------------- |
| `--name` | `-n` | Specific model name to migrate | | `--name` | `-n` | Specific model name to migrate |
| `--force` | | Force migrate in production mode | | `--force` | | Force migrate in production mode |
| `--reset` | | Drop tables before migration | | `--reset` | | Drop tables before migration |
--- ---
@ -134,8 +134,8 @@ yao version --all
**Flags:** **Flags:**
| Flag | Description | | Flag | Description |
|------|-------------| | ------- | -------------------------------------------------------------------- |
| `--all` | Print all version information (Go version, commit, build time, etc.) | | `--all` | Print all version information (Go version, commit, build time, etc.) |
--- ---
@ -146,7 +146,7 @@ Commands for testing and managing AI agents.
### `yao agent test` ### `yao agent test`
Test an agent with input cases from a JSONL file or direct message. Test an agent with input cases from a JSONL file, direct message, or script tests.
```bash ```bash
# Test with direct message (development mode) # Test with direct message (development mode)
@ -169,45 +169,104 @@ yao agent test -i tests/inputs.jsonl --parallel 4
# Verbose output # Verbose output
yao agent test -i tests/inputs.jsonl -v yao agent test -i tests/inputs.jsonl -v
# Script tests (test agent handler scripts)
yao agent test -i scripts.expense.setup -v
# Script tests with test filtering
yao agent test -i scripts.expense.setup --run "TestSystemReady"
# Script tests with custom context
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
``` ```
**Flags:** **Flags:**
| Flag | Short | Description | | Flag | Short | Description |
|------|-------|-------------| | ------------- | ----- | ---------------------------------------------------------------- |
| `--input` | `-i` | Input: JSONL file path or direct message (required) | | `--input` | `-i` | Input: JSONL file path, message, or script ID (required) |
| `--output` | `-o` | Output file path (default: `output-{timestamp}.jsonl`) | | `--output` | `-o` | Output file path (default: `output-{timestamp}.jsonl`) |
| `--name` | `-n` | Agent ID (default: auto-detect from path) | | `--name` | `-n` | Agent ID (default: auto-detect from path) |
| `--connector` | `-c` | Override default connector | | `--connector` | `-c` | Override default connector |
| `--user` | `-u` | Test user ID (default: `test-user`) | | `--user` | `-u` | Test user ID (default: `test-user`) |
| `--team` | `-t` | Test team ID (default: `test-team`) | | `--team` | `-t` | Test team ID (default: `test-team`) |
| `--reporter` | `-r` | Reporter agent ID for custom report generation | | `--ctx` | | Path to context JSON file for custom authorization |
| `--runs` | | Number of runs per test case for stability analysis (default: 1) | | `--reporter` | `-r` | Reporter agent ID for custom report generation |
| `--timeout` | | Timeout per test case (default: `5m`) | | `--runs` | | Number of runs per test case for stability analysis (default: 1) |
| `--parallel` | | Number of parallel test cases (default: 1) | | `--run` | | Regex pattern to filter which tests to run |
| `--verbose` | `-v` | Enable verbose output | | `--timeout` | | Timeout per test case (default: `5m`) |
| `--fail-fast` | | Stop on first failure | | `--parallel` | | Number of parallel test cases (default: 1) |
| `--app` | `-a` | Application directory | | `--verbose` | `-v` | Enable verbose output |
| `--env` | `-e` | Environment file | | `--fail-fast` | | Stop on first failure |
| `--app` | `-a` | Application directory |
| `--env` | `-e` | Environment file |
**Input Modes:** **Input Modes:**
1. **Direct Message Mode**: For quick development/debugging 1. **Direct Message Mode**: For quick development/debugging
```bash ```bash
yao agent test -i "Hello world" -n my.agent yao agent test -i "Hello world" -n my.agent
``` ```
- Outputs result directly to stdout - Outputs result directly to stdout
- No report file generated - No report file generated
- Ideal for iterative development - Ideal for iterative development
2. **File Mode**: For comprehensive testing 2. **File Mode**: For comprehensive testing
```bash ```bash
yao agent test -i tests/inputs.jsonl yao agent test -i tests/inputs.jsonl
``` ```
- Reads test cases from JSONL file - Reads test cases from JSONL file
- Generates detailed report - Generates detailed report
- Supports stability analysis - Supports stability analysis
3. **Script Test Mode**: For testing agent handler scripts
```bash
yao agent test -i scripts.expense.setup -v
```
- Tests TypeScript/JavaScript handler scripts (hooks, tools, setup functions)
- Input format: `scripts.<assistant>.<module>` (e.g., `scripts.expense.setup`)
- Automatically discovers and runs all `Test*` functions
- Uses Go-like testing interface with assertions
**Script Test Function Signature:**
```typescript
// assistants/expense/src/setup_test.ts
import { SystemReady } from "./setup";
export function TestSystemReady(t: testing.T, ctx: agent.Context) {
const result = SystemReady(ctx);
t.assert.True(result.success, "SystemReady should succeed");
t.assert.Equal(result.status, "ready", "Status should be ready");
}
```
**Context JSON Format (for `--ctx` flag):**
```json
{
"authorized": {
"sub": "user-12345",
"client_id": "my-app",
"user_id": "admin",
"team_id": "team-001",
"tenant_id": "acme-corp",
"constraints": {
"owner_only": true,
"team_only": false,
"extra": { "department": "engineering" }
}
},
"metadata": { "request_id": "req-123" },
"client": { "type": "web", "ip": "192.168.1.100" },
"locale": "zh-cn"
}
```
**JSONL Input Format:** **JSONL Input Format:**
```jsonl ```jsonl
@ -221,12 +280,12 @@ yao agent test -i tests/inputs.jsonl -v
**Output Formats:** **Output Formats:**
| Extension | Format | Description | | Extension | Format | Description |
|-----------|--------|-------------| | --------- | -------- | -------------------------- |
| `.jsonl` | JSONL | Streaming format (default) | | `.jsonl` | JSONL | Streaming format (default) |
| `.json` | JSON | Complete structured report | | `.json` | JSON | Complete structured report |
| `.md` | Markdown | Human-readable with tables | | `.md` | Markdown | Human-readable with tables |
| `.html` | HTML | Interactive web report | | `.html` | HTML | Interactive web report |
**Agent Resolution:** **Agent Resolution:**
@ -281,11 +340,11 @@ yao sui trans default index -l "en-US,zh-CN,ja-JP"
**SUI Flags:** **SUI Flags:**
| Flag | Short | Description | | Flag | Short | Description |
|------|-------|-------------| | ----------- | ----- | ----------------------------------------- |
| `--data` | `-d` | Session data as JSON (prefix with `::`) | | `--data` | `-d` | Session data as JSON (prefix with `::`) |
| `--debug` | `-D` | Enable debug mode | | `--debug` | `-D` | Enable debug mode |
| `--locales` | `-l` | Locales for translation (comma-separated) | | `--locales` | `-l` | Locales for translation (comma-separated) |
--- ---
@ -313,6 +372,15 @@ yao sui watch default home
# Run comprehensive agent tests # Run comprehensive agent tests
yao agent test -i tests/inputs.jsonl -o report.html -v yao agent test -i tests/inputs.jsonl -o report.html -v
# Run script tests for agent handlers
yao agent test -i scripts.expense.setup -v
# Run specific script tests with filtering
yao agent test -i scripts.expense.setup --run "TestSystem.*" -v
# Run script tests with custom context
yao agent test -i scripts.expense.setup --ctx tests/context.json -v
# Stability analysis (run each test 10 times) # Stability analysis (run each test 10 times)
yao agent test -i tests/inputs.jsonl --runs 10 -o stability-report.json yao agent test -i tests/inputs.jsonl --runs 10 -o stability-report.json
@ -337,10 +405,10 @@ yao migrate -n user --reset --force
## Exit Codes ## Exit Codes
| Code | Description | | Code | Description |
|------|-------------| | ---- | --------------------- |
| 0 | Success | | 0 | Success |
| 1 | Error or test failure | | 1 | Error or test failure |
--- ---
@ -370,4 +438,3 @@ myapp/
- [Yao Documentation](https://yaoapps.com/docs) - [Yao Documentation](https://yaoapps.com/docs)
- [Agent Test Design](../agent/test/DESIGN.md) - [Agent Test Design](../agent/test/DESIGN.md)
- [SUI Documentation](https://yaoapps.com/docs/sui) - [SUI Documentation](https://yaoapps.com/docs/sui)

2
cmd/agent/agent.go
View file

@ -22,8 +22,10 @@ var langs = map[string]string{
"Override connector": "覆盖连接器", "Override connector": "覆盖连接器",
"Test user ID (default: test-user)": "测试用户 ID (默认: test-user)", "Test user ID (default: test-user)": "测试用户 ID (默认: test-user)",
"Test team ID (default: test-team)": "测试团队 ID (默认: test-team)", "Test team ID (default: test-team)": "测试团队 ID (默认: test-team)",
"Path to context JSON file for custom authorization": "自定义认证信息的 JSON 文件路径",
"Reporter agent ID for custom report": "自定义报告生成器智能体 ID", "Reporter agent ID for custom report": "自定义报告生成器智能体 ID",
"Number of runs for stability analysis": "稳定性分析的运行次数", "Number of runs for stability analysis": "稳定性分析的运行次数",
"Regex pattern to filter which tests to run": "用于过滤测试的正则表达式",
"Default timeout per test case": "每个测试用例的默认超时时间", "Default timeout per test case": "每个测试用例的默认超时时间",
"Number of parallel test cases": "并行测试用例数", "Number of parallel test cases": "并行测试用例数",
"Verbose output": "详细输出", "Verbose output": "详细输出",

32
cmd/agent/test.go
View file

@ -25,8 +25,10 @@ var (
testConnector string testConnector string
testUser string testUser string
testTeam string testTeam string
testContext string // --ctx flag for custom context JSON file
testReporter string testReporter string
testRuns int testRuns int
testRun string // --run flag for test filtering (regex pattern)
testTimeout string testTimeout string
testParallel int testParallel int
testVerbose bool testVerbose bool
@ -140,19 +142,21 @@ var TestCmd = &cobra.Command{
// Build test options // Build test options
opts := &test.Options{ opts := &test.Options{
Input: testInput, Input: testInput,
InputMode: inputMode, InputMode: inputMode,
OutputFile: testOutput, OutputFile: testOutput,
AgentID: testAgent, AgentID: testAgent,
Connector: testConnector, Connector: testConnector,
UserID: testUser, UserID: testUser,
TeamID: testTeam, TeamID: testTeam,
ReporterID: testReporter, ContextFile: testContext,
Runs: testRuns, ReporterID: testReporter,
Timeout: timeout, Runs: testRuns,
Parallel: testParallel, Run: testRun,
Verbose: testVerbose, Timeout: timeout,
FailFast: testFailFast, Parallel: testParallel,
Verbose: testVerbose,
FailFast: testFailFast,
} }
// Merge with defaults // Merge with defaults
@ -232,8 +236,10 @@ func init() {
TestCmd.Flags().StringVarP(&testConnector, "connector", "c", "", L("Override connector")) TestCmd.Flags().StringVarP(&testConnector, "connector", "c", "", L("Override connector"))
TestCmd.Flags().StringVarP(&testUser, "user", "u", "", L("Test user ID (default: test-user)")) TestCmd.Flags().StringVarP(&testUser, "user", "u", "", L("Test user ID (default: test-user)"))
TestCmd.Flags().StringVarP(&testTeam, "team", "t", "", L("Test team ID (default: test-team)")) TestCmd.Flags().StringVarP(&testTeam, "team", "t", "", L("Test team ID (default: test-team)"))
TestCmd.Flags().StringVar(&testContext, "ctx", "", L("Path to context JSON file for custom authorization"))
TestCmd.Flags().StringVarP(&testReporter, "reporter", "r", "", L("Reporter agent ID for custom report")) TestCmd.Flags().StringVarP(&testReporter, "reporter", "r", "", L("Reporter agent ID for custom report"))
TestCmd.Flags().IntVar(&testRuns, "runs", 1, L("Number of runs for stability analysis")) TestCmd.Flags().IntVar(&testRuns, "runs", 1, L("Number of runs for stability analysis"))
TestCmd.Flags().StringVar(&testRun, "run", "", L("Regex pattern to filter which tests to run"))
TestCmd.Flags().StringVar(&testTimeout, "timeout", "5m", L("Default timeout per test case")) TestCmd.Flags().StringVar(&testTimeout, "timeout", "5m", L("Default timeout per test case"))
TestCmd.Flags().IntVar(&testParallel, "parallel", 1, L("Number of parallel test cases")) TestCmd.Flags().IntVar(&testParallel, "parallel", 1, L("Number of parallel test cases"))
TestCmd.Flags().BoolVarP(&testVerbose, "verbose", "v", false, L("Verbose output")) TestCmd.Flags().BoolVarP(&testVerbose, "verbose", "v", false, L("Verbose output"))