From ea10a7105013b661690db010c76282f596af2020 Mon Sep 17 00:00:00 2001 From: Max Date: Fri, 26 Dec 2025 11:42:45 +0800 Subject: [PATCH] Enhance README.md with Detailed Input Format Examples for Agent Testing - Expanded the input format section to include comprehensive examples organized by scenario, covering various testing cases such as simple text input, assertions, conversation history, and dynamic mode. - Added scenarios demonstrating the use of file attachments, agent-driven assertions, and options for test configuration. - Improved clarity on the structure and requirements for JSONL input, ensuring users have clear guidance on how to format their test cases effectively. - Removed outdated TODO_V2.md file to streamline documentation and focus on the updated README content. --- agent/test/README.md | 450 +++++++++++++++++++++++++++++++++++++++++- agent/test/TODO_V2.md | 122 ------------ 2 files changed, 444 insertions(+), 128 deletions(-) delete mode 100644 agent/test/TODO_V2.md diff --git a/agent/test/README.md b/agent/test/README.md index d8687362..8a530134 100644 --- a/agent/test/README.md +++ b/agent/test/README.md @@ -205,14 +205,265 @@ Simulator-driven testing with checkpoint validation. A simulator agent generates ## Input Format (JSONL) -Each line is a JSON object: +Each line is a JSON object. Below are examples organized by scenario. + +### Scenario 1: Simple Text Input + +Basic test with string input: ```jsonl -{"id": "T001", "input": "Simple text"} -{"id": "T002", "input": {"role": "user", "content": "Message with role"}} -{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]} -{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}} -{"id": "T005", "input": "Skip this", "skip": true} +{"id": "greeting-basic", "input": "Hello, how are you?"} +{"id": "greeting-chinese", "input": "你好，请问有什么可以帮助你的？"} +``` + +### Scenario 2: With Assertions + +Validate response content: + +```jsonl +{"id": "keyword-extract", "input": "Extract keywords from: AI and machine learning", "assert": {"type": "contains", "value": "AI"}} +{"id": "json-response", "input": "What's the weather?", "assert": {"type": "json_path", "path": "need_search", "value": true}} +{"id": "no-error", "input": "Help me", "assert": {"type": "not_contains", "value": "error"}} +``` + +### Scenario 3: Multiple Assertions + +All assertions must pass: + +```jsonl +{ + "id": "expense-submit", + "input": "Submit $500 travel expense", + "assert": [ + { + "type": "contains", + "value": "expense" + }, + { + "type": "not_contains", + "value": "error" + }, + { + "type": "regex", + "value": "(?i)(submitted|created|confirmed)" + } + ] +} +``` + +### Scenario 4: Conversation History + +Test with multi-turn context: + +```jsonl +{ + "id": "expense-confirm", + "input": [ + { + "role": "user", + "content": "Submit an expense" + }, + { + "role": "assistant", + "content": "What type of expense?" + }, + { + "role": "user", + "content": "Travel, $500" + }, + { + "role": "assistant", + "content": "Please confirm: $500 travel expense" + }, + { + "role": "user", + "content": "Yes, confirm" + } + ], + "assert": { + "type": "regex", + "value": "(?i)(submitted|created)" + } +} +``` + +### Scenario 5: With File Attachments + +Test with images or documents: + +```jsonl +{ + "id": "receipt-analyze", + "input": { + "role": "user", + "content": [ + { + "type": "text", + "text": "Analyze this receipt" + }, + { + "type": "image", + "source": "file://fixtures/receipt.jpg" + } + ] + }, + "assert": { + "type": "contains", + "value": "amount" + } +} +``` + +### Scenario 6: Agent-Driven Assertion + +Use LLM to validate response semantics: + +```jsonl +{ + "id": "helpful-response", + "input": "How do I reset my password?", + "assert": { + "type": "agent", + "use": "agents:tests.validator-agent", + "value": "Response should provide clear step-by-step instructions" + } +} +``` + +### Scenario 7: With Options + +Override connector or skip features: + +```jsonl +{"id": "fast-model", "input": "Quick question", "options": {"connector": "deepseek.v3", "skip": {"history": true, "trace": true}}} +{"id": "scenario-test", "input": "Query users", "options": {"metadata": {"scenario": "filter"}}, "assert": {"type": "json_path", "path": "from", "value": "users"}} +``` + +### Scenario 8: With Before/After Hooks + +Setup and teardown for each test: + +```jsonl +{ + "id": "with-user-data", + "input": "Show my expenses", + "before": "env_test.Before", + "after": "env_test.After", + "assert": { + "type": "contains", + "value": "expense" + } +} +``` + +### Scenario 9: Skip Test + +Temporarily disable a test: + +```jsonl +{ + "id": "wip-feature", + "input": "New feature test", + "skip": true +} +``` + +### Scenario 10: Dynamic Mode (Simulator) + +Multi-turn testing with user simulator: + +```jsonl +{ + "id": "coffee-order", + "input": "I want to order coffee", + "simulator": { + "use": "tests.simulator-agent", + "options": { + "metadata": { + "persona": "Regular customer", + "goal": "Order a medium latte" + } + } + }, + "checkpoints": [ + { + "id": "greeting", + "assert": { + "type": "regex", + "value": "(?i)(hello|hi|help)" + } + }, + { + "id": "ask-size", + "after": [ + "greeting" + ], + "assert": { + "type": "regex", + "value": "(?i)size" + } + }, + { + "id": "confirm", + "after": [ + "ask-size" + ], + "assert": { + "type": "regex", + "value": "(?i)confirm" + } + } + ], + "max_turns": 10 +} +``` + +### Scenario 11: Dynamic Mode with Optional Checkpoint + +Some checkpoints are optional: + +```jsonl +{ + "id": "expense-flow", + "input": "Submit expense", + "simulator": { + "use": "tests.simulator-agent", + "options": { + "metadata": { + "persona": "New employee", + "goal": "Submit $500 travel expense" + } + } + }, + "checkpoints": [ + { + "id": "ask-type", + "assert": { + "type": "regex", + "value": "(?i)type" + } + }, + { + "id": "suggest-category", + "required": false, + "assert": { + "type": "contains", + "value": "category" + } + }, + { + "id": "confirm", + "after": [ + "ask-type" + ], + "assert": { + "type": "regex", + "value": "(?i)confirm" + } + } + ], + "max_turns": 15 +} ``` ### Standard Mode Fields @@ -640,6 +891,193 @@ yao agent test -i tests/inputs.jsonl --parallel 4 - `scripts.xxx` (dot) - Run script tests (`*_test.ts` functions) - `scripts:xxx` (colon) - Generate test cases from a script +## Built-in Test Agents + +The framework provides three specialized agents for testing: + +### Generator Agent (`tests.generator-agent`) + +Generates test cases based on target agent description. + +**package.yao**: + +```json +{ + "name": "Test Case Generator", + "connector": "gpt-4o", + "description": "Generates test cases for agent testing", + "options": { "temperature": 0.7 }, + "automated": true +} +``` + +**prompts.yml**: + +```yaml +- role: system + content: | + You are a test case generator. Generate test cases based on the target agent. + + ## Input Format + - `target_agent`: Agent info (id, description, tools) + - `count`: Number of test cases (default: 5) + - `focus`: Focus area (e.g., "edge-cases", "happy-path") + + ## Output Format + JSON array of test cases: + [ + { + "id": "test-id", + "input": "User message", + "assert": [{"type": "contains", "value": "expected"}] + } + ] +``` + +**Usage**: + +```bash +yao agent test -i "agents:tests.generator-agent?count=10" -n assistants.expense +``` + +### Validator Agent (`tests.validator-agent`) + +Validates agent responses for agent-driven assertions. + +**package.yao**: + +```json +{ + "name": "Response Validator", + "connector": "gpt-4o", + "description": "Validates responses against criteria", + "options": { "temperature": 0 }, + "automated": true +} +``` + +**prompts.yml**: + +```yaml +- role: system + content: | + You are a response validator. Evaluate whether the response meets the criteria. + + ## Input Format + - `output`: The response to validate + - `criteria`: The validation rules + - `input`: Original input (optional) + + ## Output Format + JSON object (no markdown): + {"passed": true/false, "reason": "explanation"} + + ## Examples + Input: {"output": "Paris is the capital", "criteria": "factually accurate"} + Output: {"passed": true, "reason": "Statement is correct"} +``` + +**Usage in JSONL**: + +```jsonl +{ + "id": "T001", + "input": "Hello", + "assert": { + "type": "agent", + "use": "agents:tests.validator-agent", + "value": "Response should be friendly" + } +} +``` + +**Usage in script tests**: + +```typescript +t.assert.Agent(response, "tests.validator-agent", { + criteria: "Response should be helpful", +}); +``` + +### Simulator Agent (`tests.simulator-agent`) + +Simulates user behavior for dynamic mode testing. + +**package.yao**: + +```json +{ + "name": "User Simulator", + "connector": "gpt-4o", + "description": "Simulates user behavior for dynamic testing", + "options": { "temperature": 0.7 }, + "automated": true +} +``` + +**prompts.yml**: + +```yaml +- role: system + content: | + You are a user simulator. Generate realistic user messages based on persona and goal. + + ## Input Format + - `persona`: User description (e.g., "New employee") + - `goal`: What user wants to achieve + - `conversation`: Previous messages + - `turn_number`: Current turn + - `max_turns`: Maximum turns + + ## Output Format + JSON object: + { + "message": "User response", + "goal_achieved": false, + "reasoning": "Strategy explanation" + } + + ## Guidelines + 1. Stay in character + 2. Work toward the goal + 3. Be realistic (include natural variations) + 4. Set goal_achieved: true when done +``` + +**Usage in JSONL**: + +```jsonl +{ + "id": "dynamic-test", + "input": "I need help", + "simulator": { + "use": "tests.simulator-agent", + "options": { + "metadata": { + "persona": "New employee", + "goal": "Submit expense report" + } + } + }, + "checkpoints": [ + { + "id": "greeting", + "assert": { + "type": "regex", + "value": "(?i)hello" + } + } + ], + "max_turns": 10 +} +``` + +**Usage via CLI**: + +```bash +yao agent test -i tests/dynamic.jsonl --simulator tests.simulator-agent +``` + ## Exit Codes | Code | Description | diff --git a/agent/test/TODO_V2.md b/agent/test/TODO_V2.md deleted file mode 100644 index 51cd45bd..00000000 --- a/agent/test/TODO_V2.md +++ /dev/null @@ -1,122 +0,0 @@ -# Agent Test Framework V2 - TODO - -> 详细实施计划见 [UPGRADE_PLAN.md](./UPGRADE_PLAN.md) - -## Format Rules - -| Context | Format | Example | -| --------------------- | ------------------------ | ---------------------------------------------- | -| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | -| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | -| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` | -| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` | -| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.val")` | -| JSONL `before/after` | No prefix (in src/) | `"before": "env_test.Before"` | -| `--before/--after` | No prefix (in src/) | `--before env_test.BeforeAll` | - -## Phase 1: Before/After Scripts ✅ - -**新增文件**: `script_hooks.go` - -- [x] `types.go`: 添加 `Before`, `After` 字段到 `Case` -- [x] `types.go`: 添加 `BeforeAll`, `AfterAll` 字段到 `Options` -- [x] `script_hooks.go`: 实现 `HookExecutor` -- [x] `script_hooks.go`: 通过 V8 直接执行 `*_test.ts` 脚本 -- [x] `runner.go`: 集成 before/after 到 `runSingleTest` -- [x] `runner.go`: 集成 beforeAll/afterAll 到 `RunTests` -- [x] `cmd/agent/test.go`: 添加 `--before`, `--after` flags -- [x] `test/utils.go`: 添加 `LoadAgentTestScripts()` 通用函数 -- [x] 创建示例脚本 `assistants/tests/hooks-test/src/env_test.ts` -- [x] 创建单元测试 `script_hooks_test.go` (黑盒测试) - -## Phase 2: Agent-Driven Assertions ✅ - -**修改文件**: `assert.go`, `script_assert.go` - -- [x] `types.go`: 添加 `Use`, `Options` 字段到 `Assertion` -- [x] `assert.go`: 实现 `assertAgent` 方法 -- [x] `assert.go`: 在 `evaluateAssertion` 添加 `agent` 类型 -- [x] `assert.go`: 使用 `goutext.ExtractJSON` 容错解析 LLM 响应 -- [x] `script_assert.go`: 添加 `assertAgentMethod` 到 `newAssertObject` -- [x] 创建示例 validator agent (`assistants/tests/validator-agent`) -- [x] 创建单元测试 `assert_agent_test.go` (JSONL 断言 + JSAPI 断言) - -## Phase 3: Agent-Driven Input ✅ - -**新增文件**: `input_source.go` - -> 用 Agent 生成测试用例，生成后使用标准模式执行。相对简单。 - -**准备工作**: - -- [x] 创建 generator agent (`yao-dev-app/assistants/tests/generator-agent`) -- [x] 编写 generator agent 的 prompts.yml - -**实现**: - -- [x] `input_source.go`: 实现 `ParseInputSource` -- [x] `input_source.go`: 实现 `GenerateTestCases` -- [x] `loader.go`: 添加 `LoadFromAgent` 方法 -- [x] `loader.go`: 添加 `LoadFromScript` 方法 -- [x] `runner.go`: 在 `RunTests` 支持不同输入源 -- [x] `cmd/agent/test.go`: 添加 `--dry-run` flag - -**测试**: - -- [x] 创建单元测试 `input_source_test.go` - -## Phase 4: Dynamic Mode (Simulator + Checkpoints) ✅ - -**新增文件**: `dynamic_runner.go`, `dynamic_types.go` - -> 运行时使用 Simulator Agent 动态生成对话，需要多轮循环和 checkpoint 匹配。依赖 Phase 3 的 Agent 调用经验。 - -**准备工作**: - -- [x] 创建 simulator agent (`yao-dev-app/assistants/tests/simulator-agent`) -- [x] 编写 simulator agent 的 prompts.yml (模拟用户行为) - -**实现**: - -- [x] `types.go`: 添加 `Simulator`, `Checkpoints` 字段到 `Case` -- [x] `dynamic_types.go`: 定义 `Checkpoint`, `DynamicResult` 等类型 -- [x] `dynamic_runner.go`: 实现 `DynamicRunner` -- [x] `dynamic_runner.go`: 实现 checkpoint 匹配逻辑 -- [x] `dynamic_runner.go`: 实现终止条件判断 -- [x] `runner.go`: 在 `runSingleTest` 判断并调用动态模式 - -**测试**: - -- [x] 创建单元测试 `dynamic_runner_test.go` - -## Phase 5: Console Output Optimization ✅ - -**修改文件**: `output.go` - -- [x] `output.go`: 添加 `DynamicTestStart` 方法 -- [x] `output.go`: 添加 `DynamicTurn` 方法 -- [x] `output.go`: 添加 `DynamicCheckpoint` 方法 -- [x] `output.go`: 添加 `DynamicTestResult` 方法 -- [x] 动态模式输出效果已验证 - -## Already Implemented ✅ - -- [x] Message history support (`input` as array) -- [x] File attachments (`file://` protocol) -- [x] `--parallel` flag -- [x] `--fail-fast` flag -- [x] `-v` verbose mode -- [x] Script testing (`*_test.ts`) -- [x] Before/After hooks (Phase 1) -- [x] Agent-driven assertions (Phase 2) -- [x] Agent-driven input (Phase 3) -- [x] `--dry-run` flag -- [x] Dynamic mode (Phase 4) -- [x] `--simulator` flag -- [x] Console output optimization (Phase 5) - -## Open Questions - -1. **Message Generation**: 是否提供 helper 从脚本生成 message history? -2. **Snapshot Testing**: 是否支持 "golden file" 对比? -3. **Retry Logic**: 测试失败是否支持自动重试?