Enhance README.md with Detailed Input Format Examples for Agent Testing
- Expanded the input format section to include comprehensive examples organized by scenario, covering various testing cases such as simple text input, assertions, conversation history, and dynamic mode. - Added scenarios demonstrating the use of file attachments, agent-driven assertions, and options for test configuration. - Improved clarity on the structure and requirements for JSONL input, ensuring users have clear guidance on how to format their test cases effectively. - Removed outdated TODO_V2.md file to streamline documentation and focus on the updated README content.
This commit is contained in:
Max
parent
e9b3d46a6f
commit
ea10a71050
2 changed files with 444 additions and 128 deletions
|
|
@ -205,14 +205,265 @@ Simulator-driven testing with checkpoint validation. A simulator agent generates
|
|||
|
||||
## Input Format (JSONL)
|
||||
|
||||
Each line is a JSON object:
|
||||
Each line is a JSON object. Below are examples organized by scenario.
|
||||
|
||||
### Scenario 1: Simple Text Input
|
||||
|
||||
Basic test with string input:
|
||||
|
||||
```jsonl
|
||||
{"id": "T001", "input": "Simple text"}
|
||||
{"id": "T002", "input": {"role": "user", "content": "Message with role"}}
|
||||
{"id": "T003", "input": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello"}, {"role": "user", "content": "Follow-up"}]}
|
||||
{"id": "T004", "input": "Test", "assert": {"type": "json_path", "path": "field", "value": true}}
|
||||
{"id": "T005", "input": "Skip this", "skip": true}
|
||||
{"id": "greeting-basic", "input": "Hello, how are you?"}
|
||||
{"id": "greeting-chinese", "input": "你好，请问有什么可以帮助你的？"}
|
||||
```
|
||||
|
||||
### Scenario 2: With Assertions
|
||||
|
||||
Validate response content:
|
||||
|
||||
```jsonl
|
||||
{"id": "keyword-extract", "input": "Extract keywords from: AI and machine learning", "assert": {"type": "contains", "value": "AI"}}
|
||||
{"id": "json-response", "input": "What's the weather?", "assert": {"type": "json_path", "path": "need_search", "value": true}}
|
||||
{"id": "no-error", "input": "Help me", "assert": {"type": "not_contains", "value": "error"}}
|
||||
```
|
||||
|
||||
### Scenario 3: Multiple Assertions
|
||||
|
||||
All assertions must pass:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "expense-submit",
|
||||
"input": "Submit $500 travel expense",
|
||||
"assert": [
|
||||
{
|
||||
"type": "contains",
|
||||
"value": "expense"
|
||||
},
|
||||
{
|
||||
"type": "not_contains",
|
||||
"value": "error"
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"value": "(?i)(submitted|created|confirmed)"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 4: Conversation History
|
||||
|
||||
Test with multi-turn context:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "expense-confirm",
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Submit an expense"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "What type of expense?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Travel, $500"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Please confirm: $500 travel expense"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Yes, confirm"
|
||||
}
|
||||
],
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)(submitted|created)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 5: With File Attachments
|
||||
|
||||
Test with images or documents:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "receipt-analyze",
|
||||
"input": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Analyze this receipt"
|
||||
},
|
||||
{
|
||||
"type": "image",
|
||||
"source": "file://fixtures/receipt.jpg"
|
||||
}
|
||||
]
|
||||
},
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "amount"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 6: Agent-Driven Assertion
|
||||
|
||||
Use LLM to validate response semantics:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "helpful-response",
|
||||
"input": "How do I reset my password?",
|
||||
"assert": {
|
||||
"type": "agent",
|
||||
"use": "agents:tests.validator-agent",
|
||||
"value": "Response should provide clear step-by-step instructions"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 7: With Options
|
||||
|
||||
Override connector or skip features:
|
||||
|
||||
```jsonl
|
||||
{"id": "fast-model", "input": "Quick question", "options": {"connector": "deepseek.v3", "skip": {"history": true, "trace": true}}}
|
||||
{"id": "scenario-test", "input": "Query users", "options": {"metadata": {"scenario": "filter"}}, "assert": {"type": "json_path", "path": "from", "value": "users"}}
|
||||
```
|
||||
|
||||
### Scenario 8: With Before/After Hooks
|
||||
|
||||
Setup and teardown for each test:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "with-user-data",
|
||||
"input": "Show my expenses",
|
||||
"before": "env_test.Before",
|
||||
"after": "env_test.After",
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "expense"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 9: Skip Test
|
||||
|
||||
Temporarily disable a test:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "wip-feature",
|
||||
"input": "New feature test",
|
||||
"skip": true
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 10: Dynamic Mode (Simulator)
|
||||
|
||||
Multi-turn testing with user simulator:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "coffee-order",
|
||||
"input": "I want to order coffee",
|
||||
"simulator": {
|
||||
"use": "tests.simulator-agent",
|
||||
"options": {
|
||||
"metadata": {
|
||||
"persona": "Regular customer",
|
||||
"goal": "Order a medium latte"
|
||||
}
|
||||
}
|
||||
},
|
||||
"checkpoints": [
|
||||
{
|
||||
"id": "greeting",
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)(hello|hi|help)"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ask-size",
|
||||
"after": [
|
||||
"greeting"
|
||||
],
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)size"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "confirm",
|
||||
"after": [
|
||||
"ask-size"
|
||||
],
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)confirm"
|
||||
}
|
||||
}
|
||||
],
|
||||
"max_turns": 10
|
||||
}
|
||||
```
|
||||
|
||||
### Scenario 11: Dynamic Mode with Optional Checkpoint
|
||||
|
||||
Some checkpoints are optional:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "expense-flow",
|
||||
"input": "Submit expense",
|
||||
"simulator": {
|
||||
"use": "tests.simulator-agent",
|
||||
"options": {
|
||||
"metadata": {
|
||||
"persona": "New employee",
|
||||
"goal": "Submit $500 travel expense"
|
||||
}
|
||||
}
|
||||
},
|
||||
"checkpoints": [
|
||||
{
|
||||
"id": "ask-type",
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)type"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "suggest-category",
|
||||
"required": false,
|
||||
"assert": {
|
||||
"type": "contains",
|
||||
"value": "category"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "confirm",
|
||||
"after": [
|
||||
"ask-type"
|
||||
],
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)confirm"
|
||||
}
|
||||
}
|
||||
],
|
||||
"max_turns": 15
|
||||
}
|
||||
```
|
||||
|
||||
### Standard Mode Fields
|
||||
|
|
@ -640,6 +891,193 @@ yao agent test -i tests/inputs.jsonl --parallel 4
|
|||
- `scripts.xxx` (dot) - Run script tests (`*_test.ts` functions)
|
||||
- `scripts:xxx` (colon) - Generate test cases from a script
|
||||
|
||||
## Built-in Test Agents
|
||||
|
||||
The framework provides three specialized agents for testing:
|
||||
|
||||
### Generator Agent (`tests.generator-agent`)
|
||||
|
||||
Generates test cases based on target agent description.
|
||||
|
||||
**package.yao**:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Test Case Generator",
|
||||
"connector": "gpt-4o",
|
||||
"description": "Generates test cases for agent testing",
|
||||
"options": { "temperature": 0.7 },
|
||||
"automated": true
|
||||
}
|
||||
```
|
||||
|
||||
**prompts.yml**:
|
||||
|
||||
```yaml
|
||||
- role: system
|
||||
content: |
|
||||
You are a test case generator. Generate test cases based on the target agent.
|
||||
|
||||
## Input Format
|
||||
- `target_agent`: Agent info (id, description, tools)
|
||||
- `count`: Number of test cases (default: 5)
|
||||
- `focus`: Focus area (e.g., "edge-cases", "happy-path")
|
||||
|
||||
## Output Format
|
||||
JSON array of test cases:
|
||||
[
|
||||
{
|
||||
"id": "test-id",
|
||||
"input": "User message",
|
||||
"assert": [{"type": "contains", "value": "expected"}]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Usage**:
|
||||
|
||||
```bash
|
||||
yao agent test -i "agents:tests.generator-agent?count=10" -n assistants.expense
|
||||
```
|
||||
|
||||
### Validator Agent (`tests.validator-agent`)
|
||||
|
||||
Validates agent responses for agent-driven assertions.
|
||||
|
||||
**package.yao**:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Response Validator",
|
||||
"connector": "gpt-4o",
|
||||
"description": "Validates responses against criteria",
|
||||
"options": { "temperature": 0 },
|
||||
"automated": true
|
||||
}
|
||||
```
|
||||
|
||||
**prompts.yml**:
|
||||
|
||||
```yaml
|
||||
- role: system
|
||||
content: |
|
||||
You are a response validator. Evaluate whether the response meets the criteria.
|
||||
|
||||
## Input Format
|
||||
- `output`: The response to validate
|
||||
- `criteria`: The validation rules
|
||||
- `input`: Original input (optional)
|
||||
|
||||
## Output Format
|
||||
JSON object (no markdown):
|
||||
{"passed": true/false, "reason": "explanation"}
|
||||
|
||||
## Examples
|
||||
Input: {"output": "Paris is the capital", "criteria": "factually accurate"}
|
||||
Output: {"passed": true, "reason": "Statement is correct"}
|
||||
```
|
||||
|
||||
**Usage in JSONL**:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T001",
|
||||
"input": "Hello",
|
||||
"assert": {
|
||||
"type": "agent",
|
||||
"use": "agents:tests.validator-agent",
|
||||
"value": "Response should be friendly"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Usage in script tests**:
|
||||
|
||||
```typescript
|
||||
t.assert.Agent(response, "tests.validator-agent", {
|
||||
criteria: "Response should be helpful",
|
||||
});
|
||||
```
|
||||
|
||||
### Simulator Agent (`tests.simulator-agent`)
|
||||
|
||||
Simulates user behavior for dynamic mode testing.
|
||||
|
||||
**package.yao**:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "User Simulator",
|
||||
"connector": "gpt-4o",
|
||||
"description": "Simulates user behavior for dynamic testing",
|
||||
"options": { "temperature": 0.7 },
|
||||
"automated": true
|
||||
}
|
||||
```
|
||||
|
||||
**prompts.yml**:
|
||||
|
||||
```yaml
|
||||
- role: system
|
||||
content: |
|
||||
You are a user simulator. Generate realistic user messages based on persona and goal.
|
||||
|
||||
## Input Format
|
||||
- `persona`: User description (e.g., "New employee")
|
||||
- `goal`: What user wants to achieve
|
||||
- `conversation`: Previous messages
|
||||
- `turn_number`: Current turn
|
||||
- `max_turns`: Maximum turns
|
||||
|
||||
## Output Format
|
||||
JSON object:
|
||||
{
|
||||
"message": "User response",
|
||||
"goal_achieved": false,
|
||||
"reasoning": "Strategy explanation"
|
||||
}
|
||||
|
||||
## Guidelines
|
||||
1. Stay in character
|
||||
2. Work toward the goal
|
||||
3. Be realistic (include natural variations)
|
||||
4. Set goal_achieved: true when done
|
||||
```
|
||||
|
||||
**Usage in JSONL**:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "dynamic-test",
|
||||
"input": "I need help",
|
||||
"simulator": {
|
||||
"use": "tests.simulator-agent",
|
||||
"options": {
|
||||
"metadata": {
|
||||
"persona": "New employee",
|
||||
"goal": "Submit expense report"
|
||||
}
|
||||
}
|
||||
},
|
||||
"checkpoints": [
|
||||
{
|
||||
"id": "greeting",
|
||||
"assert": {
|
||||
"type": "regex",
|
||||
"value": "(?i)hello"
|
||||
}
|
||||
}
|
||||
],
|
||||
"max_turns": 10
|
||||
}
|
||||
```
|
||||
|
||||
**Usage via CLI**:
|
||||
|
||||
```bash
|
||||
yao agent test -i tests/dynamic.jsonl --simulator tests.simulator-agent
|
||||
```
|
||||
|
||||
## Exit Codes
|
||||
|
||||
| Code | Description |
|
||||
|
|
|
|||
|
|
@ -1,122 +0,0 @@
|
|||
# Agent Test Framework V2 - TODO
|
||||
|
||||
> 详细实施计划见 [UPGRADE_PLAN.md](./UPGRADE_PLAN.md)
|
||||
|
||||
## Format Rules
|
||||
|
||||
| Context | Format | Example |
|
||||
| --------------------- | ------------------------ | ---------------------------------------------- |
|
||||
| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` |
|
||||
| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` |
|
||||
| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` |
|
||||
| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` |
|
||||
| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.val")` |
|
||||
| JSONL `before/after` | No prefix (in src/) | `"before": "env_test.Before"` |
|
||||
| `--before/--after` | No prefix (in src/) | `--before env_test.BeforeAll` |
|
||||
|
||||
## Phase 1: Before/After Scripts ✅
|
||||
|
||||
**新增文件**: `script_hooks.go`
|
||||
|
||||
- [x] `types.go`: 添加 `Before`, `After` 字段到 `Case`
|
||||
- [x] `types.go`: 添加 `BeforeAll`, `AfterAll` 字段到 `Options`
|
||||
- [x] `script_hooks.go`: 实现 `HookExecutor`
|
||||
- [x] `script_hooks.go`: 通过 V8 直接执行 `*_test.ts` 脚本
|
||||
- [x] `runner.go`: 集成 before/after 到 `runSingleTest`
|
||||
- [x] `runner.go`: 集成 beforeAll/afterAll 到 `RunTests`
|
||||
- [x] `cmd/agent/test.go`: 添加 `--before`, `--after` flags
|
||||
- [x] `test/utils.go`: 添加 `LoadAgentTestScripts()` 通用函数
|
||||
- [x] 创建示例脚本 `assistants/tests/hooks-test/src/env_test.ts`
|
||||
- [x] 创建单元测试 `script_hooks_test.go` (黑盒测试)
|
||||
|
||||
## Phase 2: Agent-Driven Assertions ✅
|
||||
|
||||
**修改文件**: `assert.go`, `script_assert.go`
|
||||
|
||||
- [x] `types.go`: 添加 `Use`, `Options` 字段到 `Assertion`
|
||||
- [x] `assert.go`: 实现 `assertAgent` 方法
|
||||
- [x] `assert.go`: 在 `evaluateAssertion` 添加 `agent` 类型
|
||||
- [x] `assert.go`: 使用 `goutext.ExtractJSON` 容错解析 LLM 响应
|
||||
- [x] `script_assert.go`: 添加 `assertAgentMethod` 到 `newAssertObject`
|
||||
- [x] 创建示例 validator agent (`assistants/tests/validator-agent`)
|
||||
- [x] 创建单元测试 `assert_agent_test.go` (JSONL 断言 + JSAPI 断言)
|
||||
|
||||
## Phase 3: Agent-Driven Input ✅
|
||||
|
||||
**新增文件**: `input_source.go`
|
||||
|
||||
> 用 Agent 生成测试用例，生成后使用标准模式执行。相对简单。
|
||||
|
||||
**准备工作**:
|
||||
|
||||
- [x] 创建 generator agent (`yao-dev-app/assistants/tests/generator-agent`)
|
||||
- [x] 编写 generator agent 的 prompts.yml
|
||||
|
||||
**实现**:
|
||||
|
||||
- [x] `input_source.go`: 实现 `ParseInputSource`
|
||||
- [x] `input_source.go`: 实现 `GenerateTestCases`
|
||||
- [x] `loader.go`: 添加 `LoadFromAgent` 方法
|
||||
- [x] `loader.go`: 添加 `LoadFromScript` 方法
|
||||
- [x] `runner.go`: 在 `RunTests` 支持不同输入源
|
||||
- [x] `cmd/agent/test.go`: 添加 `--dry-run` flag
|
||||
|
||||
**测试**:
|
||||
|
||||
- [x] 创建单元测试 `input_source_test.go`
|
||||
|
||||
## Phase 4: Dynamic Mode (Simulator + Checkpoints) ✅
|
||||
|
||||
**新增文件**: `dynamic_runner.go`, `dynamic_types.go`
|
||||
|
||||
> 运行时使用 Simulator Agent 动态生成对话，需要多轮循环和 checkpoint 匹配。依赖 Phase 3 的 Agent 调用经验。
|
||||
|
||||
**准备工作**:
|
||||
|
||||
- [x] 创建 simulator agent (`yao-dev-app/assistants/tests/simulator-agent`)
|
||||
- [x] 编写 simulator agent 的 prompts.yml (模拟用户行为)
|
||||
|
||||
**实现**:
|
||||
|
||||
- [x] `types.go`: 添加 `Simulator`, `Checkpoints` 字段到 `Case`
|
||||
- [x] `dynamic_types.go`: 定义 `Checkpoint`, `DynamicResult` 等类型
|
||||
- [x] `dynamic_runner.go`: 实现 `DynamicRunner`
|
||||
- [x] `dynamic_runner.go`: 实现 checkpoint 匹配逻辑
|
||||
- [x] `dynamic_runner.go`: 实现终止条件判断
|
||||
- [x] `runner.go`: 在 `runSingleTest` 判断并调用动态模式
|
||||
|
||||
**测试**:
|
||||
|
||||
- [x] 创建单元测试 `dynamic_runner_test.go`
|
||||
|
||||
## Phase 5: Console Output Optimization ✅
|
||||
|
||||
**修改文件**: `output.go`
|
||||
|
||||
- [x] `output.go`: 添加 `DynamicTestStart` 方法
|
||||
- [x] `output.go`: 添加 `DynamicTurn` 方法
|
||||
- [x] `output.go`: 添加 `DynamicCheckpoint` 方法
|
||||
- [x] `output.go`: 添加 `DynamicTestResult` 方法
|
||||
- [x] 动态模式输出效果已验证
|
||||
|
||||
## Already Implemented ✅
|
||||
|
||||
- [x] Message history support (`input` as array)
|
||||
- [x] File attachments (`file://` protocol)
|
||||
- [x] `--parallel` flag
|
||||
- [x] `--fail-fast` flag
|
||||
- [x] `-v` verbose mode
|
||||
- [x] Script testing (`*_test.ts`)
|
||||
- [x] Before/After hooks (Phase 1)
|
||||
- [x] Agent-driven assertions (Phase 2)
|
||||
- [x] Agent-driven input (Phase 3)
|
||||
- [x] `--dry-run` flag
|
||||
- [x] Dynamic mode (Phase 4)
|
||||
- [x] `--simulator` flag
|
||||
- [x] Console output optimization (Phase 5)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Message Generation**: 是否提供 helper 从脚本生成 message history?
|
||||
2. **Snapshot Testing**: 是否支持 "golden file" 对比?
|
||||
3. **Retry Logic**: 测试失败是否支持自动重试?
|
||||
Loading…
Add table
Reference in a new issue