opencode/yao
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Enhance DESIGN_V2.md and TODO_V2.md for Message History Support

Browse source 
- Updated DESIGN_V2.md to introduce message history support in the Agent Test Framework, allowing tests to simulate multi-turn conversations without complex state management.
- Revised the test case format to include a `messages` field, enabling the passing of full conversation history directly to the agent.
- Enhanced TODO_V2.md to outline tasks for implementing message history support, including updates to the test case parser and output formats.
- Improved documentation on agent-driven assertions and error handling to reflect the new capabilities and ensure clarity for developers.
This commit is contained in:
Max 2025-12-25 18:10:52 +08:00
parent d58aa12d4d
commit d03668e876
2 changed files with 503 additions and 1193 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1619
agent/test/DESIGN_V2.md
View file

File diff suppressed because it is too large Load diff

77
agent/test/TODO_V2.md
View file

@ -2,32 +2,30 @@
## Format Rules Summary ## Format Rules Summary
| Context | Format | Example | | Context | Format | Example |
|---------|--------|---------| | --------------------- | ------------------------ | ------------------------------------------------------- |
| `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` | | `-i` flag (CLI) | Prefix required | `agents:workers.test.gen`, `scripts:tests.gen` |
| JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` | | JSONL assertion `use` | Prefix required | `"use": "agents:workers.test.validator"` |
| JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-sim"` | | JSONL `simulator.use` | No prefix (agent only) | `"use": "workers.test.user-simulator"` |
| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-sim` | | `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` |
| `t.assert.Agent()` | No prefix (method is explicit) | `t.assert.Agent(resp, "workers.test.validator", {...})` | | `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` |
## Phase 1: Static Mode ## Phase 1: Message History Support
- [ ] Add `mode` field to test case parser (`static` | `dynamic`) - [ ] Add `messages` field to test case parser
- [ ] Extend test case parser for `turns` array - [ ] Support both `input` (string) and `messages` (array) fields
- [ ] Convert `input` to `messages` format internally
- [ ] Pass messages directly to `Agent.Stream()`
- [ ] Add `options` field support (aligned with `context.Options`) - [ ] Add `options` field support (aligned with `context.Options`)
- [ ] Support test-level `options` and per-turn `options` override - [ ] Support attachments in message content parts
- [ ] Implement turn-by-turn execution with options passing - [ ] Update console output to show message count
- [ ] Implement conversation context management - [ ] Update JSONL output format
- [ ] Add per-turn assertions
- [ ] Support attachments at turn level
- [ ] Update console output for multi-turn display
- [ ] Update JSONL output format for turns
## Phase 2: Agent-Driven Input ## Phase 2: Agent-Driven Input
- [ ] Parse `agents:` prefix in `-i` flag - [ ] Parse `agents:` prefix in `-i` flag
- [ ] Parse `scripts:` prefix in `-i` flag - [ ] Parse `scripts:` prefix in `-i` flag
- [ ] Use standard `context.Options` for all agent invocations - [ ] Use standard `context.Options` for generator invocation
- [ ] Pass `test_mode: "generator"` in `options.metadata` - [ ] Pass `test_mode: "generator"` in `options.metadata`
- [ ] Pass target agent info (description, tools) in `options.metadata` - [ ] Pass target agent info (description, tools) in `options.metadata`
- [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata` - [ ] Support query parameters (`?count=10&focus=...`) → merged into `options.metadata`
@ -37,12 +35,13 @@
## Phase 3: Dynamic Mode (Checkpoints) ## Phase 3: Dynamic Mode (Checkpoints)
- [ ] Add `checkpoints` array to test case parser - [ ] Add `checkpoints` array to test case parser
- [ ] Add `simulator` field to test case parser
- [ ] Implement checkpoint matching against agent responses - [ ] Implement checkpoint matching against agent responses
- [ ] Support `after` field for order constraints - [ ] Support `after` field for order constraints
- [ ] Track pending/reached checkpoints during execution - [ ] Track pending/reached checkpoints during execution
- [ ] Implement termination conditions: - [ ] Implement termination conditions:
- [ ] All checkpoints reached → PASSED - [ ] All checkpoints reached → PASSED
- [ ] Agent completed, missing checkpoints → FAILED - [ ] Simulator signals goal_achieved but checkpoints missing → FAILED
- [ ] max_turns exceeded → FAILED - [ ] max_turns exceeded → FAILED
- [ ] timeout exceeded → FAILED - [ ] timeout exceeded → FAILED
- [ ] Implement simulator invocation via `Assistant.Stream()` - [ ] Implement simulator invocation via `Assistant.Stream()`
@ -52,24 +51,10 @@
- [ ] Pass conversation history as messages - [ ] Pass conversation history as messages
- [ ] Create example simulator agent with prompt template - [ ] Create example simulator agent with prompt template
## Phase 4: Interactive Mode ## Phase 4: Agent-Driven Assertions
- [ ] Add `--interactive` flag
- [ ] Implement terminal input prompt with context display
- [ ] Add input timeout handling
- [ ] Support input history/editing
- [ ] Add `--skip-interactive` for CI/CD mode
## Phase 5: Enhanced Detection
- [ ] Add `awaiting_input` field to agent response schema
- [ ] Implement tool-based detection (confirmation tools)
- [ ] Add configurable detection rules
- [ ] Support custom detection via script/agent
## Phase 6: Agent-Driven Assertions
### In JSONL Test Cases ### In JSONL Test Cases
- [ ] Add `agent` assertion type to assertion parser - [ ] Add `agent` assertion type to assertion parser
- [ ] Support `options` field in assertion (aligned with `context.Options`) - [ ] Support `options` field in assertion (aligned with `context.Options`)
- [ ] Implement validator agent invocation via `Assistant.Stream()` - [ ] Implement validator agent invocation via `Assistant.Stream()`
@ -79,6 +64,7 @@
- [ ] Add `suggestions` to assertion error output - [ ] Add `suggestions` to assertion error output
### In Script Tests ### In Script Tests
- [ ] Add `t.assert.Agent(response, agentID, options?)` method - [ ] Add `t.assert.Agent(response, agentID, options?)` method
- [ ] `agentID` is direct ID (e.g., `workers.test.validator`), no prefix needed - [ ] `agentID` is direct ID (e.g., `workers.test.validator`), no prefix needed
- [ ] Invoke validator agent with context - [ ] Invoke validator agent with context
@ -86,25 +72,22 @@
- [ ] Support passing conversation history in options - [ ] Support passing conversation history in options
### Shared ### Shared
- [ ] Create example validator agent with prompt template - [ ] Create example validator agent with prompt template
- [ ] Document `ValidatorResult` interface - [ ] Document `ValidatorResult` interface
## Phase 7: Error Handling & Reporting ## Phase 5: Error Handling & Reporting
- [ ] Implement turn-level error handling - [ ] Implement test-level error handling
- [ ] Implement test-level error aggregation
- [ ] Add detailed error messages with hints - [ ] Add detailed error messages with hints
- [ ] Support custom reporter agent - [ ] Support `--parallel` flag for concurrent test execution
- [ ] Support `--fail-fast` flag to stop on first failure
- [ ] Add verbose mode (`-v`) for detailed output
## Open Questions ## Open Questions
1. **Session Management**: How to handle session state across turns? Use existing session or create new per-test? 1. **Message Generation**: Should we provide a helper to generate message history from a script?
2. **Timeout Strategy**: Per-turn timeout vs. total test timeout? 2. **Snapshot Testing**: Should we support "golden file" comparison for responses?
3. **Parallel Execution**: Can multi-turn tests run in parallel, or must they be sequential?
4. **Retry Logic**: If a turn fails, retry just that turn or restart entire conversation?
5. **Snapshot Testing**: Should we support "golden file" comparison for conversation flows?
3. **Retry Logic**: If a test fails, should we support automatic retry?