Update DESIGN_V2.md and TODO_V2.md for Input Field Enhancements
- Revised DESIGN_V2.md to clarify the `input` field's capabilities, allowing for string, single message, or message array formats for conversation context. - Updated the test case format to reflect the new `input` structure, ensuring compatibility with existing single-turn tests. - Enhanced TODO_V2.md to indicate the completion of message history support and outline remaining tasks, including options field support and JSONL output format updates. - Improved documentation to ensure clarity on the new input handling and its implications for agent-driven testing.
This commit is contained in:
Max
parent
d03668e876
commit
c34cae62e8
2 changed files with 53 additions and 48 deletions
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
This document describes the design for Agent Test Framework V2, which extends the existing testing capabilities with:
|
||||
|
||||
- **Message history support** - Test agents with conversation context via `messages[]`
|
||||
- **Message history support** - Test agents with conversation context via `input` array (already implemented)
|
||||
- **Agent-driven testing** - Use agents to generate test cases and validate responses
|
||||
- **Dynamic testing** - Simulator-driven testing with checkpoint validation
|
||||
|
||||
|
|
@ -45,7 +45,7 @@ This document describes the design for Agent Test Framework V2, which extends th
|
|||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Test Case Parser │ │
|
||||
│ │ │ │
|
||||
│ │ Standard Mode: {input: "...", messages: [...], assertions} │ │
|
||||
│ │ Standard Mode: {input: "..." | [...], assertions} │ │
|
||||
│ │ Dynamic Mode: {simulator: {...}, checkpoints: [...]} │ │
|
||||
│ └───────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
|
|
@ -77,12 +77,11 @@ This document describes the design for Agent Test Framework V2, which extends th
|
|||
|
||||
Single call to agent with optional message history. **No multi-turn state management needed.**
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------ | ------ | ------------------------------------------------ |
|
||||
| `input` | string | Simple text input (shorthand for single message) |
|
||||
| `messages` | array | Full message history (overrides `input`) |
|
||||
| `assertions` | array | Assertions to validate response |
|
||||
| `options` | object | `context.Options` passed to agent |
|
||||
| Field | Type | Description |
|
||||
| ------------ | ------------------------------ | --------------------------------------------- |
|
||||
| `input` | string \| Message \| Message[] | Text, single message, or conversation history |
|
||||
| `assertions` | array | Assertions to validate response |
|
||||
| `options` | object | `context.Options` passed to agent |
|
||||
|
||||
### Dynamic Mode
|
||||
|
||||
|
|
@ -112,15 +111,15 @@ Simulator-driven testing with checkpoint validation.
|
|||
}
|
||||
```
|
||||
|
||||
### With Message History (New)
|
||||
### With Message History (Existing)
|
||||
|
||||
Test agent with conversation context - simulates multi-turn without complex state:
|
||||
The `input` field already supports message arrays for conversation context:
|
||||
|
||||
```jsonl
|
||||
{
|
||||
"id": "T002",
|
||||
"name": "Expense submission - final confirmation",
|
||||
"messages": [
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "I want to submit an expense"
|
||||
|
|
@ -170,16 +169,14 @@ To test agent behavior at different conversation stages, create separate test ca
|
|||
// Test 1: First turn - agent should ask for expense type
|
||||
{
|
||||
"id": "expense-turn1",
|
||||
"messages": [
|
||||
{"role": "user", "content": "I want to submit an expense"}
|
||||
],
|
||||
"input": [{"role": "user", "content": "I want to submit an expense"}],
|
||||
"assertions": [{"type": "contains", "value": "type"}]
|
||||
}
|
||||
|
||||
// Test 2: Second turn - agent should create expense
|
||||
{
|
||||
"id": "expense-turn2",
|
||||
"messages": [
|
||||
"input": [
|
||||
{"role": "user", "content": "I want to submit an expense"},
|
||||
{"role": "assistant", "content": "What type of expense would you like to submit?"},
|
||||
{"role": "user", "content": "Business travel, $3500"}
|
||||
|
|
@ -190,7 +187,7 @@ To test agent behavior at different conversation stages, create separate test ca
|
|||
// Test 3: Final turn - agent should confirm submission
|
||||
{
|
||||
"id": "expense-turn3",
|
||||
"messages": [
|
||||
"input": [
|
||||
{"role": "user", "content": "I want to submit an expense"},
|
||||
{"role": "assistant", "content": "What type of expense?"},
|
||||
{"role": "user", "content": "Business travel, $3500"},
|
||||
|
|
@ -206,7 +203,7 @@ To test agent behavior at different conversation stages, create separate test ca
|
|||
```jsonl
|
||||
{
|
||||
"id": "T003",
|
||||
"messages": [
|
||||
"input": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
|
|
@ -288,16 +285,19 @@ For coverage testing where conversation flow is unpredictable:
|
|||
|
||||
### Standard Mode Fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
| ------------ | ------ | -------- | --------------------------------- |
|
||||
| `id` | string | Yes | Unique test identifier |
|
||||
| `name` | string | No | Human-readable test name |
|
||||
| `input` | string | No\* | Simple text input |
|
||||
| `messages` | array | No\* | Full message history |
|
||||
| `assertions` | array | No | Assertions to validate response |
|
||||
| `options` | object | No | `context.Options` passed to agent |
|
||||
| Field | Type | Required | Description |
|
||||
| ------------ | ------------------------------ | -------- | ------------------------------------------------- |
|
||||
| `id` | string | Yes | Unique test identifier |
|
||||
| `name` | string | No | Human-readable test name |
|
||||
| `input` | string \| Message \| Message[] | Yes | Input: text, single message, or message array |
|
||||
| `assertions` | array | No | Assertions to validate response (alias: `assert`) |
|
||||
| `options` | object | No | `context.Options` passed to agent |
|
||||
|
||||
\*Either `input` or `messages` required
|
||||
**Note**: The `input` field supports three formats:
|
||||
|
||||
- `string`: Simple text (converted to `[{role: "user", content: "..."}]`)
|
||||
- `object`: Single message `{role: "...", content: "..."}`
|
||||
- `array`: Message history `[{role: "user", ...}, {role: "assistant", ...}, ...]`
|
||||
|
||||
### Dynamic Mode Fields
|
||||
|
||||
|
|
@ -327,8 +327,8 @@ For coverage testing where conversation flow is unpredictable:
|
|||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. Parse test case │
|
||||
│ ├─ Has `messages`? → Use as-is │
|
||||
│ └─ Has `input`? → Convert to [{role: "user", content: input}]│
|
||||
│ ├─ `input` is array? → Use as messages │
|
||||
│ └─ `input` is string? → Convert to [{role: "user", content}] │
|
||||
│ ↓ │
|
||||
│ 2. Call Agent.Stream(ctx, messages, options) │
|
||||
│ ↓ │
|
||||
|
|
@ -683,11 +683,11 @@ Respond in JSON format:
|
|||
Existing single-turn tests work unchanged:
|
||||
|
||||
```jsonl
|
||||
// This still works
|
||||
// Simple string input
|
||||
{"id": "T001", "input": "Hello", "assertions": [...]}
|
||||
|
||||
// Equivalent to
|
||||
{"id": "T001", "messages": [{"role": "user", "content": "Hello"}], "assertions": [...]}
|
||||
// Equivalent to array format
|
||||
{"id": "T001", "input": [{"role": "user", "content": "Hello"}], "assertions": [...]}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
|
@ -711,16 +711,17 @@ Existing single-turn tests work unchanged:
|
|||
| Simulator error | Test FAILED | `error: "simulator error: ..."` |
|
||||
| Checkpoint assertion failed | Test FAILED | `error: "checkpoint X failed"` |
|
||||
|
||||
## Comparison: Old vs New Design
|
||||
## Current Implementation Status
|
||||
|
||||
| Aspect | Old (Static Mode) | New (Messages) |
|
||||
| ------------------ | ------------------------- | --------------------------- |
|
||||
| Multi-turn testing | Sequential turn execution | Pass message history |
|
||||
| State management | Session state per test | Stateless |
|
||||
| Parallelization | Sequential within test | Fully parallel |
|
||||
| Implementation | Complex turn loop | Single agent call |
|
||||
| Debugging | Need to trace turns | Clear input/output per test |
|
||||
| Flexibility | Coupled turns | Independent tests |
|
||||
| Feature | Status | Notes |
|
||||
| ----------------------- | ---------- | ---------------------------------------- |
|
||||
| Simple text input | ✅ Done | `input: "Hello"` |
|
||||
| Message history | ✅ Done | `input: [{role, content}, ...]` |
|
||||
| File attachments | ✅ Done | `file://` protocol in content parts |
|
||||
| Static assertions | ✅ Done | contains, equals, regex, json_path, etc. |
|
||||
| Agent-driven assertions | 🔲 Planned | `type: "agent"` with validator agent |
|
||||
| Dynamic mode | 🔲 Planned | Simulator + Checkpoints |
|
||||
| Agent-driven input | 🔲 Planned | `-i agents:xxx` for test generation |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
|
|
|||
|
|
@ -10,16 +10,20 @@
|
|||
| `--simulator` flag | No prefix (agent only) | `--simulator workers.test.user-simulator` |
|
||||
| `t.assert.Agent()` | No prefix (method-bound) | `t.assert.Agent(resp, "workers.test.validator", {...})` |
|
||||
|
||||
## Phase 1: Message History Support
|
||||
## Phase 1: Message History Support ✅ (Already Implemented)
|
||||
|
||||
- [ ] Add `messages` field to test case parser
|
||||
- [ ] Support both `input` (string) and `messages` (array) fields
|
||||
- [ ] Convert `input` to `messages` format internally
|
||||
- [ ] Pass messages directly to `Agent.Stream()`
|
||||
- [ ] Add `options` field support (aligned with `context.Options`)
|
||||
- [ ] Support attachments in message content parts
|
||||
The `input` field already supports:
|
||||
- `string`: Simple text input
|
||||
- `object`: Single message `{role, content}`
|
||||
- `array`: Message history `[{role, content}, ...]`
|
||||
|
||||
See `input.go` → `ParseInputWithOptions()` for implementation.
|
||||
|
||||
**Remaining tasks:**
|
||||
- [ ] Add `options` field support (aligned with `context.Options`) - partially done via `CaseOptions`
|
||||
- [x] Support attachments in message content parts (file:// protocol)
|
||||
- [ ] Update console output to show message count
|
||||
- [ ] Update JSONL output format
|
||||
- [ ] Update JSONL output format with `messages_count`
|
||||
|
||||
## Phase 2: Agent-Driven Input
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue