From 2e2ce8bfad8190741e5b3216ad30ae141d13cfd1 Mon Sep 17 00:00:00 2001 From: Max Date: Thu, 25 Dec 2025 14:18:27 +0800 Subject: [PATCH] Refactor LoadWithRoot Function for Enhanced Clarity and Consistency - Further streamlined the LoadWithRoot function by removing redundant checks and improving the overall readability of the path resolution logic. - Ensured consistent resolution of the absolute path for the configuration root, enhancing maintainability and clarity in the codebase. --- agent/test/DESIGN_V2.md | 846 ++++++++++++++++++++++++++++++++++++++++ agent/test/TODO_V2.md | 70 ++++ 2 files changed, 916 insertions(+) create mode 100644 agent/test/DESIGN_V2.md create mode 100644 agent/test/TODO_V2.md diff --git a/agent/test/DESIGN_V2.md b/agent/test/DESIGN_V2.md new file mode 100644 index 00000000..41c50a09 --- /dev/null +++ b/agent/test/DESIGN_V2.md @@ -0,0 +1,846 @@ +# Agent Test Framework V2 Design + +## Overview + +This document describes the design for Agent Test Framework V2, which extends the existing testing capabilities with: + +- **Multi-turn conversations** - Test agents across multiple interaction rounds +- **Agent-driven testing** - Use agents to generate test cases and simulate user responses +- **Interactive testing** - Human-in-the-loop testing mode + +## Problem Statement + +Current single-turn testing cannot adequately test: + +1. **Conversational flows** - Agents that guide users through multi-step processes +2. **Confirmation dialogs** - Agents that ask for user confirmation before actions +3. **Clarification requests** - Agents that ask follow-up questions when input is ambiguous +4. **Stateful interactions** - Agents that maintain context across multiple turns + +## Design Goals + +1. **Unified format** - Single test case format that supports all modes +2. **Flexible execution** - Static, dynamic (simulator), and interactive modes +3. **Graceful degradation** - Skip tests when required input is unavailable +4. **CI/CD compatible** - Non-interactive mode for automated pipelines +5. **Agent-driven** - Both input generation and user simulation can be agent-powered + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ yao agent test │ +├─────────────────────────────────────────────────────────────────────────┤ +│ │ +│ INPUT SOURCES (-i flag) │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ JSONL File │ │ Message │ │ Generator │ │ Interactive │ │ +│ │ ./test.jsonl│ │ "Hello..." │ │ agent:xxx │ │ (stdin) │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ │ +│ └────────────────┴────────────────┴────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Test Case Parser │ │ +│ │ - Single-turn: {input, assertions} │ │ +│ │ - Multi-turn: {turns: [{input, assertions}, ...]} │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Multi-Turn Executor │ │ +│ │ │ │ +│ │ ┌─────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ +│ │ │ Turn │───▶│ Target Agent │───▶│ Response │ │ │ +│ │ │ Input │ │ (being tested) │ │ + State │ │ │ +│ │ └─────────┘ └─────────────────┘ └────────┬────────┘ │ │ +│ │ ▲ │ │ │ +│ │ │ ▼ │ │ +│ │ │ ┌─────────────────────────────────────┐ │ │ +│ │ │ │ Awaiting Input Detection │ │ │ +│ │ │ │ - Explicit declaration │ │ │ +│ │ │ │ - Tool-based detection │ │ │ +│ │ │ │ - Content heuristics │ │ │ +│ │ │ └──────────────┬──────────────────────┘ │ │ +│ │ │ │ │ │ +│ │ │ ┌─────────┴─────────┐ │ │ +│ │ │ ▼ ▼ │ │ +│ │ │ Awaiting=YES Awaiting=NO │ │ +│ │ │ │ │ │ │ +│ │ │ ▼ ▼ │ │ +│ │ NEXT INPUT ┌─────────┐ ┌─────────┐ │ │ +│ │ SOURCES: │ Get Next│ │Complete │ │ │ +│ │ │ Input │ │ Test │ │ │ +│ │ ┌──────────┐ └────┬────┘ └─────────┘ │ │ +│ │ │ Static │◀───────┤ │ │ +│ │ │ turns[] │ │ │ │ +│ │ └──────────┘ │ │ │ +│ │ ┌──────────┐ │ │ │ +│ │ │Simulator │◀───────┤ │ │ +│ │ │ Agent │ │ │ │ +│ │ └──────────┘ │ │ │ +│ │ ┌──────────┐ │ │ │ +│ │ │ Human │◀───────┤ │ │ +│ │ │ Input │ │ │ │ +│ │ └──────────┘ │ │ │ +│ │ ┌──────────┐ │ │ │ +│ │ │ SKIP │◀───────┘ │ │ +│ │ │ (no src) │ │ │ +│ │ └──────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Assertions │ │ +│ │ - Per-turn assertions │ │ +│ │ - Final assertions │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Reporter │ │ +│ │ - Console output │ │ +│ │ - JSONL output │ │ +│ │ - Custom reporter agent │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Core Challenge: Detecting "Awaiting Input" State + +The key challenge is determining when an agent is waiting for user input vs. when it has completed its task. + +### Detection Strategies + +#### Strategy 1: Explicit Declaration (Recommended) + +Agent explicitly declares its state in the response: + +```json +{ + "content": "What is the expense amount?", + "awaiting_input": true, + "input_hint": "Enter amount, e.g., $3500" +} +``` + +#### Strategy 2: Finish Reason Analysis + +Use the LLM's `finish_reason` to determine state: + +- `stop` - Agent completed normally (may or may not need input) +- `tool_calls` - Agent is executing tools (not awaiting input) +- `length` - Response truncated (not awaiting input) + +#### Strategy 3: Content Heuristics + +Analyze response content for question patterns: + +```go +func looksLikeQuestion(content string) bool { + patterns := []string{ + `\?$`, // Ends with question mark + `(?i)^(what|how|when|where|which|who|please|could you)`, + `(?i)(confirm|verify|proceed|continue)\?`, + } + for _, pattern := range patterns { + if regexp.MustCompile(pattern).MatchString(content) { + return true + } + } + return false +} +``` + +#### Strategy 4: Tool-Based Detection + +Certain tools indicate awaiting input: + +```go +func toolRequiresInput(toolCall ToolCall) bool { + confirmationTools := []string{ + "request_confirmation", + "ask_user", + "get_user_input", + } + return contains(confirmationTools, toolCall.Name) +} +``` + +### Recommended Approach: Hybrid Detection + +Combine multiple strategies with priority: + +```go +func IsAwaitingInput(result *TurnResult) (awaiting bool, reason string) { + // Priority 1: Explicit declaration + if result.AwaitingInput { + return true, "agent_declared" + } + + // Priority 2: Tool-based detection + for _, tc := range result.ToolCalls { + if toolRequiresInput(tc) { + return true, "tool_requires_confirmation" + } + } + + // Priority 3: Content heuristics + if looksLikeQuestion(result.Content) { + return true, "content_is_question" + } + + return false, "completed" +} +``` + +## Test Case Format + +### Single-Turn (Existing) + +```jsonl +{ + "id": "T001", + "input": "Hello", + "assertions": [ + { + "type": "contains", + "value": "Hi" + } + ] +} +``` + +### Multi-Turn (New) + +```jsonl +{ + "id": "T001", + "name": "Expense Reimbursement Flow", + "type": "multi_turn", + "turns": [ + { + "input": "I want to submit an expense report", + "assertions": [ + { + "type": "contains", + "value": "type of expense" + } + ] + }, + { + "input": "Business travel to Beijing, flight $2000, hotel $1500", + "assertions": [ + { + "type": "tool_called", + "name": "create_expense" + } + ] + }, + { + "input": "Yes, confirm", + "assertions": [ + { + "type": "contains", + "value": "submitted" + } + ] + } + ], + "simulator": { + "use": "agents:workers.test.user-simulator", + "persona": "New employee unfamiliar with expense process", + "goal": "Submit a $3500 travel expense", + "max_turns": 10 + }, + "interactive": { + "enabled": false, + "timeout": "5m" + }, + "on_missing_input": "skip", + "final_assertions": [ + { + "type": "json_path", + "path": "$.expense.status", + "value": "submitted" + } + ] +} +``` + +### Field Descriptions + +| Field | Type | Required | Description | +| --------------------- | ------ | -------- | ------------------------------------------------ | +| `id` | string | Yes | Unique test identifier | +| `name` | string | No | Human-readable test name | +| `type` | string | No | `"single_turn"` (default) or `"multi_turn"` | +| `turns` | array | No | Static turn definitions | +| `turns[].input` | string | Yes | User input for this turn | +| `turns[].assertions` | array | No | Assertions for this turn's response | +| `simulator` | object | No | Dynamic input generator configuration | +| `simulator.use` | string | Yes | Simulator reference: `agents:id` or `scripts:id` | +| `simulator.persona` | string | No | User persona description | +| `simulator.goal` | string | No | What the simulated user wants to achieve | +| `simulator.max_turns` | int | No | Maximum turns before timeout (default: 20) | +| `interactive` | object | No | Interactive mode configuration | +| `interactive.enabled` | bool | No | Enable human input (default: false) | +| `interactive.timeout` | string | No | Timeout for human input (default: "5m") | +| `on_missing_input` | string | No | `"skip"`, `"fail"`, or `"end"` (default: "skip") | +| `final_assertions` | array | No | Assertions after conversation completes | + +## Execution Modes + +### Mode 1: Static Turns + +Uses predefined `turns` array. Best for deterministic flows. + +``` +Turn 1: Send turns[0].input → Assert turns[0].assertions +Turn 2: Send turns[1].input → Assert turns[1].assertions +... +``` + +### Mode 2: Dynamic Simulator + +Uses an agent to simulate user responses. Best for complex/variable flows. + +``` +Turn 1: Send initial input → Get response +Turn 2: Simulator generates input based on response → Get response +... +Until: Goal achieved OR max_turns reached +``` + +### Mode 3: Interactive + +Prompts human for input when agent awaits. Best for debugging/exploration. + +``` +Turn 1: Send input → Get response +Turn 2: [Agent awaiting] → Prompt human → Get response +... +``` + +### Mode 4: Skip (Default Fallback) + +When agent awaits input but no source available, skip with explanation. + +### Mode Priority + +When multiple input sources are configured, they are used in this order: + +1. **Static turns** - If `turns[n+1]` exists, use it +2. **Simulator** - If no more static turns but simulator configured, use it +3. **Interactive** - If `--interactive` flag and no simulator, prompt human +4. **Skip/Fail/End** - Based on `on_missing_input` setting + +This allows hybrid testing: define some turns statically, then let simulator handle the rest. + +```jsonl +{ + "turns": [ + { + "input": "Start expense report" + }, + { + "input": "Travel expense, $500" + } + ], + "simulator": { + "use": "agents:workers.test.user-sim", + "goal": "Complete the expense submission" + } +} +``` + +In this example: + +- Turn 1-2: Use static inputs +- Turn 3+: Simulator generates inputs until goal achieved + +## Execution Flow + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Multi-Turn Test Execution │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ START: Get initial input │ +│ ├─ From turns[0].input if defined │ +│ └─ From test.input (single-turn compat) │ +│ ↓ │ +│ LOOP: │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ 1. Send input to Agent │ │ +│ │ ↓ │ │ +│ │ 2. Get Agent response │ │ +│ │ ↓ │ │ +│ │ 3. Execute turn assertions (if defined) │ │ +│ │ ↓ │ │ +│ │ 4. Check: Is Agent awaiting input? │ │ +│ │ │ │ │ +│ │ ├─ NO → Exit loop (conversation complete) │ │ +│ │ │ │ │ +│ │ └─ YES → Get next input: │ │ +│ │ │ │ │ +│ │ ├─ turns[n+1] exists? │ │ +│ │ │ → Use static input │ │ +│ │ │ │ │ +│ │ ├─ simulator configured? │ │ +│ │ │ → Call simulator agent │ │ +│ │ │ │ │ +│ │ ├─ interactive enabled? │ │ +│ │ │ → Prompt for human input │ │ +│ │ │ │ │ +│ │ └─ None available? │ │ +│ │ → Handle per on_missing_input: │ │ +│ │ skip: SKIP test │ │ +│ │ fail: FAIL test │ │ +│ │ end: Exit loop normally │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ ↓ │ +│ END: Execute final_assertions │ +│ Report result │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## Command Line Interface + +### Flags Reference + +| Flag | Long | Description | +| ---- | --------------- | -------------------------------------------------------- | +| `-i` | `--input` | Input source: file path, message, or `type:id` reference | +| `-n` | `--name` | Target agent ID (the agent being tested) | +| `-o` | `--output` | Output file path for results | +| `-c` | `--connector` | Override connector for the target agent | +| `-v` | `--verbose` | Verbose output showing all turns | +| | `--interactive` | Enable human input when agent awaits | +| | `--simulator` | Default simulator: `agents:id` or `scripts:id` | +| | `--timeout` | Timeout per test case (default: 5m) | +| | `--parallel` | Number of parallel test cases | +| | `--fail-fast` | Stop on first failure | +| | `--dry-run` | Generate/parse tests without running | + +### Input Sources (`-i` flag) + +The `-i` flag supports multiple input sources with unified `type:id` format: + +```bash +# 1. File path (default, no prefix needed) +yao agent test -i ./tests/multi-turn.jsonl + +# 2. Direct message (no prefix, auto-detected as non-file) +yao agent test -i "Hello, how are you?" -n assistants.chat + +# 3. Agent-generated test cases +yao agent test -i agents:workers.test.generator -n assistants.expense + +# 4. Script-generated test cases +yao agent test -i scripts:tests.generate -n assistants.expense + +# 5. With parameters (query string style) +yao agent test -i "agents:workers.test.generator?count=10&focus=edge-cases" -n assistants.expense +``` + +### Input Type Prefixes + +| Prefix | Description | Example | +| ---------- | --------------------------- | -------------------------- | +| (none) | File path or direct message | `./tests.jsonl`, `"Hello"` | +| `agents:` | Agent generates test cases | `agents:workers.test.gen` | +| `scripts:` | Script generates test cases | `scripts:tests.generate` | + +### Input Format + +``` +[prefix:][?param1=value1¶m2=value2] +``` + +- `prefix` - Input type: `agents:` or `scripts:` (optional, default is file/message) +- `id` - Agent ID or script ID +- `?params` - Query parameters passed to generator + +#### Generator Agent Interface + +```typescript +// Input to generator agent +interface GeneratorInput { + target_agent: string; // Agent being tested (from -n flag) + target_description?: string; // Agent's description/purpose + target_tools?: Tool[]; // Agent's available tools + count?: number; // Number of test cases to generate + focus?: string; // Focus area: "happy-path", "edge-cases", "errors" + complexity?: string; // "simple", "medium", "complex" +} + +// Output from generator agent +interface GeneratorOutput { + cases: TestCase[]; // Generated test cases +} +``` + +#### Example Generator Prompt + +``` +You are a test case generator for AI agents. + +Target Agent: {{target_agent}} +Description: {{target_description}} +Available Tools: {{target_tools}} + +Generate {{count}} test cases with focus on: {{focus}} + +For each test case, provide: +- id: Unique identifier +- name: Descriptive name +- input: User message or turns array for multi-turn +- assertions: Expected behaviors to verify + +Output as JSON array of test cases. +``` + +### Complete Examples + +```bash +# Basic: Run tests from file +yao agent test -i ./tests/expense.jsonl + +# With target agent specified (required for message/agent input) +yao agent test -i "Help me file an expense" -n assistants.expense + +# Agent generates tests, then runs them +yao agent test \ + -i "agents:workers.test.generator?count=20&focus=edge-cases" \ + -n assistants.expense + +# Script generates tests +yao agent test \ + -i "scripts:tests.expense.generate?scenario=approval-flow" \ + -n assistants.expense + +# Fully dynamic: Agent generates tests + Agent simulates user responses +yao agent test \ + -i "agents:workers.test.generator?count=10" \ + -n assistants.expense \ + --simulator agents:workers.test.user-simulator + +# Generate tests only, save to file (dry-run) +yao agent test \ + -i "agents:workers.test.generator?count=50" \ + -n assistants.expense \ + -o ./tests/generated.jsonl \ + --dry-run + +# Interactive mode: human provides input when agent awaits +yao agent test -i ./tests/multi-turn.jsonl --interactive + +# CI/CD mode: skip tests requiring human input +yao agent test -i ./tests/multi-turn.jsonl --skip-interactive + +# Fail instead of skip when input unavailable +yao agent test -i ./tests/multi-turn.jsonl --on-missing-input=fail + +# Verbose output showing all turns +yao agent test -i ./tests/multi-turn.jsonl -v +``` + +## Output Format + +### Console Output + +``` +═══════════════════════════════════════════════════════════════ + Agent Test (Multi-Turn) +═══════════════════════════════════════════════════════════════ +ℹ Agent: assistants.expense +ℹ Input: ./tests/expense-flow.jsonl (5 test cases) + +─────────────────────────────────────────────────────────────── + Running Tests +─────────────────────────────────────────────────────────────── + +► [T001] Expense Reimbursement Flow (3 turns) + ├─ Turn 1: "I want to submit an expense" → PASSED (2.1s) + │ Agent: "What type of expense would you like to submit?" + │ ✓ contains "type of expense" + │ + ├─ Turn 2: "Business travel, $3500" → PASSED (3.2s) + │ Agent: [tool: create_expense({amount: 3500, type: "travel"})] + │ ✓ tool_called "create_expense" + │ + ├─ Turn 3: "Yes, confirm" → PASSED (1.8s) + │ Agent: "Expense submitted. Reference: EXP-2025-001" + │ ✓ contains "submitted" + │ + └─ Final Assertions: PASSED + ✓ $.expense.status = "submitted" + +► [T002] Large Expense Approval + ├─ Turn 1: "Submit $100,000 equipment purchase" → PASSED (2.0s) + │ Agent: "This requires manager approval. Please provide PO number." + │ + ├─ Turn 2: SKIPPED + │ Reason: Agent awaiting input, no next turn defined + │ Agent asked: "Please provide PO number" + │ Hint: Add more turns, use --interactive, or configure simulator + │ + └─ Result: SKIPPED + +► [T003] Dynamic Expense Flow (simulator: workers.test.user-sim) + ├─ Turn 1: [Initial] "Help me file an expense" → PASSED (2.1s) + ├─ Turn 2: [Simulated] "It's for client dinner, $250" → PASSED (2.8s) + ├─ Turn 3: [Simulated] "Yesterday evening" → PASSED (2.2s) + ├─ Turn 4: [Simulated] "Confirm" → PASSED (1.9s) + │ Goal achieved: Expense submitted + │ + └─ Final Assertions: PASSED + +─────────────────────────────────────────────────────────────── + Summary +─────────────────────────────────────────────────────────────── + Total: 3 tests + Passed: 2 + Failed: 0 + Skipped: 1 + + Total turns: 10 + Avg turns/test: 3.3 + Total time: 18.1s +``` + +### JSONL Output + +```jsonl +{ + "id": "T001", + "name": "Expense Reimbursement Flow", + "status": "passed", + "turns": [ + { + "turn": 1, + "input": "I want to submit an expense", + "input_source": "static", + "output": "What type of expense would you like to submit?", + "awaiting_input": true, + "assertions": [ + { + "type": "contains", + "value": "type of expense", + "passed": true + } + ], + "duration_ms": 2100 + }, + { + "turn": 2, + "input": "Business travel, $3500", + "input_source": "static", + "output": "", + "tool_calls": [ + { + "name": "create_expense", + "args": { + "amount": 3500 + } + } + ], + "awaiting_input": true, + "assertions": [ + { + "type": "tool_called", + "name": "create_expense", + "passed": true + } + ], + "duration_ms": 3200 + }, + { + "turn": 3, + "input": "Yes, confirm", + "input_source": "static", + "output": "Expense submitted. Reference: EXP-2025-001", + "awaiting_input": false, + "assertions": [ + { + "type": "contains", + "value": "submitted", + "passed": true + } + ], + "duration_ms": 1800 + } + ], + "final_assertions": [ + { + "type": "json_path", + "path": "$.expense.status", + "value": "submitted", + "passed": true + } + ], + "total_turns": 3, + "duration_ms": 7100 +} +``` + +## User Simulator Agent + +### Interface + +The simulator agent receives conversation context and generates the next user input: + +```typescript +// Input to simulator +interface SimulatorInput { + persona: string; // User persona description + goal: string; // What user wants to achieve + conversation: Message[]; // Conversation history + last_response: string; // Agent's last response + turn_number: number; // Current turn (1-based) + max_turns: number; // Maximum allowed turns +} + +// Output from simulator +interface SimulatorOutput { + input: string; // Generated user input + goal_achieved: boolean; // Whether goal is complete + reasoning?: string; // Why this input was chosen +} +``` + +### Example Simulator Prompt + +``` +You are simulating a user with the following characteristics: + +Persona: {{persona}} +Goal: {{goal}} + +Current conversation: +{{conversation}} + +The agent just responded: +"{{last_response}}" + +Generate the next user message to continue toward the goal. +If the goal has been achieved, set goal_achieved to true. + +Respond in JSON format: +{ + "input": "your response as the user", + "goal_achieved": true/false, + "reasoning": "brief explanation" +} +``` + +## Backward Compatibility + +Existing single-turn tests continue to work unchanged: + +```jsonl +// This still works +{"id": "T001", "input": "Hello", "assertions": [...]} + +// Equivalent to +{"id": "T001", "type": "single_turn", "turns": [{"input": "Hello", "assertions": [...]}]} +``` + +## Error Handling + +### Turn-Level Errors + +| Error Type | Behavior | Output | +| ---------------- | -------------------- | -------------------------------- | +| Agent timeout | Mark turn as FAILED | `error: "timeout after 30s"` | +| Agent error | Mark turn as FAILED | `error: "agent error: ..."` | +| Assertion failed | Mark turn as FAILED | `assertion_errors: [...]` | +| Simulator error | Mark turn as SKIPPED | `skip_reason: "simulator error"` | + +### Test-Level Errors + +| Error Type | Behavior | Output | +| ---------------------------- | -------------------- | ---------------------------------- | +| No initial input | Mark test as FAILED | `error: "no initial input"` | +| Max turns exceeded | Mark test as FAILED | `error: "max turns (20) exceeded"` | +| All turns passed | Mark test as PASSED | `status: "passed"` | +| Any turn failed | Mark test as FAILED | `status: "failed"` | +| Skipped due to missing input | Mark test as SKIPPED | `status: "skipped"` | + +## Context and State + +### Conversation Context + +Each multi-turn test maintains a conversation context: + +```go +type ConversationContext struct { + SessionID string // Unique session for this test + Messages []Message // Full conversation history + ToolResults map[string]any // Results from tool calls + Variables map[string]any // Custom variables set during test + TurnCount int // Current turn number +} +``` + +### Context Passing to Simulator + +The simulator receives full context to generate appropriate responses: + +```json +{ + "persona": "New employee", + "goal": "Submit expense report", + "context": { + "session_id": "test-session-001", + "messages": [...], + "tool_results": { + "create_expense": {"id": "EXP-001", "status": "pending"} + }, + "turn_count": 3 + }, + "last_response": "Please confirm the expense details..." +} +``` + +## Attachments in Multi-Turn + +Multi-turn tests support attachments at the turn level: + +```jsonl +{ + "id": "T001", + "name": "Receipt Upload Flow", + "turns": [ + { + "input": "I want to submit an expense with receipt", + "attachments": [ + { + "type": "image", + "source": "file://./tests/fixtures/receipt.jpg" + } + ] + }, + { + "input": "The amount is $150" + } + ] +} +``` + +## Open Questions + +1. **Session Management**: How to handle session state across turns? Use existing session or create new per-test? + +2. **Timeout Strategy**: Per-turn timeout vs. total test timeout? + +3. **Parallel Execution**: Can multi-turn tests run in parallel, or must they be sequential? + +4. **Retry Logic**: If a turn fails, retry just that turn or restart entire conversation? + +5. **Snapshot Testing**: Should we support "golden file" comparison for conversation flows? diff --git a/agent/test/TODO_V2.md b/agent/test/TODO_V2.md new file mode 100644 index 00000000..ff7c2058 --- /dev/null +++ b/agent/test/TODO_V2.md @@ -0,0 +1,70 @@ +# Agent Test Framework V2 - Implementation TODO + +## Phase 1: Static Multi-Turn + +- [ ] Extend test case parser for `turns` array +- [ ] Implement turn-by-turn execution +- [ ] Implement conversation context management +- [ ] Add per-turn assertions +- [ ] Support attachments at turn level +- [ ] Implement awaiting input detection (heuristics) +- [ ] Add `on_missing_input` handling (`skip`, `fail`, `end`) +- [ ] Implement mode priority (static → simulator → interactive → skip) +- [ ] Update console output for multi-turn display +- [ ] Update JSONL output format for turns + +## Phase 2: Agent-Driven Input + +- [ ] Parse `agents:` prefix in `-i` flag +- [ ] Parse `scripts:` prefix in `-i` flag +- [ ] Define generator agent interface (GeneratorInput/GeneratorOutput) +- [ ] Implement generator invocation +- [ ] Support query parameters (`?count=10&focus=...`) +- [ ] Pass target agent metadata to generator (description, tools) +- [ ] Add `--dry-run` flag to save generated cases without running +- [ ] Create example generator agent + +## Phase 3: Dynamic Simulator + +- [ ] Define simulator agent interface (SimulatorInput/SimulatorOutput) +- [ ] Implement simulator invocation with full context +- [ ] Pass conversation history and tool results to simulator +- [ ] Add goal completion detection +- [ ] Add max_turns limit and timeout +- [ ] Support hybrid mode (static turns + simulator fallback) +- [ ] Create example simulator agent + +## Phase 4: Interactive Mode + +- [ ] Add `--interactive` flag +- [ ] Implement terminal input prompt with context display +- [ ] Add input timeout handling +- [ ] Support input history/editing +- [ ] Add `--skip-interactive` for CI/CD mode + +## Phase 5: Enhanced Detection + +- [ ] Add `awaiting_input` field to agent response schema +- [ ] Implement tool-based detection (confirmation tools) +- [ ] Add configurable detection rules +- [ ] Support custom detection via script/agent + +## Phase 6: Error Handling & Reporting + +- [ ] Implement turn-level error handling +- [ ] Implement test-level error aggregation +- [ ] Add detailed error messages with hints +- [ ] Support custom reporter agent + +## Open Questions + +1. **Session Management**: How to handle session state across turns? Use existing session or create new per-test? + +2. **Timeout Strategy**: Per-turn timeout vs. total test timeout? + +3. **Parallel Execution**: Can multi-turn tests run in parallel, or must they be sequential? + +4. **Retry Logic**: If a turn fails, retry just that turn or restart entire conversation? + +5. **Snapshot Testing**: Should we support "golden file" comparison for conversation flows? +