Commit graph

5 commits

Author SHA1 Message Date
Max
9326f4b747 feat(logging): enhance logger output control and add trace diagnostics
- Updated logger methods to respect the new `config.Silent` flag, allowing suppression of console output in development mode.
- Refactored `buildTurnResponse` to utilize a shared `buildTrace` function for improved tool call association.
- Introduced JSON output capabilities in `OutputWriter` for detailed trace and duration reporting.
- Enhanced reporting mechanisms to include trace details in JSON and Markdown formats for better diagnostics.
- Added support for a new `Trace` structure to capture detailed execution information, including tool calls and completion data.
2026-04-28 10:37:04 +08:00
Max
5a63d9280b Implement Tool Call Assertions in Asserter for Enhanced Validation
- Added support for `tool_called` and `tool_result` assertions in the Asserter, allowing for validation of tool execution and arguments.
- Introduced methods to check if specific tools were called and to validate their results against expected patterns.
- Updated the `README.md` to include detailed documentation on new assertion types, including usage examples and value formats.
- Enhanced the dynamic runner to set the response for tool-related assertions, improving the overall testing framework's capabilities.
2025-12-27 14:23:15 +08:00
Max
3969279f71 Add Detailed Validation for Agent Assertions in Dynamic Testing
- Introduced `ValidateWithDetails` method in the Asserter to provide comprehensive results for agent assertions, including failure messages and validation criteria.
- Updated `checkCheckpoints` to utilize the new validation method, enhancing the checkpoint validation process with detailed agent validation results.
- Enhanced `CheckpointResult` structure to include `AgentValidationResult`, capturing the agent's response and validation details.
- Expanded documentation in README.md to explain the new validation process and output structure for agent assertions, ensuring clarity on the validation workflow and expected results.
2025-12-27 13:40:15 +08:00
Max
5fb5035d8c Enhance Agent Response Structure and Dynamic Test Functionality
- Added support for tool call responses in the agent's response structure, allowing for better handling of tool execution results.
- Updated the `TurnResult` and `TurnResponse` types to include full agent responses, including tool call details and next hook data.
- Improved dynamic test execution by ensuring consistent chat session state across turns, enhancing the overall testing framework's capabilities.
- Enhanced documentation in README.md to reflect changes in response structure and dynamic testing output format.
2025-12-27 12:58:09 +08:00
Max
8de302d4e8 Implement Dynamic Testing Features in Agent Test Framework
- Introduced dynamic testing capabilities, allowing for multi-turn conversations with checkpoints through the new `DynamicRunner`.
- Enhanced the `runSingleTest` method to support dynamic mode, including the execution of before/after scripts and detailed output for dynamic test results.
- Added new output methods in `output.go` for dynamic test start, turns, checkpoints, and results, improving console feedback during testing.
- Updated `DESIGN_V2.md` and `TODO_V2.md` to reflect the new dynamic mode features, including simulator configurations and checkpoint definitions.
- Revised the `Case` struct in `types.go` to include fields for dynamic testing, such as `Simulator`, `Checkpoints`, and `MaxTurns`.
2025-12-26 11:17:54 +08:00