Commit graph

8 commits

Author SHA1 Message Date
Max
5a63d9280b Implement Tool Call Assertions in Asserter for Enhanced Validation
- Added support for `tool_called` and `tool_result` assertions in the Asserter, allowing for validation of tool execution and arguments.
- Introduced methods to check if specific tools were called and to validate their results against expected patterns.
- Updated the `README.md` to include detailed documentation on new assertion types, including usage examples and value formats.
- Enhanced the dynamic runner to set the response for tool-related assertions, improving the overall testing framework's capabilities.
2025-12-27 14:23:15 +08:00
Max
3969279f71 Add Detailed Validation for Agent Assertions in Dynamic Testing
- Introduced `ValidateWithDetails` method in the Asserter to provide comprehensive results for agent assertions, including failure messages and validation criteria.
- Updated `checkCheckpoints` to utilize the new validation method, enhancing the checkpoint validation process with detailed agent validation results.
- Enhanced `CheckpointResult` structure to include `AgentValidationResult`, capturing the agent's response and validation details.
- Expanded documentation in README.md to explain the new validation process and output structure for agent assertions, ensuring clarity on the validation workflow and expected results.
2025-12-27 13:40:15 +08:00
Max
88610da75b Implement Agent-Driven Assertions in Test Framework
- Added support for agent-driven assertions in the Asserter, allowing validation of responses using specified agents.
- Introduced the `Use` and `Options` fields in the Assertion struct to facilitate agent configuration.
- Enhanced the `evaluateAssertion` method to handle assertions of type "agent".
- Implemented the `assertAgent` method to manage agent interactions and validation logic.
- Updated `script_assert.go` to include the `assertAgentMethod` for JavaScript API integration.
- Revised documentation in DESIGN_V2.md and TODO_V2.md to reflect the new agent-driven assertion capabilities and implementation status.
2025-12-26 10:16:40 +08:00
Max
7f44e442da Enhance File Handling and Input Parsing in Test Framework
- Introduced support for file attachments in test inputs using the `file://` protocol, allowing images, audio, and documents to be loaded and converted to appropriate formats.
- Updated `ParseInput` and related functions to handle file references, ensuring seamless integration of file content into messages.
- Enhanced error handling and path resolution for file loading, considering both relative paths and the `YAO_ROOT` environment variable.
- Expanded documentation to include examples of file attachments and their usage in test cases, improving clarity for users.
2025-12-25 11:50:58 +08:00
Max
5e64c78435 Implement Script Testing Framework and Enhance Test Context Management
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code.
- Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed.
- Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests.
- Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data.
- Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
2025-12-21 10:44:59 +08:00
Max
3cf3060e0c Enhance Search Functionality with Keyword Extraction and Intent Detection
- Updated the search handling to incorporate keyword extraction with weights, improving the relevance of search results.
- Refactored the `shouldAutoSearch` method to return a `SearchIntent` struct, allowing for more nuanced control over search execution based on context.
- Enhanced the `buildSearchRequests` function to utilize extracted keywords, optimizing search queries based on user input.
- Improved the handling of search types and conditions, ensuring that the system can dynamically adjust search behavior based on intent and configuration.
- Updated documentation and prompts to reflect changes in keyword extraction and search intent classification, providing clearer guidelines for usage.
2025-12-18 20:16:56 +08:00
Max
4abc7e41ef Enhance Options and Metadata Handling in Context and Test Cases
- Added support for a new `Metadata` field in the `Options` struct to allow passing custom data to hooks, enhancing flexibility in context management.
- Updated the `ToMap` and `OptionsFromMap` methods to include serialization and deserialization of the `Metadata` field.
- Enhanced the test case structure to include an `Options` field, allowing for per-test-case configuration, including metadata and skip options.
- Updated documentation to reflect the new `options` and `metadata` fields, providing clear examples for users on how to utilize these features in test cases.
2025-12-18 11:03:55 +08:00
Max
9e4febf782 Enhance Test Case Structure with Custom Assertions
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation.
- Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs.
- Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter.
- Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
2025-12-17 14:35:57 +08:00