- Updated README files to include new commands for running tests and extracting results for review.
- Introduced the `yao agent extract` command to facilitate extraction of test results from JSONL files into Markdown or JSON formats.
- Enhanced the `FormatAvailableResources` function to support localization and detailed information for agents and MCP tools.
- Improved output formatting for better readability and usability in test result documentation.
- Added support for `tool_called` and `tool_result` assertions in the Asserter, allowing for validation of tool execution and arguments.
- Introduced methods to check if specific tools were called and to validate their results against expected patterns.
- Updated the `README.md` to include detailed documentation on new assertion types, including usage examples and value formats.
- Enhanced the dynamic runner to set the response for tool-related assertions, improving the overall testing framework's capabilities.
- Introduced `ValidateWithDetails` method in the Asserter to provide comprehensive results for agent assertions, including failure messages and validation criteria.
- Updated `checkCheckpoints` to utilize the new validation method, enhancing the checkpoint validation process with detailed agent validation results.
- Enhanced `CheckpointResult` structure to include `AgentValidationResult`, capturing the agent's response and validation details.
- Expanded documentation in README.md to explain the new validation process and output structure for agent assertions, ensuring clarity on the validation workflow and expected results.
- Added support for tool call responses in the agent's response structure, allowing for better handling of tool execution results.
- Updated the `TurnResult` and `TurnResponse` types to include full agent responses, including tool call details and next hook data.
- Improved dynamic test execution by ensuring consistent chat session state across turns, enhancing the overall testing framework's capabilities.
- Enhanced documentation in README.md to reflect changes in response structure and dynamic testing output format.
- Changed the default timeout value from 5 minutes to 2 minutes in the codebase and updated relevant sections in DESIGN_V2.md and README.md to reflect this change.
- Enhanced error handling in dynamic integration tests to account for both max turns exceeded and goal achieved scenarios without checkpoints.
- Improved formatting in the documentation for better clarity and consistency.
- Added detailed sections on Before/After hooks, including types, execution order, and function signatures.
- Provided examples for common use cases such as database setup/teardown and conditional setup based on metadata.
- Clarified parameters for hook functions, improving guidance for users on implementing hooks in their tests.
- Enhanced overall documentation to support better understanding of the testing framework's capabilities.
- Expanded the input format section to include comprehensive examples organized by scenario, covering various testing cases such as simple text input, assertions, conversation history, and dynamic mode.
- Added scenarios demonstrating the use of file attachments, agent-driven assertions, and options for test configuration.
- Improved clarity on the structure and requirements for JSONL input, ensuring users have clear guidance on how to format their test cases effectively.
- Removed outdated TODO_V2.md file to streamline documentation and focus on the updated README content.
- Updated the introduction to clarify the framework's capabilities, emphasizing support for standard testing, dynamic testing, agent-driven assertions, and CI integration.
- Revised section headers for better organization, changing "Agent Tests" to "Standard Tests" and adding new sections for "Agent-Driven Input" and "Dynamic Mode."
- Included detailed examples for generating test cases using agents and running dynamic tests with simulators.
- Improved clarity on input modes and script test modes, ensuring users understand the requirements and options available for testing.
- Added notes on the importance of the `-n` flag for agent-driven input mode and clarified the distinction between script and agent-driven test case generation.
- Introduced support for file attachments in test inputs using the `file://` protocol, allowing images, audio, and documents to be loaded and converted to appropriate formats.
- Updated `ParseInput` and related functions to handle file references, ensuring seamless integration of file content into messages.
- Enhanced error handling and path resolution for file loading, considering both relative paths and the `YAO_ROOT` environment variable.
- Expanded documentation to include examples of file attachments and their usage in test cases, improving clarity for users.
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code.
- Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed.
- Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests.
- Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data.
- Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
- Added support for a new `Metadata` field in the `Options` struct to allow passing custom data to hooks, enhancing flexibility in context management.
- Updated the `ToMap` and `OptionsFromMap` methods to include serialization and deserialization of the `Metadata` field.
- Enhanced the test case structure to include an `Options` field, allowing for per-test-case configuration, including metadata and skip options.
- Updated documentation to reflect the new `options` and `metadata` fields, providing clear examples for users on how to utilize these features in test cases.
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation.
- Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs.
- Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter.
- Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
