- Added support for `tool_called` and `tool_result` assertions in the Asserter, allowing for validation of tool execution and arguments.
- Introduced methods to check if specific tools were called and to validate their results against expected patterns.
- Updated the `README.md` to include detailed documentation on new assertion types, including usage examples and value formats.
- Enhanced the dynamic runner to set the response for tool-related assertions, improving the overall testing framework's capabilities.
- Introduced `ValidateWithDetails` method in the Asserter to provide comprehensive results for agent assertions, including failure messages and validation criteria.
- Updated `checkCheckpoints` to utilize the new validation method, enhancing the checkpoint validation process with detailed agent validation results.
- Enhanced `CheckpointResult` structure to include `AgentValidationResult`, capturing the agent's response and validation details.
- Expanded documentation in README.md to explain the new validation process and output structure for agent assertions, ensuring clarity on the validation workflow and expected results.
- Added support for tool call responses in the agent's response structure, allowing for better handling of tool execution results.
- Updated the `TurnResult` and `TurnResponse` types to include full agent responses, including tool call details and next hook data.
- Improved dynamic test execution by ensuring consistent chat session state across turns, enhancing the overall testing framework's capabilities.
- Enhanced documentation in README.md to reflect changes in response structure and dynamic testing output format.
- Changed the default timeout value from 5 minutes to 2 minutes in the codebase and updated relevant sections in DESIGN_V2.md and README.md to reflect this change.
- Enhanced error handling in dynamic integration tests to account for both max turns exceeded and goal achieved scenarios without checkpoints.
- Improved formatting in the documentation for better clarity and consistency.
- Added detailed sections on Before/After hooks, including types, execution order, and function signatures.
- Provided examples for common use cases such as database setup/teardown and conditional setup based on metadata.
- Clarified parameters for hook functions, improving guidance for users on implementing hooks in their tests.
- Enhanced overall documentation to support better understanding of the testing framework's capabilities.
- Expanded the input format section to include comprehensive examples organized by scenario, covering various testing cases such as simple text input, assertions, conversation history, and dynamic mode.
- Added scenarios demonstrating the use of file attachments, agent-driven assertions, and options for test configuration.
- Improved clarity on the structure and requirements for JSONL input, ensuring users have clear guidance on how to format their test cases effectively.
- Removed outdated TODO_V2.md file to streamline documentation and focus on the updated README content.
- Updated the introduction to clarify the framework's capabilities, emphasizing support for standard testing, dynamic testing, agent-driven assertions, and CI integration.
- Revised section headers for better organization, changing "Agent Tests" to "Standard Tests" and adding new sections for "Agent-Driven Input" and "Dynamic Mode."
- Included detailed examples for generating test cases using agents and running dynamic tests with simulators.
- Improved clarity on input modes and script test modes, ensuring users understand the requirements and options available for testing.
- Added notes on the importance of the `-n` flag for agent-driven input mode and clarified the distinction between script and agent-driven test case generation.
- Introduced dynamic testing capabilities, allowing for multi-turn conversations with checkpoints through the new `DynamicRunner`.
- Enhanced the `runSingleTest` method to support dynamic mode, including the execution of before/after scripts and detailed output for dynamic test results.
- Added new output methods in `output.go` for dynamic test start, turns, checkpoints, and results, improving console feedback during testing.
- Updated `DESIGN_V2.md` and `TODO_V2.md` to reflect the new dynamic mode features, including simulator configurations and checkpoint definitions.
- Revised the `Case` struct in `types.go` to include fields for dynamic testing, such as `Simulator`, `Checkpoints`, and `MaxTurns`.
- Added support for generating test cases using agents and scripts through the new `LoadFromAgent` and `LoadFromScript` methods in the loader.
- Enhanced the `RunTests` method to handle different input sources, including agent-driven and script-based test case generation.
- Introduced a `--dry-run` flag to allow users to preview generated test cases without executing them.
- Updated relevant documentation in DESIGN_V2.md and TODO_V2.md to reflect the new features and implementation status.
- Added support for agent-driven assertions in the Asserter, allowing validation of responses using specified agents.
- Introduced the `Use` and `Options` fields in the Assertion struct to facilitate agent configuration.
- Enhanced the `evaluateAssertion` method to handle assertions of type "agent".
- Implemented the `assertAgent` method to manage agent interactions and validation logic.
- Updated `script_assert.go` to include the `assertAgentMethod` for JavaScript API integration.
- Revised documentation in DESIGN_V2.md and TODO_V2.md to reflect the new agent-driven assertion capabilities and implementation status.
- Updated DESIGN_V2.md to reflect changes in the handling of before/after scripts, removing the `scripts:` prefix and clarifying their usage in JSONL test cases.
- Enhanced runner.go to integrate global before/after hooks, ensuring they execute correctly before and after test cases.
- Revised types.go to include new fields for before/after scripts in test case and options structures.
- Improved TODO_V2.md to track the implementation progress of before/after script functionality and related tasks.
- Added utility function LoadAgentTestScripts to facilitate loading of test scripts from the agent's src directory.
- Added `UPGRADE_PLAN.md` to .gitignore to exclude the new upgrade plan documentation from version control.
- Revised TODO_V2.md to include a detailed implementation plan for the Agent Test Framework, outlining phases for before/after scripts, agent-driven assertions, and dynamic mode features.
- Improved overall documentation clarity to facilitate understanding of upcoming enhancements and tasks within the framework.
- Updated DESIGN_V2.md to introduce support for `before` and `after` scripts in JSONL test cases, detailing their usage and execution order.
- Added examples for defining and utilizing before/after functions, including global initialization and cleanup processes.
- Revised TODO_V2.md to outline tasks for implementing before/after script functionality, ensuring clarity on remaining development efforts.
- Improved overall documentation to facilitate understanding of the new scripting capabilities in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify output formats for console and JSON, including detailed descriptions for standard, dynamic, and parallel modes.
- Updated the console output sections to provide clearer examples and summaries of test results, enhancing readability and usability.
- Modified TODO_V2.md to reflect the change from JSONL to JSON output format for message counts, ensuring consistency in documentation.
- Improved overall documentation to support better understanding of output handling in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify the `input` field's capabilities, allowing for string, single message, or message array formats for conversation context.
- Updated the test case format to reflect the new `input` structure, ensuring compatibility with existing single-turn tests.
- Enhanced TODO_V2.md to indicate the completion of message history support and outline remaining tasks, including options field support and JSONL output format updates.
- Improved documentation to ensure clarity on the new input handling and its implications for agent-driven testing.
- Updated DESIGN_V2.md to introduce message history support in the Agent Test Framework, allowing tests to simulate multi-turn conversations without complex state management.
- Revised the test case format to include a `messages` field, enabling the passing of full conversation history directly to the agent.
- Enhanced TODO_V2.md to outline tasks for implementing message history support, including updates to the test case parser and output formats.
- Improved documentation on agent-driven assertions and error handling to reflect the new capabilities and ensure clarity for developers.
- Added detailed sections on Static and Dynamic modes in DESIGN_V2.md, outlining their characteristics and execution flows for multi-turn testing.
- Introduced a Quick Reference table for format rules, clarifying the usage of flags and assertions in test cases.
- Updated TODO_V2.md to reflect tasks for implementing mode support, including the addition of checkpoints and handling of order constraints in dynamic testing.
- Improved documentation for error handling in both static and dynamic modes, ensuring clarity on expected behaviors during test execution.
- Corrected references in DESIGN_V2.md to ensure consistent usage of agent identifiers, including updates to input sources and simulator configurations.
- Enhanced the documentation in TODO_V2.md with a summary of format rules for agent testing, clarifying the usage of prefixes for various contexts and options.
- Added tasks related to the dynamic simulator implementation and metadata handling to guide future development efforts.
- Updated the DESIGN_V2.md file to clarify the usage of agent-driven assertions in JSONL test cases, including detailed examples and API specifications.
- Introduced a new section on script testing with agent assertions, outlining the implementation and usage of the `t.assert.Agent()` method.
- Modified the TODO_V2.md file to reflect the addition of JSONL support for agent assertions and outlined tasks for further development in this area.
- Introduced a comprehensive standard agent interface for agent-driven features, including generator, simulator, and validator modes.
- Added support for `context.Options` to pass parameters in test cases, allowing for flexible configuration of agent behavior.
- Updated test case format to include options at both the test and per-turn levels, enhancing customization and control over agent interactions.
- Expanded documentation to detail the usage of options in various agent modes, improving clarity for developers and users.
- Further streamlined the LoadWithRoot function by removing redundant checks and improving the overall readability of the path resolution logic.
- Ensured consistent resolution of the absolute path for the configuration root, enhancing maintainability and clarity in the codebase.
- Added MULTI_TURN_DESIGN.md to .gitignore to prevent tracking of design documents in the test directory.
- Refactored the LoadWithRoot function in config.go for improved readability by adjusting indentation for the absolute path resolution logic.
- Removed unnecessary conditional check in the LoadWithRoot function, streamlining the path resolution logic for the configuration root.
- Ensured that the function consistently resolves the absolute path of the configuration root, improving code clarity and maintainability.
- Introduced support for file attachments in test inputs using the `file://` protocol, allowing images, audio, and documents to be loaded and converted to appropriate formats.
- Updated `ParseInput` and related functions to handle file references, ensuring seamless integration of file content into messages.
- Enhanced error handling and path resolution for file loading, considering both relative paths and the `YAO_ROOT` environment variable.
- Expanded documentation to include examples of file attachments and their usage in test cases, improving clarity for users.
- Implemented GetAuthorizedMap method in the Context struct to return authorized information as a map.
- This method adheres to the AuthorizedProvider interface, facilitating authorization context access for MCP process calls.
- Enhances the ability of MCP tools to receive and utilize authorization data during Process transport interactions.
- Replaced standard log statements with structured logging using the kun/log package for improved log clarity and consistency.
- Updated log messages in the MailReceiver to include contextual information, enhancing the ability to trace issues during email processing.
- This change aims to provide better insights into the email receiver's operations and error handling.
- Introduced functionality to set shared data with authorized information for Process calls in the ScriptRunner.
- Enhanced the executeTestFunction to include error handling for setting share data, improving robustness during script execution.
- This change ensures that authorization context is properly managed when calling JavaScript functions directly, enhancing security and functionality.
- Updated the ScriptRunner to treat both StatusFailed and StatusError as failures, improving failure reporting in test results.
- Modified the fail-fast logic to stop execution on both failure statuses, ensuring quicker feedback during test runs.
- Enhanced error handling in executeTestFunction to recover from panics and provide clearer error messages for JavaScript exceptions, improving test reliability and debugging.
- Expanded the memory isolation tests to cover user, team, chat, context, and key operations, ensuring comprehensive validation of namespace behavior.
- Implemented pattern-based key retrieval and length calculation in the Namespace struct for improved efficiency and flexibility.
- Added assertions to verify that operations on one namespace do not affect others, reinforcing the integrity of memory isolation across different contexts.
- Adjusted comments in the memory leak tests to clarify the rationale behind the threshold increase to 20KB, ensuring better understanding of expected memory behavior.
- Increased the memory growth threshold per iteration in the tests to accommodate higher usage observed in business scenarios, enhancing the accuracy of leak detection.
- Increased the allowed memory growth threshold per iteration in the memory leak tests from 15KB to 20KB to accommodate higher memory usage observed in business scenarios and standard mode operations.
- Updated comments to reflect the rationale behind the new threshold and to clarify expected memory behavior during tests, ensuring better understanding and accuracy in leak detection.
- Replaced all instances of `ctx.Space` with `ctx.Memory.Context` in the context management code, ensuring a more structured approach to handling temporary request-scoped data.
- Updated related test cases to reflect the changes in context memory usage, enhancing the reliability and clarity of tests.
- Removed the deprecated `Space` references and adjusted comments and documentation to align with the new memory management strategy.
- Removed hardcoded collection IDs in `search_auth_integration_test.go` and replaced them with dynamically generated IDs to ensure uniqueness during test runs.
- Simplified the setup and cleanup processes by introducing the `authTestCollections` struct, which manages the lifecycle of test collections.
- Updated test cases to utilize the new collection management approach, enhancing test reliability and reducing potential conflicts during parallel execution.
- Removed references to Badger in the `utils.go` and `utils_test.go` files, replacing it with Xun as the primary data store.
- Updated test cases and comments to reflect the change in store preference, ensuring consistency across the codebase.
- Adjusted the testing guide to provide accurate information regarding store management and fallback options.
- Updated the cleanup process in `cleanupAuthCollections` to include a waiting mechanism for Qdrant to fully process deletions, improving reliability of test setups.
- Removed unnecessary sleep calls and added logging to warn if collections still exist after cleanup, ensuring better visibility during test execution.
- Refactored comments for clarity regarding the cleanup process in both `TestAuthSearchSetup` and `ensureAuthTestData` functions.
- Replaced Badger with Xun as the primary data store across various components, including OAuth, agent memory, and knowledge base stores, enhancing performance and flexibility.
- Updated the configuration files to reflect the new Xun store paths and removed obsolete Badger store files.
- Refactored test cases to utilize the new Xun store, ensuring compatibility and improved test reliability.
- Enhanced test setup functions to initialize the Xun store correctly, streamlining the testing process and ensuring a consistent environment.
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code.
- Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed.
- Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests.
- Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data.
- Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
- Updated multiple test cases in `jsapi_test.go` to utilize `testutils.Prepare` and `testutils.Clean` for better test setup and teardown, ensuring a consistent testing environment.
- Refactored the `parallelAny` and `parallelRace` methods in `search.go` to improve goroutine management and result handling, reducing unnecessary locking and enhancing performance.
- Implemented checks to prevent goroutines from executing after a successful result is found, optimizing resource usage during parallel searches.
- Removed redundant test environment initialization code and replaced it with a streamlined approach using `testutils.Prepare` for better clarity and maintainability.
- Introduced utility functions `ensureAuthTestData` and `createAuthTestData` to manage the setup of test collections and documents, ensuring that necessary data is available for tests.
- Updated multiple test cases to utilize the new data initialization methods, improving test reliability and reducing setup complexity.
- Renamed functions for consistency and clarity, changing `buildDBAuthWheres` to `BuildDBAuthWheres` and `filterKBCollectionsByAuth` to `FilterKBCollectionsByAuth`.
- Enhanced test cases to utilize the updated function names, ensuring proper authorization checks in the search functionality.
- Improved test environment initialization to streamline setup processes and ensure robust testing of authorization logic.
- Verified that search results adhere to authorization constraints, ensuring only accessible collections are queried based on user permissions.
- Enhanced the KB search handler to utilize the KB API for executing search queries, improving search accuracy and performance.
- Implemented authorization checks for collections in the search requests, ensuring only accessible collections are queried.
- Updated the search request structure to include metadata filtering capabilities, allowing for more refined search results.
- Refactored unit tests to validate new search functionalities, including threshold handling and collection initialization checks, ensuring robust test coverage.
- Adjusted the Makefile to streamline test coverage reporting and updated GitHub Actions workflows to include Codecov integration for better visibility on test coverage metrics.
- Updated the Dockerfiles and GitHub Actions workflows to reference version 1.0.0 of the Yao build environment and Go, ensuring compatibility with the latest features and improvements.
- Adjusted the Go version in the macOS workflow and updated the Dockerfile to install Go 1.25.0, enhancing the build process.
- Changed the Go version in multiple workflow files from 1.24 to 1.25, ensuring compatibility with the latest features and improvements in the Go programming language.
- Introduced a new unit test target for KB tests in the Makefile, allowing for dedicated testing of the KB module.
- Updated the test folder selection logic to exclude additional AI-related components, ensuring focused testing.
- Enhanced the GitHub Actions workflows to include KB tests, setting up necessary services like Qdrant, Neo4j, and MongoDB for a comprehensive testing environment.
- Refactored search test functions to improve data existence checks and streamline test setup processes, enhancing test reliability and maintainability.
- Updated the `GetCollection` method to first read from the database for existence and permissions, improving data integrity.
- Merged metadata from GraphRag into the result, ensuring backward compatibility and enhanced data representation.
- Refactored the `CollectionExists` method to check both the database and GraphRag for consistency, logging any mismatches for debugging purposes.
- Introduced new types and structures for search operations, including `SearchMode`, `Query`, and `SearchResult`, to support advanced search functionalities.
- Modified the `createTestContext` function to use `context.Background()` for improved context management in tests.
- Updated the URL in the `TestAddURL` function to point to the correct Yao Agent Caller resource, ensuring accurate test assertions for added URLs.
- Changed the query in the `TestSerpAPIProviderWithAssistantConfig` from "Yao App Engine" to "golang programming language" to align with updated test expectations.
- Updated the expected result assertion to match the new query, ensuring the test accurately verifies the functionality of the search handler.
- Modified the 'Type' field in the serpAPIKnowledge struct to use an interface{}, enabling it to accept either a string or an object based on the query context.
- This change enhances the flexibility of the knowledge graph data representation in the search handler.
- Improved the `Job` struct to support dynamic function execution with the `AddFunc` method, allowing for flexible job management.
- Enhanced the `Goroutine` struct with the `ExecuteFunc` method to manage function execution, including robust error handling and context management.
- Updated unit tests for `AddFunc` to ensure proper function registration and execution, including memory cleanup verification.
- Revised documentation to reflect the new function execution capabilities within the job system.
- Introduced `AddFunc` method to the `Job` struct for adding Go functions as job executions, allowing for dynamic execution of functions with specified arguments.
- Enhanced internal execution handling to register functions in a global registry, ensuring proper cleanup after execution.
- Implemented `ExecuteFunc` method in the `Goroutine` struct to handle the execution of registered functions, including error handling and context management.
- Added comprehensive unit tests for `AddFunc`, verifying function registration, execution, and memory cleanup post-execution.
- Updated related documentation to reflect the new functionality and usage patterns for adding and executing Go functions within the job system.
