- Added `UPGRADE_PLAN.md` to .gitignore to exclude the new upgrade plan documentation from version control.
- Revised TODO_V2.md to include a detailed implementation plan for the Agent Test Framework, outlining phases for before/after scripts, agent-driven assertions, and dynamic mode features.
- Improved overall documentation clarity to facilitate understanding of upcoming enhancements and tasks within the framework.
- Updated DESIGN_V2.md to introduce support for `before` and `after` scripts in JSONL test cases, detailing their usage and execution order.
- Added examples for defining and utilizing before/after functions, including global initialization and cleanup processes.
- Revised TODO_V2.md to outline tasks for implementing before/after script functionality, ensuring clarity on remaining development efforts.
- Improved overall documentation to facilitate understanding of the new scripting capabilities in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify output formats for console and JSON, including detailed descriptions for standard, dynamic, and parallel modes.
- Updated the console output sections to provide clearer examples and summaries of test results, enhancing readability and usability.
- Modified TODO_V2.md to reflect the change from JSONL to JSON output format for message counts, ensuring consistency in documentation.
- Improved overall documentation to support better understanding of output handling in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify the `input` field's capabilities, allowing for string, single message, or message array formats for conversation context.
- Updated the test case format to reflect the new `input` structure, ensuring compatibility with existing single-turn tests.
- Enhanced TODO_V2.md to indicate the completion of message history support and outline remaining tasks, including options field support and JSONL output format updates.
- Improved documentation to ensure clarity on the new input handling and its implications for agent-driven testing.
- Updated DESIGN_V2.md to introduce message history support in the Agent Test Framework, allowing tests to simulate multi-turn conversations without complex state management.
- Revised the test case format to include a `messages` field, enabling the passing of full conversation history directly to the agent.
- Enhanced TODO_V2.md to outline tasks for implementing message history support, including updates to the test case parser and output formats.
- Improved documentation on agent-driven assertions and error handling to reflect the new capabilities and ensure clarity for developers.
- Added detailed sections on Static and Dynamic modes in DESIGN_V2.md, outlining their characteristics and execution flows for multi-turn testing.
- Introduced a Quick Reference table for format rules, clarifying the usage of flags and assertions in test cases.
- Updated TODO_V2.md to reflect tasks for implementing mode support, including the addition of checkpoints and handling of order constraints in dynamic testing.
- Improved documentation for error handling in both static and dynamic modes, ensuring clarity on expected behaviors during test execution.
- Corrected references in DESIGN_V2.md to ensure consistent usage of agent identifiers, including updates to input sources and simulator configurations.
- Enhanced the documentation in TODO_V2.md with a summary of format rules for agent testing, clarifying the usage of prefixes for various contexts and options.
- Added tasks related to the dynamic simulator implementation and metadata handling to guide future development efforts.
- Updated the DESIGN_V2.md file to clarify the usage of agent-driven assertions in JSONL test cases, including detailed examples and API specifications.
- Introduced a new section on script testing with agent assertions, outlining the implementation and usage of the `t.assert.Agent()` method.
- Modified the TODO_V2.md file to reflect the addition of JSONL support for agent assertions and outlined tasks for further development in this area.
- Introduced a comprehensive standard agent interface for agent-driven features, including generator, simulator, and validator modes.
- Added support for `context.Options` to pass parameters in test cases, allowing for flexible configuration of agent behavior.
- Updated test case format to include options at both the test and per-turn levels, enhancing customization and control over agent interactions.
- Expanded documentation to detail the usage of options in various agent modes, improving clarity for developers and users.
- Further streamlined the LoadWithRoot function by removing redundant checks and improving the overall readability of the path resolution logic.
- Ensured consistent resolution of the absolute path for the configuration root, enhancing maintainability and clarity in the codebase.
- Introduced support for file attachments in test inputs using the `file://` protocol, allowing images, audio, and documents to be loaded and converted to appropriate formats.
- Updated `ParseInput` and related functions to handle file references, ensuring seamless integration of file content into messages.
- Enhanced error handling and path resolution for file loading, considering both relative paths and the `YAO_ROOT` environment variable.
- Expanded documentation to include examples of file attachments and their usage in test cases, improving clarity for users.
- Implemented GetAuthorizedMap method in the Context struct to return authorized information as a map.
- This method adheres to the AuthorizedProvider interface, facilitating authorization context access for MCP process calls.
- Enhances the ability of MCP tools to receive and utilize authorization data during Process transport interactions.
- Introduced functionality to set shared data with authorized information for Process calls in the ScriptRunner.
- Enhanced the executeTestFunction to include error handling for setting share data, improving robustness during script execution.
- This change ensures that authorization context is properly managed when calling JavaScript functions directly, enhancing security and functionality.
- Updated the ScriptRunner to treat both StatusFailed and StatusError as failures, improving failure reporting in test results.
- Modified the fail-fast logic to stop execution on both failure statuses, ensuring quicker feedback during test runs.
- Enhanced error handling in executeTestFunction to recover from panics and provide clearer error messages for JavaScript exceptions, improving test reliability and debugging.
- Expanded the memory isolation tests to cover user, team, chat, context, and key operations, ensuring comprehensive validation of namespace behavior.
- Implemented pattern-based key retrieval and length calculation in the Namespace struct for improved efficiency and flexibility.
- Added assertions to verify that operations on one namespace do not affect others, reinforcing the integrity of memory isolation across different contexts.
- Increased the allowed memory growth threshold per iteration in the memory leak tests from 15KB to 20KB to accommodate higher memory usage observed in business scenarios and standard mode operations.
- Updated comments to reflect the rationale behind the new threshold and to clarify expected memory behavior during tests, ensuring better understanding and accuracy in leak detection.
- Replaced all instances of `ctx.Space` with `ctx.Memory.Context` in the context management code, ensuring a more structured approach to handling temporary request-scoped data.
- Updated related test cases to reflect the changes in context memory usage, enhancing the reliability and clarity of tests.
- Removed the deprecated `Space` references and adjusted comments and documentation to align with the new memory management strategy.
- Removed hardcoded collection IDs in `search_auth_integration_test.go` and replaced them with dynamically generated IDs to ensure uniqueness during test runs.
- Simplified the setup and cleanup processes by introducing the `authTestCollections` struct, which manages the lifecycle of test collections.
- Updated test cases to utilize the new collection management approach, enhancing test reliability and reducing potential conflicts during parallel execution.
- Updated the cleanup process in `cleanupAuthCollections` to include a waiting mechanism for Qdrant to fully process deletions, improving reliability of test setups.
- Removed unnecessary sleep calls and added logging to warn if collections still exist after cleanup, ensuring better visibility during test execution.
- Refactored comments for clarity regarding the cleanup process in both `TestAuthSearchSetup` and `ensureAuthTestData` functions.
- Introduced a new script testing mode to allow testing of agent handler scripts (hooks, tools, etc.) using a Go-like interface, enabling better unit testing of TypeScript/JavaScript code.
- Enhanced the `LoadScripts` function to skip test files during script loading, ensuring only relevant scripts are processed.
- Refactored the test context creation to support custom context configurations via a JSON file, allowing for flexible authorization and metadata management during tests.
- Updated the test runner to handle script tests, including the ability to filter tests using regex patterns and manage custom context data.
- Improved documentation to include details on script testing usage, input formats, and available assertions, enhancing developer experience and clarity.
- Updated multiple test cases in `jsapi_test.go` to utilize `testutils.Prepare` and `testutils.Clean` for better test setup and teardown, ensuring a consistent testing environment.
- Refactored the `parallelAny` and `parallelRace` methods in `search.go` to improve goroutine management and result handling, reducing unnecessary locking and enhancing performance.
- Implemented checks to prevent goroutines from executing after a successful result is found, optimizing resource usage during parallel searches.
- Removed redundant test environment initialization code and replaced it with a streamlined approach using `testutils.Prepare` for better clarity and maintainability.
- Introduced utility functions `ensureAuthTestData` and `createAuthTestData` to manage the setup of test collections and documents, ensuring that necessary data is available for tests.
- Updated multiple test cases to utilize the new data initialization methods, improving test reliability and reducing setup complexity.
- Renamed functions for consistency and clarity, changing `buildDBAuthWheres` to `BuildDBAuthWheres` and `filterKBCollectionsByAuth` to `FilterKBCollectionsByAuth`.
- Enhanced test cases to utilize the updated function names, ensuring proper authorization checks in the search functionality.
- Improved test environment initialization to streamline setup processes and ensure robust testing of authorization logic.
- Verified that search results adhere to authorization constraints, ensuring only accessible collections are queried based on user permissions.
- Enhanced the KB search handler to utilize the KB API for executing search queries, improving search accuracy and performance.
- Implemented authorization checks for collections in the search requests, ensuring only accessible collections are queried.
- Updated the search request structure to include metadata filtering capabilities, allowing for more refined search results.
- Refactored unit tests to validate new search functionalities, including threshold handling and collection initialization checks, ensuring robust test coverage.
- Adjusted the Makefile to streamline test coverage reporting and updated GitHub Actions workflows to include Codecov integration for better visibility on test coverage metrics.
- Modified the `createTestContext` function to use `context.Background()` for improved context management in tests.
- Updated the URL in the `TestAddURL` function to point to the correct Yao Agent Caller resource, ensuring accurate test assertions for added URLs.
- Changed the query in the `TestSerpAPIProviderWithAssistantConfig` from "Yao App Engine" to "golang programming language" to align with updated test expectations.
- Updated the expected result assertion to match the new query, ensuring the test accurately verifies the functionality of the search handler.
- Modified the 'Type' field in the serpAPIKnowledge struct to use an interface{}, enabling it to accept either a string or an object based on the query context.
- This change enhances the flexibility of the knowledge graph data representation in the search handler.
- Added a `Results` field to the `SearchExecutionResult` struct to store raw search results, facilitating the extraction of DSL from database searches.
- Implemented logic in the `saveSearch` method to extract and store the first DSL from DB search results, improving data retention for search operations.
- Introduced a `dslToMap` method in the DB handler to convert QueryDSL to a map format for storage, enhancing the flexibility of result handling.
- Updated the `Result` struct to include a `DSL` field for generated QueryDSL, ensuring comprehensive data representation for database search results.
- Updated the search handling to incorporate keyword extraction with weights, improving the relevance of search results.
- Refactored the `shouldAutoSearch` method to return a `SearchIntent` struct, allowing for more nuanced control over search execution based on context.
- Enhanced the `buildSearchRequests` function to utilize extracted keywords, optimizing search queries based on user input.
- Improved the handling of search types and conditions, ensuring that the system can dynamically adjust search behavior based on intent and configuration.
- Updated documentation and prompts to reflect changes in keyword extraction and search intent classification, providing clearer guidelines for usage.
- Updated the AgentReporter to handle *context.Response directly instead of using type assertions, simplifying the response extraction process.
- Enhanced the extractContent method to prioritize accessing the Next field and completion content, improving robustness in data retrieval.
- Improved code readability by streamlining the response handling logic, aligning with recent refactorings in the agent's response processing.
- Updated the agent's Stream and response processing methods to return and handle *context.Response directly, eliminating the need for type assertions.
- Simplified test cases by removing unnecessary type conversions and directly accessing response fields.
- Enhanced the extraction of data from Next hook responses, ensuring more robust handling of custom data structures.
- Improved overall code readability and maintainability by streamlining response handling logic across various components.
- Updated the buildRequestMessage function to construct a JSON object for structured communication with the agent, replacing the previous string formatting approach.
- Enhanced the handling of extra parameters, scenarios, and retry context by incorporating them into the JSON structure.
- Improved the parseResult function to extract additional fields such as explain and warnings from the response, ensuring comprehensive result handling.
- Streamlined the code for better readability and maintainability while preserving existing functionality.
- Introduced new test cases to validate the behavior of the search handler when querying with nonexistent models, ensuring it returns appropriate error messages without panicking.
- Added a test for scenarios where only some of the requested models exist, confirming that the search can still succeed with valid models while handling errors gracefully.
- Updated the search handler to improve error handling by checking for model existence before proceeding with the search, enhancing robustness in search operations.
- Reorganized import statements in test files to improve clarity and consistency.
- Updated comments in test cases to provide more detailed descriptions of the test environment initialization process.
- Enhanced the `Prepare` function in the test utilities to include registration of the default query engine, ensuring proper setup for database searches.
- Improved error handling during the loading of the knowledge base and query engine, enhancing robustness in test setups.
- Refactored the Search method to support context-aware execution, allowing handlers to utilize context when performing searches.
- Introduced a new SearchWithContext method in the handler interface to facilitate context-based search operations.
- Updated the DB handler to implement the context-aware search, ensuring proper QueryDSL generation and execution.
- Enhanced test cases to validate the new context requirements and scenarios for database searches, improving error handling and robustness.
- Added scenario type support for QueryDSL generation, allowing for more complex query handling.
- Updated documentation to reflect the new context handling and scenario features in search operations.
- Added support for a new `Metadata` field in the `Options` struct to allow passing custom data to hooks, enhancing flexibility in context management.
- Updated the `ToMap` and `OptionsFromMap` methods to include serialization and deserialization of the `Metadata` field.
- Enhanced the test case structure to include an `Options` field, allowing for per-test-case configuration, including metadata and skip options.
- Updated documentation to reflect the new `options` and `metadata` fields, providing clear examples for users on how to utilize these features in test cases.
- Updated the keyword extraction and QueryDSL generation processes to require a context parameter, enhancing the robustness of the extraction methods.
- Replaced the previous frequency-based extraction with a system agent approach, utilizing the __yao.keyword and __yao.querydsl agents for improved accuracy and context awareness.
- Removed obsolete builtin extraction implementations and tests, streamlining the codebase.
- Enhanced test cases to validate the new context requirements, ensuring proper error handling when context is not provided.
- Updated documentation to reflect changes in the extraction methods and their dependencies on context.
- Updated the search intent classification prompt for the Need Search agent to provide clearer instructions and rules for classifying user queries.
- Revised the output format to specify JSON structure requirements, ensuring consistency in responses.
- Expanded classification rules to include additional categories and examples, improving the agent's ability to accurately determine the need for external searches.
- Enhanced clarity in the prompt content to facilitate better understanding and implementation by users.
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation.
- Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs.
- Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter.
- Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
- Added a `noop` check in multiple logging methods (`LLMComplete`, `ToolStart`, `ToolComplete`, `HookStart`, `HookComplete`, and `HistoryLoad`) to prevent logging when the logger is in no-operation mode.
- Improved command handling in `root.go` by removing minimum argument requirements for commands and providing help output when no arguments are given.
- Introduced an `agent` command for better organization of agent-related functionalities in the CLI.
- Implemented automatic detection of the application root directory in `run.go` to streamline the application startup process.
- Cleaned up debug print statements in `config.go` to reduce clutter in the output.
- Changed the expected names of system agents in the load test to reflect recent updates: "Keyword Extraction" to "Keyword Extractor," "QueryDSL Generator" to "Query Builder," and "Need Search" to "Reference Checker."
- Ensured that test assertions align with the latest naming conventions for improved clarity and consistency in the assistant's functionality.
- Renamed several assistant packages for clarity, including "Entity Extraction" to "Entity Extractor" and "Keyword Extraction" to "Keyword Extractor."
- Revised descriptions for various assistants to enhance understanding of their functionalities, such as changing "Extract keywords from text content" to "Extract search keywords."
- Added a "uses" field with "search" set to "disabled" in the configuration of each assistant, standardizing their setup.
- Updated the "Prompt Optimizer" description to "Optimize prompts for better results" and modified the "QueryDSL Generator" to "Query Builder" for improved clarity.
- Ensured consistent naming conventions and descriptions across all assistant packages to enhance user experience and documentation clarity.
- Updated the `shouldAutoSearch` method to include additional parameters for improved intent detection, allowing for better decision-making on whether to execute auto search.
- Introduced a new `checkSearchIntent` method to utilize the `__yao.needsearch` agent for determining the necessity of a search based on user input.
- Implemented a `ClearExcept` method in the cache to selectively clear non-system agents while preserving essential system agents during cache management.
- Updated the `LoadBuiltIn` function to maintain system agents in the cache, ensuring they remain available for use.
- Enhanced test coverage for loading system agents and validating search intent detection, ensuring robustness in the assistant's search capabilities.
- Revised localization files to include new messages for search intent feedback, improving user experience during search operations.
- Added configuration support for system agents in the assistant initialization process, allowing for custom connectors for agents like __yao.keyword and __yao.querydsl.
- Implemented the loading mechanism for system agents from bindata, ensuring that essential agents are available during runtime.
- Updated the LoadBuiltIn function to exclude system agents from being removed, enhancing the management of built-in and system agents.
- Enhanced test coverage by introducing tests for loading system agents, verifying their presence and correctness in the cache.
- Updated documentation to reflect the new system agents configuration and loading processes.
- Added retry logic to the Generate method in both AgentProvider and MCPProvider to handle failures in QueryDSL generation.
- Integrated lint validation to ensure generated QueryDSL meets required standards, with detailed error reporting for invalid DSL.
- Enhanced test coverage by introducing new tests for retry behavior in both agent and MCP contexts, ensuring robustness against lint failures.
- Updated documentation to reflect changes in QueryDSL generation processes and error handling mechanisms.
- Updated the executeAutoSearch method to improve the handling of search results, ensuring better data capture and processing.
- Enhanced the Search type to include additional metadata for improved debugging and user feedback.
- Revised related tests to align with the new search execution logic and ensure comprehensive coverage of changes.
- Updated documentation to reflect modifications in search result handling and execution processes.
- Updated the TestGPT5Vision function to support various content types in responses, including strings and slices of ContentPart.
- Implemented logic to concatenate text from multimodal responses, improving the robustness of image description handling.
- Added logging for cases where content is nil or of unexpected types, enhancing test feedback and debugging capabilities.
- Updated the CitationGenerator to produce simple integer IDs instead of formatted strings, improving clarity and consistency in citation references.
- Enhanced the executeAutoSearch method to save both successful and failed search results, capturing detailed execution data for better traceability.
- Introduced a new SearchExecutionResult type to structure search result data, including query, keywords, configuration, duration, and error information.
- Updated related tests to reflect changes in citation ID format and ensure proper functionality of the new storage mechanisms.
- Revised documentation to clarify the new citation format and search result handling processes.
- Introduced a new `Search` type to store intermediate processing results, including extracted keywords, entities, relations, and generated QueryDSL for improved debugging and citation support.
- Updated the `executeAutoSearch` method to populate the new `Search` structure, ensuring all relevant data is captured during search execution.
- Implemented methods for saving and retrieving search records in MongoDB and Redis, enhancing data persistence across sessions.
- Revised localization files to include new keys for search-related messages, improving user experience.
- Updated DESIGN.md to reflect changes in the search result structure and data flow, ensuring comprehensive documentation of the new features.
- Implemented loading and result messaging in the executeAutoSearch method to improve user experience during search operations.
- Added methods to send loading, result, and completion messages, providing real-time feedback to users.
- Integrated trace node creation and completion for search operations, enhancing transparency and debugging capabilities.
- Updated localization files to include new messages for search status updates in both English and Chinese.
- Revised DESIGN.md to document the new output flow and trace integration for search operations.
