- Added a `Results` field to the `SearchExecutionResult` struct to store raw search results, facilitating the extraction of DSL from database searches.
- Implemented logic in the `saveSearch` method to extract and store the first DSL from DB search results, improving data retention for search operations.
- Introduced a `dslToMap` method in the DB handler to convert QueryDSL to a map format for storage, enhancing the flexibility of result handling.
- Updated the `Result` struct to include a `DSL` field for generated QueryDSL, ensuring comprehensive data representation for database search results.
- Updated the search handling to incorporate keyword extraction with weights, improving the relevance of search results.
- Refactored the `shouldAutoSearch` method to return a `SearchIntent` struct, allowing for more nuanced control over search execution based on context.
- Enhanced the `buildSearchRequests` function to utilize extracted keywords, optimizing search queries based on user input.
- Improved the handling of search types and conditions, ensuring that the system can dynamically adjust search behavior based on intent and configuration.
- Updated documentation and prompts to reflect changes in keyword extraction and search intent classification, providing clearer guidelines for usage.
- Updated the AgentReporter to handle *context.Response directly instead of using type assertions, simplifying the response extraction process.
- Enhanced the extractContent method to prioritize accessing the Next field and completion content, improving robustness in data retrieval.
- Improved code readability by streamlining the response handling logic, aligning with recent refactorings in the agent's response processing.
- Updated the agent's Stream and response processing methods to return and handle *context.Response directly, eliminating the need for type assertions.
- Simplified test cases by removing unnecessary type conversions and directly accessing response fields.
- Enhanced the extraction of data from Next hook responses, ensuring more robust handling of custom data structures.
- Improved overall code readability and maintainability by streamlining response handling logic across various components.
- Updated the buildRequestMessage function to construct a JSON object for structured communication with the agent, replacing the previous string formatting approach.
- Enhanced the handling of extra parameters, scenarios, and retry context by incorporating them into the JSON structure.
- Improved the parseResult function to extract additional fields such as explain and warnings from the response, ensuring comprehensive result handling.
- Streamlined the code for better readability and maintainability while preserving existing functionality.
- Introduced new test cases to validate the behavior of the search handler when querying with nonexistent models, ensuring it returns appropriate error messages without panicking.
- Added a test for scenarios where only some of the requested models exist, confirming that the search can still succeed with valid models while handling errors gracefully.
- Updated the search handler to improve error handling by checking for model existence before proceeding with the search, enhancing robustness in search operations.
- Reorganized import statements in test files to improve clarity and consistency.
- Updated comments in test cases to provide more detailed descriptions of the test environment initialization process.
- Enhanced the `Prepare` function in the test utilities to include registration of the default query engine, ensuring proper setup for database searches.
- Improved error handling during the loading of the knowledge base and query engine, enhancing robustness in test setups.
- Bumped the version of the expr-lang/expr package from v1.17.3 to v1.17.7 to incorporate the latest features and fixes.
- Updated the go.sum file to reflect the new dependency version and its associated checksums.
- Refactored the Search method to support context-aware execution, allowing handlers to utilize context when performing searches.
- Introduced a new SearchWithContext method in the handler interface to facilitate context-based search operations.
- Updated the DB handler to implement the context-aware search, ensuring proper QueryDSL generation and execution.
- Enhanced test cases to validate the new context requirements and scenarios for database searches, improving error handling and robustness.
- Added scenario type support for QueryDSL generation, allowing for more complex query handling.
- Updated documentation to reflect the new context handling and scenario features in search operations.
- Added support for a new `Metadata` field in the `Options` struct to allow passing custom data to hooks, enhancing flexibility in context management.
- Updated the `ToMap` and `OptionsFromMap` methods to include serialization and deserialization of the `Metadata` field.
- Enhanced the test case structure to include an `Options` field, allowing for per-test-case configuration, including metadata and skip options.
- Updated documentation to reflect the new `options` and `metadata` fields, providing clear examples for users on how to utilize these features in test cases.
- Updated the keyword extraction and QueryDSL generation processes to require a context parameter, enhancing the robustness of the extraction methods.
- Replaced the previous frequency-based extraction with a system agent approach, utilizing the __yao.keyword and __yao.querydsl agents for improved accuracy and context awareness.
- Removed obsolete builtin extraction implementations and tests, streamlining the codebase.
- Enhanced test cases to validate the new context requirements, ensuring proper error handling when context is not provided.
- Updated documentation to reflect changes in the extraction methods and their dependencies on context.
- Updated the search intent classification prompt for the Need Search agent to provide clearer instructions and rules for classifying user queries.
- Revised the output format to specify JSON structure requirements, ensuring consistency in responses.
- Expanded classification rules to include additional categories and examples, improving the agent's ability to accurately determine the need for external searches.
- Enhanced clarity in the prompt content to facilitate better understanding and implementation by users.
- Introduced an `assert` field in the test case structure to allow for custom assertion rules, providing flexibility in output validation.
- Defined various assertion types, including `equals`, `contains`, `not_contains`, `json_path`, `regex`, and `script`, to cater to different validation needs.
- Updated the test runner to utilize the new assertion mechanism, replacing the previous expected output validation with a more robust asserter.
- Enhanced documentation in DESIGN.md to include detailed examples and explanations of the new assertion capabilities, improving clarity for users.
- Added a `noop` check in multiple logging methods (`LLMComplete`, `ToolStart`, `ToolComplete`, `HookStart`, `HookComplete`, and `HistoryLoad`) to prevent logging when the logger is in no-operation mode.
- Improved command handling in `root.go` by removing minimum argument requirements for commands and providing help output when no arguments are given.
- Introduced an `agent` command for better organization of agent-related functionalities in the CLI.
- Implemented automatic detection of the application root directory in `run.go` to streamline the application startup process.
- Cleaned up debug print statements in `config.go` to reduce clutter in the output.
- Replaced `json.Parse` with `text.ExtractJSON` in the `Next` function of both the `keyword` and `needsearch` assistants for improved fault-tolerant JSON extraction from LLM output.
- Simplified content handling by removing unnecessary markdown code block processing, enhancing clarity and efficiency in keyword extraction.
- Updated comments to reflect the changes in the extraction method, ensuring better understanding of the functionality.
- Changed the expected names of system agents in the load test to reflect recent updates: "Keyword Extraction" to "Keyword Extractor," "QueryDSL Generator" to "Query Builder," and "Need Search" to "Reference Checker."
- Ensured that test assertions align with the latest naming conventions for improved clarity and consistency in the assistant's functionality.
- Renamed several assistant packages for clarity, including "Entity Extraction" to "Entity Extractor" and "Keyword Extraction" to "Keyword Extractor."
- Revised descriptions for various assistants to enhance understanding of their functionalities, such as changing "Extract keywords from text content" to "Extract search keywords."
- Added a "uses" field with "search" set to "disabled" in the configuration of each assistant, standardizing their setup.
- Updated the "Prompt Optimizer" description to "Optimize prompts for better results" and modified the "QueryDSL Generator" to "Query Builder" for improved clarity.
- Ensured consistent naming conventions and descriptions across all assistant packages to enhance user experience and documentation clarity.
- Updated the `shouldAutoSearch` method to include additional parameters for improved intent detection, allowing for better decision-making on whether to execute auto search.
- Introduced a new `checkSearchIntent` method to utilize the `__yao.needsearch` agent for determining the necessity of a search based on user input.
- Implemented a `ClearExcept` method in the cache to selectively clear non-system agents while preserving essential system agents during cache management.
- Updated the `LoadBuiltIn` function to maintain system agents in the cache, ensuring they remain available for use.
- Enhanced test coverage for loading system agents and validating search intent detection, ensuring robustness in the assistant's search capabilities.
- Revised localization files to include new messages for search intent feedback, improving user experience during search operations.
- Added configuration support for system agents in the assistant initialization process, allowing for custom connectors for agents like __yao.keyword and __yao.querydsl.
- Implemented the loading mechanism for system agents from bindata, ensuring that essential agents are available during runtime.
- Updated the LoadBuiltIn function to exclude system agents from being removed, enhancing the management of built-in and system agents.
- Enhanced test coverage by introducing tests for loading system agents, verifying their presence and correctness in the cache.
- Updated documentation to reflect the new system agents configuration and loading processes.
- Added retry logic to the Generate method in both AgentProvider and MCPProvider to handle failures in QueryDSL generation.
- Integrated lint validation to ensure generated QueryDSL meets required standards, with detailed error reporting for invalid DSL.
- Enhanced test coverage by introducing new tests for retry behavior in both agent and MCP contexts, ensuring robustness against lint failures.
- Updated documentation to reflect changes in QueryDSL generation processes and error handling mechanisms.
- Updated the executeAutoSearch method to improve the handling of search results, ensuring better data capture and processing.
- Enhanced the Search type to include additional metadata for improved debugging and user feedback.
- Revised related tests to align with the new search execution logic and ensure comprehensive coverage of changes.
- Updated documentation to reflect modifications in search result handling and execution processes.
- Updated the TestGPT5Vision function to support various content types in responses, including strings and slices of ContentPart.
- Implemented logic to concatenate text from multimodal responses, improving the robustness of image description handling.
- Added logging for cases where content is nil or of unexpected types, enhancing test feedback and debugging capabilities.
- Updated the CitationGenerator to produce simple integer IDs instead of formatted strings, improving clarity and consistency in citation references.
- Enhanced the executeAutoSearch method to save both successful and failed search results, capturing detailed execution data for better traceability.
- Introduced a new SearchExecutionResult type to structure search result data, including query, keywords, configuration, duration, and error information.
- Updated related tests to reflect changes in citation ID format and ensure proper functionality of the new storage mechanisms.
- Revised documentation to clarify the new citation format and search result handling processes.
- Introduced a new `Search` type to store intermediate processing results, including extracted keywords, entities, relations, and generated QueryDSL for improved debugging and citation support.
- Updated the `executeAutoSearch` method to populate the new `Search` structure, ensuring all relevant data is captured during search execution.
- Implemented methods for saving and retrieving search records in MongoDB and Redis, enhancing data persistence across sessions.
- Revised localization files to include new keys for search-related messages, improving user experience.
- Updated DESIGN.md to reflect changes in the search result structure and data flow, ensuring comprehensive documentation of the new features.
- Implemented loading and result messaging in the executeAutoSearch method to improve user experience during search operations.
- Added methods to send loading, result, and completion messages, providing real-time feedback to users.
- Integrated trace node creation and completion for search operations, enhancing transparency and debugging capabilities.
- Updated localization files to include new messages for search status updates in both English and Chinese.
- Revised DESIGN.md to document the new output flow and trace integration for search operations.
- Renamed the ID generation function to `aigcID` for clarity and added a new function to parse AIGC IDs from file paths.
- Implemented special handling for `.ai.yml` and `.ai.yaml` extensions to correctly format the AIGC ID by removing the "_ai" suffix, improving ID accuracy in the loading process.
- Added the `OPENAI_API_KEY` environment variable to both `pr-test.yml` and `unit-test.yml` workflows for improved API access.
- Introduced a step to set up the Apple Private Key in both workflows, enhancing security for Apple-related operations.
- Integrated Redis service setup in both workflows to support caching and improve test performance.
- Added a MongoDB service to both `pr-test.yml` and `unit-test.yml` workflows to support database testing.
- Configured MongoDB with necessary environment variables for root credentials and database name, improving the test environment setup.
- Introduced a MongoDB service in both `pr-test.yml` and `unit-test.yml` workflows to facilitate database testing.
- Configured MongoDB with environment variables for root username, password, and database name, enhancing test environment setup.
- Updated `.gitignore` to exclude job logs for search agents, ensuring cleaner repository management.
- Increased the timeout for AI unit tests from 20 minutes to 20 minutes and for memory leak tests from 60 seconds to 5 minutes, improving test execution reliability.
- Adjusted test commands to ensure proper handling of long-running tests, enhancing overall testing efficiency.
- Introduced a new step in both `pr-test.yml` and `unit-test.yml` workflows to set up environment variables for database configuration.
- Enhanced the environment setup to conditionally configure database connection strings based on the selected database driver, improving test reliability and flexibility.
- Created necessary directories for SQLite database storage when not using MySQL, ensuring proper environment preparation for tests.
- Added additional failure checks in the Makefile for unit tests to improve error handling.
- Updated GitHub Actions workflows to include a new benchmark and memory leak testing job, enhancing performance validation.
- Modified environment setup in workflows to ensure correct database paths for SQLite, improving test reliability.
- Streamlined the core test execution process by removing redundant commands, focusing on unit tests only.
- Added separate unit test targets for core tests and AI-related tests in the Makefile, allowing for more granular testing.
- Updated GitHub Actions workflows to include dedicated jobs for running AI tests with SQLite and core tests, enhancing CI capabilities.
- Modified existing test commands to ensure proper execution of unit tests while excluding AI-related tests where necessary.
- Updated test paths in the agent's search tests to reflect changes in the test assistant being used, ensuring accurate test execution.
- Updated the executeAutoSearch method to include an optional parameter for Skip.Keyword, allowing for conditional keyword extraction during web searches.
- Implemented logic to extract keywords only when configured and not skipped, improving search query optimization.
- Modified the Assistant's Stream method to pass options to executeAutoSearch, ensuring seamless integration of the new functionality.
- Updated DESIGN.md to document the changes in keyword extraction logic and its impact on the search process.
- Added functionality to the Assistant's Stream method to execute auto search if enabled, enhancing the search capabilities based on user configuration.
- Introduced helper methods for determining auto search eligibility, executing the search, and injecting search context into messages.
- Updated DESIGN.md to reflect the new auto search logic, including detailed descriptions of the new methods and their integration points within the search process.
- Introduced a new search object in the context to expose search methods (Web, KB, DB, All, Any, Race) for JavaScript integration.
- Implemented individual search methods with argument validation and error handling, improving the robustness of the API.
- Updated the JSAPI implementation to utilize the new search object, ensuring seamless interaction with the search functionalities.
- Enhanced documentation in DESIGN.md to reflect the new V8 binding methods and their usage, providing clear guidance for developers.
- Refactored the SearchAPI interface to replace the Parallel method with All, Any, and Race methods, inspired by JavaScript Promise patterns.
- Updated the Searcher struct to implement these new parallel search methods, improving flexibility and performance in executing multiple searches.
- Revised the JSAPI implementation to support the new parallel search methods, ensuring consistency across the API.
- Enhanced documentation in DESIGN.md to detail the new parallel search functionalities and provide usage examples, clarifying their behavior and expected outcomes.
- Introduced a new TODO.md file to outline the implementation plan and progress for the search module.
- Updated DESIGN.md to reflect changes in the directory structure and clarify the roles of various components, including the new Handler + Registry pattern for reranking and keyword extraction.
- Refactored the Searcher struct to utilize a direct reference to the rerank package, enhancing modularity and clarity in the search process.
- Modified the Search and SearchMultiple methods to include context parameters, improving flexibility for agent mode operations.
- Revised the Reranker interface to require context for Agent and MCP modes, ensuring compatibility with different reranking strategies.
- Enhanced documentation to provide comprehensive guidance on the updated search architecture and its components.
- Added initialization for the Search JSAPI factory in the Assistant module to streamline search operations.
- Updated the DESIGN.md to reflect the new architecture of the search module, including detailed descriptions of the keyword extraction and QueryDSL generation processes.
- Enhanced the interfaces for keyword extraction to require context, improving flexibility for different extraction modes.
- Revised directory structures in the documentation to clarify the organization of search-related components, ensuring comprehensive guidance for developers.
- Added logging for hook start and completion in the Assistant's Stream method to improve traceability during execution.
- Refactored the AgentGetterFunc to utilize the caller package, enhancing modularity and reducing circular dependencies.
- Updated the CallAgent function to check for the initialized AgentGetterFunc from the caller package, ensuring proper agent loading.
- Enhanced the Search handler to support an optional context parameter, improving flexibility for agent mode operations.
- Refined the agentSearch function to delegate search requests to a new AgentProvider, streamlining the search process.
- Updated DESIGN.md to reflect changes in search modes and the integration of the caller package, ensuring comprehensive documentation.
- Introduced new environment variables for TAVILY_API_KEY, SERPAPI_API_KEY, and SERPER_API_KEY in both `pr-test.yml` and `unit-test.yml` workflows to support additional search providers.
- Updated `DESIGN.md` to reflect the inclusion of SerpAPI as a search provider, detailing its usage and configuration options, including support for multiple search engines.
- Enhanced the `builtinSearch` function in `handler.go` to accommodate the new SerpAPI provider, ensuring proper handling of search requests.
- Revised the `WebConfig` struct in `config.go` to include an `Engine` field for specifying the search engine when using SerpAPI, improving flexibility in search configurations.
- Updated documentation to clarify the roles of new search providers and their integration within the search module, ensuring comprehensive guidance for developers.
- Updated the `QueryDSLGenerator` interface to use `gou.QueryDSL` and `model.Model` types for improved compatibility with Yao's query system.
- Revised the `Request` struct to incorporate GOU types for `Wheres` and `Orders`, enhancing the integration with the GOU QueryDSL format.
- Removed deprecated `QueryWhere` and `QueryOrder` types, streamlining the codebase and reducing redundancy.
- Enhanced documentation in `DESIGN.md` to reflect these changes and provide guidance on using GOU types directly.
- Introduced a new search configuration structure, allowing for detailed settings for web, knowledge base, database, citation, and weights.
- Updated the `Assistant` model to include a `Search` field, enabling assistant-specific search configurations.
- Enhanced the loading and merging logic for search configurations, ensuring global defaults can be overridden by assistant-specific settings.
- Added comprehensive tests for loading, saving, and updating assistants with search configurations, verifying the integrity of search settings.
- Updated documentation in DESIGN.md to reflect the new search configuration hierarchy and usage, clarifying the interaction between global and assistant-level settings.
- Updated the flowchart to enhance the representation of the search process, clarifying the conditions under which search is skipped or auto-triggered.
- Revised labels in the flowchart for better readability and understanding of the search control logic, ensuring consistency with recent documentation updates.
- Introduced a `uses` field in the return structure of the `Create` function to manage auto search behavior, setting it to "disabled" when handled by hooks.
- Revised flowchart and sequence diagrams to reflect the new `Uses.Search` control logic, enhancing clarity on search decision-making processes.
- Updated the `SearchUses` struct to encapsulate search-specific configurations, improving documentation on search mode options.
- Enhanced the documentation to detail the hierarchy of search configuration, clarifying how global, assistant, and hook-level settings interact.
- Added examples demonstrating the new search control values and their implications for auto search behavior.
- Revised the `Uses` struct documentation to reflect the new format for MCP tool references, changing from `mcp:<server-id>` to `mcp:<server>.<tool>`.
- Enhanced examples and descriptions throughout the document to illustrate the updated MCP tool usage in various search modes.
- Improved clarity on the configuration options for keyword extraction, QueryDSL generation, and reranking, ensuring consistency with the new MCP format.
- Updated error handling documentation to address the new MCP tool parsing logic, enhancing developer understanding of potential issues.
- Introduced a new `Uses` struct to encapsulate configuration for search modes, including `builtin`, `agent`, and `mcp`.
- Updated the `Searcher` initialization to utilize the new `Uses` struct for improved clarity and flexibility in search handling.
- Expanded documentation to detail the three web search modes, their functionalities, and the corresponding configuration options.
- Added examples illustrating the AI-powered search flow when using the agent mode, enhancing understanding of intent-aware search capabilities.
- Clarified the roles of built-in providers and the MCP server in the search process, ensuring comprehensive guidance for developers.
- Renamed the `config/` directory to `defaults/` to better reflect its purpose for default configuration values.
- Updated references throughout the documentation to align with the new directory structure.
- Clarified the configuration loading process, detailing how global and assistant-level configurations are merged.
- Enhanced the explanation of the `Searcher` struct and its initialization, emphasizing the use of merged configuration.
- Added new sections to document the configuration merging process for both global and assistant-specific settings, improving overall understanding of the search module's architecture.
- Modified the test case in agent_next_test.go to use a deterministic sub-scenario, reducing flakiness in tests caused by unpredictable LLM responses.
- Enhanced DESIGN.md to clarify the structure and organization of the search module, including detailed descriptions of new types, interfaces, and configuration options.
- Updated the documentation to reflect changes in the citation system, source weighting, and the overall architecture of the search module, ensuring consistency and better understanding for developers.
- Added detailed guidelines for citation output format, including HTML link attributes for reference integration.
- Introduced examples demonstrating the correct usage of citation links in LLM outputs, improving clarity for developers.
- Updated documentation to ensure consistency in citation practices across the search module, aiding in the integration of references.
