- Added separate unit test targets for core tests and AI-related tests in the Makefile, allowing for more granular testing.
- Updated GitHub Actions workflows to include dedicated jobs for running AI tests with SQLite and core tests, enhancing CI capabilities.
- Modified existing test commands to ensure proper execution of unit tests while excluding AI-related tests where necessary.
- Updated test paths in the agent's search tests to reflect changes in the test assistant being used, ensuring accurate test execution.
- Updated the executeAutoSearch method to include an optional parameter for Skip.Keyword, allowing for conditional keyword extraction during web searches.
- Implemented logic to extract keywords only when configured and not skipped, improving search query optimization.
- Modified the Assistant's Stream method to pass options to executeAutoSearch, ensuring seamless integration of the new functionality.
- Updated DESIGN.md to document the changes in keyword extraction logic and its impact on the search process.
- Added functionality to the Assistant's Stream method to execute auto search if enabled, enhancing the search capabilities based on user configuration.
- Introduced helper methods for determining auto search eligibility, executing the search, and injecting search context into messages.
- Updated DESIGN.md to reflect the new auto search logic, including detailed descriptions of the new methods and their integration points within the search process.
- Introduced a new search object in the context to expose search methods (Web, KB, DB, All, Any, Race) for JavaScript integration.
- Implemented individual search methods with argument validation and error handling, improving the robustness of the API.
- Updated the JSAPI implementation to utilize the new search object, ensuring seamless interaction with the search functionalities.
- Enhanced documentation in DESIGN.md to reflect the new V8 binding methods and their usage, providing clear guidance for developers.
- Refactored the SearchAPI interface to replace the Parallel method with All, Any, and Race methods, inspired by JavaScript Promise patterns.
- Updated the Searcher struct to implement these new parallel search methods, improving flexibility and performance in executing multiple searches.
- Revised the JSAPI implementation to support the new parallel search methods, ensuring consistency across the API.
- Enhanced documentation in DESIGN.md to detail the new parallel search functionalities and provide usage examples, clarifying their behavior and expected outcomes.
- Introduced a new TODO.md file to outline the implementation plan and progress for the search module.
- Updated DESIGN.md to reflect changes in the directory structure and clarify the roles of various components, including the new Handler + Registry pattern for reranking and keyword extraction.
- Refactored the Searcher struct to utilize a direct reference to the rerank package, enhancing modularity and clarity in the search process.
- Modified the Search and SearchMultiple methods to include context parameters, improving flexibility for agent mode operations.
- Revised the Reranker interface to require context for Agent and MCP modes, ensuring compatibility with different reranking strategies.
- Enhanced documentation to provide comprehensive guidance on the updated search architecture and its components.
- Added initialization for the Search JSAPI factory in the Assistant module to streamline search operations.
- Updated the DESIGN.md to reflect the new architecture of the search module, including detailed descriptions of the keyword extraction and QueryDSL generation processes.
- Enhanced the interfaces for keyword extraction to require context, improving flexibility for different extraction modes.
- Revised directory structures in the documentation to clarify the organization of search-related components, ensuring comprehensive guidance for developers.
- Added logging for hook start and completion in the Assistant's Stream method to improve traceability during execution.
- Refactored the AgentGetterFunc to utilize the caller package, enhancing modularity and reducing circular dependencies.
- Updated the CallAgent function to check for the initialized AgentGetterFunc from the caller package, ensuring proper agent loading.
- Enhanced the Search handler to support an optional context parameter, improving flexibility for agent mode operations.
- Refined the agentSearch function to delegate search requests to a new AgentProvider, streamlining the search process.
- Updated DESIGN.md to reflect changes in search modes and the integration of the caller package, ensuring comprehensive documentation.
- Introduced new environment variables for TAVILY_API_KEY, SERPAPI_API_KEY, and SERPER_API_KEY in both `pr-test.yml` and `unit-test.yml` workflows to support additional search providers.
- Updated `DESIGN.md` to reflect the inclusion of SerpAPI as a search provider, detailing its usage and configuration options, including support for multiple search engines.
- Enhanced the `builtinSearch` function in `handler.go` to accommodate the new SerpAPI provider, ensuring proper handling of search requests.
- Revised the `WebConfig` struct in `config.go` to include an `Engine` field for specifying the search engine when using SerpAPI, improving flexibility in search configurations.
- Updated documentation to clarify the roles of new search providers and their integration within the search module, ensuring comprehensive guidance for developers.
- Updated the `QueryDSLGenerator` interface to use `gou.QueryDSL` and `model.Model` types for improved compatibility with Yao's query system.
- Revised the `Request` struct to incorporate GOU types for `Wheres` and `Orders`, enhancing the integration with the GOU QueryDSL format.
- Removed deprecated `QueryWhere` and `QueryOrder` types, streamlining the codebase and reducing redundancy.
- Enhanced documentation in `DESIGN.md` to reflect these changes and provide guidance on using GOU types directly.
- Introduced a new search configuration structure, allowing for detailed settings for web, knowledge base, database, citation, and weights.
- Updated the `Assistant` model to include a `Search` field, enabling assistant-specific search configurations.
- Enhanced the loading and merging logic for search configurations, ensuring global defaults can be overridden by assistant-specific settings.
- Added comprehensive tests for loading, saving, and updating assistants with search configurations, verifying the integrity of search settings.
- Updated documentation in DESIGN.md to reflect the new search configuration hierarchy and usage, clarifying the interaction between global and assistant-level settings.
- Updated the flowchart to enhance the representation of the search process, clarifying the conditions under which search is skipped or auto-triggered.
- Revised labels in the flowchart for better readability and understanding of the search control logic, ensuring consistency with recent documentation updates.
- Introduced a `uses` field in the return structure of the `Create` function to manage auto search behavior, setting it to "disabled" when handled by hooks.
- Revised flowchart and sequence diagrams to reflect the new `Uses.Search` control logic, enhancing clarity on search decision-making processes.
- Updated the `SearchUses` struct to encapsulate search-specific configurations, improving documentation on search mode options.
- Enhanced the documentation to detail the hierarchy of search configuration, clarifying how global, assistant, and hook-level settings interact.
- Added examples demonstrating the new search control values and their implications for auto search behavior.
- Revised the `Uses` struct documentation to reflect the new format for MCP tool references, changing from `mcp:<server-id>` to `mcp:<server>.<tool>`.
- Enhanced examples and descriptions throughout the document to illustrate the updated MCP tool usage in various search modes.
- Improved clarity on the configuration options for keyword extraction, QueryDSL generation, and reranking, ensuring consistency with the new MCP format.
- Updated error handling documentation to address the new MCP tool parsing logic, enhancing developer understanding of potential issues.
- Introduced a new `Uses` struct to encapsulate configuration for search modes, including `builtin`, `agent`, and `mcp`.
- Updated the `Searcher` initialization to utilize the new `Uses` struct for improved clarity and flexibility in search handling.
- Expanded documentation to detail the three web search modes, their functionalities, and the corresponding configuration options.
- Added examples illustrating the AI-powered search flow when using the agent mode, enhancing understanding of intent-aware search capabilities.
- Clarified the roles of built-in providers and the MCP server in the search process, ensuring comprehensive guidance for developers.
- Renamed the `config/` directory to `defaults/` to better reflect its purpose for default configuration values.
- Updated references throughout the documentation to align with the new directory structure.
- Clarified the configuration loading process, detailing how global and assistant-level configurations are merged.
- Enhanced the explanation of the `Searcher` struct and its initialization, emphasizing the use of merged configuration.
- Added new sections to document the configuration merging process for both global and assistant-specific settings, improving overall understanding of the search module's architecture.
- Modified the test case in agent_next_test.go to use a deterministic sub-scenario, reducing flakiness in tests caused by unpredictable LLM responses.
- Enhanced DESIGN.md to clarify the structure and organization of the search module, including detailed descriptions of new types, interfaces, and configuration options.
- Updated the documentation to reflect changes in the citation system, source weighting, and the overall architecture of the search module, ensuring consistency and better understanding for developers.
- Added detailed guidelines for citation output format, including HTML link attributes for reference integration.
- Introduced examples demonstrating the correct usage of citation links in LLM outputs, improving clarity for developers.
- Updated documentation to ensure consistency in citation practices across the search module, aiding in the integration of references.
- Replaced the previous text-based data flow representation with a flowchart using Mermaid syntax for improved readability and understanding.
- Streamlined the depiction of data sources and their relationships, clarifying the flow from content module to LLM input.
- Updated the documentation to ensure consistency with the new visual format, aiding in the comprehension of the search module's architecture.
- Added new fields to the ResultItem struct for source type, weight, and relevance score to improve context building for LLM.
- Updated the documentation to clarify the source weighting system, detailing how user, hook, and auto sources are prioritized.
- Introduced a unified context protocol for handling references across different data sources, ensuring consistent formatting and processing flow.
- Enhanced examples and behavior rules to reflect the new structure and clarify the integration of source weighting in search results.
- Refined the description of Global Configuration to specify the roles of `agent/agent.yml` and `agent/search.yao` in search options.
- Added detailed information on the `uses` configuration, including keyword extraction, QueryDSL generation, and reranking methods.
- Enhanced documentation to improve understanding of how configuration layers interact and override each other, aiding in the customization of the search module.
- Modified the reranking section to specify "builtin" as the default score-based reranking method, replacing previous terminology.
- Updated the RerankOptions structure to reflect changes in parameter names and types, including the transition from `topK` to `topN`.
- Enhanced documentation to clarify how reranker types are determined in the configuration file, improving overall understanding of the search module's reranking capabilities.
- Updated comments and structure in the configuration section to better distinguish between keyword extraction, QueryDSL generation, and rerank options.
- Removed redundant comments and streamlined the configuration keys for improved clarity and consistency.
- Enhanced documentation to reflect the new organization of configuration settings, aiding in understanding the search module's functionality.
- Changed the configuration key from `uses.dsl` to `uses.query` for clarity in the search processing tools section.
- Updated documentation to ensure consistency in the description of query processing methods and their corresponding configurations.
- Enhanced examples to align with the new configuration terminology, improving understanding of the search module's functionality.
- Revised the configuration details for keyword extraction, QueryDSL generation, and reranking tools, specifying the roles of built-in, agent, and MCP server options.
- Enhanced the formatting section to improve clarity on tool usage and examples, ensuring better understanding of the search module's capabilities.
- Updated examples to reflect the new configurations, providing clearer guidance on how to utilize the search processing tools effectively.
- Updated the configuration section to outline a three-layer hierarchy for settings: System Built-in Defaults, Global Configuration, and Assistant Configuration.
- Expanded the uses configuration details in `agent/agent.yml`, specifying processing tools for keyword extraction, QueryDSL generation, and reranking.
- Enhanced the explanation of system defaults and their role in the configuration process, providing clearer guidance on how to override settings at different levels.
- Added detailed examples for each configuration layer, improving understanding of the search module's behavior and customization options.
- Updated the section on user message processing to clarify that users only specify data source IDs, with filters generated by the Search module from natural language input.
- Added a detailed example illustrating the extraction of queries and the generation of QueryDSL for data sources, improving understanding of the Search module's functionality.
- Introduced a new section in DESIGN.md detailing the integration of the content module for handling user messages with data source references.
- Defined various DataSource types including model, KB collection, KB document, table, API, and MCP resource.
- Provided an example of a user message containing data references and outlined the processing flow in the content.Vision() function.
- Specified the implementation details for the processDataContent() function in content/content.go, enhancing the search module's reusability for auto search and explicit data content references.
- Adjusted sleep durations to 1100ms to enhance test stability in CI environments.
- Modified timestamp calculations to ensure millisecond precision, improving event timing accuracy in tests.
- Updated the TestHistoryLoading function to clarify role filtering, ensuring only user and assistant messages are included in the history.
- Adjusted message creation timestamps for consistency in test scenarios.
- Removed deprecated context creation methods to streamline context management and improve code clarity.
- Enhanced logging levels in context management for better traceability and debugging.
- Added a new method to set store settings during assistant initialization, allowing for configuration of storage parameters such as MaxSize and TTL.
- Updated context creation methods to streamline the setup process, ensuring that essential fields are populated consistently across various test contexts.
- Revised tests to validate the new initialization behavior and context management, ensuring proper handling of assistant settings and context properties.
- Added `locale` query parameter for assistant information retrieval, allowing for localization based on user preferences.
- Introduced `assistants` field in the response to provide localized details (name, avatar, description) for unique assistant IDs, improving frontend display capabilities.
- Updated documentation to clarify locale resolution priority and the significance of the new `assistants` field in chat responses.
- Introduced `GetInfo` method in the Assistant struct to return basic assistant information with optional locale support for internationalization.
- Implemented `GetInfoByIDs` function to retrieve information for multiple assistants based on their IDs, enhancing batch processing capabilities.
- Updated chat buffer management to include a mode parameter, allowing for dynamic switching between chat and task modes.
- Enhanced message handling to store and retrieve the mode associated with each message, improving context tracking during chat sessions.
- Revised tests to validate the new functionalities, ensuring accurate retrieval of assistant information and proper mode management in chat operations.
- Updated the `Create` and `Next` hook documentation to clarify their execution flow and usage, including new optional parameters for customization.
- Added detailed examples demonstrating the hooks' functionality, including error handling and tool processing.
- Introduced new notes on resource management, emphasizing the correct usage of `ctx.Release()` for manually created contexts.
- Enhanced overall clarity and guidance for developers on the agent execution lifecycle and context API features.
- Added new methods to the `ctx.trace` object for improved tracing capabilities, including `EndBlock`, `MessageID`, `BlockID`, and `ThreadID`.
- Expanded the documentation to clarify the purpose of the `ctx.trace` object, detailing its properties and methods for user transparency and developer debugging.
- Updated examples to reflect changes in method usage and added new sections for trace lifecycle and space operations, enhancing overall clarity and guidance for developers.
- Changed property names from `MCP` to `mcp` and `Trace` to `trace` in the Context object for consistency with JavaScript naming conventions.
- Updated related tests and documentation to reflect these changes, ensuring clarity and uniformity across the API.
- Enhanced examples in JSAPI.md to demonstrate the new property names, improving developer guidance.
- Deleted the outdated JSAPI_OUTPUT.md file, consolidating documentation efforts.
- Revised JSAPI.md to reflect changes in the Context object, including the removal of the `connector` and `retry` fields.
- Updated method descriptions and examples for clarity, particularly around message handling and streaming capabilities.
- Added new sections for the Space API, detailing methods for data sharing between requests and agent calls.
- Enhanced hook documentation to clarify usage and execution flow, ensuring better guidance for developers.
- Updated the `ChatBuffer` to support streaming messages, allowing for content to be appended and finalized with `SendStream` and `End` methods.
- Modified the `AddAssistantMessage` method to include a message ID, improving message tracking and retrieval.
- Implemented new methods for appending content to streaming messages and completing them, ensuring accurate message storage and event handling.
- Revised tests to validate the new streaming functionality and ensure proper integration with existing message handling processes.
- Updated `CHAT_STORAGE_DESIGN.md` to reflect changes in message storage and indexing, including unique constraints for message IDs within requests.
- Enhanced chat buffer and message handling to support dynamic connector switching, allowing users to change the connector during a chat session.
- Updated the `ChatBuffer` and `BufferedMessage` structures to include a `connector` field, enabling tracking of the connector used for each message.
- Modified the `EnsureChat` method to skip chat creation when history is disabled, improving chat session management.
- Revised tests to validate the new connector functionality, ensuring accurate message retrieval and connector state management.
- Updated `CHAT_STORAGE_DESIGN.md` to reflect the addition of the `last_connector` field in chat metadata and message structures.
- Added comprehensive documentation for the new RESTful HTTP APIs for managing chat sessions and messages, including endpoints for listing, retrieving, updating, and deleting chat sessions.
- Included detailed request and response examples for each endpoint, along with query parameters and permission filtering guidelines.
- Updated the `CHAT_STORAGE_DESIGN.md` to reflect these changes, ensuring clarity on the API's functionality and usage.
- Introduced methods for initializing and managing a chat buffer, allowing for efficient storage of user inputs and assistant messages during chat sessions.
- Added functionality to track execution steps, including beginning and completing steps, with support for capturing space snapshots and handling errors.
- Enhanced the FlushBuffer method to save buffered messages and steps to the database, ensuring data integrity and recovery capabilities.
- Updated the Stream method to integrate buffer management, ensuring proper handling of chat sessions and message storage.
- Added comprehensive tests to validate buffer initialization, user input handling, and step tracking functionalities.
- Updated the ChatFilter structure to include advanced permission filters for UserID and TeamID, allowing for more granular chat retrieval.
- Added examples in the documentation to demonstrate new filtering capabilities, including combinations of user and team filters, as well as complex conditions using QueryFilter.
- Implemented batch saving and retrieval functionalities for messages and resumes, improving data management efficiency.
- Revised related tests to validate the new filtering features and ensure robust functionality across chat management operations.
- Added CreateChat, GetChat, UpdateChat, and DeleteChat methods to manage chat sessions effectively.
- Implemented validation for required fields and handling of nullable fields during chat creation and updates.
- Enhanced chat retrieval with pagination and filtering options, including time-based grouping.
- Introduced helper functions for converting database rows to Chat structs and grouping chats by time.
- Updated message and resume models to support soft deletes, improving data management and integrity.
- Updated the `CHAT_STORAGE_DESIGN.md` to define the `ChatStore` and `AssistantStore` interfaces, providing clear operations for managing chats, messages, resumes, and assistants.
- Introduced new methods for creating, retrieving, updating, and deleting chats and assistants, along with batch operations for messages and resumes.
- Enhanced the documentation to clarify the responsibilities of each store interface and the associated data structures, ensuring better understanding for future development.
- Revised related functions and tests to support the new design, reinforcing the integrity and performance of chat interactions and assistant management.
- Removed the `history` model and replaced it with a new `message` model to streamline chat message storage and retrieval.
- Introduced a `resume` model to enhance the chat's ability to recover from interruptions, capturing essential execution state data.
- Updated the `CHAT_STORAGE_DESIGN.md` to reflect these changes, including new fields and relationships for the message and resume models.
- Revised related functions and tests to support the new data structures, ensuring improved integrity and performance in chat interactions.
- Added time range filters (StartTime, EndTime, TimeField) to the ChatFilter structure for improved chat retrieval based on specific time criteria.
- Introduced sorting options (OrderBy, Order) to allow customization of chat list order.
- Implemented grouping functionality (GroupBy) to enable time-based organization of chat results.
- Updated the ChatList structure to include groups for better UI representation of chats.
- Added examples in the documentation to demonstrate the new filtering and grouping capabilities in chat retrieval.
- Removed the `preset` field from the chat metadata structure to simplify the design.
- Added new indices for `role` and `thread_id` in the message storage section to enhance query performance and organization.
- Introduced a new index for `type` in the resume functionality to improve retrieval efficiency during chat state restoration.
- Revised the documentation to emphasize that all message types, including built-in and custom types, are stored, with specific notes on the handling of lifecycle events and custom types.
- Enhanced the timeline section to reflect changes in message processing, including updates to the sequence of stored messages and the introduction of concurrent operations storage.
- Improved clarity on the storage approach for concurrent operations, detailing the use of `block_id` and `thread_id` for grouping messages during parallel tasks.
- Updated the write strategy in CHAT_STORAGE_DESIGN.md to transition from a two-write to a single-write approach, enhancing efficiency by writing all messages (user input and assistant responses) to the database only once upon stream exit.
- Revised related documentation to clarify the new single-write process and its implications for message handling and database interactions.
- Adjusted implementation details in the Stream function to reflect the new strategy, ensuring that steps are only saved on error or interruption, thereby reducing unnecessary database writes.
- Introduced a new `retrieval` message type in `BUILTIN_TYPES.md` to handle knowledge base and web search results, including structured properties for query, sources, and metadata.
- Updated `CHAT_STORAGE_DESIGN.md` to document the storage approach for retrieval results, emphasizing user feedback, quality analytics, and source attribution.
- Enhanced examples and use cases for retrieval messages to clarify implementation and integration within the chat system.
- Revised related documentation to ensure consistency and understanding of the new retrieval capabilities.
