- Introduced `file_id` field in the document model to link documents with their corresponding files in the attachment system.
- Updated `AddFileProcess` to include `file_id` in the response, improving document tracking.
- Modified `GetDocument` function to support the new `file_id` field in document retrieval.
- Removed the unused `ScrollDocuments` endpoint to streamline the API.
- Improved error handling and response structure for document-related operations.
- Enhanced ListDocuments function to support pagination with customizable page size and sorting options.
- Added filtering capabilities for keywords, tags, collection IDs, and status, allowing for more refined document retrieval.
- Introduced validation for requested fields and sorting parameters to ensure only valid options are processed.
- Improved error handling for document search operations, returning appropriate error responses when necessary.
- Updated AddFileProcess, AddTextProcess, and AddURLProcess functions to accept an optional job ID for async operations.
- Enhanced error handling by requiring document IDs to be provided by the caller before processing.
- Streamlined document creation and upsert operations, ensuring better validation and rollback mechanisms.
- Implemented async processing capabilities in AddFileAsync, AddTextAsync, and AddURLAsync functions, returning job IDs and document IDs in responses.
- Improved code maintainability and readability by encapsulating business logic and reducing dependencies on Gin context.
- Changed MaxConcurrent from 10 to 1 in test cases to align with updated configuration.
- Added new properties: Separator and EnableDebug, with default values set in tests.
- Refactored semantic options handling to include Connector and Toolcall properties, ensuring correct defaults are validated in tests.
- Enhanced test structure for semantic properties to improve clarity and maintainability.
- Updated modification timestamps for various asset files to reflect recent changes.
- Modified chunking options to include new parameters: Separator and EnableDebug, with default values set.
- Enhanced the Options method in Structured and Semantic providers to handle additional properties from the option map, improving flexibility in configuration.
- Added new fields in the document model for converter and fetcher option IDs, allowing for better integration with provider configurations.
- Introduced separate processing functions (AddFileProcess, ProcessAddTextRequest) to encapsulate business logic for file and text additions, making them Gin-agnostic.
- Updated AddFile, AddText, and AddURL functions to handle request parsing and validation more effectively.
- Enhanced error handling and rollback mechanisms during document creation and upsert operations.
- Implemented async processing capabilities for AddFileAsync, AddTextAsync, and AddURLAsync functions using pre-parsed request data.
- Streamlined utility functions for preparing requests and validating input, improving code maintainability and readability.
- Added ListDocuments and ScrollDocuments functions for paginated document retrieval.
- Implemented GetDocument function to fetch document details by ID, including error handling for missing IDs.
- Introduced RemoveDocs function to handle document deletion requests.
- Removed outdated AddFile, AddText, and AddURL functions to streamline document management logic.
- Renamed extractor provider to extraction provider across the codebase for consistency and clarity.
- Updated references in configuration files, provider factories, and asset management to reflect the new terminology.
- Removed the extractor provider implementation and associated test files, streamlining the provider structure.
- Adjusted test cases and documentation to align with the new extraction provider framework.
- Updated the resolveProviderOption function to include providerType as a required parameter, enhancing error handling for missing provider types.
- Simplified the provider retrieval logic to directly fetch providers based on the specified type, improving clarity and maintainability.
- Removed redundant comments and streamlined the code for better readability.
- Updated modification timestamps for various asset files to reflect recent changes.
- Added support for "application/octet-stream" MIME type in the attachment.local.yao file to enhance file upload capabilities.
- Added GetConfig function to retrieve the knowledge base configuration, ensuring proper initialization checks.
- Refactored CreateCollection, AddFile, AddText, and AddURL functions to utilize the new GetConfig method for improved error handling and database record management.
- Introduced preparation functions (PrepareCreateCollection, PrepareAddFile, PrepareAddText, PrepareAddURL) to streamline request handling and database data preparation.
- Enhanced error responses and rollback mechanisms for document and collection operations, improving robustness and clarity in error handling.
- Introduced default values for CollectionModel and DocumentModel in the Config struct to ensure proper initialization.
- Updated UnmarshalJSON method to set these defaults if not explicitly configured, enhancing the robustness of the configuration handling.
- Replaced the existing kb.mod.yao file with two new files: collection.mod.yao and document.mod.yao, enhancing the organization of knowledge base models.
- Updated references in system and test models to reflect the new structure, ensuring compatibility with the updated model paths.
- Adjusted asset loading functions to accommodate the new model files, improving the overall asset management system.
- Introduced a Validator interface and a validateRequest function to streamline request validation across handlers.
- Added checkKBInstance function to ensure the knowledge base instance is initialized before processing requests.
- Implemented handleAsync function to manage asynchronous processing for file and text addition.
- Updated AddFile and AddText functions to utilize the new validation and async handling logic.
- Added async variants for AddFile and AddText to support non-blocking operations.
- Enhanced AddURL and its async variant with similar validation and processing improvements.
- Modified test cases for CreateCollection, RemoveCollection, and CollectionIntegration to use the new embedding provider and option values.
- Removed outdated fields related to the previous embedding model and added locale support for provider reading.
- Ensured consistency in the configuration structure across all relevant tests.
- Introduced ProviderSettings type to encapsulate provider configuration details.
- Implemented getProviderSettings function to resolve provider settings based on provider ID and option value.
- Updated CreateCollection function to utilize provider settings, including dimension and metadata enhancements.
- Refactored CreateCollectionRequest to include a new CreateCollectionConfig struct for better organization of embedding provider details.
- Introduced new test cases for Structured and Semantic providers to validate schema retrieval functionality.
- Updated existing tests across various providers to ensure they check for non-nil schemas, enhancing error handling and test coverage.
- Ensured consistency in error messages for schema validation across different provider tests.
- Updated Schema methods for various providers (OpenAI, Fastembed, MCP, OCR, Office, UTF8, Video, Vision, Whisper) to retrieve schemas from bindata, improving localization capabilities.
- Removed outdated JSON files for embedding, extractor, and fetcher providers, streamlining the data structure.
- Enhanced the overall provider management by ensuring schemas are dynamically loaded based on locale, facilitating better multi-language support.
- Modified the test case for UnmarshalJSON in config_test.go to specify that features are not computed during the unmarshalling process but rather after providers are loaded.
- Added comments to explain the manual computation of features for testing purposes, ensuring clarity in the test logic.
- Moved the computation of features in the Config struct to occur after loading providers, ensuring that all necessary data is available for accurate feature computation.
- Updated comments for clarity regarding the loading sequence of providers and feature computation.
- Implemented loading of providers from directories, allowing for multi-language configurations.
- Updated the KnowledgeBase struct to include a Providers field for managing provider configurations.
- Refactored GetProviders and GetProvider methods to utilize the new multi-language provider system, improving localization support.
- Added comprehensive test cases to validate provider loading and retrieval functionality across different languages.
- Implemented GetProviders and GetProvider methods to retrieve providers based on type and ID, improving the knowledge base's provider management capabilities.
- Enhanced the Schema methods for chunking providers to retrieve schemas from bindata, ensuring better integration with the overall system.
- Updated OpenAPI routes to include endpoints for provider management, allowing for easier access to provider information via HTTP requests.
- Modified Schema methods across various providers (chunking, embedding, extractor, fetcher, converters) to accept an additional locale parameter, enhancing localization support.
- Updated corresponding test cases to reflect the changes in method signatures and ensure proper functionality with the new locale argument.
- Updated the Config struct to include `omitempty` for optional fields in JSON and YAML serialization, ensuring that empty values are omitted from the output.
- Enhanced the Provider struct and related configurations to maintain consistency in field definitions and improve clarity in the codebase.
- Implemented UpdateCollectionMetadata function to handle metadata updates for existing collections.
- Added request structure and validation for updating collection metadata.
- Integrated new endpoint in the router for updating collection metadata via PUT request.
