Commit graph

2432 commits

Author SHA1 Message Date
Max
bb64eef20d Implement environment variable resolution in configuration parsing and enhance collection management API
- Added a new test for resolving environment variables during configuration parsing, ensuring that environment variables are correctly substituted in the configuration.
- Refactored the configuration handling to resolve environment variables immediately after parsing, improving the clarity and usability of the configuration structure.
- Enhanced the collection management API by standardizing error responses and success responses using a custom response structure, improving consistency across endpoints.
- Implemented cleanup logic for test collections to ensure proper resource management during testing.
2025-07-24 17:08:58 +08:00
Max
8d35e824ae Remove deprecated test files and refactor OAuth response handling
- Deleted obsolete test files for various OpenAPI components, including config_test.go, dsl_test.go, hello_test.go, oauth_test.go, oauth_token_test.go, and openapi_test.go, to streamline the codebase.
- Refactored OAuth response handling by integrating response methods from the response package, ensuring consistent error and success responses across OAuth endpoints.
- Enhanced error handling and response structure for improved clarity and maintainability, aligning with best practices for API responses.
2025-07-24 15:53:29 +08:00
Max
40519e8429 Refactor Knowledge Base loading in Load and Reload functions
- Moved Knowledge Base loading logic from the Load function to ensure it is consistently handled in both Load and Reload functions.
- Improved error handling by printing errors during Knowledge Base loading in the Reload function, enhancing user feedback.
2025-07-23 17:56:51 +08:00
Max
d25ce68ab9 Enhance CI workflows and Knowledge Base integration
- Updated GitHub Actions workflows to use the latest Ubuntu version and added services for Neo4j, Qdrant, and FastEmbed.
- Integrated FFmpeg and additional utilities into the CI environment, ensuring they are installed and tested during the workflow.
- Implemented environment variable resolution in the Knowledge Base configuration, allowing dynamic configuration based on environment settings.
- Added tests for environment variable resolution in the Knowledge Base configuration, ensuring correct parsing and handling of variables.
2025-07-23 17:42:10 +08:00
Max
b76afe23bf Add Knowledge Base configuration and integration with GraphRag
- Introduced a new configuration structure for the Knowledge Base, supporting vector and graph database configurations.
- Implemented JSON parsing methods for loading configurations from files and converting them to JSON format.
- Enhanced the Knowledge Base instance to integrate with GraphRag, allowing for dynamic configuration of vector and graph stores.
- Added feature detection capabilities to determine available functionalities based on the current configuration.
- Updated the Knowledge Base loading process to read from a specified configuration file, improving flexibility and usability.
2025-07-23 17:07:38 +08:00
Max
aae1bccf88 Add Knowledge Base API endpoints and integrate with OpenAPI router
- Introduced the Knowledge Base (KB) API by creating a new kb package with various endpoints for collection and document management, segment management, and search functionalities.
- Updated the OpenAPI router to attach the KB API, ensuring OAuth protection for all endpoints.
- Implemented logging for configuration validation to enhance error handling and user feedback.
2025-07-23 15:19:40 +08:00
Max
a921a76778 Deprecate studio API functions and comment out implementation details
- Marked the studio API functions (Start, Load, loadDSL, loadScripts) as deprecated, returning an error message to indicate their status.
- Commented out the implementation details of these functions to prevent execution of deprecated functionality, ensuring clarity in the codebase.
2025-07-22 17:41:03 +08:00
Max
7bdeec85f9 Deprecate studio API functions and comment out test cases
- Marked the studio API functions (Start, Load, loadDSL, loadScripts) as deprecated, returning an error message to indicate their status.
- Commented out all test cases in studio_test.go to prevent execution of deprecated functionality, ensuring clarity in the test suite.
2025-07-22 17:29:42 +08:00
Max
92d21c5389 Remove hello world endpoints and refactor routing to use new hello package
- Deleted the existing hello world handlers from hello.go to streamline the codebase.
- Updated openapi.go to attach the new hello package for handling hello world routes, ensuring OAuth protection is applied correctly.
- Enhanced the README.md to reflect the new structure and provide comprehensive documentation for the hello world API endpoints.
2025-07-22 16:55:24 +08:00
Max
d54572a7d1 Refactor OAuth response handling and improve content type management
- Updated response methods to standardize Content-Type header to "application/json" across OAuth endpoints, enhancing compliance with JSON standards.
- Refactored error and success response methods to streamline response generation without unnecessary wrappers, improving clarity and maintainability.
- Enhanced security by ensuring all responses include appropriate OAuth security headers, aligning with best practices for sensitive endpoints.
- Simplified test assertions for Content-Type in OAuth tests, ensuring consistency in response validation.
2025-07-22 15:36:41 +08:00
Max
7e8d4a9ba9 Enhance OAuth token management with PKCE support and refactor tests
- Updated OAuth token handling to include PKCE (Proof Key for Code Exchange) parameters, improving security for authorization code grants.
- Refactored token management tests to incorporate PKCE code verifier and challenge, ensuring compliance with OAuth 2.1 standards.
- Enhanced refresh token handling to validate requested scopes against originally granted scopes, improving security and compliance.
- Updated various methods to support optional scope parameters, streamlining token management and validation processes.
- Improved test coverage for token introspection and exchange scenarios, ensuring robust validation of token handling logic.
2025-07-22 12:55:44 +08:00
Max
41c44cb726 Implement JWKS endpoint and enhance OAuth tests
- Added the JWKS endpoint to return JSON Web Key Set in compliance with RFC 7517, including necessary security headers.
- Refactored the JWKS generation logic to retrieve signing certificates and construct the JWK from the RSA public key.
- Introduced comprehensive tests for the JWKS endpoint, validating response format, compliance, and security headers.
- Updated go.mod to include the MongoDB driver as a required dependency.
2025-07-21 20:07:46 +08:00
Max
966e0cfd00 Enhance OAuth test setup with reusable certificate management
- Introduced global test certificate paths to avoid redundant certificate generation across tests, improving efficiency.
- Implemented a function to create temporary certificates once for all tests, ensuring consistent usage of signing certificates.
- Updated test configurations to utilize the new certificate management, enhancing clarity and maintainability.
- Added cleanup functionality for global test certificates to ensure proper resource management after tests.
2025-07-21 19:44:00 +08:00
Max
e1428551ba Refactor OAuth token management tests and remove deprecated methods
- Removed outdated token management tests from the user provider, including tests for storing, revoking, and retrieving tokens.
- Updated the user provider interface to reflect the removal of token management methods, ensuring cleaner code and improved maintainability.
- Streamlined the test suite by focusing on relevant user management functionalities, enhancing overall test clarity and effectiveness.
2025-07-21 18:59:26 +08:00
Max
8d1174d566 Refactor hello world endpoints and add OAuth protection
- Renamed existing hello world endpoints to public and added a new protected endpoint with OAuth guard.
- Updated test cases to reflect the new endpoint structure and added tests for protected endpoint access with and without valid tokens.
- Enhanced response handling for public and protected endpoints to ensure consistent output and proper status codes.
2025-07-21 18:49:20 +08:00
Max
55931bb59b Add OAuth token revocation and introspection tests
- Implemented comprehensive tests for the OAuth token revocation and introspection endpoints, ensuring correct handling of valid, invalid, and missing token scenarios.
- Enhanced the oauthRevoke and oauthIntrospect methods to comply with RFC specifications, returning appropriate status codes and responses.
- Introduced a utility function for obtaining access tokens directly in tests, streamlining the testing process for OAuth endpoints.
- Improved error handling and logging for better debugging and verification during tests.
2025-07-21 17:57:34 +08:00
Max
2b171c3540 Refactor OAuth token handling and enhance grant type support
- Consolidated token grant handling into a unified method for authorization code, client credentials, and device code grants, improving code organization and reducing duplication.
- Introduced new methods for handling token exchange and refresh token grants, ensuring compliance with relevant RFCs.
- Enhanced error handling and validation for client credentials and grant types, improving robustness and security.
- Updated tests to utilize real authorization codes and ensure comprehensive coverage of the new functionality.
2025-07-21 17:40:08 +08:00
Max
e4a02d6b9c Add OAuth authorization tests and improve error handling 2025-07-21 16:21:40 +08:00
Max
e936407452 Enhance OAuth functionality with registration tests and response improvements
- Added a comprehensive test for the OAuth client registration endpoint, ensuring proper handling of valid requests and responses.
- Updated the response handling methods to comply with RFC standards, including a new method for direct OAuth responses.
- Refactored existing response methods to improve security header management and streamline response generation.
- Enhanced test logging for better debugging and verification of response data during OAuth registration tests.
2025-07-21 16:07:40 +08:00
Max
517a4cf5b0 Implement OAuth endpoint logic and enhance test setup
- Added core functionality for OAuth authorization, token handling, and user info retrieval, including error handling and response generation.
- Introduced helper functions for managing different grant types, improving code organization and clarity.
- Enhanced the test setup with a new Prepare function to initialize the OpenAPI test environment and a Clean function for proper resource management.
- Updated tests to ensure the OpenAPI server is correctly loaded and operational during testing.
2025-07-21 15:34:13 +08:00
Max
458391f5b4 Refactor OAuth endpoint structure and add well-known handlers
- Removed OAuth discovery and metadata endpoints from the attachOAuth function to streamline the code.
- Introduced a new method to handle well-known endpoints, improving organization and clarity in the OAuth implementation.
- Updated the Attach method to include the new well-known handlers, ensuring proper routing for OAuth-related metadata.
2025-07-20 19:11:16 +08:00
Max
6256e131b3 Add OAuth documentation for versioned paths and discovery endpoints
- Added notes in the OAuth handler to clarify the requirements for using versioned paths, including the placement of discovery endpoints and server metadata.
- Emphasized the importance of mounting discovery endpoints at the root level for proper MCP client configuration and OAuth functionality.
2025-07-20 19:05:41 +08:00
Max
b96a400869 Add OpenAPI support and enhance configuration handling
- Integrated OpenAPI loading functionality into the engine's Load and Reload processes, allowing for better API management.
- Updated the OpenAPI configuration to set a default BaseURL and ensure it does not have a trailing slash.
- Implemented the Attach method to connect the OpenAPI server to the Gin router, facilitating API endpoint management.
- Removed the obsolete hello package to streamline the OpenAPI module.
2025-07-20 18:52:30 +08:00
Max
0fff602c93 Implement JSON marshaling and unmarshaling for configuration with duration parsing
- Enhanced the Config struct to support JSON marshaling and unmarshaling with human-readable duration strings for various OAuth settings.
- Introduced temporary structures to facilitate the conversion of string duration fields to time.Duration types during JSON operations.
- Added utility functions for parsing and formatting duration strings, ensuring accurate handling of time-related configurations.
- Updated tests to validate the correct parsing and formatting of duration fields in the configuration.
2025-07-20 17:49:01 +08:00
Max
fa58bf42dc Enhance test for Load function by adding data store cleanup
- Implemented a cleanup step in the TestLoad function to remove the data store created during tests, ensuring a clean test environment.
- Added error handling for the Load function to improve robustness and provide feedback in case of failures.
2025-07-20 12:03:25 +08:00
Max
5d3c03db1f Update OAuth store management and enhance cache configurations
- Added support for a new OAuth data store in the system store management, allowing for better organization and retrieval of OAuth-related data.
- Updated the OAuth cache store configuration to increase the cache size from 4096 to 8192, improving performance and storage capacity.
- Enhanced test utilities to include the new OAuth store, ensuring comprehensive testing coverage for the updated store management features.
2025-07-20 11:47:57 +08:00
Max
8523170992 Implement system store management and enhance store loading functionality
- Introduced a new system store management feature, allowing for the loading of predefined system stores such as cache and OAuth client stores.
- Added a `loadSystemStores` function to handle the loading of system stores with variable replacement for configuration paths.
- Enhanced the `Load` function to include the loading of system stores, improving the overall store management process.
- Updated test utilities to support loading system stores for testing, ensuring comprehensive coverage and functionality.
- Refactored the `replaceVars` function to facilitate variable replacement in JSON strings, enhancing flexibility in store configurations.
2025-07-20 11:23:00 +08:00
Max
72b732348d Enhance OAuth configuration types with optional defaults
- Updated the SigningConfig, TokenConfig, SecurityConfig, and ClientConfig structures to include optional default values for various fields, improving clarity and usability.
- Added comments to specify default values and optionality for each configuration parameter, enhancing documentation within the code.
- This change aims to streamline the configuration process for OAuth service implementations.
2025-07-18 17:05:13 +08:00
Max
ef07622e7e Add user and security tests 2025-07-18 15:48:50 +08:00
Max
2610036082 Refactor token exchange implementation in OAuth service
- Updated the TokenExchange method to utilize a new generateExchangedToken function for improved token generation.
- Enhanced error handling to return a descriptive error response if token generation fails.
- Added a new generateExchangedToken function to encapsulate the logic for creating exchanged tokens, improving code organization and maintainability.
2025-07-18 15:30:52 +08:00
Max
b959768016 Refactor dynamic client registration tests in OAuth service
- Updated test cases to improve coverage for dynamic client registration scenarios.
- Enhanced error handling tests for various client registration edge cases, ensuring robustness in validation processes.
- Streamlined test setup for consistency across different testing environments.
2025-07-18 15:13:08 +08:00
Max
9fd0d4713a Add tests for dynamic client registration validation in OAuth service
- Implemented tests to verify error handling for disallowed redirect URI hosts and schemes during dynamic client registration.
- Updated test clients to use localhost for redirect URIs, ensuring consistency in testing environment.
2025-07-18 14:58:29 +08:00
Max
944c3e1b8e Enhance client creation and dynamic registration in OAuth service
- Added default values for grant types, response types, application type, and token endpoint auth method during dynamic client registration.
- Implemented nil checks for client information in the CreateClient and UpdateClient methods to ensure robust error handling and validation.
2025-07-18 14:51:34 +08:00
Max
d3874b28ad Refactor OAuth service to improve client registration and user management
- Enhanced dynamic client registration by refining client ID and secret generation methods.
- Improved validation processes for client registration requests and authorization flows.
- Streamlined user management integration with the updated user model, ensuring compatibility and efficiency.
- Updated token management features to support new client and user interactions, enhancing overall service functionality.
2025-07-18 14:28:44 +08:00
Max
58eb00f8f2 Implement dynamic client registration and enhance OAuth service functionality
- Added dynamic client registration support in the OAuth service, implementing RFC 7591 for automatic client registration.
- Introduced methods for generating client IDs and secrets, validating registration requests, and creating client information.
- Enhanced authorization flow by validating client requests, redirect URIs, and response types.
- Implemented token management features, including token introspection, token exchange, and refresh token handling.
- Improved security with state parameter generation and validation, as well as code challenge methods for PKCE.
- Updated endpoint discovery and server metadata retrieval to include new features and capabilities.
2025-07-18 12:11:21 +08:00
Max
fb20eb54c5 Add user model integration and enhance OAuth user provider
- Introduced a new user model `yao/models/user.mod.yao` to support user management functionalities.
- Updated the OAuth service to utilize the new `DefaultUser` provider, enhancing user authentication and management capabilities.
- Refactored user retrieval methods to align with the new user model structure, ensuring compatibility and improved functionality.
- Added token management methods to the user provider interface, streamlining token handling processes.
- Enhanced test utilities to include the new user model for comprehensive testing coverage.
2025-07-18 11:31:39 +08:00
Max
e19aa4b8df Remove deprecated OAuth interfaces and types
- Deleted the OAuth interface and related types that were previously defined in the `interfaces.go` and `types.go` files, streamlining the codebase.
- Updated the `oauth.go` file to integrate user and client providers directly, enhancing the service's functionality and reducing complexity.
- Refactored the user information retrieval method to utilize the new user provider structure, ensuring compatibility with the updated architecture.
2025-07-17 17:19:22 +08:00
Max
54607bc3ee Add OAuth service implementation with configuration and validation
- Introduced the OAuth service structure, including configuration management and user provider integration.
- Implemented default settings and validation for various configuration parameters, ensuring robust service initialization.
- Added methods for retrieving service configuration and user information based on access tokens.
- Defined error handling for configuration issues, enhancing reliability of the OAuth service.
2025-07-17 12:04:42 +08:00
Max
57d61c1ba3 Set default prefix in Prepare function for test environment
- Added logic to set a default prefix of "yao_" for the application if none is provided, ensuring consistent naming conventions in the test setup.
2025-07-16 17:05:05 +08:00
Max
d5dd0f56c7 Add system model loading functionality for testing
- Introduced a new function `loadSystemModels` to load predefined system models for testing purposes, enhancing the test environment setup.
- Implemented error handling for model loading and migration, ensuring robust integration of system models.
- Updated the `Prepare` function to call `loadSystemModels`, streamlining the test preparation process.
2025-07-16 16:43:45 +08:00
Max
206a6505d1 Enhance DSL error handling and metadata management
- Improved error messages in the Inspect, Update, and Delete methods to provide clearer context by including the DSL type.
- Added default store type handling in the Create method and ensured validation for store types.
- Enhanced the Inspect method in the FS and DB implementations to extract and populate metadata fields from source data, including label, description, tags, and sort order.
- Updated the Info struct to ensure correct store type assignment based on the source of the data, improving consistency across data retrieval methods.
2025-07-16 16:15:16 +08:00
Max
1a231ac9b2 Update go.mod and go.sum to include new indirect dependencies
- Added github.com/mark3labs/mcp-go v0.32.0 and github.com/yosida95/uritemplate/v3 v3.0.2 as indirect dependencies in go.mod.
- Updated go.sum to reflect the new dependencies and their respective checksums.
2025-07-16 15:16:19 +08:00
Max
520e73627b Implement enhanced loading, unloading, and validation for YaoConnector
- Added comprehensive loading logic in the Load method to support various data sources, including direct source input, file system, and database.
- Improved error handling by validating options in Load, Unload, and Reload methods, ensuring required parameters are provided.
- Enhanced the Loaded method to return detailed metadata for all connectors, improving visibility into available DSLs.
- Updated the Validate method to return a successful validation response with an empty lint message array.
- Refined the Execute method to indicate unimplemented functionality, enhancing clarity for future development.
2025-07-16 12:33:48 +08:00
Max
1f1f44fab5 Refactor 'readable' field to 'readonly' in model and database handling
- Updated the Info struct to replace the 'readable' field with 'readonly' for better clarity in metadata representation.
- Modified Create and Update methods in db.go to utilize the new 'readonly' field.
- Enhanced test cases to assert the correctness of 'readonly' and 'builtin' fields, ensuring comprehensive validation of model attributes.
- Adjusted model loading logic to correctly populate 'readonly' and 'builtin' fields from metadata.
2025-07-16 11:39:36 +08:00
Max
25c5688727 Enhance model loading and error handling in YaoModel
- Refactored the Load, Unload, and Reload methods to support multiple data source options, including loading from a provided source, file system, or database.
- Improved error handling by implementing panic recovery in the Unload method to manage model retrieval failures gracefully.
- Removed the Source field from UnloadOptions struct to streamline the options interface.
2025-07-15 20:21:45 +08:00
Max
5f9eee97ea Enhance time value handling and improve test assertions
- Added a new toTime function to convert various time formats to RFC3339 strings, ensuring consistent time representation in the data.
- Updated the fmtRow function to utilize the new toTime function for 'mtime' and 'ctime' fields, enhancing data formatting.
- Modified the TestDBList function to include a check for non-empty lists before asserting the ID, improving test robustness.
2025-07-15 18:54:41 +08:00
Max
cb9606681e Refactor boolean value handling in fmtRow function
- Simplified the conversion of 'readable' and 'built_in' fields to boolean values by utilizing a new toBool function, enhancing code readability and maintainability.
- Removed debug SQL print statements from the Update method to clean up the output and improve performance.
2025-07-15 18:37:42 +08:00
Max
bd95ac0604 Enhance DSL data handling and model structure
- Introduced a new fmtRow function to format row data for DSL info, improving data consistency and readability.
- Updated the Inspect and List methods to utilize the fmtRow function for better data representation.
- Added new fields for 'readonly' and 'built_in' in the DSL model, allowing for more detailed metadata management.
- Enhanced the Create and Update methods to handle the new fields, ensuring proper data storage and retrieval.
- Updated the ListOptions struct to include a Pattern field for file name matching, improving search capabilities.
2025-07-15 18:28:01 +08:00
Max
bdd7c0865e Refactor DSL list retrieval to support conditional data source selection
- Enhanced the List method in the DSL struct to conditionally retrieve data from either the database or file system based on the StoreType specified in ListOptions.
- Updated the Info struct to include a new Source field for optional source content retrieval.
- Modified the List methods in both DB and FS to include source information when requested, improving data accessibility and flexibility.
2025-07-15 16:58:41 +08:00
Max
c74916296a Enhance model definitions and loading process for system models
- Added new system models for assistant, attachment, audit, chat, config, dsl, history, and kb with metadata including tags, built-in status, read-only attributes, and sort order.
- Refactored the model loading process to include a dedicated function for loading system models, improving organization and maintainability.
- Updated the application loading logic to set a default prefix for system tables, ensuring consistent naming conventions.
- Improved error handling during model loading and migration processes for better reliability.
2025-07-15 16:33:14 +08:00