- Changed the default timeout value from 5 minutes to 2 minutes in the codebase and updated relevant sections in DESIGN_V2.md and README.md to reflect this change.
- Enhanced error handling in dynamic integration tests to account for both max turns exceeded and goal achieved scenarios without checkpoints.
- Improved formatting in the documentation for better clarity and consistency.
- Introduced dynamic testing capabilities, allowing for multi-turn conversations with checkpoints through the new `DynamicRunner`.
- Enhanced the `runSingleTest` method to support dynamic mode, including the execution of before/after scripts and detailed output for dynamic test results.
- Added new output methods in `output.go` for dynamic test start, turns, checkpoints, and results, improving console feedback during testing.
- Updated `DESIGN_V2.md` and `TODO_V2.md` to reflect the new dynamic mode features, including simulator configurations and checkpoint definitions.
- Revised the `Case` struct in `types.go` to include fields for dynamic testing, such as `Simulator`, `Checkpoints`, and `MaxTurns`.
- Added support for generating test cases using agents and scripts through the new `LoadFromAgent` and `LoadFromScript` methods in the loader.
- Enhanced the `RunTests` method to handle different input sources, including agent-driven and script-based test case generation.
- Introduced a `--dry-run` flag to allow users to preview generated test cases without executing them.
- Updated relevant documentation in DESIGN_V2.md and TODO_V2.md to reflect the new features and implementation status.
- Added support for agent-driven assertions in the Asserter, allowing validation of responses using specified agents.
- Introduced the `Use` and `Options` fields in the Assertion struct to facilitate agent configuration.
- Enhanced the `evaluateAssertion` method to handle assertions of type "agent".
- Implemented the `assertAgent` method to manage agent interactions and validation logic.
- Updated `script_assert.go` to include the `assertAgentMethod` for JavaScript API integration.
- Revised documentation in DESIGN_V2.md and TODO_V2.md to reflect the new agent-driven assertion capabilities and implementation status.
- Updated DESIGN_V2.md to reflect changes in the handling of before/after scripts, removing the `scripts:` prefix and clarifying their usage in JSONL test cases.
- Enhanced runner.go to integrate global before/after hooks, ensuring they execute correctly before and after test cases.
- Revised types.go to include new fields for before/after scripts in test case and options structures.
- Improved TODO_V2.md to track the implementation progress of before/after script functionality and related tasks.
- Added utility function LoadAgentTestScripts to facilitate loading of test scripts from the agent's src directory.
- Updated DESIGN_V2.md to introduce support for `before` and `after` scripts in JSONL test cases, detailing their usage and execution order.
- Added examples for defining and utilizing before/after functions, including global initialization and cleanup processes.
- Revised TODO_V2.md to outline tasks for implementing before/after script functionality, ensuring clarity on remaining development efforts.
- Improved overall documentation to facilitate understanding of the new scripting capabilities in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify output formats for console and JSON, including detailed descriptions for standard, dynamic, and parallel modes.
- Updated the console output sections to provide clearer examples and summaries of test results, enhancing readability and usability.
- Modified TODO_V2.md to reflect the change from JSONL to JSON output format for message counts, ensuring consistency in documentation.
- Improved overall documentation to support better understanding of output handling in the Agent Test Framework.
- Revised DESIGN_V2.md to clarify the `input` field's capabilities, allowing for string, single message, or message array formats for conversation context.
- Updated the test case format to reflect the new `input` structure, ensuring compatibility with existing single-turn tests.
- Enhanced TODO_V2.md to indicate the completion of message history support and outline remaining tasks, including options field support and JSONL output format updates.
- Improved documentation to ensure clarity on the new input handling and its implications for agent-driven testing.
- Updated DESIGN_V2.md to introduce message history support in the Agent Test Framework, allowing tests to simulate multi-turn conversations without complex state management.
- Revised the test case format to include a `messages` field, enabling the passing of full conversation history directly to the agent.
- Enhanced TODO_V2.md to outline tasks for implementing message history support, including updates to the test case parser and output formats.
- Improved documentation on agent-driven assertions and error handling to reflect the new capabilities and ensure clarity for developers.
- Added detailed sections on Static and Dynamic modes in DESIGN_V2.md, outlining their characteristics and execution flows for multi-turn testing.
- Introduced a Quick Reference table for format rules, clarifying the usage of flags and assertions in test cases.
- Updated TODO_V2.md to reflect tasks for implementing mode support, including the addition of checkpoints and handling of order constraints in dynamic testing.
- Improved documentation for error handling in both static and dynamic modes, ensuring clarity on expected behaviors during test execution.
- Corrected references in DESIGN_V2.md to ensure consistent usage of agent identifiers, including updates to input sources and simulator configurations.
- Enhanced the documentation in TODO_V2.md with a summary of format rules for agent testing, clarifying the usage of prefixes for various contexts and options.
- Added tasks related to the dynamic simulator implementation and metadata handling to guide future development efforts.
- Updated the DESIGN_V2.md file to clarify the usage of agent-driven assertions in JSONL test cases, including detailed examples and API specifications.
- Introduced a new section on script testing with agent assertions, outlining the implementation and usage of the `t.assert.Agent()` method.
- Modified the TODO_V2.md file to reflect the addition of JSONL support for agent assertions and outlined tasks for further development in this area.
- Introduced a comprehensive standard agent interface for agent-driven features, including generator, simulator, and validator modes.
- Added support for `context.Options` to pass parameters in test cases, allowing for flexible configuration of agent behavior.
- Updated test case format to include options at both the test and per-turn levels, enhancing customization and control over agent interactions.
- Expanded documentation to detail the usage of options in various agent modes, improving clarity for developers and users.
- Further streamlined the LoadWithRoot function by removing redundant checks and improving the overall readability of the path resolution logic.
- Ensured consistent resolution of the absolute path for the configuration root, enhancing maintainability and clarity in the codebase.
