Sync with sipeed/picoclaw upstream/main (~1095 commits since last merge),
including major restructures:
- Launcher migrated to web/backend (#1275): cmd/picoclaw-launcher/* removed.
Our launcher is dropped (MagicForm doesn't use it; web/backend is the
upstream replacement when needed).
- Agent loop split (#2585, 12d5421c): pkg/agent/loop.go deleted, replaced by
pkg/agent/agent_*.go files. Multi-tenancy customizations to be forward-ported
in follow-up commits on this branch.
- Tool packages reorganized (4c133dc2): pkg/tools/{filesystem,web}.go moved
to pkg/tools/{fs,integration}/. Customizations to be forward-ported.
- Channels switched to map+Settings model: MagicFormConfig → MagicFormSettings
registered in config_channel.go alongside other channels; channel constructor
updated to (channelName, channelType, *Config, *MessageBus) factory pattern.
- Config schema updates: removed deprecated AgentDefaults.Model field;
removed Config.Bindings (legacy migration in upstream); SessionConfig now
uses Dimensions instead of DMScope; ExecConfig adds AllowRemote.
Conflicts resolved:
- pkg/agent/loop.go + loop_test.go: accepted upstream deletion (split)
- pkg/tools/{filesystem,web}.go: accepted upstream deletion (moved)
- cmd/picoclaw-launcher/*: dropped (deps deleted upstream, unused by MagicForm)
- cmd/picoclaw/internal/gateway/helpers.go: accepted upstream deletion
- pkg/channels/manager.go: switched to upstream's map-driven channel init
- pkg/config/config.go: merged AgentDefaults fields, kept WorkspaceRoot,
removed deprecated Model
- pkg/bus/types.go: merged OutboundMessage with upstream's new fields
(Context, AgentID, SessionKey, Scope, ContextUsage) plus our magicform
callback fields (Type, Metrics, Progress, Escalation)
- pkg/tools/{cron,shell}.go: combined our customizations with upstream changes
- pkg/skills/installer.go: dropped our LimitReader path (upstream now uses
chunked DownloadToFile which is size-bounded)
The magicform channel was rebuilt to match upstream's new API:
- *config.MagicFormSettings (SecureString token) instead of MagicFormConfig
- Send returns ([]string, error) per upstream Channel interface
- InboundMessage built with Context (with Raw map for tenancy hints) instead
of bus.Peer + Metadata field which were removed.
Multi-tenancy hints (workspace_override, config_dir, allowed_tools,
allowed_skills) are still flowing via Context.Raw — the agent loop side of
the integration is forward-ported in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: support explicit model list providers
* fix(web): preserve explicit model providers
* fix(web): preserve legacy provider prefixes on model updates
fix(models): normalize explicit provider-prefixed ids
fix(api): preserve legacy model updates across providers
fix(agent): preserve config identity for explicit provider refs
* fix ci
* * completed
* * optimzie
* * fix format
* * fix pr check
* try to fix ci
* * Indicates that Windows does not support expos_paths, adding more mount paths for the Linux platform.
* fix isolation startup lifecycle and MCP transport wrapping
* fix isolation startup cleanup and optional Linux mounts
* fix isolation path handling for relative hooks
Preserve relative command and working-directory semantics when Linux isolation wraps subprocesses, and restore absolute argv path exposure to avoid startup regressions. Add hook coverage and docs updates so isolation-enabled process hooks keep working as configured.
* * fix ci
* fix: use per-candidate provider for model_fallbacks
Each fallback model now uses its own api_base and api_key from
model_list instead of inheriting the primary model's provider config.
Previously, a single LLMProvider was created from the primary model's
ModelConfig and reused for all fallback candidates — only the model ID
string was swapped. This caused all fallback requests to be routed to
the primary provider's endpoint, making cross-provider fallback chains
non-functional (e.g., OpenRouter primary with Gemini fallback would
send the Gemini request to OpenRouter's API).
Fix: pre-create a per-candidate LLMProvider at agent initialization
time by looking up each candidate's ModelConfig from model_list. The
fallback run closure now selects the correct provider per candidate
via CandidateProviders map, falling back to agent.Provider when no
override is found.
Fixes#2140
Made-with: Cursor
test: add test for instance.go
fix: fix test
refactor: optimize
fix: fix Golang lint issues
chore: comment cleanup
* refactor: use resolvedModelConfig() instead of buildModelIndex()
* fix
* Add multi-message sending via split marker
* Add marker and length split integration tests
Tests that SplitByMarker and SplitMessage work together correctly, and
that code block boundaries are preserved during marker splitting.
* Simplify message chunking logic in channel worker
Extract splitByLength helper function and remove goto-based control
flow.
The logic now flows more naturally - try marker splitting first, then
fall
back to length-based splitting.
* Update multi-message output instructions in agent context
* Add split_on_marker to config defaults
* Add split_on_marker config option
* Rename 'Multi-Message Sending' setting to 'Chatty Mode'
* Add SplitOnMarker config option
* feat(session): add SessionStore interface and JSONL backend adapter
Extract a SessionStore interface from the methods the agent loop uses
(AddMessage, GetHistory, SetSummary, TruncateHistory, Save, etc.).
Both SessionManager and the new JSONLBackend satisfy this interface,
allowing the persistence layer to be swapped transparently.
JSONLBackend wraps memory.Store and maps its error-returning API to
the fire-and-forget contract that the agent loop expects — write
errors are logged, reads return empty defaults on failure. Save()
triggers compaction to reclaim space after logical truncation.
Part of #1169
* test(session): add JSONLBackend integration tests
8 tests covering the full SessionStore contract through the JSONL
backend: message roundtrip, tool calls, summary, truncation with
compaction, history replacement, empty sessions, session isolation,
and the complete summarization flow (SetSummary → TruncateHistory →
Save).
Includes compile-time interface satisfaction checks for both
SessionManager and JSONLBackend.
Part of #1169
* feat(agent): wire JSONL session store into agent loop
Replace the concrete *SessionManager field with the SessionStore
interface and initialize the JSONL backend by default. Legacy .json
session files are auto-migrated on first startup. Falls back to
SessionManager if the JSONL store cannot be initialized.
The agent loop code (loop.go) requires zero changes — all method
calls work identically through the interface.
Closes#1169
* fix(session): propagate compact error from Save
Save() was swallowing the error returned by Compact and always
returning nil. Callers checking Save's return value would never
see a compaction failure. Return the error directly so the agent
loop can log or handle it as needed.
* feat(session): add Close to SessionStore interface
Add Close() error to SessionStore so callers can release resources
through the interface. JSONLBackend already had Close; this adds
a no-op implementation to SessionManager for compatibility.
* fix(session): close session stores on shutdown and harden migration
- Add Close() to AgentInstance, AgentRegistry, and AgentLoop so JSONL
file handles are released during gateway shutdown and CLI exit.
- Fall back to SessionManager when migration fails, preventing a split
state where some sessions live in JSONL and others remain in JSON.
- Add defer agentLoop.Close() in the CLI agent command path.
- Document SessionStore interface methods (fire-and-forget contract).
* feat(cli): add workspace, config-dir, tools, and skills override flags
Add --workspace, --config-dir, --tools, and --skills flags to
`picoclaw agent` for single-shot invocations. Supports workspace
override with bootstrap file injection (AGENTS.md, IDENTITY.md,
SOUL.md, USER.md), tool allowlisting, and skills filtering.
Also extracts FormatSkillsSummary from SkillsLoader for reuse,
adds SetSkillsFilter to ContextBuilder, and wires SkillsFilter
from agent config into the context builder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(channels): add MagicForm webhook channel plugin
Implement MagicForm channel for webhook-based agentic task delegation.
Supports Bearer token auth, async processing with HTTP callback,
workspace path validation against configurable workspace_root,
bootstrap file injection, tool/skill filtering via metadata, and
per-conversation session key isolation.
Includes Bus() accessor on BaseChannel for direct message publishing,
MagicFormConfig with workspace_root for path traversal prevention,
request body size limits, and TTL cleanup for stale request contexts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(agent): support per-request workspace, tool, and skill overrides
Add workspace override, tool allowlisting, and skills filtering to the
agent loop via processOptions metadata. When a workspace override is
active (e.g. from MagicForm channel), creates isolated SessionManager
and ContextBuilder instances per request.
Threads effSessions/effContextBuilder through all downstream paths
including runLLMIteration, forceCompression, maybeSummarize, and
summarizeSession to ensure full workspace isolation. Adds defense-in-
depth tool execution guard alongside LLM-facing tool definition filter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(auth): add response body size limits to OAuth HTTP calls
Wrap all io.ReadAll(resp.Body) calls in the OAuth flow with
io.LimitReader capped at 1 MB to prevent memory exhaustion from
malicious or unexpectedly large server responses.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(tools): add response body size limits to search providers
Add 2 MB LimitReader to BraveSearch, Tavily, DuckDuckGo, and
Perplexity search providers to prevent memory exhaustion from
unexpectedly large responses. Matches existing pattern used by
GLMSearchProvider.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(tools): add 20 MB file write size limit
Reject writes exceeding 20 MB in WriteFileTool to prevent disk
exhaustion from unexpectedly large LLM-generated file content.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(tools): use crypto/rand for temp file naming
Replace predictable PID+nanosecond temp file names with
cryptographically random hex strings to prevent symlink
pre-placement attacks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(skills): validate GitHub repo format and limit download size
Add regex validation for 'owner/repo' format to prevent URL injection
in InstallFromGitHub. Add 5 MB LimitReader on skill file downloads
to prevent memory exhaustion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(auth): allow OAuth credentials to be overridden via env vars
Add PICOCLAW_OPENAI_CLIENT_ID, PICOCLAW_GOOGLE_CLIENT_ID, and
PICOCLAW_GOOGLE_CLIENT_SECRET env var overrides. Hardcoded values
remain as fallback defaults for backward compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(auth): sanitize OAuth error messages
Replace raw HTTP response bodies in error messages with generic
status-code-only messages. Full response details are logged at
debug level for troubleshooting.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(launcher): HTML-escape error messages in auth callback
Use html.EscapeString on all error text rendered in HTML responses
to prevent XSS via crafted OAuth error parameters or error messages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(launcher): remove internal details from HTTP error responses
Log detailed errors server-side and return generic messages to
clients to prevent information disclosure of file paths and
internal state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(launcher): add HTTP security headers middleware
Add SecurityHeaders middleware setting X-Content-Type-Options,
X-Frame-Options, and Content-Security-Policy on all responses
to mitigate MIME sniffing, clickjacking, and content injection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(skills): validate ClawHub registry base URL scheme
Reject non-HTTPS registry URLs (unless localhost) to prevent
config-based redirection to malicious skill servers. Falls back
to the default https://clawhub.ai with a warning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(skills): prevent workspace skills from shadowing builtins
Builtin skill names are now reserved — workspace and global skills
with the same name as a builtin are skipped with a warning log.
LoadSkill also checks builtins first. This prevents supply chain
attacks where a malicious skill replaces a trusted builtin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(skills): warn when safety metadata is unavailable
Add MetadataAvailable flag to InstallResult. When the registry
metadata fetch fails (silent fallback), the user now sees a
warning that safety checks could not be completed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* security(shell): add opt-in environment variable filtering
When filter_env is enabled in config, shell commands run with a minimal
allowlist of environment variables (PATH, HOME, LANG, TERM, etc. plus
PICOCLAW_* prefix), preventing accidental leakage of sensitive env vars
like API keys to spawned processes. Disabled by default for backward compat.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(config): add workspace-local config.json overlay for per-tenant isolation
Support per-workspace config overrides via --config-dir (CLI) or configDir
(webhook), enabling different API keys, models, and agent settings per tenant.
- Add LoadWorkspaceConfig/MergeWorkspaceConfig with raw JSON overlay to
preserve unmentioned bool fields (prevents clobbering tool enabled flags)
- Add Config.Clone() for safe per-request config copies in gateway mode
- Gateway: per-request provider creation from workspace config, with
effProvider/effModel threaded through all LLM call sites and summarization
- CLI: workspace config merged before provider creation, CLI flags win
- MagicForm: replace inline bootstrap fields with configDir path;
agent loop copies bootstrap files and loads config.json from configDir
- Fix --session flag: format as agent:main:cli:{key} so router honors it
- Fix cron session key: format as agent:main:cron:{id} for isolation
- Add shared CopyBootstrapFiles helper in pkg/agent/bootstrap.go
- Add config/workspace.config.example.json
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(agent): honor workspace config overrides in model routing
selectCandidates uses agent.Model/Candidates baked in at startup, which
ignores per-request workspace config overrides (effProvider/effModel).
When the workspace overrides the provider, skip routing and fallback
candidates entirely to avoid cross-provider credential issues. When only
the model is overridden, allow routing but use the effective model.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix linter errors (gci, gofumpt, golines, govet, unused)
- gci: alphabetize magicform import in gateway/helpers.go
- gofumpt: reformat multi-line function signature in agent/helpers.go
- golines: wrap lines exceeding 120 chars across 5 files
- govet: rename shadowed variables (err → wcErr, s → trimmed)
- unused: remove maybeSummarize, forceCompression, summarizeSession
wrappers superseded by their *With variants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: nuestraai <nuestraai@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: admin-mf <admin@magicform.ai>
Upstream added ThinkingLevel, SummarizeMessageThreshold,
SummarizeTokenPercent, MaxMediaSize, and maybeSummarize.
Our branch added Router, LightCandidates, and selectCandidates.
Both sets of changes are kept. Dead updateToolContexts removed
(upstream deleted it; no callers exist).
* feat: add extended thinking support for Anthropic models
Support configurable thinking levels (off/low/medium/high/xhigh/adaptive)
via `agents.defaults.thinking_level` config field.
- "adaptive": uses Anthropic's adaptive thinking API (Claude 4.6+)
- "low/medium/high/xhigh": uses budget_tokens (all thinking-capable models)
- "off": disables thinking (default)
API constraints handled:
- Temperature cleared when thinking is enabled
- budget_tokens clamped to max_tokens-1
- Thinking response blocks parsed into Reasoning field
Relates to #645, #966
* fix: address PR review feedback for thinking support
- Add ThinkingCapable interface for provider capability detection
- Warn when thinking_level is set but provider doesn't support it
- Warn when temperature is cleared due to thinking enabled
- Adjust budget values per Anthropic best practices (medium=16K, xhigh=64K)
- Add budget clamp warning and 80% threshold warning
- Add parseResponse thinking block tests
- Add thinking_level field to config.example.json
* refactor: move ThinkingLevel from AgentDefaults to ModelConfig
Thinking is a model-level capability, not a global agent property.
Per-model config avoids silent ignoring on non-Anthropic providers
and eliminates spurious warning logs in multi-provider setups.
Addresses PR #1076 review feedback from @yinwm.
instance.go:
- Add Router *routing.Router and LightCandidates []FallbackCandidate
to AgentInstance.
- At agent creation, when routing.enabled and light_model resolves
successfully in model_list, pre-build the Router and resolve the
light model candidates once. If the light model isn't in model_list,
log a warning and disable routing for that agent gracefully.
loop.go:
- Add selectCandidates(agent, userMsg, history) helper.
It calls Router.SelectModel and returns either agent.Candidates /
agent.Model (primary tier) or agent.LightCandidates / light_model
(light tier). Returns primary unchanged when routing is disabled.
- In runLLMIteration, resolve (activeCandidates, activeModel) once
before entering the tool-iteration loop. The model tier is sticky
for the entire turn so a multi-step tool chain doesn't switch
models mid-way.
- Replace hard-coded agent.Candidates / agent.Model references in
callLLM and the debug log with the resolved active values.
The fallback chain and retry logic are untouched. When light_model
returns an error the fallback chain handles escalation normally.
* fix(tools): allow /dev/null redirection and add read/write sandbox split
- Remove deny pattern that incorrectly blocked redirects to /dev/null
- Expand block device write pattern to cover nvme, mmcblk, vd, xvd,
hd, loop, dm-, md, sr and nbd in addition to sd
- Add safe path whitelist for kernel pseudo-devices so workspace path
check does not reject /dev/null, /dev/zero, /dev/random, /dev/urandom,
/dev/stdin, /dev/stdout and /dev/stderr
- Add allow_read_outside_workspace config option (default true) so file
read and list tools are unrestricted while write tools stay sandboxed
Closes https://github.com/sipeed/picoclaw/issues/964
Closes https://github.com/sipeed/picoclaw/issues/965
Signed-off-by: Huang Rui <vowstar@gmail.com>
* feat(tools): add configurable allow patterns and path whitelists
- Add custom_allow_patterns to exec config so users can exempt specific
commands from deny pattern checks
- Add allow_read_paths and allow_write_paths regex lists to tools config
for whitelisting specific paths outside the workspace
- Introduce whitelistFs that wraps sandboxFs and falls through to hostFs
for paths matching whitelist patterns
- Use variadic constructor signatures to keep backward compatibility
Suggested-by: lxowalle
Signed-off-by: Huang Rui <vowstar@gmail.com>
---------
Signed-off-by: Huang Rui <vowstar@gmail.com>
* fix: remove redundant tools definitions from system prompt
Tools are already provided to the LLM via JSON schema through
ToProviderDefs(), so the text-based tools section in the system
prompt is redundant.
This removes the buildToolsSection() logic and the tools field
from ContextBuilder, reducing system prompt length while maintaining
the "ALWAYS use tools" rule reminder.
Fixes#731
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: correct spelling 'initialized' (was 'initialised')
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The configuration field for specifying the model has been renamed from
"model" to "model_name" for better clarity and consistency with the
model_list configuration.
A GetModelName() accessor method has been added to maintain backward
compatibility. Existing configurations using the old "model" field will
continue to work correctly.
This change affects:
- Configuration structure (AgentDefaults struct)
- All references across the codebase
- Documentation in all language variants
- Example configuration files
* fix: add MaxTokens and Temperature fields to AgentInstance and update related logic
* feat: add MaxTokens and Temperature options to SubagentManager and update tool loop logic
* feat: add default temperature handling and update related tests
* feat: allow temperature 0 and distinguish unset
* fix: format MockLLMProvider struct in subagent_tool_test.go
Resolve conflicts in pkg/providers/types.go and pkg/agent/loop.go:
- types.go: use protocoltypes aliases from PR #213, keep fallback types
- loop.go: drop old single-agent createToolRegistry (replaced by multi-agent pattern)
Refactor to align with PR #213 patterns:
- instance.go: use NewExecToolWithConfig (accept full config for deny patterns)
- registry.go: pass full config to NewAgentInstance
- loop.go: add Perplexity web search options to registerSharedTools
