Enhance Agent and LLM APIs with Agent-to-Agent Communication
- Introduced `ctx.agent` and `ctx.llm` objects in the context API for improved agent-to-agent (A2A) and direct LLM calls. - Added methods for single and parallel agent calls, including `Call`, `All`, `Any`, and `Race`, enabling flexible communication between agents. - Enhanced documentation with detailed method summaries, parameters, and examples for better developer guidance on using the new APIs.
This commit is contained in:
Max
parent
02e813af0b
commit
cff93116ec
2 changed files with 707 additions and 0 deletions
|
|
@ -38,6 +38,8 @@ interface Context {
|
|||
memory: Memory; // Agent memory with four namespaces: user, team, chat, context
|
||||
trace: Trace; // Trace object for debugging and monitoring
|
||||
mcp: MCP; // MCP object for external tool/resource access
|
||||
agent: Agent; // Agent-to-Agent calls (A2A)
|
||||
llm: LLM; // Direct LLM connector calls
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -1795,6 +1797,524 @@ const sample = ctx.mcp.GetSample("echo", "tool", "ping", 0);
|
|||
console.log(sample.name, sample.input); // Sample name and input data
|
||||
```
|
||||
|
||||
## Agent API
|
||||
|
||||
The `ctx.agent` object provides methods to call other agents from within hooks, enabling agent-to-agent communication (A2A). This allows building complex multi-agent workflows where agents can delegate tasks, consult specialists, or orchestrate parallel operations.
|
||||
|
||||
### Methods Summary
|
||||
|
||||
| Method | Description |
|
||||
| ------------------------------- | ---------------------------------------- |
|
||||
| `Call(agentID, messages, opts)` | Call a single agent |
|
||||
| `All(requests, opts?)` | Call multiple agents, wait for all |
|
||||
| `Any(requests, opts?)` | Call multiple agents, first success wins |
|
||||
| `Race(requests, opts?)` | Call multiple agents, first complete wins|
|
||||
|
||||
### Single Agent Call
|
||||
|
||||
#### `ctx.agent.Call(agentID, messages, options?)`
|
||||
|
||||
Calls a single agent and streams the response to the current context's output.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `agentID`: String - The target agent/assistant ID
|
||||
- `messages`: Array - Messages to send to the agent
|
||||
- `options`: Object (optional) - Call options including callback
|
||||
|
||||
**Options:**
|
||||
|
||||
```typescript
|
||||
interface AgentCallOptions {
|
||||
connector?: string; // Override LLM connector
|
||||
mode?: string; // Agent mode ("chat", "task", etc.)
|
||||
metadata?: Record<string, any>; // Custom metadata passed to hooks
|
||||
skip?: {
|
||||
history?: boolean; // Skip loading chat history
|
||||
trace?: boolean; // Skip trace recording
|
||||
output?: boolean; // Skip output to client
|
||||
keyword?: boolean; // Skip keyword extraction
|
||||
search?: boolean; // Skip search
|
||||
content_parsing?: boolean; // Skip content parsing
|
||||
};
|
||||
onChunk?: (msg: Message) => number; // Callback for each message chunk
|
||||
}
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Basic call
|
||||
const result = ctx.agent.Call("specialist.agent", [
|
||||
{ role: "user", content: "Analyze this data" }
|
||||
]);
|
||||
|
||||
// With callback
|
||||
const result = ctx.agent.Call("specialist.agent", messages, {
|
||||
connector: "gpt-4o",
|
||||
onChunk: (msg) => {
|
||||
console.log("Received:", msg.type, msg.props?.content);
|
||||
return 0; // 0 = continue, non-zero = stop
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
|
||||
```typescript
|
||||
interface AgentResult {
|
||||
agent_id: string; // Agent ID that was called
|
||||
response?: Response; // Full agent response
|
||||
content?: string; // Extracted text content
|
||||
error?: string; // Error message if failed
|
||||
}
|
||||
```
|
||||
|
||||
**Message Object (received in onChunk callback):**
|
||||
|
||||
The `onChunk` callback receives a `Message` object with the following structure:
|
||||
|
||||
```typescript
|
||||
interface Message {
|
||||
type: string; // Message type: "text", "thinking", "tool_call", "error", etc.
|
||||
props?: Record<string, any>; // Message properties (e.g., { content: "Hello" })
|
||||
|
||||
// Streaming identifiers
|
||||
chunk_id?: string; // Unique chunk ID (C1, C2, ...)
|
||||
message_id?: string; // Logical message ID (M1, M2, ...)
|
||||
block_id?: string; // Output block ID (B1, B2, ...)
|
||||
thread_id?: string; // Thread ID for concurrent calls (T1, T2, ...)
|
||||
|
||||
// Delta control
|
||||
delta?: boolean; // Whether this is an incremental update
|
||||
delta_path?: string; // Update path (e.g., "content")
|
||||
delta_action?: string; // Update action: "append", "replace", "merge", "set"
|
||||
}
|
||||
```
|
||||
|
||||
Common message types:
|
||||
- `"text"` - Text content (`props.content` contains the text)
|
||||
- `"thinking"` - Reasoning/thinking content (o1, DeepSeek R1 models)
|
||||
- `"tool_call"` - Tool/function call
|
||||
- `"error"` - Error message (`props.error` contains error details)
|
||||
|
||||
### Parallel Agent Calls
|
||||
|
||||
The parallel methods allow calling multiple agents concurrently, similar to JavaScript Promise patterns.
|
||||
|
||||
#### `ctx.agent.All(requests, options?)`
|
||||
|
||||
Executes all agent calls and waits for all to complete (like `Promise.all`).
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `requests`: Array of request objects
|
||||
- `options`: Object (optional) - Global options including callback
|
||||
|
||||
**Request Structure:**
|
||||
|
||||
```typescript
|
||||
interface AgentRequest {
|
||||
agent: string; // Target agent ID
|
||||
messages: Message[]; // Messages to send
|
||||
options?: AgentCallOptions; // Per-request options (excluding onChunk)
|
||||
}
|
||||
|
||||
// Note: Per-request onChunk is NOT supported in batch calls.
|
||||
// Use the global onChunk callback in the second argument instead.
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Call multiple agents in parallel
|
||||
const results = ctx.agent.All([
|
||||
{ agent: "analyzer", messages: [{ role: "user", content: "Analyze X" }] },
|
||||
{ agent: "summarizer", messages: [{ role: "user", content: "Summarize Y" }] }
|
||||
]);
|
||||
|
||||
// Results array matches request order
|
||||
results.forEach((r, i) => {
|
||||
if (r.error) {
|
||||
console.log(`Agent ${r.agent_id} failed:`, r.error);
|
||||
} else {
|
||||
console.log(`Agent ${r.agent_id} response:`, r.content);
|
||||
}
|
||||
});
|
||||
|
||||
// With global callback for all responses
|
||||
const results = ctx.agent.All([
|
||||
{ agent: "agent-1", messages: [...] },
|
||||
{ agent: "agent-2", messages: [...] }
|
||||
], {
|
||||
onChunk: (agentId, index, msg) => {
|
||||
console.log(`Agent ${agentId} [${index}]:`, msg.type, msg.props?.content);
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
#### `ctx.agent.Any(requests, options?)`
|
||||
|
||||
Returns as soon as any agent call succeeds (like `Promise.any`). Other calls continue in background.
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Try multiple agents, use first successful response
|
||||
const results = ctx.agent.Any([
|
||||
{ agent: "primary.agent", messages: [...] },
|
||||
{ agent: "fallback.agent", messages: [...] }
|
||||
]);
|
||||
|
||||
// First successful result is returned
|
||||
const success = results.find(r => !r.error);
|
||||
if (success) {
|
||||
console.log("Got response from:", success.agent_id);
|
||||
}
|
||||
```
|
||||
|
||||
#### `ctx.agent.Race(requests, options?)`
|
||||
|
||||
Returns as soon as any agent call completes, regardless of success/failure (like `Promise.race`).
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Race multiple agents for fastest response
|
||||
const results = ctx.agent.Race([
|
||||
{ agent: "fast.agent", messages: [...] },
|
||||
{ agent: "slow.agent", messages: [...] }
|
||||
]);
|
||||
|
||||
// First completed result (may be error or success)
|
||||
const first = results.find(r => r !== null);
|
||||
console.log("Fastest agent:", first.agent_id);
|
||||
```
|
||||
|
||||
### Use Cases
|
||||
|
||||
```javascript
|
||||
// Use case 1: Specialist consultation
|
||||
function Next(ctx, payload) {
|
||||
const { completion } = payload;
|
||||
|
||||
if (completion?.content?.includes("complex analysis")) {
|
||||
// Delegate to specialist
|
||||
const result = ctx.agent.Call("specialist.analyzer", [
|
||||
{ role: "user", content: completion.content }
|
||||
]);
|
||||
|
||||
return {
|
||||
data: {
|
||||
status: "delegated",
|
||||
specialist_response: result.content
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// Use case 2: Parallel processing
|
||||
function Create(ctx, messages) {
|
||||
const userQuery = messages[messages.length - 1]?.content;
|
||||
|
||||
// Query multiple knowledge sources in parallel
|
||||
const results = ctx.agent.All([
|
||||
{ agent: "kb.technical", messages: [{ role: "user", content: userQuery }] },
|
||||
{ agent: "kb.business", messages: [{ role: "user", content: userQuery }] },
|
||||
{ agent: "kb.legal", messages: [{ role: "user", content: userQuery }] }
|
||||
]);
|
||||
|
||||
// Combine results
|
||||
const combinedKnowledge = results
|
||||
.filter(r => !r.error)
|
||||
.map(r => r.content)
|
||||
.join("\n\n");
|
||||
|
||||
// Add to messages
|
||||
return {
|
||||
messages: [
|
||||
...messages,
|
||||
{ role: "system", content: `Relevant knowledge:\n${combinedKnowledge}` }
|
||||
]
|
||||
};
|
||||
}
|
||||
|
||||
// Use case 3: Fallback strategy
|
||||
function Next(ctx, payload) {
|
||||
if (payload.error) {
|
||||
// Try backup agents
|
||||
const results = ctx.agent.Any([
|
||||
{ agent: "backup.gpt4", messages: payload.messages },
|
||||
{ agent: "backup.claude", messages: payload.messages }
|
||||
]);
|
||||
|
||||
const success = results.find(r => !r.error);
|
||||
if (success) {
|
||||
return { data: { recovered: true, content: success.content } };
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
## LLM API
|
||||
|
||||
The `ctx.llm` object provides direct access to LLM connectors for streaming completions. This allows calling LLM models directly without going through the full agent pipeline, useful for quick completions, model comparisons, or building custom workflows.
|
||||
|
||||
### Methods Summary
|
||||
|
||||
| Method | Description |
|
||||
| --------------------------------- | -------------------------------------- |
|
||||
| `Stream(connector, messages, opts)` | Stream LLM completion |
|
||||
| `All(requests, opts?)` | Call multiple LLMs, wait for all |
|
||||
| `Any(requests, opts?)` | Call multiple LLMs, first success wins |
|
||||
| `Race(requests, opts?)` | Call multiple LLMs, first complete wins|
|
||||
|
||||
### Single LLM Call
|
||||
|
||||
#### `ctx.llm.Stream(connector, messages, options?)`
|
||||
|
||||
Calls an LLM connector with streaming output to the current context's writer.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `connector`: String - The LLM connector ID (e.g., "gpt-4o", "claude-3")
|
||||
- `messages`: Array - Messages to send to the LLM
|
||||
- `options`: Object (optional) - LLM options including callback
|
||||
|
||||
**Options:**
|
||||
|
||||
```typescript
|
||||
interface LlmOptions {
|
||||
temperature?: number; // Sampling temperature (0-2)
|
||||
max_tokens?: number; // Max tokens (legacy, use max_completion_tokens)
|
||||
max_completion_tokens?: number; // Max completion tokens
|
||||
top_p?: number; // Nucleus sampling
|
||||
presence_penalty?: number; // Presence penalty (-2 to 2)
|
||||
frequency_penalty?: number; // Frequency penalty (-2 to 2)
|
||||
stop?: string | string[]; // Stop sequences
|
||||
user?: string; // User identifier for tracking
|
||||
seed?: number; // Random seed for reproducibility
|
||||
tools?: object[]; // Function/tool definitions
|
||||
tool_choice?: string | object; // Tool choice strategy
|
||||
response_format?: { // Response format
|
||||
type: string; // "text" | "json_object" | "json_schema"
|
||||
json_schema?: {
|
||||
name: string;
|
||||
description?: string;
|
||||
schema: object;
|
||||
strict?: boolean;
|
||||
};
|
||||
};
|
||||
reasoning_effort?: string; // For reasoning models (e.g., "low", "medium", "high")
|
||||
onChunk?: (msg: Message) => number; // Callback for each chunk
|
||||
}
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Basic streaming call
|
||||
const result = ctx.llm.Stream("gpt-4o", [
|
||||
{ role: "system", content: "You are a helpful assistant." },
|
||||
{ role: "user", content: "Explain quantum computing" }
|
||||
]);
|
||||
|
||||
// With options and callback
|
||||
const result = ctx.llm.Stream("gpt-4o", messages, {
|
||||
temperature: 0.7,
|
||||
max_tokens: 2000,
|
||||
onChunk: (msg) => {
|
||||
console.log("Chunk:", msg.type, msg.props?.content);
|
||||
return 0; // 0 = continue, non-zero = stop
|
||||
}
|
||||
});
|
||||
|
||||
console.log("Full response:", result.content);
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
|
||||
```typescript
|
||||
interface LlmResult {
|
||||
connector: string; // Connector ID used
|
||||
response?: CompletionResponse; // Full completion response
|
||||
content?: string; // Extracted text content
|
||||
error?: string; // Error message if failed
|
||||
}
|
||||
```
|
||||
|
||||
### Parallel LLM Calls
|
||||
|
||||
The parallel methods allow calling multiple LLM connectors concurrently, useful for model comparison, ensemble methods, or fallback strategies.
|
||||
|
||||
#### `ctx.llm.All(requests, options?)`
|
||||
|
||||
Executes all LLM calls and waits for all to complete (like `Promise.all`).
|
||||
|
||||
**Request Structure:**
|
||||
|
||||
```typescript
|
||||
interface LlmRequest {
|
||||
connector: string; // LLM connector ID
|
||||
messages: Message[]; // Messages to send
|
||||
options?: LlmOptions; // Per-request options (excluding onChunk)
|
||||
}
|
||||
```
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Compare responses from multiple models
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: [...], options: { temperature: 0.7 } },
|
||||
{ connector: "claude-3", messages: [...], options: { temperature: 0.7 } },
|
||||
{ connector: "gemini-pro", messages: [...] }
|
||||
]);
|
||||
|
||||
results.forEach((r) => {
|
||||
console.log(`${r.connector}: ${r.content?.substring(0, 100)}...`);
|
||||
});
|
||||
|
||||
// With global callback
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] }
|
||||
], {
|
||||
onChunk: (connectorId, index, msg) => {
|
||||
console.log(`LLM ${connectorId} [${index}]:`, msg.props?.content);
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
#### `ctx.llm.Any(requests, options?)`
|
||||
|
||||
Returns as soon as any LLM call succeeds (like `Promise.any`).
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Use first successful response from any model
|
||||
const results = ctx.llm.Any([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "gpt-4o-mini", messages: [...] }
|
||||
]);
|
||||
|
||||
const success = results.find(r => !r.error);
|
||||
if (success) {
|
||||
ctx.Send(success.content);
|
||||
}
|
||||
```
|
||||
|
||||
#### `ctx.llm.Race(requests, options?)`
|
||||
|
||||
Returns as soon as any LLM call completes (like `Promise.race`).
|
||||
|
||||
**Example:**
|
||||
|
||||
```javascript
|
||||
// Get fastest response
|
||||
const results = ctx.llm.Race([
|
||||
{ connector: "gpt-4o-mini", messages: [...] }, // Usually faster
|
||||
{ connector: "gpt-4o", messages: [...] } // Usually slower
|
||||
]);
|
||||
|
||||
const first = results.find(r => r !== null);
|
||||
console.log("Fastest model:", first.connector);
|
||||
```
|
||||
|
||||
### Use Cases
|
||||
|
||||
```javascript
|
||||
// Use case 1: Quick classification without full agent pipeline
|
||||
function Create(ctx, messages) {
|
||||
const userMessage = messages[messages.length - 1]?.content;
|
||||
|
||||
// Quick intent classification
|
||||
const result = ctx.llm.Stream("gpt-4o-mini", [
|
||||
{ role: "system", content: "Classify intent as: question, command, or chat" },
|
||||
{ role: "user", content: userMessage }
|
||||
], { temperature: 0, max_tokens: 10 });
|
||||
|
||||
const intent = result.content?.toLowerCase();
|
||||
ctx.memory.context.Set("intent", intent);
|
||||
|
||||
return { messages };
|
||||
}
|
||||
|
||||
// Use case 2: Model comparison for quality assurance
|
||||
function Next(ctx, payload) {
|
||||
const { completion } = payload;
|
||||
|
||||
// Get second opinion from different model
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: payload.messages },
|
||||
{ connector: "claude-3-opus", messages: payload.messages }
|
||||
]);
|
||||
|
||||
// Compare responses
|
||||
const gptResponse = results[0].content;
|
||||
const claudeResponse = results[1].content;
|
||||
|
||||
return {
|
||||
data: {
|
||||
primary: completion.content,
|
||||
comparisons: {
|
||||
gpt4o: gptResponse,
|
||||
claude: claudeResponse
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Use case 3: Ensemble with voting
|
||||
function Create(ctx, messages) {
|
||||
// Get multiple model opinions for important decisions
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] },
|
||||
{ connector: "gemini-pro", messages: [...] }
|
||||
]);
|
||||
|
||||
// Simple majority voting (in real use, implement proper consensus)
|
||||
const responses = results.filter(r => !r.error).map(r => r.content);
|
||||
|
||||
return {
|
||||
messages: [
|
||||
...messages,
|
||||
{
|
||||
role: "system",
|
||||
content: `Multiple model opinions:\n${responses.map((r, i) => `Model ${i+1}: ${r}`).join('\n')}`
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
|
||||
// Use case 4: Fallback with latency optimization
|
||||
function Next(ctx, payload) {
|
||||
if (payload.error) {
|
||||
// Race multiple fallback models
|
||||
const results = ctx.llm.Race([
|
||||
{ connector: "gpt-4o-mini", messages: payload.messages },
|
||||
{ connector: "claude-3-haiku", messages: payload.messages }
|
||||
]);
|
||||
|
||||
const fastest = results.find(r => r !== null);
|
||||
if (fastest && !fastest.error) {
|
||||
ctx.Send(fastest.content);
|
||||
return { data: { recovered: true, model: fastest.connector } };
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
## Hooks
|
||||
|
||||
The Agent system supports two hooks that can be defined in the assistant's `index.ts` file: `Create` and `Next`.
|
||||
|
|
|
|||
|
|
@ -19,6 +19,8 @@ interface Context {
|
|||
trace: Trace; // Tracing API
|
||||
mcp: MCP; // MCP operations
|
||||
search: Search; // Search API
|
||||
agent: Agent; // Agent-to-Agent calls (A2A)
|
||||
llm: LLM; // Direct LLM calls
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -312,3 +314,188 @@ interface SearchResult {
|
|||
error?: string;
|
||||
}
|
||||
```
|
||||
|
||||
## Agent API
|
||||
|
||||
The `ctx.agent` object provides methods to call other agents from within hooks, enabling agent-to-agent communication (A2A).
|
||||
|
||||
### Single Agent Call
|
||||
|
||||
```typescript
|
||||
// Basic call
|
||||
const result = ctx.agent.Call("assistant-id", messages);
|
||||
|
||||
// With options and callback
|
||||
const result = ctx.agent.Call("assistant-id", messages, {
|
||||
connector: "gpt-4o",
|
||||
mode: "chat",
|
||||
metadata: { source: "hook" },
|
||||
skip: { history: false, trace: false, output: false },
|
||||
onChunk: (msg) => {
|
||||
console.log("Received:", msg.type, msg.props);
|
||||
return 0; // 0 = continue, non-zero = stop
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Agent Options
|
||||
|
||||
```typescript
|
||||
interface AgentCallOptions {
|
||||
connector?: string; // Override LLM connector
|
||||
mode?: string; // Agent mode ("chat", "task")
|
||||
metadata?: Record<string, any>; // Custom metadata passed to hooks
|
||||
skip?: {
|
||||
history?: boolean; // Skip loading chat history
|
||||
trace?: boolean; // Skip trace recording
|
||||
output?: boolean; // Skip output to client
|
||||
keyword?: boolean; // Skip keyword extraction
|
||||
search?: boolean; // Skip search
|
||||
content_parsing?: boolean; // Skip content parsing
|
||||
};
|
||||
onChunk?: (msg: Message) => number; // Callback (0=continue, non-zero=stop)
|
||||
}
|
||||
```
|
||||
|
||||
### Parallel Agent Calls
|
||||
|
||||
```typescript
|
||||
// Wait for all agents to complete (like Promise.all)
|
||||
const results = ctx.agent.All([
|
||||
{ agent: "agent-1", messages: [...] },
|
||||
{ agent: "agent-2", messages: [...] }
|
||||
]);
|
||||
|
||||
// Return first successful result (like Promise.any)
|
||||
const results = ctx.agent.Any([
|
||||
{ agent: "agent-1", messages: [...] },
|
||||
{ agent: "agent-2", messages: [...] }
|
||||
]);
|
||||
|
||||
// Return first completed result (like Promise.race)
|
||||
const results = ctx.agent.Race([
|
||||
{ agent: "agent-1", messages: [...] },
|
||||
{ agent: "agent-2", messages: [...] }
|
||||
]);
|
||||
|
||||
// With global callback for all responses
|
||||
const results = ctx.agent.All([
|
||||
{ agent: "agent-1", messages: [...] },
|
||||
{ agent: "agent-2", messages: [...] }
|
||||
], {
|
||||
onChunk: (agentId, index, msg) => {
|
||||
console.log(`Agent ${agentId} [${index}]:`, msg.type);
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Result Structure
|
||||
|
||||
```typescript
|
||||
interface AgentResult {
|
||||
agent_id: string;
|
||||
response?: Response;
|
||||
content?: string;
|
||||
error?: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Message Object (onChunk callback)
|
||||
|
||||
```typescript
|
||||
interface Message {
|
||||
type: string; // "text", "thinking", "tool_call", "error"
|
||||
props?: Record<string, any>; // e.g., { content: "Hello" }
|
||||
chunk_id?: string; // C1, C2, ...
|
||||
message_id?: string; // M1, M2, ...
|
||||
delta?: boolean; // Incremental update flag
|
||||
}
|
||||
```
|
||||
|
||||
## LLM API
|
||||
|
||||
The `ctx.llm` object provides direct access to LLM connectors for streaming completions.
|
||||
|
||||
### Single LLM Call
|
||||
|
||||
```typescript
|
||||
// Basic streaming call
|
||||
const result = ctx.llm.Stream("gpt-4o", [
|
||||
{ role: "user", content: "Hello" }
|
||||
]);
|
||||
|
||||
// With options and callback
|
||||
const result = ctx.llm.Stream("gpt-4o", messages, {
|
||||
temperature: 0.7,
|
||||
max_tokens: 2000,
|
||||
onChunk: (msg) => {
|
||||
console.log("Chunk:", msg.props?.content);
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Parallel LLM Calls
|
||||
|
||||
```typescript
|
||||
// Wait for all LLM calls (like Promise.all)
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] }
|
||||
]);
|
||||
|
||||
// Return first successful result (like Promise.any)
|
||||
const results = ctx.llm.Any([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] }
|
||||
]);
|
||||
|
||||
// Return first completed result (like Promise.race)
|
||||
const results = ctx.llm.Race([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] }
|
||||
]);
|
||||
|
||||
// With global callback
|
||||
const results = ctx.llm.All([
|
||||
{ connector: "gpt-4o", messages: [...] },
|
||||
{ connector: "claude-3", messages: [...] }
|
||||
], {
|
||||
onChunk: (connectorId, index, msg) => {
|
||||
console.log(`LLM ${connectorId} [${index}]:`, msg.type);
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### LLM Options
|
||||
|
||||
```typescript
|
||||
interface LlmOptions {
|
||||
temperature?: number;
|
||||
max_tokens?: number;
|
||||
max_completion_tokens?: number;
|
||||
top_p?: number;
|
||||
presence_penalty?: number;
|
||||
frequency_penalty?: number;
|
||||
stop?: string | string[];
|
||||
user?: string;
|
||||
seed?: number;
|
||||
tools?: object[];
|
||||
tool_choice?: string | object;
|
||||
response_format?: { type: string; json_schema?: object };
|
||||
reasoning_effort?: string;
|
||||
onChunk?: (msg: Message) => number;
|
||||
}
|
||||
```
|
||||
|
||||
### Result Structure
|
||||
|
||||
```typescript
|
||||
interface LlmResult {
|
||||
connector: string;
|
||||
response?: CompletionResponse;
|
||||
content?: string;
|
||||
error?: string;
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue