yao/sandbox/docker/chrome/tests/README.md

Chrome Browser Automation Demo Tests

Example scripts demonstrating browser automation inside the sandbox-claude-chrome Docker image.

Scripts

Script	Description
demo-llm-vision.py	LLM-driven universal automation — works with any search engine, no hardcoded selectors. LLM reads page DOM and decides what to click.
demo-baidu.py	Baidu search demo — hardcoded selectors
demo-duckduckgo.py	DuckDuckGo search demo — hardcoded selectors

demo-llm-vision.py

The main demo. Uses a layered architecture where each component does what it's best at:

LLM reads HTML → returns CSS selectors → DOM locates elements → CDP clicks

• Playwright: Opens pages, extracts DOM, keyboard input
• LLM: Reads page structure, returns CSS selectors for target elements (any cheap text model works)
• DOM: Uses LLM's selectors to get precise bounding boxes
• CDP: Chrome DevTools Protocol mouse events (isTrusted=true) for clicking

No hardcoded selectors — LLM figures out the page structure dynamically. Works with Google, Bing, Baidu, DuckDuckGo, Sogou, and any other search engine.

Key Features

• Concurrent LLM Race: DOM is split into chunks, sent to LLM concurrently. First valid response wins — faster than sequential.
• CDP Click: Uses Input.dispatchMouseEvent via Chrome DevTools Protocol. Coordinates match bounding_box() exactly, no offset issues.
• Ctrl+Click New Tab: Search results open in new tabs, keeping the results list intact for clicking more links.
• Fallback Chain: CDP click → PyAutoGUI OS-level click → Playwright .click() → form submit. Always gets through.

Environment Variables

Variable	Description
LLM_API_KEY	API key for the LLM service
LLM_API_BASE	OpenAI-compatible endpoint URL
LLM_MODEL	Model name (e.g. gpt-4o-mini)

Quick Start

# Start the container
docker run -d --name chrome-test \
  --platform linux/amd64 \
  -p 6080:6080 \
  yaoapp/sandbox-claude-chrome:latest

# Wait for VNC to start
sleep 5

# Copy the script
docker cp tests/demo-llm-vision.py chrome-test:/workspace/

# Run with any search engine
docker exec \
  -e LLM_API_KEY="your-key" \
  -e LLM_API_BASE="https://api.openai.com/v1/" \
  -e LLM_MODEL="gpt-4o-mini" \
  chrome-test bash -c \
  'DISPLAY=:99 python3 /workspace/demo-llm-vision.py "https://www.bing.com" "Yao App Engine"'

Open http://localhost:6080 in your browser to watch the automation in real-time via VNC.

Tested Search Engines

Engine	Status	Notes
Bing	Passed	gpt-4o-mini, ~100s
Sogou	Passed	gpt-4o-mini, ~83s
DuckDuckGo	Passed	gpt-4o-mini, ~87s
Baidu	Passed	glm-4-7, ~160s
Google	Passed	May show CAPTCHA on shared IPs

Flow

Phase 1  Open search engine homepage
Phase 2  [LLM Race] Analyze homepage DOM → get input/button selectors
Phase 3  [CDP] Click search input, type query
Phase 4  [CDP] Click search button (fallback: Enter key → form submit)
Phase 5  [LLM Race] Analyze results page DOM → get link selector
Phase 7+ [CDP Ctrl+Click] Open results in new tabs, screenshot, close

Screenshots

Each demo saves screenshots to /workspace/ at key steps:

File	Content
llm-01-homepage.png	Search engine homepage
llm-02-typed.png	Query typed in search box
llm-03-results.png	Search results page
llm-detail.png	Result detail page (new tab)

Notes

• Google may show reCAPTCHA due to IP-based rate limiting. Use a clean IP or proxy.
• Model choice: gpt-4o-mini recommended for speed. Slower models (e.g. glm-4-7) may timeout on large DOMs.
• Concurrent Race splits DOM into ~2000-char chunks and sends all chunks + full DOM to LLM simultaneously. First valid JSON response wins.