Canvas Visualization Interface and A2UI

What You'll Learn

After completing this lesson, you will be able to:

Configure Canvas Host and deploy custom HTML/CSS/JS interfaces
Use canvas tool to control node Canvas (show, hide, navigate, execute JS)
Master A2UI protocol to push dynamic UI updates from AI
Capture Canvas screenshots for AI context
Understand Canvas security mechanisms and permission controls

Your Current Challenge

You have an AI assistant, but it can only interact with you through text. You want to:

Let AI display visual interfaces like tables, charts, forms
See dynamic UI generated by Agent on mobile devices
Create "app-like" interactive experiences without separate development

When to Use This

Canvas + A2UI is suitable for these scenarios:

Scenario	Example
Data Visualization	Display charts, progress bars, timelines
Interactive Forms	Let users confirm actions, select options
Status Panels	Real-time task progress, system status
Games and Entertainment	Simple games, interactive demos

A2UI vs. Static HTML

A2UI (Agent-to-UI): AI dynamically generates and updates UI, suitable for real-time data
Static HTML: Pre-written interface, suitable for fixed layouts and complex interactions

🎒 Prerequisites

Before starting, please ensure you have completed:

[ ] Gateway is running: Canvas Host starts automatically with Gateway (port 18793)
[ ] Node is paired: macOS/iOS/Android nodes are connected to Gateway
[ ] Node supports Canvas: Confirm node has canvas capability (clawdbot nodes list)

Prerequisites

This tutorial assumes you are familiar with:

Core Concepts

Canvas system contains three core components:

┌─────────────────┐
│   Canvas Host  │ ────▶ HTTP Server (port 18793)
│   (Gateway)   │        └── Serves ~/clawd/canvas/ files
└─────────────────┘
         │
         │ WebSocket Communication
         │
┌─────────────────┐
│    Node App   │ ────▶ WKWebView renders Canvas
│ (iOS/Android) │        └── Receives A2UI pushes
└─────────────────┘
         │
         │ userAction events
         │
┌─────────────────┐
│   AI Agent    │ ────▶ canvas tool calls
│  (pi-mono)   │        └── Pushes A2UI updates
└─────────────────┘

Key Concepts:

Canvas Host (Gateway side)
- Provides static file service: http://<gateway-host>:18793/__clawdbot__/canvas/
- Hosts A2UI server: http://<gateway-host>:18793/__clawdbot__/a2ui/
- Supports hot reload: auto-refresh after file changes
Canvas Panel (Node side)
- macOS/iOS/Android node embeds WKWebView
- Connects to Gateway via WebSocket (real-time reload, A2UI communication)
- Supports eval JS execution, snapshot screen capture
A2UI Protocol (v0.8)
- Agent pushes UI updates via WebSocket
- Supports: beginRendering, surfaceUpdate, dataModelUpdate, deleteSurface

Follow Along

Step 1: Verify Canvas Host Status

Why Ensure Canvas Host is running so nodes can load Canvas content.

bash

# Check if port 18793 is listening
lsof -i :18793

You should see:

text

COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
node     12345  user   16u  IPv6  0x1234      0t0  TCP *:18793 (LISTEN)

Configuration Paths

Canvas Root Directory: ~/clawd/canvas/ (can be changed via canvasHost.root)
Port: 18793 = gateway.port + 4 (can be changed via canvasHost.port)
Hot Reload: Enabled by default (can be disabled via canvasHost.liveReload: false)

Step 2: Create First Canvas Page

Why Create custom HTML interface to display your content on nodes.

bash

# Create Canvas root directory (if it doesn't exist)
mkdir -p ~/clawd/canvas

# Create a simple HTML file
cat > ~/clawd/canvas/hello.html <<'EOF'
<!doctype html>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Hello Canvas</title>
<style>
  body {
    font-family: -apple-system, BlinkMacSystemFont, system-ui, sans-serif;
    padding: 20px;
    background: #000;
    color: #fff;
    text-align: center;
  }
  h1 { color: #24e08a; }
</style>
<h1>🎉 Hello from Canvas!</h1>
<p>This is your first Canvas page.</p>
<button onclick="alert('Button clicked!')">Click Me</button>
EOF

You should see:

text

File created: ~/clawd/canvas/hello.html

Step 3: Display Canvas on Node

Why Let the node load and display the page you just created.

First, find your node ID:

bash

clawdbot nodes list

You should see:

text

ID                                  Name          Type       Capabilities
──────────────────────────────────────────────────────────────────────────
abc123-def456-ghi789               iOS Phone     canvas, camera, screen
jkl012-mno345-pqr678               Android Tab   canvas, camera

Then display Canvas (using iOS node as example):

bash

# Method 1: Via CLI command
clawdbot nodes canvas present --node abc123-def456-ghi789 --target http://127.0.0.1:18793/__clawdbot__/canvas/hello.html

You should see:

A borderless panel pops up on iOS device showing your HTML content
Panel appears near menu bar or mouse position
Content is center-aligned with green heading and button

AI Invocation Example:

AI: I've opened a Canvas panel on your iOS device displaying a welcome page.

Canvas URL Format

Local file: http://<gateway-host>:18793/__clawdbot__/canvas/hello.html
External website: https://example.com (requires node network access)
Return to default: / or empty string to show built-in scaffold page

Step 4: Push Dynamic UI with A2UI

Why AI can push UI updates directly to Canvas without modifying files, suitable for real-time data and interaction.

Method A: Quick Text Push

bash

clawdbot nodes canvas a2ui push --node abc123-def456-ghi789 --text "Hello from A2UI"

You should see:

Canvas displays blue A2UI interface
Text centered: Hello from A2UI

Method B: Complete JSONL Push

Create A2UI definition file:

bash

cat > /tmp/a2ui-demo.jsonl <<'EOF'
{"surfaceUpdate":{"surfaceId":"main","components":[{"id":"root","component":{"Column":{"children":{"explicitList":["title","status","button"]}}}},{"id":"title","component":{"Text":{"text":{"literalString":"A2UI Demo"},"usageHint":"h1"}}},{"id":"status","component":{"Text":{"text":{"literalString":"System Status: Running"},"usageHint":"body"}}},{"id":"button","component":{"Button":{"label":{"literalString":"Test Button"},"onClick":{"action":{"name":"testAction","sourceComponentId":"demo.test"}}}}}]}
{"beginRendering":{"surfaceId":"main","root":"root"}}
EOF

Push A2UI:

bash

clawdbot nodes canvas a2ui push --node abc123-def456-ghi789 --jsonl /tmp/a2ui-demo.jsonl

You should see:

┌────────────────────────────┐
│     A2UI Demo         │
│                        │
│  System Status: Running    │
│                        │
│   [ Test Button ]          │
└────────────────────────────┘

A2UI JSONL Format Explanation

JSONL (JSON Lines) has one JSON object per line, suitable for streaming updates:

jsonl

{"surfaceUpdate":{...}}   // Update surface components
{"beginRendering":{...}}   // Start rendering
{"dataModelUpdate":{...}} // Update data model
{"deleteSurface":{...}}   // Delete surface

Step 5: Execute Canvas JavaScript

Why Execute custom JS in Canvas, such as modifying DOM, reading state.

bash

clawdbot nodes canvas eval --node abc123-def456-ghi789 --js "document.title"

You should see:

text

"Hello from Canvas"

JS Execution Examples

Read element: document.querySelector('h1').textContent
Modify style: document.body.style.background = '#333'
Calculate value: innerWidth + 'x' + innerHeight
Closure execution: (() => { ... })()

Step 6: Capture Canvas Screenshot

Why Let AI see current Canvas state for context understanding.

bash

# Default format (JPEG)
clawdbot nodes canvas snapshot --node abc123-def456-ghi789

# PNG format + max width limit
clawdbot nodes canvas snapshot --node abc123-def456-ghi789 --format png --max-width 1200

# High-quality JPEG
clawdbot nodes canvas snapshot --node abc123-def456-ghi789 --format jpg --quality 0.9

You should see:

text

Canvas snapshot saved to: /var/folders/.../canvas-snapshot.jpg

File path is automatically generated by system, usually in temp directory.

Step 7: Hide Canvas

Why Close Canvas panel and free up screen space.

bash

clawdbot nodes canvas hide --node abc123-def456-ghi789

You should see:

Canvas panel disappears on iOS device
Node status restored (if previously occupied)

Checkpoint ✅

Verify Canvas functionality works properly:

Check Item	Verification Method
Canvas Host running	`lsof -i :18793` has output
Node Canvas capability	`clawdbot nodes list` shows `canvas`
Page loads successfully	Node displays HTML content
A2UI push successful	Canvas displays blue A2UI interface
JS execution returns result	`eval` command returns value
Screenshot generates file	Temp directory has `.jpg` or `.png` file

Common Pitfalls

Foreground/Background Restrictions

iOS/Android nodes: canvas.* and camera.* commands must run in foreground
Background calls return: NODE_BACKGROUND_UNAVAILABLE
Solution: Wake device to foreground

Security Considerations

Directory Traversal Protection: Canvas URL prohibits .. to access parent directories
Custom Scheme: clawdbot-canvas:// is restricted to node internal use
HTTPS Restrictions: External HTTPS URLs require node network permissions
File Access: Canvas Host only allows access to files under canvasHost.root

Debugging Tips

View Gateway logs: clawdbot gateway logs
View node logs: iOS Settings → Debug Logs, Android app logs
Test URL: Access directly in browser http://<gateway-host>:18793/__clawdbot__/canvas/

Summary

In this lesson, you learned:

Canvas Architecture: Understanding the relationship between Canvas Host, Node App, and A2UI protocol
Configure Canvas Host: Adjust root directory, port, and hot reload settings
Create Custom Pages: Write HTML/CSS/JS and deploy to nodes
Use A2UI: Push dynamic UI updates via JSONL
Execute JavaScript: Run code in Canvas, read and modify state
Capture Screenshots: Let AI see current Canvas state

Key Points:

Canvas Host starts automatically with Gateway, no additional configuration needed
A2UI is suitable for real-time data, static HTML for complex interactions
Nodes must be in foreground to execute Canvas operations
Use canvas snapshot to pass UI state to AI

Next Lesson

In the next lesson, we'll learn Voice Wake and Text-to-Speech.
You'll learn:
Configure Voice Wake keywords
Use Talk Mode for continuous voice conversations
Integrate multiple TTS providers (Edge, Deepgram, ElevenLabs)

Appendix: Source Code Reference

Click to expand source code locations

Last updated: 2026-01-27

Function	File Path	Lines
Canvas Host Server	`src/canvas-host/server.ts`	372-441
A2UI Protocol Handler	`src/canvas-host/a2ui.ts`	150-203
Canvas Tool Definition	`src/agents/tools/canvas-tool.ts`	52-179
Canvas Path Constants	`src/canvas-host/a2ui.ts`	8-10

Key Constants:

A2UI_PATH = "/__clawdbot__/a2ui": A2UI host path
CANVAS_HOST_PATH = "/__clawdbot__/canvas": Canvas file path
CANVAS_WS_PATH = "/__clawdbot__/ws": WebSocket hot reload path

Key Functions:

createCanvasHost(): Start Canvas HTTP server (port 18793)
injectCanvasLiveReload(): Inject WebSocket hot reload script into HTML
handleA2uiHttpRequest(): Handle A2UI resource requests
createCanvasTool(): Register canvas tool (present/hide/navigate/eval/snapshot/a2ui_push/a2ui_reset)

Supported Canvas Actions:

present: Display Canvas (optional URL, position, size)
hide: Hide Canvas
navigate: Navigate to URL (local path/HTTP/file://)
eval: Execute JavaScript
snapshot: Capture screenshot (PNG/JPEG, optional maxWidth/quality)
a2ui_push: Push A2UI updates (JSONL or text)
a2ui_reset: Reset A2UI state

Configuration Schema:

canvasHost.root: Canvas root directory (default ~/clawd/canvas)
canvasHost.port: HTTP port (default 18793)
canvasHost.liveReload: Enable hot reload (default true)
canvasHost.enabled: Enable Canvas Host (default true)

A2UI v0.8 Supported Messages:

beginRendering: Start rendering specified surface
surfaceUpdate: Update surface components (Column, Text, Button, etc.)
dataModelUpdate: Update data model
deleteSurface: Delete specified surface

Quick Start

Platforms and Channels

Unlock AI Superpowers

FAQ

Appendix

Canvas Visualization Interface and A2UI

What You'll Learn

Your Current Challenge

When to Use This

🎒 Prerequisites

Core Concepts

Follow Along

Step 1: Verify Canvas Host Status

Step 2: Create First Canvas Page

Step 3: Display Canvas on Node

Step 4: Push Dynamic UI with A2UI

Step 5: Execute Canvas JavaScript

Step 6: Capture Canvas Screenshot

Step 7: Hide Canvas

Checkpoint ✅

Common Pitfalls

Summary

Next Lesson

Appendix: Source Code Reference

Canvas Visualization Interface and A2UI ​

What You'll Learn ​

Your Current Challenge ​

When to Use This ​

🎒 Prerequisites ​

Core Concepts ​

Follow Along ​

Step 1: Verify Canvas Host Status ​

Step 2: Create First Canvas Page ​

Step 3: Display Canvas on Node ​

Step 4: Push Dynamic UI with A2UI ​

Step 5: Execute Canvas JavaScript ​

Step 6: Capture Canvas Screenshot ​

Step 7: Hide Canvas ​

Checkpoint ✅ ​

Common Pitfalls ​

Summary ​

Next Lesson ​

Appendix: Source Code Reference ​

Canvas Visualization Interface and A2UI

What You'll Learn

Your Current Challenge

When to Use This

🎒 Prerequisites

Core Concepts

Follow Along

Step 1: Verify Canvas Host Status

Step 2: Create First Canvas Page

Step 3: Display Canvas on Node

Step 4: Push Dynamic UI with A2UI

Step 5: Execute Canvas JavaScript

Step 6: Capture Canvas Screenshot

Step 7: Hide Canvas

Checkpoint ✅

Common Pitfalls

Summary

Next Lesson

Appendix: Source Code Reference