以下为本文档的中文说明agent-tui 是 Google Gemini 团队开发的终端用户界面TUI程序化驱动技能。其主要功能是通过编程方式控制、测试和自动化终端 UI 应用程序的交互过程。该技能不是直接面向终端用户的工具而是供 AI 代理使用的底层基础设施。它的核心价值在于让 AI 能够像人类一样操作终端界面程序包括启动守护进程、发送命令、捕获输出、管理会话等。技能提供了会话管理机制使用 session_id 而非 PID 来追踪和重连会话这使得在守护进程崩溃或网络中断后仍能恢复会话。该技能的使用场景包括自动化 CLI/TUI 交互测试、回归测试终端应用程序、验证交互式程序的行为、以及演示和展示 TUI 应用程序的功能。典型的工作流程包括检查守护进程是否存活如果不在则通过 tmux 启动、使用 agent-tui run 执行命令、通过 session_id 管理会话、以及处理守护进程崩溃后的恢复逻辑。该技能还支持对多个终端会话的并发管理适合批量测试场景。核心原则是可靠性和可恢复性确保在各种异常情况下都能维持或恢复与终端会话的连接。该技能特别适合需要大规模自动化测试终端应用的开发团队。Check if daemon is alive, start it in tmux if it is notif ! agent-tui sessions /dev/null 21; thentmux kill-session -t agent-tui 2/dev/null || trueagent-tui daemon stop 2/dev/null || truerm -f /tmp/agent-tui*tmux new-session -d -s agent-tui ‘agent-tui daemon start --foreground /tmp/agent-tui-daemon.log 21’sleep 1fi### Session ID vs PID (Crucial for Reconnection) When agent-tui run returns JSON, it includes both a session_id and a pid. The pid is purely informational (the OS process ID of the child command). You **do not** use the pid to reconnect or issue commands. You must always use the session_id (e.g., --session id). If the daemon crashes (os error 61), the pseudo-terminal is destroyed. Even if the child pid survives as an orphan, you cannot reconnect to it. You must restart the daemon using the workaround above and start a completely new session. ### Testing the Gemini CLI When testing the Gemini CLI with agent-tui, there are several strict requirements to ensure deterministic and accurate behavior: 1. **Build Before Running**: agent-tui runs the built JS files, not TypeScript. You **MUST** run npm run build or npm run build:all after making code changes and before launching the CLI with agent-tui. 2. **Bypass Trust Modals**: Always pass GEMINI_CLI_TRUST_WORKSPACEtrue in the environment. If you dont, any new project-level agents or extensions will trigger a full-screen Acknowledge and Enable modal. This modal steals focus, swallows automation keystrokes, and causes agent-tui wait commands to time out. 3. **Isolated Environments**: If you need to test without real user credentials or existing agents interfering, isolate the global settings using GEMINI_CLI_HOMEsome-test-dir. 4. **Testing State Deltas (e.g., Reloads)**: If you are testing features that report deltas (e.g., /agents reload outputting 1 new local subagent), you **MUST**: - Start the CLI *first* so it establishes its baseline registry. - Use a separate shell command (outside of agent-tui) to write the new agent .md/.toml file. - Use agent-tui type and press to trigger the /agents reload command inside the running session. - (If you add the files before starting the CLI, they become part of the baseline and wont trigger the delta logic). bash # Example: Standard isolated run (sandboxed config bypass trust modals) env GEMINI_CLI_TRUST_WORKSPACEtrue GEMINI_CLI_HOMEtest-gemini-home agent-tui run -d $(pwd) node packages/cli/dist/index.jsTerminal Automation MasteryPrerequisitesSupported OS: macOS or Linux (Windows not supported yet).Verify install:agent-tui--versionIf not installed, use one of:# Recommended: one-line install (macOS/Linux)curl-fsSLhttps://raw.githubusercontent.com/pproenca/agent-tui/master/install.sh|sh# Package managernpmi-gagent-tuipnpmadd-gagent-tui bunadd-gagent-tui# Build from sourcecargoinstall--githttps://github.com/pproenca/agent-tui.git--pathcli/crates/agent-tuiIf you used the install script, ensure~/.local/binis on your PATH.Philosophy: Why Terminal Automation Is DifferentTerminal UIs arestateless from the observer’s perspective. Unlike web browsers with a persistent DOM, terminal automation works with a constantly-refreshed character grid. This fundamental difference shapes everything:Web AutomationTerminal AutomationDOM persists across interactionsScreen buffer is redrawn constantlySelectors are stableText positions may shiftQuery once, act many timesMust re-verify before EVERY actionNetwork events signal completionMust detect visual stabilityThe Core Insight: agent-tui gives you vision without memory. Each screenshot is a fresh observation. Previous state means nothing after the UI changes. This isn’t a limitation—it’s the nature of terminal interaction.Mental Model: The Feedback LoopThink of terminal automation as aclosed-loop control system:┌──────────────────────────────────────────────┐ │ │ ▼ │ OBSERVE ──► DECIDE ──► ACT ──► WAIT ──► VERIFY ───┘ │ │ │ │ └─────── NEVER skip ◄────────────────────┘Each phase is mandatory.Skipping verification is the #1 cause of flaky automation.The “Fresh Eyes” PrincipleEvery time you need to interact with the UI:Take a fresh screenshot— your previous one is now staleLocate your target visually— text positions may have changedVerify the state— the UI may have changed unexpectedlyAct only when stable— animations and loading states cause failuresThis feels slower, but it’s the only reliable approach. Optimistic reuse of stale state causes intermittent failures that are painful to debug.Critical Rules (Non-Negotiable)RULE 1: Atomic Execution (No Pipelining)You are FORBIDDEN from chaining commands with(e.g.,type x press Enter wait). Modals or UI updates can intercept your keystrokes. You MUST execute one atomic action, wait, screenshot, and verify before taking the next action in a new turn.RULE 2: Re-snapshot after EVERY actionThe UI state is invalidated by any change. Always take a fresh screenshot before acting again.RULE 3: Never act on unstable UIIf the UI is animating, loading, or transitioning,wait --stablefirst. Acting during transitions because race conditions.RULE 4: Verify before claiming successUsewait expected text --assertto confirm outcomes. Don’t assume an action worked—prove it.RULE 5: Error RecoveryIf awaitcommand times out, DO NOT blindly restart or kill the session. Executescreenshotto visually diagnose what unexpected UI element (modal, error dialog, lost focus) intercepted the flow.RULE 6: Clean up sessionsAlways end withagent-tui kill. Orphaned sessions consume resources and can interfere with future runs.Decision FrameworkWhich Screenshot Mode?Usescreenshot --format jsonwhen parsing automation output, or plainscreenshotfor human readable text.How to Wait?What are you waiting for? │ ├─► Specific text to appear │ └─► wait text --assert (fails if not found) │ ├─► Specific text to disappear │ └─► wait text --gone --assert │ ├─► UI to stop changing (animations, loading) │ └─► wait --stable │ └─► Multiple conditions └─► Chain waits sequentiallyHow to Act?What do you need to do? │ ├─► Type text into the terminal │ └─► type text │ ├─► Send keyboard shortcuts/navigation │ └─► press CtrlC or press ArrowDown EnterCore WorkflowThe canonical automation loop:# 1. START: Launch the TUI appagent-tui runcommand[-- args...]# 2. OBSERVE: Get current UI stateagent-tui screenshot--formatjson# 3. DECIDE: Based on text, determine next action# (This happens in your head/code)# 4. ACT: Execute the actionagent-tuitypetextagent-tui press Enter# 5. WAIT: Synchronize with UI changesag ent-tuiwaitExpected--assert# or wait --stable# 6. VERIFY: Confirm the outcome (often combined with step 5)# If verification fails, handle the error# 7. REPEAT: Go back to step 2 until done# 8. CLEANUP: Always clean upagent-tuikillAnti-Patterns (What NOT to Do)❌ Acting During Animation/Loading# WRONG: Acting immediately on dynamic UIagent-tui run my-app agent-tui screenshot--formatjson# UI might still be loading!agent-tuitypevalue# ❌ Might miss the input field# RIGHT: Wait for stability firstagent-tui run my-app agent-tuiwait--stable# Let UI settleagent-tui screenshot--formatjson# Now its reliableagent-tuitypevalue❌ Assuming Success Without Verification# WRONG: Assuming the type workedagent-tuitypevalueagent-tui press Enter# ...proceed as if success... # ❌ What if it failed silently?# RIGHT: Verify the outcomeagent-tuitypevalueagent-tui press Enter agent-tuiwaitSuccess--assert# ✓ Proves the action worked❌ Skipping Cleanup# WRONG: Forgetting to kill the sessionagent-tui run my-app# ...do stuff...# script ends # ❌ Session left running!# RIGHT: Always clean upagent-tui run my-app# ...do stuff...agent-tuikill# ✓ Clean exitBefore You Start: Clarify RequirementsBefore automating any TUI, gather this information:Command: What exactly to run? (my-app --flagornpm start?)Success criteria: What text/state indicates success?Input sequence: What keystrokes/data to enter, in what order?Safety: Is it safe to submit forms, delete data, etc.?Auth: Does it need login? Test credentials?Live preview: Does the user want to watch? (agent-tui live start --open)If any of these are unclear, ask before running.Demo Mode: Showing What agent-tui Can DoWhen a user asks what agent-tui is, wants a demo, or asks “show me how it works”:Don’t explain—demonstrate.Actions speak louder than words.Use the live previewso they can watch in real-time.Runtop—it’s universal and shows dynamic real-time updates.Quick demo trigger phrases:“What is agent-tui?” / “What does agent-tui do?”“Demo agent-tui” / “Show me agent-tui”“How does agent-tui work?” / “See it in action”Failure RecoverySymptomDiagnosisSolution“Text not found”Stale view or text movedRe-snapshot, locate text againWait times outUI didn’t reach expected stateCheck screenshot, verify expectations“Daemon not running”Daemon crashed or not startedagent-tui daemon startUnexpected layoutWrong terminal sizeagent-tui resize --cols 120 --rows 40Session unresponsiveApp crashed or hungagent-tui kill, then re-runRepeated failuresSomething fundamentally wrongStop after 3-5 attempts, ask userSelf-Discovery: Use --helpYou don’t need to memorize every flag. The CLI is self-documenting:agent-tui--help# List all commandsagent-tui run--help# Options for runagent-tui screenshot--help# Options for screenshotagent-tuiwait--help# Options for waitWhen in doubt, ask the CLI.This skill teacheswhenandwhyto use commands. For exact flags and syntax,--helpis authoritative.Quick Reference# Start appagent-tui runcmd[-- args]# Launch TUI under control# Observeagent-tui screenshot# Plain text viewagent-tui screenshot--formatjson# Machine-readable output# Actagent-tui press Enter# Press key(s)agent-tui press CtrlC# Keyboard shortcutsagent-tuitypetext# Type text# Wait/Verifyagent-tuiwaittext--assert# Wait for text, fail if not foundagent-tuiwaittext--gone--assert# Wait for text to disappearagent-tui w ait--stable# Wait for UI to stop changing# Manageagent-tui sessions# List active sessionsagent-tui live start--open# Start live previewagent-tuikill# End current session3c:[“,,,L3f”,null,{“content”:“$40”,“frontMatter”:{“name”:“agent-tui”,“description”:Main Agents: Do NOT use this skill directly. If you need to test the TUI, invoke thetui_testersubagent. Drive terminal UI (TUI) applications programmatically for testing, automation, and inspection. Use when: automating CLI/TUI interactions, regression testing terminal apps, or verifying interactive behavior. Also use when: user asks \“what is agent-tui\”, \“what does agent-tui do\”, \“demo agent-tui\”, \“show me agent-tui\”, \“how does agent-tui work\”, or wants to see it in action.}}]3d:[“KaTeX parse error: Expected }, got EOF at end of input: …,children:[[”,“div”,null,{“className”:“flex items-center justify-between border-b border-border bg-muted/30 px-4 py-2.5”,“children”:[[“KaTeX parse error: Expected }, got EOF at end of input: …,children:[”,“span”,null,{“className”:“truncate text-xs font-medium text-muted-foreground”,“children”:“同仓库更多 Skills”}]}],[“KaTeX parse error: Expected EOF, got } at position 88: …ldren:同仓库}]]}̲],[”,“div”,null,{“className”:“p-4 sm:p-5”,“children”:[[“,h2,null,id:related−skills−heading,className:text−2xlfont−semiboldtracking−normaltext−foreground,children:同仓库更多Skills],[,h2,null,{id:related-skills-heading,className:text-2xl font-semibold tracking-normal text-foreground,children:同仓库更多 Skills}],[,h2,null,id:related−skills−heading,className:text−2xlfont−semiboldtracking−normaltext−foreground,children:同仓库更多Skills],[”,“div”,null,{“className”:“mt-4 grid gap-3 sm:grid-cols-2”,“children”:[“L41,L41,L41,L42”,“L43,L43,L43,L44”,“L45,L45,L45,L46”]}]]}]]}]47:I[206516,[“/_next/static/chunks/051aanbhrv4br.js”,“/_next/static/chunks/0mizr60h7ayzt.js”,“/_next/static/chunks/0v9lm1dmbdoo-.js”,“/_next/static/chunks/0rxr1j1j3j-.r.js”,“/_next/static/chunks/02ftybezfvqjd.js”,“/_next/static/chunks/0.v9ksvnnj8ia.js”,“/_next/static/chunks/0bn6id96nx3k.js,“/_next/static/chunks/13ybnhn37c.tc.js”,“/_next/static/chunks/0_fnrdtruz8uf.js”,“/_next/static/chunks/0r6l15utt1mwb.js”,“/_next/static/chunks/0dm9a5into854.js”,/_next/static/chunks/07k6hqoibtcn.js”,“/next/static/chunks/0b4cao.4y…j.js”,“/_next/static/chunks/02i-n28z7kjd0.js”],“default”]
