Capture Screen State Node

C

Automation/Computer/Capture

Observes a window or display for an agent: a screenshot, the same screenshot with numbered boxes on every actionable element, and a compact element list (accessibility tree plus optional OCR text) whose ids Click Screen Element accepts

computer_capture_stateautomationLocal only

Inputs10

Outputs8

Security exposure5/10

Packageautomation

Ratings

Scores range from 0 to 10. Higher values mean more impact, exposure, or operational weight.

SecurityAttack surface and exposure impact.

5/10Medium

PrivacyPotential sensitivity of processed data.

3/10Low

PerformanceRuntime or resource pressure.

5/10Medium

GovernancePolicy, audit, or compliance impact.

6/10Medium

ReliabilityOperational stability considerations.

7/10High

CostExternal or compute cost impact.

10/10High

Input Pins

10

▶

Execution

exec_in

Trigger

Session

Struct

session

Computer session handle

SchemaAutomationSessionAutomationSessionAutomationSession11 fields

session_refstringrequired

platformPlatformrequired

enum "Windows", "MacOS", "Linux"

default_delay_msinteger:uint64required

format uint64min 0

click_delay_msinteger:uint64required

format uint64min 0

debug_modebooleanrequired

browser_typeanyOf (2)

Browser context info if browser is attached

variant 1BrowserTypevariant

enum "Chrome", "Firefox", "Edge", "Safari"

variant 2nullvariant

browser_headlessboolean | null

browser_user_data_dirstring | null

current_page_refstring | null

Current page info if a page is open

current_window_handlestring | null

browser_frame_selectorsArray<Selector>

default []

itemsSelectorarray item

kindSelectorKindrequired

variant 1enumvariant

enum "Css", "Xpath", "TextExact", "TestId"...

variant 2constvariant

Visible text contained in the element; script, style and head content is ignored.

const "Text"

variant 3constvariant

ARIA role, optionally filtered by accessible name: `button`, `button|Sign in` (case-insensitive substring), `button|=Sign in` (exact) or `button|/^sign/i` (regex).

const "Role"

variant 4constvariant

Element ref from the latest Browser Snapshot, such as `e12`.

const "Ref"

valuestringrequired

confidencenumber | null

format double

scopestring | null

Target

String

target

focused_window: the focused window outside Flow-Like; window: the window titled below; display: a whole display

Default focused_window

focused_windowwindowdisplay

Window Title

String

window_title

Title (or part of it) of the window for the window target

Display Index

Integer

display_index

Display for the display target

Default 0

Accessibility Elements

Boolean

include_ax

List elements from the accessibility tree

Default true

OCR Text

Boolean

include_ocr

Add recognized text that no accessibility element already covers

Default false

Interactive Only

Boolean

interactive_only

Only list controls that can be clicked, typed into or toggled

Default true

Max Elements

Integer

max_elements

Upper bound on listed elements (1–500)

Default 150

Range 1 to 500Step 1

OCR Languages

String

languages

Comma-separated OCR languages such as en-US, de; empty lets the OS engine choose

Output Pins

8

▶

Execution

exec_out

Continue

Session

Struct

session_out

Computer session handle (pass-through)

SchemaAutomationSessionAutomationSessionAutomationSession11 fields

session_refstringrequired

platformPlatformrequired

enum "Windows", "MacOS", "Linux"

default_delay_msinteger:uint64required

format uint64min 0

click_delay_msinteger:uint64required

format uint64min 0

debug_modebooleanrequired

browser_typeanyOf (2)

Browser context info if browser is attached

variant 1BrowserTypevariant

enum "Chrome", "Firefox", "Edge", "Safari"

variant 2nullvariant

browser_headlessboolean | null

browser_user_data_dirstring | null

current_page_refstring | null

Current page info if a page is open

current_window_handlestring | null

browser_frame_selectorsArray<Selector>

default []

itemsSelectorarray item

kindSelectorKindrequired

variant 1enumvariant

enum "Css", "Xpath", "TextExact", "TestId"...

variant 2constvariant

Visible text contained in the element; script, style and head content is ignored.

const "Text"

variant 3constvariant

ARIA role, optionally filtered by accessible name: `button`, `button|Sign in` (case-insensitive substring), `button|=Sign in` (exact) or `button|/^sign/i` (regex).

const "Role"

variant 4constvariant

Element ref from the latest Browser Snapshot, such as `e12`.

const "Ref"

valuestringrequired

confidencenumber | null

format double

scopestring | null

Image

Struct

image

Screenshot of the target

SchemaNodeImageNodeImageNodeImage1 fields

image_refstringrequired

Annotated Image

Struct

annotated

Screenshot with a numbered box on every listed element

SchemaNodeImageNodeImageNodeImage1 fields

image_refstringrequired

Frame

Struct

frame

Desktop rectangle and pixel size of both images

SchemaScreenFrameScreenFrameScreenFrame7 fields

display_indexinteger | null

format uint32default nullmin 0

xinteger:int32required

format int32

yinteger:int32required

format int32

widthinteger:uint32required

format uint32min 0

heightinteger:uint32required

format uint32min 0

pixel_widthinteger:uint32required

format uint32min 0

pixel_heightinteger:uint32required

format uint32min 0

Elements

Struct Array

elements

Listed elements with role, name, value, states and desktop bounds

SchemaScreenElementScreenElementScreenElement8 fields

idinteger:uint32required

1-based number drawn on the annotated image.

format uint32min 0

sourcestringrequired

`ax` (accessibility tree) or `ocr` (recognized text).

rolestringrequired

namestring | null

default null

valuestring | null

default null

statesArray<string>

default []

itemsstringarray item

bboxInputRectrequired

Visible part of the element.

xinteger:int32required

format int32

yinteger:int32required

format int32

widthinteger:uint32required

format uint32min 0

heightinteger:uint32required

format uint32min 0

centerInputPointrequired

xinteger:int32required

format int32

yinteger:int32required

format int32

Summary

String

summary

One line per element for a language model, e.g. [3] button "Save" @ (812,433)

Generation

Integer

generation

Identifies this capture; Click Screen Element rejects ids from older captures

Node Info

Internal name

computer_capture_state

Category

Automation/Computer/Capture

Version

1