Type Alias: IntentEvent
IntentEvent = {
at:number;on:boolean;type:"armed"; } | {at:number;mode:Mode;type:"mode"; } | {at:number;trigger:"talk"|"ink"|"shot"|"contribution"|"explicit";type:"thread-open"; } | {at:number;reason:"send"|"cancel"|"timeout";type:"thread-close"; } | {at:number;segment:number;type:"talk-start"; } | {at:number;ms:number;segment:number;type:"talk-end"; } | {at:number;segment:number;text:string;type:"transcript-delta"; } | {at:number;correction?:boolean;latencyMs:number;model:string;segment:number;text:string;type:"transcript-final";words?:TranscriptWord[]; } | {at:number;segment:number;text:string;type:"segment-replace";words?:TranscriptWord[]; } | {at:number;bounds:Rect;points:number;type:"stroke"; } | {at:number;auto:boolean;reason?:"navigation";type:"ink-clear"; } | {at:number;from:string;kind?:"push"|"replace"|"traverse"|"reload"|"hash";tab?:TabRecord;to:string;type:"navigation"; } | {at:number;from:string;fromTab?:number;tab?:TabRecord;to:string;toTab?:number;type:"tab-switch"; } | {at:number;components:LocatedComponent[];marker:string;origin?:"paste";path?:string;rect:Rect;share?:ShotShare;takenAt?:number;thumb?:string;type:"shot";viewport?:boolean; } | {at:number;marker:string;type:"shot-drop"; } | {at:number;from:number;instruction:string;latencyMs?:number;model?:string;original:string;patch?:string;scope?: {fromLine:number;toLine:number; };to:number;type:"correction";via:"speech"|"typed"; } | {at:number;type:"correction-undo"; } | {at:number;cadenceMs?:number;mode?:VideoCaptureMode;on:boolean;ordinal?:number;type:"video-share"; } |object&AppSelection| {at:number;marker?:string;type:"app-selection-drop"; } |object&CodeSelection| {at:number;marker:string;type:"code-selection-drop"; } | {at:number;type:"turn-pause"; } | {at:number;type:"turn-resume"; } | {at:number;text:string;type:"note"; } | {at:number;segment?:number;text:string;type:"linter-note"; } | {args:Record<string,unknown>;at:number;tool:string;type:"linter-tool-call"; } | {at:number;ok:boolean;summary:string;tool:string;type:"linter-tool-result"; } | {at:number;segment?:number;type:"linter-turn-complete"; }
Defined in: aiui-lowering-pipeline/src/types.ts:192
Union Members
Type Literal
{ at: number; on: boolean; type: "armed"; }
Type Literal
{ at: number; mode: Mode; type: "mode"; }
Type Literal
{ at: number; trigger: "talk" | "ink" | "shot" | "contribution" | "explicit"; type: "thread-open"; }
Type Literal
{ at: number; reason: "send" | "cancel" | "timeout"; type: "thread-close"; }
Type Literal
{ at: number; segment: number; type: "talk-start"; }
Type Literal
{ at: number; ms: number; segment: number; type: "talk-end"; }
Type Literal
{ at: number; segment: number; text: string; type: "transcript-delta"; }
Type Literal
{ at: number; correction?: boolean; latencyMs: number; model: string; segment: number; text: string; type: "transcript-final"; words?: TranscriptWord[]; }
at
at:
number
correction?
optionalcorrection?:boolean
Set when this segment was spoken while a correction target was lassoed.
latencyMs
latencyMs:
number
model
model:
string
segment
segment:
number
text
text:
string
type
type:
"transcript-final"
words?
optionalwords?:TranscriptWord[]
Word-level timestamps + confidence, when the transcriber reports them.
Type Literal
{ at: number; segment: number; text: string; type: "segment-replace"; words?: TranscriptWord[]; }
at
at:
number
Replace a segment's transcript WHOLESALE — the panel's segment editor (fixing bad STT, pasting text) speaks this. The compiler treats it as latest-wins IN PLACE: the segment's text item keeps its stream position, its text (and word timestamps, when provided — best-effort re-timestamped by the editor) are superseded, and the timestamp interleave then reflows any anchored shots against the new words. The original transcript stays in the stream for the trace, exactly like a dropped shot.
segment
segment:
number
text
text:
string
type
type:
"segment-replace"
words?
optionalwords?:TranscriptWord[]
Type Literal
{ at: number; bounds: Rect; points: number; type: "stroke"; }
Type Literal
{ at: number; auto: boolean; reason?: "navigation"; type: "ink-clear"; }
at
at:
number
auto
auto:
boolean
reason?
optionalreason?:"navigation"
Why the ink went away, when a cause beyond the user's ✕ / the fade is worth recording. "navigation": the page SPA-navigated and strokes must not float over a route they weren't drawn on — the logged strokes stay in the stream (correctly attributed by their position before the navigation event); screenshots remain the durable form of deixis.
type
type:
"ink-clear"
Type Literal
{ at: number; from: string; kind?: "push" | "replace" | "traverse" | "reload" | "hash"; tab?: TabRecord; to: string; type: "navigation"; }
at
at:
number
The page navigated within the same document mid-turn — an SPA router push, a hash jump to a section, a back/forward traversal (the intent client's navigation watcher). Context riding a turn, never a turn opener (the app-selection rule): emitted only while a thread is open. Ordering in the log is the attribution: strokes, shots, and selections before this event belong to from, after it to to — which also makes the stream self-describing after the hello's one-shot location.href snapshot goes stale.
This is the first of the context-boundary family — the host that drives real tabs adds the tab-switch sibling below.
from
from:
string
location.href before.
kind?
optionalkind?:"push"|"replace"|"traverse"|"reload"|"hash"
How it happened, when the watcher could cheaply attribute it.
tab?
optionaltab?:TabRecord
The DESTINATION tab's full record, when the client gathered one.
to
to:
string
location.href after.
type
type:
"navigation"
Type Literal
{ at: number; from: string; fromTab?: number; tab?: TabRecord; to: string; toTab?: number; type: "tab-switch"; }
at
at:
number
The user changed WHICH TAB they are looking at mid-turn — a different boundary from navigation (same tab navigating in place). Its own event so the compiler can phrase "you switched tabs" distinctly from "the page navigated", and so the tab identities travel: fromTab/ toTab are the driver's tab handles (CDP targetId-derived ids / MV3 chrome.tabs ids), from/to the two tabs' location.href. Like navigation, it is context riding a turn — emitted only while a thread is open, and its POSITION is the attribution (content above it belongs to the tab you left). See the browser-extension intent-tool proposal §2 (git history).
from
from:
string
location.href of the tab left behind.
fromTab?
optionalfromTab?:number
The driver's handle for the tab left behind, when known.
tab?
optionaltab?:TabRecord
The DESTINATION tab's full record, when the client gathered one.
to
to:
string
location.href of the tab switched to.
toTab?
optionaltoTab?:number
The driver's handle for the tab switched to, when known.
type
type:
"tab-switch"
Type Literal
{ at: number; components: LocatedComponent[]; marker: string; origin?: "paste"; path?: string; rect: Rect; share?: ShotShare; takenAt?: number; thumb?: string; type: "shot"; viewport?: boolean; }
at
at:
number
components
components:
LocatedComponent[]
marker
marker:
string
Ordinal token, e.g. "shot_1" — identifier-shaped on purpose: it names the shot across the stream (the key a shot-drop retracts by) and surfaces in the [screenshot shot_1 located at MISSING] fallback when the pixels were never captured. (The retired Option-C rendering also used it as a {shot_n} body token + meta key — see ComposedIntent.prompt.)
origin?
optionalorigin?:"paste"
Where the pixels came from, when NOT a screen capture: "paste" = the user pasted an image into the turn (the panel's segment editor / end-of-turn paste). Same marker space, same disk blob, same takenAt anchoring — but the lowering labels it a pasted image, so the model never mistakes clipboard content for what was on screen.
path?
optionalpath?:string
Absolute path of the saved image on disk (the thing the prompt hands the session).
rect
rect:
Rect
share?
optionalshare?:ShotShare
Present when the screen share's sampler took this frame rather than a human pressing S. Sampled frames are shots in every other respect — same marker space, same disk blob, same labeled injection into the linter, same takenAt anchoring — so the model sees one kind of image, and this descriptor says which of them came from a sequence.
takenAt?
optionaltakenAt?:number
Wall-clock of the capture GESTURE (pointerup / S keydown) — not of this event's emission, which trails it by the compositor wait + encode (and by the getDisplayMedia picker on the first shot). The compiler uses it to place the shot INSIDE a still-open segment's text via the transcript-delta timeline; absent (legacy streams, idle shots) → arrival-order placement.
thumb?
optionalthumb?:string
Data-URL thumbnail (absent when no capture stream was granted).
type
type:
"shot"
viewport?
optionalviewport?:boolean
True for a whole-viewport shot (S). Viewport shots deliberately skip the locator and render with no element metadata — "everything" is not a useful point of reference.
Type Literal
{ at: number; marker: string; type: "shot-drop"; }
at
at:
number
Retract a shot from the turn (the preview thumbnail's ✕). Append-only like everything else: the shot event (and any uploaded bytes) stay in the stream and the trace; composeIntent — shared with the channel's lowering — just excludes the marker from the composition.
marker
marker:
string
type
type:
"shot-drop"
Type Literal
{ at: number; from: number; instruction: string; latencyMs?: number; model?: string; original: string; patch?: string; scope?: { fromLine: number; toLine: number; }; to: number; type: "correction"; via: "speech" | "typed"; }
at
at:
number
from
from:
number
Character range in the rendered transcript at selection time.
instruction
instruction:
string
latencyMs?
optionallatencyMs?:number
model?
optionalmodel?:string
Corrector model + its latency, for the timing pane.
original
original:
string
patch?
optionalpatch?:string
The V4A patch the correction micro-pipeline produced (see patch.ts). Absent when the pipeline failed — appliers then fall back to replacing original with instruction.
scope?
optionalscope?:object
The window of transcript lines (text items, [fromLine, toLine)) this correction is scoped to — the chunk that was active in the editor. The corrector model sees ONLY these lines and the plain-replacement fallback searches only inside them; absent → the whole transcript (pre-chunk-editor events, and the engine-level tests).
scope.fromLine
fromLine:
number
scope.toLine
toLine:
number
to
to:
number
type
type:
"correction"
via
via:
"speech"|"typed"
Type Literal
{ at: number; type: "correction-undo"; }
at
at:
number
Undo the most recent still-active correction (LIFO — a stack pop). Append-only like everything else: the correction event stays in the stream and the trace; composeIntent — shared with the channel's lowering — pops it from the applied set, so the preview AND the lowered prompt agree about what Escape took back.
type
type:
"correction-undo"
Type Literal
{ at: number; cadenceMs?: number; mode?: VideoCaptureMode; on: boolean; ordinal?: number; type: "video-share"; }
at
at:
number
The screen share toggled (V). While on, the client samples the display-capture stream on a cadence and each sampled frame enters the turn as an ordinary shot — same marker space, same disk blob, same injection into the live linter. So this event is not the frames; it is the bracket around them, and it records the terms the sequence was taken under.
cadenceMs?
optionalcadenceMs?:number
The cadence ceiling in ms (the HUD's slider).
mode?
optionalmode?:VideoCaptureMode
Which sampling discipline was in force.
on
on:
boolean
ordinal?
optionalordinal?:number
The share's ordinal (1 = the turn's first V), on the on event.
type
type:
"video-share"
object & AppSelection
Type Literal
{ at: number; marker?: string; type: "app-selection-drop"; }
at
at:
number
Retract exactly one app selection (the chip's ✕ / a cleared watcher): the one named by marker. Append-only like everything else: the app-selection events stay in the stream and the trace; composition just excludes the marker. A markerless drop (pre-marker traces) retracts the most recent selection, the legacy behavior.
marker?
optionalmarker?:string
type
type:
"app-selection-drop"
object & CodeSelection
Type Literal
{ at: number; marker: string; type: "code-selection-drop"; }
at
at:
number
Retract a code selection from the turn (the chip's ✕ — the same gesture as deleting a screenshot). Append-only like everything else: the code-selection event stays in the stream and the trace; composeIntent just excludes the marker from the composition.
marker
marker:
string
type
type:
"code-selection-drop"
Type Literal
{ at: number; type: "turn-pause"; }
at
at:
number
The client PAUSED collection mid-turn (the panel's ⏸ / b — and, when it lands, the oracle detour): from here to the matching turn-resume, no new audio, shots, or selections enter the turn. A bracket in the stream, deliberately REASON-FREE — a manual pause and an oracle detour read identically; the panel's banner is where the reason lives, never the trace (owner, 2026-07-30). Never composed: the lowering skips it. Boundary events (navigation / tab-switch) are suppressed while paused; the client compares tab identity at resume and emits at most one boundary AFTER the turn-resume (the intent client's capture lanes). A thread may close while paused — the close is the outer bracket, so an unmatched turn-pause before a thread-close is well-formed.
type
type:
"turn-pause"
Type Literal
{ at: number; type: "turn-resume"; }
Type Literal
{ at: number; text: string; type: "note"; }
Type Literal
{ at: number; segment?: number; text: string; type: "linter-note"; }
A prompt-linter observation — the realtime model's spoken diagnostic, folded into the stream so the trace and the preview both carry what the linter said. NEVER composed into the prompt: the compiler's fold skips every linter-* kind (the linter observes the briefing; it does not write it). segment correlates the note to the talk window it lints.
Type Literal
{ args: Record<string, unknown>; at: number; tool: string; type: "linter-tool-call"; }
The linter asked to use a tool (e.g. read_file) — the request half, recorded first-class so the trace shows exactly what the linter did. Trace/debug material only: the compiler skips it, and the client renders no chip for it (the trace viewer is its surface).
Type Literal
{ at: number; ok: boolean; summary: string; tool: string; type: "linter-tool-result"; }
The tool's answer to a linter-tool-call — summary is a short human gloss ("src/x.ts — 4.1KB" / an error string), never the content (which lives in the trace stage's data). Compiler-skipped like its request.
Type Literal
{ at: number; segment?: number; type: "linter-turn-complete"; }
A lint turn completed: the model finished replying to a lint now (the linter's one turn trigger since the overhear retirement, 2026-07-19) and the floor is free. Purely informational — the client's pulse settles on it; the linter STAYS ON (talk reopens the window, the button lints again). Compiler-skipped like every linter-* kind. segment anchors the turn it completed, when known.