Acting on a Moving Target: Reliability Under Dynamic Web Conditions

For camofox-browser, reliable interaction means more than retrying a failed browser call. The current problem is to identify whether a failure belongs to the control, the tab, the user's browser context, the proxy path, or the browser process, then recover at the smallest scope that preserves trustworthy work. This chapter follows that diagnosis from an action on a changing page through structured errors, serialization, context rotation, and process cleanup. The aim is practical: readers should be able to decide what to refresh, what to recreate, and what a successful response still does not prove.

When a page makes a simple action complicated

A page can be difficult even when the requested action sounds simple. A button may be covered by an overlay, a JavaScript handler may depend on the complete sequence of mouse events, or a single-page application may replace the control between the time it appears in a snapshot and the time the agent tries to use it. A text field can also behave differently depending on whether it accepts a direct fill or needs real keyboard events.

Camofox handles these cases as different action paths rather than as one universal retry. For a normal interaction, it first uses Playwright's ordinary actionability checks: the target should be visible, enabled, attached, and not obscured. If an overlay intercepts the pointer, it can try a forced interaction. If that still fails, it can calculate the control's bounding box and dispatch a native mouse sequence. That sequence moves to the control and sends the down/up events that trigger the browser's ordinary mouse-event path. The fallback is useful for pages with stubborn handlers, but it is not automatically more correct. A forced or native action can bypass an application affordance that the normal path would have respected, so the result still needs to be observed.

Typing has the same distinction. A fill operation sets the value through the input abstraction. Keyboard mode focuses the target and sends characters as real key events, which can matter for contenteditable regions and frameworks that listen to individual keystrokes. Camofox also checks whether the target is a fillable control instead of allowing a request to treat a button, hidden field, or other non-text element as a text box. These checks turn a vague “typing failed” message into a more useful diagnosis: the field may be stale, the selector may be invalid, or the chosen element may not support the requested operation.

The next question is whether the agent's reference is still meaningful. A ref such as e7 comes from an earlier page observation. When the page changes, the server can refresh its ref table before giving up. If the requested ref still cannot be resolved, Camofox returns a stale-ref outcome and directs the caller toward a new snapshot. It can also distinguish a page-change or navigation race from an ambiguous selector or a control that is not actionable. Those categories matter because they imply different next steps.

An ambiguous selector means the request does not identify one control reliably. A stale ref means the agent's earlier observation no longer maps to a current control. A navigation race means the page changed while the operation was being evaluated. An element-actionability failure means the target was found but could not be used in the requested way. Retrying the same bytes is not equally sensible in all four cases. The first asks for a more precise target; the second and third ask for fresh observation; the fourth may ask for a different interaction mode or a different control.

The server's bounded timeouts are part of this diagnosis.