Ken Pfeuffer

In the history of Human-Computer Interaction (HCI), major interfaces have been based on the coordination of eyes and hands, following a clear division of labour: hands provide commands via input devices, while eyes perceive the visual output on a screen. This has been a long-lasting paradigm transcending eras of computing — from command-line interfaces to desktop and mobile devices.

Eye-tracking technology is rapidly advancing, enabling increasingly usable, accurate, and secure sensing of gaze. Eye-tracked headsets, glasses, smartphones, and in principle any computing device are on the horizon.

One of the most evolved relationships in our body exists between the eyes and hands. We can intuitively coordinate what we see with what we touch, hold, and manipulate. Our hands are highly expressive and flexible — capable of learning skill and gesture — while our eyes indicate intent, attention, and interest effortlessly and instantly. Both combined form the basis for powerful human abilities to experience and manipulate the world.

Hand-controlled input can be classified as direct or indirect. A mouse is indirect — the device mediates a cursor. A touchscreen is direct — input immediately acts on the object beneath the finger. This can be further decomposed using spatial and temporal indirection (Beaudouin-Lafon, CHI'00):

• Spatial indirection: distance between physical input and the virtual object.
• Temporal indirection: delay between starting the action and affecting the object.

Eye+Hand UIs — such as Gaze-Touch, Gaze+Pen, or Gaze+Pinch (as in Apple Vision Pro) — are spatially indirect but temporally direct. You can interact as quickly as with direct input, yet reach any target regardless of distance.

Categorisation of input methods into degrees of spatial and temporal indirection (adopted and extended from Beaudouin-Lafon, CHI'00).

As described in my 2014 UIST paper, Eye-Hand Symbiosis represents an advance for potentially all manual interfaces. A fundamental aspect is an extension of the direct manipulation paradigm:

• Direct input: control one object from one hand position (1:1)
• Eye+Hand UI: control any object you see, from any hand position (N:N)

This decomposes into two powerful cases: 1:N — one target, any hand position (flexibility, two-handed mappings); and N:1 — any target, one hand position (reach any object at a glance, context-switching without moving the hand).