Ken Pfeuffer

In the history of Human-Computer Interaction (HCI), major interfaces have been based on the coordination of eyes and hands, following a clear division of labour: hands provide commands via input devices, while eyes perceive the visual output on a screen. This has been a long-lasting paradigm transcending eras of computing — from command-line interfaces to desktop and mobile devices.

Eye-tracking technology is rapidly advancing, enabling increasingly usable, accurate, and secure sensing of gaze. Eye-tracked headsets, glasses, smartphones, and in principle any computing device are on the horizon.

One of the most evolved relationships in our body exists between the eyes and hands. We can intuitively coordinate what we see with what we touch, hold, and manipulate. Our hands are highly expressive and flexible — capable of learning skill and gesture — while our eyes indicate intent, attention, and interest effortlessly and instantly. Both combined form the basis for powerful human abilities to experience and manipulate the world.

Hand-controlled input can be classified as direct or indirect. A mouse is indirect — the device mediates a cursor. A touchscreen is direct — input immediately acts on the object beneath the finger. This can be further decomposed using spatial and temporal indirection (Beaudouin-Lafon, CHI'00):

• Spatial indirection: distance between physical input and the virtual object.
• Temporal indirection: delay between starting the action and affecting the object.

Eye+Hand UIs — such as Gaze-Touch, Gaze+Pen, or Gaze+Pinch (as in Apple Vision Pro) — are spatially indirect but temporally direct. You can interact as quickly as with direct input, yet reach any target regardless of distance.

Categorisation of input methods into degrees of spatial and temporal indirection (adopted and extended from Beaudouin-Lafon, CHI'00).

As described in my 2014 UIST paper, Eye-Hand Symbiosis represents an advance for potentially all manual interfaces. A fundamental aspect is an extension of the direct manipulation paradigm:

• Direct input: control one object from one hand position (1:1)
• Eye+Hand UI: control any object you see, from any hand position (N:N)

This decomposes into two powerful cases: 1:N — one target, any hand position (flexibility, two-handed mappings); and N:1 — any target, one hand position (reach any object at a glance, context-switching without moving the hand).

XR user input is fragmented across devices — but over time, clear paradigms have emerged. The modern XR era, beginning with Oculus and the subsequent wave of consumer devices, has seen input control schemes fall broadly into four categories: an initial experimental phase, controller-based input, hand tracking, and the emerging eye-hand paradigm.

Each approach involves meaningful trade-offs. Head pointing is precise and reliable; controllers are fast and accurate; hand tracking is natural but can induce fatigue and reduce precision; and gaze-based selection reduces physical effort and can enable faster target acquisition. Crucially, newer paradigms do not simply replace older ones — controllers remain essential for high-precision tasks and gaming, hand input enables expressive interaction, and gaze+pinch suits everyday spatial UI navigation.

References: [1] Head vs controller vs gaze pointing · [2] Gorilla arm / mid-air fatigue · [3] Head vs gaze performance · [4] Gaze vs hand ray · [5] Gaze vs hand ray (2) · [6] Gaze+pinch principles

With Apple Vision Pro and Google/Samsung Galaxy XR adopting gaze+pinch as a core UI model, the paradigm is rapidly entering mainstream XR. This section distills five design principles that characterise how gaze+pinch is effectively implemented, building on the original formulation in the 2017 SUI paper.

A key insight is that the gestures in gaze+pinch mirror those of direct manipulation, eliminating the need to learn new input commands. Gaze is used only for the initial onset of a gesture — minimising the risk of accidental activation, a classic problem in gaze interfaces known in the literature as the Midas Touch problem (Jacob, CHI'90). The coexistence of direct and indirect gaze+pinch within the same UI further adds flexibility, allowing users to choose the most appropriate mode for each interaction context.

Beyond what makes gaze+pinch work well, it is equally important to understand where it breaks down. Five key design issues characterise the current limitations of gaze+pinch UIs — each of which can be framed as a user-facing problem, a design challenge, or an open research direction. As the paradigm becomes the default interaction model on leading XR platforms, addressing these issues becomes increasingly relevant for both practitioners and researchers.