[SUI’17] Gaze + Pinch Interaction in Virtual Reality

An exploration of 3D eye gaze interactions in VR. Focus is on what kind of capabilities one would gain when using gaze with freehand gestures. It’s different to my prior work on 2D touchscreens – less than a traditional input device, more like a supernatural ability! Lots of examples, scenarios, build with low-fidelity prototypes.

Interaction concept: Look to select, gesture to manipulate.

The concept supports all degrees of freedom for the typical RST tasks (Rotate, Scale, Translate), as it only relies on the user’s gesture capabilities.

Fundamental interaction possibilities.



Virtual reality affords experimentation with human abilities beyond what’s possible in the real world, toward novel senses of interaction. In many interactions, the eyes naturally point at objects of interest while the hands skilfully manipulate in 3D space. We explore a particular combination for virtual reality, the Gaze + Pinch interaction technique. It integrates eye gaze to select targets, and indirect freehand gestures to manipulate them. This keeps the gesture use intuitive like direct physical manipulation, but the gesture’s effect can be applied to any object the user looks at — whether located near or far. In this paper, we describe novel interaction concepts and an experimental system prototype that bring together interaction technique variants, menu interfaces, and applications into one unified virtual experience. Proof-of-concept application examples were developed and informally tested, such as 3D manipulation, scene navigation, and image zooming, illustrating a range of advanced interaction capabilities on targets at any distance, without relying on extra controller devices.

Conference Talk


Gaze + Pinch Interaction in Virtual Reality
Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi and Hans Gellersen. 2017. In Proceedings of the 2017 Symposium on Spatial User Interaction (SUI ’17). ACM, Brighton, UK, to appear. doi, pdf, video


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: