[CHI’17 / MSR] Thumb + Pen Interaction on Tablets

This paper was the result of the amazing internship in MSR 2016. Developed lots of interaction concepts using pen and touch modalities, in a typical exploratory design approach from Ken Hinckley & co. The video reached >30000 views!

Tablets should be designed with their use in mind – laid back, relaxed, on the couch! This means users are holding the device with at least one hand. But how can we bring the power of bimanual pen and touch interaction to the tablets then?

Here we propose the UI should better adapt to the grip, particularly provide enhanced menus that users control with their thumb only.

ms pic.png
User leveraging simultaneous thumb and pen interactions.

We followed up with a deep theoretical analysis of this kind of interaction, what it means for bimanual input, marking menus, indirect input, and quasi modes.

ms theory.png
Design space of the UI/menu design (does anyone actually understand this? :-))


Modern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and touch interaction, with both hands, in such situations. Our approach engages the thumb of the device-holding hand, such that the thumb interacts with the touch screen in an indirect manner, thereby complementing the direct input provided by the preferred hand. For instance, the thumb can determine how pen actions (articulated with the opposite hand) are interpreted. Alternatively, the pen can point at an object, while the thumb manipulates one or more of its parameters through indirect touch. Our techniques integrate concepts in a novel way that derive from marking menus, spring-loaded modes, indirect input, and multi-touch conventions. Our overall approach takes the form of a set of probes, each representing a meaningfully distinct class of application. They serve as an initial exploration of the design space at a level which will help determine the feasibility of supporting bimanual interaction in such contexts, and the viability of the Thumb + Pen techniques in so doing.


Thumb + Pen Interaction on Tablets
Ken Pfeuffer, Ken Hinckley, Michel Pahud and Bill Buxton. 2017. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, Denver, CO, USA, 3254-3266, doi, pdfvideo, WindowsReport, onMSFT, WindowsCentral , SlashGear

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s