One of the most fundamental issues in any object-based interaction paradigm, be it GUI or NUI, is the issue of selection. It isn’t sexy, but it is an incredibly important piece to get right – and many, many touch systems get it wrong. The solution for mice is so engrained in our thinking that it can be difficult to understand what the issue is at all. To select something in a mouse-based system, the user positions the pointer above the object to be selected, depresses the mouse button, then, without moving the pointer off of the object, releases the mouse button. To cancel mid-selection, the user can move the pointer off of the object before releasing. There are some minor variations (eg: lasso selection, either free-form or rectangular), but, in general, the ‘click’ paradigm is well observed.
Sounds simple enough – why not just copy this mechanism for touch-based systems? There are several complicating factors when working with a touch-based system, factors which make the choice of selection mechanisms slightly more complicated. Further, by carefully choosing the appropriate selection mechanism, we can actually overcome fundamental limitations of touch-based systems.
In this entry, we’ll examine a full taxonomy of possible selection events on a touch system, discuss the various implementations that have explored, and ultimately come to an understanding of the pros and cons of each selection method.
Anatomy of a Selection Event
Intuitively, selection is typically mapped to a moment of contact transition between the user’s finger and the object to be selected. To understand the myriad of selection methods, I will refer to Figure 1 regularly:
Figure 1. Various contract transition types on a touch display. Classic ‘tap’ (or ‘click’) selection occurs on a B-C event: the user lowers their finger onto the object (B) and removes their finger from the display while still in contact (C). A and D represent the user sliding their finger along the screen to enter the object (A) or exit it (D).
The various combinations of these transitions into selection events have been studied by a variety of researchers and technologists. Several combinations of entry events (A,B) and exit events (C,D) have been combined to create selection events.
Tap: B-C Selection
Tap selection is a classic, in that it mimics the mouse: the finger comes down on the object, and is lifted from the screen while still within. Here at Surface, our UR team has found that users tend to try this one first, and are sometimes blocked if Tap selection is not available.
Advantage: technique is consistently the first one tried by novice users. It also provides a clear ‘cancel’ mechanism: after touching the object, sliding the finger off of it but maintaining contact with the display aborts the selection.
Disadvantage: does not allow for chained selections (selection of more than one item in a single gesture). Requires precise touching, so on-screen targets must be large. Does not support parallelism very well, so multi-touch tap selections are likely to be sequential, rather than simultaneous.
First-Touch: A Selection
First-touch selection comes to us from Potter et al’s classic paper on touch-screen selection. The idea is simple: allow the user to put their finger down anywhere on the display. They then slide it around until they successfully touch an object, which is then selected immediately (the exit transition has no effect on the selection).
Advantages: In target-sparse environments (aka screens without a lot of selectable stuff), this can help to overcome some of the problems associated with fat fingers, discussed in detail a few weeks back. Essentially, if the user misses the target when they first touch the screen, they can simply wiggle their finger a bit to acquire it. Further, the selection event is made before the user lifts their finger from the screen, so they can continue to work. This makes multiple-selection easy, and is the method of selection in a hierarchical menu.
Disadvantages: in target-rich environments, first-touch makes it very difficult to accurately select a target. There is no cancellation mechanism.
Take-Off: C Selection
Take-off selection takes-place when the finger is lifted from the screen. This is the selection method used to enter text on the iPhone keyboard. In the original paper, Potter et al. describe the use of an offset cursor in combination with take-off, which is also done by the iPhone keyboard.
Advantages: allows for a preview of what will be selected when the finger lifts, and thus greater accuracy than First-Touch selection.
Disadvantages: unlike First-Touch selection, the finger is now off the display, so continuous actions are impossible.
Land-On: B Selection
Land-on selection is the method used by physical keyboards: as soon as the finger lands on the target, the selection is made.
Advantages: allows for intuitive selection, immediate system response when the finger lands on the screen.
Disadvantages: no opportunity to cancel the selection (as with 'tap', where the user can slide their finger off before lifting), and also is much less precise because the system cannot preview for the user which object will be selected.
Crossing: A-D Selection
Crossing selection is to 'first touch' what 'tap selection' is to land-on: the user must both enter and exit the target using the same method.
Advantage: allows the user to cancel their selection, because they can make a 'B' exit from the target to avoid selecting it. Multiple target selection is possible, as the user slides along the screen.
Disadvantage: no preview of selected items before selection is made (unless twinned with a 'first touch' visualization). Unintuitive, in that landing a finger direction on the target does not select it.
Slip-Off: B-D Selection
This selection method is traditionally used by pen-based interfaces, as it resembles a "checkmark": the user touches the stylus to the target, then slides it off of the target while on the screen.
Advantage: more intuitive than crossing, since the goal is to touch the object.
Disadvantage: multiple selection is not possible, because the stylus must be lifted before selecting an object.
Aside: What does ‘Selection’ Mean in Multi-User Systems?
Now armed with a deep understanding of various selection methods, it is important to remember not to blindly apply your favourite to a new multi-user application. It is critical that you remember that in a multi-touch system, traditional select-apply (aka ‘noun-verb’) interaction sequences don’t work without careful thought. Remember: each user may have one or more objects selected independently of one another. Providing a tool palette to then apply some action or property to those objects will lead to obvious task interference between your users (eg: user 1 wants to colour an object blue, so he selects the object, and moves his hand to select the ‘blue’ button. At the same time, user 2 decides to colour a different object red, so he selects it, and then selects the ‘red’ button. If you have not carefully designed your system, both of the objects selected by of user 1 and user 2 will now be red. Oops!).
Our Team’s Work
The Surface design team is redesigning our touch primitives, and selection mechanisms feature highly on the list. Making the right choices from among the selection event types is critical for success, and we will be focusing our energies on making the right choice.
