Interaction Styles & Devices
Interaction Styles & Devices !
Touch / multi-touch
!
Handwriting
!
Voice
!
Gesture
!
Eye-tracking
!
Virtual
Touch / multi-touch !
Pros " " " "
!
" " " "
Other inputs not practical
When not to use "
!
Fat finger and ccclusion Ambiguous actions Cost Reduced screen brightness No haptic feedback
When to use "
!
Better real-estate (no buttons) Multi-touch -> Multiuser
Cons "
!
More natural (direct control) More convenient (display is input)
High precision work
Standards/Guidelines " "
Follow consistent behavior patterns Large target sizes
Handwriting !
Pros "
!
"
Natural/familiar Useful for non-linear input
"
Pen device or touch
Cons " " "
Messy “doctor” penmanship Small writing space
Attention-heavy " Available & reliable recognition software software " Continuous text
Handwriting When to use "
!
"
Mathematics learning Note taking
"
Mobile devices (+ pen)
When not to use "
!
"
Other attention-heavy activities Small screens w/o pen
"
Long passage/data input
Standards/Guidelines "
Large writing area for touch input
"
Tested & reliable recognition software supports correction of recognition errors
"
Voice !
Pros "
!
"
Hands free Fast and accurate for text
"
Little/no training
Cons "
Trouble for accents and weak support for some languages
"
Limited vocabulary / applicability Computationally expensive
"
!
When to use " "
!
Small devices (smart watches) Text-only input
When not to use "
Large operation set
Standards/Guidelines ! Other information !
"
Semantics / natural language processing
Gesture !
Pros " "
" " "
!
Pairable with other input modalities Touch gesture or near-field gesture (NFG) Natural Pen-style tools or body only Wearable devices
Cons "
" " " " "
Transition from 2d pointing to 3d motion Fatigue Mental demand on gesture recall False positive commands Limited gesture-viewing field Limited by standard point-n-click UI
Gesture When to use " "
Sterile environments Hand/arm disabilities (+ pen tool)
"
3D models/graphics " AR/VR immersive environments environments !
When not to use " " "
!
Low tracking/sensing environment Standard UI Narrow paths/menus
Standards/Guidelines " " " " " "
Begin with a clean slate Larger targets (or expanding) (NFG) Middle # of the display (NFG) Obvious/intuitive gesture commands Provide feedback Robust tracking system
Eye-tracking !
Pros "
Hands free
"
Easy to learn " Extra input channel
!
Cons "
Deliberate eye-control is mentally demanding
"
Head-tracking / accuracy challenge, saccadic movements " Involuntary indicators of action
!
When to use "
Motor disabilities " Hands-free environments
!
When not to use "
High precision tasks " Complicated action set
Virtual / Augmented Reality (VR & AR) !
Pros " "
Digital real-world overlay Full environment immersion
"
Combines with other input modalities " Additional information to environment " Vision-deficit assistance (VDR) !
Cons " "
" "
"
"
Tracking technique Ergonomics (size, weights, power, ruggedness) Portability Alignment of AR object to realrealworld environment, Limited FOV (depends on tracking technique) Display problems (color depth, resolution, contrast, luminance)
Virtual / Augmented Reality (VR & AR) When to use " " " "
" " " "
!
Medical training tool Virtual instructions Information visualization Production sets for entertainment (e.g. theatre) Gaming Military aid Navigation Many many more$.
When not to use "
" "
"
Non-user-present environments Non-user-present (e.g. flying) (AR only) Long-duration use Poor tracking environments (depends on techniques used) No access to wireless or data networks
Virtual / Augmented Reality (VR & AR) Standards/Guidelines "
Information enhances real world
"
Supports collaborative work No special apparatus Supports need to naturally display 3D images Portable & minimally invasive (handheld device, glasses) Interfaces supports: % Manipulation of virtual objects % Drawing paths or trajectories % Assigning quantitative values & text input
" "
"
"
%
Direct manipulation of physical objects
Head-to-Head: Eye tracking vs Gesture vs Traditional
Head-to-Head: Eye tracking vs Gesture vs Traditional
Other Interaction Styles & Devices Sip-n-puff ! BCI ! Multi-modal ! Sensory substitution/augmentation !
Thought Question You want to design a user interface for a specific population (your choice). c hoice). What device(s) are best for that population, and what user interface design considerations do you have to make?
References Touch/Multitouch: https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/MobileHIG/ Touch/Multitouch: https://www.google.com/design/spec/layout/metrics-keylines.html Touch/Multitouch: Boring, S., Ledo, D., Chen, X. A., Marquardt, N., Tang, A., & Greenberg, S. (2012, September). The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services (pp. services (pp. 39-48). ACM. Touch/Multitouch: Jung, E. S., & Im, Y. (2015). Touchable area: An empirical study on design approach considering perception size and touch input behavior. International Journal of Industrial Ergonomics, Ergonomics, 49 49,, 21-30. Handwriting: Anthony, Handwriting: Anthony, L., Yang, J., & Koedinger, K. R. (2006, July). Towards the application of a handwriting interface for mathematics learning. In Multimedia and Expo, 2006 IEEE International Conference on (pp. on (pp. 2077-2080). IEEE. Handwriting: Bharath, A., & Madhvanath, S. (2008, January). FreePad: a novel handwriting-based text input for pen and touch interfaces. In Proceedings of the 13th international Conference on intelligent User interfaces (pp. interfaces (pp. 297-300). ACM. Handwriting (PPT): Snowdon, J. L. (2003). Pen Computing: Challenges and Applications. New York . Handwriting: Subrahmonia, J., & Zimmerman, T. (2000). Pen computing: Challenges and applications. In Pattern Recognition, 2000. Proceedings. 15th International Conference on (Vol. on (Vol. 2, pp. 60-66). IEEE. Voice: Katangur, A. K., Akkaladevi, S., & Osei, H. (2013). Voice remote computer control using speech recognition through PSTN. Journal of Applied Global Research, Research, 6 (18). (18). Voice: Marukami, Voice: Marukami, T., Tani, S., Matsuda, A., Takemoto, K., Shindo, A., & Inada, H. (2012). A basic study on application of voice recognition input to an electronic nursing record system-evaluation of the function as an input interface. Journal of medical systems, systems, 36 (3), (3), 1053-1058. Gesture: Dhawale, Gesture: Dhawale, P., Masoodian, M., & Rogers, B. (2006, July). Bare-hand 3D gesture input to interactive systems. In Proceedings of the 7th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: design centered HCI (pp. (pp. 25-32). ACM. Gesture: Wigdor, D., & Wixon, D. (2011). Brave NUI world: designing natural user interfaces for touch and gesture. gesture. Elsevier. Eye Tracking: Sesin, A., Adjouadi, M., Cabrerizo, M., Ayala, M., & Barreto, A. (2008). Adaptive eye-gaze tracking using neural-network-based user profiles to assist people with motor disability. J Rehabil Res Dev , 45 (6), (6), 801-817. Eye Tracking: Zhai, Tracking: Zhai, S. (2003). What's in the eyes for attentive input. Communications of the ACM , 46 (3), (3), 34-39. AR: Azuma, AR: Azuma, R. T. (1997). A survey of augmented reality. Presence Presence,, 6 (4), (4), 355-385. AR: Van AR: Van Krevelen, D. W. F., & Poelman, R. (2010). A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality , 9(2), 1. AR: Zhou, AR: Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008, September). Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (pp. (pp. 193-202). IEEE Computer Society.
References Head-to-head: Canare, D., Chaparro, B., & He, J. (2015). A comparison of gaze-based and gesture-based input for a point-and-click task. In Universal Access in Human-Computer Interaction. Access to Interaction (pp. Interaction (pp. 15-24). Springer International Publishing. Device Comparison Discussion: Hinckley, K., & Wigdor, D. (2002). Input technologies and techniques. The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, applications, 151-168. Mutli-Modal: Jaimes, Mutli-Modal: Jaimes, A., & Sebe, N. (2007). Multimodal human–computer interaction: A survey. Computer Vision and Image Understanding , 108 (1), (1), 116-134. Mutli-Modal: Nigay, Mutli-Modal: Nigay, L., & Coutaz, J. (1993, May). A design space for multimodal systems: concurrent processing and data fusion. In Proceedings of the INTERACT'93 and CHI'93 Conference on Human Factors in Computing Systems (pp. Systems (pp. 172-178). ACM. Tongue Operated: Ghovanloo, Operated: Ghovanloo, M. (2007). Tongue operated assistive technologies. IEEE Engineering in Medicine and Biology Magazine, Magazine , 1, 4376 Tongue Operated: Huo, Operated: Huo, X., & Wang, J. (2008). Introduction and preliminary evaluation of the Tongue Drive System: wireless tongue-operated assistive technology for people with little or no upper-limb function. Journal of Rehabilitation Research and Development , 45 (6), (6), 921. Tongue Operated: Yousefi, Operated: Yousefi, B., Huo, X., Veledar, E., & Ghovanloo, M. (2011). Quantitative and comparative assessment of learning in a tongue-operated computer input device. Information Technology in Biomedicine, IEEE Transactions on, on, 15 (5), (5), 747-757. BCI: Wolpaw, BCI: Wolpaw, J. R., McFarland, D. J., Neat, G. W., & Forneris, C. A. (1991). An EEG-based brain-computer interface for cursor control. Electroencephalography and clinical neurophysiology , 78 (3), (3), 252-259. BCI: Schettini, BCI: Schettini, F., Riccio, A., Simione, L., Liberati, G., Caruso, M., Frasca, V., ... & Cincotti, F. (2015). Assistive device with conventional, alternative, and brain-computer interface inputs to enhance interaction with the environment for people with amyotrophic lateral sclerosis: a feasibility and usability study. Archives Archives of physical medicine and rehabilitation, rehabilitation, 96 (3), (3), S46-S53. Sensory Aug: Bach-y-Rita, P. (1967). Sensory plasticity. Acta Neurologica Scandinavica, 43(4), 417-426. Sensory Aug: Bach-y-Rita, P. (2001). Nonsynaptic diffusion neurotransmission in the brain: functional considerations. Neurochemical research, 26(8), 871-873. Sensory Aug: Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. Journal of rehabilitation research and development, 35, 427-430.
