file.pdf (871.06 kB)
Download fileGestural Communication over Video Stream: Supporting Multimodal Interaction for Remote Collaborative Physical Tasks
journal contribution
posted on 01.01.2003, 00:00 authored by Jiazhi Ou, Susan R. Fussell, Xilin Chen, Leslie D. Setlock, Jie YangWe present a system integrating gesture and live video to support collaboration on physical tasks. The architecture
combines network IP cameras, desktop PCs, and tablet PCs to allow a remote helper to draw on a video feed of a workspace as he/she provides task instructions. A gesture recognition
component enables the system both to normalize freehand drawings to facilitate communication with remote partners and
to use pen-based input as a camera control device. Results of a preliminary user study suggest that our gesture over video communication system enhances task performance over traditional video-only systems. Implications for the design of multimodal systems to support collaborative physical tasks are also discussed.