Carnegie Mellon University
Browse

Computational Understanding of User Interfaces

Download (59.58 MB)
thesis
posted on 2024-08-26, 20:27 authored by Jason WuJason Wu

A grand challenge in human-computer interaction (HCI) is constructing user in terfaces (UIs) that make computers useful for all users across all contexts. Conventional UI development processes have approached this goal by iteratively converging  towards a single “final” UI through prototyping, implementation, and testing. However, even when following best practices, this approach locks in a set of assumptions that often cannot accommodate the diversity of user abilities, usage contexts,  or computing technologies, ultimately limiting how we can use computers. In this dissertation, I propose a new approach that uses machine-learning-driven systems  that automatically understand and manipulate existing UIs as a way to overcome  this challenge. Using content and functionality inferred from an existing application, combined with sensed usage context, the UI can be dynamically tailored to the immediate needs of use, e.g., through interface adaptation or generation.  

The work presented in this dissertation represents the initial technical foundation for this vision. First, I describe approaches for understanding user ability and context  (user understanding), which HCI suggests is the basis for building good interfaces. I  describe a recommendation system that recommends device settings (e.g., accessibility features) based on sensed usage behaviors and user interaction logs. While users often found these suggestions helpful, this approach of adapting interfaces through  configuration changes has traditionally been limited, since applications often do not  properly expose their semantics to external services. To this end, I describe several  projects in the area of UI understanding, which shows that it is possible to overcome  this barrier using data-driven ML models that predict interface layout, structure, and functionality from visual information, which is how UIs are generally assumed to  be used. These predicted semantics can enable many forms of existing computing  infrastructure, such as accessibility and UI agents to work more reliably and robustly. Finally, I combine both user and UI understanding to dynamically generate  and adapt UIs that meet the specific needs of users. I describe ML-driven systems  that generate UIs by modifying existing application layouts and generating UI code  based on personalized user profiles and design objectives. Ultimately, through my work, I show that computational understanding of user interfaces allows UIs to be  transformed from static objects into malleable representations that can be dynamically reshaped for new devices, modalities, and users.  

History

Date

2024-07-26

Degree Type

  • Dissertation

Department

  • Human-Computer Interaction Institute

Degree Name

  • Doctor of Philosophy (PhD)

Advisor(s)

Jeffrey P. Bigham