Characterizing, Modelling and Augmenting Datasets for Simulation of Multi-Modal Internet-of-Things Applications

The Internet of Things (IoT) applications rely on sensors to monitor their environment and actuators to react to environmental changes or user instructions. As IoT devices become more common, the multiple available sensor and actuator modalities can augment information about the surroundings or provide new services.

In my thesis, I demonstrate how the benefits of multi-modal interactions between devices come at the cost of increased complexity when designing and testing new IoT system applications. I address two main types of multi-modal interaction in IoT systems. The first type is sensor fusion, where multiple modalities are used to observe the same phenomenon in different ways. The second type is automated control, where actuators in the system are used to respond to detected events and themselves create detectable events. Without expert domain knowledge of the physical processes behind sensed events, energy transfer between modalities, and the physical response of the chosen hardware, multi-modal IoT event detection applications are commonly developed with extensive empirical evaluation. Real-world testing cannot exhaustively check every configuration of devices and actuators for possible interactions; however, without expert domain knowledge, it is difficult to verify the accuracy of any model or simulation built from a limited sample of real-world data. 

Using data from multiple sources, including public research datasets and real-world system implementations, I address the challenge of modelling event detection application performance in multi-modal IoT systems. First, I characterize the performance of camera + IMU sensor fusion algorithms and distinguish errors due to physical/sensor limitations from algorithm inaccuracy. Second, I characterize the sound produced from small UAV motors in order to use this multi-modal "side effect" as a communication channel. Through a combination of simulation and empirical validation, I was able to integrate the new capability into a UAV swarm without hindering flight. Third, I characterize the common features of digital sensor signals for event detection applications that allow for simulation without expert domain knowledge. Based on the common features of event detection signals, I develop a machine learning framework for augmenting sensor signal datasets for simulation with limited domain knowledge.