A Mixed Reality System Combining Augmented Reality, 3D Bio Printed Physical Environments, and Inertial Measurement Unit Sensors for Task Planning

Medical errors are the third leading cause of death in the United States of America, accounting for approximately 9.2% (250,000) of all total deaths. To combat this, successful surgical operations are characterized by preplanning routines to be executed during actual surgical operations. To achieve this, surgeons rely on the experience acquired from cadavers, enabling technologies like virtual reality (VR) and clinical years of practice. However, cadavers lack dynamism and realism as they lack blood and can exhibit limited access, tissue degradation, and shrinkage while current VR systems do not provide amplified haptic feedback during operations. In addition, the clinical years of practice required for mastery vary across surgical techniques for different comorbidities. This can impact surgical training increasing the likelihood of medical errors during surgical operations.

This dissertation looks at these deficiencies and provides a unified solution composed of three components. The first of these components guarantees user navigation in confined spaces using an inertial measurement unit sensor attached to a user’s tool during task execution. The second component is the use of a mixed reality device to stream imagery in the form of augmented reality to a user field of view to assist with task execution. The third component is to provide a platform for haptic feedback from user actions during task execution in the form of a 3D bio print. Taking all these solutions together, a novel Mixed Reality Combination System (MRCS) that pairs Augmented Reality (AR) technology and an inertial measurement unit (IMU) sensor with 3D bio-printed, collagen-based specimens that can enhance task performance like planning and execution is proposed as an overall solution. 

Next, to demonstrate the effectiveness of the MRCS Platform as an overall solution, two studies are set up that compare the task execution outcomes between groups that have either been or not been exposed to the MRCS training and/or groups have either basic or advanced medical training. The first study looks at graduate students with no medical training and compares their surgical task proficiency gain curves with an identical group exposed to the MRCS Platform. The second study is run as a Qualitative study that replaces the graduate student population with medical students who have just received/ are receiving medical instruction in the form of a medical degree. This group is presented with the MRCS Platform. It is asked to provide a qualitative assessment that provides insight into the adoption and benefit of using the platform as a medical training curriculum tool to improve skill acquisition and prevent surgical errors during operations.