Generative AI for Critical Digital Twins

 Training and evaluating autonomous robots within the real world present significant challenges and risks, emanating from unpredictable environments, safety  concerns, ethical dilemmas, and limited human oversight. As a mitigation strategy,  the use of realistic simulations, also known as digital twins, offers virtual duplication  of the actual system or environment, thus fostering the development of trustworthy  autonomy.  

Digital twins enable developers to evaluate the performance of systems in various  scenarios to identify potential risks or failure cases. It facilitates the accumulation  and subsequent analysis of datasets, which serve to validate and calibrate the au tonomous system’s perception and decision-making algorithms. By comparing the  behavior of the digital twin with real-world data, developers can identify discrepan cies, improve accuracy, and enhance the system’s safety and reliability.  

Scenarios that embody dynamic and interactive components reflect the intrica cies of digital twins and take precedence in significance. One main value of digital  twins is helping us understand how objects interact and behave. For example, in  autonomous driving, the behavior of vehicles, pedestrians, and traffic conditions are  crucial components of scenarios that need to be accurately modeled. However, not  all scenarios in digital twins are created equal. In the pursuit of developing trustwor thy autonomy, ordinary scenarios often prove insufficient in subjecting autonomous  systems to extreme conditions where safety and robustness are paramount. Although  critical scenarios hold the potential to expose model vulnerabilities, their rare occur rence creates a challenge. The process of manually identifying or extrapolating such  critical scenarios from normal data or expert design proves not only inefficient but  also contains substantial human biases.  

My doctoral research seeks to harness the potential of generative AI to explore  two pivotal questions: (1) Which scenarios are critical in existing data and (2) How  to generate such scenarios in digital twins? The proposal begins with the definition  of critical scenarios and the corresponding optimization problem and subsequently  delves into three distinct categories of scenario generation frameworks: data-driven  generative models, adversarial generative models, and knowledge-guided gen erative models. Concluding this thesis is future directions that effectively combine  generation resources from different perspectives and improve the data flywheel by