Detecting as-built information model errors using unstructured images

Keeping an as-built information model of infrastructure, such as a Building Information Model (BIM), up to date is valuable for the whole life cycle in the architecture, engineering, and construction (AEC) industries. During construction, an up-to-date model can facilitate the coordination process among project participants and help avoid project delays and rework. During operation and management, an accurate as-built information model could help reduce facility management inefficiencies and plays an important role in enabling digital twin.

One of the primary reasons causing outdated information in an as-built model is that changes made during the life cycle of a building are not well documented, resulting in model errors that could not accurately reflect as-built information. Existing industry practices often address such model errors by capturing photos during construction or maintenance and comparing them with the model in use manually. This process is time-consuming and error-prone due to many reasons: 1) photos used in AEC projects are often captured in an unstructured and distributed manner, meaning that they do not have camera poses, overlapping regions amongst images, or camera intrinsic parameters that are necessary for associating components in an image to a digital twin; 2) it is inefficient to manually interpret and register as-built information in a large number of images to a building information model; 3) buildings are continuously changing, especially during construction and renovation when incomplete and temporary objects could be present in images. In summary, unstructured images can only provide snapshots of as-built conditions at a certain time and location and it is difficult to understand and associate as-built information shown in images to their corresponding digital representations for detecting potential model errors.

Considering the aforementioned engineering challenges, the approach described in this thesis incorporates the following three steps to detect model errors: 1) Register a 2D construction image to a 3D digital model, 2) Detect components of interest in images, 3) Compare as-built information in an image with the as-designed information stored in a model to check if there is a possible model error. Though many previous studies in the computer vision and machine learning communities have partially addressed the registration and the semantic understanding problem respectively, they could not be directly applied to an AEC scene with changing and incomplete components. Therefore, the thesis aims at addressing the following three research questions: 1) how to register unstructured images captured in a changing environment to a 3D as-built model and achieve a similar localization accuracy compared to the existing approach using structured images captured in a static scene; 2) what algorithm could accurately recognize incomplete and changing components in unstructured images; 3) what approach can compare as-built information shown in unstructured images to an as-built model and detect model errors automatically.

The contributions of the thesis include a new workflow that leverages existing unstructured images to detect model errors. The thesis addressed the aforementioned research questions from three perspectives. First, for image-to-BIM registration, this thesis proposed a pure data-driven and a domain knowledge-driven registration approach that could be applied to different phases of a building’s life cycle. Second, for image-based scene understanding, the thesis proposed to leverage existing semantic information in a BIM to help improve the ability of scene understanding in unstructured images. Third, for model error detection, the paper presented a method for comparing as-built components in images with their corresponding digital representations and detect model errors through semantic information reprojection. It is envisioned that the complete workflow presented in this thesis could help to maintain an up-to-date model for project coordination during construction and for facility management during operation and management, facilitating information exchange among contractors, subcontractors, and owners, and reducing wasted work time and cost due to incorrect model information.