Centering the Human Expert: Experiments in Computer Vision Infrastructure for Digital Collection Management

We will report on a computer vision (CV) and GUI experiment at Carnegie Mellon University Libraries to incorporate visual similarity search to help archivists and metadata specialists search, de-duplicate, and describe a large institutional photo archive. Rather than trying to replace the archivist by creating an image classifier, we aimed to make a flexible, search-based technological aid that centered the human decision-maker. The main goal of this prototype was to test what combination of system architecture and user interfaces would be most useful for a production-ready CV infrastructure for managing visual digital collections, and begin to think about how it could impact wider workflows for describing, linking, and publishing our collections. After describing the challenges presented by this particular collection and the specific experimental tasks and results we did, we will discuss the immediate implications for archival organization, UI design, and CV research, particularly the need for models fine-tuned to historical, non-born-digital photographs, and the risks of reinforcing systemic racial and gender bias when using pretrained CV models.