A Collaborative Framework for Machine Learning-Based Toolmaking for Creative Practices

The latest boom of Machine Learning (ML) in the early-2010s has raised a new wave of interest among creative practitioners to explore the intersection of Art and Artificial Intelligence (AI), specifically Generative Machine Learning. A growing number of artists, designers, and architects appropriated these algorithms to make new tools for their creative practices. 

This dissertation introduces and documents a collaborative framework to make machine learning-based tools for creative practices. The framework embraces the idiosyncratic nuances and elements of the physical context of the creative practice. It takes a new point of view on data and data curation as the primary method of interacting with ML algorithms. The framework achieves this goal by utilizing small user-generated datasets, which are biased toward the creative practitioners’ personal preferences, subjective measures, and elements of the physical context of their practice. Through collaboration with machine learning expert toolmakers, the framework makes ML algorithms more accessible to these creative practitioners. It highlights the affordances of ML algorithms, specifically Conditional Variational AutoEncoders (C-VAE), that can be efficiently trained and overfit on small datasets to produce outcomes that are closely tied to the creative practitioners and their context.

In the two case studies, the framework serves as a high-level blueprint to develop bespoke tools that support various stages of machine learning-based toolmaking for creative practitioners. In SecondHand, I collaborated with a group of participants to develop handwriting typeface generation tool. A dashboard, based on Dash Plotly, featuring interactive data visualization and data curation tools, was developed for this study. In ThirdHand, I collaborated with a musician to create a robotic tool to play santur, a traditional Persian musical instrument, using an ABB IRB 120 robotic arm and a real santur. 

The case studies demonstrated that the proposed collaborative framework meaningfully brings ML experts’ technical literacy to complement creative practitioners’ domain knowledge and skills to overcome the technical ML challenges, and help integrate various idiosyncratic aspects, elements of the physical context, and nuances of creative practice in the toolmaking process.