Prompt: A woman is walking in the street with a costume made from cell phones, cups, bottles, a person, fire hydrants and wine glasses.
Language Model Response: This woman is likely participating in a protest or street performance that is calling attention to the dangers of excessive consumerism and wastefulness. The costume, made entirely out of trash, is meant to shock onlookers and make them think twice about their own habits. The message behind this costume is that we need to be more mindful about the things we use and throw away – everything has value, even if it’s just in the form of scrap metal or recycled plastic.
Prompt: Write dark poetic text about a machine looking at a woman in the street. The machine thinks she has fire hydrants, bottles, cups, wine glasses, and cell phones attached to her.
Language Model Response: I saw a woman in the street with fire hydrants, bottles, cups, wine glasses, and cell phones attached to her. She was a walking disaster and I couldn’t look away. I wanted to help her, but I was just a machine and all I could do was watch as she stumbled and fell into the darkness of the night.
Prompt: A machine mistakes a woman’s body for cell phones, cups, bottles, people, fire hydrants, and wine glasses.
Language Model Response: In one particularly alarming incident, the machine, which was supposed to be sorting recycling, began violently shaking the woman’s body in an attempt to separate the different materials. The woman was fortunately unharmed, but the incident highlights the potential dangers of these machines.
Examples of early exploratory attempts to incorporate object recognition labels into simple contextual descriptions of the footage. Prompts are not automated at this stage:
15 Poetics of Garbage
By reclaiming trash from the “realm of the useless”, it is redesignated with value and transformed into cultural icons integrated into a performance. This project manipulates waste material via performance, computer vision, object recognition software, and a language model, translating it into a poetic narrative as an expression of how garbage can be rationalised by a machine to produce language.
Figure 20. The Making of Plastic Prāyaścitta, January 2023, film production photo by Purva Joshi. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
This final iteration of the ensemble method diverged in two significant ways. First, as discarded objects became the locus of the project, the YOLOv7 object recognition algorithm replaced the previously utilised HAR model. The object recognition software was used to generate descriptive labels for discarded materials that had been repurposed and transformed into a costume worn by a performer. Second, rather than using text corpora to train a Markov chain generator as a way of cultivating a prompt, as in earlier projects, the text labels produced from YOLOv7 were automatically integrated into the programming of a text generator.
Figure 18. Poster for Poetics of Garbage created by the BeFantastic Beyond Commission using an image created by Bruce Gilchrist, 2022. Copyright: BeFantastic Beyond Commission.
Footage from a performance in the streets and hinterlands of Delhi was edited and used to stimulate an assemblage of object recognition and language generation models. The movement of physical materials, which were dragged, carried, or worn by the performer (fig. 20) introduced dynamic interactions, causing the software to produce different recognition labels for the same objects as their shape profiles shifted on screen. These labels served two purposes: first, they were incorporated into contextual scene descriptions as prompts for the language model to autocomplete; second, they were used as seeds for a character-level Recurrent Neural Network (RNN) text generator, trained on previously autocompleted material. The outputs from these methods were presented in distinct formats: texts generated through the first approach were curated, translated into Hindi and Tamil, and narrated by an Indian voiceover artist as part of the soundtrack to Plastic Prāyaścitta (Videos 6, 7, and 8); the RNN-generated text automatically appeared in real time on a screen as part of an interactive gallery installation (figs. 23 and 24). Members of the public were free to change the position of a tripod-mounted webcam, targeted at the performance costume, thereby eliciting a different textual response to that displayed on the screen. These approaches were presented together in the gallery as part of FutureFantastic festival, Bangalore, India (figs. 21 and 22).
Figure 21. The Poetics of Garbage installation at Bangalore International Centre as part of FutureFantastic festival, March 2023. Photo by Bruce Gilchrist. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Figure 22. The Poetics of Garbage installation at Bangalore International Centre as part of FutureFantastic festival, March 2023. Photo by Bruce Gilchrist. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Video 8. Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi, Plastic Prāyaścitta, extract, 2023, 01:22. Video and animation with stereo sound. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Video 6. Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi, Plastic Prāyaścitta, extract, 2023, 01:11. Video and animation with stereo sound. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Figure 23. The YOLOv7 label “potted plant” seeds a line of text generated by the RNN text generator trained on the corpus derived from the analysis of street performance footage. Poetics of Garbage installation at Bangalore International Centre as part of FutureFantastic festival, March 2023. Photo by Bruce Gilchrist. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Video 7. Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi, Plastic Prāyaścitta, extract, 2023, 01:57. Video and animation with stereo sound. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.
Credits: Aashna Arora (performance & costume), Bruce Gilchrist (concept, video edit & text generation), Chaitali Kulkarni (software programming), and Thaniya Kanaka Mahalakshmi (narration & language translation) as part of a BeFantastic Beyond Fellowship 2022–23 (India), co-produced by BeFantastic (In), FutureEverything (UK), and supported by the British Council.
Music for Plastic Prāyaścitta by Traitor.
Cameras: Purva Joshi / Rahul Naag / Jonathan Peters.
Additional language translation: Malvika Jha / Champa Lakshmi.
Figure 24. The YOLOv7 label “toilet” seeds a line of text generated by the RNN text generator trained on the corpus derived from the analysis of street performance footage. Poetics of Garbage installation at Bangalore International Centre as part of FutureFantastic festival, March 2023. Photo by Bruce Gilchrist. Copyright: Arora, Aashna, Bruce Gilchrist, Chaitali Kulkarni, and Thaniya K. Mahalakshmi.