Demolish Monsters on the Rocks: Prompting Through an Ensemble

bruce gilchrist

As interaction with corporate artificial intelligence increasingly becomes a precondition for contemporary life, artists need to see beyond Generative AI (GenAI) technology as a discrete tool that makes generic products. Instead, they can imagine combinatorial approaches and conceptual frameworks for AI-enabled artworks. Through my practice-based research, the act of prompting multimodal GenAI models has been informed by comprehending an assemblage as a “framework of instruction” held together through poetic alliances, within which the output from one component feeds the process of another. Practical experiments explored an interrelation of body, text, and predictive technology, where an algorithmic prediction of human action conjured “biometric poetry” that was used to stimulate a language model. Working with archival film footage and digital puppets animated with motion-capture files gave rise to the idea of a camera’s field of view – with its bounded contents acting like a key – eliciting value from a language model in a novel form of story making. Potential erroneous inferences were perceived as a new form of chance operation and a characteristic of algorithmic remix as defined by Steve F. Anderson. This method has been further developed in a project that combines performance, waste material, object recognition, and a language model to explore how the manipulation of garbage can be rationalised by a machine to produce poetic texts as a commentary to action portrayed on a screen.

16 Summing Up

• Inscrutable AI: Despite its ubiquity, GenAI remains enigmatic. Large language models in particular are strangers in our midst, as even their creators lack a full understanding of their productions. Computer scientists are trying to reverse engineer them to understand why they come up with specific outputs, and massive effort is required to create guardrails to prevent them from churning out bias and misinformation.

• Speculative Understanding (Fictioning): This opacity demands a speculative leap, or an act of fictioning, to conceptualise their workings. Artists who critically engage with AI by subverting its typical uses create unique spaces for reflection on the societal and creative implications of these technologies.

• Misclassification: The use of an outdated human-action recognition (HAR) model serves not as a technical limitation but as a productive force. By embracing the HAR model’s tendency for disjointed inference and inability to neatly categorise ambiguous gestures, the ensemble becomes a performative mechanism – a stage for the contested meaning and fractured representation that emerges when computational logics confront the nuances of human action.

• Promptism: Language is considered the metaphorical machinery that delivers cultural messages (Hoad 2001, 37). In the context of GenAI, this metaphorical machinery is largely occluded within several millions of lines of unfathomable code. Sparking up the machine has become a form of arcane prompt-engineering that offers a tenuous influence over outcomes.

• Repertoire and Rupture: Through an interplay between automation and human agency, randomness and structure, GenAI-enabled art recasts the essential role of the artist as a selector, curator, and translator across systems. Artists are called to leverage their cultivated repertoire to identify conceptual resonances, unexpected alignments, and poetic disjunctions in new forms of meaning-making.

• Movement through Data: Conceptually, GenAI can be understood as a movement through data, with a text-prompt interface acting as a contemporary form of instruction art, where language is prioritised in the making of things.

Contents Back | Next

Back | Next

• Instruction Art Analogy: Instruction art is defined as a set of instructions created by the artist for the spectator or performer to follow, or as a score to be reproduced. In GenAI-enabled art, the instruction is provided as a way of stimulating or prompting further action by the software to generate something new based on a distillation of its training data.

• Iteration: Iterative methods emphasise the interplay between textual outputs and their subsequent transformation. The textual output from one component is a prompt for the next, echoing the performative, recursive, and generative processes found in instruction art.

• Historical Continuity: Variability, introduced through the interaction of predicted labels, Markov chains, and language models, aligns this project with the historical traditions of combinatorial creativity, in which algorithmic transformations and chance interactions drive innovation.

• Framework of Instruction: Combinatory poetry “openly exposes and addresses its combinatorics by changing and permuting its text according to fixed rules” (Cramer 2001, 1). This project is driven by a framework of instruction that incorporates fixed rules, specifically relating to the core aspect of three components within an ensemble: the HAR (or object recognition) algorithm, the Markov chain generator, and the language model. The text being “permuted” is understood to be encoded in the statistical vectors contained in the language model’s latent space.

• Creative Uncertainty: The contentious juncture of human and “machine creativity” is subjected to a reframing, where automation, contingency, erroneous inference, and human intervention coalesce in the knowledge that not all things are, or should be, predictable. “Something else may, and does happen, and it is also our task to respond to such a provocation of the unknown” (Dubreuil 2025, 95).

• Reimagining Artistic Practice: Traditional art forms struggle to capture the scope of our evolving epistemology. This project proposes the invention of combinatorial approaches and conceptual frameworks to explore how speculative artistic methods can respond to an escalating and seemingly obligatory entanglement with proprietary AI systems.

Tip

Tip