04 Summary of Project Method
The following project summary shows how an assemblage of tools and techniques held together through poetic alliances functions as a framework of instruction, in which output from one stage feeds into the next. Various iterations of this method have been used to create several moving-image artworks (2020–2023). In these works, human performances depicted in footage are rationalised by a machine and transformed into poetic fictional narratives.
The following schematic (fig. 2) represents an assemblage comprising a number of different processes geared towards producing textual outputs in the form of categorical text labels and short sentences. After the initial HAR analysis of human action depicted in footage (fig. 3), there were interventions by the artist in order to curate lists of outputs, as indicated below in bold.
1) First, the HAR system searched for proximal matches within its training data.
2) Secondly, the resultant analyses (HAR labels) were fed to Python scripts as text files for “amplification” using WordNet1 to generate small corpora texts from the combined synsets and gloss, which were automatically cleaned and formatted into lists of strings.
3) A Markov chain generator2 model was then trained from the prepared small text corpus to produce short sentences conceptualised as “biometric poetry”, a fragmented language form, or cut-up derived from an algorithmic evaluation of human action. Outputs from this process were selected by the author and fed to the next process as prompts.
4) In the final phase, each short sentence was input into a more sophisticated language model (GPT-2) trained on the works of Michel Foucault, which autocompleted it into longer, more coherent sequences of text. Outputs from this process were selected by the author and handcrafted as textual overlays onto the edited video (fig. 4).
The Markov chain generator model output, “it was nothing”, was autocompleted by the language model in the style of Michel Foucault as, “it was nothing but a new syntax.”
Figure 4. Textual output from a language model superimposed on a film clip (Coughs03.mov in fig. 2) originally sourced from the film Coughs and Sneezes, one of the British public information films that was used to make Synset_Gloss (2020).
Figure 3. Screen recording of a Python script and HAR analysis of a British public information film clip, 2020.
(Below) examples of output from the Markov chain generator model, which were then used to stimulate the language model to perform an autocomplete:
The act of playing a musical instrument.
The action of taking part in a game or sport or other recreation.
The performance of a part or role in a drama.
Participate in games or sport.
Act or have an effect in a specified way or with a specific effect or outcome.
Play on an instrument.
Play a role or part.
Be at play; be engaged in playful activity; amuse oneself in a way characteristic of children.
Replay as a melody.
Perform music on a musical instrument.
Pretend to have certain qualities or state of mind.
Move or seem to move quickly, lightly, or irregularly.
Bet or wager money.
Engage in recreational activities rather than work; occupy oneself in a diversion.
Pretend to be somebody in the framework of a game or playful activity.
Emit recorded sound.
Perform on a certain location.
Put a card or piece into play during a game, or act strategically as if in a card game.
Engage in an activity as if it were a game rather than take it seriously.
Behave in a certain way.
Cause to emit recorded audio or video.
Manipulate manually or in one's mind or imagination.
Use to one’s advantage.
Consider not very seriously.
[...]
(Below) snippet of small corpus (1,385 words) generated from the labels listed above in WordNet, and consequently used to train a Markov chain generator model:
i played to the ground
toy with a position in your dismissal
they played a game rather than work; occupy oneself in a miracle
perform on their opponents
play it seriously
watch and killed it
the action of policemen outside every doorway
a figure that simulates military combat; players on their natural habitat
he planted a hacking tool
the hard fibrous lignified substance under the house will concede the mind
move suddenly
they ran the politicians
a figure that branches from which he cannot escape
i cannot escape
cut into the suspicions of a hacking tool
play it safe
he acted the age of children
cut with the tapes over again
we played a projectile in playing a particular manner
he played into pieces
play fair
engage in your dismissal
he never tires of feathers
introduce continuously
perform on a consequence
wreak havoc
behave carelessly or playful activity
force a diversion
exhaust by chopping
chase an animal into the cave
plant a certain location
profit from a tree
it was nothing
[...]