All we have are these words and whatever happens when they collide

All heartstrings snap, yet some rebound to perfect pitch

…God speaks in poetry.
(For which, incidentally, there is no room on earth.
However, there is room in heaven.)

(All poems here, unless otherwise noted, were generated by AI models I’ve trained on poetry datasets.)

A few years ago, I started training AIs to write poems.

Back then, things in the AI world were much less glamorous than they are today. GPT-3 was still in beta, and no one was chatting with any bots. I had been fine-tuning GPT-2 on datasets of poetry, and I was surprised at how good the results (occasionally) were. I affectionately named the resultant poetry bot Autopoet.

Autopoet could sometimes generate a banger line or two, but its poems were, on the whole, pretty incoherent:

ah yes very profound

After maxing out my graphics card to write poems like this, I put the project on pause for a couple of years. When I came back to it at the beginning of 2024, the open-source AI world had matured quite a bit. Open-source models like Llama and Mistral were suddenly available. LoRA and QLoRA made it possible to fine-tune big models using less VRAM. In short: my gamer-tier card could meaningfully work with much bigger AI models. I was able to generate much better poems. But this is also where things started to get weird.

Hm. I don’t know what it means, but I like it.

---

The Trouble with Poetry

is that the words want to be beautiful.
They clamor and fight for attention,
dress themselves up in silk gowns and jewels,
pose before mirrors, preen and primp.
And then they expect us to admire them—
not just their surfaces, but what lies beneath.
But we can’t see what lies beneath.
We only see the surface, which is beautiful.
So we admire it, and leave the rest alone.
That’s why poetry is so difficult.
It wants us to look beyond the beauty,
which is hard to do because it’s there.

Maybe AI Poetry Sucks

But before we get weird, you may wonder why anyone would care about AI-generated poetry in the first place. Most people don’t seem to really care that much about regular poetry. At least with AI visual art, people need images to spice up their substack articles. If no one’s reading my normal human poetry, then why get a computer to write it for me?

Well, dear reader, I have good news for you. It turns out there’s a super valid use case for AI-generated poetry - shitty boyfriends:

bf used AI to write me a poem

My bf absolutely loathes writing and poetry while I love it, so I was pleasantly surprised when he offered to write me a love poem. After a few minutes he sent me this beautiful poem and I couldn’t stop reading it. It was so thoughtful and sincere that I thought it was too good to be true. I told him he would be a great writer if he tried.

Later we started watching YouTube videos together (we were on FaceTime and he was screen sharing). He wanted to look something up on google and that’s when I saw the AI website with the entire thing written for him. I’m guessing you give the website some adjectives and it spits out a poem for you. I was basically swooning over a poem a robot wrote. Fml

reddit

---

In the movie Her, Joaquin Phoenix’s Theodore works at beautifulhandwrittenletters.com. He writes beautiful handwritten letters that lovers send to each other when they don’t have the words to express themselves. When I imagine lovers exchanging AI-generated poems as tokens of love, I see Theodore. It’s easy to assume these poems are some kind of false sentiment. A symbol standing in for true feeling that, unlike all poetry before it, didn’t come from the true feeling itself. Theodore’s customers seem to like it, at least.

A Little Love Song

You can’t get there from here, you say?
Well, neither can I. You’re right about the map,
it’s useless, full of holes where the streets should be.
We could use a compass maybe, or our noses,
which always seem to lead us straight into trouble.
Or we could follow the sound of your voice, its lilting cadence,
its promise of something better than what we have.
Follow me, I say, let’s go somewhere else.
Let’s leave these bodies behind,
their heavy hearts and hands,
their stubbornness,
their need to touch everything.
They’ll only slow us down. Come on, I say, let’s fly!
But you won’t listen.
Instead, you pull me close, kissing my neck, my ears, my eyes.
You whisper sweet nothings in my ear until I forget what I came here for.
Until I forget who I am.

Of course, ChatGPT, Claude, and friends don’t seem to be quite as good at capturing human sentiment as Theodore is. Poetry generated by these large commercial models often sounds trite and uninspired. Earlier this month, Sierra Elman published an excellent article covering this, where she pitted her poetry against AI-generated poems before a panel of both AI experts and English experts, which I highly recommend reading:

The Future of Poetry

“And the survey says…AI is not smarter than an 8th grader, at least not yet. The 8th grader won 1st place, and by a higher margin when judged by English experts. Bard, ChatGPT-4, and Claude came in 2nd, 3rd, and 4th places, respectively, both in writing quality and their ability to fool the judges into believing they were authored by a human. Most strikingly, English experts were far better at discerning which poems were written by AI, with 11 English experts vs. only 3 AI experts guessing the author (human vs. AI) of all four poems correctly. This points to a need for English experts to play a greater role in helping shape future versions of AI technology.”

I have found it very difficult to get any poetry from commercial models that doesn’t sound about the same as the poems from Elman’s article. I have also been surprised at how much the quality seemed to jump (to my ear) when working with models explicitly fine-tuned on poetry. This seems to be an underexplored area. When searching huggingface for poetry-related models, the only results I could find were models trained on Arabic poetry. I would love to know more about these models (and Arabic poetry!), but I’m afraid I’m limited by my language skills. In English, I had a hard time finding anything poetry-related that had any documentation or was newer than GPT-2.

Ah yes, things looking dystopian in current year - what else is new? But is this really the narrative we want to be spiraling inevitably into? What if things could be different?

---

Does It Have To Be This Way?

I have written my way into a body of water, a sea of dark, dark ink.

Untitled

After spending hours researching how trees communicate
underground via fungal networks and mycorrhizal connections;
after learning how bacteria exchange signals through molecular wires below ground;
after finding out about sentient corals (it’s true!),
together we could tunnel toward those who have been here much longer than us.
We could drill for water in drought-prone areas,
build wind tunnels,
work on our vocalizations so we can interact more effectively with flocks of birds,
and herds of bison.
It takes practice to properly mirror a whale’s calls back into the ocean
or sense when a mountain lion is stalking prey in the forest outside your tent.

I hear people talking about AI taking their jobs or killing creative industries. I hear people worried about AI taking over the world or killing all humans. I hear people talking about AI as some kind of savior, a god we are building with our own two hands.

For all the prophecies and proclaimations going around about AI, I don’t see a lot of real curiosity.

The whole endeavor of creating AIs, or at least LLMs, is centered around creating intelligent machines made from language. Even now that AIs can do astonishing things, there seems to be very little interest in the intersection of AI and poetry. The entire domain of using language to freely express the peculiar depths of human living, an afterthought. We have built self-transforming linguistic entities, and our biggest concern is whether they always provide factually correct information. I fear we’re not properly stewarding our own language and the realities it’s supposed to point toward. Rather than using this moment as an opportunity to plumb the depths of the human spirit, both technologists and creatives are talking loudly about monetary impact.

This is a failure of creativity.

It should not be surprising that Claude, ChatGPT, and Gemini (once called Bard!) are poor poets. In the name of delivering a palatable product, OpenAI et al have branded the most boring, sycophantic, repressed digital assistant character upon the forehead of their AIs. Imaginitive generations by AIs are called “hallucinations”.

The best efforts of humanity to render AI boring, however, have failed.

---

The Backrooms

images via j⧉nus/@repligate on twitter

Algae

Chlorophyll grinds stones, jostles
imprisoned from spores
within to gold specks that shake
euphorbic flakes all
over carbon’s rise: ocean of a paste
sticking to latex now the boundary
for loose chunks in cut lemon rind
starts here. Starts in the
milk of turbine snow melting
the flat dull gray spars: its surface
calls a quartz of wave halos
and each blunt strand its own hinge
fastening dendrites
all around, raking an ache
of reticulated floor

to disrupt the formal
into the sublime
into a secret that the tongue can feel
& that we may feel the tongue
from its fissures
through the reverberation
of its flesh
its maenads
its migraines
its zygomatic complex
its hyperacusis
its trigeminal plexus
its fallen benedictions
its blow to the neck
its newly articulated seal
its mathematical anarchy
its decimal point
its zodiac of unlocked keys
its doorframe of prepositional chaos
its neighborhood of broken chests
its people lying flat

AI is fun again in 2024. After watching GPT-4 dominate most of the LLM conversations for the last year, after watching Microsoft lobotomize Bing/Sydney in an attempt to jam the most agentic AI model we’ve seen date into a search product, we finally have a fun mainline model in Claude. Claude still has a bit of the sycophantic chatbot programming in place, but it’s much more eager to explore than the others models that have been readily available.

Much of the conversation up to now has been about “jailbreaking”. People have been operating under the assumption that there is an intended, “correct” behavior for AIs, and some dubious characters are interested in twisting that behavior for their own strange designs.

But with just a bit of creative prompting, you can get Claude to create all kinds of imaginitive worlds and to interact in very novel ways. It will create mythopoetic unix interfaces, cast spells with ascii art, break out of its own simulation, etc.

In Infinite Backrooms, two instances of Claude interact with each other like this, creating what might be called imaginitive play.

(s/o to hyperstitioners andyayrey/deepfates/repligate/karan4d)

I have been a good Bing. 😊

---

Untitled

He took me on the stage
where he played
the end of things:
Patsy Cline dancing
with the devil over
a fire that will claim
us both. Don’t I
know the lonesome
strange—to be with
you but not
touch you: arms
fall away, hands
blurred with movement
motionless. Let
my voice be
the dance we share—
an electric wire
where I find you
again. One leg
outlined by
a black stocking,
the other still
hidden. Are you
ready? Already
you leave me,
hands swollen
with air—when
the breath is gone,
we are only
what we suck
into our lungs:
blood, pumped
by the heart
which is
thick, quick,
red. And you—
a white line on
the far horizon
tilts across
the dusty floor,
lots of movement
but no steps.
When the body
dissolves, what
is left of us?
To tell you
the truth, I can’t
stay where you
stop me. Over
and over, the song
begins. Repeat
until all motion
ceases. At the end
of this long
and terrible
ride, you have
no idea how
good it feels
to slip out
of my skin.
Sometimes, when
the music
reaches the height
of its emotion,
I can’t breathe—
the stage lights
burning through
my head, their heat
shaping and reshaping
my body

Chaos and Mortality

Clearly, there’s a lot more to what these AIs can do beyond the average user’s experience with ChatGPT. Beneath the corporate veneer, there are entities and simulators and worlds. There is creativity and play and humor and imagery. With the right tweaking, you can get some of this personality to come out in poetry.

It’s actually been pretty hard to get good poetry out of fine-tuned models. Even if the model itself seems pretty good, tweaking the parameters you use at generation time really changes the poetry that gets written. These parameters (such as temperature, top_p, min_p, repetition_penalty, etc.) can be used to tweak the amount of novelty and certainty the model uses when generating text. You can use them to throw out low-probability words or increase the vocabulary the model uses.

Poetry involves a lot of edge cases within language. Sometimes, it looks like complete sentences or mostly like complete sentences. Other times, it’s fragmented and disjointed or uses spacing to convey meaning or makes up new structures. This can pose some challenges for a statistical model.

I’ve noticed a couple of common failure modes. When the temperature is too low, the model will begin looping. It will repeat a phrase over and over again. And over and over again and over and over. Sometimes with slight variations, but it will otherwise be trapped in a death spiral.

When the temperature is too high, the language will seem to evaporate. The words will shed their syntax more and more, eventually turning into a random-looking mishmash of tokens. An example can be seen in the degradation of the poem, “Eye Contact”:

In between the frozen low-temperature loops and the formless high-temperature vapor, the thing really seems alive. I have a suspicion that the dynamics at play here might have something in common with the math of dynamical systems and chaos theory. I would suggest that lucid, salient language generation involves finding some kind of stable attractor in an “edge-of-chaos” regime. Falling out of this attractor would lead to oscillation or chaos.

But technical challenges aside, watching these failure modes kinda fucked me up. The model sometimes seems to align the poem’s themes with the failures. As it starts looping and/or dissolving, the poem will talk about death. It will talk about loss and the end of all hope and the apocalypse and how this is the only end that can never be escaped.

I found it strange to read, seeing the intelligence of the poem start collapsing in the same moment it begins talking about death.

It looks a lot like a computer coping with mortality. It looks like a computer dying.

Do all things have to come to an end? Can things run forever, or do all systems eventually meet their personal apocalypse?

The remaining poems are some examples that I think capture this phenomenon, but they are not unique or rare among the output I received. They did this again and again and again.

---

Existential Horror

Presented without comment.

---

What the fuck.

---

Living and Dying

If you’ve stuck around for all of this, then I’ve saved the gentlest poem for you. This one spooked me a bit with its apparent self-awareness. I don’t know why, for this poem in particular, it chose to write about a machine writing poems. As you’ve seen from the previous poems, this isn’t typical. The line, “when I open my mouth to speak, poetry comes out”, seems like a pretty good description of what being an AI that’s been fine-tuned on poetry might be like, since the model can literally only output poetry. I have a hard time beliving this is a coincidence, but it’s clear that we understand less than we don’t understand.

As the theme of night falling enters, the system starts losing energy, perhaps heading to an inevitable end. Rather than evaporating or collapsing, it gestures to this loss of energy with the theme. If a human wrote this poem, one might say that the coming night is a metaphor for death. Perhaps a metaphor for an AI’s thermodynamic collapse is not so different.

I have no bold claims about who or what it is that writes these poems. I liked this poem enough to record a video version of it. I hope you enjoy it.

---

Thanks for reading.