By 2020, OpenAI’s GPT-3 had demonstrated that AI could generate astonishingly fluent text. The natural next question inside the lab was: what if we applied the same ideas to images?

The project was led by researcher Aditya Ramesh. The core insight was deceptively simple: instead of training an AI to predict the next word, what if you trained it to understand the relationship between words and images? Show it hundreds of millions of image-text pairs scraped from the internet — a photo of a red apple with the caption “a red apple” — and let it learn what words correspond to what visual concepts.

DALL-E launched in January 2021 — named as a portmanteau of Salvador Dalí (the surrealist artist) and WALL-E (Pixar’s imaginative robot). The name was deliberate: imaginative, slightly strange, and capable of things that shouldn’t be possible. The first version was limited to researchers, but even its early demos were jaw-dropping: avocado armchairs, snails made of harps, baby penguins wearing sombreros. Images that had never existed and could never be photographed.

When you type a description and DALL-E creates an image, four things happen in sequence — each building on the last.

If you’re satisfied with the plain-English explanation above, you can skip this section. If you want to actually understand what’s inside DALL-E, read on — still no equations, just the real mechanics.

CLIP — the bridge between words and images. DALL-E’s ability to understand what you mean by a prompt comes from CLIP (Contrastive Language-Image Pre-training), which OpenAI also built. CLIP was trained on 400 million image-text pairs collected from the internet. It learned to map images and text descriptions into the same numerical space — so that the vector representing a photo of a dog ends up close to the vector for the text “a dog.” This shared space is what allows DALL-E to translate your words into a visual target.

Why diffusion beats older approaches. Earlier image generation models (GANs — Generative Adversarial Networks) worked by having two competing neural networks: one that generates images, one that tries to distinguish fake from real. This worked but was famously difficult to train and produced specific failure modes (mode collapse, training instability). Diffusion models side-step these problems entirely. They’re trained to reverse a noise-addition process, which turns out to be both more stable to train and capable of higher-quality outputs. DALL-E 2 was one of the first large-scale demonstrations that diffusion outperforms GANs at this scale.

The prior — connecting text to image space. DALL-E 2 has a component called the “prior” that maps from the CLIP text embedding to a CLIP image embedding — essentially translating “what this description means” into “what an image matching this description would look like.” This prior is what allows such precise adherence to complex prompts. When you ask for “a photorealistic oil painting of a corgi wearing a Victorian top hat, warm lighting, bokeh background,” the prior understands each of these qualifiers and encodes them into the image target.

DALL-E 3, released in October 2023, addressed the single biggest frustration with earlier AI image generators: following complex instructions accurately.

Earlier models would interpret your prompt loosely — give them “a red ball on a blue table next to a green vase” and you’d often get a red thing near some blue with a splash of green. The exact spatial relationships, colours, and object counts would be wrong. DALL-E 3’s major improvement was training specifically on highly descriptive captions rather than the short, often vague descriptions that came with internet images.

The ChatGPT integration was arguably more important than any technical improvement. By embedding DALL-E 3 directly into ChatGPT, OpenAI removed the need to learn prompt engineering. You can describe what you want in natural conversation: “I need an image for my blog post about the importance of sleep — something warm, cosy, with a person in bed, maybe a book and a cup of tea nearby, soft illustrated style.” ChatGPT translates your conversational description into a detailed technical prompt and generates the image. The barrier to entry dropped to near zero.

Hands and faces are still hard. DALL-E has improved significantly but still struggles with human hands (too many fingers, wrong joints) and faces at small scales. This is a fundamental challenge for diffusion models — the training data contains enormous variation in how hands appear, and the model hasn’t fully learned their consistent structure.

Precise composition is difficult. Ask for exactly three red balls arranged in a triangle and you might get two or four balls in a rough cluster. Image AI understands concepts and styles more reliably than precise spatial logic. For highly specific compositional requirements, professional design tools are still more reliable.

The copyright question is genuinely unresolved. DALL-E was trained on images scraped from the internet — including the work of professional artists, photographers, and illustrators who never consented. Multiple lawsuits are ongoing. The legal and ethical framework for training AI on copyrighted creative work hasn’t been established. If you’re generating commercial work, this is a risk to understand. OpenAI argues this is fair use. Courts are still deciding.

Consistency across multiple images is hard. If you generate an image of a character and then want another image of the same character from a different angle, you won’t automatically get a consistent result. DALL-E has no concept of “the same character” between prompts. Maintaining visual consistency across a series requires careful prompt crafting and often still produces imperfect results.

Safety restrictions can be frustrating. DALL-E 3 is noticeably conservative about what it will generate — it refuses content it considers potentially sensitive, which sometimes catches legitimate creative requests in the filter. Photographers, artists, and game developers working with mature themes find this limiting.