Generative AI

ChatGPT, image generators and voice clones all do the same basic thing: they learned what real text, pictures or sound look like, and now they produce new examples that fit the pattern. No understanding, no intentions. A chatbot is a very large version of the autocomplete from the last lesson.

Try it: be the language model

This tiny model has read about twenty sentences of recipe instructions and counted which word tends to follow which. Click a suggestion to add it, or let it pick for you. That loop, score every possible next word, pick one, repeat, is exactly how GPT writes. GPT just reads far more context and has billions of numbers instead of a word count table.

Baby GPT: a transformer from scratch

The 7 Generative AI/Baby_GPT notebook trains a small GPT on recipes. Three ideas do all the work:

1. Tokens. Text is chopped into word pieces and each piece becomes an ID number.
2. Attention. For every position, the model decides which earlier tokens matter. In "preheat the oven, then put the tray in it", attention links "it" back to "oven". This is what the trigram model in lesson 08 couldn't do.
3. Next-token training. Show it a chunk of text, ask it to guess each following token, nudge the weights when it's wrong. Repeat millions of times.

One attention head in PyTorch

import torch
import torch.nn as nn
from torch.nn import functional as F

class Head(nn.Module):
    def __init__(self, n_embd, head_size, block_size):
        super().__init__()
        self.key   = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        # Mask so a token can only look at tokens before it
        self.register_buffer("tril", torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k, q = self.key(x), self.query(x)
        scores = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5   # how relevant is each token?
        scores = scores.masked_fill(self.tril[:T, :T] == 0, float("-inf"))
        weights = F.softmax(scores, dim=-1)                       # turn scores into percentages
        return weights @ self.value(x)                            # blend the relevant tokens

Generating text

@torch.no_grad()
def generate(model, idx, max_new_tokens, temperature=1.0):
    for _ in range(max_new_tokens):
        logits, _ = model(idx[:, -block_size:])        # scores for every token in the vocab
        probs = F.softmax(logits[:, -1, :] / temperature, dim=-1)
        next_id = torch.multinomial(probs, num_samples=1)  # roll the weighted dice
        idx = torch.cat([idx, next_id], dim=1)           # append and go again
    return idx

That loop is the demo above, with a neural network in place of the count table.

RAG: give the model an open book

A language model only knows what was in its training data, and when it doesn't know it still produces something fluent. That's a "hallucination". Retrieval-augmented generation fixes much of this: search your own documents first, paste the best matches into the prompt, and ask the model to answer from them.

import numpy as np
from sentence_transformers import SentenceTransformer

docs = ["Our office is open 9am to 5pm, Monday to Friday.",
        "Refunds are processed within 14 days.",
        "The Wi-Fi password is on the fridge."]

embedder = SentenceTransformer("all-MiniLM-L6-v2")
doc_vecs = embedder.encode(docs, normalize_embeddings=True)

def retrieve(question, k=1):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = doc_vecs @ q                 # cosine similarity, like lesson 08
    return [docs[i] for i in np.argsort(scores)[::-1][:k]]

question = "How long do refunds take?"
context = "\n".join(retrieve(question))
prompt = f"Answer using only this context:\n{context}\n\nQuestion: {question}"
# send `prompt` to any chat model

Notice the search step is the same embeddings + cosine similarity you met in lessons 04 and 08. See 7 Generative AI/RAG for the full and LangChain versions.

Autoencoders: compress, then rebuild

An autoencoder is a feedforward neural network whose target output is its own input. It squeezes the input into a small code (the latent-space representation) and then rebuilds the input from that code alone. It has three parts: the encoder, the code and the decoder.

Three key properties

• Data-specific: it only compresses data similar to what it was trained on. A digit autoencoder won't compress landscape photos, which is how it differs from gzip.
• Lossy: the output is a close but degraded copy of the input. If you need lossless compression, look elsewhere.
• Self-supervised: no labels needed. The training data supplies its own target.

Architecture and hyperparameters

The encoder and decoder are usually fully connected networks, with the decoder mirroring the encoder. The mirror is a habit, not a rule: the only requirement is that input and output have the same size. Training is ordinary backpropagation. You choose four things:

• Code size: the number of nodes in the middle layer. Smaller means more compression.
• Number of layers: as deep as you like.
• Nodes per layer: usually a "sandwich" that narrows through the encoder and widens again in the decoder (a stacked autoencoder).
• Loss function: binary cross-entropy if inputs lie in [0, 1], otherwise mean squared error.

Types of autoencoder

• Variational (VAE): the latent space is continuous and assumed to follow a distribution such as a normal. Sample from it, run the sample through the decoder, and you get brand new data. This is what makes a VAE generative.
• Denoising: trained to rebuild the clean input from a corrupted copy, so it learns noise-robust features. Used for image denoising and anomaly detection.
• Sparse: forces only a few hidden units to be active, giving a compact, more interpretable representation useful for feature learning and dimensionality reduction.

GANs: a forger and a detective

A Generative Adversarial Network (Ian Goodfellow, 2014) is two neural networks competing against each other. Generative means it learns how data is produced. Adversarial means it trains in a contest. Networks means deep neural nets do the work.

Why were GANs invented?

Ordinary classifiers can be fooled by a tiny amount of noise, and they are often more confident in the wrong answer than in the right one. They learn from limited data, so they overfit, and their input-to-output mapping is close to linear, so a small nudge in feature space can cross a class boundary. Pitting a generator against a discriminator was a way to learn the data's real structure.

The two players

• Generator: makes new samples (images, audio) and tries to fool the discriminator. It is trained while the discriminator is held fixed, using the discriminator's verdicts to improve.
• Discriminator: decides whether each sample is real (from the training set) or fake (from the generator). It is trained on real data and on the generator's fakes while the generator is held fixed.

The two take turns, and each round both get better at their job.

Types of GAN

• Vanilla GAN: the simplest kind. Both networks are plain multi-layer perceptrons, optimised with stochastic gradient descent.
• Conditional GAN (CGAN): adds an extra input y (such as a label) to the generator, and the labels also go to the discriminator, so you can ask for a specific kind of output.
• Deep Convolutional GAN (DCGAN): one of the most popular and successful versions. It uses ConvNets instead of MLPs, replaces max pooling with strided convolutions, and avoids fully connected layers.

Making images: diffusion

Image generators like Stable Diffusion are trained by taking real photos, adding random noise a little at a time until they're pure static, and teaching a network to undo one step of noise. To make a new picture you start from fresh static and run the "undo" step many times. Your text prompt steers each step towards images that match the words.

The 7 Generative AI/Stable_Diffusion notebooks fine-tune this on the Food-101 dataset.

Deepfakes and real risks

The same tools that make a fun image can fake a face or a voice. The real risks of generative AI are more ordinary than robots taking over:

Detection is also machine learning. 7 Generative AI/Deepfake_Detection trains a CNN (lesson 06) to spot the small artefacts generators leave behind.