Natural Language Processing

NLP gives computers the ability to understand, generate, and reason about human language. It underpins search engines, chatbots, translation systems, and modern LLMs. This section covers the foundational building blocks: turning words into vectors, measuring word association, and building simple language models.

Word2Vec

Word2Vec learns dense word embeddings from raw text. The core idea: "a word is known by the company it keeps." Words that appear in similar contexts get similar vectors.

The skip-gram model predicts context words given a target word. Training adjusts the embedding matrix so that similar words end up close together in vector space.

NumPy Implementation (Skip-gram)

import numpy as np

class Word2Vec:
    def __init__(self, vocab_size, embed_dim=10, lr=0.01):
        self.w1 = np.random.randn(vocab_size, embed_dim) * 0.01
        self.w2 = np.random.randn(embed_dim, vocab_size) * 0.01
        self.lr = lr

    def softmax(self, x):
        e = np.exp(x - x.max())
        return e / e.sum()

    def forward(self, x):
        h = x @ self.w1          # hidden layer (embedding lookup)
        u = h @ self.w2           # output scores
        return self.softmax(u), h

    def train_step(self, x, y_true):
        y_pred, h = self.forward(x)
        error = y_pred - y_true   # output error
        # Backprop
        dw2 = np.outer(h, error)
        dw1 = np.outer(x, error @ self.w2.T)
        self.w1 -= self.lr * dw1
        self.w2 -= self.lr * dw2
        return -np.log(y_pred[y_true.argmax()] + 1e-9)  # cross-entropy loss

Using Gensim (Production)

from gensim.models import Word2Vec
from nltk.tokenize import word_tokenize

corpus = [word_tokenize(s.lower()) for s in sentences]

model = Word2Vec(corpus, vector_size=100, window=5,
                 min_count=1, workers=4, epochs=10)

# Find most similar words
print(model.wv.most_similar('machine', topn=5))

# Word arithmetic: king - man + woman ≈ queen
result = model.wv.most_similar(
    positive=['king', 'woman'], negative=['man'], topn=1
)
print(result)

PMI & Word Collocations

Pointwise Mutual Information (PMI) measures how much more often two words co-occur than if they were statistically independent. High PMI = strong association.

from nltk.collocations import BigramCollocationFinder, BigramAssocMeasures
from nltk.tokenize import word_tokenize

text = """natural language processing and machine learning are
          subfields of artificial intelligence and machine learning
          overlaps with natural language processing significantly"""

tokens = word_tokenize(text.lower())
finder = BigramCollocationFinder.from_words(tokens)

# Score all bigrams by PMI
measures = BigramAssocMeasures()
for bigram, score in finder.score_ngrams(measures.pmi):
    print(f"{bigram[0]} + {bigram[1]:<20} PMI = {score:.2f}")

Collocations with high PMI — like "machine learning" or "natural language" — are phrases that carry meaning together. Knowing this is crucial for tokenization and phrase detection.

Cosine Similarity Explorer

Drag the sliders to set two user interest vectors across 4 topics. Watch how cosine similarity changes — it measures the angle between vectors, not their magnitude.

Language Models

A language model assigns a probability to a sequence of words. Given previous words, it predicts what comes next — the foundation behind autocomplete, GPT, and text generation.

Trigram Language Model

from collections import defaultdict
import random

def build_trigram_model(corpus):
    model = defaultdict(lambda: defaultdict(float))
    for sentence in corpus:
        words = sentence.split()
        for i in range(len(words) - 2):
            w1, w2, w3 = words[i], words[i+1], words[i+2]
            model[(w1, w2)][w3] += 1

    # Convert counts to probabilities
    for context, nexts in model.items():
        total = sum(nexts.values())
        for w in nexts:
            nexts[w] /= total
    return model

def generate_text(model, seed, n=20):
    words = seed.split()
    for _ in range(n):
        context = (words[-2], words[-1])
        next_words = model.get(context)
        if not next_words:
            break
        words.append(random.choices(
            list(next_words.keys()),
            weights=list(next_words.values())
        )[0])
    return ' '.join(words)

model = build_trigram_model(reuters_sentences)
print(generate_text(model, seed="the stock market"))

Why Trigrams Fall Short

Neural language models (LSTMs, Transformers) solve all three — they generalise across vocabulary and capture arbitrarily long context via attention.

Knowledge Check

Test your understanding — pick the best answer, then click Check.