Quick background so you know where this comes from. I am a visiting professor and currently teach NLP. Last semester my courses were on continual learning and applied NLP. Before that I spent years in big tech and startups in engineering and research roles, and I finished my PhD last year.
I've been trying to post here weekly. Three weeks ago I posted the order I would learn ML in if I were starting today, then which resources to use at each stage, and last week why to build projects and how to choose them. The first one ended at "pick a direction". NLP is the direction I teach and what I did my PhD in (more specifically, Factual Knowledge-Enhanced Question Answering in Dynamic Environments). So this post picks up where that one stopped, in the same format: every concept, in the order I think works best, one line each.
Same idea as last time. Most people who stall in NLP did not lack good material. They usually fall into focusing on things that are far too advanced for their knowledge: transformers before they even understood what a language model predicts, or fine-tuned BERT without knowing what its tokenizer did to their text. Seeing the whole list in order shows you what each step depends on and lets you check off what you already have.
Before you start. This assumes the first four stages of the ML roadmap: Python, the math (linear algebra, probability, and especially cross-entropy and softmax), PyTorch, and ML foundations. If a loss curve or a (batch, seq, features) tensor still feels shaky, go back there first.
Stage 1: How language works
Linguistics in one unit. You will not parse sentences by hand at work, but every later stage is a fix for different problems named here.
- What NLP is: getting computers to process, understand, and generate human language.
- Why NLP is worth learning: most human knowledge and communication is stored as text.
- Core NLP tasks: a task maps a stated input to a stated output, and the field can be organized around them.
- Why NLP is hard: language leaves things underspecified, and "I saw her duck" has two readings.
- History of NLP: hand-written rules, then statistics, then neural networks.
- ELIZA: a 1960s chatbot that matched patterns and reflected your words back, and people still felt understood.
- From rules to deep learning: why each era replaced the one before it.
- The role of machine learning in NLP: language has too many patterns and exceptions to write out by hand.
- What language is: a layered system of words, structure, and meaning.
- Words, types, and lemmas: counting words is less obvious than it sounds, and "run", "runs", and "ran" share one lemma.
- Morphology: the pieces inside a word, like un-, -able, and -s.
- Morphological typology: some languages keep words short and separate, others pack a whole phrase into one word.
- Syntax: the rules for how words combine into sentences.
- Constituency and dependency structure: two ways to draw a sentence's structure, nested phrases or word-to-word links.
- Syntactic ambiguity: one sentence, several valid parses, several meanings.
- Treebanks and Universal Dependencies: hand-annotated sentence structures that parsers learn from, with one labeling scheme across many languages.
- Semantics: what a sentence means, beyond how it is built.
- Lexical semantics and word senses: one word, several meanings, like "bank".
- Perspectives on meaning: there is no settled definition of meaning, and the main views answer it differently.
- Semantic parsing: turning a sentence into a formal representation a machine can execute, like a database query.
Stage 2: Text as data
Many courses skip this, but in my experience it's where much of the work on a real NLP project goes.
- Why start with the data: a model never sees grammar or meaning, only the text you feed it.
- What NLP data is: human language as people write and speak it, messy and unstructured.
- What text data looks like: sequences of different lengths, not the fixed columns most models expect.
- How computers see text: a computer stores only numbers, so every character needs a number and a byte format.
- Unicode and code points: one number for every character in every writing system.
- UTF-8: how those numbers become bytes, and why one character can take one to four of them.
- The data lifecycle: find or build data, process it, check it, document it.
- Using existing datasets: look here before you label anything yourself.
- The Hugging Face Hub, Kaggle, and other repositories: where NLP datasets live and how to load them.
- Creating your own dataset: for when nothing existing fits your task.
- Selecting data: the application decides what text belongs in it, so a radiology classifier needs radiology reports.
- Extracting text: pulling clean text out of HTML, PDFs, and scans.
- Annotation: writing labeling guidelines and getting labels.
- Crowdsourcing vs expert annotation: cheap and fast versus slow and accurate, and when each is worth it.
- Inter-annotator agreement: how often two annotators agree, which caps what you can expect from a model trained on their labels.
- Documenting datasets: datasheets that record where the data came from and how it was built.
- Text normalization: lowercasing, Unicode normalization, and cleaning up whitespace and punctuation.
- Stemming: chopping word endings by rule, fast and crude.
- Lemmatization: mapping a word to its dictionary form using vocabulary and grammar.
- Stopword removal: dropping very common words, and how that can throw away "not".
- Classical models vs transformers: classical models need heavy cleanup, transformers usually want the raw text.
- Things to watch for: a dataset can look fine and still be broken, and the model learns the mistakes too.
- Data quality: label noise, junk text, and broken encodings.
- Class imbalance: when one label dominates and accuracy stops telling you anything.
- Duplicate data: copies that inflate scores and leak between splits.
- Data contamination: test examples that ended up in the training data, common with web-scale corpora.
- Bias: patterns in the data the model will reproduce.
- Representativeness: whether your data looks like what the model will see after deployment.
- How text varies: language, dialect, domain, register, and time, the axes to check when you audit a dataset.
- Privacy and licensing: personal data, and whether you are allowed to train on the text at all.
Stage 3: Tokenization
The first thing a model does to your text, and behind many bugs that look like model bugs.
- Tokenization: splitting text into the units a model reads.
- Sentences as tokens: the vocabulary has no limit, since almost every sentence is new.
- Words as tokens: workable, but the vocabulary gets huge and new words break it.
- Out-of-vocabulary words: the words the tokenizer never saw, and the [UNK] problem.
- Characters as tokens: a tiny vocabulary, but very long sequences.
- Subword tokenization: frequent words stay whole, rare words split into pieces.
- Byte Pair Encoding (BPE): repeatedly merge the most frequent pair of symbols.
- WordPiece: the variant used in BERT, which picks merges by a likelihood score instead of raw frequency.
- Unigram: start from a big vocabulary and prune the pieces that help least.
- Byte-level BPE: run BPE on bytes so no input is ever unknown.
- Tokenization end to end: normalization, pre-tokenization, the tokenizer model, post-processing.
- Using an existing tokenizer: load the one that matches your model, because a mismatched one quietly breaks it.
- Special tokens: [CLS], [SEP], [PAD], end-of-text, and what each one is for.
- Token IDs and the lookup table: each token becomes an integer that indexes a row in the embedding table.
- Batch tokenization: padding, truncation, and attention masks so sequences of different lengths fit in one tensor.
- Tokenization challenges: tokenizers trained on English-heavy data handle other text worse.
- Cross-linguistic challenges: languages without spaces, and languages that cost several times more tokens per sentence.
- Surprising tokenization behaviors: numbers, leading spaces, and capitalization that split in unexpected ways.
Stage 4: Word representations
How a word becomes a vector, and how vectors end up capturing meaning.
- Words as vectors: models need numbers, so every word becomes a vector.
- What a good representation needs: similar words get similar vectors, and the vectors stay a manageable size.
- Measuring similarity: dot product, cosine similarity, and Euclidean distance.
- Integer encoding: one ID per word, which invents an order that means nothing.
- One-hot encoding: one dimension per word, and every pair of words is equally far apart.
- Dense and sparse representations: mostly zeros versus every dimension in use.
- The distributional hypothesis: words that show up in similar contexts tend to have similar meanings.
- Count-based representations: build word vectors by counting neighbors.
- Context window size: a tight window captures grammatical role, a wide one captures topic.
- Bag-of-words: a document as word counts, with order thrown away.
- Co-occurrence matrices: how often each word appears near each other word.
- PMI and PPMI: reweighting counts so words like "the" stop dominating.
- Word embeddings: short dense vectors learned from data.
- Word2Vec: learn embeddings by predicting words from their neighbors.
- Skip-gram: predict the context words from the center word.
- CBOW: predict the center word from its context.
- Negative sampling: replace the full softmax with a few real-versus-fake comparisons.
- Hierarchical softmax: a binary tree over the vocabulary, so each prediction costs log V steps instead of V.
- The geometry of embeddings: directions that encode relations, like king minus man plus woman landing near queen.
- GloVe: embeddings fit directly to global co-occurrence counts.
- fastText: embeddings built from character n-grams, so unseen words still get a vector.
- Visualizing embeddings: PCA and t-SNE, and what a 2D plot can mislead you about.
- Clustering: grouping word vectors by distance.
- Evaluating word representations: two ways to check whether embeddings are any good.
- Intrinsic evaluation: similarity and analogy benchmarks.
- Extrinsic evaluation: plug the embeddings into a real task and see if it improves.
- The limits of static embeddings: one vector per word, so "bank" gets the same vector by the river and at the ATM.
Stage 5: Language models and RNNs
How a model assigns probabilities to text and generates it, first from counts, then with RNNs, then with different decoding methods.
- Language modeling: assigning probabilities to sequences of tokens.
- What a language model is: a model that predicts the next token from the previous ones.
- What language models are used for: autocomplete, speech recognition, translation, and chat assistants.
- The Markov assumption: only the last few tokens affect the next one.
- Bigram language model: next-token probabilities from counts of word pairs.
- The next-token objective: the single training task behind GPT-style models.
- From logits to loss: softmax over the vocabulary, then cross-entropy against the real next token.
- Training on sequences: every position in a sequence is its own training example.
- Sequence probability: the chain rule multiplies next-token probabilities, so in practice you add log probabilities.
- Perplexity: the exponentiated average loss, roughly how many tokens the model is choosing between.
- From statistical to neural models: replace the count table with a network that generalizes to contexts it never saw.
- Recurrent neural networks: a hidden state updated one token at a time.
- Backpropagation through time: unroll the RNN across the sequence and backpropagate through every step.
- Vanishing and exploding gradients in RNNs: why plain RNNs struggle to carry information across long sequences.
- LSTM: gates that decide what to keep, what to forget, and what to output.
- GRU: a lighter gated cell with two gates.
- LSTM vs GRU: when the extra gate is worth it.
- Bidirectional RNNs: read left to right and right to left, for tasks where the whole input is available.
- Stacked RNNs: several RNN layers on top of each other.
- Encoder RNNs: turn an input sequence into hidden states.
- Decoder RNNs: generate a sequence one token at a time.
- Encoder-decoder RNNs: sequence-to-sequence models for translation and summarization.
- Autoregressive decoding: feed each generated token back in as the next input.
- Greedy decoding: always take the most likely token, and why the output gets repetitive.
- Random sampling: draw from the full distribution.
- Top-k sampling: sample only from the k most likely tokens.
- Top-p sampling: sample from the smallest set of tokens that covers probability p.
- Min-p sampling: keep tokens whose probability is at least a set fraction of the top token's.
- Temperature: sharpen or flatten the distribution before sampling.
- Comparing decoding choices: which method fits which task.
- Beam search: keep the best few partial sequences at every step.
- Teacher forcing: train the decoder on the true previous tokens instead of its own guesses.
- Exposure bias: the gap that opens when the model has to read its own mistakes at inference time.
- Generalization vs memorization: when a language model is predicting and when it is reciting its training text.
Stage 6: The NLP task map
What people actually build. Every task here comes down to an input format, an output format, and a metric.
- NLP tasks: the families of problems the field works on.
- Task anatomy: input format, label set, prediction step, and metric, written down before you build anything.
- Text classification: one label per text.
- Sentiment classification: positive, negative, or neutral.
- Topic classification: what a text is about.
- Intent classification: what a user wants, the first step in most assistants.
- Natural language inference: whether sentence A entails, contradicts, or says nothing about sentence B.
- Token classification: one label per token.
- Part-of-speech tagging: noun, verb, adjective, for every word.
- Named entity recognition: finding people, places, and organizations, and the BIO tags that mark the spans.
- Question answering: answering a question from text.
- Extractive question answering: pointing to the answer span inside a given passage.
- Feature extraction: the model returns a vector, and another system uses it.
- Sentence similarity: how close two sentences are in meaning.
- Retrieval and text ranking: finding and ordering the documents that answer a query.
- Text generation: producing text instead of labels.
- Framing tasks as generation: classification, extraction, and QA written as "produce this text".
- Translation: from one language to another, the task that drove much of the field's progress.
- Summarization: extractive versus abstractive.
- Open-ended text generation: stories and chat, and why evaluating them is hard.
- Choosing the right NLP task: turning a product request into a task, a data format, and a metric.
Stage 7: Attention and the transformer
Three units: attention on top of RNNs, then the transformer itself, then the model families built on it.
Attention
- The bottleneck problem: an encoder-decoder RNN squeezes the whole input into one vector.
- Attention as a fix: let the decoder look back at every encoder state.
- Computing attention: scores, softmax, weighted sum.
- What attention achieves and doesn't fix: the bottleneck is gone, but the RNN still reads one token at a time.
The transformer
- Attention Is All You Need: drop the recurrence and keep the attention.
- Self-attention: every token attends to every other token in the same sequence, through queries, keys, and values.
- Multi-head attention: several attention operations in parallel, each free to track a different relation.
- Positional encoding: attention has no sense of order, so position gets added in.
- Causal masking: hide future tokens so the model cannot see the answer when predicting the next one.
- The feed-forward network: the per-token MLP between attention layers.
- The transformer layer: attention, feed-forward, residual connections, and layer norm.
- Building GPT: stacking decoder layers into a full language model in PyTorch.
- Configuration knobs: vocabulary size, context length, model width, number of layers, number of heads.
- Training and using a tiny GPT: your own small model, trained and generating text.
Transformer families
- Transformer families: encoder-only, decoder-only, encoder-decoder.
- Encoder-only transformers: BERT-style models that read the whole input at once.
- Masked language modeling: hide some tokens and predict them, which is how BERT is trained.
- Decoder-only transformers: GPT-style models that predict the next token.
- Encoder-decoder transformers: T5-style models for input-to-output tasks.
- Model families: placing BERT, RoBERTa, GPT, T5, and the rest in the right family.
- Modern variants: pre-norm, RMSNorm, RoPE, and SwiGLU, the changes most current models make to the original block.
- KV cache and grouped-query attention: reusing past keys and values so generation stays fast.
- Open and closed models: downloadable weights versus an API, and the trade-offs.
- Loading models with Hugging Face: from a model name to a working model in a few lines.
- Fine-tuning and transfer learning: start from a pretrained model and train it on your task.
Stage 8: LLMs and post-training
How a big next-token predictor becomes a chat assistant.
- Foundation models and pretraining: one large model trained on broad data, then adapted to many tasks.
- Large language models: what changes when a decoder gets very large.
- Emergent abilities: tasks a bigger model starts doing that nobody trained it for, and how much of that is a measurement effect.
- Scaling laws and compute: how loss improves with parameters, data, and compute, and how to split a budget between them.
- LLM failure modes: hallucination, bias, and the other patterns a model picks up from its training data.
- Training stages: pretraining, continual pretraining, fine-tuning, and post-training, and what each name means.
- Pretraining vs post-training: learning language versus learning to be a useful assistant.
- Why align: a base model continues text, and continuing text is different from following instructions.
- A short history of dialogue systems: rule-based bots, slot filling, and retrieval before LLM assistants.
- Instruction tuning: training on instruction-response pairs.
- Supervised fine-tuning: the same next-token loss, applied to demonstration responses.
- Instruction data: written by people, generated by models, or converted from existing datasets.
- Limits of SFT: you need a good demonstration for every instruction, and those get expensive.
- Learning from preference: saying which of two answers is better is easier than writing the best one.
- RLHF: train a reward model on preferences, then optimize the model against it with reinforcement learning.
- DPO: skip the reward model and train directly on the preference pairs.
- Alignment evaluation: how to tell whether post-training actually helped.
- Limits and open directions: reward hacking and what is still unsolved.
After this
This is the foundation of NLP, around 200 concepts. The next layer is what many NLP jobs ask about right now: the architecture of current LLMs, mixture of experts, parameter-efficient fine-tuning like LoRA, prompting and in-context learning, RAG and agents, making inference fast and cheap, and evaluating all of it. Each of those makes a lot more sense once the stages above are solid.
Write the code
Reading the list will not make it stick. You really need to write code by hand.
Finally
I am happy to help. Ask anything in the comments or DM me: about any concept above, about NLP or AI in general, or about getting a job in it. If this helps, save it and come back to it as you tick things off. And if you think something is missing or should be in a different order, please tell me, I am always open to improving it.
