From Words to Vectors

Before we can talk about transformers, we need to talk about the problem they solve. Computers don’t understand words; they understand numbers. The whole history of NLP is really the history of people finding cleverer ways to turn words into numbers while keeping what the words mean.

Previously, on NLP

The earliest approach was embarrassingly simple: count the words. “Bag-of-words” throws every word in a document into a bag, counts the frequencies, and calls it a representation. TF-IDF is the same thing but slightly smarter, weighting words by how rare they are across documents.

This is basically what keyword search does. When Google works well, it’s bag-of-words plus PageRank plus a lot of engineering; when it fails badly, it’s because meaning needs more than word frequency.

Then in 2013, Word2Vec changed everything. The idea was to represent each word as a dense vector (a list of, say, 300 numbers) in a space where nearness means likeness. You train it by predicting neighboring words in a sentence, so words that show up in similar contexts end up with similar vectors.

The famous result: $\text{king} - \text{man} + \text{woman} \approx \text{queen}$. The vector arithmetic captures semantic relationships, and yes, every Word2Vec explainer uses this example because it’s a cliche. The more interesting thing is where it breaks.

That’s the bigger problem: one vector per word, period. “I went to the bank to deposit money” and “I sat on the river bank” give “bank” the same vector, with no context sensitivity at all.

Then came RNNs and LSTMs. These process words one at a time, keeping a hidden state that should carry context forward. In practice, the information decays exponentially; by the time you’re 50 words into a sentence, the model has essentially forgotten the beginning. They were also painfully slow because you can’t parallelize sequential processing.

This is why pre-2017 chatbots had such terrible memory: it wasn’t a design choice but a hard constraint of the architecture.

The attention breakthrough

In 2014, Bahdanau et al. had a simple but powerful idea for machine translation: instead of compressing the whole input sentence into a single hidden state vector, let the decoder look back at all the encoder states and decide which ones matter for each word it’s generating.

This worked very well, but it was still coupled with the slow sequential processing of RNNs. The real breakthrough came in 2017, when Vaswani et al. asked: what if we only use attention? No recurrence, no sequential processing, just attention all the way down. That paper, “Attention Is All You Need,” introduced the transformer.

Tokens

Something that surprises most people: LLMs don’t see words. They see tokens.

Here’s what tokenization looks like in practice:

1
import tiktoken
2

3
enc = tiktoken.encoding_for_model("gpt-4")
4

5
# Simple sentence
6
tokens = enc.encode("The transformer architecture is elegant.")
7
print([enc.decode([t]) for t in tokens])
8
# ['The', ' transform', 'er', ' architecture', ' is', ' elegant', '.']
9

10
# The classic failure
11
tokens = enc.encode("strawberry")
12
print([enc.decode([t]) for t in tokens])
13
# ['str', 'awberry']
14

15
# Numbers are weird
16
tokens = enc.encode("123456789")
17
print([enc.decode([t]) for t in tokens])
18
# ['123', '456', '789']

Look at “strawberry.” It becomes two tokens: “str” and “awberry.” The model literally cannot see the individual letters, so when you ask “how many r’s are in strawberry?” and it gets it wrong, the representation it works with simply doesn’t contain the information you’re asking about. It would be like asking you to count the pixels in a word while looking at it from across the room.

Numbers are strange too. “123456789” becomes three tokens: “123”, “456”, “789.” The model doesn’t see this as one number; it sees three chunks, which is partly why LLMs struggle with precise arithmetic on large numbers.

When you’re working with LLMs and hitting character-level failures, it’s not a fixable bug but a fundamental representation limitation. The model would need to be retrained with character-level tokenization to fix it, which would make every other task worse because you’d need far more tokens to represent the same text.

Embeddings

Once text is split into tokens, each token gets turned into a vector (a list of numbers). This is the embedding.

Every token in the model’s vocabulary has a learned embedding vector. For GPT-4, these are probably 12,288-dimensional vectors (12,288 numbers per token). The model learned these during training: they started random and were adjusted billions of times until tokens with similar meanings ended up at similar points in this high-dimensional space.

The embedding dimensions don’t have human-readable meanings (it’s not like dimension 47 means “how formal is this word”). The meaning comes from the relationships between vectors, not from individual dimensions.

Positional encoding: where are you in the sequence?

There’s a subtle problem here. A transformer processes all tokens at once (that’s what makes it fast, unlike RNNs which go one at a time), but that means it has no built-in sense of word order. “The cat sat on the mat” and “mat the on sat cat the” would look the same to a position-unaware transformer.

We fix this by adding positional encoding to each token’s embedding. The original transformer used sinusoidal functions:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$ $PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$

The practical upshot is that context window limits come from positional encoding. The model was trained with positions up to some maximum $N$, and it has never seen position $N + 1$; it doesn’t know what that position means. This is why context windows have hard limits and why extending them is an active research problem. The model has no representation for “the 130,000th token” if it was trained with a 128K context window.

At this point, you understand how an LLM gets from raw text to a sequence of vectors. Text goes to tokens (via BPE), then to embeddings (via lookup table), then to embeddings plus position (via positional encoding). The result is a sequence of vectors, one per token, that encode both what the token means and where it sits in the sequence.

This is the input to the transformer; next, we’ll look at what the transformer actually does with it.