What Counts as a Word? · Computational Linguistics Lab

Step 1

Find the boundaries

A space is evidence for a boundary, but it is not a universal definition of a word.

Example text Ignore capitalizationTreat “Word” and “word” alike.

Text to tokenize

Before running the comparison

How many tokens should this text contain? Identify one boundary that a computer could reasonably handle in more than one way.

Step 2

Compare three rules

The same visible text becomes three different datasets.

Keep contractions togetherFor example, treat “can’t” as one token. Keep hyphenated forms togetherFor example, treat “re-enter” as one token.

Which output is correct?

No output is universally correct. A useful tokenizer preserves distinctions needed for a particular analysis while remaining consistent.

Notice: Changing tokenization also changes frequency counts, vocabulary size, and every model trained from those observations.

Step 3

Learn BPE tokens, one merge at a time

Begin with characters, rank adjacent symbol pairs, and preview how each merge changes the corpus and vocabulary.

BPE training text This text is shared with Steps 1–2. Repeated word forms contribute repeated pair counts.

Pairs to rank Choose any value from 1 to 20.

What is being counted?

Round 0 counts adjacent characters. Later rounds count adjacent symbols, so a pair may include a previously learned multi-character token. ▁ marks a word beginning. For visibility, this teaching version begins with Unicode grapheme characters rather than raw bytes.

Completed merge rounds0

0written words

0merges completed

0subword tokens

0learned vocabulary

Current segmentation

Character starting point

Learned vocabulary

0 symbols

Most frequent adjacent pairs

Current round

Rank	Pair	Count

Next-round preview

Merge the highest-ranked pair

Segmentation after the merge

Preview only

Top pairs in the new round

Recounted after merging

Rank	Pair	Count

Vocabulary after the next merge

0 symbols

Merge history

Open a round to inspect its vocabulary

Did the algorithm discover morphemes?

Sometimes a frequent character sequence aligns with a morpheme. That does not mean the procedure understands its grammatical function or meaning.

Test it: Choose “Related word forms.” Advance several rounds and compare how teach, teacher, and teaching are segmented.

Step 4

Stress-test the assumptions

A rule that looks natural in English may encode the wrong boundaries elsewhere.

Select a stress test, then inspect the three tokenizers above.

Exit question

Is tokenization merely preprocessing—or is it already a theory about linguistic structure?

Use one disagreement between the tokenizers as evidence.