Hello everyone, welcome to the sixteenth issue of The Main Thread. In our last issue, we explored how Word2Vec learns meaning by observing which words keep company with each other. Today, we descend from the conceptual to the mechanical; to the actual forces that arrange words in semantic space, and the practical choices that determine whether this arrangement serves our needs.

Let’s begin with the problem that nearly prevented this approach from working at all. The naÏve implementation requires something computationally absurd: for every word pair like (“fox“, “quick“), the model must evaluate how “fox“ relates to the every other word in the vocabulary - all 50,000 of them, just to learn this association.

Imagine if, to learn that foxes are quick, you had to consciously consider whether the foxes are also “asteroids“, “zygotes“, “quasars“, “milk“, and every other word you know. This is extremely inefficient.

This is where negative sampling transforms an impractical idea into a workable algorithm through a philosophical shift: we don’t need to know how “fox“ relates to every word; we just need to know its real companions from impostors.

Negative sampling reframes the problem as a series of simple distinctions. For each genuine pair (“fox“, “quick“), we:

Geometrically, this creates a push-pull dynamic in the semantic space. "Fox" and "quick" experience an attractive force, pulling their vectors closer. Meanwhile, "fox" pushes away from "asteroid," "calculus," and other randomly sampled words.

Over millions of such adjustments, words gradually settle into stable arrangements. Semantically related words develop smaller angle between their vectors. Unrelated words stand at oblique or opposing angles. The entire vocabulary self-organizes into a geometry that reflects usage patterns.

This is where Word2Vec offers us a meaningful choice between two learning philosophies, each with distinct characteristics:

Skip-Gram approaches learning with focused intensity. When it encounters the rare word “quokka“, it treats this word as the center of attention and asks, “What words surround you?“ The gradient updates flow directly to “quokka“‘s embeddings, giving even rare words strong, distinctive representations.

CBOW takes a more contextual approach. It gathers all the words around a position, averages their meanings and asks, “What word belongs here?“ This smoothing effect makes CBOW particularly adept with common words that appear in diverse contexts, as it integrates signals from many directions.

The difference is subtle but profound. Skip-Gram excels at capturing what makes each word unique. CBOW excels at capturing what makes words interchangeable in context.

These aren't just implementation details, they are decisions that shape what your embeddings can and cannot do.

Choose Skip-Gram when:
You're working with specialized language where precise distinctions matter. In medical documentation, the difference between "benign" and "malignant" needs to be sharply defined. In legal contracts, "shall" and "may" carry different obligations. Skip-Gram's focused learning preserves these crucial distinctions.

Choose CBOW when:
You're analyzing broad patterns where individual word identity matters less than categorical membership. For sentiment analysis, knowing that "excellent," "outstanding," and "superb" cluster together is more important than preserving their subtle differences. CBOW's averaging naturally creates these categorical clusters.

This brings us to the fundamental compromise at the heart of practical machine learning: sharpness vs. stability.

Skip-Gram produces sharp, distinctive embeddings that are perfect for telling similar things apart. But this very sharpness makes it sensitive to data sparsity. Rare words get strong signals when they appear, but they appear infrequently.

CBOW produces stable, robust embeddings that are excellent for general categorization. But this stability comes from averaging, which can wash out subtle distinctions. The very process that makes it good with common words makes it less precise with rare ones.

There's no universal best choice, only the choice that best aligns with your particular balance of needs.

What’s remains remarkable is how these simple mechanics; push here, pull there, choose your learning style. The culminate in semantic space where analogies become vector arithmetic:

This is not programmed into the system. It emerges because the relationships "king to man" and "queen to woman" represent similar contextual adjustments.

Yet for all its elegance, this approach remains fundamentally limited to learning from words about words. It captures the map but not the territory. The word "warmth" might be close to "sunlight" and "blanket," but no embedding captures the actual sensation.

Next in our series, we will explore how GloVe builds on these insights while incorporating global statistics, and why even these improved methods still leave us craving connections to the world beyond text.

Until then, I leave you with this thought: Every embedding is a reflection of the corpus it was trained on; a fossil record of word relationships in a particular slice of language. What does your embedding collection say about the texts that formed it?

— Anirudh

P.S. Detailed treatment of these two techniques with robust mathematical derivations that build intuition are on my blog post - Understanding Embeddings: Part 2 - Learning Meaning from Context.