Fixed Labels Limit Vision

The Closed Label List

Traditional vision systems were trained to predict a fixed set of predetermined object categories.

This worked well. A model trained on ImageNet's 1,000 classes recognizes those 1,000 classes reliably.

But the label space is closed. Anything outside the list simply does not exist for the model.

Notice that images falling outside the fixed list have nowhere to land, which is exactly what a closed label space means.

That closure is the bottleneck CLIP set out to remove.

When A Phrase Has No Slot

Imagine asking an ImageNet classifier for a diagram of contrastive learning.

It cannot answer. The phrase is not in its taxonomy, so the model has no way to represent the request.

The classifier is forced to choose from a known label list, whatever you actually asked for.

Specifying a new visual concept requires new labeled data or a new task-specific classifier.

Bigger Lists Are Still Lists

A common reaction is to say: just label more classes.

Larger taxonomies do help. They do not remove the problem.

Someone still has to define and label each desired category in advance.

Going from 1,000 classes to 10,000 makes the fixed list longer, never open.

Labels Are Not The Enemy

This is not an argument that labels are useless.

Carefully labeled data remains valuable, especially in high-stakes or narrow domains where categories are known and precision matters.

CLIP's claim is narrower and sharper.

Fixed labels are an incomplete route to general visual intelligence, not a failed one.

The Transfer Question

A representation is more useful when it transfers to many downstream tasks.

Transfer asks whether learned features support new tasks, datasets, or concepts with little or no task-specific retraining.

Succeeding only on the dataset you were trained to predict is a narrower achievement.

The CLIP paper evaluates transfer across more than 30 datasets rather than reporting a single benchmark.

What Product Teams Actually Need

Think about a team that wants to search images by phrases.

Or classify unusual objects. Or compare images directly to captions.

None of these fit a fixed label list cleanly. Each one needs a representation that connects open-ended text with visual evidence.

Language, not a taxonomy, becomes the interface.

Why This Sets Up CLIP

Pull the thread together.

Fixed categories restrict generality and usability, because every extra concept demands more labeled data.

Larger label sets extend the list without opening it.

Transfer across many datasets, not one score, is the real test of a visual representation. That is the gap language supervision was built to close.

A classifier trained on 1,000 ImageNet classes can directly recognize any phrase describing a visual concept.

• True
• False

Answer: False

False. The classifier can choose only among its 1,000 predefined categories. A phrase such as “a diagram of contrastive learning” may have no label slot.

A product team wants users to search a photo collection with arbitrary phrases, including “a diagram of contrastive learning.” Why is a traditional fixed-label classifier a poor fit?

• It can predict only categories defined before deployment.
• It cannot recognize any image unless the image has no text.
• It always performs worse than an unsupervised model on known categories.
• It needs a separate camera for every category in its taxonomy.

Answer: It can predict only categories defined before deployment.

A fixed-label classifier can select only categories defined in its taxonomy. Arbitrary phrases may not have corresponding labels, so supporting them would require new labeled data or a task-specific classifier. A representation that connects language and images is better suited to this use.

Increasing a fixed taxonomy from 1,000 classes to 10,000 classes removes the fixed-taxonomy bottleneck.

• True
• False

Answer: False

False. A larger taxonomy provides more predefined choices, but it remains closed. Each additional category still must be imagined, defined, and labeled in advance.