When CNNs Dominated Computer Vision
Imagine we want to teach a computer to recognize a cat in a photo.
Before Deep Learning became mainstream, Computer Vision systems relied heavily on hand-crafted features such as edges, corners, textures, or algorithms like SIFT and HOG.
CNN (Convolutional Neural Network) changed everything.
Instead of manually telling the computer which features to look for, CNNs automatically learn useful features directly from data during training.
From LeNet to AlexNet, VGG, and ResNet, CNNs gradually became the foundation for a wide range of core tasks, including Image Classification, Object Detection, and Semantic Segmentation.
The secret behind CNN's success lies in the convolution operation.
A small filter (kernel) slides across the image to extract local features. Across multiple layers, the model combines these simple features into increasingly complex representations.
For example, early layers might learn basic edges and lines, while deeper layers assemble them into parts, shapes, and high-level representations used for object recognition.
This approach works remarkably well because images have an inherent spatial structure: pixels that are close together are usually closely related.
However, this also raises an interesting question.
What happens when a model needs to understand the relationship between regions that are far apart within the same image?
CNNs can still capture long-range dependencies by stacking many processing layers. Still, is there a more direct approach?
Vision Transformer: Treating Images as a Sequence of Tokens
In 2017, the Transformer architecture revolutionized Natural Language Processing with the landmark paper Attention Is All You Need.
By 2020, a research team at Google published An Image Is Worth 16x16 Words, introducing the Vision Transformer (ViT).
The core idea is fascinating:
If a Transformer can process a sequence of words, why not turn an image into a sequence of patches?
Instead of processing images primarily through convolutional filters, ViT splits an image into fixed-size patches.
For instance, a 224 × 224 pixel image is divided into 196 patches, each measuring 16 × 16 pixels.
Each patch is flattened into an embedding vector, combined with positional embeddings, and fed directly into a Transformer Encoder.
Here, the Self-Attention mechanism allows the model to integrate information across patches—even when they are located on opposite sides of the image.
This marks a fundamental difference between the two architectures:
- CNNs exploit local spatial structure through convolutions and gradually aggregate information in deeper layers.
- ViTs use attention to model relationships between patches, enabling them to interact directly within global attention layers.
ViT opened up a powerful new paradigm for Image Recognition, especially when pre-trained on massive datasets.
Yet, it is not without its weaknesses.
Is ViT Really Better Than CNN?
The rise of the Vision Transformer has led many to ask: Are CNNs becoming obsolete?
The answer isn't that simple.
CNNs are built with clever "inductive biases" tailored specifically for images: they naturally group neighboring pixels to capture local detail and reuse the same filter across the entire image. Thanks to this design, CNNs learn remarkably fast—even with limited data or modest computing power.
In contrast, a vanilla ViT approaches an image like a blank slate and has to figure out spatial rules from scratch. It becomes incredibly powerful when fed millions of images during pre-training, but it can easily struggle when data is scarce or when high-resolution images are split into too many patches.
Then, in 2022, ConvNeXt arrived. Built purely on traditional CNN architecture—but modernized with design and training techniques inspired by Transformers—ConvNeXt proved that CNNs can still compete head-to-head with ViTs across numerous benchmarks.
As it turns out, the race isn't simply Convolution vs. Self-Attention. Model design, data scale, and training recipes are what truly tip the scales.
So, Should You Choose CNN or ViT?
Suppose we need to distinguish between delivery drivers from two companies whose uniforms look almost identical.
A naive model might focus on shirt color, whereas the actual distinguishing features are tiny details like logos, badges, or subtle patterns.
In real-world scenarios like this, the problem isn't just picking between a CNN and a ViT.
We have to care about data quality, object cropping strategies, image resolution, feature representation, and how the model is evaluated on production data.
Sometimes, a well fine-tuned CNN is more than enough. In other cases, embeddings extracted from a pre-trained Vision Transformer might deliver superior results.
The right answer should come from empirical experimentation—not from which architecture is newer.
Conclusion
Looking back at the evolution of Computer Vision, every generation of models has reshaped how we learn and represent visual information.
CNNs proved the power of exploiting spatial structure. Vision Transformers expanded our ability to model global relationships across image regions using Self-Attention. Today, Vision Foundation Models continue to push the field closer to building reusable, general-purpose visual representations.
CNNs did not die when ViT arrived. Computer Vision simply gained more powerful tools to tackle increasingly complex problems.
Ultimately, the best architecture isn't necessarily the newest one—it is the one that best fits your data, your compute budget, and the specific problem you need to solve.