Overview

Granite Vision uses a staged curriculum rather than one undifferentiated update. Stage 1 trains only the projector on 558,000 LLaVA-Pretrain image-text pairs while the vision encoder and language model stay frozen; Stage 2 tunes projector and language model with vision frozen, retains the projector, and restores the language model; Stage 3 tunes projector and language model on approximately 20 million mixed examples. The sequence establishes alignment before document instruction tuning, but the paper does not prove it is universally optimal.

Learning outcomes

Stage 1 aligns visual features through a projector while the visual encoder and language model remain frozen.
Stage 2 retains the adapted projector but restores the language model's original weights before Stage 3.
Stage 3 connects the prepared interface to document-rich instruction data; the paper does not prove the sequence or reset is universally optimal.

Keywords

Granite VisionAlignmentProjectorCurriculumInstructionReset

Alignment Before Instruction: The Training Curriculum

Granite Vision's data and visual interface do not become a working multimodal model simply by being placed next to one another. The training procedure determines which component learns first, what remains fixed while another component adapts, and what is deliberately discarded at a transition. This chapter follows that lifecycle so a reader can understand why Granite Vision moves from projector alignment to joint adaptation and then to document-rich instruction tuning, rather than treating training as one undifferentiated update.

stateDiagram-v2 [*] --> ProjectorAlignment ProjectorAlignment: Stage 1: projector-only alignment ProjectorAlignment --> JointAdaptation: aligned visual-language interface JointAdaptation: Stage 2: tune language model and projector; vision frozen JointAdaptation --> ResetLanguageWeights: retain projector weights ResetLanguageWeights: discard tuned language model and restore original weights ResetLanguageWeights --> InstructionTuning: continue with retained projector InstructionTuning: Stage 3: tune projector and language model; vision frozen InstructionTuning --> MultimodalModel: final trained state MultimodalModel: final projector plus language model; vision remains frozen

Stage one establishes a common visual-language interface

Before the stages make sense, it helps to identify the thing they are trying to align. Granite Vision receives an image and a text instruction, then generates a text response. The visual encoder produces image features, and the projector turns those features into visual tokens with the same dimensionality as the language model's word embeddings. The language instruction becomes language tokens. Training adjusts selected parameters so that the resulting sequence supports the desired answer.

In practical terms, the first problem is not yet whether the model knows how to answer a difficult chart question. It is whether the language model can use the visual tokens at all. A language model starts with patterns learned from text. Visual features arrive in a different representational form, even after the visual encoder has organized them into a sequence. The projector is the adjustable interface between those spaces. If that interface is poorly aligned, later instruction examples have to teach both basic communication and task behavior at the same time.

Granite Vision addresses this first with projector pre-training. In Stage 1, the projector is the only trainable part of the network; the visual encoder and language model remain frozen. The paper reports using 558,000 image-text pairs from LLaVA-Pretrain, along with selected image scales and a multi-turn conversation format. The stage is therefore a constrained alignment exercise. The projector must learn to present visual features in a form that the already-trained language model can relate to captions and instructions, while the other components provide stable endpoints.

Freezing the endpoints makes the transition easier to interpret. The vision encoder is not changing its feature space while the projector is trying to map it. The language model is not changing its word-embedding and generation behavior while the visual tokens are being introduced. A successful Stage 1 does not make the complete system a document expert. It creates a starting interface in which visual information can participate in the language model's sequence.

That distinction is important for a compact system.

Published by DeepMox

FAQ

What happens in Stage 1 of Granite Vision training?

Stage 1 trains only Granite Vision's projector on 558,000 LLaVA-Pretrain image-text pairs while the visual encoder and language model remain frozen. Its purpose is to establish a usable visual-language interface, not complete document expertise.

Why does Granite Vision restore the language model before instruction tuning?

Granite Vision restores the language model's original weights because the authors report that this gave Stage 3 a better starting point while retaining the adapted projector. The paper provides no full ablation or effect size, so this is an empirical choice rather than a universal rule.

Which Granite Vision components are updated during instruction tuning?

During Stage 3, Granite Vision updates the projector and language model on approximately 20 million mixed image-text examples while keeping the visual encoder frozen. The mixture includes document and general-image behaviors.

Previous9 minGranite Vision: Compactness Through Coordinated Design

Next12 minGranite Vision: What Its Benchmarks Actually Measure

Granite Vision: From Alignment to Instruction Tuning

Alignment Before Instruction: The Training Curriculum

Stage one establishes a common visual-language interface

FAQ