ICCV 2025 · MULTIMODAL LONG-CONTEXT LEARNING

More context.Less compromise.

A new perspective on visual position encoding. V2PE helps vision-language models see further, connecting images and language across sequences of up to one million tokens.

Read the paper Explore the code

Explore the researchSCROLL TO DISCOVER

THE BIGGER PICTURE01 / ∞

V2PEA longer view.

VISIONLANGUAGE

Standard positions

Variable visual positions

THE RESEARCHImproving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

Junqi Ge^1,4*Ziyi Chen^1,4*Jintao Lin^3,4*Jinguo Zhu^4*Xihui Liu³Jifeng Dai^1,4Xizhou Zhu^1,2,4†

1 Tsinghua University2 SenseTime Research3 University of Hong Kong4 Shanghai AI Laboratory

* Equal contribution † Corresponding author

1MTokens in long-context inference

256KTokens per training sample

2BParameters in the base model

01 / THE IDEA

A small shift in position. A much wider perspective.

Vision-language models understand increasingly complex combinations of images and text. Yet their performance degrades when a multimodal sequence extends beyond the context window encountered during training.

V2PE uses smaller, variable positional increments for visual tokens. Integrated with InternVL2-2B and augmented long-context training data, this design enables the model to process multimodal sequences up to 1M tokens, while training on sequences up to 256K.

02 / HOW IT WORKS

Make room for what matters.

Text and images share a sequence. They do not have to share the same positional increment.

Diagram comparing standard position encoding with Variable Visual Position Encoding — 01V2PE uses smaller and variable positional increments for visual tokens, while preserving textual position increments.

THE EXPERIMENTAL SETUP
Component	Configuration	Purpose
Base model	InternVL2-2B	Multimodal understanding
Training sequence	Up to 256K tokens	Long-context adaptation
Inference sequence	Up to 1M tokens	Extended multimodal context
Visual positions	Smaller, variable increments	Efficient use of positional space

Configuration summary from the project. See the paper for complete training and evaluation details.

03 / THE EVIDENCE

Keep the bigger picture in view.

From million-token retrieval to positional increment ablations, explore how V2PE behaves as context grows.

Values rounded to one decimal. GPT-4o has no reported point at 1M.

View data table

Image retrieval at million-token scale — data
Context length	InternVL2-2B + V2PE (Ours)	InternVL2-2B without V2PE	GPT-4o
32K	96.6	96.6	100
64K	97.1	93.1	92.7
128K	99.1	56.1	62.1
256K	95.7	3.7	60
512K	95.1	10.8	58.6
1M	64.5	0	—

Approximate digitized values; overlapping points may introduce uncertainty. Values rounded to one decimal.

View data table

Results — data
Panel	Category	Series	Value
MM-NIAH · positional increments	1	δ = 1/256	96.7
MM-NIAH · positional increments	1	δ = 1/16	94.9
MM-NIAH · positional increments	1	δ = 1	80
MM-NIAH · positional increments	1	linear	85.2
MM-NIAH · positional increments	1	NTK	78.9
MM-NIAH · positional increments	2	δ = 1/256	93.9
MM-NIAH · positional increments	2	δ = 1/16	95.8
MM-NIAH · positional increments	2	δ = 1	81
MM-NIAH · positional increments	2	linear	85.1
MM-NIAH · positional increments	2	NTK	80.6
MM-NIAH · positional increments	4	δ = 1/256	95.5
MM-NIAH · positional increments	4	δ = 1/16	92.8
MM-NIAH · positional increments	4	δ = 1	78.5
MM-NIAH · positional increments	4	linear	82.8
MM-NIAH · positional increments	4	NTK	80.2
MM-NIAH · positional increments	8	δ = 1/256	93.3
MM-NIAH · positional increments	8	δ = 1/16	92.6
MM-NIAH · positional increments	8	δ = 1	78.4
MM-NIAH · positional increments	8	linear	78.7
MM-NIAH · positional increments	8	NTK	79.2
MM-NIAH · positional increments	12	δ = 1/256	97.2
MM-NIAH · positional increments	12	δ = 1/16	94.9
MM-NIAH · positional increments	12	δ = 1	75.2
MM-NIAH · positional increments	12	linear	77.7
MM-NIAH · positional increments	12	NTK	73.6
MM-NIAH · positional increments	16	δ = 1/256	94.2
MM-NIAH · positional increments	16	δ = 1/16	91.8
MM-NIAH · positional increments	16	δ = 1	68.7
MM-NIAH · positional increments	16	linear	76.1
MM-NIAH · positional increments	16	NTK	70.8
MM-NIAH · positional increments	24	δ = 1/256	93.5
MM-NIAH · positional increments	24	δ = 1/16	90.9
MM-NIAH · positional increments	24	δ = 1	52.8
MM-NIAH · positional increments	24	linear	64.1
MM-NIAH · positional increments	24	NTK	56
MM-NIAH · positional increments	32	δ = 1/256	86.6
MM-NIAH · positional increments	32	δ = 1/16	87.8
MM-NIAH · positional increments	32	δ = 1	38.8
MM-NIAH · positional increments	32	linear	56.7
MM-NIAH · positional increments	32	NTK	40.8
MM-NIAH · positional increments	40	δ = 1/256	80.5
MM-NIAH · positional increments	40	δ = 1/16	82.8
MM-NIAH · positional increments	40	δ = 1	30.8
MM-NIAH · positional increments	40	linear	46.8
MM-NIAH · positional increments	40	NTK	38.4
MM-NIAH · positional increments	48	δ = 1/256	68.2
MM-NIAH · positional increments	48	δ = 1/16	66.6
MM-NIAH · positional increments	48	δ = 1	27.1
MM-NIAH · positional increments	48	linear	38.5
MM-NIAH · positional increments	48	NTK	26.2
MM-NIAH · positional increments	64	δ = 1/256	53
MM-NIAH · positional increments	64	δ = 1/16	57.9
MM-NIAH · positional increments	64	δ = 1	27.6
MM-NIAH · positional increments	64	linear	34.8
MM-NIAH · positional increments	64	NTK	25.8
Long-VQA · positional increments	16	δ = 1/256	58.2
Long-VQA · positional increments	16	δ = 1/16	62.9
Long-VQA · positional increments	16	δ = 1	62.3
Long-VQA · positional increments	16	linear	62.5
Long-VQA · positional increments	16	NTK	63.2
Long-VQA · positional increments	32	δ = 1/256	52.8
Long-VQA · positional increments	32	δ = 1/16	55.6
Long-VQA · positional increments	32	δ = 1	49
Long-VQA · positional increments	32	linear	51.3
Long-VQA · positional increments	32	NTK	54.2
Long-VQA · positional increments	40	δ = 1/256	48.1
Long-VQA · positional increments	40	δ = 1/16	51.2
Long-VQA · positional increments	40	δ = 1	30
Long-VQA · positional increments	40	linear	42.1
Long-VQA · positional increments	40	NTK	44.9
Long-VQA · positional increments	48	δ = 1/256	45.4
Long-VQA · positional increments	48	δ = 1/16	48.3
Long-VQA · positional increments	48	δ = 1	26.5
Long-VQA · positional increments	48	linear	34.7
Long-VQA · positional increments	48	NTK	41
Long-VQA · positional increments	56	δ = 1/256	43.8
Long-VQA · positional increments	56	δ = 1/16	47
Long-VQA · positional increments	56	δ = 1	27
Long-VQA · positional increments	56	linear	31.4
Long-VQA · positional increments	56	NTK	40.4
Long-VQA · positional increments	64	δ = 1/256	40.3
Long-VQA · positional increments	64	δ = 1/16	42.5
Long-VQA · positional increments	64	δ = 1	25.9
Long-VQA · positional increments	64	linear	25.9
Long-VQA · positional increments	64	NTK	34.3

The positional increment experiments compare different visual increments alongside linear interpolation and NTK-aware scaled RoPE. Explore individual data points, toggle series, or open the data table below each chart.

04 / BENCHMARKS

Beyond a single task.

Results on general and long-context multimodal benchmarks.

Table 4 · General multimodal benchmarks
Model	#Param	ChartQA	DocVQA	AI2D	InfoVQA	SQA	POPE	MMMU (val)	MMBench (EN)	SEED (I)	Avg
InternVL2-2B	2.0B	71.7	86.9	74.1	58.9	94.1	85.2	36.3	73.4	70.9	72.4
DeepSeek-VL-1.3B	2.0B	47.4	—	51.5	—	68.4	85.9	33.8	66.4	66.0	—
Qwen2-VL-2B	2.0B	73.5	90.1	74.7	65.5	—	—	41.1	74.9	—	—
Aquila-VL-2B	2.2B	32.0	85.0	75.1	58.3	95.1	83.1	46.9	79.0	73.9	69.8
MiniCPM-V-2	2.8B	55.6	71.9	62.9	—	80.7	86.3	38.2	64.1	67.1	—
Vintern-3B-beta	3.7B	68.3	—	69.1	—	75.0	87.4	46.7	70.6	70.0	—
Llama 3.2 11B	11B	83.4	88.4	91.1	—	—	—	50.7	68.0	—	—
Qwen2-VL-72B	73B	88.3	96.5	88.1	84.5	91.2	87.2	64.5	86.9	77.9	85.0
GPT-4o	—	85.7	92.8	84.7	—	90.1	97.2	69.1	82.1	76.7	—
InternVL2-V2PE-32K	2.0B	77.3	84.9	73.7	56.5	94.9	88.7	36.1	73.8	71.2	73.0

#Param denotes model parameters. — indicates an unreported score.

Table 5 · Long-context multimodal benchmarks
Model	#Param	MM-NIAH			Milebench				VideoMME
Model	#Param	Image	Text	Avg	T	S	NI	Avg	VideoMME
InternVL2-2B	2.0B	23.0	18.9	21.0	58.2	54.5	37.0	49.9	—
Phi-3-Vision	2.7B	—	—	—	46.9	50.0	—	—	—
OmChat	3.9B	—	—	—	51.4	52.0	—	—	45.9
LongLLaVA	9B	—	—	—	47.3	46.8	—	—	43.7
LongLLaVA	13B	—	—	—	52.7	52.1	—	—	51.6
VILA	13B	14.5	40.5	27.5	—	—	—	—	—
Gemini-1.5	—	28.5	82.1	55.2	50.2	58.3	97.9	68.8	69.6
GPT-4V	—	—	84.1	—	45.6	58.9	99.4	68.0	59.9
GPT-4o	—	—	—	—	56.2	63.5	—	—	64.7
Claude3-Opus	—	—	—	—	37.4	48.1	85.3	56.9	59.7
InternVL2-V2PE-32K	2.0B	72.3	68.1	70.2	58.5	58.2	51.1	55.9	48.2

#Param denotes model parameters. — indicates an unreported score.

05 / THE DATA

A richer world to learn from.

Augmented long-context multimodal data connects document understanding, charts, and retrieval across longer sequences.

An overview of the components in our custom-made training dataset.

Long-VQA document question answering example — 01Long-VQA · Document question answering.

Long-VQA chart question answering example — 02Long-VQA · Chart question answering.

Long-MR image needle retrieval with distractors — 03Long-MR · Image retrieval with distracting needles.

Long-MR challenging image needle retrieval example — 04Long-MR · Challenging image retrieval.

Explore the details

06 / OPEN RESEARCH

Take a closer look.

Everything you need to explore the work and build on it.

01 / PAPER

If this work helps yours.

BIBTEX

@misc{ge2024v2peimprovingmultimodallongcontext,
  title={V2PE: Improving Multimodal Long-Context Capability of
         Vision-Language Models with Variable Visual Position Encoding},
  author={Junqi Ge and Ziyi Chen and Jintao Lin and Jinguo Zhu
          and Xihui Liu and Jifeng Dai and Xizhou Zhu},
  year={2024},
  eprint={2412.09616},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2412.09616}
}

A LONGER VIEW STARTS HERE

See further. Understand more.

Read the paper View the code

More context.Less compromise.

A small shift in position. A much wider perspective.

Make room for what matters.

Keep the bigger picture in view.

Beyond a single task.

A richer world to learn from.

An overview of the components in our custom-made training dataset.

Take a closer look.

Read the full story

Build on V2PE

Explore the data

Meet the model

If this work helps yours.

See further. Understand more.