ViT-Up：用于视觉变换器的忠实特征上采样方法
ViT-Up: Faithful Feature Upsampling for Vision Transformers
摘要
视觉变换器（ViTs）已成为视觉表示学习领域的主流架构，它能够提供非常强大且可广泛使用的特征。然而，由于全局自注意力的二次计算成本，ViTs通常需要在相对较小的块-令牌网格上运行，这给语义分割和深度估计等密集预测任务带来了持续的瓶颈问题。因此，人们开始开发与任务无关的特征放大方法。虽然最新的先进方法能够生成视觉上清晰且密集的特征表示，但它们依赖浅层图像编码器进行放大操作，这可能导致特征泄漏、破碎以及模糊现象。我们提出了ViT-Up这一隐式特征放大框架，它用ViT中间隐藏状态中的层级查询来替代外部图像指导，从而能够在任意连续的图像坐标上进行特征预测，同时保持与主干特征空间的一致性。实验表明，在密集预测和语义匹配任务中，ViT-Up始终优于各种基于图像引导的放大方法。在DINOv3-S+数据集上，ViT-Up在Cityscapes数据集上提升了2.07 mIoU，在SPair-71k数据集上提升了4.17 PCK@0.10。对于更大的DINOv3-B主干网络，这些提升效果分别达到3.36 mIoU和8.09 PCK@0.10，这表明ViT-Up能够随着主干网络的容量而有效扩展。
English Abstract
Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 PCK@0.10 on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 PCK@0.10, demonstrating that ViT-Up scales favorably with backbone capacity.