ReCalCon BMVC 2026

BMVC 2026

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

Describe each image’s transformation in its text prompt, and re-calibrate the contrastive loss against positive-gradient dilution.

  1. Seungmin Oh1
  2. Seunghun Kang1
  3. Jongbin Ryu1,2†
  1. 1Department of Artificial Intelligence, Ajou University
  2. 2Department of Computer Engineering, Ajou University
  3. †Corresponding author

Top ID accuracy, best OOD mean

52.5 55.0 57.5 60.0 62.5 65.0 81 81.5 82 82.5 83 83.5 ImageNet accuracy (%) OOD mean (%) better ↗ FFT LP-FT WiseFT FLYP ARF Lipsum-FT RAda-FT Ours
Fine-tuning CLIP ViT-B/16 on ImageNet. OOD is the mean over five shifted test sets. Hover a point for its numbers.
OOD mean and H.M. on ImageNet distribution shift, tied best ID (ViT-B/16)
SOTA
points of ImageNet OOD mean over the best prior method (ViT-B/16)
+0.9
points of iWildCam H.M. (macro F1) over the best prior method (ViT-B/16)
+1.4

Try it: describe the augmentation

Describe the augmentationRe-calibrate the loss I′ I T′ v₁v₂v₃v₄rotateshearxshearytranslatexcolorposterizecontrastbrightnessdog same as T T v₁v₂v₃v₄dog Contrastive space · batch: dog, dog, cat T′ I′ T I′ dogdogcat T′ I dogdogcat T I dogdogcat dogdogcat Learning signal for image I (dog) Positives |P(i)| 2 Soft-target CE, weight per positive 1/2 Adaptive margin −log ψ(i) not ÷ |P(i)|

Step 1 of 5 · PairA clean image I and its prompt T: four learnable context tokens and the class name. In a labeled batch of two dogs and a cat, the other dog's prompt is a positive too, so soft-target cross-entropy splits the target in half.

Drag across the progress bar, tap a step or press Play, and pick any two RandAugment operations (the paper uses k = 2). The photo is the one in the paper's Figure 2; image operations are browser approximations and the descriptor words are illustrative.

Describe the augmentation, then re-calibrate

Training scheme: clean and transformed images with clean and transformation-conditioned prompts form a 2 by 2 augmented contrastive space, trained with the re-calibrated loss.
Figure 1.Each clean image and prompt also produces a transformed image and a transformation-conditioned prompt. The four matches form an augmented contrastive space, trained with the re-calibrated loss. Paper, Figure 2.

Transformation-aware prompt conditioning

The text prompt is M = [V, A, CLS]: learnable context V, a fixed transformation descriptor A that describes the RandAugment operations applied to the paired image, and the class token. A comes from the RandAugment dictionary and is never updated; within the prompt only V is learned. The descriptor gives the text transformation-level consistency without altering class semantics.

Augmented contrastive space

Each clean image and prompt also produces a transformed image and a transformation-conditioned prompt. The loss uses all four matches, [I I′] ↔ [T T′], and treats same-class samples as positives rather than as distinct instances.

Re-calibrated contrastive loss

With many positives per anchor, soft-target cross-entropy gives each one a target weight of 1/|P(i)|, so its gradient shrinks as positives are added: positive-gradient dilution. The re-calibrated loss adds one adaptive margin, −log ψ(i), where the re-calibration factor ψ(i) is the ratio of positive to negative evidence for that anchor. The margin is not divided by |P(i)|, so it keeps full strength as positives grow.

Better OOD transfer

ReCalCon ties the best ID accuracy with ViT-B/16 and leads it with ViT-B/32, gives the best OOD mean, harmonic mean and iWildCam score with both backbones, and the best transfer mean with ViT-B/16.

  • LP-FTICLR’22
  • WiseFTCVPR’22
  • FLYPCVPR’23
  • ARFCVPR’24
  • Lipsum-FTICLR’24
  • RAda-FTICCV’25
  • Ours
Gain (pp) 0.0 +2.5 +5.0 +7.5 +10.0 +2.0 ImageNet ID +8.5 ImageNet OOD +6.5 ImageNet H.M. +2.9 iWildCam H.M. +3.3 Transfer (6 sets)
Figure 2.Gain over full fine-tuning (FFT) with CLIP ViT-B/16, in accuracy (macro F1 for iWildCam). ImageNet OOD is the mean over ImageNet-R, -A, -V2, -Sketch and ObjectNet; H.M. is the harmonic mean of ID and OOD; transfer is the mean over six downstream datasets. Missing bars are not reported. Paper, Tables 1 and 2.
ImageNetiWildCam
ID-R-A-V2-SketchObjectNetOOD meanH.M.IDOODH.M.
ViT-B/16, FFT ±0.0 −4.3 +3.2 +0.5 ±0.0 +1.6 +0.9 +0.7 +1.0 +0.6 +1.4
ViT-B/32, FFT +1.8 −4.0 +4.4 +1.3 −0.9 +4.0 +1.0 +1.3 +1.3 +2.3 +2.0
behind the best prior method ahead of the best prior method tied 17 of 22 cells ahead, 2 tied
Figure 3.Our score minus the best prior method with the same backbone, both fully fine-tuned: accuracy on every ImageNet test set, macro F1 on iWildCam. Prior methods stay ahead only on ImageNet-R (both backbones) and on ImageNet-Sketch with ViT-B/32; ±0.0 marks a tie. Hover a cell for both values. Paper, Table 1.

Keeps improving with longer training

From 10 to 25 epochs, FFT, WiseFT and FLYP lose ID accuracy and harmonic mean. ReCalCon gains on both.

ID accuracy (%)

  • FFT
  • WiseFT
  • FLYP
  • Ours
80 81 82 83 84 10152025 Epoch Ours

H.M. of ID and OOD (%)

  • FFT
  • WiseFT
  • FLYP
  • Ours
60.0 62.5 65.0 67.5 70.0 72.5 10152025 Epoch Ours
Figure 4.Accuracy under extended training on ImageNet with ViT-B/16. Hover an epoch for every method's value, read from the paper's vector figure (Figure 4c-d).

The descriptor has to match

Describing the wrong transformation is worse than no descriptor at all; only the matching descriptor beats having none.

  • No descriptor
  • Mismatched descriptor
  • Matched descriptor (ours)
H.M. gain (pp) 0 +2 +4 +6 +8 Random crop +7.3 RandAugment
Figure 5.Harmonic mean of ID and OOD accuracy on ImageNet with ViT-B/32, as a gain over zero-shot CLIP (55.1%). No descriptor: prompts without a transformation descriptor. Mismatched: a descriptor for a different transformation. Matched: the descriptor of the applied transformation. Paper, Table 3.

Citation

@inproceedings{oh2026recalibrated,
  title     = {Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models},
  author    = {Oh, Seungmin and Kang, Seunghun and Ryu, Jongbin},
  booktitle = {Proceedings of the British Machine Vision Conference},
  year      = {2026},
  publisher = {BMVA},
  url       = {https://arxiv.org/abs/2609.06967}
}