GLINT-CT

📝 Paper • 💻 Code • 🌐 Project page • 🤗 GLINT-CXR

GLINT-CT is the chest CT model of GLINT (GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations, NeurIPS 2026).

• Vision encoder: V-JEPA 2.1 ViT-L/16, 384×384, 80 frames / 240 slices
• Text encoder: sentence-transformers/all-mpnet-base-v2 (fine-tuned)
• Alignment: linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head
• Training: CT-RATE, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen V-JEPA 2.1 ViT-G teacher

⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition.

Usage

Both examples use the bundled inference.py / utils.py; run them from the downloaded repo.

Full pipeline

hf download Deepnoid/GLINT-CT --local-dir glint-ct && cd glint-ct
python inference.py --volume volume.nii.gz --text "There is a lung nodule."
# prints the sentence probability and saves similarity_map.nii.gz

--volume takes a NIfTI file or a DICOM series directory. The map has the input CT's shape and affine, so it overlays the input in ITK-SNAP / 3D Slicer.

AutoModel

import torch
from transformers import AutoModel, AutoTokenizer
from utils import load_ct_volume, model_inference, save_similarity_map, similarity_scores_to_volume   # bundled helpers

repo = "Deepnoid/GLINT-CT"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)

pixel_values, geometry = load_ct_volume("volume.nii.gz", return_geometry=True)   # [1, 80, 3, 384, 384]
prob, scores = model_inference(pixel_values, "There is a lung nodule.", model=model, tokenizer=tokenizer)
similarity_map = similarity_scores_to_volume(scores, model, geometry)           # input CT shape, e.g. (512, 512, 205)
save_similarity_map(similarity_map, geometry["affine"], "similarity_map.nii.gz")

• Outputs: prob is the sentence probability (sigmoid); scores is the per-token similarity in [0, 1] (40 × 24 × 24 tokens); similarity_scores_to_volume resamples it onto the input CT grid (0 outside the 360 mm preprocessing crop), or keeps the 240×384×384 preprocessed grid without geometry.
• geometry: returned by load_ct_volume(..., return_geometry=True): shape and affine of the input CT (the map's NIfTI header) and to_model, the 4×4 input-voxel → preprocessed-voxel mapping used for the resampling; for array input affine is None and the map stays on the preprocessed grid.
• Inputs: a NIfTI file or a DICOM series directory (one series); utils.load_ct_volume applies the training preprocessing (IPL, HU [-1000, 1000] → [-1, 1], 1.5 × 0.75 × 0.75 mm, crop/pad to 240×480×480, resize to 240×384×384) and returns [1, 80, 3, 384, 384].
• Unreliable affine: pass the voxel spacing along the reoriented I, P, L axes, e.g. raw_spacing=(1.5, 0.7, 0.7), to load_ct_volume (--raw_spacing in the script).
• Underlying call: model.compute_logits(pixel_values, [encoded_text]) returns logits and similarity_scores.

Vision encoder only (vision_only=True):

import torch
from transformers import AutoModel
from utils import load_ct_volume   # bundled helper

repo = "Deepnoid/GLINT-CT"
vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval()

pixel_values = load_ct_volume("volume.nii.gz")   # [1, 80, 3, 384, 384]
with torch.no_grad():
    out = vision(pixel_values)
print(out.last_hidden_state.shape)               # (1, 23040, 1024)

Requirements

transformers==4.56.1, torch, entmax, monai, nibabel; pydicom for DICOM series input.

⚠️ Keep transformers==4.56.1, the version the bundled code was validated with.

Notes

• Training and fine-tuning code: GitHub; split ID lists: data/.
• dfr.py and its projector weights are used only in training; the V-JEPA 2.1 ViT-G teacher is not included.
• vision_only=True still downloads the full checkpoint; save_pretrained on the loaded model keeps a vision-only copy.

Acknowledgments

This work was supported by the Technology Innovation Program (RS-2025-02221011, Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE, South Korea), and by the Advanced GPU Utilization Support Program funded by the Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330).

License

• Weights and code: CC BY-NC 4.0.
• Vision encoder: fine-tuned from V-JEPA 2.1 ViT-L (© Meta Platforms, Inc., MIT); vjepa21_encoder.py adapts facebookresearch/vjepa2 (MIT, LICENSES/MIT.txt).
• Text encoder: fine-tuned from sentence-transformers/all-mpnet-base-v2 (Apache-2.0, modified; LICENSES/Apache-2.0.txt).

Citation

@inproceedings{park2026glint,
  title     = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
  author    = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
               Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
               Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.03180}
}