lean27b

Qwen/Qwen3.6-27B LoRA-finetuned on Lean 4 / Mathlib4 source, merged and released as a single Q8_0 GGUF.

Training curves

Weight-decay sweep at rank 32, all layers. The released model is wd = 0.4 (darkest line). Final full-validation loss: 0.652. The tool-call sanity check passed 3/3 in every epoch.

Data

  • Every .lean file under Mathlib/ in mathlib4 at tag v4.33.0.
  • 80/20 train/val split by file (seed 42).
  • Files shorter than 1024 tokens are kept whole. Longer files are split only at blank lines (\n\n) and packed greedily into chunks of at most 1024 tokens.
  • Plain causal-LM loss on all tokens. No chat template, no instruction data.

Training

Parameter Value
Method LoRA (peft), bf16 base, gradient checkpointing
Rank / alpha / dropout 32 / 32 / 0.05
Target modules q_proj k_proj v_proj o_proj in_proj_qkv out_proj gate_proj up_proj down_proj
Layers all 64
Optimizer AdamW, lr 1e-5 (constant, no warmup), weight decay 0.4
Grad clip 1.0
Batch size 8 (no accumulation)
Max sequence length 1024
Epochs 5
Seed 42 (data order reshuffled with seed + epoch)
Hardware 1ร— 80 GB GPU
Libraries torch 2.13.0, transformers 5.15.1, peft 0.20.0, accelerate 1.14.0

After each epoch, a check confirms the model still emits valid tool calls (grep / web_search / ls). Training stops early if 2 of the 3 calls fail in two consecutive epochs. This never triggered.

Export. The LoRA was merged into the bf16 base, and the vision tower was dropped. The result was converted with llama.cpp convert_hf_to_gguf.py --outtype q8_0. The GGUF metadata was then patched from block_count=65, nextn_predict_layers=1 to 64 / 0, because the MTP head is not included. No tensor data was changed.

Usage

The file is lean27b-r32_wd40_full-Q8_0.gguf (28.6 GB). It needs a llama.cpp build that supports qwen35.

llama-server -m lean27b-r32_wd40_full-Q8_0.gguf -ngl 99 -c 262144 --jinja
  • Text only: no mmproj, no MTP head.
  • VRAM: about 26.6 GiB for the weights, about 41.6 GiB at the full 262k context.
  • Sampling defaults embedded in the GGUF: temperature 1.0, top_p 0.95, top_k 20.
Downloads last month
39
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for noahsta/lean27b

Base model

Qwen/Qwen3.6-27B
Finetuned
(440)
this model