HelloSun/Experiment_1

openPangu-R-7B-2512-Q4_K_M (GGUF)

Powered by openPangu. openPangu is a trademark of Huawei Technologies Co., Ltd.

Q4_K_M GGUF conversion of openpangu/openPangu-R-7B-2512, together with a CPU inference backend study (llama.cpp vs oneDNN vs OpenVINO). This is an unofficial community conversion, not released or endorsed by Huawei.

Repository layout

path	contents
openPangu-R-7B-2512-Q4_K_M.gguf	the Q4_K_M model (4.72 GiB)
openpangu-r-llama.cpp-v0.7.0.patch	llama.cpp v0.7.0 patch adding the openpangu-r architecture
build_openpangu_r.sh	one-click clone + patch + build + model download
onednn/	CPU backend comparison: oneDNN backend patch, OpenVINO programs, benchmarks, raw logs
LICENSE	OpenPangu Model License Agreement 1.0

Important

• Upstream llama.cpp does not support this model. The GGUF uses the architecture name openpangu-r, which exists only in the patch in this repository. The patch targets llama.cpp v0.7.0 (latest stable release).
• Tested on CPU only (x86-64, AVX-512). GPU backends were not tested.

Quick start

bash build_openpangu_r.sh          # clones llama.cpp v0.7.0, applies the patch, builds, downloads the GGUF

Options: SKIP_MODEL=1 skips the download; HF_TOKEN is only needed if this repository is private. The script downloads from this repository (HelloSun/Experiment_1) and checks the sha256 of the patch and the GGUF.

Run (raw prompt, Chinese chat template of the model):

./llama.cpp/build/bin/llama-completion -m models/openPangu-R-7B-2512-Q4_K_M.gguf -no-cnv --temp 0 -n 128 --no-display-prompt \
  -p "[unused9]系统：你是一个有帮助的助手。[unused10][unused9]用户：用一句话介绍什么是大语言模型。 /no_think[unused10][unused9]助手：[unused16]"

Prompt format: [unused9]系统：…[unused10][unused9]用户：…[unused10][unused9]助手：[unused16]. Append /no_think to a user message for fast (non-thinking) answers. The answer follows [unused17]; generation ends with [unused10].

Conversion

1. convert_hf_to_gguf.py (patched) with --outtype f16 from the BF16 safetensors. The MTP module (layer 27) and the second final norm are dropped.
2. llama-quantize <f16.gguf> <out.gguf> Q4_K_M. Sinks and the conv weight are kept in F32.

Verification (done in this environment)

• Reference check: a CPU fp32 reference was written from the HF modeling code. For a 33-token prompt, the top-10 logits match llama.cpp F16 (max |Δlogit| ≈ 0.01). For a 484-token prompt (sliding-window layers active), the top-10 match with max |Δlogit| ≈ 0.03. Turning the sliding window off in the reference changes the logits by about 6, so the window is verified.
• Greedy decoding: llama.cpp F16 matches the fp32 reference token by token (11 tokens checked).
• Batching: F16 gives the same results with prompt decoding in one batch, in chunks of 4, and token by token.
• Long context: a fact placed about 400 tokens before the question was answered correctly (7312).
• Multi-turn: a second user turn appended to the same context is answered in context.

Sample outputs (Q4_K_M):

prompt	output
用一句话介绍什么是大语言模型（快思考）	大语言模型是一种基于大量文本数据训练而成的人工智能模型，能够理解和生成人类语言，广泛应用于自然语言处理任务。
写一首关于秋天的四行短诗（快思考）	秋风轻拂叶纷飞，/ 金黄铺满小径长。/ 寒蝉低唱夕阳下，/ 一缕炊烟入梦乡。
一个班有30名学生，其中2/5是女生，女生有多少人？（慢思考）	12人
123 + 456 等于多少？（慢思考）	579

Note: in fast mode the model once answered 123 + 456 = 679. The fp32 reference gives the same answer for that prompt, so this is a model behavior, not a conversion or implementation error. Thinking mode answers 579.

CPU backend comparison (Q4_K_M)

Measured on an Intel Xeon Platinum 8559C (192 threads host, 16-CPU container, AMX BF16/INT8, AVX-512), 80 GiB memory guard, no GPU. pp512 / tg128 are steady-state throughput; OpenVINO tg128 is the median of 128 decode steps. Full tables, accuracy checks and methodology: onednn/README.md, raw numbers in onednn/results.csv.

backend	weights	pp512 tok/s	tg128 tok/s	ppl512	argmax vs f32
llama.cpp (plain AVX512)	Q4_K_M	180.50	13.39	-	-
llama.cpp (default AMX+repack)	Q4_K_M	112.77	12.19	-	-
oneDNN	int8	90.09	7.67	-	-
oneDNN	bf16	75.52	4.79	-	-
OpenVINO	f32	125.60	2.22	28.82	1.000
OpenVINO	bf16-hint	209.43	3.93	28.85	0.975
OpenVINO	int8-asym	219.17	4.70	29.11	0.951
OpenVINO	int8-sym	226.68	-	29.34	0.941
OpenVINO	int4-asym	223.65	5.20	32.68	0.703
OpenVINO	int4-sym	222.48	-	42.95	0.701

Takeaways:

• llama.cpp Q4_K_M is the fastest overall on this 16-CPU container: 13.39 tg128, 180.50 pp512.
• The oneDNN hook (in onednn/onednn.patch) is bandwidth-bound: bf16 weights are 16.6 GB/token vs 5 GB/token for Q4_K, int8 8.3 GB/token. The AMX int8 kernel itself is fast (7.1 TFLOPS at M=512); the loss comes from the weight format and per-call activation quantization.
• OpenVINO has the fastest prefill but the slowest decode here, because its decode graph runs attention over the full 640-slot KV cache and moves all KV tensors across the host/plugin boundary each step. int8/int4 are NNCF weight-only compression of the f32 IR: int8 stays close to f32 (PPL +0.3, ~95% argmax), int4 keeps short factual answers but loses distributional fidelity (PPL +3.9, ~70% argmax).

Reproduce: build the oneDNN backend with onednn/build.sh <llama.cpp dir>, benchmark with onednn/bench.sh <Q4_K_M.gguf> (needs a plain AVX512 build and an oneDNN build alongside a default build). The OpenVINO route is a direct ov.convert_model trace of the HF model (stateless prefill + decode IRs); optimum-cli, openvino-genai LLMPipeline and torch.compile(backend="openvino") all reject this architecture — see onednn/README.md and onednn/openvino_quantization.md.

Known limitations

• One sequence per batch. A batch with more than one sequence aborts. The conv history of the merge layers is kept inside the model and follows one sequence decoded contiguously. Prompt-cache reuse, seq_rm, and multi-slot serving are not supported.
• The MTP (next-token) head is not included and not used.
• Hadamard-rotated (quantized) KV cache is not supported; use the default f16 KV cache.
• llama-server was not validated.
• The oneDNN backend handles only 2D quantized-weight x 2D activation MUL_MAT (projections and lm_head); batched attention matmuls stay on the ggml path. int4 has no optimized kernel in oneDNN 3.7.

License and notices

The model and derivative works are licensed under the OPENPANGU MODEL LICENSE AGREEMENT VERSION 1.0 (see LICENSE).

• EU restriction (Section 3): you must not access, download, install, run, deploy, integrate, modify, or otherwise use this model within the European Union.
• Acknowledgment (Section 4.2): "Powered by openPangu" and the trademark notice "openPangu is a trademark of Huawei Technologies Co., Ltd." must appear on pages, manuals, or materials that describe features of the model.
• The model is provided "AS IS" without warranty. Outputs are generated automatically and may be inaccurate.

Files and checksums

file	sha256
openPangu-R-7B-2512-Q4_K_M.gguf	efeb81edbf64ad9d7a80254332f3128d86deec589ee39315a3c1f7ec1aa5f10a
openpangu-r-llama.cpp-v0.7.0.patch	b5f302da0cece9c6bc717982d820f8b3e6ca0970f8f32b549efb41ef0d6b8773
LICENSE	747083c8c24995c75ed641d5c031a95a859b1ec37cd3deeb033425a75249a3cb

Backend-study material (all small text files):

path	contents
onednn/README.md	full comparison, environment, accuracy and per-backend notes
onednn/results.csv	every measured configuration
onednn/onednn.{cpp,h}, onednn/onednn.patch	oneDNN MUL_MAT backend for ggml-cpu
onednn/build.sh, onednn/bench.sh	build and benchmark helpers
onednn/openvino/	prefill/decode IR builders, NNCF compression, PPL/argmax and isolation tests
onednn/logs/	raw llama-bench, OpenVINO and NNCF logs (incl. logs/quant/)
onednn/openvino_quantization.md	why int8/int4 needed NNCF weight compression