PiCo-7B

PiCo-7B is a 7 billion parameter (7B) large language model featuring a novel Adaptive Hierarchical Mixture of Experts (AHMoE) architecture. This model aims to significantly reduce inference costs while maintaining powerful expressive capabilities through dynamic routing and expert compression techniques.

---

🚀 Model Highlights

• High-Performance AHMoE Architecture: With approximately 6.95B total parameters, the model activates only about 2.63B parameters per token, achieving a "large capacity, lightweight" inference experience.
• Adaptive Hierarchical Routing: Introduces a two-level routing mechanism to optimize expert selection efficiency.
• Context-Aware Meta-Router: Dynamically analyzes input complexity and context to adjust routing decisions.
• Ultra-Long Context Support: Natively supports context lengths up to 131,072 (128K) tokens.
• OzmosToken Tokenizer: Custom-developed multilingual tokenizer supporting 20+ language families, with specialized thought and reasoning markers.

---

🏗️ Architecture Specifications

Core Parameters

Parameter	Value
Total Parameters	~6.95B
Non-Embedding Parameters	~6.32B
Active Parameters per Token	~2.63B (Top-2 Experts + Shared Expert)
Hidden Size	2048
Intermediate Size	5888
Number of Layers	30 (15 MoE Layers + 15 Dense Layers)
Attention Heads	16 (Q) / 4 (GQA)
Context Window	131,072 tokens
Vocabulary Size	152,064

AHMoE Technical Details

1. Hierarchical Routing: Experts are grouped, allowing for coarse-grained group selection followed by fine-grained expert selection within groups, reducing routing computational complexity.
2. Meta-Router: Utilizes a context encoder to evaluate token difficulty, dynamically invoking more appropriate expert combinations for complex tasks.
3. Shared Knowledge Base: A shared knowledge query module across all layers reduces redundant information storage among experts.
4. Dynamic Expert Pruning: Automatically prunes low-contributing experts during runtime based on importance scores, improving computational efficiency.
5. Expert Compression: Compresses expert weights (default 50%) to further optimize memory footprint.

---

🔤 OzmosToken Tokenizer

PiCo-7B uses the specially optimized OzmosToken, supporting over 20 language families including East Asian (Chinese, Japanese, Korean), European, Middle Eastern, and South Asian languages.

Special Tokens

The model incorporates rich functional tokens to guide it in performing specific tasks:

• <|thought|> / <|reasoning|>: Triggers Chain-of-Thought (CoT) and logical reasoning.
• <|summary|> / <|translation|>: Task-oriented markers.
• <|code|> / <|math|>: For programming and mathematical problem-solving.
• <|user|> / <|assistant|> / <|system|>: Standard conversational role markers.

---

🛠️ Installation and Usage

Environment Setup

pip install torch accelerate transformers safetensors datasets bitsandbytes peft trl lm-eval

Quick Start (Inference)

import torch
from pico import PiCoConfig, PiCoForCausalLM
from ozmos_tokenizer import OzmosTokenizer

# Load configuration and model
config = PiCoConfig.from_json("pico-7b/configs/pico_7b.json")
model = PiCoForCausalLM(config)
tokenizer = OzmosTokenizer.from_pretrained("pico-7b/ozmos_tokenizer_model")

# Example generation
inputs = tokenizer("<|user|>Please explain what a Mixture of Experts (MoE) model is?<|assistant|><|thought|>", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0]))

---

📜 License and Citation

This project is licensed under a custom license based on OpenRAIL.

@misc{pico7b,
    title={PiCo-7B: Adaptive Hierarchical Mixture of Experts},
    author={PiCo Team},
    year={2025},
    url={https://github.com/pico-ai/pico-7b}
}