VADRK155/Cortex-1-Code

Cortex 1 Code

A small GPT model for code generation, trained completely from scratch (custom PyTorch architecture, no transformers library involved).

• Parameters: 36.5m (36,505,728 learnable; the "30m" figure omitted the untied output head)
• Weights format: fp32
• Architecture: Decoder-only Transformer (custom implementation)
• Training: from scratch, no pretrained weights used
• Language: writes Python code, all comments and output in English

⚠️ Note: the context window is only 512 tokens, so the model will forget earlier parts of the code quickly.

⚠️ Note: this is a small model for code, so don't expect great results from it — but it can be fun to try out of curiosity.

How to run

2. Install dependencies:

pip install -r requirements.txt

3. Run:

python chat.py

Limitations

This is a small model, so it may produce broken, incomplete, or non-functional code — expected behavior for this parameter count.