README.md

---
pipeline_tag: text-generation
inference: true
widget:
- text: '<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = elem - elem2\n                if distance < threshold:\n                    return True\n\n    return False<commit_message>Fix bugs in has_close_elements.<commit_after>'
  example_title: Fix has_close_elements
  group: Python
license: bigcode-openrail-m
datasets:
- bigcode/commitpack-subset-cf
metrics:
- code_eval
library_name: transformers
tags:
- code
model-index:
- name: SantaCoderPack
  results:
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix Python
    metrics:
    - name: pass@1
      type: pass@1
      value: 3.2
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix JavaScript
    metrics:
    - name: pass@1
      type: pass@1
      value: 4.9
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix Java
    metrics:
    - name: pass@1
      type: pass@1
      value: 1.8
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix Go
    metrics:
    - name: pass@1
      type: pass@1
      value: 3.6
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix C++
    metrics:
    - name: pass@1
      type: pass@1
      value: 4.2
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix Rust
    metrics:
    - name: pass@1
      type: pass@1
      value: 1.7
      verified: false
  - task:
      type: text-generation
    dataset:
      type: bigcode/humanevalpack
      name: HumanEvalFix Average
    metrics:
    - name: pass@1
      type: pass@1
      value: 3.3
      verified: false
---
![Octopack](https://github.com/bigcode-project/octopack/blob/31f3320f098703c7910e43492c39366eeea68d83/banner.png?raw=true)

# Table of Contents

1. [Model Summary](#model-summary)
2. [Use](#use)
3. [Training](#training)
4. [Citation](#citation)

# Model Summary

SantaCoderPack is an pre-trained model with the same architecture of SantaCoder on <th><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a> using this format: 

```
<commit_before>code_before<commit_msg>message<commit_after>code_after
```

- **Repository:** [bigcode/octopack](https://github.com/bigcode-project/octopack)
- **Paper:** [OctoPack: Instruction Tuning Code Large Language Models](https://arxiv.org/abs/2308.07124)
- **Languages:** Python, JavaScript, Java, C++, Go, Rust
- **SantaCoderPack:**
<table>
<tr>
<th>Data</t> 
<th><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a></th>
<td>4TB of GitHub commits across 350 programming languages</td>
</tr>
<tr>
<th>Model</t> 
<th><a href=https://huggingface.co/bigcode/octocoder>SantaCoderPack</a></th>
<td>SantaCoderPack (1.1B parameters) pre-trained on CommitPack</td>
</tr>
<tr>
<th>Evaluation&nbsp;&nbsp;</t> 
<th><a href=https://huggingface.co/datasets/bigcode/humanevalpack>HumanEvalPack/HumanEvalFix</a></th>
<td>Extension of OpenAI's HumanEval to HumanEvalFix</td>
</tr>
</table>


# Use

## Intended use

The model follows instructions provided in the input. We recommend prefacing your input with "<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = elem - elem2\n                if distance < threshold:\n                    return True\n\n    return False<commit_message>Fix bugs in has_close_elements.<commit_after>"

**Feel free to share your generations in the Community tab!**

## Generation

```python
# pip install -q transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "bigcode/santacoderpack"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer.encode("Q<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n    for idx, elem in enumerate(numbers):\n        for idx2, elem2 in enumerate(numbers):\n            if idx != idx2:\n                distance = elem - elem2\n                if distance < threshold:\n                    return True\n\n    return False<commit_message>Fix bugs in has_close_elements.<commit_after>", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```

# Training

## Model

- **Architecture:** GPT-2 model with multi-query attention
- **Steps:** 250k pretraining
- **Pretraining tokens:** 131B
- **Precision:** bfloat16

## Hardware

- **Pretraining:**
  - **GPUs:** 32 Tesla A100
  - **Training time:** 15 days

## Software

- **Orchestration:** [Megatron-LM/Transformers](https://github.com/bigcode-project/santacoderpack#training)
- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)

# Citation

```bibtex
@article{muennighoff2023octopack,
      title={OctoPack: Instruction Tuning Code Large Language Models}, 
      author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},
      journal={arXiv preprint arXiv:2308.07124},
      year={2023}
}
```