MGnify Protein Catalogues

This repository contains the LiteFold/Mgnify MGnify protein catalogue payload plus a compact default index for the Hugging Face Dataset Viewer.

The raw sequence and table payload is TB-scale, so the default config is intentionally a file/shard index rather than a duplicate of every raw row. The raw files remain in sequences/ and tables/; use the index to discover sources, shards, part files, sizes, and download patterns, then stream or download only the payload files you need.

Dataset Summary

Metric	Value
Default index rows	3,226
Default index columns	34
Repository files indexed	3,226
Repository bytes indexed	3,156,136,546,084
Sequence source files	26
Sequence shards	3,148
Sequence shard bytes	342,815,994,580
Logical table sources	28
Logical table bytes	2,380,980,172,257
Table files in repo	76
Table repo bytes	2,813,320,536,987

The table repo bytes include both top-level table JSONL files and split part files where both are present. The logical table bytes count each upstream table once.

Default Splits

The default index split is deterministic by file id:

sha256(file_id) % 10

Bucket 0 is test; buckets 1 through 9 are train.

Split	Rows
train	2,902
test	324

These are file-index splits, not biological train/test sequence splits. For model training, create sequence-level or cluster-level splits appropriate to your task after loading the relevant MGnify payload.

Loading With datasets

Load the default file/shard index:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify")
print(index)
print(index["train"][0])

Load one split directly:

from datasets import load_dataset

train_index = load_dataset("LiteFold/Mgnify", split="train")

Find sequence shards for one source family:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify", split="train")
mgy_clusters = index.filter(
    lambda row: row["role"] == "sequence_shard"
    and row["source_family"] == "mgy_clusters"
)
print(mgy_clusters[0]["download_pattern"])

Find split table parts:

from datasets import load_dataset

index = load_dataset("LiteFold/Mgnify", split="train")
parts = index.filter(lambda row: row["role"] == "table_split_part")
print(parts[0]["path"], parts[0]["size_bytes"])

Streaming Raw FASTA Shards

Download one source family with the Hub client:

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="LiteFold/Mgnify",
    repo_type="dataset",
    allow_patterns=[
        "sequences/sequence_mgnify_current_release_mgy_clusters.fa.gz/shard-*.fasta.zst"
    ],
)
print(local_dir)

Stream a shard without downloading the whole source family:

from huggingface_hub import HfFileSystem
import zstandard as zstd

fs = HfFileSystem()
path = (
    "datasets/LiteFold/Mgnify/"
    "sequences/sequence_mgnify_current_release_mgy_clusters.fa.gz/"
    "shard-000001.fasta.zst"
)

dctx = zstd.ZstdDecompressor()
with fs.open(path, "rb") as f, dctx.stream_reader(f) as reader:
    chunk = reader.read(1 << 20)
    print(chunk[:200])

Downloading Raw Table Parts

For split tables, use the download_pattern column or a direct include pattern:

hf download LiteFold/Mgnify --repo-type dataset \
  --include 'tables/sequence_mgnify_current_release_mgy_proteins_pfam.tsv.gz.jsonl.parts/part-*.jsonl' \
  --local-dir ./mgnify

For unsplit tables:

hf download LiteFold/Mgnify --repo-type dataset \
  --include 'tables/sequence_mgnify_current_release_mgy_seq_metadata_2.tsv.gz.jsonl' \
  --local-dir ./mgnify

The raw table files are not registered as datasets configs because they are multi-TB nested JSONL payloads. Keeping the default config as a compact Parquet index prevents accidental full-repo scans and keeps the Dataset Viewer responsive.

Default Columns

Column	Description
file_id	Stable file identifier, currently the repository path.
repo_id	Hugging Face dataset repository id.
source_sha	Source commit used to build the index.
dataset_id	mgnify_proteins.
source_family	Parsed source family such as mgy_clusters, mgy_proteins_1, or mgy_seq_metadata_2.
source_slug	Source slug/path component used by the repository.
source_file	Original MGnify source path when derivable.
path	File path in the repository.
role	File role: sequence_shard, table_jsonl, table_split_part, table_split_manifest, readme, or git_attributes.
shard_index	FASTA shard index, otherwise -1.
part_index	Split table part index, otherwise -1.
size_bytes	File size in bytes.
compression	File/container format.
logical_table_size_bytes	Logical source table size when applicable, otherwise -1.
split_part_count	Number of table split parts when applicable, otherwise -1.
split_chunk_bytes	Target split chunk size when applicable, otherwise -1.
sequence_source_shard_count	Number of shards in the sequence source, otherwise -1.
sequence_source_bytes	Total bytes for that sequence source, otherwise -1.
repo_file_count	Total repository files indexed.
repo_total_bytes	Total indexed repository bytes.
sequence_shard_count_total	Total sequence shards.
sequence_shard_bytes_total	Total sequence shard bytes.
table_repo_file_count_total	Total table files in the repo, including split manifests and parts.
table_repo_bytes_total	Total table repo bytes, including duplicated top-level and split files where both exist.
logical_table_count_total	Logical upstream table count.
logical_table_bytes_total	Logical upstream table bytes.
is_sequence_shard	Whether the row is a FASTA shard.
is_table_file	Whether the row is a table file or table manifest.
is_split_part	Whether the row is a split table part.
is_split_manifest	Whether the row is a split manifest.
is_original_table_copy	Whether a top-level table also has split parts.
download_pattern	Glob or exact path for downloading related payload files.
access_note	Short usage note.
split_bucket	Deterministic bucket used for the default train/test split.

Files

• data/*.parquet: default file/shard index for Dataset Viewer.
• metadata/source_files.parquet: full index copy.
• sequences/*/shard-*.fasta.zst: raw compressed MGnify FASTA shards.
• tables/**/*.jsonl: raw normalized table payloads and split parts.
• _MANIFEST.json: index build summary.
• dataset_summary.json: same summary in a Dataset Viewer-adjacent file.
• scripts/prepare_mgnify_dataset.py: script used to build the default index.

Source

Derived from LiteFold/Mgnify, originally sourced from EMBL-EBI MGnify.

License

CC BY 4.0.

Citation

If you use the MGnify data, cite MGnify:

Mitchell AL, et al. MGnify: the microbiome analysis resource in 2020. Nucleic Acids Research, 48(D1):D570-D578, 2020.