commoncrawl/web-graph-embeddings

Web Graph Embedings from Common Crawl's Host-level Hyperlink Graph

Dense 128-dimensional embeddings for 52,913,544 web hosts, learned by link prediction on the Common Crawl host-level hyperlink graph release cc-main-2025-26-nov-dec-jan. Vectors are L2-normalized and served in float16; similarity is cosine (a dot product on the unit vectors). The dataset contains hosts with link degree >= 8 (total in+out degree), a 52.9 M / ~19 % induced subgraph that carries ~97 % of the edges. The source code is available on Github.

This is an experimental dataset release and additional versions are currently not planned.

What each row is

One host, keyed by two stable string identifiers derived deterministically from the Common Crawl host name (no per-release integer ids — those are ephemeral):

column	type	meaning
row_id	int64	global row index; shard = row_id // rows_per_shard
host_key	string	forward host, e.g. www.example.com (the stable identity key)
domain_key	string	registrable domain / PLD (eTLD+1), e.g. example.com
embedding	fixed_size_list<float16>[128]	the L2-normalized vector

Files

vectors/emb-00000-of-00014.parquet … emb-00013-of-00014.parquet   # ~1 GB each, 4 M rows/shard
manifest.json                                                     # schema/model version, per-shard row ranges + SHA-256

Shards are contiguous row ranges (not hash-partitioned), so row_id order is preserved across shards. To locate one host without downloading everything, read only the host_key/row_id columns of the shards (Parquet projects columns), then fetch the single shard row_id // 4_000_000.

Usage

Stream as a 🤗 datasets table:

from datasets import load_dataset
ds = load_dataset("commoncrawl/web-graph-embeddings", "cc_deg8_shallow_v1", split="cc_main_2025_26_nov_dec_jan", streaming=True)
row = next(iter(ds))
print(row["host_key"], len(row["embedding"]))

Materialize a contiguous matrix for FAISS / numpy (cosine = inner product on these unit vectors):

import numpy as np, pyarrow.parquet as pq, glob
mats, keys = [], []
for f in sorted(glob.glob("vectors/emb-*.parquet")):
    t = pq.read_table(f, columns=["host_key", "embedding"])
    keys += t["host_key"].to_pylist()
    mats.append(np.asarray(t["embedding"].to_pylist(), dtype=np.float32))
X = np.vstack(mats)                      # [52_913_544, 128], already L2-normalized

Provenance & model

• Source graph: Common Crawl host webgraph cc-main-2025-26-nov-dec-jan, aggregating crawls CC-MAIN-2025-47, CC-MAIN-2025-51, CC-MAIN-2026-04.
• Encoder: shallow (per-node) embedding table trained by link prediction (WholeGraph-sharded, in-batch negatives), dim=128, 3 epochs.
• Link-prediction quality (held-out edges): MRR 0.957, Hits@1 0.939, Hits@10 0.984.

Explore it in the browser

Two Hugging Face Spaces let you inspect the released vectors without downloading the full dataset.

2D map — an interactive projection of all 52.9 M hosts, streamed by tile. Pan and zoom, hover any point for its hostname and stats, or jump to one of the top 20k hosts by name. The same layout recolours by link degree, topic, content language or quality, which makes it easy to see where the embedding separates a property and where it does not — the view above is coloured by language.

Nearest-neighbour search — type a hostname and get its closest hosts in the full 128-dimensional space rather than in the 2D projection. All 52,913,544 hosts are indexed. It is the quickest way to check what the embedding considers "similar" for a site you already know well.

Evaluation

The model is trained only to predict links, so every result below is a probe of what link structure alone turns out to encode. Held-out link prediction is near-saturated (test_mrr 0.9574); the interesting question is what transfers.

Topic — the strongest transfer

A logistic probe on the raw vectors reaches macro-F1 0.388 over 24 WebOrganizer topics, against 0.062 for a majority-topic-per-TLD baseline — about 6× the identity baseline, on 170,399 labelled hosts. Link structure makes a site's topic close to linearly separable.

Language — near-perfect on well-connected hosts

A single 120-class macro-F1 badly understates this, because it averages over many classes with almost no held-out support. Stratified by link degree the signal is unambiguous (300,000 held-out hosts, cosine kNN):

host degree	accuracy	neighbour-precision	weighted-F1	macro-F1 @sup≥20
8–16	0.613	0.455	0.525	0.095
128–512	0.740	0.662	0.710	0.326
512–4096	0.921	0.892	0.913	0.596
≥ 4096	0.980	0.970	0.976	0.713

Overall accuracy is 0.673 and weighted-F1 0.612. Mid-resource languages climb from the floor to strong as degree rises — Spanish 0.02 → 0.45, Polish 0.03 → 0.95, Indonesian 0.02 → 0.83. The more links a host has, the more precisely the embedding places its language.

Spam — an independent and unusually durable signal

The embedding never saw a spam label. Compared with Anti-TrustRank (ATR), the classic link-based spam algorithm, over 52.6 M candidate hosts:

• They are not the same signal. Spearman(embedding, ATR) = −0.04.
• The embedding is the better clean-vs-spam discriminator. Against operator-reviewed clean sites, gold-AUROC is 0.937 for the graph-propagated embedding, 0.874 for a plain logistic probe, and 0.865 for ATR.
• It detects what ATR cannot. At a 1% false-positive budget against hard, spam-shaped sites that reviewers had manually cleared, the embedding finds 40% of known spam where ATR finds ~0%; at a 5% budget, 74% vs ~0%.
• It cuts false positives 4.5×. Using ATR for recall and the embedding to vet its top 1,000 drops legitimate false positives from 0.122 to 0.027, holding spam precision at 0.967.
• It survives a crawl change. On spam labelled from a later crawl, ATR's gold-AUROC falls to 0.56 — chance — while the embedding still ranks it (enrichment 173×, AUROC 0.91).

ATR remains the stronger raw ranker on the easy full-web pool (enrichment@1k 117.8×). The practical recipe is both: ATR for recall, the embedding to vet its hits and to catch the spam that guilt-by-association misses.

In one line: link structure alone carries a surprising amount of what a site is — its topic, its language wherever the host is well connected, and a spam signal independent of the classic link-based one.

Coverage and limitations

• Only the well-connected core. Training used hosts with total degree ≥ 8: 52.9 M of the crawl's 279.4 M hosts (19%), covering 13.07 B of ~13.4 B edges. The 81% long tail — hosts with almost no links — is not in this artifact. There is no vector for a host that was not trained.
• Transductive. There is no encoder to apply to a new host; a host absent from the training crawl cannot be embedded without retraining.
• Direction-blind, as above.
• One crawl window. Vectors from different releases are not comparable — the model space is arbitrary up to rotation, so a new release means a new space (tracked by model_space_version in the manifest).
• Spam metrics are a conservative floor. The label sets are partial, so unlabelled true spam is scored as a false positive; real precision is better than the numbers above.

License

Derived from the Common Crawl host webgraph. Its use is subject to the Common Crawl Terms of Use.