Common Crawl Foundation
@commoncrawl.bsky.social
Common Crawl is a non-profit foundation dedicated to the Open Web.
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June, and July 2026. commoncrawl.org/blog/host--a...
Common Crawl - Blog - Host- and Domain-Level Web Graphs May, June, and July 2026
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June, and July 2026, consisting of 240.4 million nodes and 3.7 billion edges at the host ...
commoncrawl.org
The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce some improvements and changes. commoncrawl.org/blog/july-20...
Common Crawl - Blog - July 2026 Crawl Archive Now Available
The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce som...
commoncrawl.org
Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and culturally informed AI development. commoncrawl.org/blog/common-...
Common Crawl - Blog - Common Crawl Joins Project Tapestry
Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and ...
commoncrawl.org
How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think. commoncrawl.org/blog/measuri...
Common Crawl - Blog - Measuring Crawled Coverage of a Website in Common Crawl
How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think.
commoncrawl.org
We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 31.7% work from everywhere, one vantage point turns out to be enough for the headline rate, and 5,530 hosts reveal why it isn't enough for the rest of the story. commoncrawl.org/blog/is-one-...
Common Crawl - Blog - Is one vantage point enough? IPv6 across the top million web hosts
We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 31.7% work from everywhere, one vantage point turns out to be enough for the headline rate, and 5,530 hosts...
commoncrawl.org
Turning 30,000 Arabic Domains Into a Better Crawl: How we filtered, geolocated and categorised a donation of Arabic seed domains commoncrawl.org/blog/turning...
Common Crawl - Blog - Turning 30,000 Arabic Domains Into a Better Crawl
How we filtered, geolocated and categorised a donation of Arabic seed domains
commoncrawl.org
The Common Crawl team attended the 16th International Conference on Language Resources and Evaluation in Palma, Mallorca, co-organizing a tutorial, presenting recent published work, and strengthening links with the research community. commoncrawl.org/blog/common-...
Common Crawl - Blog - Common Crawl Foundation at LREC 2026
The Common Crawl team attended the 16th International Conference on Language Resources and Evaluation in Palma, Mallorca, co-organizing a tutorial, presenting recent published work, and strengthening ...
commoncrawl.org
The latest @commoncrawl.bsky.social crawl indexes way more than HTML pages. 20.9M PDFs. Plus calendars, BibTeX, markdown... The whole index now lives in a @hf.co Bucket, so I pulled this with one SQL query straight over it (new S3 API). 2.1B rows, nothing downloaded, $0 to read.
The WaC-13 workshop invites research submissions on web data, corpus building, and linguistic analysis. commoncrawl.org/blog/13th-we...
Common Crawl - Blog - 13th Web-as-Corpus Workshop @ EMNLP 2026
The WaC-13 workshop invites research submissions on web data, corpus building, and linguistic analysis.
commoncrawl.org
The May 2026 crawl archive (CC-MAIN-2026-21) is now also available on our HF bucket. 🤗 huggingface.co/buckets/comm...
commoncrawl/commoncrawl - Storage Bucket
Storage bucket commoncrawl/commoncrawl on Hugging Face
huggingface.co
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of April, May, and June 2026. commoncrawl.org/blog/host--a...
Common Crawl - Blog - Host- and Domain-Level Web Graphs April, May, and June 2026
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of April, May, and June 2026. The graphs consist of 247.3 million nodes and 6.3 billion edges at ...
commoncrawl.org
We are happy to announce the release of the June 2026 crawl archive, consisting of 2.10 billion web pages, or 354.59 TiB of uncompressed content. commoncrawl.org/blog/june-20...
Common Crawl - Blog - June 2026 Crawl Archive Now Available
We are happy to announce the release of the June 2026 crawl archive, consisting of 2.10 billion web pages, or 354.59 TiB of uncompressed content.
commoncrawl.org
CommonLID, a community-built language ID benchmark, has a new website and interactive leaderboard. Its paper was accepted to ACL 2026, with a poster session on 7 July. Source code, a PyPI package, and the dataset are now available. commoncrawl.org/blog/commonl...
Common Crawl - Blog - CommonLID Update: New Tools, Growing Impact
CommonLID, a community-built language ID benchmark, has a new website and interactive leaderboard. Its paper was accepted to ACL 2026, with a poster session on 7 July. Source code, a PyPI package, and...
commoncrawl.org
Common Crawl was well represented with contributions at the 2026 IIPC Web Archiving Conference and General Assembly. commoncrawl.org/blog/common-...
Common Crawl - Blog - Common Crawl Foundation at IIPC-WAC 2026
Common Crawl was well represented with contributions at the 2026 IIPC Web Archiving Conference and General Assembly.
commoncrawl.org
We are very happy to announce our next seminar: Pedro Ortiz Suarez @pjox.bsky.social ( @commoncrawl.bsky.social ) "Expanding Linguistic and Cultural Coverage in Common Crawl" on Friday 12th June 2026, 11am CEST. Details here 👉 almanach.inria.fr/seminars-en....
The Columnar Index Is Now the URL Index! We have renamed the Columnar Index to the URL Index, to be clearer about its purpose and to pave the way for more datasets in a columnar format. commoncrawl.org/blog/the-col...
Common Crawl - Blog - The Columnar Index Is Now the URL Index
We have renamed the Columnar Index to the URL Index, to be clearer about its purpose and to pave the way for more datasets in a columnar format.
commoncrawl.org
Introducing the AI Visibility Audit! A free guide for SEOs and GEOs on how to check whether AI systems can actually reach a site, and how to stay visible in the crawl that trains them. commoncrawl.org/blog/introdu...
Common Crawl - Blog - Introducing the AI Visibility Audit
A free guide for SEOs and GEOs on how to check whether AI systems can actually reach a site, and how to stay visible in the crawl that trains them.
commoncrawl.org
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, April, and May 2026. commoncrawl.org/blog/host--a...
Common Crawl - Blog - Host- and Domain-Level Web Graphs March, April, and May 2026
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, April, and May 2026. The graphs consist of 262.4 million nodes and 8.1 billion edges at...
commoncrawl.org
Under-represented languages deserve better tools! On June 4th, The Common Crawl Foundation and Mozilla Data Collective will host a webinar to test language identification for the languages you care about.
We are happy to announce the release of the May 2026 crawl archive, consisting of 2.16 billion web pages, or 365.56 TiB of uncompressed content. www.commoncrawl.org/blog/may-202...
Common Crawl - Blog - May 2026 Crawl Archive Now Available
We are happy to announce the release of the May 2026 crawl archive, consisting of 2.16 billion web pages, or 365.56 TiB of uncompressed content.
commoncrawl.org
As an early experiment in distributing Common Crawl data through another channel, the April 2026 crawl archive is now available in a Hugging Face Storage Bucket, alongside its existing home on AWS S3. commoncrawl.org/blog/april-2...
Common Crawl - Blog - April 2026 Crawl Archive Now Available in a Hugging Face Storage Bucket
As an early experiment in distributing Common Crawl data through another channel, the April 2026 crawl archive is now available in a Hugging Face Storage Bucket, alongside its existing home on AWS S3.
commoncrawl.org
Browsers can now fetch Common Crawl data directly, no backend needed. Build SQL explorers, snapshot viewers and diff tools as static pages. commoncrawl.org/blog/you-can...
Common Crawl - Blog - You can now build directly on Common Crawl from the browser
Browsers can now fetch Common Crawl data directly, no backend needed. Build SQL explorers, snapshot viewers and diff tools as static pages.
commoncrawl.org
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March, and April 2026. commoncrawl.org/blog/host--a...
Common Crawl - Blog - Host- and Domain-Level Web Graphs February, March, and April 2026
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March, and April 2026. The graphs consist of 269.0 million nodes and 9.4 billion edg...
commoncrawl.org
We are pleased to announce that the crawl archive for April 2026 is now available, containing 2.19 billion web pages or 379.2 TiB of uncompressed content. commoncrawl.org/blog/april-2...
Common Crawl - Blog - April 2026 Crawl Archive Now Available
We are pleased to announce that the crawl archive for April 2026 is now available, containing 2.19 billion web pages or 379.2 TiB of uncompressed content.
commoncrawl.org
Check out our newsletter for April 2026, with updates on what we've been up to. www.commoncrawl.org/blog/april-2...
Common Crawl - Blog - April 2026 Common Crawl Newsletter
Check out our newsletter for April 2026, with updates on what we've been up to.
commoncrawl.org
Common Crawl is switching to reporting dataset sizes in nibbles. As an organisation dedicated to data preservation, we feel it would be remiss to allow this underrepresented unit to fall out of use. Our latest crawl now exceeds 689 tebibbles. commoncrawl.org/blog/announc...
Common Crawl - Blog - Announcing a Change to Common Crawl Dataset Size Reporting
Common Crawl is switching to reporting dataset sizes in nibbles. As an organisation dedicated to data preservation, we feel it would be remiss to allow this underrepresented unit to fall out of use. O...
commoncrawl.org
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of January, February, and March 2026. commoncrawl.org/blog/host--a...
Common Crawl - Blog - Host- and Domain-Level Web Graphs January, February, and March 2026
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of January, February, and March 2026. The graphs consist of 270.2 million nodes and 9 billion edg...
commoncrawl.org
We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content. We also observed a dramatic increase in fetches over IPv6, explained by the enabling of Happy Eyeballs in the OkHttp library. commoncrawl.org/blog/march-2...
Common Crawl - Blog - March 2026 Crawl Archive Now Available
We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content. We also observed a dramatic increase in fetches over IPv6, exp...
commoncrawl.org
Blocking the Internet Archive Won’t Stop AI, But It Will Erase the Web’s Historical Record www.eff.org/deeplinks/20... @eff.org
Blocking the Internet Archive Won’t Stop AI, But It Will Erase the Web’s Historical Record
Imagine a newspaper publisher announcing it will no longer allow libraries to keep copies of its paper. That’s effectively what’s begun happening online in the last few months. The Internet Archive—th...
eff.org
We probed the 100,000 most-linked web hosts for IPv6 support using the Common Crawl Web Graph. Only 36.9% are fully reachable over IPv6, with adoption ranging from 71% among the top 100 to 32% in the long tail. commoncrawl.org/blog/ipv6-ad...
Common Crawl - Blog - IPv6 Adoption Across the Top 100K Web Hosts
We probed the 100,000 most-linked web hosts for IPv6 support using the Common Crawl Web Graph. Only 36.9% are fully reachable over IPv6, with adoption ranging from 71% among the top 100 to 32% in the ...
commoncrawl.org