Open-source tools power nearly every web archive, including Perma.cc. From bot defenses to shrinking funding, sustaining this critical preservation infrastructure can't be an afterthought. Takeaways from #iipcWAC26 keynote: lil.law.harvard.edu/blog/2026/06... #WebArchiveWednesday

Sustaining Open-Source Web Archiving Infrastructure: Takeaways from IIPC Conference 2026 | Library Innovation Lab

The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.

lil.law.harvard.edu

Announcing WARCbench: an open-source command-line tool for exploring, analyzing, transforming, and extracting data from WARC files—built on a decade of Perma.cc experience to help archivists wrangle real‑world, messy web archives. #WebArchiveWednesday lil.law.harvard.edu/blog/2026/06...

WARCbench: A Swiss Army Knife for WARC Processing | Library Innovation Lab

The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.

lil.law.harvard.edu

In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...

In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...

In partnership with dataindex.us, LIL's #PubDataProject Team investigates solutions for detecting and monitoring changes in data over time at scale. Read more about our methods in our newest blogpost: lil.law.harvard.edu/blog/2026/06...

New blogpost from the #PublicDataProject! Project Lead Molly Hardy reflects on our #AI benchmarking exercise, which grew out of our partnership with @dataindex.us and is part of our efforts to build a data monitoring tool. Read more on the LIL blog: lil.law.harvard.edu/blog/2026/06...

An Automated Data Monitoring Toolkit and the AI Benchmarking Exercise at the Public Data Project | Library Innovation Lab

The Library Innovation Lab is growing knowledge and community by bringing library principles to technological frontiers.

lil.law.harvard.edu

When might digital provenance matter? Could we imagine it being used to right past wrongs, to return objects to their rightful places, to restore justice? Our Public Data Project's @mollyhardy.bsky.social reflects on copying gov data & principles of provenance lil.law.harvard.edu/blog/2025/12...

Replication of Government Datasets and the Principles of Provenance | Library Innovation Lab

As part of our Public Data Project, LIL recently launched Data.gov Archive Search. In this post, we consider the importance of provenance for large, replicat...

lil.law.harvard.edu

If you'd like an informative and interesting conversation to enjoy try "Inside Harvard’s Data.gov Archive – A Conversation with Jack Cushman" @jed.co interviewing Jack Cushman @harvardlil.bsky.social about Data.gov archiving on @source.coop Now live www.youtube.com/watch?v=XYMb...

Our EOT2024 partner @harvardlil.bsky.social was interviewed by @jed.co on @source.coop about archiving government data. Listen & learn how 300,000+ federal datasets are archived for posterity Inside Harvard’s Data.gov Archive, A Conversation with Jack Cushman: www.youtube.com/watch?v=XYMb...

If you missed our conversation with Jack Cushman from @harvardlil.bsky.social‬, you can catch up now. We discussed the Data.gov Archive and the challenge of preventing federal data loss – it's about more than just web pages. youtube.com/live/XYMbQru...

Jack Cushman: Inside Harvard’s Data.gov Archive (EP 3)

YouTube video by Great Data Products

youtube.com

Library Innovation Lab@harvardlil.bsky.social · 9mo ago

And join Jack live on @source.coop's Great Data Products podcast on 11/19 to hear how we're making cultural memory collections easier to access and harder to delete. greatdataproducts.com/housekeeping...

Workshop Report: "Resilience in Times of Crisis - Strengthening Open Science Against Geopolitical Pressures" (via Research Group Information Management" at Humboldt-Universität zu Berlin) infomgnt.org/posts/2025-1... @datarescueproject.org @harvardlil.bsky.social #libraries #openscience

Podcast: Jack Cushman joins “Pioneers & Pathfinders” to discuss libraries shaping legal tech, digital preservation, and realities of legal AI. Listen: www.seyfarth.com/news-insight...

Pioneers and Pathfinders: Jack Cushman

Today, we’re joined by Jack Cushman, director of the Harvard Library Innovation Lab, where he and his team are reimagining how library principles can shape the future of legal technology. Jack is a…

seyfarth.com

LIL fellow @maxy.bsky.social asked 14 scholars, archivists, designers, business leaders & engineers: "If you were given unlimited funding to design a system for storing and preserving digital information for at least a century, what would you do?" Their answers: lil.law.harvard.edu/generational...

Generational Data Interviews | Library Innovation Lab

14 Designs for Digital Preservation in 2025

lil.law.harvard.edu

Our Public Data Project Lead @mollyhardy.bsky.social writes about public data preservation and how it’s complicated by the artificial contemporary distinction between science and the humanities @scholarlykitchen.bsky.social

Guest Post - Rethinking Disciplinary Data Regimes - The Scholarly Kitchen

Between a political policy environment focused on defunding and deleting data collections – an environment in which little can be trusted – and an onslaught of new AI tools that feed indiscriminately ...

scholarlykitchen.sspnet.org

Join our team! LIL is looking for a Product and Research Manager to help create, shape, and execute on our portfolio of open knowledge projects. PRMs work across every piece of the LIL ecosystem, from software experimentation to convening of events. Learn more at careers.harvard.edu/job/product-...

Product and Research Manager

careers.harvard.edu

Thrilled to share that @maxy.bsky.social's "Century-Scale Storage" was nominated for a Webby Award! You can vote in the "Best Individual Editorial Feature" category here: vote.webbyawards.com/PublicVoting...

Vote for the best of the internet

I just voted in The Webby People's Voice Awards and checked my voter registration.

vote.webbyawards.com

As the @institutionaldatainitiative.org expands its mission, we’re announcing a collaboration with @bpl.boston.gov to develop AI-driven tools capable of accelerating new digitization at libraries across the world, starting at the Boston Public Library. institutionaldatainitiative.org/posts/using-...

Using AI to Accelerate Digitization at Boston Public Librarys

Today, as part of our mission expansion, we’re announcing a collaboration with BPL to develop AI-driven tools capable of accelerating new digitization of large collections at libraries across the worl...

institutionaldatainitiative.org

I'm pleased to announce we're expanding our mission at the @institutionaldatainitiative.org with an open call for institutional collaborators, new digitization at Harvard Law School Library, and additional support to advance this work. institutionaldatainitiative.org/posts/open-c...

Expanding Our Mission: An Open Call for Collaborators

Today, we’re pleased to announce an open call for institutional collaborators as new support expands the research capacity of the Institutional Data Initiative.

institutionaldatainitiative.org

Ed Summers at Stanford wrote this great deep dive of how and why we designed our data.gov archiver the way we did. Thanks for digging in, Ed, this is excellent. inkdroid.org/2025/02/17/n...

We just launched a 16TB archive of every dataset that has been available on data.gov since November. This will be updated day by day as new datasets appear. It can be freely copied, and we're sharing the code behind it to help others make their own archives of data they depend on.

Announcing the Data.gov Archive | Library Innovation Lab

Today we released our archive of data.gov on Source Cooperative. The 16TB collection includes over 311,000 datasets harvested during 2024 and 2025, a complet...

lil.law.harvard.edu

Penn is getting a lot of questions about Data Refuge. That effort no longer exists, but several efforts are currently active. I've created a doc from what I & others have suggested. I'll update as I hear more. Feel free to share or suggest: docs.google.com/document/d/1...

Data Rescue Efforts

Data / Website Rescue Efforts End of Term Crawl - The main coordinated effort to archive websites, but datasets have been more of a challenge. EDGI - They have been focused on environmental data. A ...

docs.google.com

Last year we started a project to download and preserve public data. lil.law.harvard.edu/blog/2025/01... Since saving public data is in the news today — but is always needed — let’s talk about what you can do to help.

Preserving Public U.S. Federal Data | Library Innovation Lab

lil.law.harvard.edu

What insights emerge when a librarian, a software engineer, and a legal scholar come together to experiment with Retrieval Augmented Generation (RAG) to explore over 800,000 French legal articles 🇫🇷? Blog post: lil.law.harvard.edu/blog/2025/01... Case study: lil.law.harvard.edu/open-french-...

I’ve been thinking so much lately about data storage, formats, inscription, archives & deep time for the #dataloss project. So cool to see this piece come out from @harvardlil.bsky.social on CENTURY-SCALE STORAGE by @maxy.bsky.social (ty @louravn.bsky.social!) lil.law.harvard.edu/century-scal...

Century-Scale Storage

If you had to store something for 100 years, how would you do it?

lil.law.harvard.edu

"We picked a century scale because most physical objects can survive 100 years in good care. It is attainable, and yet we selected it because the design of mainstream digital storage mediums are nowhere close to even considering this mark." lil.law.harvard.edu/century-scale-storage

Century-Scale Storage

If you had to store something for 100 years, how would you do it?

lil.law.harvard.edu

Library Innovation Lab@harvardlil.bsky.social · 2y ago

The question is "simple". This essay starts at the birthplace of the IBM 305 RAMAC, looks at the cloud, clay tablets, tape drives, distributed storage...But technology alone will never be the whole answer. Maxwell Neely-Cohen is a writer, technologist & fellow at LIL. Link below!

More news! @institutionaldatainitiative.org officially launched today. This massive effort was conceived at LIL by Greg Leppert, who's now leading the charge as executive director. Our mission to bring library principles to technological frontiers is embedded in IDI's purpose. Read more below!

This is such a cool piece and presented in a beautiful format

Maxwell Neely-Cohen@maxy.bsky.social · 2y ago

Here it is, the longest piece of nonfiction I’ve ever written, CENTURY-SCALE STORAGE. The kind folks at @harvardlil.bsky.social let me investigate the present and future of digital storage, and how we steward and preserve our most treasured cultural objects lil.law.harvard.edu/century-scal...

Happy to be here! 👋

Here it is, the longest piece of nonfiction I’ve ever written, CENTURY-SCALE STORAGE. The kind folks at @harvardlil.bsky.social let me investigate the present and future of digital storage, and how we steward and preserve our most treasured cultural objects lil.law.harvard.edu/century-scal...

Century-Scale Storage

If you had to store something for 100 years, how would you do it?

lil.law.harvard.edu

It has crazy good design from friends Shelby Wilson and Alex Miller. If you like reading about libraries, archives, hard drives, codexes, monasteries, wars, disasters, space probes, and indictments of the current corporate and government order, this piece might be for you...