It would be best if absolutely everything was immortalized onchain forever… but it’s just not practical. Permanent data is around $0.015 per MB. Not expensive for something vitally important that needs to live for 200 years; very expensive for a few hidden gems amid terabytes of garbage.
In an ideal world, we’d have a way to automatically determine an artifact’s significance before making it permanent. But how do we know what is worth keeping forever?
One idea we had at Decent Land Labs was an agent-based way to plug into a temporary storage layer and immortalize some of its data on Arweave if certain conditions were met. Before we get into the theory, here’s a quick overview of the technicals.
How conditional storage works
We recently developed a HyperBEAM-powered S3 client with a native pipeline to Arweave via ar.io’s Turbo bundles settlement layer. The first iteration of the future network - the beta release for Load S3 - an object-storage temporal data storage layer is live, with ~300TB of storage space available to be rented.
The S3 client uses a S3 cluster spun in the same location as the HyperBEAM node, or connected to an external cluster and time-bound storage terms to provide non-permanent storage guarantees. The uploader can choose at any time to commit that same data to Arweave, retaining its ID and integrity across both networks.
In the Load-S3 beta release, temporary DataItem storage guarantees are inherited from the MinIO cluster: native erasure-coded redundancy, fault-tolerance and data availability proofs. With that, Load S3’s beta release offers guarantees that are similar to what centralized bundling services on Arweave offer.
In some cases, the moment when data must become permanent is obvious. When flipping the switch between testing and production. When a user asks you to permanently archive their content. When a draft becomes the final. But some cases are more fuzzy. When is data valuable?
Not all data is valuable… but who’s to say?
“The archive has always been a pledge, and like every pledge, a token of the future. To put it more trivially: what is no longer archived in the same way is no longer lived in the same way.” – Derrida, 1996
If we knew what was important, we might have archived it. If we knew it was at risk, we might have made it permanent. Barely at the dawn of the digital age, Derrida is saying that the way we view data’s significance changes how we store it. Every time we decide what to keep and what to delete, we’re making bets about what future generations will find important.
But here’s the problem: the accountant uploading a trove of incriminating corporate documents doesn’t know if it will matter. The researcher saving preliminary data can’t predict which findings will be significant. The blogger reporting on what’s happening in their city has no idea if it will be historically significant.
This is really a version of what economist Friedrich Hayek called the “knowledge problem.” In a 1945 essay Hayek pointed out that “the knowledge of the circumstances of which we must make use never exists in concentrated or integrated form but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess.”
Hayek was talking about economic planning, but the same logic applies to digital preservation. No single archivist or algorithm can know what will be important in the future. The knowledge about significance is scattered across time and people. And much of it doesn’t even exist yet.
Markets are oracles for the value of data
As relevant for Hayek in the 40s as it is for us today, markets \= information. Markets are simple no-AI-required decision makers that poll the wisdom of the crowds. Markets (“a kind of machinery for registering change, or a system of telecommunications”) don’t just find out the truth about the value of an asset, they can find the truth about any information. Prediction markets like Polymarket provide indicators and final decisions about the truthiness of any event.
Conditional permanence systems respond to existing markets: Polymarket resolving prediction outcomes, academic citations as an indicator of veracity, regulatory agencies opening investigations, Wikipedia editors reaching consensus on contested articles. These are all ongoing processes that filter for significance.
The system watches these triggers and runs a basic operation: move file from temporary storage to permanent storage. When the Polymarket question “Will SBF be convicted?” resolves to “Yes,” documents related to FTX earn permanence. When the SEC announces an enforcement action against a tech company, stored communications about that company’s compliance practices get promoted to long-term archives. When a Wikipedia article about a scientific controversy reaches editor consensus after months of debate – or even gets large sections censored – archive it. That’s one side of the vision.
Glued together by AI (of course)
Whether it’s a market, an authority, or consensus amongst a crowd, we can hook into existing indicators without needing to be our own arbiters of permanence. Or, since it’s 2025, we can hook into AI.
We can train models to spot significance patterns. An AI trained on historical archives could flag documents that match patterns of previously important information. “This reads like a whistleblower report that later proved accurate”, it could internally remark.
But AI alone has the same knowledge problem as human archivists, just with better pattern matching. It can spot what looks like past significance, but it can’t predict future importance any better than we can. That’s why the real power is in combining AI with external oracles: let the model flag potentially important documents, then let prediction markets or institutional processes determine if they actually matter.
Aided by markets and signals from authorities, an AI could ingest a user’s whole archive, be equipped with a wallet, funded by others based on its prescience.
What this all means (data agents are coming to Load’s temporary S3 storage layer)
The conditional permanence plugin, offered by the WIP ao-powered data-agent-framework, lets users define storage rules for objects. Rules can be as simple as “if it \<came from this source / has this tag>, make it permanent”. As nuanced as “if this rumour turns out to be true, make this relevant data permanent”, or automated based on a custom-trained model.
Zooming out to see the full upcoming picture, we are building a data-agents framework powered by the ao network. Those data-agents will have several distinct plugins and templates, and the Load S3 temporary data storage layer will make primary use of conditional permanence data agents, forming a “smart data router” between HyperBEAM’s Load S3 and Arweave’s permanent storage, coordinated by ao processes (data agents).
Look out for the framework soon to get started with data agents.