KYVE Network, the de facto web3 data lake and blockchain infrastructure for ledger data validation and storage, has recently added support for Load S3 in its Storage Providers set.
Load S3’s native support for ANS-104 DataItems, flexible integration with Arweave, and cost efficiency brings KYVE users a new way to store data. In this blog post, we’ll explore what this integration unlocks and how it works.
What is KYVE?
KYVE is a Layer 1 blockchain network built on the Cosmos SDK that provides decentralized infrastructure for validating, storing, and retrieving on-chain and off-chain data.
The network operates through two distinct layers: a Proof of Stake consensus layer that secures the blockchain, and a protocol layer where validators collect data from various sources, bundle it, upload it to decentralized storage solutions like Arweave, and validate its accuracy.
This architecture enables KYVE to function as a Web3 data lake, offering developers and data engineers trustless access to permanently stored, validated blockchain data across multiple partner networks including Polkadot, Solana, Avalanche, and Cosmos. The network utilizes its native $KYVE token for staking, governance, and funding data pools.
What is Load S3?
Load S3 is an object data storage layer (S3 compliant) built on top of HyperBEAM with ANS-104 default data format for the S3 objects. The way an LS3 node functions is through a micro-modularity protocol built in the HyperBEAM node code.
LS3 leverages HyperBEAM’s custom device feature by offering the s3_nif device: an S3 Rust based NIF device that powers the HyperBEAM’s http server and “stores” with S3 storage capabilities. A LS3 node within the Load S3 storage layer consists of running a HyperBEAM node with s3_nif customization, a natively-spawned local MinIO cluster (or optionally, connect to any external cluster), and an ANS-104 dataitems low-level data streaming gateway sidecar (offchain dataitems resolver).
The Rationale: why Load S3?
The main reasons to integrate Load S3 in KYVE’s storage clients is the integration between Load S3 and Arweave in terms of data formatting – xANS-104. Load S3 differentiates in terms of cost efficiency and data retention, but shares the same DataItem encoding, provenance, and ID.
As described in the xANS-104 research article, it’s a DataItem with a state transition: it starts as tANS-104 DataItem (temporary) then could be dropped (void) or posted to Arweave (pANS-104 - permanent). Load S3’s dynamic DataItem state gives KYVE storage providers a totally new flexible paradigm.
With Load S3 SP, KYVE nodes today can use an Arweave-like temporary storage client, cheaper than AWS S3, and with a native data pipeline to Arweave’s permanency if needed. What does this mean in practice?
Let’s take a real world example: you are a KYVE operator running a pool for XYZ chain. You want to start archiving and validating this chain’s ledger data, however you:
- Don’t have enough AR to cover the one time upfront storage costs
- Want to hedge holding AR with its USD price fluctuations
- Don’t want to store permanently all the data ingress (data noise filtering)
- Want to crowdfund the upfront Arweave storage costs
- Want to have the provenance of the data in time
- Want S3 compatibility and Arweave compatibility
With temporary dataitems that can be pushed 1:1 to Arweave at any time, you can effectively submit data to permanent storage at your own pace rather than with one upfront payment, retaining provenance and data IDs. This gives teams flexibility over what data to commit and when to pay the full price for permanence.
To learn more about xANS-104’s data provenance, lineage and governance, check out this research article by Decent Land Labs.
Load S3 in Action
PR #177 in kyvejs repository has added the Load S3 as storage provider under ID 5 of the storage providers factory. The integration makes use of the load-s3-agent to store signed ANS-104 dataitems in HyperBEAM Load S3, over HTTP.
To retrieve the offchain temporary ANS-104 dataitems, the s3 agent resolver can be used to download the dataitem’s data. An example is here.
Additionally, the load-s3-agent can be used to colocate the offchain LS3 dataitem to Arweave, onchain. Check out the API docs to get started.
As a PoC, the KYVE’s Korellia pool #161 is live with Load S3 as storage provider.
The load-s3-agent inherits the SLA offered by LoadHyperBEAM S3 cluster SLA. If you are a KYVE node operator and you are interested in using Load S3 storage, please reach out to us on X or Telegram or apply directly to the free 100GB S3 Partner Program.
Wrapping up
At Load Network we continue the work on Load S3’s adoption, the first Arweave adjacent temporary storage layer – if you are interested in HyperBEAM-powered S3 storage, the xANS-104 data paradigm, cost efficiency and future Arweave data-anchoring, Load S3 is here for you.