Today marks the v0.2.0 release of the HyperBEAM NIF (Native Implemented Function) that powers Load’s S3 temporary storage layer.
In this blog post, we will go over the updates and features introduced in v0.2.0 of s3_nif - especially the blazing fast offchain ANS-104 data streaming sidecar!
For more context, check our original HyperBEAM S3 announcement, S3 SDK, and cloud vision.
Note: you can upload directly to the offchain-dataitems bucket hosted on Load’s HyperBEAM S3 node using the s3-agent service! Check it out here.
Why dataitems?
Dataitems are a kind of signed bundle-ready data defined for use in Arweave in the ANS-104 spec. Bundlers like Turbo use them, and we’re making them first-class citizens in the Load ecosystem too.
What makes dataitems particularly powerful is their flexibility across the storage lifecycle. You can create dataitems entirely offchain, e.g. in temporary storage, and later post them onchain with the exact same dataitem ID. This means your application logic, references, and indexing remain completely consistent whether the data lives temporarily in S3 or permanently on Arweave.
What is s3_nif
s3_nif is the Native Implemented Function (NIF) that powers the object-storage layer of Load S3, from spawning local S3 MinIO clusters in the same location as the HyperBEAM node, to connect to external clusters, proxying S3 commands from the Rust NIF to the Erlang dev_s3.erl module, and more.
The focus on s3_nif in this release is on how you can access and read the S3 objects, especially the objects serialized as signed ANS-104 dataitems, or as we named it, the offchain-dataitems protocol.
offchain-dataitems protocol in simple terms
The offchain-dataitems protocol defines a set of rules on how to work with S3 objects temporarily stored as signed valid ANS-104 dataitems, how to store them, and how to access them publicly from a HyperBEAM node.
In simple terms, the offchain-dataitems is a protocol that standardises how to work and handle temporarily offchain ANS-104 dataitems at the intersection of HyperBEAM micro-modularity, ANS-104 and S3 object storage.
The rules that defines storing dataitems in a HyperBEAM S3 cluster are pretty straightforward, ANS-104 dataitems should be stored in a public bucket named offchain-dataitems in a dataitems folder, and the ANS-104 object should be named as $dataitem_id.ans104
Now the interesting other part is the different ways to access the offchain dataitems from the public offchain-dataitems bucket.
Different ways and levels of trust on accessing offchain dataitems
The s3_nif in HyperBEAM, alongside the complementary erlang modules, defines several ways to access an offchain dataitem, across different trustlessness, verification tiers and use cases.
1. Full serialized ANS-104 (original) dataitem download
The Rust s3_nif lets the developer set the whitelisted as public, read-only buckets, including by default the offchain-dataitems bucket to be compliant with the protocol. Given that, the s3_dev.erl module offers 2 different ways to access or download the original full ANS-103 object: the first option is to generate a presigned get_object URL, with expiry set by default to 1hr, and the other is the basic S3 get_object handler.
Accessing the full serialized ANS-104 dataitem object usually fits developer’s interest to verify the integrity of a dataitem, or the necessity to have the full dataitem in order to push it from offchain-hyperbeam to onchain-arweave – let’s now go over the normal user’s tailored access points.
2. The Hybrid Gateway
The hybrid gateway is the default way to access small-to-mid sized dataitems, in the same way a user would access a transaction data from an Arweave gateway (e.g. arweave.net). The hybrid gateway is an extension to the hb_gateway_client.erl that makes it possible for the HyperBEAM node to retrieve both onchain (Arweave) and offchain (s3_nif - Load) ANS-104 dataitems, a smart router through HyperBEAM’s Stores configuration.
The hybrid gateway leverage the ar_bundles:deserialize/1 function to add a minimal check on the format validity of the dataitem, however this minimal integrity add a constraints of being able to handle only Arweave (RSA-4096) signed dataitems due to some hardcoded support of ANS-104 signature type in the canonical ar_bundles.erl module. Another very important constraint is the lack of streaming capabilities (yet) from the hybrid gateway, thus the initial “small-to-mid sized dataitems”.
3. The hyper-optimized S3 sidecar
The latest addition to the data accessibility routes is the blazing fast Rust-based low-level ANS-104 deserializer & sidecar gateway. The sidecar leverages bundles-rs ANS-104 SDK to be able to construct any dataitem regardless of the signature type (unlike arbundles:deserialize/1).
Additionally, the hyper-optimization is in the sidecar’s strategy to precalculate where the ANS-104’s data body starts (start-end bytes offsets) without actually deserializing the dataitem, and then range stream it to the sidecar gateway server, directly from the S3 cluster.
Obviously, this approach has its pros and cons, while the pros is clear, latency efficiency, streaming and all-sigs types support, as well as separating the heavy large dataitems streaming from the canonical HyperBEAM http server, the con here is being optimistic regarding the cryptographic integrity of the dataitem - it’s as optimistic as its hybrid gateway.
The S3 sidecar is available on the Load S3 hyperbeam node (s3-node-1) under the gateway.s3-node-1.load.network/resolve/$DATAITEM-ID endpoint.
Check it out streaming a video: https://gateway.s3-node-1.load.network/resolve/nWPNZxVrAR68i1kttoontvJICBCRjws6o-lREpVHTRM
What’s next
We are actively developing the s3_nif and the related HyperBEAM modules stack for a better and more complete Load S3 temporary storage layer, the goal is clear, S3 on ao-hyperbeam. The final goal of our ongoing efforts is the have a decentralized, distributed, incentives and verifiable network of HyperBEAM nodes running the Load S3 micro-modular protocol beside the canonical ao-hyperbeam stack.
Full integrity helper checks are coming next to the S3 streaming sidecar without compromising the work done on latency and performance, stay tuned for the next release!