Since the launch of the HyperBEAM-powered temporary storage layer, Load S3 (LS3), we have been discussing the features and advantages of having a temporary to permanent ANS-104 data route that preserves the provenance of the dataitems.
Before LS3 and the offchain-dataitems data protocol, the route for ANS-104 dataitems in the Arweave ecosystem was to directly hit the permanent storage (Permaweb) however, after LS3, ANS-104 dataitems can be stored temporarily then dropped, or temporarily then colocated to Arweave’s Permaweb while preserving the provenance of the dataitem.
In this theory-based article we are going to explain what the new coined “xANS-104” term stands for, LS3 data provenance, data governance and data lineage. Furthermore, we will outline how we made that possible, the current work put into it, and the near-future roadmap.
What is an ANS-104 DataItem?
An ANS-104 DataItem is a self-contained, cryptographically signed data container that wraps binary data with metadata and cryptographic proof of authenticity. It includes the data payload, owner’s public key, tags for metadata, and a signature generated using Arweave’s deep-hash algorithm, with the DataItem ID being the SHA256 digest of this signature.
However, the signing mechanism supports multiple cryptographic standards - the DataItem can be signed with Ethereum secp256k1 keys, Solana Ed25519 keys, or Arweave’s RSA-PSS keys, making it blockchain-dynamic while maintaining cryptographic guarantees of authenticity and immutability regardless of where it’s stored.
What is Load S3 (LS3)?
Load S3 is an object data storage layer (S3 compliant) built on top of HyperBEAM. The way an LS3 node functions is through a micro-modularity protocol built in the HyperBEAM node code.
LS3 leverages the HyperBEAM’s custom devices feature by offering the s3_nif device: an S3 Rust based NIF device that powers the HyperBEAM’s http server and “stores” with S3 storage capabilities. A LS3 node within the Load S3 storage layer consists of running a HyperBEAM node along the s3_nif customization, a natively-spawned local MinIO cluster (the device also offer the feature to connect to external clusters), an ANS-104 dataitems low-level data streaming gateway sidecar (offchain dataitems resolver).
On top of that, the node operator can run a set of utility APIs such as the load-s3-agent which offers an HTTP-based ANS-104 native data upload pipeline for an LS3 node.
Most importantly, the ability to run Turbo-compliant data upload service on top of the LS3 node. That means an LS3 node can be used as a Turbo-compliant bundler endpoint in the official Turbo-SDK, offering OOTB compatibility with the wider Arweave dev stack, and the first ever offchain temporary ANS-104 bundling service (bundler). We released the loaded-turbo-api, a Turbo-compliant upload service, last week. Read more about it here.
Coining xANS-104
xANS-104 (pronounced as eks-AY-en-ESS-wun-oh-FOR) is the term that refers to an ANS-104 dataitem that gets first stored temporarily offchain, on LS3. And its lifecycle is subject to changes based on the following storage flow:
xANS-104 Data Provenance, Lineage and Governance
To simplify it in a mathematical way, xANS-104 is a valid signed dataitem where x stands for t (temporal) first. X can change to ∅ (void sign) or to ∞ (as in permanency).
Mathematical Representation
Given a signed dataitem D, we can represent its state transitions as:
Where:
- t = temporal storage state (Load S3)
- ∅ = void/null state (deleted)
- ∞ = permanent state (Arweave)
State Transition Function
The function f represents the decision point that determines whether the temporally stored dataitem transitions to deletion or permanency.
Data Provenance
The data provenance inherited from the ANS-104 data protocol to LS3 S3 object storage layer means that the S3 objects have a native cryptographic proof maintained through its state transitions that prove the data integrity (hasn’t been tampered with), data origin (who created it) and the timestamp of the creation. Throughout all state transitions, the cryptographic signature σ remains invariant:
A real life example of xANS-104 data provenance benefits in S3 storage is when a user’s load_acc keys get compromised and someone (the malicious user) starts uploading objects (natively are in ANS-104 format) to the victim user’s bucket. In the case of traditional S3 storage, for the client-side the data being served from the S3 bucket is “authentic” because it has been uploaded via a valid (access-controlled access key) – however, in the case of LS3, it’s pretty straightforward to verify the authenticity of the object creator and any data being served from a certain bucket due to ANS-104 object cryptography proofs: has it been signed by xyz address? When was it signed?
As you can see, xANS-104 offers a soft “key leakage protection” that can prove the authenticity of data uploads even after a bucket hack.
Let’s take this to AWS CloudTrail forensics comparison. xANS-104 offers a better CloudTrail to LS3 when it comes to map an object upload action to a real persona. You have a team of devs each with different access keys putting data in a bucket, a dev claims that his access key was leaked and the malicious actor started uploading a malicious object, the real question here: did the dev go rogue or his keys were really leaked? (in the case where only dev-signed data is considered valid)
With xANS-104 it is easy to verify the provenance of the object, without relying on the paid CloudTrail service, and not being limited to 90 days of history.
Data Lineage
In Load S3, data lineage is the actual Extract Transform Load (ETL) pipeline that an S3 object passes through. First, an S3 object is created by a user and stored in a bucket with S3 and ANS-104 associated metadata, then if it’s pushed to Arweave, we have two record in the case of bundling service usage: the API request to the bundling service, and the Arweave gateway API response (dataitem onchain metadata: transaction settling, block inclusion, etc)
The lineage would show the complete journey: Original Data → ANS-104 Signing → S3 Storage → Arweave Migration → Permanent Storage
Data Governance
In LS3 and xANS-104, the data governance consists of the followings:
- ANS-104 data structure, or what we call it, the offchain-dataitems data protocol on S3 storage
- S3 bucket policies, access controls, and lifecycle management
- The decision logic for when/how to migrate from S3 to Arweave
- Retention policies and compliance requirements (the cluster’s SLA)
Therefore, xANS-104 dataitems data governance is a hybrid approach of data structure (ANS-104), compute unit (HyperBEAM micro-modularity via NIFs), and data policies (SLA, bucket policies, etc).
Applied xANS-104
At the time of writing, xANS-104 is already live across several Load Network developed open source softwares, primarily the Load S3 customized HyperBEAM node and the Turbo-compliant upload service.
Load S3 powers the Load Cloud Platform (cloud.load.network), the user’s dashboard to interact with LS3 storage from creating private buckets, uploading private objects (dataitems) and posting the temporary to Arweave or deleting the temporary objects. Read more about LCP V2 release here
Additionally, Load S3 powers the Turbo-compliant ANS-104 upload service, building the ground for the very first offchain temporary Arweave data bundler with OOTB compatibility with the onchain permanent route.
And outside the Load Network ecosystem, xANS-104 is adopted with a serious volume demand from Load’s partners, such as Apus Network with its Digital Twin product, and KYVE Network supporting LS3’s xANS-104 as storage provider.
(pool)
xANS-104 in the near future
We are actively working towards the adoption of xANS-104 DataItems by pushing the usage of the first Arweave-aligned S3 storage layer. xANS-104 forms a great cost-efficient data entry for large, unfiltered and noisy datasets, stored temporarily in S3, with ANS-104’s provenance, and with the built-in native colocation to Arweave’s permanency (data selection).
LS3’s xANS-104 offers several forms with read and write data, either via official (and unofficial) AWS S3 SDKs, Turbo-compliant SDKs, or over raw HTTP. Our next big ship would be completing the loaded-turbo-api roadmap in order to offer a complete Load S3 (HyperBEAM) powered cost-efficient Arweave bundler.
And if you need up to 100 GB free per month, apply to the Load S3 Partner Program.
Load it up!