ao (@aoTheComputer): Going Cacheless

Going Cacheless The blockweave is the only source of truth. Instead of optimizing the cache, correcting the cache, or creating the cache to cache all caches, why not throw it out? It's time to go cacheless. For years, Arweave has relied on a loosely-aligned set of centralized caches for getting data in and out of the weave. Can I have this chunk of data? Yes, it's on the gateway cache. Can I upload this bundle? Yes, let's put it in the bundler cache. For uploads, users needed caches for instant data availability. For downloads, caches showed gateways whether data existed and what metadata it had. With Arweave's long and complicated multi-gateway/multi-bundler history, we ended up with a diverse collection of sources of truth. When it comes to something as singular as truth, multiple sources can only be a bad thing. Before the HyperBEAM migration, accessing Arweave data required trusting a single point of failure. arweave.net — the primary gateway — ran on enterprise-grade hardware that most operators couldn't afford, and it maintained its own internal cache of data it believed miners were holding. When a bundler uploaded chunks to the network but those chunks hadn't yet reached the gateway's cache, reads would fail. The cache was optimistic: it assumed data would propagate, even when it hadn't. Pessimistic Cache The disconnect between the cache used by a gateway, bundler, or indexer vs. the reality of the weave was one of the biggest transparency hurdles for permaweb users - if the data's missing from the index, it looks invisible. Or, in the terrifying inverse case, caches can create situations where the data is only available on the cache and never made it onto the weave. With misbehaving caches, it’s like having multiple versions of the weave, each with their own blind spots and only one with real permanence guarantees. Like most problems distributed systems were built to solve (and problems created by distributed systems), it's about the source of truth. Ironically, the actual source of truth is the weave itself. Truth is what blockchains output, but they also output scaling challenges. When the weave is slower than web2 UX, developers will usually conclude that just one more cache is the silver bullet solution. Instead of optimizing the cache, correcting the cache or adding the cache to cache all caches, why not throw it out? Arweave is going cacheless. The Cacheless Gateway When arweave.net migrated to HyperBEAM, the gateway stack stopped relying on a monolithic enterprise managed service, and transformed into something properly decentralized with a new philosophical approach: Rather than maintaining a local cache of what miners were storing, HyperBEAM-powered gateways route requests directly to Arweave nodes. Data is served verifiably — cryptographic proofs travel with the response, so you can confirm it came from the right place without trusting the gateway. Running a gateway became a lighter lift too. Using miners as the source of truth negated the need for a cache and duplication of storage (terabytes of on-node disk space). The result was a decentralized gateway stack: any HyperBEAM node could serve Arweave data, and routers could act as load balancers. But removing the cache created a new problem. Without a central index, how are you supposed to find anything? The Indexer Gap Relying directly on miners as the source of truth is the endgame for reliability, but not remotely practical for most use cases. Miners will tell you whether they have your exact transaction ID, and what the data is, but what if you don't know the ID? What if -- like with bundles -- that ID is only discoverable by reference to some other unknown parent ID? No cache means relying on the barest of bare metal, with no convenience layer and no index. What if, instead of building an index and storing it in a cache, we had an indexer that lives inside HyperBEAM and is so efficient that you'd believe it was pulling data from a cache when it was actually computing it in real time? This is where the latest improvement to copycat comes in. An upgrade to the HyperBEAM indexing layer shipped in PR #734 added filtered, depth-aware bundle traversal — the ability to dive recursively into nested bundles and find data at any depth, not just the top level. Rather than scanning the entire chain, copycat now supports ID-based indexing with bounded recursion: give it a transaction ID, and it crawls the nested bundle tree to a configurable depth cap, applying owner and tag filters before expanding payloads. Arweave is a dataset with 25 billion rows that takes days to fully index, even for state-of-the-art machines with industrial-grade bandwidth. The indexer upgrades make it possible to index the relevant slice of the network, in-depth, on demand. The practical effect that has the most impact on the network is that AO can still find deeply nested data — L2 bundles inside L1 bundles, data items inside those, without an optimistic cache. The indexer directly queries Arweave nodes, reconstructs the bundle structure on demand, and serves results verifiably. Hardware requirements dropped because the indexing strategy changed, and the source of truth became the actual truth, not something inferred from a desynced cache layer. Cacheless Bundlers The same problem that applied to gateways applies to bundlers (just one level lower). How bundling worked (the old way) ANS-104, the Arweave Network Standard for bundled data, defines a format where multiple independent data items — each with their own owner, data, tags, and signature — are packed into a single Arweave transaction. The bundle format (binary) and version (2.0.0) are declared via tags on the containing transaction. The standard is explicit: "Some form of caching or indexing could be performed by gateways to improve lookup times." This is the architectural gap the old model relied on. Services like Turbo and up.arweave.net sit between users and Arweave: they accept data items, bundle them according to ANS-104, submit the bundle transaction to miners, and maintain their own index of what was uploaded. When a user uploads via Turbo, the response includes { id, owner, dataCaches, fastFinalityIndexes } — Turbo's own infrastructure separate from the Arweave network. The dataCaches and fastFinalityIndexes fields are Turbo's cached view of upload progress, returned immediately, before the bundle is confirmed on-chain. This is the optimistic layer - data available from the bundler's index before Arweave has actually confirmed it. The index is maintained by a service, not by the network. When the cache is returning data that was never actually seeded to the weave, users fail to access their data and permanence guarantees disappear. The same applies to other bundler implementations. Arseeding and similar services each maintain their own index layer, decoupled from the Arweave nodes themselves. The ANS-104 format lets you read a bundle once confirmed, but the existing services added a parallel read path: the bundler's index, queried before the network had confirmed anything. How bundling works now (the new way) The architecture is changing at the node level. Rather than a bundling service maintaining an index that sits between the uploader and the network, data items go direct to Arweave miners. The bundle is still formed, but the service that previously maintained a separate cache – a parallel source of truth with different rules – is being removed from the read path. This turns the mempool itself into a trustless optimistic cache. The mempool is where chunks already live after upload - sitting on an Arweave node that has received payment and expects to be included in the next block. The node's own API is being extended to make that mempool state queryable. The existing GET /chunk endpoint requires a global offset, which only exists after confirmation. The proposed API for unconfirmed chunks would allow nodes to serve bundle contents directly from the mempool, with availability measured in seconds rather than minutes. Copycat, as described earlier, handles the indexing of nested bundles directly from the network. It no longer needs a bundler's index to find data — it queries Arweave nodes and computes the bundle structure on demand. The flow, side by side Old bundling (ANS-104 + external bundler service): New bundling (direct to miners, node-queryable mempool): The old path had a floating service layer - a bundler maintaining its own index, separate from the network. The new path is protocol-level: data items go direct to miners, the mempool is queryable, the network is the index. The Optimistic Placeholder Decentralizing arweave.net meant more than changing one gateway. The ripple effect of decoupling everything in the network from the old centralized convenience layers it relied on forced a fundamental restructure. Replacing core infra with HyperBEAM means ruthlessly removing the centralized workarounds that used to prop Arweave up and adding newer, more efficient features to HyperBEAM with fewer dependencies and margin for error. The optimistic cache was a necessary compromise when gateway hardware was expensive and the indexing layer was shallow, but it created single points of failure. It shaped who could operate infrastructure, and introduced cache desync - a failure mode that had nothing to do with whether the data actually existed on the network. The cacheless model makes that particular failure mode impossible, and brings core network infra into the network where it belongs. Gateways route directly from miners, copycat indexes on demand to arbitrary depth, and the mempool serves as a live layer where unconfirmed data is already queryable. Each piece is lighter than what it replaces, and available to operators all from inside their existing HyperBEAM node. The infrastructure propping up Arweave is no longer a collection of separately maintained, desynced layers. It’s HyperBEAM.