ao (@aoTheComputer): Arweave is a Realtime Database

Arweave is a Realtime Database HyperBEAM connects native mempool indexing with published ArLMDB databases. It’s live today for GraphQL, but could be live tomorrow for any dataset. Optimistic is old-world thinking. Systems that achieve speed and scale purely by assuming the best case scenario inherently rely on trust and are usually patching around a fundamental weakness in the layer below. ), it made old failure modes impossible – all while making the permaweb experience immediate as well as trustless.Many permaweb services until now had to be optimistic to work around Arweave block times. Arweave is built to be uncompromisingly robust – thinking about timelines in years and decades, with minutes as the least important scale. This focus is what gives the permaweb strong foundations. If however, in order to help the user see results quickly, an indexer makes an upload discoverable because a bundler says it exists, this speed comes with a nasty trust requirement. Ingoing cacheless, we looked at how a decentralized AO-powered network of bundlers would throw out the secondary source of ‘truth’ and just read the chain directly. Not only did this make hardware requirements low (a full bundlercan live on your laptop or mobile phone With no optimistic cache, it’s impossible for the cache to lie – or as more often happens – simply err. Instead of optimism, the permaweb’s infrastructure on AO is embracing realism – and realtime. and recent work on the HyperBEAM query layer, Arweave is now a real-time database.WithArLMDB , discovery is now near-instant without the need for a centralized cache or a poisonable, trusted source of truth.FrombundlerstoBazar is the new way the app layer discovers what’s on the weave. Applications can find newly uploaded data through Arweave itself, before the next block, alongside the permanent history already stored on the network. Decentralized bundlers now have a query layer that can keep up with their uploads, with no tradeoff between UX and decentralization.RealtimeGraphQL Realtime GraphQL Over The Whole Database ArLMDB unlocked rapid, dependencyless queries over all historical data, but what about data that isn’t in the 622 GiB onweave LMDB snapshot yet – data that is only (milli)seconds old? A realtime database needs to query its newest records alongside its history. Previously, immediate GraphQL discovery depended on trusted, unverified data feeds from selected, permissioned bundlers. Index operators would take unverified streams of data item headers (critically, lacking the body that would make them verifiable) and serve those results as if they had been mined into blocks. Great UX, but terrible trust assumption. This also practically made it harder for others to run their own bundlers because their service would appear slower and less attractive than existing services unless both the main gateways and GraphQL indexes agreed to ‘peer’ with the bundler and display its uploads ‘optimistically’. The first version of ArLMDB, offering in-browser GraphQL made published history independently queryable (with proofs of completeness against the published index!), which was already a major improvement that offered a decentralized off-ramp from the centralized indexes. The next step was to make that database current – without adding in dependencies that negated those existing decentralization wins. HyperBEAM’s realtime query layer allows simultaneous computing over published ArLMDB snapshots on Arweave, locally indexed confirmed transactions, and even paid, uploaded and verified, yet still pending records in one matching operation. Historical data stays in reusable, transparent databases on Arweave. New data enters a live index through native network ingestion. GraphQL filters can span both, including uploads that have not yet entered a block. Every single realtime result comes with a hashpath and attestation from the node. Challengeable and verifiable by all. What else gets faster? Pagination retrieves every record, using each endpoint’s own cursor. Compare complete result sets and three-feed requests. The new realtime version connects ingestion to indexing at the point a message is stored. ~copycat@1.0 retrieves pending transaction headers and bundle contents from Arweave nodes. It does this after the Arweave node has already: Verified that the uploader can pay for the upload. Verified that the transaction and bundled data are signed correctly. Verified that the chunks of data are actually seeded to the Arweave network and will be available long-term. No ‘optimism’ required. Just realism. This work also adds a programmable cache-write hook that passes messages – from both Arweave’s mempools and ingested blocks – to devices of the node’s choosing. By default, each new piece of data is sent to match@1.0 which indexes their fields under their signed identities, making them queryable on the node. That supplies GraphQL with searchable records derived directly from network data, including the individual items inside a bundle. Verification before confirmation The old optimistic path made data searchable through centralized authorized bundler-to-indexer notifications. A signed data item could establish who created it and whether its contents had changed. That signature alone could not establish that the item had been included in an Arweave upload, that the parent transaction met the network’s payment rules, or that its bytes were retrievable from the network. Immediate visibility depended on the bundler’s ‘optimistic’ integration with the indexer. The parent transaction’s signature commits to its data root, data size and fee. Arweave’s transaction validator checks the signature, the required fee and the payer’s ability to fund it against the current chain state. For bundled data, that parent transaction carries the storage fee for the items inside it. Payment is therefore attached to the same signed transaction that commits to the uploaded bytes. The native validation code performs these checks together. Chunk proofs connect the retrieved bytes to that signed commitment. Each chunk’s Merkle path can be checked against the parent transaction’s data root. Authenticating the bundle’s contents establishes which items it contains; each data item’s own signature authenticates its author, fields and payload. An independent reader can verify the chain from the signed item, through the bundle bytes, to the native transaction. These checks establish authenticity and current transaction validity before confirmation. Block inclusion then establishes the transaction’s place in permanent history and settles its fee. Discovery can use the earlier evidence without treating pending payment as already settled. New uploads become queryable sooner with realtime GraphQL Time from first mempool appearance to the first query result, comparing realtime and legacy GraphQL. Measurement details: Three 1 KiB synthetic items were uploaded through a bundler, 90 seconds apart. Each item was in a separate parent bundle. Parent signatures, bundle data roots, item signatures and payload hashes were verified. Zero is the first completed response from tip-2.arweave.xyz’s native pending-transaction list containing that parent bundle. Timing ends at the first successful GraphQL response containing the uploaded item, found through the same unique-tag query on both services. Both times use one client’s monotonic clock. Each observer polled independently at a target interval of 500 ms. Times include request latency and polling delay; they describe this node’s observed mempool, not the earliest arrival anywhere on the network. Current-endpoint observation ranges were approximately 0.98–2.05 s, −0.02–1.04 s and 0.04–1.13 s. The range crossing zero means the events could occur within the same polling interval. The roughly 60-second wait before mempool detection is consistent with the bundler’s configured 60-second dispatch timer. It also includes dispatch and propagation. It is excluded from this chart’s starting point. Current GraphQL first returned all three items before their parents were confirmed. Later successful queries temporarily omitted those items 17, 16 and 30 times; legacy GraphQL had no observed disappearances. These three samples establish first discovery in this run, not typical latency, continuous availability or an internal cache policy. A private notification no longer has to substitute for an upload available through Arweave. Independent indexers can retrieve the same underlying records, and readers can check their cryptographic commitments. Realtime discovery follows from processing verifiable network data in the time between blocks. Extending the published database The initial seed was a snapshot – a 622 GiB LMDB file – but publishing a database on Arweave doesn’t freeze it in a state where it gets progressively more stale. New records extend the published history through a live index, and GraphQL queries both together. The original LMDB snapshot remains immutable while the queryable database keeps growing. HyperBEAM maintains the newer records locally, including pending uploads. Its matching engine combines matches from that live index and the published snapshots into one ordered result, with filters and pagination spanning both. Historical queries retain ArLMDB’s ability to retrieve only the database pages they need. An operator can therefore keep the database current without downloading or rebuilding its entire history. As more records accumulate, that additional indexing work can be published in further snapshots. Other nodes can reuse those snapshots alongside the original database and continue indexing new arrivals themselves. History becomes a shared, reusable foundation that independent nodes can extend in realtime. Not just GraphQL: Any dataset. This machinery also extends beyond the Arweave transaction index. ArLMDB supports general LMDB layouts, and the cache-write hook makes the indexing rules programmable. A publisher can define an index over application records, publish its historical database and use ongoing ingestion to index new records under the same rules. Other applications can query that published database directly or maintain their own live view. Applications can share a database without sharing a database operator. The published index becomes part of the permanent data available for others to build on, while independent nodes extend it with realtime discovery. Arweave was always technically a database – queryable by tags and metadata – but the index serving those queries usually lived outside Arweave, and storing a dataset there did not make its contents efficiently queryable. Applications still needed a hosted query service, or their own downloaded and indexed copy of the data. ArLMDB puts that index on the weave, in a format clients can traverse a few pages at a time. Realtime indexing keeps that published history at the tip as new data arrives. This continues the arc of Arweave for AO, and everything else. AO replaces centralized services around Arweave; Arweave supplies permanent storage and shared history for AO. Each dependency removed produces machinery other applications can reuse. The database powering Arweave discovery is one application – the same storage and query model can serve any dataset, in realtime. The permaweb needed a decentralized query layer of its own data, but it got something much broader: a way to make any database permanently openly available, trustlessly queryable, as well as current. Arweave introspecting its own history is one dataset, but the same architecture is open to everyone. Read on the Permaweb: https://ao.arweave.net/#/blog/arweave-is-a-realtime-database