Accessing blocks from runtime

[nazar-pc]

TL;DR: Is there a plan of recommended way of architecting things if there is a need to access to blocks history in runtime environment?

Background

I'm working on Subspace, which uses proof of archival storage of the blockchain itself as a consensus mechanism.

There is one piece of the puzzle that requires to store some information about post-processed blocks back to the chain (as a consensus log item).

For consensus part it would be beneficial to have that as part of the runtime to make sure everyone follows the same rules and mechanism is actually upgradable instead of spilling over all of the logic into the client. It is relatively costly (erasure coding of a few MiBs of data and building a Merkle Tree over it), but shouldn't be too bad even if implemented in WASM and performance should be fine too.

However, right now there is no way to access anything except the state at the block being handled in the runtime (and even offchain worker).

I'm certain it would be possible to expose an API like this with custom piece of logic (like offchain indexing does to storage while being callable from runtime), but it is way above my head at the moment in terms of complexity.

Maybe there is already an issue/feature request about this (I have searched and didn't find anything relevant).

[burdges]

We cannot fit arbitrary past blocks into polkadot's model because our validators only verify PoV blocks and cannot access other historical blocks. In principle, we could've access to recent blocks under 24 hours old since they still live in the availability store, but after 24 hours parachains become responsible for their own history.

An off-chain worker accessing pact blocks makes more sense of course, but yeah maybe this requires some architectural changes. We'd some storage parachain work which then wound up unsuitable likely afaik due to not being properly designed. #9598

At a high level, I'd assume protocols should split into purely off-chain real work on user data vs the partially on-chain enforcement of incentives or economics. We then hit gaps where our "real work" has weaknesses in the on-chain representation, like trusting attestations by suitably random peers, but the protocol designer should argue this works out fine within the user's threat model

A priori, I'd assume a storage chain cannot both be efficient for regular user data and also secure against availability adversaries in blockchain validation. I think filecoin ran down this zero-knowledge proof rabbit hole because they wanted a tighter compromise between both, but this always seemed silly to me. We do availability in polkadot so others do not have to do so.

In brief, I do not believe users and blockchains have the same threat model, so one should specify the threat model when one starts discussing protocols that do "real work" as opposed to "economics".

[nazar-pc]

Yeah, for the moment I forgot that this would have to be working on the relay chain as well, so current limitations make even more sense now.

[burdges]

We should discuss your specific use case more on matrix because I'm already interested in some related protocols. I spepect expanding the sampler set to include willing users addresses the concerns that likely motivated this post.

[h4x3rotab]

The storage chain pallet builds a accumulator of the past tx. Probably it can be used to feed data while no need to put the past data in PoV?