Benchmarking initial DB data

[ggwpez]

I am trying to find out which data to use for benchmarking the Databases (Rocks and Parity).
Please take a look and leave some feedback and/or ideas.
Ideally we want something that works in Substrate and any specific chain.

A. Historical Data

Normally I would opt for this, but maybe there are some obvious reasons against it, which I want to clear out with this discussion.

Polkadot:
Load up a validator DB and use that.

Substrate:
Load a Polkadot validator DB. Since Substrate does not have the Polkadot chain spec, the values would not be decodeable.
Should still work for populating it.

pros:

• Real results which should come close to historical weights
• Can be easily re-run every 1M blocks to update the weights

cons:

• Needs access to a synced DB for accurate results
• Harder re-reproducibility because since you need an exact DB snapshot

B. Modelled Fake Data

Build a statistical model from the data at block #x from Polkadot and then populate a DB with it.

pros:

• Should be easy to re-produce
• Probably faster than A.

cons:

• Results cannot be fully trusted unless compared it to historic weights
• Still needs a DB snapshot to calculate the statistical model

C. Pure Fake Data

Just write 10M or whatever random key/values into the DB and use that.
Key and Value sizes could be set to blake2_concat128(AccountId) and AccountInfo.

pros:

• Easiest to implement and maintain
• Very simple approach (less room for errors)

cons:

• Results could be completely bogus if not verified with historic weights
• Harder to fine-tune to accurately represent actual weights

[arkpar]

Substrate:
Load a Polkadot validator DB. Since Substrate does not have the Polkadot chain spec, the values would not be decodeable.
Should still work for populating it.

I suppose it only makes sense to benchmark and provide weights for real network. The final figure largely depends on the already existing in the network. Abstract "substrate" benchmark is not important really.

With approach A it is harder to extrapolate into the future. We want to compute weight not for the current state, but for the state in e.g. a year. You could extrapolate with a coefficient, but it is hard to get right. I guess the slowdown will be O(log(n)) for rocksdb at least, as long as the data still fits into RAM, which should be the case for polkadot and kusama for a couple of years. However if some blockchain is expected to grow by a 100Gb state within a year, the slowdown will be dramatic at some point.

So with A the most solid approach will be to update the weight measurements more frequently. Maybe automate it as part of the runtime release process.

[ggwpez]

Thanks for the reply!

I suppose it only makes sense to benchmark and provide weights for real network. The final figure largely depends on the already existing in the network. Abstract "substrate" benchmark is not important really.

True... so I could maybe even use approach C as baseline for Substrate and A for real chains.

So with A the most solid approach will be to update the weight measurements more frequently. Maybe automate it as part of the runtime release process.

We could add x% on top of it to account for DB growth and update regularly.
But getting projections for the future would be out of reach. Maybe we can use approach B for future estimations if we really want to.