can't restore cause system.replicas entries

[karthik-thiyagarajan]

I'm restoring the schema in each and every replica and shard (3 shard 3 replica - total 9 pods) and restoring the data in 0th replica of every shard. I'm trying to do only one table restoration and it fails with the below error.

I checked the clickhouse-backup pod logs and i see the same error as given below.

Any help would be appreciated.

Restoration failure reason - 2024-10-08 08:08:21 2024-10-08 08:08:26 error one of restoreDataRegular go-routine return error: can't attach data parts for table 'db.stats_local': db.ap_stats_local can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=14648, log_max_index=14647, absolute_delay=0, queue_size=0
2024-10-08 08:18:24 2024-10-08 08:18:29 error one of restoreDataRegular go-routine return error: can't attach data parts for table 'db.stats_local': db.stats_local can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=14648, log_max_index=14647, absolute_delay=0, queue_size=0
jenkins@ip-10-58-130-135:/home/ubuntu$

[Slach]

'db.stats_local': db.stats_local can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=14648, log_max_index=14647, absolute_delay=0, queue_size=0

It means other replicas continue to write

Look to https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#restore

for one tables you need to execute in all replicas
restore_remote --schema --table=db.table backup-name
and then
run
restore_remote --data --table=db.table backup-name in first replica in each shard

[gavishdewangan]

1. it is delete local

2. this workaround worked but we are facing another error now
   Restoration failure reason - 2024-10-10 04:07:41 2024-10-10 04:07:47 error one of restoreDataRegular go-routine return error: can't attach data parts for table 'db.stats_local': code: 242, message: Table is in readonly mode (replica path: /clickhouse/tables/cnxcluster-01/0/db/stats/replicas/chi-clickhouse-server-cnxcluster-01-0-0)

3. We are executing --data for first replica only

4. ReplicatedMergeTree('/clickhouse/tables/{cluster}/{shard}/db/stats', '{replica}')

[Slach]

Looks like you doing something wrong outside clickhouse-backup
check grep -C 50 -E "read.{0,1}only" /var/log/clickhouse-server/clickhouse-server.log

[Slach]

could share full logs from clickhouse-backup container, all replicas in one shard for restore command?

[karthik-thiyagarajan]

We dropped the table completely and then added the env flag "check_replicas_before_attach" while doing the restore and it seems to be working now. Can you explain what does this flag do and why was it not working previously and now it s working ? We have stopped all the writers definitely and I'm pretty sure this restore was working some 6 months ago though not sure what version we were using at that time but this seems to be happening from 2.5.20 and above onwards including the current version 2.6.1

[Slach]

This flag check system.replicas state and if it show us "replication in progress" return error

during restore --schema
in all replicas clickhouse-backup should execute DROP TABLE ... SYNC in all replicas
add log_level: debug in your backup config and check it in clickhouse-backup log