CloudFlare Node Backup uploads fail

[blakebyrnes]

Hi, thanks for publishing this project. It's been super helpful to learn from to deploy our own chain.

The node backups role does not work on Cloudflare r2. It fails at the upload of files.txt. Oddly, it had worked to synchronize the db folder fine. Researching this, it seemed to be a common s3 compatibility issue.

<6>INFO  : full/000930.sst: Copied (new)
<6>INFO  : 
Transferred:      357.202 MiB / 357.202 MiB, 100%, 69.344 MiB/s, ETA 0s
Transferred:          123 / 123, 100%
Elapsed time:         6.0s

+ sort
+ rclone lsf -R --fast-list --files-only R2backups:argon-testnet-backup/argon-testnet/20241209-192549
+ set +x
+ rclone copyto -v /tmp/remote-files-argon-testnet-20241209-192549 R2backups:argon-testnet-backup/argon-testnet/20241209-192549/files.txt
<3>ERROR : remote-files-argon-testnet-20241209-192549: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<3>ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<3>ERROR : remote-files-argon-testnet-20241209-192549: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<3>ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<3>ERROR : remote-files-argon-testnet-20241209-192549: Failed to copy: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<3>ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Access Denied
        status code: 403, request id: , host id: 
<6>INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.5s

Failed to copyto: AccessDenied: Access Denied
        status code: 403, request id: , host id: 

It fails if you upload directly as well rclone copyto -v /tmp/remote-files-argon-testnet-20241209-192549 R2backups:argon-testnet-backup/argon-testnet/20241209-192549/files.txt

Single file uploads will apparently only work if you add --s3-no-check-bucket to the upload (this is using rclone 1.63.1).

As an aside, the rclone specific features seem quite minimal. Why did you choose to configure this role to have specific configurations for Cloudflare but not other s3 providers? It seems like it would only be a matter of configuring rclone with the right provider, which could easily be left to the ansible user.

[blakebyrnes]

You can apparently also add a configuration entry no_check_bucket = true, which is referenced in many of the tickets rclone/rclone#5271
https://forum.rclone.org/t/cannot-copy-file-to-cloudflare-r2/43702/7

[nprt]

Hi @blakebyrnes, thanks for reaching out!

Thanks for raising this issue. We are planning on revamping the current configuration in the near future. There's an issue opened for it in #82.

In the meantime, I'll try to take a look at this asap. Sorry for the late reply.

[blakebyrnes]

No worries! I have a fork where I fixed this and added general s3 compatibility if you're interested. https://github.com/argonprotocol/ansible-polkadot

[nprt]

No worries! I have a fork where I fixed this and added general s3 compatibility if you're interested. https://github.com/argonprotocol/ansible-polkadot

Awesome, this is great, thanks!

Would you mind creating a PR from it? I'd love to give it a closer look and merge it.