Automatic backup failed to upload to cloud exactly 45 minutes after backup was created, then filled logs with job exceptions

[Petro31]

The problem

I was trying out the automatic cloud backups last night, set up for 4:45 am. Backup completed but was unable to upload to cloud.

The initial error was a 403 error:

2024-12-27 05:30:17.962 ERROR (MainThread) [homeassistant.components.backup] Error during backup upload - Failed to upload backup
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/cloud/backup.py", line 182, in async_upload_backup
    upload_status.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/aiohttp/client_reqrep.py", line 1161, in raise_for_status
    raise ClientResponseError(
    ...<5 lines>...
    )
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url='https://nc-user-files-weur.161122e6f55854edd16b585ba68d4972.r2.cloudflarestorage.com/5f4d45cc-76dc-445f-8441-9f110d2eb229/backup/cb8f11775bbe42878839a021bdcf890e.tar?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=10bd110cf51f2ae32893d33bbb042449/20241227/auto/s3/aws4_request&X-Amz-Date=20241227T094614Z&X-Amz-Expires=60&X-Amz-Signature=4266f0e810aacbd1f858a40769406f95a0205b6436cc21c7ad2f5c62ab7f9125&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-meta-b64json%3Bx-amz-meta-storage-type&x-id=PutObject'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/cloud/backup.py", line 184, in async_upload_backup
    raise BackupAgentError("Failed to upload backup") from err
homeassistant.components.backup.agent.BackupAgentError: Failed to upload backup

Afterwards, this error spammed the logs every microsecond until 05:31:20.827:

2024-12-27 05:30:18.004 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback None() (None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

ending with this error

2024-12-27 05:31:20.827 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback _UnixReadPipeTransport._add_reader() (None)
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/asyncio/unix_events.py", line 532, in _add_reader
    self._loop._add_reader(fd, callback)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/asyncio/selector_events.py", line 283, in _add_reader
    self._selector.modify(fd, mask | selectors.EVENT_READ,
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          (handle, writer))
                          ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/selectors.py", line 372, in modify
    self._selector.modify(key.fd, selector_events)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

What version of Home Assistant Core has the issue?

core-2025.1.0b2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @home-assistant/cloud, mind taking a look at this issue as it has been labeled with an integration (cloud) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of cloud can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign cloud Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

cloud documentation
cloud source
_{^{(message by IssueLinks)}}

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (backup) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of backup can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign backup Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

backup documentation
backup source
_{^{(message by IssueLinks)}}