Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data Liberation] "Fetch from a different URL" button for failed media downloads, Interactivity API support #2040

Merged
merged 16 commits into from
Dec 4, 2024

Conversation

adamziel
Copy link
Collaborator

@adamziel adamziel commented Dec 1, 2024

Description

Ships:

  • Fetch from a different URL" button that puts the user in control of media download failures
  • Automatically refreshing the progress details on the wp-admin page via the interactivity API.

User-driven error handling

When a remote asset cannot be downloaded, most importers either stop or ignore the error. This PR adds a user interaction to make an explicit decision about what should happen next – do we ignore the missing asset? Do we use another file instead?

CleanShot.2024-12-04.at.00.36.52.mp4

Auto-Refreshing Import Status

This PR also re-expresses the entire data liberation wp-admin page using the interactivity API, and auto-refreshes the progress:

CleanShot.2024-12-04.at.00.25.54.mp4

A part of #1894

Technical overview

User-driven error handling

During the frontloading stage, the WP_Stream_Importer exposes all the frontloaded entities to the API consumer. The consumer then creates a post of type frontloading_placeholder with an initial status awaiting_download for each asset, and updates it with progress information and status (success, failure, skipped) as the import progresses.

The frontloading stage is not finished until all the frontloaded assets have been processed with a non-error outcome. There's a few ways to recover from errors:

  • Retry the download – WP_Stream_Importer now retries the failed assets URL (via WP_Retry_Frontloading_Iterator) before moving on to entities provided by the usual entity source such as a WXR file.
  • Changing the downloaded URL – done by the user on the wp-admin page
  • Choosing to skip the download – done by the user on the wp-admin page

Sometimes we don't want to require user interactions, e.g. when running the importWxr Blueprint step. In those scenarios, we could choose a default error outcome, e.g. "skip failed downloads".

Auto-refreshing admin page

Two fetch() requests running in an infinite loop are:

  • Updating the JavaScript interactivity store with the latest import state from the server
  • Running the next import step

Other changes

  • Adds php_userstreamop_read to the Asyncify list – it crashed the importer in @wp-playground/cli running in bun.

Follow-up work

  • Pretty UI transitions. Right now it's all sudden and jerky. We need progress bars, smooth animations, clear visual causality.
  • Prevent running the same import in two concurrent requests. This is a serial importer not designed for parallelization.
  • Run each import step in a transaction – either it all worked and we can commit the changes and an updated cursor, or it didn't work and we roll back the last step. Ideally we'll never see a scenario where an entity was processed, but a crash happened before storing the updated cursor and the next run reprocesses the same entity.

Testing instructions

  • Go to the data liberation admin page
  • Upload a WXR export file
  • Confirm the import processes automatically and doesn't error out

@adamziel adamziel changed the title [Data Liberation] UI to manage attachments download errors [Data Liberation] Interactive import UI to manage media frontloading errors Dec 3, 2024
@adamziel adamziel marked this pull request as ready for review December 4, 2024 12:24
@adamziel adamziel requested a review from a team as a code owner December 4, 2024 12:24
@adamziel adamziel changed the title [Data Liberation] Interactive import UI to manage media frontloading errors [Data Liberation] "Fetch from a different URL" button for failed media downloads Dec 4, 2024
@adamziel adamziel changed the title [Data Liberation] "Fetch from a different URL" button for failed media downloads [Data Liberation] "Fetch from a different URL" button for failed media downloads, Interactivity API support Dec 4, 2024
@adamziel
Copy link
Collaborator Author

adamziel commented Dec 4, 2024

The CI failure is an unrelated, flaky test. This mostly works so I'll go ahead and merge. Any issues will be addressed in follow-up PRs.

@adamziel adamziel merged commit 6ebd956 into trunk Dec 4, 2024
9 of 10 checks passed
@adamziel adamziel deleted the frontloading-errors branch December 4, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

1 participant