Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

faster failfast by cancelling all test-items running in parallel when one fails #189

Open
nickrobinson251 opened this issue Sep 18, 2024 · 0 comments
Labels
new feature A feature we would like to add perf About improving the performance of the package

Comments

@nickrobinson251
Copy link
Collaborator

nickrobinson251 commented Sep 18, 2024

When running with nworkers > 1 there may be multiple test-items running when one fails. Currently, when failfast=true , a failure prevents any further testitems starting, but will wait for the currently running ones to complete... which might be as long as timeout * nretries...

Instead, when one test-item fails we should proactively interrupt the other test-items (and kill the workers, and side-step any retry-logic, and log that these tests were interrupted/cancelled). This would allow failfast to report the failure and stop the tests much faster

We need to decide how the testitems that are interrupted (because a different testitem failed) should appear in the printed test summary (and the xml test report) -- perhaps they should appear as skipped? Or simply omit them? Omitting them might be confusing since it'd be inconsistent with the logs, but could still be an okay option... we can't mark them passed or failed (because they didn't complete), so omitting them or marking them skipped are i think our only options.

Follow-up to #133

@nickrobinson251 nickrobinson251 added new feature A feature we would like to add perf About improving the performance of the package labels Sep 18, 2024
@nickrobinson251 nickrobinson251 changed the title failfast should cancel all running test-items when one fails faster failfast by cancelling all test-items running in parallel when one fails Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature A feature we would like to add perf About improving the performance of the package
Projects
None yet
Development

No branches or pull requests

1 participant