Implement DELETE /api/v1/crates/:crate_id endpoint

[Turbo87]

This PR implements the first step of #9352: a new DELETE /api/v1/crates/:crate_id endpoint that can be used by crate owners to delete their crate under certain conditions (see linked RFC)

Unresolved Questions:

• Should names of deleted crates be blocked so that they can't be re-used? – This PR does not implement any blocking for now, similar to how it already works for our crates-admin tool.

Related:

• Tracking Issue for "Crate Deletions" #9352
• crates.io: Crate Deletions rfcs#3660
• Extract DeleteCrateFromStorage background job #9549
• Add deleted_crates database table #9912
• Add minimum version tracking to deleted crates #10016

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.57%. Comparing base (3669ad6) to head (925e41d).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9904      +/-   ##
==========================================
+ Coverage   89.36%   89.57%   +0.21%     
==========================================
  Files         294      295       +1     
  Lines       31314    31690     +376     
==========================================
+ Hits        27983    28387     +404     
+ Misses       3331     3303      -28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[LawnGnome]

Going to review momentarily, but I wanted to wrestle with the open question first:

Should names of deleted crates be blocked so that they can't be re-used? – This PR does not implement any blocking for now, similar to how it already works for our crates-admin tool.

The downside in punting an unresolved question in an RFC is that eventually you have to resolve it. 😆

That might be a good one to get outside opinions on. NPM you already noted in the RFC. I don't know what PyPI does off hand — I can't find any reference to blocking names after deletion, but I guess we could confirm that with @di et al. Might be one we could take to a relevant OpenSSF channel.

My feeling is probably very slightly towards allowing re-use: at first blush, lockfiles should prevent obvious supply chain attacks, and it opens up the useful use case of a crate author wanting to free a crate name back up that they no longer intend to use. If we go down the road of blocking versions, we'll probably have to do it by blocking any version that is semver-compatible with any version that was previously published, which feels challenging both implement and (more importantly) explain.

@walterhpearce, you probably have feelings here too?

[LawnGnome]

LGTM. I've got a few nits (mostly around handling of the download limit constant), but the core logic to check the RFC conditions and validate that the user has the correct permissions looks solid to me.

🫥

[LawnGnome]

Given that these rules are public anyway, I wonder if we shouldn't short circuit, but instead report all the conditions that are failing (if there's more than one).

[Turbo87]

I generally agree. it's somewhat related to #3054, which requests something similar for the publish endpoint. I'm not aware of any ergonomic Error implementations for this use case though. Do you have something in mind? I played around with a custom implementation in the past, but it didn't quite compose as well as I had hoped.

[LawnGnome]

The really quick and dirty solution here would be something private to this module that represents the error states as enum values, puts them together into a container, and then just spits out a string that can be given to custom().

It feels like we could extend CustomApiError to support multiple detail messages easily enough, though, if we want something more reusable. Something like this (except with tests, and some renaming of types).

Not too bothered either way, but it feels like we should do at least one of them.

[eth3lbert]

The implementation looks correct to me 👍

I've left a note and a small suggestion, but they're not blockers.

[di]

I don't know what PyPI does off hand — I can't find any reference to blocking names after deletion, but I guess we could confirm that with @di et al.

PyPI sort of does a mix: we immediately release the project name after deletion, but we prevent the reuse of filenames, which has the effect of requiring a new release if someone were to re-use the project name. We also have controls which prevent the creation of new project names that are too similar with existing projects, so sometimes there is an effect of the project name becoming immediately unavailable after deletion due to this.

What we've found is that a common use case for our users is to make a bunch of releases to a project, delete the project to "wipe the slate clean", and then immediately attempt to make a "proper" release. It's also a common use case for a user to delete an unused or unmaintained project in order to return it to the pool of available names for another user to use.

Due to the limits on filename reuse and our general recommendation to enable hash-checking, the risk of a resurrection attack is relatively low.

All that said, we are currently having a discussion about changing this policy, which you might find relevant: https://discuss.python.org/t/pep-763-limiting-deletions-on-pypi/69487

Might be one we could take to a relevant OpenSSF channel.

Please feel free to take the discussion to the #wg-securing-software-repos channel if you want!

[Turbo87]

I've rebased this on top of #9912 now and integrated the INSERT INTO deleted_crates query into the proposed endpoint.

[Turbo87]

for cross-referencing purposes:

we discussed how we want to treat republishing previously existing versions of crates that were deleted in the last crates.io team meeting. the summary is available at #10016 (comment)