Release recovery¶
What to do when a release goes wrong — a publish fails, a wheel build fails, a step after the publish fails. Companion to Releasing.
The one rule¶
Published versions are immutable. Once a file exists on PyPI or a version exists on npm, you can never overwrite it. So every recovery is one of two moves:
- Finish the same version — fill the gaps for a version whose tag exists.
- Release a new version — required whenever the content must change. You cannot "fix and
re-publish
0.3.1"; you release0.3.2.
What ship does on failure¶
ship release runs, in order: write the version → before-build → build → tests → publish →
commit + tag + push → GitHub release → <version>+dev back. Any failure before the publish is
reverted: the working tree is restored, nothing was pushed, no tag exists. Fix the cause and
dispatch again; ship computes the same version from the unchanged last tag.
A failure after the publish is not reverted (the registry already has the version). ship
prints the remaining commands — the commit, the tag, the push, the release — so they can be run
by hand from a checkout of main. Because the publish happens before the tag is pushed, the
tag lagging the registry is the one state to watch for: ship deps check and the next release
both read tags, so finish the tag and push before anything else.
Before a release, ship asks the registry that the next version does not exist yet. If it does (a half-finished release), finish that version rather than releasing again.
Failure playbook¶
| Where it failed | State | Do this |
|---|---|---|
CI gate (the commit's ci.yml is red, missing, or timed out) |
Nothing done | Fix CI and dispatch again, or dispatch with skip_ci_check=true if you accept the risk. |
before-build, build, or tests |
Nothing published; tree restored | Fix the cause, dispatch again (same version). |
| publish (PyPI or npm rejected the upload) | Possibly partial | A PyPI 403: PYPI_API_TOKEN is missing, revoked or not scoped to that project (gh secret list). An npm error: NPM_TOKEN needs write on the whole @habemus-papadum scope; provenance needs a public repository and id-token: write, which the caller grants. If nothing landed, dispatch again. If files landed, treat it as "after the publish". |
after the publish (commit, tag, push, GitHub release, +dev write-back) |
Registry has the version; tag or release missing | Run the commands ship printed from a clean checkout of main; at minimum commit the version, tag <package>-v<version>, push commit and tag together, gh release create <tag> --generate-notes, then write <version>+dev back and push. |
build-nvenc-sdk-wheel.yml / build-vtenc-wheel.yml (a build = false package: ship tagged and released, the wheel build or its publish failed) |
Tag and GitHub release exist; wheels missing | Fix the cause if it is the runner's, then re-run the workflow for the tag: gh workflow run build-nvenc-sdk-wheel.yml -f ref=habemus-papadum-nvenc-v<version> (or re-run the failed run). uv publish --check-url skips files PyPI already has and gh release upload --clobber is idempotent. Do not release again for a build failure; the version is cut. |
build-wheels dispatch job in release.yml did not run or failed |
Tag and release exist; no wheel run | Dispatch the wheel workflow by hand with ref=<tag> as above. |
| Need to change shipped content | Version burned | Release a new version (pick the bump). Do not attempt to overwrite. Avoid npm unpublish (72-hour window, discouraged, breaks anyone who installed it). |
Checking state¶
uv run ship packages # manifest version and last tag per package
uv run ship deps check # every manifest must say <last tag>+dev
git ls-remote --tags origin 'habemus-papadum-rfb-v*'
gh release view habemus-papadum-nvenc-v<version>
Auth notes (token publishing)¶
CI currently publishes with token secrets (not OIDC — the maintainer is locked out of PyPI
2FA; see the migration proposal). The same
secrets serve the wheel workflows' publish jobs. There is no local publish path.