Release Checklist

This is the release checklist (when creating a new tagged minor revision):

Note: the EC release build environment is on ci.betrusted.io and relies on the toolchain at the following path:

RISCV_TOOLS=/tools/xpack-riscv-none-elf-gcc-12.2.0-3
FOMU_TOOLS=/opt/fomu-toolchain-updated
export PATH=$RISCV_TOOLS/bin:$FOMU_TOOLS/bin:$PATH

Prepping the release

• Roughly 0.5 hrs (+1.0 hrs if SoC has an update)
• No warnings in hosted, renode, or hardware builds
• Trim buildable apps (currently mtxchat is in CI but should not be in release)
• All CI tests passing
• Update tag on EC, rebuild, tag & push (on ci.betrusted.io in bunnie/code/betrusted-ec, see note about build env above)
  • git tag -a "v0.9.x" -m "commit message" Tag to release version
  • betrusted_ec.py --revision=pvt to generate an FPGA image with the tag in it
  • cargo xtask copy-precursors to absorb the FPGA image
  • git commit -a to commit the updated FPGA image to the tree
  • do not delete the current tag and re-tag -- the parser expects a +n (in this case +1) at the end, a clean tag omits that and version reporting breaks.
  • cargo xtask hw-image to test the final composition
  • git push --tags to send tags to the cloud
  • git push origin to update the actual repo
  • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
  • Mark build as "keep build forever" for both betrusted-ec and betrusted-ec-fw
• Update tag on SoC, rebuild, tag & push
  • git tag -a "v0.9.x" -m "commit message" Tag to release version
  • .\betrusted_soc.py -e .\dummy.nky Generate a fresh SoC image that contains the tag
  • git push --tags to send the new tag to the cloud
  • git push origin to push the actual repo
  • Check that CI triggered. It may not trigger if it's just a tag update and no code changes.
  • Check slack histograms on release build for any suspicious timing closures (max and min)
  • Check the candidate SoC file across 3 hardware test cases: nom, corner-0.95 and corner-1.0 (requires swapping out the mainboard)
  • Mark the build artifacts as "keep forever" for betrusted-soc
• Absorb SoC artifact into xous-core; rebuild, tag & push
  • cd precursors && ./update_soc_pvt.sh - absorb the production SoC image into the Xous tree. Note: this does not absorb any SVD changes, this is assumed to have been done previously to the release process!
  • Push latest SoC ref to crates.io:
    • edit utralib to reflect the change in git rev
    • use publish.py to bump version numbers & publish new revs
    • edit verifier.rs/main.rs in xtask to absorb latest version numbers
  • cargo xtask app-image - confirm everything builds
  • git commit -a to commit the updated SoC precursors to the tree
  • git tag -a "v0.9.x" -m "commit message" Tag to release version
  • git push --tags then git push to send new tag to the cloud and kick off the final CI

Testing the release

Roughly 2-3 hours if all goes well. Note that it may make sense to push the release prior to all testing being done, because the backup/restore tests rely on finding the release kernel in the release location.

• Simple proof-of-run in hosted mode
  • cargo xtask run vault
• Simple proof-of-run with Japanese and Chinese locale in hosted mode
  • cargo xtask run vault --feature locales/lang-[ja,zh]
• Simple proof-of-run in renode
  • cargo xtask renode-image.
  • renode emulation/xous-release.resc
• Update from previous version to current version using production update mechanism: python3 -m precursorupdater -b
• Functionality tests not covered by CI:
  • Check audio (now done with OQC test)
  • Check vibe (now done with OQC test)
  • Test net server, net tcpget, time, and ping
  • Run suspend/resume loop for 50 cycles minimum without failure
• Simple proof-of-run with text-to-speech enabled on hardware
  • Rely on the TTS image built on the CI server
  • python3 -m precursorupdater -b -l en-tts - load the TTS image onto the hardware
  • Note: if TTS fails consider doing cargo update in espeak-embedded, and re-running the espeak-embedded build task to refresh the APIs on the external server artifact
• Updating tests: (note: important to run on next EC update after v0.9.15 as changes were made to the EC update loop in v0.9.15 - remove this note once we have passed that point)
  • Provision unit with previous "stable" image, and update successfully to the new tag.
    • Now:
      • Use factory-newify repo and run rollback.sh script to roll a device back to the previous release without erasing the PDDB
      • Force a gateware update to move the SoC to the older version without erasing keys
      • Confirm all artifacts (soc, xous, ec, wf200) are at the previous version.
      • Perform an update using python3 -m precursorupdater -b and confirm all artifacts are at the correct version.
    • Previously (for historical records):
      • Use factory-newify repo and run the newify.sh script to roll a device back to 0.9.8 release.
      • Run update_ci -s and confirm that the auto-update flow succeeds (note: this has to be done after the "Latest" release has been updated on the server, so it sort of blends in with the first step of "pushing the release")
• Run the OQC factory test
  • Make sure the test peer AP is booted and active
  • Run test oqc on device that is provisioned with the candidate release firmware.
• Confirm that the factory-test jenkins CI flow has no errors this without erasing PDDB)
• Backup/restore test:
  • Take a device that is running the current tag + initialized keys + PDDB.
  • If the BIP-39 flow was modified in this release, provision BBRAM keys with bbram_helper.py.
  • Create a secret basis and store something in it.
  • Prepare and run a backup: prepare backup from menu and then run tools/backup.py
  • Newify the device.
  • Restore the backup by running tools/restore.py
  • Confirm the secret basis data is present
  • Check backup integrity: python3 ./backalyzer.py -c -f <path to backup>
  • Note: this test flow does not test the following: If BBRAM key step is skipped, then BIP-39 key entry of BBRAM/eFuse key is not tested; rekeying to a different device is not tested.

Pushing the Release

• About 0.5hrs
• Make the release official:
  • Copy the release artifacts on the server:
    • mkdir /var/releases/v0.9.x
    • cp /var/latest-ci/* var/releases/v0.9.x
    • rm /var/releases/latest - remove old symlink
    • cd /var/releases && ln -s v0.9.x latest - create new symlink
  • Update the CI expected version:
    • edit, on ci-pi, the version.expect script to have the new version inside the ci branch on bootstrap-mainboard repo
  • Update restore.py to include the latest release in URL_LIST
  • Update and tag the release on the factory tester:
    • In the factory test environment, check out main branch and change to bootstrap-mainboard/precursors/ and run update_ci.sh
    • This is done so that any potential regressions/changes since the test of the factory test are captured (ideally nothing has changed, though)
    • Commit the artifacts, tag & push. Revert to ci branch.
    • (best practice) newify, reboot & test one last time using the START button
    • Call the factory and ask them to absorb the changes.
  • Snapshot a factory-newify repository -- so we can quickly reproduce an RMA that may be specific to a quirk of the repository state.
    • create a new branch for the relevant version in factory-newify
    • inside the precursors directory run fetch_binaries.sh v0.9.x. This will snag the latest binaries from the named release directory and also update the VERSION file.
    • commit and push the branch. Do not merge to main. main is the first version shipped, version 0.9.5
  • Capture the crate audit
    • These commands are run inside the crate_scraper repo
    • cp ../xous-core/Cargo.lock . - update the .lock file to the latest version
    • rm ./crates/* - clear out the old crates fetched
    • ./crate_scraper.py --download - fetch the crates
    • rm -r ./builds/* && rm builds.rs and then ./crate_scraper.py --analyze - create the build.rs audit. The "Not found in archive" errors are normal.
    • Update the README.md with the xous-core tag used in the scrape and other notes
    • Commit & push
    • There is a potential TOCTOU because the crates are scraped sometime after the release build is actually built (within hours of the build, but still, a window for a timing attack).