Sporadic errors with armhf-gnu CI remote test runs

[ehuss]

The armhf-gnu CI runner has been having sporadic errors while running remote tests. The errors typically look something like:

---- [ui] src/test/ui/modules/mod_dir_implicit.rs stdout ----

error: test run failed!
status: exit status: 101
command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/remote-test-client" "run" "0" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/modules/mod_dir_implicit/a"
--- stdout -------------------------------
uploaded "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/modules/mod_dir_implicit/a", waiting for result
------------------------------------------
--- stderr -------------------------------
thread 'main' panicked at 'client.read_exact(&mut header) failed with Connection reset by peer (os error 104)', src/tools/remote-test-client/src/main.rs:310:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------------------------------------------

There's not much to go on here, as we can't really see the qemu server side.

I don't see a pattern in which tests are failing, so it could be just a general issue with the qemu setup?

• 2021-11-21T00:49:48 https://github.com/rust-lang-ci/rust/actions/runs/1485716983: ui/simd/intrinsic/inlining-issue67557.rs
• 2021-11-25T12:01:07 https://github.com/rust-lang-ci/rust/actions/runs/1503586127: ui/modules/mod-view-items.rs
• 2022-01-23T05:35:55 https://github.com/rust-lang-ci/rust/actions/runs/1735092010: ui/generics/generic-fn-infer.rs
• 2022-02-16T17:19:17 https://github.com/rust-lang-ci/rust/actions/runs/1854221995: ui/issues/issue-42210.rs
• 2022-03-03T09:44:04 https://github.com/rust-lang-ci/rust/actions/runs/1927255833: ui/modules/mod_dir_implicit.rs
• 2022-03-23T08:23:38 https://github.com/rust-lang-ci/rust/actions/runs/2027186581: ui/imports/glob-use-std.rs
• 2022-03-24T05:38:15 https://github.com/rust-lang-ci/rust/actions/runs/2032465233: ui/numbers-arithmetic/next-power-of-two-overflow-debug.rs
• 2022-05-19T11:55:23 https://github.com/rust-lang-ci/rust/actions/runs/2351785646: src/test/ui/binding/match-pattern-lit.rs
• 2022-05-29T10:43:44 https://github.com/rust-lang-ci/rust/actions/runs/2404024881: src/test/ui/generic-associated-types/const-generics-gat-in-trait-return-type-2.rs
• 2022-06-01T14:15:07 https://github.com/rust-lang-ci/rust/actions/runs/2422184056: src/test/ui/modules/mod_dir_implicit.rs

[Noratrieb]

This happened a lot recently:
#109220 (comment)
#109557 (comment)
#109692 (comment)

[saethlin]

Another one: #118942 (comment)

[saethlin]

Another one: #121662 (comment)