[internal] Don't download Go third-party dependencies multiple times (Cherry-pick of #13352) #13378
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Turns out that #13339 didn't actually work - we are redownloading the same modules several times with Go! Downloads happen when:
GoModInfo
(once pergo.mod
)AllDownloadedModules
(once pergo.mod
)This PR fixes it so that we only download modules a single time, once per
go.mod
.To fix this, we stop treating third-party modules like first-party modules, i.e. we stop
cd
-ing into its downloaded directory and runninggo list
directly in it, using its owngo.mod
andgo.sum
. That requires that the chroot has all of the module's transitive dependencies present, and it also resulted in issues like #13138. Instead, the much simpler thing to do is rungo list '...'
to do all third-party analysis in a single swoop. That gets us all the analysis we need.We also extract the relevant
.go
files from all of the downloadedGOPATH
, i.e. all the downloaded modules. For compilation, all we need is the.go
files + the metadata we had earlier calculated. Compilation doesn't need access to anything else like other package's.For first-party analysis, we copy the whole
GOPATH
into the chroot. (This is really slow! We need something like #12716 to fix this.)Benchmark
Running in https://github.com/toolchainlabs/remote-api-tools.
Before:
Fixing only third-party analysis:
Fixing everything:
[ci skip-rust]
[ci skip-build-wheels]