Go: support resource embedding

[Eric-Arellano]

We need to support resource targets being included via Go's embed directive. https://pkg.go.dev/embed

Unclear how files should be handled.

[Eric-Arellano]

This can be split into three steps:

1. Model a Go target depending on resource targets. This should be automated via analysis from FirstPartyPkgInfo and ThirdPartyPkgInfo.
   • Should Pants generate resources targets based on this? Probably not, at least at first, given that the resources could be used by other languages and this risks being confusing? It could definitely warn though!
2. Include the raw resources files into the chroot for build_pkg.py
3. Transform the raw resource file into a .go file

I will take the first two steps, @tdyas has offered to do the third step.

[tdyas]

I investigated further, and as it turns out, we don't actually need to do the transformation from raw resource file to Go file; the compiler will do that for us. We just have to provide an "embedcfg" to the compiler to map the patterns to actual file paths.

#13509 adds an EmbedConfig dataclass to handle the embedcfg. Once that PR lands, the only remaining step is (2) i.e. put the raw resource files into the sandbox and create an EmbedConfig to map the file patterns extracted into FirstPartyPkgInfo to the file paths of the raw resource files.

[Eric-Arellano]

Amazing! I can do part 2 and also complete part 1 to hook up go list for third-party + add inference and tailor based on embed.

Thanks!

[tdyas]

Amazing! I can do part 2 and also complete part 1 to hook up go list for third-party + add inference and tailor based on embed.

That will help re third-party; I haven't thought through exactly how file embedding should work in third-party packages. I assume the file patterns will just be evaluated relative to the package directory in that case. You will need to implement your own globbing logic to construct the EmbedConfig.

Bazel rules_go implements its matching logic using its own virtual directory tree. See https://github.com/bazelbuild/rules_go/blob/bd7fbccc635af297db7b36f6c81d0e7db7921cca/go/tools/builders/embedcfg.go#L291

The go tool's embed logic is here: https://github.com/golang/go/blob/b74f2efc47bbfcc4aa301ebda1033948d8b6b63e/src/cmd/go/internal/load/pkg.go#L2001. It uses globbing logic from https://github.com/golang/go/blob/master/src/cmd/go/internal/fsys/fsys.go. This also uses a virtual directory tree since it supports "overlay" files to define embeddable resources.

[tdyas]

https://pkg.go.dev/embed has a description of the glob format:

The patterns are interpreted relative to the package directory containing the source file. The path separator is a forward slash, even on Windows systems. Patterns may not contain ‘.’ or ‘..’ or empty path elements, nor may they begin or end with a slash. To match everything in the current directory, use ‘*’ instead of ‘.’. To allow for naming files with spaces in their names, patterns can be written as Go double-quoted or back-quoted string literals.
If a pattern names a directory, all files in the subtree rooted at that directory are embedded (recursively), except that files with names beginning with ‘.’ or ‘_’ are excluded.

Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links. Matches for empty directories are ignored. After that, each pattern in a //go:embed line must match at least one file or non-empty directory.

If any patterns are invalid or have invalid matches, the build will fail.

[tdyas]

I'm currently implementing this. The best approach was to use glob matching code in Go to ensure the same semantics as the go tool.

[Eric-Arellano]

Closed by #13743 and #13801. Thanks Tom for the help!