[internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest #13599
Conversation
src/python/pants/core/goals/repl.py
Outdated
| @@ -139,8 +147,10 @@ async def run_repl( | |||
| InteractiveProcess( | |||
| argv=request.args, | |||
| env=env, | |||
| run_in_workspace=True, | |||
| input_digest=request.digest, | |||
There was a problem hiding this comment.
@Eric-Arellano / @stuhood: This line uncovered the issue with the Python repls. The repl goal was not passing the input root through.
There was a problem hiding this comment.
Mmm.
The repl should probably be doing whatever we do for run here...? Or we should add explicit support to InteractiveProcess for materializing in a temporary subdir? But it's not clear that that removes very much boilerplate.
Disabling "in workspace" for either repl would be confusing I think... that's probably a blocker.
There was a problem hiding this comment.
Or we should add explicit support to InteractiveProcess for materializing in a temporary subdir?
InteractiveProcess already has the ability to materialize in a temporary directory. That is what its input_digest attribute enables. But it requires run_in_workspace=False. The Python repls were providing an input root, but the core repl goal was ignoring it when constructing the InteractiveProcess and also hard-coding run_in_workspace=True.
There was a problem hiding this comment.
Why does run create a separate temporary subdirectory versus just using the input_digest support?
There was a problem hiding this comment.
but the core repl goal was ignoring it when constructing the InteractiveProcess and also hard-coding run_in_workspace=True.
They're not. If you look right above, they are writing the digest into the Workspace's temp dir.
We use the build root for both run and repl because we want to give users the flexibility of accessing the build root. For example, run needs to be able to write files to the build root, without forcing users to have their scripts use absolute paths to the build root. For repl, our argument was that it's for experimentation so you should be able to do things like open("f.ext") even if you don't have a file/files target for that.
If Scala cannot run in the build root because append only caches, such it is. Your new ReplRequest.run_in_workspace field is a good one though, and I think here in core/goals/repl.py you will want to use an if statement: if running in workspace, use the temp dir stuff, unmodified from before. Else, use input_digest and append_only_caches via the engine's mechanism.
There was a problem hiding this comment.
Simpler solution was just to set input_digest to EMPTY_DIGEST (similar to the existing code which doesn't pass it) if run_in_workspace=True since the data in the input_digest would have already been written to the temporary directory. The temporary directory still exists even if run_in_workspace=False since it is passed to the rules generating the ReplRequest and I didn't want to disturb that API contract.
[ci skip-build-wheels]
[ci skip-rust] [ci skip-build-wheels]
tdyas
force-pushed
the
repl_digest_pass_thru
branch
from
88f0ad4 to
4c33cb8
Compare
|
ping |
tdyas
changed the title
Yes. Updated. |
Internal changes: * [internal] Remove superfluous f-string specifiers ([#13821](#13821)) * [internal] scala: extract annotations as consumed types ([#13810](#13810)) * [jvm] Split nailgun digest from input file digests ([#13813](#13813)) * [internal] jvm: add jre_major_version and use stderr to properly extract version ([#13812](#13812)) * [internal] Clean up Go `embed` support's handling of dependencies ([#13801](#13801)) * [internal] scala: handle package object syntax in parser ([#13809](#13809)) * [internal] java: fix junit sentinel rule setup ([#13815](#13815)) * [internal] upgrade to rust v1.57.0 ([#13807](#13807)) * [internal] add failing test for FrozenDict equality issue ([#13389](#13389)) * [internal] Use `PyObject` instead of `Value` in more places ([#13802](#13802)) * Remove MultiPlatform Process abstractions ([#12725](#12725)) * [internal] add `JvmToolBase` for lockfile handling for JVM tools ([#13777](#13777)) * [internal] Port `MergeDigests` to Rust ([#13773](#13773)) * [jvm] Spawn nailgun servers outside the pool lock ([#13796](#13796)) * [internal] DRY loading internal Go binaries ([#13800](#13800)) * [internal] Convert unit tests to use pytest ([#13798](#13798)) * [internal] remove dead code - socket util. ([#13797](#13797)) * [internal] Reorganize Go parser scripts ([#13791](#13791)) * Adds Jackson core/datatype to `JVM_ARTIFACT_MAPPINGS` ([#13792](#13792)) * [internal] go: initial support for embedding resources ([#13743](#13743)) * [internal] Refer to `go.mod` path when downloading packages ([#13786](#13786)) * [internal] More robust Go dependency inference tests ([#13785](#13785)) * [internal] `tailor` doesn't add `go_package` for `testdata` folder ([#13783](#13783)) * [internal] Add Scala backend to dryrun for wheel builds. ([#13772](#13772)) * [internal] Unify JVM thirdparty resolves ([#13771](#13771)) * [internal] scala: infer dependencies from consumed symbols and intermediate scopes ([#13758](#13758)) * [internal] java: infer scala encoded symbols ([#13739](#13739)) * [internal] scala: parse and report package scopes ([#13738](#13738)) * [internal] go: configure included env vars for `GoSdkProcess` ([#13734](#13734)) * Fix some `./cargo audit` vulnerabilities. ([#13728](#13728)) * [internal] fix non-empty __init__.py ([#13730](#13730)) * Compute RepositoryPex directly from addresses. ([#13716](#13716)) * Upgrade to cargo-audit 0.16.0. ([#13729](#13729)) * Simplify `NativeHandler`. ([#13727](#13727)) * [internal] Switch to a maintained fork of the `fuse` crate for `brfs`. ([#13679](#13679)) * [internal] Add infrastructure to support deprecating field names ([#13666](#13666)) * [internal] Introduce OptionalPex/OptionalPexRequest. ([#13712](#13712)) * [internal] tailor adds go_package targets ([#13703](#13703)) * [internal] Remove unused testproject for pants-plugin ([#13704](#13704)) * [internal] Rename ambiguous `subpath` variable for Go code ([#13701](#13701)) * [internal] scala: generate the JVM names seen by Java code for Scala code ([#13696](#13696)) * Use RequirementsPexRequest in run_pex_binary.py. ([#13693](#13693)) * [internal] Refactor finding owning `go_mod` for first-party packages ([#13695](#13695)) * [internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest ([#13599](#13599)) * [internal] switch back to official `cargo-ensure-prefix` crate ([#13692](#13692)) * [internal] scala: extract type names from all Type.* AST nodes ([#13685](#13685)) * [internal] Convert unit tests to use pytest ([#13652](#13652)) * Unblock codegen support for java export analysis (#13645) ([#13675](#13675)) * [internal] upgrade to Rust 2021 Edition ([#13644](#13644)) * [internal] Don't store derived values on `go_first_party_package` targets ([#13676](#13676)) * Upgrade to py03 0.15.1. ([#13725](#13725)) * Add PyPDF2 to module mapping ([#13717](#13717)) * Upgrade console and indacatif. ([#13726](#13726))
Changes to the core
replrules needed by the Scala repl support:Add
run_in_workspaceandappend_only_cachesattributes toReplRequestand pass them into theInteractiveProcessfor the repl.Previously, the repl goal did not pass the input digest through to the
InteractiveProcess. This causes an issue for the Scala repl since it needs to materialize Coursier (which references paths in an append-only cache). If an input digest is provided in theReplRequestandrun_in_workspace=True, then the rule continues to write the digest to the temporary directory (and does not pass the input digest to theInteractiveProcess). Forrun_in_workspace=False, the input digest will now be passed into theInteractiveProcess.[ci skip-rust]