flake: add private inputs & defer locking #164
Conversation
|
Generally in favor of this. I see problems with not committing lock files, unless we rely on direnv with |
|
Well we would commit a lock file on the consuming cluster flake, just not on the upstreams to make them easier to update from the registry. if NixOS/nix#6550 were to be merged at some point, it wouldn't be necessary, but without that functionality, not commiting a lock on bitte is the only way for the consuming flake to easily control and update bitte's inputs (assuming they are all registry inputs, and not direct references). But yeah, some fine tuning of the registry will definitely be necessary. Thankfully it's easy to modify the json registry with the nix cli if you just pass |
|
Added some addtional, more specific pins to the registry, and gave this idea a testrun in a smaller project as a poc here: |
Inputs can now be declared without propagating to a consumers lock file by adding inputs to the flake in the private sub directory. The inputs defined here can be managed just like other flake inputs. The narHash in the main flake.nix for the private inputs must be updated anytime the private flake is touched.
nrdxp
changed the title
nixConfig section
|
ping @manveru, @blaggacao when you have time to look at this, I've implemented the majority of the idea, that can be safetly consumed by clusters, so long as they define our registry. In order to get the most out of it, we'll have to do something similar for most of our other projects, but on the plus side, it's really not too difficult to do so. |
| @@ -5,6 +5,7 @@ | |||
| .std | |||
|
|
|||
| # nixos/nix | |||
| flake.lock | |||
There was a problem hiding this comment.
Unfortunately, this does not do much. nix by default force adds the lock file
|
@nrdxp can you create separate PRs; one per motivation? These are unfamiliar patterns, we should scrutinize and field one at a time... ... probably starting with private inputs that won't then propagate to downstream consumers' lock files, anymore. This is relatively straight forward and will help bring lock file sizes down, but also needs for full effect its As for the registry, I'd like to better understand what abstract root problem a registry would solve and what undesired side-effects this might have. But we can discuss this on the respective PR. There is also a third consideration: bitte iteslf is a private input to the |
Includes our fork of the official nix registry:
https://github.com/input-output-hk/flake-registry/commits/iog
Here is a PoC of a scheme that allows for defining private flake inputs that do not propagate to the consuming flake, for easily management of dependencies without the overhead of cluttering everybodies flake.lock. Inputs that are defined in the top-level flake.nix should be expected to be defered to the consumer for locking, and preferably the consuming cluster should have our registry set in the
nixConfigto keep values consistent and bumps explicit.This is only the first step, but if we do this consistently throughout our projects, it will solve a majority of the lock related problems we've had so far while giving us a much easier path for updating dependencies on the cluster level. For example, it's been a pain in the past to update ssh keys from ops-lib, because we first had to bump the bitte dep and then again bump bitte on the cluster. Now that locking is deferred to the cluster flake will simply update ops-lib whenever a user requests an update to it, without having to channel through bitte at or define any special semantics.
This also gives us a reasonable path to avoid massive lock files that impose heavy costs: NixOS/nix#6626
For the sake of not breaking too many things, I have kept the API the same, that is, public inputs and private inputs are combined at the top of the flake.nix so minimal code changes to the rest of the codebase are needed. Once we
stdize we may be able to break this up a bit if desired.