[RFC 0148] Pipe operator

[piegamesde]

Rendered

Discussion notice: please try to attach all discussions to a thread by using the code review feature. If your comment doesn't refer a specific line to attach to, use the header line instead.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/3

[roberth]

Nix is already considered to be overly complex by many. Of course we know that Nix solves hard problems and can't do so without exposing intrinsic complexity, but this needed complexity is already more than most people expect to have to deal with, coming from the broken but "easy" traditional methods.

This makes the addition of accidental complexity to Nix disproportionately harmful. It makes newcomers more likely to reject Nix, depriving themselves of a real solution to their packaging and deployment problems, while depriving us from valuable contributions.

I am opposed to the pipe function, and skeptical of a function application operator; especially one that reverses the evaluation order compared to normal function application.

[roberth]

pipe is not a good function because the order is not obvious. It's already available practically everywhere as lib.pipe. It's also hardly ever used, so doing this for performance strips it of the last argument for making it a builtin.

Builtins need to satisfy more stringent requirements, they take resources from the Nix team, and they can never be changed or removed.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned.

[KFearsoff]

It's also hardly ever used

I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned

This... makes sense, but then I'd make an argument that parts of nixpkgs.lib that manipulate generic data should exist outside of builtins and outside of nixpkgs - in a separate repository. It is not weird: we already have library flakes like flake-utils or flake-parts.

This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe not seeing much usage is due to technical reasons.

[rhendric]

Shepherds currently believe this conversation is resolved, outcome: inclined to leave pipe in lib.

[Profpatsch]

pipe is not a good function because the order is not obvious.

@roberth for what it’s worth, the commit where I introduced it has a rationale for the order in its message.

[roberth]

It's not a problem with your contribution, but with the constraints of the Nix syntax.

lib.pipe is a necessary step, and its adoption shows that it is useful to many, so thank you for creating it!

[roberth]

I think you could call this curried style.
While it is point free in the technical sense of not having a variable name to carry the data flow, point-free is generally only used in a context where higher order functions are used for more arbitrary data flows. For many, it also carries the connotation of the synonymously used pointless style.

[piegamesde]

I've never heard that term and the resources I've read so far all talked about "point-free" programming style. Google does not seem to know much either, so some links would be appreciated.

[roberth]

The flipped pipe function is just function composition on a list, eta expanded. Eta expansion / reduction do not change the meaning, so these two functions would be indistinguishable.

That means that we can divide the problem and drop some unnecessary terminology.

1. pipe = flip composeFunctions (and composeFunctions = flip pipe by virtue of a flip law)
2. composeFunctions is isomorphic to pipe
3. composeFunctions returns a function, which makes it easier to use where a function is expected: the application of higher order functions such as map.

A lot of that is just overly formal thought. Instead we can simplify this section to:

flip pipe is equivalent to function composition applied to a list. By using a function that composes a list of functions instead of pipe, we get back a function, which can be readily used in higher order functions such as map.

[Profpatsch]

point-free is also point-less, in a semantic sense 😛

[rhendric]

Shepherds currently believe this conversation can be resolved by accepting or rejecting the suggestion below.

[piegamesde]

I am very well aware of the language's complexity. I am also aware of the fact that some of its users are very technical and into functional programming languages, and for many others this is a first contact with these concepts.

This is why I suggest picking only one of these operators as a compromise. I don't have any overly strong preferences here, but I make a point against function composition instead of argument piping, because the former leads to the so-called "point-free" programming style which tends to be more confusing for new users. I picked |> because of its similarity to the existing lib.pipe and the |> operator in Nickel, but if you are concerned about reversing control flow, then <| is a viable alternative too.

I strongly oppose calling this proposal "accidental" complexity. I spent a lot of time thinking about the options and trying to balance expressiveness and complexity, finding a compromise that makes solving real Nix problems easier. The Nix language is full of weird quirks users eventually have to face: we even have a wiki article collecting such instances, and it is far from exhaustive. Feel free to call these accidental complexity. But nothing about this proposal is accidental.

[roberth]

To be clear, I don't use accidental to describe this RFC. I only used it in accidental complexity, which I've used as a synonym for extrinsic complexity. I can see that you've put a lot of thought into it, and I respect that, but that does not necessarily make the change a net positive.

expressiveness

A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.

makes solving real Nix problems easier.

What is a real Nix problem? At least the lack of a syntax won't end up on the quirks page.

Are the quirks a problem? Probably. I'd be happy to see a proposal that simplifies the language or makes the syntax easier to learn, but those are hard problems with a lot of inertia, and up-front costs that aren't "repaid" for years into the future.
Pragmatically, I think the real Nix problems aren't in the syntax. We might reach a stage where we could simplify the syntax and possibly redo it to make it more accessible to people who use more commonplace languages. That's not in the coming five years though, unless we radically expand the Nix team, which is rather resource constrained today.

[suhr]

Nix is already considered to be overly complex by many.

And more people consider Nix awkward to read and write rather than being complex. Mostly because the standard library is poor and not discoverable.

|> would definitely make Nix less awkward to use.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/10

[piegamesde]

A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.

I disagree with your interpretation of the word "expressiveness". Because you can (almost) always already solve all existing problems with existing syntax in (almost) any programming language. That's the point of being Turing complete. Therefore the question then becomes, how well can a problem be expressed in the language. And I think this holds true even for changes that are purely syntactical, like this one.

When adding new syntax to the language, one goal may indeed be to make it easier to write. Another one, and IMO much more important, is does it make the language easier to read. Depending on the feature and the language those may coincide, or one may have the other as a side benefit, or they may contradict each other. I recently learned Haskell, so I am very well aware of the trap of having a lot of powerful operators that allow it to write programs concisely and without parentheses, but which comes at the cost of readability.

What is a real Nix problem?

Problems that one might face when writing Nix code, either for some flake/shell/system configuration or when contributing to Nixpkgs. What I mean with this, is that I don't want to solve problems that for example mostly occur when trying to solve AdventOfCode in Nix.

[KFearsoff]

While the lib.pipe function is pretty cool (thanks for introducing it to me!), I think going through with this RFC would take gigantic effort, and the positive results (if any) will be seen only after a few years. And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.

[KFearsoff]

It's also hardly ever used

I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.

I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned

This... makes sense, but then I'd make an argument that parts of nixpkgs.lib that manipulate generic data should exist outside of builtins and outside of nixpkgs - in a separate repository. It is not weird: we already have library flakes like flake-utils or flake-parts.

This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe not seeing much usage is due to technical reasons.

[KFearsoff]

I think this is a great argument against going through with this RFC.

[piegamesde]

This would be an argument for never making any changes to improve the language at all, ever. Unless you see some shiny alternative language to which we'll realistically all migrate to within the next five years, any improvements to the Nix language are still expected to be beneficial.

I've actually thought about proposing this change for quite some time now, and the main thing that held me back was the idea is that Nix is a "lost cause" and that I should rather spending my energy towards some successor. But eventually, I realized that realistically, we'll be stuck with Nix for a long time to come, so better invest the energy to make it a bit more comfortable …

Also note that outside of Nixpkgs, the feature will be available almost immediately. There is a lot of Nix code outside of Nixpkgs, all these flakes, shells, and users' system configurations. Projects like home-manager, devenv, nur, etc. are all free to migrate to newer Nix versions at their own pace. We shouldn't ignore these.

[piegamesde]

I think going through with this RFC would take gigantic effort

Honestly, the most effort in this RFC is convincing people, and maybe also the coordination across many projects. But from the technical side of things, this is actually pretty easy to implement.

and the positive results (if any) will be seen only after a few years.

This only applies to Nixpkgs, see #148 (comment)

And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.

They also won't make it significantly harder for beginners to read Nix code IMO, while bringing quality of life improvements to everyday Nix development.

I don't understand why only "technical" problems should be worth solving.

[edolstra]

This RFC is now open for shepherd nominations!

[piegamesde]

@AndersonTorres because I don't know every programming language and just didn't come across OCaml during my research. Furthermore, the related work section is intended to roughly cover the design space, not to be an exhaustive list of every programming language.

[piegamesde]

Updated the text according to some of the initial feedback. The more I think of it, the more I'm leaning towards also having <| in the language. I'll try to conduct a larger Nixpkgs survey soon to get some more data on that question. (Function concatenation operators are still out IMO)

[AndersonTorres]

I was thinking on it, precisely, two pipe operators! It makes the language more orthogonal: if we have arg |> function, it looks reasonable to have function <| arg as its reverse operator.

The problems are:

• if we are to elevate pipe to builtins and pipe == |>, where is the builtin for reverse pipe? The apply previously suggested looks good to me;

• how the pipes should interact when in a same statement (a |> b <| c, a <| b |> c)?
  I suggest to APLize: same precedence, right-to-left associativity (a |> (b <| c), a <| (b |> c))

• a problem for the parser team: how should the errors be reported?

[piegamesde]

I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design.

[piegamesde]

@roberth, what about you, would you like to be a shepherd? I'd rather like to have critical voices on board from the beginning …

[maralorn]

I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design.

I am pretty certain there are a lot of people with more experience on this in the wider community. That doesn’t prevent me from having opinions on this particular bikeshed, though. 🤣

My availability depends on the timeframe here. I will be very busy in the next few weeks, after that I’d be on board.

[piegamesde]

I've procrastinated writing this RFC for at least half a year, so waiting two more months won't be the end of the world

[roberth]

would you like to be a shepherd?

I can do that

[rhendric]

(Attempting to corral feedback into a thread.)

@illustris [#148 (comment)]

nix-repl> double = x: x*2

# current syntax
nix-repl> 3 |> (x: 1 + x) |> double |> toString
"8"

# proposed alternative
nix-repl> 3 |> x: 1 + x |> double |> toString

# current syntax
nix-repl> 3 |> (x: 1 + (x |> double)) |> toString
"7"

# proposed alternative
nix-repl> 3 |> x: 1 + (x |> double) |> toString

Both of those proposals seem, to me, to be ‘more likely’ to mean something else:

3 |> (x: (1 + x |> double |> toString))
3 |> (x: (1 + (x |> double) |> toString))

In other words, if |> binds more loosely than the lambda-forming :, it would be the first thing in the language to do so and thus would be very surprising to me. So far in the language, a lambda always extends as far rightward as it can.

[piegamesde]

Partially applied pipes

To me this smells like function composition with extra steps. Function composition is discussed in the RFC as well, and while I am not completely opposed to it, my fear is that it reduces code readability given a lack of type annotations. In x = |> inc |> toString it is a lot less intuitively clear that x is a function.

This is one of these questions where I'd like to see more usage to find out whether this is a thing that comes up sufficiently often in practice that it is worth dealing with.

(Side note: I think we would get this feature for free if Nix had operator sections)

Lambdas inside pipe

While I agree that not requiring parentheses here would be nice, especially given that one goal of the pipe operator is to reduce parentheses, I don't see any way this would be realistically implementable without throwing the entire language under the bus. Making the binding strength of abstractions context dependent just doesn't sound great

[Profpatsch]

Making the binding strength of abstractions context dependent just doesn't sound great

agreed, it’s a recipe for disaster

[illustris]

Lambdas inside pipe

Making the binding strength of abstractions context dependent

Right. I didn't think that through. This is a bad idea.

Partially applied pipes

In x = |> inc |> toString it is a lot less intuitively clear that x is a function

In my opinion it is slightly more clear than, for example,

concatLines = concatStringsSep "\n"
getBin = getOutput "bin"

or any other partially applied function. The open |> (or |) at the start suggests to me that this is a function.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-08-19/50831/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-02/51514/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-16/52224/1

[jian-lin]

Is it allowed to backport the pipe operator to the Nixpkgs minimum nix version?

[MattSturgeon]

Do major feature additions like this usually get a backport?

Even if it was backported, wouldn't it defeat the purpose of having a minimal version that's likely to be installed by most users?

A backport would still be a new release, even if it was based on an old major version number.

[alyssais]

Sometimes. Support for zstd cache compression was backported to 2.3, so it's within the realms of possibility.

The reason we have a minimum version of 2.3 not so much to support people who haven't updated Nix in several years, it's because later versions of Nix have unresolved regressions in things that some people depend on.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-09-30/53690/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-10-28/55095/1

[GetPsyched]

RFCSC:

@rhendric, what's the status of this RFC? What is the next step to move this RFC forward?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-11-11/55888/1

[rhendric]

@rhendric, what's the status of this RFC? What is the next step to move this RFC forward?

Per these notes, we are nominally now in a period of experimentation, collecting feedback, and iterating. I had ventured some thoughts in this direction here but to my knowledge nothing formal has actually been done.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-11-25/56591/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2024-12-09/57636/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/lastmodified-attribute-mismatch-in-input-lastmodified-0/58395/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2025-01-06/58396/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2025-01-20/59113/1

[GetPsyched]

RFCSC:

@rhendric has anything progressed since the last status update? Please share any updates since then.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2025-02-03/59763/1

[VTimofeenko]

I absolutely love this feature. Helps a lot with rewriting long rambling
functions into a much more concise form.

Something that's been on my mind while refactoring code from lib.pipe is (and
please let me know if this would be off-topic here) is that I wish there were a
map-pipe infix notation. I often write functions in a sequence so that they
would do one thing and one thing only. When working on lists, it sometimes
results in syntax like so:

doSomething = x: x; # example unary function
doSomethingElse = x: y: z: x + y + z; # example n-ary function
foo =
  [ ... ] 
  |> map doSomething # apply `doSomething` to every element of the incoming list
  |> map (doSomethingElse foz baz) # apply `doSomethingElse` bound to `foz` and `baz` to every element of the incoming list
  |> ...

What I wish existed is a pipe+map operator that would allow me to write the same
code as(\> symbol is chosen as an example):

foo = 
  [ ... ]
  \> doSomething # applies `doSomething` to every element of the incoming list
  \> (doSomethingElse foz baz)

For attrsets it would be applying mapAttrs:

foo =
  { ... }
  |> builtins.mapAttrs (name: value: doSomething value)
  |> builtins.mapAttrs (name: value: doSomethingElse foz name value)

Becomes:

foo =
  { ... }
  \> (name: value: doSomething value)
  \> (name: value: doSomethingElse foz name value)

Pros

• Makes code cleaner.

Cons

• More development efforts.
• Pipe syntax is new enough, this would be an addition on top of it that may
  confuse people even more.
• Attrset map-pipe specific: less obvious than the list case (is it applying
  mapAttrs, mapAttrs', a combination of attrValues and map?)
• (minor) I don't think there is a good font ligatures for \>.

Notes

• I am not aware of any language with an infix pipe that has map-pipe as a separate operator.
• Not sure if inverse operator (<\ or </) makes sense at all.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/lix-mismatch-in-feature-name-compared-to-nix/59879/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2025-02-17/60444/1

[piegamesde]

The pipe operators have been now implemented for quite a while, now I'd like to know about common usage patterns. Especially:

• What are the real-world use cases for <|?
• Are there use cases where mixing |> and <| is actually useful / would have been useful? (Lix allows it, CppNix forbids it)

[sersorrel]

What are the real-world use cases for <|?

While writing open-location-code.nix, I frequently found myself reaching for Haskell's $, which AIUI is essentially what <| is. For example, the following snippets:

let
  digitVal = digit: builtins.floor (builtins.stringLength (builtins.elemAt (builtins.match "(.*)${digit}.*" digits) 0) / 2);
  # ...
  lat' = builtins.floor ((lat + 90) * 8000);
  long' = builtins.floor ((long + 180) * 8000);
  encodeLatMsd = lat: valDigit (mod lat 20);
  encodeLongMsd = long: valDigit (mod long 20);

could have some parens removed:

let
  digitVal = digit: builtins.floor <| builtins.stringLength (builtins.elemAt (builtins.match "(.*)${digit}.*" digits) 0) / 2;
  # ...
  lat' = builtins.floor <| (lat + 90) * 8000;
  long' = builtins.floor <| (long + 180) * 8000;
  encodeLatMsd = lat: valDigit <| mod lat 20;
  encodeLongMsd = long: valDigit <| mod long 20;

On the other hand, I don't know if you would count open-location-code.nix as a "real-world use case"...

Examining the results for the GitHub code search query language:nix "<| ", you see some slightly more interesting ideas, including use with lib.mkForce/lib.mkIf:

{
  "containers/registries.conf".source =
    lib.mkForce
    <| toml.generate "registries.conf" {
      /* multi-line expression elided */
    };
};

lib.mkIf config.shell.direnv.enableCompletionAutoloadingWorkaround
  <| lib.mkOrder 2000 "..."

There's also some more complex usage in an Advent of Code solution here: https://github.com/V1K1NGbg/Advent-Of-Code-2024/blob/755b780a3416dc03508a209dc8c072a8e2aa938c/2.2.nix, including this fun line using both <| and |>:

concat [ (range 0 (n - 1) |> (map (elemAt xs))) ] <| groupsOf n (tail xs)

[VTimofeenko]

Should the feedback be provided in this issue or through some other means?

I don't want to spam here, but I do want to let the record show that I absolutely love this feature just as it is.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfcsc-meeting-2025-03-03/61097/1