feat: introduce next-gen planner & optimizer

[wangrunji0408]

This PR introduces a brand new optimizer based on egg, a powerful crate to build cascades-style optimizers in Rust.

With egg, we can define a complex rewrite rule using pattern matching in a few lines:

// predefine join node: (join type condition left right)
rewrite!("join-reorder";
    "(join ?type ?cond2 (join ?type ?cond1 ?left ?mid) ?right)" =>
    "(join ?type ?cond1 ?left (join ?type ?cond2 ?mid ?right))"
    if columns_is_disjoint("?cond2", "?left")
    // ^ a custom function to test whether the column set of two nodes are disjoint
),

As a result, the new optimizer is super small but complete.

In about 600 lines of code (+ 200 lines for test), it implements the following optimizations:

• expression simplification
• constant folding
• predicate pushdown
• nested-loop-join -> hash-join
• join reordering
• column pruning

and resolves two annoying problems in the old planner:

• extracting aggregations from select list

  SELECT sum(a) + b FROM t GROUP BY b;
  => 
  SELECT s + b FROM          # projection
    (SELECT b, sum(a) AS s   # aggregation
     FROM t GROUP BY b);

• converting column references to physical indices (InputRef)
  and maintaining them during optimization

  =>
  SELECT $1 + $0 FROM 
    (SELECT b, sum(a) FROM t GROUP BY b);

As a comparison, the current planner + optimizer has nearly 4100 lines of code (+ 600 lines for test) with a similar feature set.

In my opinion, the egg framework is definitely a game changer for building database optimizers.
If it proves to work well in RisingLight, we can consider migrating it to RisingWave in the future.

[skyzh]

Looks great to me! Will take a look soon.

[skyzh]

Generally LGTM, thanks for your great work! It looks like the planner has not been fully integrated with the current query processing code path. Is there a tracking issue / TODO list for this?

[skyzh]

We should make ColumnRefId more generic (e.g. Vec<String>) so as to represent a column in subquery. e.g. subquery.table1.x. I'll do this probably when refactoring binder.

[wangrunji0408]

I gave up taking subquery into account. It's a little complicated. 🤯

[wangrunji0408]

It looks like the planner has not been fully integrated with the current query processing code path. Is there a tracking issue / TODO list for this?

Not yet. I'll create a tracking issue soon.

A rough plan is to:

1. fork the current binder, convert the sqlparser AST into the new egg AST.
2. rewrite the type checking as an egg analysis. (optional)
3. construct executors from the egg AST.
4. remove the old binder, planner and optimizer

[skyzh]

Sounds great. I'll start working on the new binder soon. Constructing executors from egg AST also doesn't look hard.

[skyzh]

ready to merge?

[fuyufjh]

Really blow my mind!

[fuyufjh]

Well, to be honest, I still had some concerns about the capability of DSL for a productional database system.

https://twitter.com/fuyufjh/status/1509182041828790274?s=20&t=LBlWsQKaNTiU2QBXbDtGBg

[wangrunji0408]

I think these kinds of DSL are suitable to describe the high-level structural trait of a rule. The rest detailed and complex logic is usually left to the native language. In fact, this combination is a common pattern in the Programming Language field (e.g. lex, yacc...). Because functional, declarative languages are good at language processing, while low-level procedural languages are more efficient. In this term, although Rust has pattern matching, we found it's not expressive enough for optimizer rules. So in my opinion, some kinds of metaprogramming are still needed, whatever it is macro, codegen, or DSL.

PS. When I was writing the rules of this PR, I felt like I was writing LISP, not Rust. Then I began to realize why LISP is so powerful. Greenspun's tenth rule 😇

[CAJan93]

Thank you very much for your work! Looks amazing!