[Format][FlightRPC] Flight SQL evolution

[lidavidm]

Describe the enhancement requested

From apache/arrow-rs#5731 (comment)

Originally Flight RPC was implemented as a framework wrapping gRPC. This was especially expedient for the C++ implementation. By now it's mostly a weight dragging down Flight users, especially Flight SQL.

If we have the chance to evolve Flight SQL and/or Flight RPC, some changes may include:

• Use a proper gRPC service definition, instead of opaque bytes payloads
• [Format][FlightRPC] Transfer FlightData in pieces #34485
• [C++][FlightRPC] Memory tracking for arrow flight over grpc #37900

Component(s)

FlightRPC, Format

[lidavidm]

Other potential things

• There's a mix of stateful and stateless components, e.g. transactions use explicit handles but sessions are ambient (some of this stems from trying to accommodate JDBC/ODBC more directly)
• The name needs to be something that doesn't make people think it's a SQL dialect
• The metadata methods are incomplete (e.g there's no procedures, types, etc)
• There's no support for multiple result sets (this is also a limitation of using IPC streams in Flight, this has been complained about before too)
• The underlying IPC stream could use enhancements
  • the small result proposal from Micah, which would embed trivial result sets into the initial response
  • IPC data often isn't aligned
  • IPC data has to deal with the gRPC message size limits
• Duplication between GetFlightInfo/PollFlightInfo due to evolution over time (though you could argue it makes sense to have both anyways)
• There's no first-class support for result sets/queries as an actual API resource
• There's no support for result set pagination, limited support for retries
• There's no way to extend the protocol with custom actions
• There's no way to pass extra metadata alongside queries (e.g. a query ID to use)
• It's hard for proxies to interject info into requests/responses