PX4 workqueue add simple reference counting to shutdown empty queues

[dagar]

When probing optional drivers at startup the corresponding workqueue per bus is created if it doesn't already exist. With the current init structure of attempting to start many possible external devices the result is a workqueue for every bus in the system. This doesn't actually waste that much memory, but we are starting to bump up against the NuttX task limit. #11792 (comment)

The downside of this PR is that in some cases the WQ for the same bus might be created, killed, then created again later as each optional driver is probing.

Example

INFO [px4_work_queue] creating: wq:I2C1, priority: 248, stack: 1200 bytes
WARN [bst] no devices found
INFO [px4_work_queue] wq:I2C1: last active WorkItem closing, stopping thread
INFO [px4_work_queue] wq:I2C1: exiting

[dagar]

Catkin CI failure might be real.

[dagar]

TODO: go back through the WQ manager to initiate shutdown.

[dagar]

New idea

• keep a count of active WorkItems per WorkQueue
• work queue manager add cleanup function to shutdown WorkQueues without any WorkItems
• add wq_manager command to call status and cleanup
• at end of boot call wq_manager cleanup (similar to mavlink boot_complete)

cleanup (end of init)

INFO  [px4_work_queue] wq:SPI4 empty, stopping
INFO  [px4_work_queue] wq:SPI4: exiting
INFO  [px4_work_queue] wq:I2C1 empty, stopping
INFO  [px4_work_queue] wq:I2C1: exiting

status

nsh> wq_manager status
INFO  [px4_work_queue] wq:SPI2 running, priority (relative to max): -2, stack: 1200 bytes, open: 1
INFO  [px4_work_queue] wq:SPI1 running, priority (relative to max): -1, stack: 1200 bytes, open: 3
INFO  [px4_work_queue] wq:hp_default running, priority (relative to max): -11, stack: 1200 bytes, open: 2
INFO  [px4_work_queue] wq:lp_default running, priority (relative to max): -50, stack: 1200 bytes, open: 1

[davids5]

New idea

• keep a count of active WorkItems per WorkQueue
• work queue manager add cleanup function to shutdown WorkQueues without any WorkItems
• add wq_manager command to call status and cleanup
• at end of boot call wq_manager cleanup (similar to mavlink boot_complete)

cleanup (end of init)

INFO  [px4_work_queue] wq:SPI4 empty, stopping
INFO  [px4_work_queue] wq:SPI4: exiting
INFO  [px4_work_queue] wq:I2C1 empty, stopping
INFO  [px4_work_queue] wq:I2C1: exiting

status

nsh> wq_manager status
INFO  [px4_work_queue] wq:SPI2 running, priority (relative to max): -2, stack: 1200 bytes, open: 1
INFO  [px4_work_queue] wq:SPI1 running, priority (relative to max): -1, stack: 1200 bytes, open: 3
INFO  [px4_work_queue] wq:hp_default running, priority (relative to max): -11, stack: 1200 bytes, open: 2
INFO  [px4_work_queue] wq:lp_default running, priority (relative to max): -50, stack: 1200 bytes, open: 1

I am concerned with fragmentation.

[mcsauder]

I haven't reviewed these changes, but here is a flight log from this PR:
https://review.px4.io/plot_app?log=80ee6b6e-abaa-468b-b7a1-06798555c5b4

[bkueng]

We can also get rid of the wq:manager thread which is only idling after bootup, and move thread start/stop handling into one of the wq that is running anyway.

[dagar]

Continued in #12853.