concurrency-limit: use tokio-util's PollSemaphore

[hawkw]

Currently, our concurrency limit middleware uses
tokio::sync::Semaphore to implement the concurrency limit. Because
acquiring permits from the Semaphore type is only possible via the
acquire and acquire_owned methods, which are async fns returning
unameable futures that are !Unpin, we cannot simply poll the semaphore
in poll_ready. Instead, to allow polling the concurrency limit service
to acquire a permit, we box the acquire_owned future in poll_ready.
This means that every time we try to acquire a new concurrency limit
permit, we must allocate a new box.

The tokio-util crate has a PollSemaphore type which allows
wrapping a tokio::sync::Semaphore to add a poll_acquire method to
poll the semaphore to acquire a permit. Unlike our implementation,
PollSemaphore uses the tokio_util::sync::ReusableBoxFuture type
for boxing the acquire_owned future returned by the Semaphore.
ReusableBoxFuture is a safe abstraction around unsafe code that allows
one allocation to be reused for storing multiple type-erased
Box::pinned futures. This means creating a single allocation for each
clone of a ConcurrencyLimit service, rather than having each clone of
the concurrency limit service allocate once every time a new permit is
acquired and drop that allocation once the permit is acquired.

This should reduce the overhead of the concurrency limit layer a bit. It
also makes the code much simpler! :)

Signed-off-by: Eliza Weisman eliza@buoyant.io

[hawkw]

logging the number of available permits here depends on upstream PR tokio-rs/tokio#3683

[kleimkuhler]

Nice! Thanks for the helpful description.