Zilla doesn't communicate with bitnami/kafka in Taxi demo

[vordimous]

Describe the bug
Zilla (ZILLA_VERSION=0.9.64) is not brokering messages down to kafka in the Taxi-demo. The same zilla.yaml works when pointing to a redpanda instance at the same kafka:9092 address. A separate kafka-init container is successfully connecting and creating topics in both instances. I have checked the repo at the state before adding the redpanda feature and before the previous dependabot version updates and produced the same results.

gRPC messages are on the request topic but are not getting delivered to the response topic:

I updated to the previous version ZILLA_VERSION=0.9.63 and the issue does not exist.

To Reproduce
Steps to reproduce the behavior:

1. Run the Taxi-demo with the ./startup.sh script
2. Observe no messages are getting to kafka from the mqtt simulator and the /taxi/locations GET request from the UI never finishes
3. Stop the demo with ./teardown.sh
4. Run the Taxi-demo with the ./startup.sh --redpanda script
5. Everything works as expected

Expected behavior
Taxi Demo should function as stated in README

[vordimous]

Tested against lastest (ZILLA_VERSION=0.9.65) and the issue still exists.

[vordimous]

I found an additional issue while replacing Zilla with a new instance there was an extra message delivered to the reply topic:

[attilakreiner]

We have done some analysis on this with @vordimous with various zilla versions to look for regressions and try to narrow down the issue.

It looks like there is more than one issue at play here causing a mix of unexpected behaviors:

1. The issue where curl hangs seems to be a regression from 0.9.63 to 0.9.64, having been fixed in 0.9.65. When bumping zilla to 0.9.65 this issue is not present anymore.

2. There is another issue where at some point messages stop being propagated from the route-requests topic to the route-replies topic. This seems a regression from 0.9.61 -> 0.9.62 and the behavior seems consistent with the other bug described in Zilla Quickstart gRPC RouteGuide service hangs after lots of messages #719 which is already scheduled to be fixed.

At this point I suggest that we take the following course of action:

1. Bump zilla version to 0.9.65 in the example now to make the first issue go away.
2. After Zilla Quickstart gRPC RouteGuide service hangs after lots of messages #719 is fixed, let's revisit this if the second issue still persists.

[jfallows]

Blocked by #719

[akrambek]

I tried with latest release which has #719 fix and issue doesn't reproduce.