Using the Zilla MQTT broker the producer periodically disconnect and only reconnects after pod restart

[vordimous]

Describe the bug
A mqtt client once connected to the zilla broker is getting disconnected periodically.
The only way to recover is by doing a rollout resatrt of the zilla deployment

To Reproduce
Steps to reproduce the behavior:

1. Create an MQTT Client and connect to the zilla broker
2. Once Connected start producing mqtt messages
3. Messages are being continuously and after x amount of time, the MQTT Client gets a disconnect
4. Turned on stdout and the only messages the broker emits are these..
   "name: mqtt-quickstart

bindings:

Proxy service entrypoint

north_tcp_server:
type: tcp
kind: server
options:
host: 0.0.0.0
port:
- 7183
routes:
- when:
- port: 7183
exit: north_mqtt_server
telemetry:
metrics:
- stream.*

MQTT Server to Kafka topics

#region mqtt_broker_mapping
north_mqtt_server:
type: mqtt
kind: server
exit: north_mqtt_kafka_mapping
north_mqtt_kafka_mapping:
type: mqtt-kafka
kind: proxy
options:
topics:
sessions: mqtt-sessions
messages: mqtt-messages
retained: mqtt-retained
#endregion mqtt_broker_mapping
exit: north_kafka_cache_client
telemetry:
metrics:
- stream.*

Kafka sync layer

north_kafka_cache_client:
type: kafka
kind: cache_client
exit: south_kafka_cache_server
telemetry:
metrics:
- stream.*
south_kafka_cache_server:
type: kafka
kind: cache_server
options:
bootstrap:
- mqtt-messages
- mqtt-retained
exit: south_kafka_client
telemetry:
metrics:
- stream.*

Connect to local Kafka

south_kafka_client:
type: kafka
kind: client
options:
servers:
#- signoz.istio-kubernetes-gateway.acs314159.com:9092
- ${{env.KAFKA_BOOTSTRAP_SERVER}}
exit: south_tcp_client
south_tcp_client:
type: tcp
kind: client

telemetry:

Desired metrics to track

metrics:
- stream.opens.sent
- stream.opens.received
- stream.closes.sent
- stream.closes.received
- stream.errors.sent
- stream.errors.received
- stream.active.sent
- stream.active.received
exporters:
stdout_logs_exporter:
type: stdout

started
mqtt-quickstart.north_mqtt_server [18/Oct/2024:02:38:29 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (vsmqtt_client_4c65).
mqtt-quickstart.north_mqtt_server [18/Oct/2024:02:38:33 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (ACMq7lvl3p14awobakr).
mqtt-quickstart.north_mqtt_server [18/Oct/2024:10:47:31 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (test).
mqtt-quickstart.north_mqtt_server [18/Oct/2024:10:49:26 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (vsmqtt_client_4c65).
mqtt-quickstart.north_mqtt_server [18/Oct/2024:21:07:14 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (test)."

Expected behavior
The MQTT Client isn't able to establish connection.
The only way a connection can be stabalished is when the zilla broker is restarted.
The MQTT Client should be able to connect again without having to resatrt the MQTT Broker

Zilla Environment:
zilla-10182024.tar.gz

[bmaidics]

According to the user who reported this issue:
"with the latest changes (version 0.9.99), it has been almost 24 hours and no disconnects."