Avro to JSON conversion problem with REST Proxy

[ttimot24]

Describe the bug

When I create a REST CRUD service with Zilla I'd like to convert the request and response JSON to AVRO and back. The request to AVRO works fine, but the response to JSON not looking good. During the serialization of the object Zilla include type information as sub fields into the JSON.

To Reproduce

name: zilla-demo

telemetry:
  metrics:
    - http.active.requests
    - http.duration
    - http.request.size
    - http.response.size
    - stream.opens.sent
    - stream.closes.sent
    
  exporters:
    stdout_logs_exporter:
      type: stdout
    prometheus_metric_exporter:
      type: prometheus
      options:
        endpoints:
          - scheme: http
            path: /metrics
            port: 7190

catalogs:
  schema-registry:
    type: karapace
    options: 
      url: ${{env.SCHEMA_REGISTRY_URL}} 
      context: default

bindings:
  mndwrk_tcp_server:
    type: tcp
    kind: server
    options:
      host: 0.0.0.0
      port: 
        - 8080
        - 61616
    routes:
      - when:
          - port: 8080
        exit: mndwrk_http_server
      - when:
          - port: 61616
        exit: mndwrk_mqtt_server
    telemetry:
      metrics:
        - http.*
        - stream.*
  
  mndwrk_http_server:
    type: http
    kind: server
    routes:
      - when:
          - headers:
              :scheme: http
              :authority: localhost:8080
        exit: mndwrk_sensor_service
    telemetry:
      metrics:
        - http.*

  mndwrk_sensor_service:
    type: http-kafka
    kind: proxy
    telemetry:
      metrics:
        - http.*
    routes:
      - when:
          - method: POST
            path: /api/v1/paymenthub/simple-pay/ipn
        exit: mndwrk_kafka_cache_client
        with:
          capability: produce
          topic: core-paymenthub-simple-pay-ipn-request
          key: ${idempotencyKey}
          reply-to: core-paymenthub-simple-pay-ipn-response

  mndwrk_kafka_cache_client:
    type: kafka
    kind: cache_client
    options:
      topics:
        - name: core-paymenthub-simple-pay-ipn-request
          value:
            model: avro
            view: json
            catalog:
              schema-registry:
                - strategy: topic
                  version: latest
        - name: core-paymenthub-simple-pay-ipn-response
          value:
            model: avro
            view: json
            catalog:
              schema-registry:
                - strategy: topic
                  version: latest
    exit: mndwrk_kafka_cache_server
  mndwrk_kafka_cache_server:
    type: kafka
    kind: cache_server
    telemetry:
      metrics:
        - http.*
        - stream.*
    options:
      bootstrap:
        - mqtt-sessions
        - mqtt-messages
        - mqtt-retained
        - mndwrk-zilla-response
      topics:
        - name: core-paymenthub-simple-pay-ipn-request
          value:
            model: avro
            view: json
            catalog:
              schema-registry:
                - strategy: topic
                  version: latest
        - name: core-paymenthub-simple-pay-ipn-response
          value:
            model: avro
            view: json
            catalog:
              schema-registry:
                - strategy: topic
                  version: latest
    exit: mndwrk_kafka_client
  mndwrk_kafka_client:
    type: kafka
    kind: client
    telemetry:
      metrics:
        - http.*
        - stream.*
    options:
      servers:
        - ${{env.KAFKA_BOOTSTRAP_SERVER}}
    exit: mndwrk_kafka_tcp_client
  mndwrk_kafka_tcp_client:
    type: tcp
    kind: client
    telemetry:
      metrics:
        - http.*
        - stream.*
    routes:
      - when:
          - cidr: 0.0.0.0/0 

Expected behavior
When I set AVRO to JSON conversion do not include the type information.

{
    "orderRef": {
        "string": "150-6622c5591a320"
    },
    "method": {
        "string": "CARD"
    }
}

{
    "orderRef": "150-6622c5591a320",
    "method": "CARD"
}

Zilla Environment:
Latest Zilla version

Kafka Environment:

• Provider: OpenSource Kafka
• Version: 3.7
• Config: default

Client Environment:

• Service: Postman Test

[vordimous]

@ttimot24 Are you able to share the Avro schema and how are the messages being produced to the topic?

[vordimous]

@ttimot24 I have tried to replicate your use case based on your zilla.yaml config by adapting the http.kafka.karapace example. I attached a zip of the setup that properly converts json to avro and back. You will find the steps to reproduce my findings in the README.md. Let me know if this unblocks you or Identifies what was causing the issue.

http.kafka.karapace.zip

[ttimot24]

Yes it works fine, ksqlDB creates the wrong response format.

Thank you @vordimous!