Support messages in Avro format

[berndruecker]

Context:

• Kafka users often prefer Avro as the message format instead of JSON (was also recommended since the beginning: https://www.confluent.io/blog/avro-kafka-data/).
• Kafka Connect Zeebe is currently hardcoded to use JSON, which limits the applicability of the connector in some use cases
• Zeebe internally stores data as JSON (https://docs.camunda.io/docs/product-manuals/concepts/variables)

Now there are two possibilities:

1. Convert messages from/to Json/Avro when they come in/go out. Maybe transforms (Use Kafka Connect "Single Message Transforms" instead of custom JsonPath #18) can do this and there is no need to do changes in this connector
2. Extend the connector to be more flexible on this, for example in the sink it might be the https://github.com/camunda-community-hub/kafka-connect-zeebe/blob/master/src/main/java/io/zeebe/kafka/connect/sink/message/JsonRecordParser.java#L49 and in the source around

   kafka-connect-zeebe/src/main/java/io/zeebe/kafka/connect/source/ZeebeSourceTask.java

   Line 143 in 177f164

   private SourceRecord transformJob(final ActivatedJob job) {

We should first explore if "transforms" do the trick (in this case adding some documentation might be sufficient).

[berndruecker]

Googling around I found this interesting thread: https://groups.google.com/g/confluent-platform/c/OGuOCkEKoeA.

My summary:

• Easiest would be to use KSQL to convert Avro to JSON and vice versa before using the Converter
• It is possible to build a custom SMT (https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/) to do this, but this does not exist out of the box yet (some inspiration can be found in https://github.com/ottomata/kafka-connect-jsonschema, https://github.com/epheatt/kafka-connect-jsonavroschema/blob/master/src/main/java/com/github/epheatt/kafka/connect/JsonToAvroSchemaConverter.java or probably also ewencp/kafka@3abb54a or https://stackoverflow.com/questions/65517359/kafka-connect-custom-transforms-to-convert-schema-less-json-to-avro). This probably needs a bit more understanding of how SMT and formats work in Kafka.
• All of this would be independent of the connector itself