apache kafka - Exception for non-existent topic in MassTransit

Question

When topic in Kafka doesn't exist and there is an endpoint configuration done for it with TopicEndpoint method, it throws an exception: Confluent.Kafka.ConsumeException: Broker: Unknown topic or partition. After the topic is created, the exception doesn't occur anymore. How to set up the automatic topic creation for endpoints configured with TopicEndpoint method?

Answer 1

Automatic topic creation is not currently supported, and based on experience with a number of enterprises using Kafka, they don't recommend it either. Typically it is something that is managed so that proper partitioning and ownership can be applied to the topic (rather than just guessing).

In fact, I'm not sure it's even possible for a Kafka consumer (using the Confluent client) to automatically create the topic. There are aspects such as the schema registry, etc. that are taken into consideration as well.

I am open to the discussion, but the choices made were based upon interviewing teams using Kafka who said they wouldn't usually allow topics to be created automatically.