Good overview, particularly the latency/durability tradeoffs between sync/async.
Do you have any views on the "reverse"? i.e., an "extraction API" or, in other words, the ability to consume from Kafka via HTTP? I believe the Confluent REST Proxy supports it but I haven't used it. I was always under the impression that the HTTP/KAFKA semantics aligned better when producing than when consuming.
Great read. If you ever finish the book, Ill be first in line to buy it. I'm working on an ingestion service for a project of mine currently with go, redpanda, and scylla, so this is right up my alley. Thanks for sharing!!
Good overview, particularly the latency/durability tradeoffs between sync/async.
Do you have any views on the "reverse"? i.e., an "extraction API" or, in other words, the ability to consume from Kafka via HTTP? I believe the Confluent REST Proxy supports it but I haven't used it. I was always under the impression that the HTTP/KAFKA semantics aligned better when producing than when consuming.
I agree it makes more sense to use it for producing.
Since Kafka doesn't support lookup queries you'd be limited to "tailing" the output of the topics with an HTTP consumer, so it's not always useful.
In my opinion a better pattern is writing the data to a database, index/partition/sort it properly and add an HTTP API on top of that.
Great read. If you ever finish the book, Ill be first in line to buy it. I'm working on an ingestion service for a project of mine currently with go, redpanda, and scylla, so this is right up my alley. Thanks for sharing!!