Karate SQL? KSQL vs. Kafka Streams

Just kidding -:) No Karate SQL I am aware of..

Naturally you would use Kafka Streams if your code runs on Java where your code requires SQL like access to the data.

“Kafka Streams is the core API for stream processing on the JVM: Java, Scala, Clojure etc. It is based on a DSL (Domain Specific Language) that provides a declaratively-styled interface where streams can be joined, filtered, grouped or aggregated using the DSL itself. It also provides functionally-styled mechanisms — map, flatMap, transform, peek, etc”

(Building a Microservices Ecosystem with Kafka Streams and KSQL
https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/
via Instapaper)

Checkout KSQL, the Kafka Streams client for cases where you want to run SQL queries vs Kafka outside a JVM.

You can set a KSQL container as a side car along with your app container and let the app act upon regular Kafka Topic events, discarding the need for the app to deal with the data query logic needed to find relevant data off the stream.

Example: Your micro service needs to act upon a new customer order. Your sidecar container will run KSQL DSL select and stream only relevant event data to your app one at a time (configurable).

KSQL will get a copy of the same data across your micro services replicas.

Sounds like fun? Well because it is!

Maybe it should be called Karate SQL after all..

P.S.

If you use AWS, and need Kafka (otherwise you would use AWS Kinesis), here is a nice basic starter automation for setting Kafka on AWS.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s