Crowd Source Best DevOps tools diagram
Just kidding -:) No Karate SQL I am aware of..
Naturally you would use Kafka Streams if your code runs on Java where your code requires SQL like access to the data.
“Kafka Streams is the core API for stream processing on the JVM: Java, Scala, Clojure etc. It is based on a DSL (Domain Specific Language) that provides a declaratively-styled interface where streams can be joined, filtered, grouped or aggregated using the DSL itself. It also provides functionally-styled mechanisms — map, flatMap, transform, peek, etc”
(Building a Microservices Ecosystem with Kafka Streams and KSQL
Checkout KSQL, the Kafka Streams client for cases where you want to run SQL queries vs Kafka outside a JVM.
You can set a KSQL container as a side car along with your app container and let the app act upon regular Kafka Topic events, discarding the need for the app to deal with the data query logic needed to find relevant data off the stream.
Example: Your micro service needs to act upon a new customer order. Your sidecar container will run KSQL DSL select and stream only relevant event data to your app one at a time (configurable).
KSQL will get a copy of the same data across your micro services replicas.
Sounds like fun? Well because it is!
Maybe it should be called Karate SQL after all..
If you use AWS, and need Kafka (otherwise you would use AWS Kinesis), here is a nice basic starter automation for setting Kafka on AWS.
YES! ! AWS Secrets!!
I’d say secrets in parameter store are like Serverless credentials in Jenkins while secrets in secrets manager are like Serverless hashicorp vault. The difference for now is in the limits of use – SSM is free but would not work well when saturated with many calls – you are expected to use it moderately, while in AWS secrets you are not limited cause you pay. I believe in the future AWS secrets will be more feature-rich.
- Because they pre-set with correct permissions, no effort on your part
- You cannot delete those roles by mistake if an AWS service uses them
- Easier to audit use compared to roles you set yourself
More info here – look where the column for services linked roles says “Yes”
MongoDB gets mature aiming to grab at SQL markets.. adds support for ACID – allowing regular transactions reliability in parallel to its traditional support for fast noSQL abilities:
Info on ACID: https://en.m.wikipedia.org/wiki/ACID
Kubernetes 1.9 includes powerful admission extension abilities that are part of the golden principles Kubernetes is being built on – you want to look for those principals in other solutions you are considering.
What is Admission?
Admission is the phase of handling an API server request that happens before a resource is persisted, but after authorization. Admission gets access to the same information as authorization (user, URL, etc) and the complete body of an API request (for most requests).
What are they good for?
Webhook admission plugins allow for mutation and validation of any resource on any API server, so the possible applications are vast. Some common use-cases include:
Mutation of resources like pods. Istio has talked about doing this to inject side-car containers into pods. You could also write a plugin which forcefully resolves image tags into image SHAs.
Name restrictions. On multi-tenant systems, reserving namespaces has emerged as a use-case.
Complex CustomResource validation. Because the entire object is visible, a clever admission plugin can perform complex validation on dependent fields (A requires B) and even external resources (compare to LimitRanges).
Security response. If you forced image tags into image SHAs, you could write an admission plugin that prevents certain SHAs from running.
More information here: http://blog.kubernetes.io/2018/01/extensible-admission-is-beta.html
New feature in AWS Redshift allows automatic audit logs shipping to S3 for complete backward analysis beyond the few days log kept inside Redshift