Author Archives: jackbezalel

About jackbezalel

Senior Technical Leader, DevOps Specialist, {"AI & ML"​:”Data Science Pro"​}, Security Expert, Tech Books Author, Teams and Senior Management Career Coach, Blending Leadership and Innovation

The Cloud Cost Optimization Void and connecting FinOps and DevOps teams

Keeping operational costs optimized in the Cloud is one of the main pillars for cloud based companies success. This goes along with resilience, agility, and security. There are many resources discussing cost optimization, but they often lack the critical component—the human factor. Your people and processes will determine your financial success much more than the tools you use. Today we’ll focus on the former.

Let’s look at the basic scenario happening at your company as well: The FinOps (Financial Operations) team taking care of cost control and optimization for your cloud operations picks up on a component or a process that requires a change to reduce cost and improve operations. For example, it could be changing your instance types from the existing generation to the latest one. Moving to the new instance type can cut cost in up to 20%.

The FinOps team reaches out to the relevant development (or DevOps) teams with recommendations to change the instance types they are using. This is where the optimization process can break. Sometimes the development team simply does not respond, or they could state they have challenges (technical, time related or other) to address the required changes. There are cases where development teams accept and execute changes in a timely manner, but there is lack of visibility for the change and its impact on cost. Practically, The cost cut disappears into the “void”. Maybe it will be back sometime, probably too late, same as the passengers of the TV Drama “Manifest” of flight 828 have—and it did not end up well (so far at season 3 :-))

During the many journeys I lead with companies, I found the following process valuable in addressing the cost optimization challenges described here.

First, make sure you have continuous monitoring of your application workloads across their environments (development, QA, Staging, Pre-Production, Production). In each environment, your application workload (resources it uses) may change. Tag your application deployments with their change number (revision, version etc.). That way, you can observe the performance and resource use of your workloads across their change numbers and compare them.

Your FinOps team should be able to open a software/configuration change request for the development teams, when they find a workload that requires that change to improve its cost. That change request should enter the development team tasks. The task should have a cost-reduction estimation assigned to it. You can compute the estimated cost reduction according to the workload current cost, and then applying the estimated cost cut.

If you are a developer, you may say, “but my development team’s tasks don’t have any financial figure attached to them—how would I prioritize the cost reduction task versus a marketing or customer ask task?”. My answer is that most of your tasks could be set with a financial gain figure to them. So adding functionality would have a “5% income gain” assigned to it and could be compared to a “cost cut of 10%”. Lacking that, you should still strive to find a way to prioritize cost-cut related tasks.

Once the development team priorities allow it, they develop the code to implement the cost reduction. Since the task is in the development queue everyone knows about it, FinOps team as well- and we have visibility and accountability. Once the code change deploys into an environment (provided it is NOT mixed with other changes) you can clearly observe its effect on cost in your monitoring system: cost of your workload before and after the change.

Of course, this process is still challenging in many cases and having all the components I described can take time and effort to set and maintain, but I believe aiming to it brings valuable benefits.

We are all eager to learn from your experience on that topic, so please share!

Yours,

Jacky Bezalel, Senior Technical Leader at Amazon Web Services ; Teams and Senior Management Career Coach.

Karate SQL? KSQL vs. Kafka Streams

Just kidding -:) No Karate SQL I am aware of..

Naturally you would use Kafka Streams if your code runs on Java where your code requires SQL like access to the data.

“Kafka Streams is the core API for stream processing on the JVM: Java, Scala, Clojure etc. It is based on a DSL (Domain Specific Language) that provides a declaratively-styled interface where streams can be joined, filtered, grouped or aggregated using the DSL itself. It also provides functionally-styled mechanisms — map, flatMap, transform, peek, etc”

(Building a Microservices Ecosystem with Kafka Streams and KSQL
https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/
via Instapaper)

Checkout KSQL, the Kafka Streams client for cases where you want to run SQL queries vs Kafka outside a JVM.

You can set a KSQL container as a side car along with your app container and let the app act upon regular Kafka Topic events, discarding the need for the app to deal with the data query logic needed to find relevant data off the stream.

Example: Your micro service needs to act upon a new customer order. Your sidecar container will run KSQL DSL select and stream only relevant event data to your app one at a time (configurable).

KSQL will get a copy of the same data across your micro services replicas.

Sounds like fun? Well because it is!

Maybe it should be called Karate SQL after all..

P.S.

If you use AWS, and need Kafka (otherwise you would use AWS Kinesis), here is a nice basic starter automation for setting Kafka on AWS.

AWS Secrets… Yes!!

YES! ! AWS Secrets!!

https://aws.amazon.com/blogs/aws/aws-secrets-manager-store-distribute-and-rotate-credentials-securely/

I’d say secrets in parameter store are like Serverless credentials in Jenkins while secrets in secrets manager are like Serverless hashicorp vault. The difference for now is in the limits of use – SSM is free but would not work well when saturated with many calls – you are expected to use it moderately, while in AWS secrets you are not limited cause you pay. I believe in the future AWS secrets will be more feature-rich.

Kubernetes 1.9 admission extension – What is it?

Kubernetes 1.9 includes powerful admission extension abilities that are part of the golden principles Kubernetes is being built on – you want to look for those principals in other solutions you are considering.

What is Admission?

Admission is the phase of handling an API server request that happens before a resource is persisted, but after authorization. Admission gets access to the same information as authorization (user, URL, etc) and the complete body of an API request (for most requests).

What are they good for?

Webhook admission plugins allow for mutation and validation of any resource on any API server, so the possible applications are vast. Some common use-cases include:

Mutation of resources like pods. Istio has talked about doing this to inject side-car containers into pods. You could also write a plugin which forcefully resolves image tags into image SHAs.

Name restrictions. On multi-tenant systems, reserving namespaces has emerged as a use-case.

Complex CustomResource validation. Because the entire object is visible, a clever admission plugin can perform complex validation on dependent fields (A requires B) and even external resources (compare to LimitRanges).

Security response. If you forced image tags into image SHAs, you could write an admission plugin that prevents certain SHAs from running.

More information here: http://blog.kubernetes.io/2018/01/extensible-admission-is-beta.html