Category Archives: Big Data

Solved: DE-Centralized Puppet Asset Management – factbeat

Using a DE-Centralized (Master-Less) Puppet stack has its benefits for dynamic fast morphing environments.

Yet you’d still love to get all changes made to your environment recorded in a central repo.

Check out factbeat from Elasticsearch community. It’s a beat that  ships Puppet Facter facts to Elasticsearch, where they can be stored, analyzed, displayed and compared over time.

Factor can be easily customized to ship new types of configuration information as your heart desires.

What are you using?

State of the ‘Spark’

I got first hands on with Apache Spark about a year ago and it seemed cool. Yet going through my updated quick notes here, I felt falling in love with it 😎 It grew much more in integration options as well as features..

  1. The Zeppelin IDE checks for syntax errors syntax and shows you the data as well as lets you submit jobs to a spark cluster
  2. Scala is the default language, but can be used from python SQL and others
  3. Spark is Newer than Hadoop and positioned to replace it
  4. Spark Optimizes data shifting using memory mapping and reduces the move data across cluster nodes using partitions
  5. Runs on top of jvm
  6. Scala is based on Functional programming where you would use X = collection of Y filter by.. Instead ofFor loop in Y  If then add to X
  7. Spark uses RDD – Resilient Distributed Datasets: fault-tolerant collection of elements that can be operated on in parallelis and produce the data processing we want
  8. Spark supports many formats for the data: hive json cassandra Elasticsearch
  9. Spark can be used with Mlib for machine learning
  10. Spark Streaming allows data frames manipulations on the fly – letting you write streaming jobs the same way you write batch jobs. It supports Java, Scala and Python. 
  11. SparkR let’s you interact with spark via R. Still not fully functional
  12. You can use those to submit Spark jobs: EMR step, Lambda, AWS pipeline, Airflow, Zeppelin, R Studio
  13. You can reduce cost and keep data off the cluster on S3 and by using emrfs as well
  14. In AWS you can hook Spark with DynamoDB RDS Kinesis and many others
GENESIS - Big Data Super Nova - Part One (Eve)

Big Data Super Nova Book on Kindle for FREE (next 23 hours)

I just published my first Novella on Amazon and would like to share it with you.

If you go to this link: http://amzn.to/17mLBi4 you can get it for free on your Kindle (in the next 24 hours or so)

*** I’d really appreciate if You Could Rate it and Write a Comment ***

It is called “GENESIS – Big Data Super Nova” (40 min read) – insights and ideas on the future of Big Data, wrapped in a Sci-Fi tech Novella:

*The Death of passwords
*Data dissolving agents
*The new security paradigm and the merge of humans and computers
*Inter Body Nano Bots, Brain Dumps, Light Speed Travel
*And more…

GENESIS - Big Data Super Nova - Part One (Eve)

GENESIS – Big Data Super Nova – Part Three (Apple)

Hey and welcome!

The “Genesis – Big Data Super Nova and the Journey Back to Privacy and Security” broadcast is about to begin.

This time I decided to share with you my insights and ideas through a story.
So tighten your seatbelt as we jump into hyperspace through time.
It’s now exactly 37 years forward – the morning of March 13, 2050, 11 AM: Earth Standard Galaxy time.

This is chapter 3, here is where you can find The first chapter of this Big Data Sci-Fi Novella (Eve)

Chapter 3: Apple

GENESIS - Big Data Super Nova - A Sci-fi Tech Novella, Chapter 3: Apple

Listen to this postListen or Download audio of this chapter here: 

It was the 1st of April 2013.

One year before, there were rumors that the world is going to end, that is, if you believed the ancient Mayan literature.

It turned up to be a bit later and more digital than physical.

Back then, social networks such as “Facebook” and “Twitter” allowed people to share their thoughts and feelings through external personal outer-body devices.
People called those cumbersome ugly devices “iPhone” and “Android”.

Everyone was hooked to those networks. Companies started letting people get access to services, based on their always-on authentication, to the social networks.
But people were using “Pass-words” to get connected to the social networks. They chose easy pass-words and did not care replacing them occasionally.

So if you found out one’s social network password (let’s say on Twitter), you also got access to any other service they had authorized (O-AUTH) to get logged on to, based on authentication token stored in their Twitter account.

Basically if you had someone’s twitter password, you could buy stuff and have them pay for it, look at the medical records, and review their personal assets. You actually became them.

When the big Digital Tsunami happened, it started by a huge surge of credit card and bank transactions fraud. Then as money pore into the attackers bank accounts (there were many used, so you could not track a single source), it started spreading away as a massive purchase wave.

Amazon, eBay and similar services were hammered with endless waves of purchases, and then rebounded into cancellations. Everyone tried to cancel the fraud purchase attempts.
There were similar attacks on health, government, manufacturing and other essential services, all directed to make them useless.

Later on this kind of attack was labeled “Application Based Denial of Service” – where instead of driving lots of requests to a service, you would overload the remediation system of a service with transactions that require huge effort to remediate (such as reversing a fraud transaction).

There was lots of confusion and slow response, addressing the global, world-wide break-in, because of the miss-leading reports, that it was all a 1st April joke.

In less than one hour all the financial institutions, digital merchant and many other critical systems were completely ruined.

It took 4 months to get most of the damages fixed. But by then many lives were lost.
You could die, because you could not get food or medical treatment, or you could get attacked, by desperate people who tried to forcefully take what you had.

No Anti Malware system could find out the secret key-loggers which had everyone’s passwords, because they were part of many operating systems.

The attackers worked many years, getting to the right people in every operating-system manufacturer’s staff. Then the secret key-logger code was added to the operating system’s codebase. Every developer the attackers acquired, had to add or change just a small piece of code, entirely blind to the demonic intension driving all those small changes (Time Shifted Attack).

The attackers had even penetrated the NSA (National Security Agency) staff, and implanted their own code on top of the home-land security hooks. Those hooks were originally designed to allow law-agencies get access to people’s cloud stored data as well as communications, in case those people were suspects. Now those law enforcement hooks had a parasite code, wire-tapping everything, ready to spray all this information to the malicious crackers.

It all sat there, silent, waiting, collecting and using Big Data to analyze and further, and reveal additional access details. Gone were the days of brute force password cracking – you guess passwords much faster, using Big Data analysis.

And then it was time for the software octopus to wake.

It started ignition on March 13, and up it woke on April 1.

Adam recalled that there was one more important event which took place on the 1st of April 2013.
The “Naturalists Group” was born.

The attack was carried on through Windows, Linux, Oracle, Java infrastructure suppliers and Cloud Providers.
Cloud services failed one by one and the true lack of the Cloud concept was revealed: We did not have “The Cloud”. What we had was many disparate cloud services, each one with its own weakness and finalized amount of resources.

Each Cloud provider had a final central set of “engines” operating all its services, which could easily hacked and brought down, essentially making it useless.
You really had nowhere to fail over to.

The lack of Data Integration was apparent across Big Data archives, so you couldn’t really see what was going on. We had no “Big Eye in the Sky”, no “Digital Defense Satellite”, watching our data and computers.

That was the day when true “Universal Computing” and the “Neuro-Fibre Net”, were born.

Lots of server systems halted that day, and many personal devices got their share of blackout as well.

But most of the victims had the mobile device of choice of that time.

It was produced by Apple.

Get additional chapters by reading the book by Clicking Right Here

GENESIS - Big Data Super Nova - Part One (Eve)

GENESIS – Big Data Super Nova – Part Two (Adam)

Hey and welcome!

The “Genesis – Big Data Super Nova and the Journey Back to Privacy and Security” broadcast is about to begin.

This time I decided to share with you my insights and ideas through a story.
So tighten your seatbelt as we jump into hyperspace through time.
It’s now exactly 37 years forward – the morning of March 13, 2050, 11 AM: Earth Standard Galaxy time.

Previous Chapter is here (Eve)

Chapter Two: Adam

GENESIS – Big Data Super Nova – Part Two (Adam)

Listen to this postListen or Download audio of this chapter here: 

He didn’t like to Wake-on-LAN people, although Eden Industries code of conduct allowed it, in case of emergencies.

As a Naturalist, long term member of the “Real Thing”, the last thing he wanted is any relation to using inter-body Nano bots or other artificial enhancements.

Adam was 27 years old, and preferred to die young, rather than have his body parts replaced by bot parts.

After 200 hundred years or so, no bot can keep your body alive, and you have to mind-beam yourself into the mind cloud, that is, if you discount the choice of artificial body part replacements.

The other option is to use a whole body replacement.

“That’s when you fully stop being a human, turning yourself into a dumb robot”, he thought. You look like a Frankenstein, no matter how precise your human body imitation is.

Of course, your mind is still there, if you don’t mind an emotional glitch here and there, since data loss can happen when your brains is scanned.

“Nothing is perfect, including 3-D brain scanners.”

Thinking about the Nano Mind- Scan bots crawling through his brain, as they turn mind into data, turned Adam’s brown eyes even darker.

“I don’t mind being a bit over-weight and baldy, or even called ‘Slow Thinker’. I won’t let any of those crap bots get under my skin”.

Eve looked always in great shape. But she never had to actually make an effort to look good. Her Muscle Trainer bots did all the work.

Eve was 20 years older than him, and still looked like she was in her early twenties.

“But she was not real. Not a real human being. And still I like her a lot.”

Then he felt ashamed for waking Eve by communicating the alert message, through her Inter Body Bots.

But he didn’t have a choice.

There was no time to wait, as more reports came in, it turned up to be a huge disaster.
He recalled the last time a global alert was broadcast on planet earth.

This time it was worse, enormously worse.

Get additional chapters by reading the book by Clicking Right Here

GENESIS - Big Data Super Nova - Part One (Eve)

GENESIS – Big Data Super Nova – Part One (Eve)

Hey and welcome!

The “Genesis – Big Data Super Nova and the Journey Back to Privacy and Security” broadcast is about to begin.

This time I decided to share with you my insights and ideas through a story.
So tighten your seatbelt as we jump into hyperspace through time.
It’s now exactly 37 years forward – the morning of March 13, 2050, 11 AM: Earth Standard Galaxy time.

Listen to this postListen or Download audio of this chapter here: 

Chapter 1: Eve

Eve woke up.

But as she puffed her dark long hair off her face, she felt strange, a sense of fuzziness.
The inter-body adrenaline injection Nano system did its job and she opened her eyes.

GENESIS - Big Data Super Nova - Part One (Eve)

“No need for Alarms clocks any more”, she thought.

Then she felt as if she lost it.

It was as if her thoughts got encrypted through an SSH5 Gateway so she could not really understand what she was thinking.

Lights came up, as the under-skin environment communication Nano chip sent the message about her awakening to her room’s climate control unit.

But she was still lost.

Her body was still awakening, eyes turned purple, muscles twitching, heart beating faster.
But none of this was in her control. The Nano bots were doing it all.
No one needed to use contact lenses to get their choice of eye color, or go to the sea to get their body’s skin tanned.

Well except the Naturalists of course. They were objecting just any kind of body enhancements.

Then suddenly, as if someone hit the lights on, Eve was really awaken.
Her first thought was “It is too early”.

Then the Red Blinking message appeared right in front of her eyes, blinking, mesmerizing.

“From: Adam@eden-industries.com
Subject: Big trouble Boss
Better get here right away, while you still can.
Adam.”

That’s why she lost it for a moment.
There was no time for the usual gradual wake up.

Adam had probably activated the Emergency wake up process, communicating the secret “Wake-on -LAN” message right into the Nano chips in her body.

“I hate it”, she thought. “And I don’t have time to get to the office.”

“I have to mind-beam”.

Get additional chapters by reading the book by Clicking Right Here