Pride, Prejudice, Production and Cloud-Native Applications

It was those few minutes of twilight before which the sun would cut through the skies and be fully visible, and having just reached home after a tiring journey(and a stressful fortnight full of work before that), the phone rings!

This is one such everyday thing you might have heard from “Admins”, but, why do they always appear to be busy? Why do things behave erratically in the Production Environment? Why do they always have to work on weekends?

Having worked as an Application Administrator(DevOps, ProdOps) for two years, to keep it short — it is not easy.

What do we do?

In brief —

• Keeping the services running (or) better put as Keeping the lights on
• Installation, Configuration, Management
• Monitoring
• Troubleshooting and fixing issues

Additional skill-set required :

• Good knowledge of infra
• Expertise in supported Tech Stack
• A holistic understanding of application being supported
• Scripting
• Agility and critical thinking
• Keeping calm in the most impossible times

Let’s think of this, its a Sunday morning, you were supposed to wake up to birds chirping, spring breezes kissing your forehead while you stare at the feel of twilight, fresh with great vigour; instead, you wake up at 2 AM in the night to a call where you have your business crashing and teams complaining about the degradation in your systems.

You wouldn’t be able to figure out if it was another Hiroshima in the DataCenter, or if it was an Anakrakatoa, but, you would understand that it now is thus on your shoulders to make sure your systems work normally and that business continues to go on perpetually as if everything were normal (thus ensuring that peace prevails).

Pride :

- Critical Thinkers and agile problem solvers

In operations, you have but a few mins to “find the root cause”.In an ocean full of possible problems you are expected to find the Nemo(RCA). Sounds lunatic? well, we admins are capable of doing it and doing it RIGHT. All this must happen in a matter of moments.

Having handled and administered utterly critical workflows for almost two full years, in strongest of possible words I can ascertain that it not only is a challenge for a business to get the outage fixed but, it is also a question on your ability to get it sorted ASAP. In those two minutes of initial observations and speculations, you must distil apart the possibilities, known points of failure, the absolute no’s and mere certainties. You must calculate the impact radius and handle the pressure WHILST you continue to identify the root cause and also think on possible fixes.

In fell clutch of circumstance, my head is bloody but unbowed — every admin ever

- Creative thinkers and team makers

In many outages, what helps? It is not money, it surely is not always just luck but, it often is your team. If you are a developer you might own responsibility for a certain artefact, but, in operations, it is not an individual that resolves the issue, it is the Ops-Team and hence the responsibility is thus shared, always. Only together we stand a chance against the dreaded mutual enemy — Outage.

By nature, outages are unequivocally tricky, exceptionally unique, and incomparably brutal. What makes it more disastrous is the overarching complexity that grows with the system over time. To settle down such unrest, it is impossible to always stick to convention. To go out of the way and fix issues comes but comes with “thine nature” and thus, to thrive in Ops, one must and must be compellingly creative.

Teams maketh Ops

- FIREFIGHTERS and PEACEKEEPERS

Multi-Taskers by phenomena, admins are also responsible to ensure the state of systems; And thus a task of arduous automation with humongous abstraction is thrust on every admin, by none but their own selves. This helps in peacekeeping. When peace disrupts in the darkest of the nights and fire spreads over the realms of production, like a superhero, like a ghost from a tomb and a baby from a womb, every admin joins the production bridge. But certainly, not in those cloaks of superheroes, every admin jumps to fight the fire and win over it.

Prejudices :

- “It is easy”

Even for those who understand the length and breadth of the underlying systems, to an extent it is easy, but, largely, it is quite challenging. Although with cloud, cloud-native applications and cloud-native architectures, the dependency does come down by an exponent, yet not the complexity.

-”But, give me the root access and I can do it too”

Very accurate, boy, thou art. But, are you ready to take a dip on your productivity? It takes more than root access to get things done right. Installation, configuration, management, setup, directory structure, and access management, server maintenance, network operations, and at least a million more things go into being Ops.

“With great power comes great responsibility”

“It worked in my system”

It did? sure, it might have. There is one certainty with production environments. They are like the first element of every group of the periodic table. The first element is always bizarre in its properties and reactions. To a great extent, all this can be solved by containerizing, that being our next point of debate.

“I will put it on AWS/GCP/Azure/VMware Cloud”

Although there is incomparable ease and smoothness in clouds, it is still a fact that you are only off-boarding a fixed set of operations to the public cloud vendor. This is by far the best “win-win” situation.

“War rooms?”

Why do we need a war room to troubleshoot problems, is there any science behind it? The answer is yes. War rooms are battlegrounds. The common enemy(OUTAGE/ISSUE) stands tall initially, but, with right coordination between the various teams, all adversities shall be bulldozed.

Another example of criticality -

versus

Production DevOps, Docker, Kubernetes and CNA!

All that we have seen thus far is a foundation for what we are now about to witness. If the IT operations was a cinema, “VIRTUALIZATION” was the first plot, happy times and peace. But, as Yuval Noah Harari says,

“One of history’s few iron laws is that luxuries tend to become necessities and to spawn new obligations.”

the architectures, maintenances, dependencies etc. were troubling the protagonist of the show, “DEVELOPER/IT WORLD” and thus, like “Samaritans” admins came for help. But, oh boy, admins were out of weapons by the time it was the interval that moment, that very frame is the advent of Docker.

Docker changed things, perhaps for ever.

Kubernetes is the “Aparaajita” thus far in the context of DevOps, a philosophy set ablaze in recent times. Kubernetes is leading the way towards an ideal and sheer near 100% automation.

All those problems we would face in the administration would vanish, in the container world. Especially with an orchestration tool like Kubernetes, containerization becomes smooth. This also paves a way to a change in the ideology of application development and urges us gently to go stateless.

Kubernetes also a revolution to the legacy mindset not only from an application standpoint but also from infra, storage and network perspectives.

But at the laste, as every thing hath ende

Let us look forward to drill deeper into each of those in the forthcoming chain of stories!

Please note, this is not just an ode to the admins but is also a reflection of what goes behind the curtains.