Managing Inventory, Change and Compliance

Are you responsible for managing a large collection of resources in AWS? Resources that other people often change without you even knowing about it. You have got a list of rules that your infrastructure has to comply with; perhaps HIPAA. With all those change going on it is almost impossible for your team to keep track of their compliant state on their own. Your brain feels like it is about to explore.

Generally there are three parts of problem we hope to solve with whatever tool we pick.

• First, we need to know at any given time if our infrastructure is compliant with whatever rule is relevant for our organization.
• Next part of the problem is resolution. That is bringing your overall system back to a compliant state as soon as possible. Especially for compliant rules that deal with security, the timeliness of this action is essential.
• Finally after we fix the issue, the next part of the problem comes in. Determining at what point and through what actions your system became non-compliant.

So what tools are there for us to solve these problems? Inventory and compliance management space is pretty mature at this point. It is a problem that’s been there for a very long time. So there are quite a variety of tools you can use to help solve this problem. I would like to break them down in three categories:

• Enterprise-focused tool.
  • Example: Solarwinds, Spiceworks, Microsoft SCCm.
  • All-in-one solution.
  • Expensive.
  • OS-Specific.
  • Good for large organizations.
  • Not so goof for small organizations.
• DevOps-focused tool.
  • Example: Puppet, Chef, Ansible.
  • OS-agnostic.
  • Not all-in-one.
  • Integrating can be difficult.
  • Often open-source.
  • Highly automatable.
  • Benefits beyond inventory/compliance.
• Fully-integrated tool.
  • All-in-one solution.
  • Small or large organization.
  • Highly Automatable.
  • Trusted vendor.
  • Pay for what you use.
  • Made for the cloud.
  • Scale near infinitely and handle familiar resources with no changes required in your part.

Any of the three categories is better than no inventory tool at all. But in my option the third category is the best bank for your buck. And one of the best fully integrated tool is: AWS Config.

I will elaborate AWS Config in our next post. So stay tuned and watch out for it.

Read Full Post »

Demystifying DevOps : Difference between Agile and DevOps

Recently organizations are embracing DevOps which is a great thing. However the whole adoption is causing a lot of confusion as well. Some of you might have heard the term “Agile and DevOps”. With that it looks like Agile and DevOps are different. To over-simplify further people assume Agile is all about processes (like Scrum and Kanban) and DevOps is all about technical practices like CI, CD, Test Automation and Infrastructure Automation.

This is causing a lot of harm as some organizations now have Agile and DevOps as two separate streams as part of their enterprise Agile transformation. Agile by definitions disrupts silos and you see, in this case people are creating new silos in the name of Agile and DevOps.

With that background in mind, let’s try to understand what exactly DevOps is all about:

• DevOps is mainly the widening of Agile’s principles to include systems and operations instead of stopping its concerns at code check-in. Apart from working together as a cross-functional team of designer, tester and developer as part of an Agile team, DevOps suggests to add operations as well in the definition of cross-functional team.
• DevOps strives to focus on the overall service or software fully delivered to the customer instead of simply “working software”.
• It emphasizes breaking down barriers between developers and operations teams, and getting them to collaborate in a way where they benefit from combined skills.
• Agile teams used automated build, test automation, Continuous Integration and Continuous Delivery.
• With DevOps that extended further to “Infrastructure as Code”, configuration management, metrics and monitoring schemes, a tool chain approach to tooling, virtualization and cloud computing to accelerate change in the modern infrastructure world. DevOps brings some tools on the block as well like configuration management (puppet, chef, ansible), orchestration (zookeeper, noah, mesos), monitoring, virtualization and containerization (AWS, OpenStack, vagrant, docker) and many more.
• So you see DevOps is not a separate concept but a mere extension of Agile to include operations as well in the definition of cross-functional Agile team, collaborate together and work as one team with an objective to delivery software fully to the customer.

Creating separate Agile and DevOps horizontals in any organization just defeats the whole purpose (removing silos) of DevOps.

Read Full Post »

Is DevOps a good career?

It depends on what you are passionate about. I would choose passion over all other considerations or else you will burn out and the money won’t help. Any position you pursue in tech is going to require your complete dedication to achieve success.

However, I think the DevOps area has a lot of growth potential in the future. While I cannot speak to your interests I can tell you why I am drawn to the field after being a web based software developer.

I have learned through that if you are operations staff you’d better be automating your job, and if you are a developer you have to face the inevitability of getting down and dirty with operations if you are to stay relevant. Developers who won’t administer/monitor and admins who won’t develop will increasingly become less and less valuable to organizations needing to stay competitive.

DevOps is exciting because you are always working with and integrating new technologies and solving new challenges. Essentially your job is to find a happy balance between operations and developers. This relationship is delicate and can blow up if not regulated. As a devops specialist your job is to integrate these two different mindsets. This requires that aspects of IT be securely shared so that you don’t have the blame game. Developers need to continually push code and operations want to keep everything running smoothly. The more integrated the systems and processes in use, the easier it is for each to do their job.

I personally like to think of IT as three separate phases that all contribute to the ultimate success of the enterprise tech ecosystem; packaging, automation, and scaling.

Packaging:

DevOps is great if you like to explore and work with a variety of technologies and processes. I think one of the first things to consider is the packaging of IT that the tech teams use to provide the organizations products and services. The better packaged and more malleable the packaging the easier it is to keep everything standardized and reusable.

If you like playing with configuration management systems (Puppet, Chef, Ansible, etc…) and digging into imaging systems such as Docker you will like DevOps. I would caution that it is very important to create highly configurable packaging of the IT systems in use so that they can evolve as the organization’s needs change. This also makes it easier to modify for production, QA, staging, and development environments.

If you think about it, the amount of new technologies and services being released into the market is growing exponentially (especially with the add-on potential of all the open source frameworks in existence). In DevOps no technology is off limits and you find yourself continuously working with, integrating, and automating different technologies. As the amount of tech and services grow, so is the demand for people who can put it all together into golden images (configuration managed images on different environments).

Automation:

Your automation potential is only as good as your ability to package the infrastructure in a form machines can work with effectively. If you come from a development background you most likely have had to deal with brittle environments (at least in testing new technologies, which you should be continuously doing).

The DevOps specialist makes it easy for programmers and operations to automate their jobs so that we don’t have to reinvent the wheel over and over again. Ultimately, if the automation is good enough we can realize a scalable architecture (which is the end goal).

You should like scripting a lot. You don’t need to be the best programmer to accomplish this but the more integrated your approach the easier it will be to build on your previous work (which I like). Automation brings the machines to life and if you like seeing a bunch of moving pieces come together to achieve some measurable outcome you will like this part of the job.

I would recommend that you know at least one glue language: Python, Ruby, Go, etc. The more flexible the language, the better. Although, the beauty of automation is that many different languages can be brought together to create a unified system. If something needs to be built for speed, it’s easy enough to design that part in a language like C or Go while allowing other tasks that need more flexibility to be written in a higher level language. You definitely want to become very good at shell scripting which many times ties everything together.

I personally cannot see automation becoming less in demand in the future. The promise of the cloud is built on automation, and enterprise usage of the cloud is growing rapidly throughout organizations of all sizes and types.

Scaling:

If reusability is a passion of yours, I think you would definitely like DevOps. I believe the biggest factor in the successful tech organizations of the future will be their ability to scale rapidly while being able to deflate when not needed to minimize costs in downtime. Customers want speed. They don’t care about the tech behind the application as long as the application is reliable, zippy, and meets their needs.

If you can create packages of IT that can be easily automated in a portable fashion then I think you will have great prospects in the tech world in the future. Companies like Google and Facebook would never have gotten as popular as they are if they had not learned to scale their IT effectively.

Scalability is not easy to achieve and many would rather not have to worry about it, which explains the growth of scalability as a service offerings. But somebody has to know how. Think about the problems of the future: Data analysis, AI, internet of things, mobile consumption, scalable web driven apps, etc. While all of these tech areas require different skills to develop on their own, each is absolute garbage without the same fundamental building blocks. Want to jump from mobile to AI? DevOps could allow that. Want to play with that new SaaS service that is all the rage these days? DevOps can allow that.

DevOps is about being the glue that holds everything and everyone together, and to me that is what makes it so exciting. The possibilities are limitless and the technologies are always growing and evolving. And if you don’t focus on DevOps, you will still have to manage infrastructure as a developer anyway.

When I first started programming I started with a passion for machine learning in C, then over time I started creating web sites using asp.net for my organization. Over time I felt suffocated by the limited nature of the technologies people expected me to work with day in and day out. It wasn’t that I did not like the technologies, but I felt like I was in single technology hell. And once you gain a lot of experience in a specific technology or system people just expect you to focus on that area: recruiters, managers, developers, everyone. With DevOps variety is part of the job description so if you ever feel trapped by technology and find yourself looking to the stars wondering what the hell did I get myself into, DevOps can free you from that limited mindset.

That is why I got into DevOps. I certainly don’t claim to be the best or the most experienced out there, but I am a heck of a lot happier these days. And I am constantly learning new things that I can apply to any new project, whether it is a new AI platform or a mobile application.

At the end of the day your happiness and the passion you feel for what you do is all that really matters.

Read Full Post »

Definition of DevOps

DevOps is one of the hottest topics in the technical world for some time now. When I tried to dig out the real reason behind it, I came to know that it is because it hurts the most. I always believe that the thing you should fix first is what hurts you the most. Historically, the inability to produce code that our users wanted within budget and in a timely manner was the biggest pain.  Development took months, if not years, and at the end of all that time and effort, what we delivered was not what our users expected. Our development process had to be fixed first because that was what hurt most then. Because the process of software development was so bad, the quality of the code was also in question. So the need for innovation on the system side of technology work was increasing. And then a giant was born: DevOps.

When I tried to search and find out the real definition of DevOps, I found out hundreds of different ones. Some of them are really long. But if you ask me personally, I would go for one of the simplest ones to define DevOps.

“DevOps is the union of People, Process, and Products to enable Continuous Delivery of Value to our End Users.”

The writer is very deliberate in the terms used in this definition. He chooses value over software. DevOps is not just automating a pipeline so we can quickly deliver software. Our goal is to deliver value. The term “End Users” was also very carefully chosen. The value we produce must reach our end users. If the value only reaches the Development and QA environments but is held up before reaching production where it can be realized by our end users, we are still failing.

It is very important to realize that DevOps is not a product. You cannot buy DevOps and install it. DevOps is not just automation or infrastructure as code. DevOps is people following a process enabled by products to deliver value to our end users.

The best way to define DevOps in depth is to use a parallel method to the definition of a similarly complex term, agile development.  Agile development, according to the agile manifesto, consists of four different “levels” of concern.

Agile Values: Top level philosophy, usually agreed to be embodied in the Agile Manifesto. These are the core values that inform agile.

Agile Principles: Generally agreed upon strategic approaches that support these values.  The Agile Manifesto cites a dozen of these more specific principles. You do not have to buy into all of them to be Agile, but if you do not subscribe to many of them, you are probably doing something else.

Agile Methods: More specific process implementations of the principles.  XP, Scrum, your own home-brew process – this is where the philosophy gives way to operational playbooks of “how we intend to do this in real life.” None of them are mandatory, just possible implementations.

Agile Practices: Highly specific tactical techniques that tend to be used in conjunction with agile implementations.  None are required to be agile but many agile implementations have seen value from adopting them. Stand-ups, planning poker, backlogs, CI, all the specific artifacts a developer uses to perform their work.

DevOps Values: I believe the fundamental DevOps values are effectively captured in the Agile Manifesto – with perhaps one slight emendation to focus on the overall service or software fully delivered to the customer instead of simply “working software.”

DevOps Principles: There is not a single agreed upon list, but there are several widely accepted attempts. “Infrastructure as code” is a commonly cited DevOps principle. I personally believe that DevOps at the conceptual level is mainly just the widening of Agile’s principles to include systems and operations instead of stopping its concerns at code check-in.

DevOps Methods: Some of the methods here are the same; you can use Scrum with operations, Kanban with operations, etc. (although usually with more focus on integrating ops with dev, QA, and product in the product teams). There are some more distinct methods, like Visible Ops-style change control and using the Incident Command System for incident response. The set of these methodologies are growing; a more thoughtful approach to monitoring is an area where common methodologies have not been well defined, for example.

DevOps Practices: Specific techniques used as part of implementing the above concepts and processes. Continuous integration and continuous deployment, “Give your developers a pager and put them on call,” using configuration management, metrics and monitoring schemes, a tool chain approach to tooling. Even using virtualization and cloud computing is a common practice used to accelerate change in the modern infrastructure world.

In the end, DevOps is a little tricky to define, just like its older brother Agile. But it is worth doing. “I do what Scrum book says so I am doing Agile” is as erroneous as “I am using Chef so I am DevOps”. To be a successful Agile or DevOps practitioner it is really important to understand all the layers that go into it, and what a given DevOps implementation might contain or not contain. In the end, what DevOps hopes to bring to Agile is the understanding and practice that software is not done until it is successfully delivered to a user and meets their expectations around availability, performance, and pace of change.

Read Full Post »

Not DevOps

I am seeing around me that lots of people have a serious misconception about what is DevOps and what it is actually not. So today let us try to figure out what is not DevOps.

Not NoOps

It is not “They are taking our jobs!”  Some folks think that DevOps means that developers are taking over operations and doing it themselves.  Part of that is true and part of it is not.

It is a misconception that DevOps is coming from the development side of the house to wipe out operations – DevOps, and its antecedents in agile operations, are being initiated out of operations teams more often than not.  This is because operations folks have realized that our existing principles, processes, and practices have not kept pace with what is needed for success.  As businesses and development teams need more agility as the business climate becomes more fast paced, we have often been providing less as we try to solve our problems with more rigidity, and we need a fundamental reorientation to be able to provide systems infrastructure in an effective manner.

Now, as we realize some parts of operations need to be automated, that means that either we ops people do some automation development, or developers are writing “operations” code, or both.  That is scary to some but is part of the value of the overall collaborative approach. All the successful teams I have come across have both people with deep dev skill sets and deep ops skill sets working together to create a better overall product. And I have yet to see anyone automate themselves out of a job in high tech – as lower level concerns become more automated, technically skilled staff start solving the higher value problems up one level.

Not (Just) Tools

DevOps is also not simply implementing a set of tools.  One reason why I feel that a more commonly accepted definition of DevOps is needed is that having various confusing and poorly structured definitions increases the risk that people will pass by the “theory” and implement the processes or tools of DevOps without the principles in mind, which is definitely an anti-pattern. Automation is just the exercise of power, and unwise automation can do as much damage as wise automation can bring benefit.

Similarly, Agile practitioners would tell you that just starting to work in iterations or adopting other specific practices without initiating meaningful collaboration is likely to not work out real well. Sure, a tool can be useful in Agile (or DevOps), but if you do not know how to use it then it is like giving an assault weapon to an untrained person.

But in the end, fretting about “Tools should not be called DevOps” is misplaced. Is poker planning “Agile” in the sense that doing it magically gets you Agile?  No.  But it is a common tool used in various agile methodologies, so calling it an “Agile Tool” is appropriate. Similarly, just because DevOps is not just a sum of the tools does not mean that tools specifically designed to run systems in accordance with a DevOps mindset are not valuable.

Not (Just) Culture

Many people insist that DevOps “Is just Culture” and you cannot apply the word to a given principle or practice, but I feel like this is overblown and incorrect. Agile has not helped thousands of dev shops because the work on it stopped at “Culture,” with admonitions to hug coworkers and the lead practitioners that identified the best practices simply declaring it was all self-evident and refusing to be any more prescriptive. We might be able to figure out all those best practices ourselves given the cultural direction and lots of time to experiment, but sharing information is why we have the Internet.

Not (Just) Devs and Ops

In the end, it’s not exclusionary.  Some people have complained “What about security people!  And network admins!  Why leave us out!?!”  The point is that all the participants in creating a product or system should collaborate from the beginning – business folks of various stripes, developers of various stripes, and operations folks of various stripes, and all this includes security, network, and whoever else.  There are a lot of different kinds of business and developer stakeholders as well; just because everyone does not get a specific call-out (“Don’t forget the icon designers!”) does not mean that they are not included. The original agile development guys were mostly thinking about “biz + dev” collaboration, and DevOps is pointing out issues and solutions around “dev + ops” collaboration, but the mature result of all this is “Everyone Collaborating”. In that sense, DevOps is just a major step for one discipline to join in on the overall culture of agile collaboration that should involve all disciplines in an organization. So whoever is participating in the delivery of the software or service is part of DevOps.

Not (Just) A Job Title

Simply taking an existing ops team and calling them “The DevOps Team” does not actually help anything by itself; neither does changing a job title to “DevOps Engineer.” If you do not adopt the values and principles above, which require change at an overall system level not simply within a given team, you would not get all the benefits.

However, I am not in the camp that rails that you cannot have DevOps in a job title. It is often used in a job title as a way to distinguish “New style DevOps-thinking, Automation-first, Dev-collaborating, CI-running, System Admin etc.” from “grouchy back room person who aggressively does not care what your company does for a living.” Some people find value in that, others do not, and that is fine.

Not Everything

Sometimes, DevOps people get carried away and make grandiose claims that DevOps is about “Everything Everywhere!” Since DevOps plugs into the overall structure of a lot of lean and agile thinking, and there are opportunities for that kind of collaboration throughout an organization, it is nice to see all the parallels, but going and reengineering your business processes is not really DevOps per se.  It is part of an overall, hopefully collaborative and agile corporate culture, but DevOps is specifically about how operations plugs into that.  Some folks overreach and end up turning DevOps into a super watered down version of Lean, Agile, or just love for everyone. Which is great at the vision level, but as you march down the hierarchy of granularity, you end up mostly dealing with operational integration – other efforts are worrying about the other parts. But there are still a lot of unsolved problems around the delivery of software and maintenance of services and making it fast, reliable, secure, et al. – if someone wants to use what they have learned from DevOps to go be a larger scope corporate consultant that is fine, but most people involved in DevOps are technical practitioners who are looking for better ways to do their job, not someone else’s.  In Agile there is “Agile Software Development” and then there is the larger Agile organization work. I think DevOps is best defined as “Agile Software Delivery and Operations,” which should similarly work in concert with others working on larger organizational initiatives, but without losing sight of its primary value proposition for the organization.

Read Full Post »