A few weeks ago I participated in a panel for Continuous Discussions organized by Electric Cloud where we discussed how organizations can support development velocity downstream of development teams. You can watch the discussion to get everyones thoughts but I wanted to elaborate a bit here on my own thoughts around these topics.

You can watch the discussion here:

The general idea in this discussion was that while we can optimize development workflows to improve velocity for those teams, work often gets held up downstream of development during testing or deployment into production. We wanted to focus on what can be done to improve downstream throughput.

In general the topics covered were as follows:
* Culture and Team Structure
* Value Stream mapping
* Optimize inter-team handoffs
* Automate³
* Version & share binaries, environments and processes
* Blameless post-mortems

My $.02 starting point

As an industry we’ve become really good at making things faster & more iterative but we haven’t yet figured out how to make light move faster. Most work toward improving the speed of a thing involve eliminating waste – improving efficiency & effectiveness. You’ll probably hear this theme over and over in my responses below but the simplest way to improve velocity through a series of steps is to eliminate steps & let computers do the ones you cannot eliminate. It’s not always possible – but should alway be considered a possibility.

Culture and Team Structure

This, in my view, is the most important aspect of the overall velocity of development. Assuming that we view “development velocity” as the time it takes from an idea being prioritized so that someone can spend time on it, to the time a customer can touch it, then we have to consider the entire delivery pipeline when we evaluate our velocity.

If we look at an entire delivery pipeline and all the handoffs that might occur, there are a lot of opportunities for waste. We could focus on making handoffs more efficient or we could focus on how we automate them – which often mostly eliminates the traditional idea of a handoff. This is usually a core objective of Continuous Deployment.

The most effective and high velocity team structures I’ve seen are those where developers have the ability to make changes which have a fully automated pipeline to production. Typically putting something like this in place requires that the development team be a cross-functional team with development, testing, product, operations and leadership expertise. The more of those functions you remove from the immediate team, the less efficient the team becomes. Whether that means they have lower velocity or are less effective depends on the organization – but there’s a good chance those traits are impacted as well.

When a development team has representatives from the areas of the business they need to interact with on the team & have an ability to rapidly experiment with changes to their software in a production environment then their ability to make accurate and timely decisions about product changes is vastly improved.

Value Stream mapping

Watching the video you’ll see that I don’t spend much time on this topic because I’ve never really done much of it. I have, however, listened to others talk about how effective it can be at bringing groups together to talk about and understand what happens at each stage of a delivery pipeline. Again, I refer to a “delivery pipeline” as the entire process of getting some idea into customer hands. If you are a software company, your delivery pipeline includes nearly your entire organization. It is very rare for a single person to have a complete and accurate view of all the steps that occur in this process – so bringing folks together to document the current state of things can bring about some surprising results.

Have you ever watched the making of Jelly Belly beans and thought “Holy Cow, I had no idea there were so many steps in making a Jelly Bean!”? You might be surprised how little you know about your own processes. You might also be surprised to find out who in your organization is crucial to those processes completing correctly & on time. Are those folks doing ok?

Watch Jelly Bellys get made!

Optimize Inter-team handoffs

As I mentioned above, my first approach to this problem is to eliminate the handoff, but this isn’t always possible. When it isn’t possible, the above value stream mapping exercise can help you understand what opportunities there are for automation. Often the most time consuming part of any handoff between groups is discussion & establishing trust. If I am receiving a thing from a group I trust a lot & I have received things from them in the past with great success, I’m probably not going to require a lot of discussion. On the other hand, if every thing that group has handed off to me has erupted into a tire fire, then I will probably have some questions. Worse, my distrust may be based on nothing more than my own prejudice – making it a difficult situation to correct.

Trust is a fickle thing for a company because it’s usually between two individuals. There can be trust between groups, but that trust is often contingent on each groups membership – as members change then so too does the trust level. Humans are also imperfect – they forget, they have bad days, and sometimes they just don’t show up. For this reason it’s more effective to automate any of these decision points you can.

When you automate a handoff you could be simply documenting a set of possible outcomes and the conditions for each – like a protocol for handling things. This allows more autonomy and flexibility in handling the handoff – if the documented expectations are met – a set of steps can be followed to complete an action. You could also be adding some technical automation by having computers make some of the decisions for you & maybe do some of the work (if they aren’t already).

When a development team implements continuous delivery – they are essentially automating the handoff to Operations by saying “We’ve tested & met these set of expectations – this software will work if you press this button and deploy it”. When a team implements Continuous Deployment they’re taking it one step further and saying “We are taking more responsibility for the software and largely eliminating the handoff – we’ll both watch for problems and work together to resolve”.

Automate³

Automation – “the use of largely automatic equipment in a system of manufacturing or other production process”

Automation takes that process you repeat over and over and allows some computer/mechanical processor to handle it. You can automate decision making (automated test suites/continuous integration) and you can automate actions (automated provisioning/deployment). I’m a super duper big fan of automation – it’s much of what people think I get paid to do. But the act of automating something you understand is not the hard part – or even really the interesting part to me anymore. The interesting part is understanding where you can apply automation, and then understanding how to tweak things here and there to make it possible to automate them.

An example of this is Feature Toggles. I can implement the automation to deploy the software a team builds on every commit. You push a commit and like magic – computers will whisk your code on out to production… where it will promptly provide a fine example of why “Continuous Deployment is BAD”. Outages, dying kittens, it all happens and then “Stop! We need a change control process!”. The issue isn’t that automating software deployment is hard, the issue is that making the software deployment automatable in a way that allows a high degree of success is hard.

Feature Toggles (Flags, Flippers, Switches, whatever) allow you to make changes to software while maintaining existing customer-facing behavior. This means that you can try some new thing in a Continuous Deployment environment in a way where customers should never be impacted. Yes, there are caveats – but it’s beside my point. The capability to automate the software deployment is contingent on a capability of the software, not the deployment automation.

There’s a similar dependency on automated testing, not because test quality is required – it’s required either way – but because testing must be automated in order for Continuous Deployment to even be possible. If I have to wait for Jimmy to finish his testing before my commit can go out to production – then by definition that’s not “Continuous Deployment”.

So automation is important – but not always in the way people think. Folks think they’ll just hire a DevOp to DevOp the heck out of this problem and automate it all away – but unless you also have a dev team that has built software capable of being deployed in this way, it’s not gonna happen.

Humane Software systems

This is a point I bring up during the discussion and I wanted to touch on it. If you haven’t, you should go and watch Jeff Hackert’s talk (below) on building humane systems. I see this failure so often and it’s truly avoidable if we treat our automation systems like we treat our products – building them with an understanding that our customers are humans, and they have feelings & experiences that differ from ours, and getting their feedback is good. I can’t add much to the talk – watch it.

Version & Share binaries, environments and processes

This is perhaps more obvious to some than others – this is table stakes for building a modern development workflow. Very often the “environments” and “processes” pieces are harder than others – especially when different groups handle the production environment vs. the development environment. If you can make them the same – do it – but I have yet to be somewhere that can do this 100%.

My preference is to focus on the production environment. This is more meaningful in a SaaS environment than perhaps someone building desktop software, but I think it’s legitimate to consider either way. The effort involved in completely replicating your production environment – the traffic, the processes, the systems, the network, everything – is… so. much. work. I would argue that in many (most?) cases you can put that same degree of effort (or less) into making it possible to evaluate software changes in production. Further, by doing so you take advantage of all the organic variations which are experienced in a production environment.

Much has been written about doing this – for your googling pleasure take a look at the idea of a “dark launch”. The basic premise is exposing a system to production traffic in a way which doesn’t put customers at risk. The process is too involved to discuss here – but it’s a way to leverage the existing environment and traffic you have to evaluate changes in a way that provides better realism than any test environment I’ve seen while still allowing experimentation.

Evaluating changes in a production environment means you are using the same process, the same systems, the same network & ideally the same traffic patterns to evaluate the results of change. And as those things evolve, so too does your testing environment.

Blameless post-mortems

This is a big one for me. I prefer retrospectives, not everyone does, but I like to look back on the good and bad to learn from the past on a regular basis. There are times when an event was clearly bad, but there are usually positive things that happened that you can acknowledge and enforce during a retrospective.

That said, the “blameless” part is super important. If you want to bother to look back on what went wrong and what didn’t then it’s worthwhile to invest in making sure everyone is honest and open. This is hard because different people have different tolerance for criticism – so we minimize it. I’ve written in the past about this as have others – I’d suggest further reading if you aren’t familiar with the idea.

This impacts velocity by identifying areas where improvements can be made & identifying where there is friction today. If you aren’t looking at the past then it’s much more difficult to improve the future.

Wrapping up

I care about these topics a lot and I always look forward to chatting with others about their different experiences. If you want to talk about any of this – just give me a shout.

I am constantly pulled toward this idea, an idea I’m beginning to think of as delusional, that you can pivot the cultural direction of a company without that company first hitting rock bottom – or their version of it. I think if you are a CEO this doesn’t apply – but I’m not. Companies who do not naturally follow a particular trajectory also don’t respond very favorably to changing direction toward that trajectory. The existing direction is there for a reason, because the majority of the company, for whatever reason, has made decisions large and small which pointed the ship that way. Changing that direction requires changing hearts and minds (or staff) toward some new horizon.

insert over-used and possibly inaccurate Einstein quote here

On the other side of the coin – I’ve worked for companies where stuff worked pretty OK. I’ve participated in what I thought was a reasonably good version of a high functioning team. I left that company because I got bored, or just didn’t feel like the company itself was giving me the opportunity to do what I wanted. What I wanted was to grow a team into that high functioning place.

So I found myself recently being frustrated that my attempts to change a team aren’t meeting my expectations. Unlike me from a number of years ago I’ve become a bit more introspective about things and so I started asking myself some questions:

1. What is your basis for expecting people to agree with you and change in the way you think they should within some window of time? Is there evidence supporting this expectation?
2. Would the job be interesting if they all turned on a dime and started doing the right thing naturally tomorrow? No more problems to solve?
3. Assuming they too want to succeed and do their best, what pressures are preventing them from changing?
4. Are you being an asshole about it?

Item #1 is that I have expectations that have been established on assumptions not in evidence. This is much the same as being frustrated that when you let go of a bowling ball it falls to the ground – but you expected it to go up. Have you ever observed an object fall up? Why would you expect it? Wanting it, thinking it’s so obvious that it should, does not change physics. Few folks get all bent out of shape when it rains, you can’t control it, you accept it. So too are other people – cannot be controlled, have to be understood and accepted, no real value in getting frustrated.

Teams do change, and there are many stories of teams changing. What is often missing from those stories is some perspective on the effort, time, and patience required to get to that end. This is usually measured in years unless there’s dramatic (and usually very disruptive) action. So, patience.

Item #2 is why I’m here in the first place, this is what I want. I could go work for some company that already has their shit together but what fun is that? Nope, I need to learn how to find joy in this journey and how to become successful at it because this is what drives me. I need only look at my selection of books in amazon for evidence of where my interests lean. Improving my ability to do this means being in an environment where things aren’t right & practicing. When you want to improve your code you find a problem to solve – same goes here.

Again, patience. Whether I like to believe it or not, I’m changing & learning as much as the rest of the team. By adjusting my behavior, others adjust theirs – we learn and adapt to each other, we build rapport. As this happens it unlocks new opportunities and makes possible things that weren’t before.

Item #3 is my fundamental belief that all people want to do good. With few exceptions, people are driven by the same desires I am – to do well & succeed at their goals. When we observe people who aren’t doing what we expect them to do, we have to ask why. Sometimes they don’t even know why, so we have to allow them to show us why. Sometimes what we expect isn’t correct – so we have to be willing to learn.

This is also a core tenant of how I believe this process can work. There’s another blog post brewing about building a “pit of success” – about making the right thing the easiest thing. Doing this requires setting aside your belief about what works & observing and learning from behavior of others. It requires lettings things go wrong & then asking how it can be improved.

Item #4 is my reality check. I can absolutely get passionate, maybe even dogmatic, about how I think things should happen. Given enough resistance I can turn into a righteous asshole. Do you listen to people like that? I don’t. I’m not doing anyone any favors if I’m not listening, learning, and asking how I can help people do well.

I have to sit down with people and ask how I’m doing. I have to check in and make sure I’m not destroying relationships. It shouldn’t be necessary to create adversaries in this process – if we are listening & learning we should be finding common ground. This takes effort, but it’s worth it.

All of this gave me some perspective to look back over the last 6-8 months and what progress has been made. Actually, there’s been a lot. There are definitely some problems, but there’s now evidence that the play dough actually does move – it’s not rigid. The process might not be as fast or as direct as I expect, and it might not even look like what I expect when we get there – but there is movement. I need to realize that it is this movement & the positive results of it that are my journey and when we reach our destination, so too does my journey end and I go looking for another tribe to walk with.

No sense in rushing to that point.

I’ve pushed version 1.0.0 of Docker spoon. I moved the version to 1.0 for a few reasons – the release is pretty significant:

• This release pulls in a version of docker-api (0.17) which is not backward compatible with older docker versions. If you need to run with older versions of Docker, use 0.8.x.
• I’ve added a ton of test coverage using cucumber and aruba. The tests require a local docker daemon but otherwise should be portable. This is the most thorough coverage spoon has ever had.
• The original framework used to build spoon – methadone – has been completely removed. This was done because it was getting in the way of setting up a correct default / config file / command line argument parsing hierarchy.

So what’s new?

• New commands including --kill and --restart to ease stopping spoon containers without destroying them or having to use the docker cli.
• Support for config/cli defined port specifications both for docker-forwarded ports or ssh-forwarded ports. This supports both dynamic and 1:1 port mappings for docker ports.
• Privileged mode support when creating new instances.
• Config support for a variety of docker params including:
  • cpu shares
  • memory limits
  • dns servers
  • volume mappings
  • privileged mode

The release has been tested against Docker 1.4 and is known not to work with docker 1.2.

Whether you pair or not I think this post is worth reading. Do you ever have needs like this?

• Ability to create on-demand dev environments with all your dependencies in place?
• Ability to collaborate with local or remote folks in a low latency environment that doesn’t suck?
• Want bare metal performance instead of using a VM?
• Want to kick off that build, close your laptop and head home to check on it later?
• Want to share your work on your dev environment w/ others easily, even with your laptop in your backpack?

There are a bunch more reasons you might like docker-spoon, those are just a few.

This tool grew out of a desire to streamline developer workstation setup but it went even further to the point where developer environments are disposable, can be provisioned in seconds & allow local and remote pairing at any time. Are there other ways to do this? Yep. I’ve seen this tool used successfully by entire dev teams at two companies now – it’s new to you but it’s been around a bit.

The idea behind docker-spoon is pretty basic, you create a docker image with all your developer needs built into it, then you run spoon <name> where name is the name you assign to a new spoon instance running in Docker. Spoon does all the heavy lifting – currently including:

• Check to see if the container exists, if not create it
• Forward any exposed ports
• Copy any files you want copied on creation into the new instance & add your ssh public key for future connections
• Run any commands you want run on container creation
• establish an ssh connection into the container

So after installing spoon and creating a config, the time from running spoon to being at a command prompt in your new dev environment is about a few seconds. How easy is it? Demo time!

Video can be viewed directly here

There are some features not shown in this demo which already exist:

• Forward ports over ssh which aren’t exposed via ssh
• Run commands on container creation
• Using spoon to execute docker builds

That demo used the example docker container included in the docker-spoon repository. If you want to try it out quickly just follow the directions in the docker-spoon repo.

So here’s the thing, there are some caveats – all this awesomeness doesn’t come without some conditions. Spoon takes advantage of the idea that working in a terminal with tmux is low latency & easily shared. There is some work being done to use VNC inside spoon to allow for the use of GUI apps but that’s not the optimal use case.

If you find docker-spoon useful let me know. If you want to see something different, submit a pull request or shoot me a note. The usage should be thoroughly documented in the README.

Spending nearly my entire working day (and many evenings) with Open Source projects I don’t maintain I have come to form an opinion or two on the use of OSS – specifically when it comes to contributing to projects.

We’ve come a long way as a community with tools like Github, making it easier and easier to get an OSS project out in the wild and accept contributions. As we’ve done this though, projects seem to support users with varying degrees of success. Some projects flourish, graciously accepting community contributions. Other projects do quite well accepting feedback but largely having maintainers drive the direction of the project. Still others fall to extremes of accepting every pull request that comes along with reckless abandon or apathetically allowing the project to rot and become another abandoned github relic.

Along with this more and more companies are building their businesses on OSS. Not just a piece here and there, but entire complex ecosystems of software maintained by all kinds of different folks. Each of these business have their own unique quirks, personalities, and constraints.

I’ve worked for a few of these businesses and I know that those constraints and those personalities, they aren’t easy for me to change. I have to work with what I have and try to find tools that do what I need. When I find a tool that’s 90% of what I need and I’m willing to put in the effort to push it across the line and make it work for me I get to come face to face with the maintainers of that project.

I’d like to say this always goes smoothly, that I’m always able to express my use case in a way that folks understand, and that my use case always falls inline with the intent of the project. I’d love if my skillset in a particular language was always up to par, and that my ability to write quality tests was always appropriate. I’d love to always understand the roadmap for the project & the maintainers expectations around how I should contribute. This just aint so.

As much as I may feel like I don’t understand a particular project well enough, sometimes it feels like the maintainers don’t understand me. What’s worse is when I interpret (accurately or not) an attitude from maintainers that I “just don’t get it” rather than making an attempt to help me understand.

This sounds really familiar to me. This sounds like that whole observation the world made that Dev & Ops need to work more closely. The observation that the people who build the software need to interact more closely with the people who use the software. That building empathy and a collaborative environment where everyone can communicate and be involved will help get us past our differences. For me, this problem isn’t just with the developers in my company. This problem extends to the developers who build the systems I use every day and most of those developers don’t work for my company.

Just as we have to invest in building relationships within a company, when you use OSS you have to invest in building a relationship with those you work with outside your company as well. This goes both ways, and the more effort maintainers put into understanding users, the better this is for everyone. This isn’t easy – so if you aren’t a maintainer and you’re reading this, understand that very often the interactions you have with maintainers is done outside their full time job, outside their ordinary deadlines and pressures, and it’s done because they enjoy working on a project. If folks are like me, when something stops being enjoyable for too long, I stop doing it. Keep them in mind when you have a complaint.

Further, for a popular project the ratio of users to maintainers is pretty imbalanced and not in the favor of the maintainers. They have a hard job and for those that do that job with grace and empathy for their users, my hat is off to you.

So when we talk about DevOps and Empathy and all these great ideas to make your company work better – don’t stop there. Think about all those projects that make your job exist, and all those users that make your project a success. Try to take a little time and understand each other and work together to make more awesome.

kthxbye