The Experiment Canvas is a simple means of planning, tracking, and responding to your experiments.
Simple Things
When I say simple, I don’t necessarily mean easy. And I certainly do not mean crude in form or incomplete. Simple indicates something that does not have superfluous parts or multiple responsibilities, is easy to understand, is as independent from the rest of the solution as possible, and meets a need as is.
Simple may not address all use cases, but it does address some use cases.
Over the past several years, as I’ve been helping teams and organizations improve their ability to deliver software products that are desirable, viable, and feasible, I have been experimenting with a Behavior Framework that has proven to be rather effective. And I’d like to share it with you in hopes that you find it useful and that you provide me feedback on your experiences with it.
My plan is to write a series of blog posts all related to the behaviors framework. Some of them will be about a specific behavior. Some of them will be about tools or techniques that help teams express one or more of the behaviors. Some of them will be my own experiences. And some will be damn near complete fiction.
Status Meetings - they happen every day, to thousands of people. Millions upon millions of corporate dollars are spent every year on them. Countless hours dedicated to attendance. And they suck. They are most often mind-numbing, pointless, disruptive wastes of time.
What if we could make them suck less?
Now, metrics are not bad. But, they are often used in bad ways.
It might help to be aware of some of the side effects of mismanagement of metrics. From inadvertently creating behaviors that actively work against our best interest, to altering the meaning of the metric, mismanagement can do more harm than good.
Charles Goodhart is an economist and former advisor to the Bank of England. In 1975, Goodhart delivered two papers to a conference at the Reserve Bank of Australia. In those papers, Goodhart was discussing research and theory related to monetary policy and control in the United Kingdom. In the years leading up to 1975, existing monetary targets and the controls used to achieve the goals were no longer producing the results desired or expected. There had been what most considered to be evidence of a stable money demand in the United Kingdom. It was believed that the growth of money could be controlled through the setting of short-term interest rates. Higher interest rates correlated with lower money growth.
Goodhart warned, however, that policies and practices based on specific targets were flawed. Goodhart stated,
“Any statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
“Any statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
A common paraphrasing is, “When a measure becomes a target, it ceases to be a good measure.” When I talk about this, I tend to add, “And the target therefore no longer means what you think it does.”
Goodhart’s law is a critical piece of information when we think about metrics. No matter how tempting it might be, the moment we set a target for a measure, we’ve changed the system, thereby changing what the measurement means, thereby changing what the target means.
The lesson here is pretty simple. Don’t set targets for metrics. And please don’t give teams incentives towards targets if you do set them. I know. I know. Management 101 says this works. But, science says it doesn’t. Seriously. Setting targets and providing incentives for knowledge work lowers performance. Don’t do it.
Instead, provide guidelines to the teams. My favorite guideline for metrics is, “Monitor trending. Dig in when the trend changes and you aren’t absolutely certain why.”
This article is an excerpt from the book, “Escape Velocity”, available on LeanPub, Amazon, and elsewhere.
Now, metics are not bad. But, they are often used in bad ways.
It might help to be aware of some of the side effects of mismanagement of metrics. From inadvertently creating behaviors that actively work against our best interest, to altering the meaning of the metric, mismanagement can do more harm than good.
Western Electric had commissioned an extensive study that ran from 1924 to 1932 at their Hawthorne Works in Cicero, IL. The intent of the study was to determine the impact of ambient lighting on worker productivity. Would employees be more productive under high levels of light or low levels of light? The workers in the factory were divided into two groups based on physical location within the plant. For one group, the lighting was increased dramatically while for the other group (the control) lighting levels remained the same. Researchers found that productivity improved among the group for whom lighting changed whereas the control group had no statistically significant change.
Employee working conditions were then changed in other ways. Working hours were adjusted, rest breaks were changed, floors were rearranged, workstations were kept cleaner, and several other adjustments were made, including returning the lighting back to normal levels and changing practices and policies back to original standards.
With every change, productivity made small improvements. By early 1932, and the end of the studies, the factory productivity was at an all-time high and employee attendance and retention were at record-setting levels. Some groups seemed to do better than others, but across the factory, all measures were improved.
When the studies ended, productivity, attendance, and retention soon returned to original levels.
The key takeaway from the Hawthorne studies is - that which gets measured will improve, at least temporarily. “The Hawthorne Effect” is described as the phenomenon in which subjects in behavioral studies change their performance in response to being observed.
This, at first, seems like a precious nugget of management gold.
Measure productivity.
Make it known.
Ka-Pow! Increased productivity.
The perfect management formula.
But the reality was (and is), that while that which is being measured shows improvement, it does not mean the overall system has improved. Working longer hours can lead to employee fatigue and burn out, as well as lower quality. Lack of attention in areas not measured, such as quality or workplace safety, can lead to other negative outcomes.
If your team is slacking so significantly that merely measuring their velocity can result in a marked increase in velocity with no ill- effects, then you’ve a more serious issue at play than velocity.
What’s more, there is no guarantee that the thing being measured has actually improved. Velocity might have gone up because the team inflated story points. We should rephrase the key takeaway to that which gets measured will (appear to) improve.
This article is an excerpt from the book, “Escape Velocity”, available on LeanPub, Amazon, and elsewhere.
In the past few years, we at OnBelay have had the honor of working with companies who are looking to improve their engineering culture.
One key tool we use today is the Experiment Canvas. My partner, Diane Zajac, and I co-developed the canvas. It is based heavily on our experience with A3s. It is still a work in progress, but I want to share with you where we are to date. Please feel free to use it and give us feedback.
In November of 2015, HBR published an article entitled, "We Like Leaders Who Underrate Themselves". The authors describe an extensive study based on 69,000 managers, 750,000 respondents and hundreds of companies where through an analysis of 360-degree feedback data, they found that leaders who rate themselves poorly compared to how their subordinates rate them are not only seen more favorably by their employees, but actually have more engaged employees.
Bottom line to the article:
"The more people overrated themselves, the higher the probability that they have fatal flaws and the lower the probability they have any strengths. The more people underrate themselves, however, the higher the probability they have strengths and the lower the probability they have fatal flaws."
A hat tip to Tyler Jennings for the title of this post. He and I were in a meeting some time ago, along with a lot of other interesting people from Groupon Engineering. We were sharing our thoughts on team leadership and the role of managers. There was talk about how decisions need to be made close to the work and how managers need to not just seek advice, but actually provide others the opportunity to make decisions.
I was recently contacted by a colleague looking for a bit of advice.
C: "We are thinking about merging two teams together and we're not sure how to message the change."
D: "What do you think will be your biggest challenges?"
C: "Well, for one, we're not sure how the funnel flow team will respond to being folded into the purchase page team."
Some time in mid-January, I was thinking about various organizational patterns and how companies run. I was thinking, in particular, about the idea of a "leadership team". I tweeted that leadership teams may be an indication of dysfunction.
I got a few different responses, all of which amounted to "please say more about this."
I think I saw Daniel Pink's TED Talk on "The Puzzle of Motivation" for the first time in 2011. I'd been reading some about leadership, management, and organizational psychology up to that point, but Pink's talk and his distillation of these complex concepts into a simple framework (Autonomy, Connection, and Excellence) inspired me to read more on the topics. Over the course of the next couple of years, I consumed a decent amount of material. You can view my Goodreads account to see what books I was reading. Unfortunately, there is no easy way to share all of the scientific articles and other sources I also consumed.
It is not only the things we reward that shape culture, but the things we allow. Perhaps the easiest way to shape a culture is to do nothing at all.
When a rockstar employee yells at, denigrates, or refuses to help teammates and you let it slide because the rockstar is valuable, you are shaping a culture. When a teammate tells a racist or sexist joke and you say nothing because nobody present is a member of the target group, you are shaping a culture. When an executive abuses power, when a coworker engages in gossip, when a team cuts corners to make deadlines and you decide it isn't your problem, you are shaping a culture.
We have an application internal to Groupon called the "Love Monster". It was written, in large part by Devin Breen. There are other contributors, but Devin is the one that made it happen. He didn't do it because it was on a roadmap or because it's part of our quarterly objectives. He did it because he and others wanted something like this to exist. So he willed it into existence.
