Introduction

If you go to Google and search for "measuring software developer productivity" you will find a whole lot of nothing. Seriously -- nothing.

Well, okay, not exactly nothing. You'll get a ton of links. But almost all of the links you find will talk about how measuring the productivity of software developers can't be done effectively. Some people will even argue that it shouldn't be attempted at all. Some others will describe techniques to measure developer productivity, that, well, everyone else knows don't really work.

There have been many valiant attempts to measure developer productivity, but all seem to end in less than successful territory. (We all know to laugh at "Lines of Code" as a productivity measure). Virtually any objective measurement you can apply to the software development process can be "gamed" into submission.

There are just too many variables to account for everything that goes into the development process. Measuring any one of them, or even a combination of them, simply cannot begin to capture effectively everything that is involved. You can't pick a surrogate thing to measure, but because of human nature, developers will react in ways that will skew the measurement.

Many studies have been done on this topic, with little to show for it. Every software development manager who has to fill out a developer's evaluation or determine who gets bonuses has struggled with this. Many a leading expert and guru has thought about and researched this thorny topic.

Experts Agree

And not one of the experts has come up with a good idea or method to measure individual productivity. There are plenty of good and effective ideas on measuring team and project productivity, but there is a severe if not complete shortage of ways to measure individual performance and productivity.

For instance, Robert D. Austin wrote an entire book on this subject called Measuring and Managing Performance in Organizations and came to the conclusion that unless you can measure 100% of something -- that is, every aspect of it -- then there is no point in measuring at all. If you attempt to measure less than 100% of the things involved in a given activity, you will get what he refers to as "dysfunction", or what you and I might call gaming the system. And since software development is an incredibly complex and creative endeavor, it's virtually impossible to measure every aspect of it. Often it quite literally involves sitting and staring at a computer screen. How do you measure the productivity of that?

Here's Martin Fowler's conclusion in an article he entitles " CannotMeasureProductivity":

"This is somewhere I think we have to admit to our ignorance."

Neil McAllister of InfoWorld writes in an article entitled "The Futility of Developer Productivity Metrics"

"For most other companies, however, it might be best simply to forget about the idea of measuring developer productivity and rely instead on tried and true methods. That's right: A highly effective, productive developer workforce is often the result of high-quality, effective management. Unfortunately, nobody has developed a metric for that yet. Coincidence?"

Joel Spolsky writes:

"It's rather hard to measure programmer productivity; almost any metric you can come up with (lines of debugged code, function points, number of command-line arguments) is trivial to game, and it's very hard to get concrete data on large projects because it's very rare for two programmers to be told to do the same thing."

Steve McConnell, a widely regarded academic, author, and consultant says:

"So while I see the value of measuring individual performance in research settings, I think its difficult to find cases in which the effort is justified on real projects."

Like I said, it's hard to find a leading expert making the case for objective measurements, and easy to find plenty that say you shouldn't even try.

So I don't think that it's a tough leap to agree that there is no objective way to measure productivity in software developers. I know this is controversial, but it seems quite clear to me that software is more art than science, and measuring productivity in art is, well, simply not done. One does not tell Michelangelo that it took him too long to paint the Sistine Chapel because he should have been painting 1.4 square feet per day instead of 1.2. That would be preposterous.

So, What Are We to Do?

Anyone who is charged with managing and leading developers is left in a quandary. We want to be fair, but if everything is subjective, how can we be?

The fact that there is no objective measure of developer productivity doesn't mean you can't measure it. It just means that you have to measure it subjectively . And we do that, don't we. You can look around your shop and just know who the productive developers are. There are probably developers you know that simply crank out good code faster and fix bugs more efficiently than the rest. For ease of discussion, I'll give this developer the name "Jason".

And just how do you know Jason is more productive and more valuable? You just do. You can just tell. You know it when you see it. But if you stop and think about it, Jason has some characteristics that you can figure out and write down.

What I mean is that if you stop and examine Jason and his work, there are things that you can say about him that describe what he is all about. They are subjective, sure, but you can write them down and give an "objective" measure to these subjective things.

A Modest Proposal

So yeah, what I am proposing is a way to measure developer productivity and value via a subjective set of measurements. Think of it as a way to qualify the notion of "you just know that Jason is productive".

Here is the list:

Productivity

• Does the developer get a reasonable amount of work done in a given period of time?
• Is the developer's velocity on bug fixing sufficient?

Engagement

• Is the developer dedicated to his/her craft?
• Is the developer committed to delivering software on time?
• Is the developer dedicated to company success?

Attention to Quality

• To what degree does the developer's code work as designed?
• Does the developer thoroughly test code and believe it to be correct before checking it in?
• Do a minimal number of bugs get reported against his/her code?
• Does the developer write unit tests for all new code?
• Does the developer follow the Boy Scout Rule and leave a module cleaner than it was before he or she worked on it?

Code Base Knowledge and Management

• To what degree does the developer understand the code base assigned to him/her?
• Does the developer take responsibility for his/her team's code base, improving it at every opportunity?
• Adherence to coding guidelines and techniques
• Does developer's code routinely meet coding standards?
• Do code reviews reveal a minimum of problems and discrepancies?
• Does the developer use Dependency Injection to ensure decoupled code?

Learning and Skills

• Is the developer constantly learning and improving his/her skills?
• Does the developer show a passion for the craft of software development?

Personal Responsibility

• Does the developer first assume that the error lies within his or her code?
• Does the developer understand that he or she is solely responsible for their code working correctly?
• Does the developer take pride in their code, ensuring it is clean, tested, easy to read, and easy to maintain?

Again, these seven items are all subjective measurements. All the questions require the evaluator to make subjective judgments. But a good lead/manager is presumably in their position because of their good judgment about such things, and so they should be able to make fair and accurate assessments about Jason's productivity using these "measurements".

Most of the questions ask for a yes or no answer, but all can spark a discussion with management or even the developer being evaluated. You could use these measures on a formal evaluation form. Whatever you do with them, it's probably better than the "just know it when I see it" approach you are probably taking now.

Conclusion

In the end, any evaluation, judgment, or other assessment of the productivity or value of a single developer will be subjective. No one has yet found an objective way to measure individual productivity in our excessively complex profession. But I am saying that it is possible to qualify and make judgments about a specific set of criteria as a small and humble step towards being able to do the thing that our industry has so far found very difficult.

My suggestion is that we simply accept reality and embrace the subjectivity -- and lack of objectivity -- in measuring developer performance.

Note: This article originally appeared on my personal blog at http://nickhodges.com/post/Measuring-Developer-Productivity.aspx