Combating information failure during software development

2015-02-16T00:00:00-05:00

Developer productivity is a product of program understanding, a developer's (1) mental model, and (2) access to information for refining that model.

Former Defense Secretary Donald Rumsfeld's famous intuition is actually roughly correct for understanding program understanding:

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.

Because software development involves constant decision-making, it's also important to consider whether information is discoverable — that we can, when required, make "known unknowns" become "known knowns". This yields three dimensions:

Awareness: does your (the developer's) mental model consider the information?
Presence: is the information present with the software (i.e., known)?
Computability: can you compute and update the information automatically (i.e., knowable)?

Across these dimensions, there are six situations that occur during software development:

Level	Situation	Aware?	Present?	Computable?
0	Unknown Unknowns	No	No	Maybe
1	Uncomputable Information	Yes	No	No
2	Stale / Misplaced Information	Yes	Maybe	No
3	Developer-Supplied Information	Yes	Yes	No
4	Uncomputed Information	Yes	Maybe	Yes
5	Perfect Information	Yes	Yes	Yes

In general, to increase the quality of information available, you must supply additional information in the right form. The information you supply signals intent, and can serve as machine-checked documentation (with the right tools).

The rest of this post describes the six situations, providing some intuition behind their causes. It starts with Level 0, the most dangerous level.

Level 0: Unknown Unknowns. Properties of which the developer is unaware. These typically occur when you have not anticipated a use case, or do not understand a platform well enough to be aware of potential issues. Examples:

Thread-safety
The (unintuitive) behavior of Javascript's operators
Argument aliasing

This is the worst situation to be in. If and when your software crashes, your only hope is to re-create and observe the effects of the behavior. Because the behavior is not part of your mental model, you won't know how to observe it directly.

You eliminate "unknown unknowns" by expanding your awareness in two ways: (1) experience, and (2) a broad exposure to software engineering topics and frameworks. For most types of software, your mental model doesn't need to be complete! You just need to be aware of potential "gotchas" so that you can direct your development and debugging.

Level 1: Uncomputable Information. Properties of which the developer is aware, but does not have an easy way of deciding automatically. Examples:

The correctness of a sorting algorithm implementation
Program termination
How clients use your library

Level 2: Stale / Misplaced Information. Information that either might be out-of-date or is not present with the development artifact. Examples:

Design decisions made by a developer who was fired
Documentation for an old version of an API
Requirements captured in an email thread

If you have no way of telling which information is accurate, you're back at "Level 1: Uncomputable Information". If you have no way of telling if the information is complete, you're back at "Level 0: Unknown Unknowns".

Level 3: Developer-Supplied Information. Information that is up-to-date, but the developer has no easy way of knowing how a change might affect that information. Examples:

A written proof of an algorithm
Class documentation
Renaming a field in a dynamically-typed program

If you don't update this information when making a change, you'll quickly find yourself at "Level 2: Stale Information". By instead capturing information in a machine-readable form, you can instead approach "Level 4: Uncomputed Information".

Level 4: Uncomputed Information. Information that can be computed automatically, but is not yet available for the current version of the program. Examples:

Non-incremental compilation (e.g., compiling a C++ program)
A long-running test suite
A long-running static analysis

When a failure occurs, you might be back at "Level 3: Developer-Supplied Information" or "Level 2: Stale / Misplaced Information". For example, if a test fails, you need to be able to determine (1) if the failure is valid, and (2) if so, how to modify the program and/or test suite.

For many types of information, you can effectively bring yourself to "Level 5: Perfect Information" by applying the Pareto principle. In the case of a long-running test suite, for example, you might prioritize the execution order of the test suite.

Level 5: Perfect Information. Information that is up-to-date and is updated automatically whenever the developer makes a change. Examples:

Incremental compilation with type checking
Continuous testing
Speculative analyses that compute the effects of a change before you even make the change

Remember that to increase the quality of information available, you must supply additional information. For example, typed languages require type annotations; more-detailed type analyses (e.g., the Checker Framework) require yet more (and more precise) annotations.

Summary

Program understanding depends on information awareness, presence, and computability. The next time you hit a wall programming or debugging, take a step back and ask: "where is the information failure?" By answering this question, you can set yourself and your team up to be more productive.

This post is based on an early framing of my PhD research. As part of my research, I designed the Cupid framework for giving developers immediate insight into their project and team.

Todd Schiller - mental model

Combating information failure during software development

Summary