On the value of reflection

The research methodologies of Action Research and Reflective Practice instruct that 'reflections' (as in contemplation, not as in type introspection) from a previous software development phase should drive the planning for the next phase (much like the industry methodology of Iterative Development). It is perhaps worthwhile to reinforce the value of this reflection. After all, reflection is expensive. Conducting experiments, interviews and case studies consumes valuable time and resources, and it is legitimate to question whether its benefits outweigh its cost.

One of the most important factors in software development is scope: deciding what to include and what to leave out. Scope creep and feature bloat are recognised risks, impacting development costs and release schedules. Good architects carefully apply rules of thumb: every design decision should 'carry its own weight', and strive to 'kill several birds with one stone'. But an implicit difficulty in evaluating this is knowing what the 'birds' are. Once out of its initial planning phases, software development has a tendency to lurch from immediate issue to immediate issue, dealing with each new requirement as it arises. Considering new requirements in isolation invariably means the burden of large-scale redesign to satisfy any one requirement will seem onerous: a smaller-scale, less impactful alternative will always seem the better option. Reflection, on the other hand, allows the practitioner to consider many weeks worth of problems in a holistic light: he can see all the birds at once, and an approach that once seemed over-engineered now appears justified. Surfacing all the issues at the same time clears a path forward that otherwise would have seemed prohibitive.

This phenomena is analogous to neural networks. While progressing to solve a given problem, a neural network may get trapped, still short of the best solution, in a local minima. The local minima itself does not represent the best answer, but none of the immediate ways out of the minima are enough of an improvement to overcome the walls of the valley. It takes a combined push, a sort of disruptive excitation, to escape the trough so that a better solution can be found:

So much for the theory - is it demonstrable in practice? Here I will give personal testimony. One of the themes from Metawidget's alpha cycle reflections was support for 1-to-M relations. In itself, this seemed a corner case: difficult to support within the current architecture without a slippery slope of requirements around sorting, pagination and summary-to-detail navigation. Another theme was support for third-party UI components. The most challenging case study indicated this would have improved adoption, though it was not a primary factor. A third issue was around supporting the SWT library: the current design of 'return null to render nothing, return a dummy Metawidget to trigger nesting' was backwards for SWT's purposes, though this was being worked around in a sub-optimal way.

Individually, none of these requirements seemed enough to justify a significant reworking of the widget creation subsystem. Indeed, the theme of 1-to-M relations gnawed at me for months with no obvious solution within the existing architecture. It was only reframing it within the context of the additional requirements of 'supporting third-party components' and 'turning widget creation inside out' that a new path presented itself (see WidgetBuilders). Looking back, I realise I was probably especially resistant to seeing this path because it was in an area I had already considered and decided against.

In summary, I have found explicit reflection to be an enlightening and worthwhile use of a project's time. It is easy to skip this phase in the heady rush of pumping out release and release, but when one takes the time to properly pause for breath important insights can be gained.