Agile Values

“Individuals and interactions over processes and tools.”

Paying attention to individuals, the people on the team as opposed to roles in the process chart. Not all practices in software world are appropriate for every project, team and situation; while it is of utmost importance to understand the people in the team, how they work together and how each individual’s work impacts everyone else.

Although a process description is needed to get a group of people started, people are going to develop the program, and can go wrong when blindly following a process, or making use of tools that drive them to get faster an incorrect result.

But all the work is not carried out individually, being of relevance attending to the interactions between the individuals. New solutions and flaws in old solutions come to life in discussions between
people. The quality of the interactions matters, being preferable to use an undocumented process with good interactions than a documented process with hostile interactions.


“Working software over comprehensive documentation.”

There are binders full of complete and comprehensive software documentation sitting unopened on shelves all over the world. There is so much that can be documented in a software project, and it’s often difficult during the heat of the project to predict what’s going to be useful in the future, and what will gather dust. Because of that, a lot of teams—and especially their managers—will decide on a comprehensive approach, where every little thing must be documented, no matter whether or not
there’s a potential reader for the document.

On the other hand, a working system is the only way to measure the real work done built the team. Running code is ruthlessly honest, while documents showing the requirements, analysis, design, screen flows, sequence charts… are handy as hints to aid the team, together with their own experience, to guess what the future will look like

What does the word “working” really mean? To an agile practitioner, working software is software that adds value to the organization. It could be software that a company sells to make money, or it could be software that people who work at the company use to do their jobs more efficiently.

Valuing working software over comprehensive documentation does not mean that no documentation is needed; there are many kinds of documents that are very useful for the team. But it’s important to keep in mind that the people writing the documentation are often the same people who are writing the software.

Documents serve as markers in the game, used to build an image of the unreliable future. On the other hand, the composite act of gathering requirements, designing, coding, and debugging the software, reveals information about the development team, the development process, and the nature of the problem to be solved. Those things together with the running final result provide the only reliable measure of the speed of the team, the shortcomings of the group, and a glimpse into what the team really should be building.


“Customer collaboration over contract negotiation.”

There should not be a “us” or “them” when referring to people involved in the project. Independently of the task of building or providing requirements the team is performing, there should only be “us”. Both sides are needed to produce good software.

Although contracts are useful at times, collaboration strengthens development both when there is a contract in place and when there is none. Good collaboration can save a contract situation when it is in jeopardy. Good collaboration can sometimes make a contract unnecessary. Either way, collaboration is the winning element.


“Responding to change over following a plan.”

There’s an old project management saying: “plan the work, work the plan.” Unfortunately, if you work the wrong plan, you’ll build the wrong product. That’s why teams need to constantly look for changes, and to make sure that they respond appropriately when there’s a change in what the users need, or in how the software needs to be built. If the circumstances change, the project needs a new plan.

Building a plan is useful, and each of the agile methodologies contains specific planning activities. They also contain mechanisms for dealing with changing priorities, guaranteeing that the team has the time and peace of mind to develop working software.



How these values shape Agile

Out of these four values, the following points can be used to summarize the Agile basic tenets.

  1. Redefined roles for developers, managers and customers.
  2. No “Big Upfront” steps.
  3. Iterative development.
  4. Limited, negotiated functionality.
  5. Focus on quality, understood as achieved through testing.

The first tenet affects a fundamental feature of project development: the role of developers and managers. Agile methods redefine and limit the manager’s job by transferring many of the duties to the team as a whole, including one of the most important responsibilities: selecting tasks to be performed and assigning them to developers. It is possible to give a sociological interpretation of the agile movement as a “revolt of the cubicles”: the rejection of rigid, top-down, Dilbert’s-boss-like techniques for managing software projects. Programmers in the trenches — the cubicles — often resent these attempts as ignorant of the specific nature of software development. The Dilbert types know that documents and diagrams do not make a system: code does. Agile methods are, in part, the rehabilitation of code.

The redefinition of roles also affects customers, who in the agile world are not passive recipients of the software but active participants. Most methods advocate including a customer representative in the development team itself.

The second tenet is the rejection of “Big Upfront Anything”, a term used derogatorily for standard software engineering techniques involving extensive planning at the beginning of a project; the principal examples are requirements, to define the goals of the system, and design, to define its architecture. In the agile view:

  • Requirements cannot be captured at the beginning of a project, because users do not know what they want. Even if one managed to write a requirements document, it would be useless because requirements will change through the project.
  • Building a design upfront is a waste of time because we do not know what will work and what will not.

Instead of a requirements document, agile methods recommend constant interaction with the customer — hence the benefit of a customer representative in the team — to get both insights into the problem and feedback on what has been produced so far. Instead of design, the recommendation is to build the system iteratively, devising at each step the “simplest solution that can possibly work” (an Extreme Programming slogan) for the task at hand; then, if the solution turns out to be imperfect, improving its design through a process known as refactoring. Agile development, as a consequence, is iterative, time-boxed development.

The agile alternative to a requirements document is, at the beginning of each iteration, a prioritized list of functions from which the team will select for implementation the function that has the highest Return on Investment (ROI). In the absence of big upfront tasks, this choice will be made in successive steps, (e.g.: the “sprints” in Scrum) each taking a fixed time — a few weeks — hence “time-boxed”. The development thus proceeds by iterative addition of functionality.

The “negotiation” occurs at the step of choosing the functionality for each iteration. Just as it is impossible, in the agile view, to determine full requirements in advance, it is unrealistic to commit to both functionality and delivery time. With time-boxed development, any tradeoffs (“do you want it all or do you want it next month?”) will tend to be resolved in favor of the second criterion: if not all the functions planned for an iteration can be delivered by the deadline, it is the functionality that goes; the deadline stays. The missed functionality will either be reassigned to a subsequent phase or — if further analysis deems its ROI insufficient — dropped. This process of planning and adjusting requires constant negotiation with the customer.

The final tenet is the focus on quality, which in the agile view essentially means continuous testing (rather than other approaches to quality, in particular those based on design techniques, formal programming methodology, or whatever smacks of “Big Upfront”).

The agile approach has little patience with what it sees as the languid attention to quality in traditional development; it especially dislikes the practice of continuing to develop functionality even when the code already developed does not pass all the tests.

One of its contributions is to emphasize the role of a project’s regression test suite: the set of tests that must pass, including all tests that at some point did not pass and hence revealed faults that were then fixed. Regression testing has been known and applied for a long time, but agile methods have given this task a central place in the development process.

 

 

Chapter 4: Aristocracy, Democracy and System Design


This great church is an incomparable work of art. There is neither aridity nor confusion in the tenets it sets forth. . , ,

It is the zenith of a style, the work of artists who had understood and assimilated all their predecessors’ successes, in complete possession of the techniques of their times, but using them without indiscreet display nor gratuitous feats of skill.

It was Jean d ‘Orbais who undoubtedly conceived the general plan of the building, a plan which was respected, at least in its essential elements, by his successors. This is one of the reasons for the extreme coherence and unity of the edifice.

REIMS CATHEDRAL GUIDEBOOK



Conceptual Integrity

The typical situation for a cathedral building is being made during several generations by several builders, where each period shows the ideas and “improvements” of those in command at that moment.

The Reims Cathedral is the counterexample: the building integrity was achieved by the self-abnegation of eight generations of builders whom sacrificed some of his ideas in benefit for the construction pure design.

Even though they have not taken centuries to build, most programming systems reflect conceptual disunity far worse than that of cathedrals. Usually this arises not from a serial succession of master designers, but from the separation of design into many tasks done by many men.

Conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.


Achieving Conceptual Integrity

The purpose of a programming system is to make a computer easy to use.

Ease of use is enhanced only if the time gained in functional specification exceeds the time lost in learning, remembering, and searching manuals. With modern programming systems this gain does exceed the cost, which did not happen in software development old days. Because ease of use is the purpose, the ratio of function- conceptual complexity is the ultimate test of system design. Neither function alone nor simplicity alone defines a good design.

For a given level of function, however, that system is best in which one can specify things with the most simplicity and straightforwardness.

It is not enough to learn the elements and rules of combination; one must also learn the idiomatic usage, a whole lore of how the elements are combined in practice. Simplicity and straightforwardness proceed from conceptual integrity. Every part must reflect the same philosophies and the same balancing of desiderata.

Every part must even use the same techniques in syntax and analogous notions in semantics. Ease of use, then, dictates unity of design, conceptual integrity.


Aristocracy and Democracy

Conceptual integrity in turn dictates that the design must proceed from one mind, or from a very small number of agreeing resonant minds. Schedule pressures, however, dictate that system building needs many hands. Two techniques are available for resolving this dilemma.

  • Division of labour between architecture and implementation.
  • The Surgical Team structuration

The separation of architectural effort from implementation is a very powerful way of getting conceptual integrity on very large projects. Complete and detailed specification of the user interface.  (For the entire system it is the union of the manuals the user must consult to do his entire job).

The architect of a system has to bring professional and technical knowledge to bear in the unalloyed interest of the user, as opposed to the interests of the salesman, the fabricator, etc.

“Where architecture tells what happens, implementation tells how it is made to happen.”

In regard to the deeply emotional question of aristocracy versus democracy:

  • Are not the architects a new aristocracy, an intellectual elite, set up to tell the poor dumb implementers what to do?
    • Yes, in the sense that there must be few architects, their product must endure longer than that of an implementer, and the architect sits at the focus of forces which he must ultimately resolve in the user’s interest. If a system is to have conceptual integrity, someone must control the concepts. That is an aristocracy that needs no apology.
    • No, because the setting of external specifications is not more creative work than the designing of implementations. It is just different creative work. The design of an implementation, given an architecture, requires and allows as much design creativity, as many new ideas, and as much technical brilliance as the design of the external specifications.
  • Has not all the creative work been sequestered for this elite, leaving the implementers as cogs in the machine?
  • Won’t one get a better product by getting the good ideas from all the team, following a democratic philosophy, rather than by restricting the development of specifications to a few?
    • Not only the architects will have good architectural ideas. Often the fresh concept does come from an implementer or from a user. However, the conceptual integrity of a system determines its ease of use. Good features and ideas that do not integrate with a system’s basic concepts are best left out.
    • If there appear many such important but incompatible ideas, one scraps the whole system and starts again on an integrated system with different basic concepts.

The external provision of an architecture enhances, not cramps, the creative style of an implementing group. They focus at once on the part of the problem no one has addressed, and inventions begin to flow. In an unconstrained implementing group, most thought and debate goes into architectural decisions, and implementation proper gets short shrift.


What Does the Implementer Do While Waiting?

“It is a very humbling experience to make a multimillion-dollar mistake, but it is also very memorable”

When it is proposed that a small architecture team in fact write all the external specifications for a computer or a programming system, the implementers raise three objections:

  • The specifications will be too rich in function and will not reflect practical cost considerations.
  • The architects will get all the creative fun and shut out the inventiveness of the implementers.
  • The many implementers will have to sit idly by while the specifications come through the narrow funnel that is the architecture team.

In the computer systems business the pace is quicker than in construction (where design comes first, building comes after), and one wants to compress the schedule as much as possible. How much can specification and building be overlapped?

The total creative effort involves three distinct phases: architecture, implementation, and realization, which can in fact begun in parallel and proceed simultaneously. Meanwhile, on the realization level there is much to be done also. Programming has a technology, too. Much work must be done on subroutine conventions, supervisory techniques, searching and sorting algorithms.

Conceptual integrity does require that a system reflect a single philosophy and that the specification as seen by the user flow from a few minds. Because of the real division of labour into architecture, implementation, and realization, however, this does not imply that a system so designed will take longer to build. Experience shows the opposite, that the integral system goes together faster and takes less time to test. In effect, a widespread horizontal division of labour has been sharply reduced by a vertical division of labour, and the result is radically simplified communications and improved conceptual integrity.

Chapter 3: The Surgical Team

These studies revealed large individual differences between high and low performers, often by an order of magnitude.

 SACKMAN. ERIKSON. AND GRANT


When managing a team there is a debate between small teams made out of sharp first-class people, or bigger teams with people of every kind. Appart from that there are the needs and size of the project, for which a small team runs short or reseources for doing it in a meaningful schedule.


The Problem

Is it preferable to have small, sharp teams of very efficient programmers or a large team of no so efficient people able to deal with big tasks by, at the end, brute force?

The dilemma is a cruel one. For efficiency and conceptual integrity, one prefers a few good minds doing design and construction. Yet for large systems one wants a way to bring considerable manpower to bear, so that the product can make a timely appearance. How can these two needs be reconciled?


Mills’s Proposal – THE SURGICAL TEAM

Harlan Mills proposes that each segment of a large job be tackled by a team, but that the team be organized like a surgical team rather than a hog-butchering team. That is, instead of each member cutting away on the problem, one does the cutting and the others give him every support that will enhance his effectiveness and productivity.

Much as a surgical team during surgery is led by one surgeon performing the most critical work, while directing the team to assist with less critical parts, it seems reasonable to have a “good” programmer develop critical system components while the rest of a team provides what is needed at the right time

THE SURGEON  (The chief programmer).

  • Defines the functional and performance specifications, designs the program, codes it, tests it, and writes its documentation.
  • Writes in a structured programming language such as PL/I
  • Has effective access to a computing system which not only runs his tests but also stores the various versions of his programs, allows easy file updating, and provides text editing for his documentation.
  • Needs great talent (10+ years experience) and considerable systems and application knowledge, whether in applied mathematics, business data handling, or whatever.

THE COPILOT. (The alter ego of the surgeon)

  • Able to do any part of the job, but is less experienced.
  • Main function is to share in the design as a thinker discussant, and evaluator. The surgeon tries ideas on him, but is not bound by his advice.
  • Represents his team in discussions of function and interface with other teams.
  • Knows all the code intimately.
  • Researches alternative design strategies.
  • May even write code, but he is not responsible for any part of the code.

THE ADMINISTRATOR

  • Is boss, and he must have the last word on personnel, raises, space, and so on, but he must spend almost none of his time on these matters.
  • Needs a professional administrator who handles money, people, space, and machines, and who interfaces with the administrative machinery of the rest of the organization.
  • Has a full-time job only if the project has substantial legal, contractual, reporting, or financial requirements because of the user-producer relationship.

THE EDITOR

  • Takes the draft or dictated manuscript produced by the surgeon and criticizes it, reworks it, provides it with references and bibliography, nurses it through several versions, and oversees the mechanics of production.

TWO SECRETARIES

  • The administrator and the editor will each need a secretary; the administrator’s secretary will handle project correspondence and non-product files.

THE PROGRAM CLERK

  • Responsible for maintaining all the technical records of the team in a programming-product library.
  • The clerk is trained as a secretary and has responsibility for both machine-readable and human-readable files.
  • Logs and keys it all computer input. The output listings go back to him to be filed and indexed.
  • Making all the computer runs visible to all team members and identifying all programs and data as team property, not private property.
  • Relieves programmers of clerical chores, systematizes and ensures proper performance of those oft-neglected chores, and enhances the team’s most valuable asset—its work-product.
  • Logs all updates of team program copies from private working copies, still handles all batch runs, and uses his own interactive facility to control the integrity and availability of the growing product.

THE TOOLSMITH

  • Responsible for ensuring the adequacy of “File-editing, text-editing, and interactive debugging” services and for constructing, maintaining, and upgrading special tools—mostly interactive computer services—needed by his team.
  • Each team will need its own toolsmith
  • The tool-builder will often construct specialized utilities, catalogued procedures, macro libraries.

THE TESTER

  • Is both an adversary who devises system test cases from the functional specs, and an assistant who devises test data for the day-by-day debugging.
  • Also plans testing sequences and set up the scaffolding required for component tests.

THE LANGUAGE LAWYER

  • Masters the intricacies of a programming language, which uses to do difficult, obscure and tricky things
  • Does small studies on good technique
  • At the service of different surgeons


How It Works

The team just defined meets the desiderata in several ways. Ten people, seven of them professionals, are at work on the problem, but the system is the product of one mind—or at most two, acting uno animo.

Differences between a team of two programmers conventionally organized and the surgeon-copilot team.

  • In the conventional team the partners divide the work, and each is responsible for design and implementation of part of the work. In the surgical team, the surgeon and copilot are each cognizant of all of the design and all of the code. This saves the labor of allocating space, disk accesses, etc. It also ensures the conceptual integrity of the work.
  • In the conventional team the partners are equal, and the inevitable differences of judgment must be talked out or compromised. Since the work and resources are divided, the differences in judgment are confined to overall strategy and interfacing, but they are compounded by differences of interest. In the surgical team, there are no differences of interest, and differences of judgment are settled by the surgeon unilaterally.

These two differences—lack of division of the problem and the superior-subordinate relationship—make it possible for the surgical team to act uno animo.

Yet the specialization of function of the remainder of the team is the key to its efficiency, for it permits a radically simpler communication pattern among the members.

Chapter 1: The Tar Pit

A ship on the beach is a lighthouse to the sea.

DUTCH PROVERB

Over the years, all kind of animals have sunk in tar pits despite their efforts to scape.

Large-system programming has over the past decade been such a tar pit, and many great and powerful beasts have thrashed violently in it. Most have emerged with running systems—few have met goals, schedules, and budgets. Large and small, massive or wiry, team after team has become entangled in the tar. No one thing seems to cause the difficulty—any particular paw can be pulled away. But the accumulation of simultaneous and interacting factors brings slower and slower motion. Everyone seems to have been surprised by the stickiness of the problem, and it is hard to discern the nature of it. But we must try to understand it if we are to solve it.


Programming System Products

Obsession with transition from garage programming to real programming system product

We start with a program, THAT is the thing commonly produced in garages, and what is used by the individual programmer for doing productivity estimations

There are two ways a program can be converted into a more useful, but more costly, object:

PROGRAM – PROGRAMMING PRODUCT

  • A programming product can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data.
  • To become a generally usable programming product, must be written in a generalized fashion.
  • The range and form of inputs must be generalized as much as the basic algorithm will reasonably allow.
  • Then the program must be thoroughly tested, so that it can be depended upon. (A substantial bank of test cases, exploring the input range and probing its boundaries, must be prepared, run, and recorded)
  • Finally, promotion of a program to a programming product requires its thorough documentation, so that anyone may use it, fix it, and extend it.
  • I estimate that a programming product costs at least three times as much as a debugged program with the same function.

PROGRAM – PROGRAMMING SYSTEM

  • A programming system is a collection of interacting programs, coordinated in function and disciplined in format (the assemblage constitutes an entire facility for large tasks.)
  • To become a programming system component, a program must be written so that every input and output conforms in syntax and semantics with precisely defined interfaces.
  • The program must also be designed so that it uses only a prescribed budget of resources—memory space, input-output devices, computer time.
  • Finally, the program must be tested with other system components, in all expected combinations. This testing must be extensive, for the number of cases grows combinatorially. It is time-consuming, for subtle bugs arise from unexpected interactions of debugged components.
  • A programming system component costs at least three times as much as a stand-alone program of the same function.

PROGRAMMING SYSTEM / PRODUCT – PROGRAMMING SYSTEM PRODUCT 

  • This differs from the simple program in all of the above ways.
  • It costs nine times as much. But it is the truly useful object, the intended product of most system programming efforts.


programming-system-products


The Joys of the Craft

Why is programming fun? What delights may its practitioner expect as his reward?

  • The sheer joy of making things.
  • The pleasure of making things that are useful to other people. Deep within, we want others to use our work and to find it helpful.
  • The fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning.
  • The joy of always learning, which springs from the nonrepeating nature of the task. In one way or another the problem is ever new, and its solver learns something: sometimes practical, sometimes theoretical, and sometimes both.
  • The delight of working in such a tractable medium. Almost pure thought-stuff. Building castles in the air, from air, creating by exertion of the imagination.

Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.

Yet the program construct is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

Programming then is fun because it gratifies creative longings built deep within us and delights sensibilities we have in common with all men.


The Woes of the Craft

  • Not all is delight, however, and knowing the inherent woes makes it easier to bear them when they appear.
  • One must perform perfectly. Human beings are not accustomed to being perfect, and few areas of human activity demand it. Adjusting to the requirement for perfection is the most difficult part of learning to program.
  • Other people set one’s objectives, provide one’s resources, and furnish one’s information. One rarely controls the circumstances of his work, or even its goal. In management terms, one’s authority is not sufficient for his responsibility.
  • The dependence upon others has a particular case that is especially painful for the system programmer. He depends upon other people’s programs (often maldesigned, poorly implemented, incompletely delivered (no source code or test cases), and poorly documented). Forcing to  spend hours studying and fixing things that in an ideal world would be complete, available, and usable.
  • Designing grand concepts is fun; finding nitty little bugs is just work. With any creative activity come The Woes of the Craft dreary hours of tedious, painstaking labour.
  • Debugging has a linear convergence, so testing drags on and on, the last difficult bugs taking more time to find than the first.
  • The product over which one has laboured so long appears to be obsolete upon (or before) completion. The technological base on which one builds is always advancing. As soon as one freezes a design, it becomes obsolete in terms of its concepts. But implementation of real products demands phasing and quantizing. The obsolescence of an implementation must be measured against other existing implementations, not against unrealized concepts.

The challenge and the mission are to find real solutions to
real problems on actual schedules with available resources.


This then is programming, both a tar pit in which many efforts have floundered and a creative activity with joys and woes all its own. For many, the joys far outweigh the woes, and for them the remainder of this book will attempt to lay some boardwalks across the tar.