Extreme Programming
Extreme Programming (XP) is a method or approach to software engineering and the most popular of several agile software development methodologies. It was formulated by Kent Beck, Ward Cunningham, and Ron Jeffries. Kent Beck wrote the first book on the topic, Extreme programming explained: Embrace change, published in 1999. The second edition of the book, which appeared in 2005, delves more into the philosophy of eXtreme Programming and describes it as being:
- a mechanism for social change
- a style of development
- a path to improvement
- an attempt to reconcile humanity and productivity
- a software development discipline
History
Context
The 1990s had seen the great promise of object technologies launch ambitious projects that in many cases ended in failure. Objects themselves, while conferring numerous advantages of reuse, were not the panacea that many had hoped they would be. Fluid, rapidly-changing requirements demanded shorter lifecycles, and did not go well with more traditional methods of development, which usually required extensive design up front and penalized later design changes with higher costs or missed deadlines.
Origins
The Chrysler Comprehensive Compensation project was started in order to determine the best way to use object technologies, using the payroll systems at Chrysler as the object of research, with Smalltalk as the language and GemStone as the persistence layer. They brought in Kent Beck, a prominent Smalltalk practitioner, to do performance tuning on the system, but his role expanded as he noted several issues they were having with their development process. He took this opportunity to propose and implement some changes in their practices based on his work with his frequent collaborator, Ward Cunningham.
- The first time I was asked to lead a team, I asked them to do a little bit of the things I thought were sensible, like testing and reviews. The second time there was a lot more on the line. I thought, "Damn the torpedoes, at least this will make a good article," [and] asked the team to crank up all the knobs to 10 on the things I thought were essential and leave out everything else. —[Kent Beck]
Beck invited Ron Jeffries to the project to help develop and refine these methods. Jeffries thereafter acted as a kind of coach to instill the practices as habits in the C3 team.
Information about the principles and practices behind XP was disseminated to the wider world through discussions on the original WikiWiki, Cunningham's Portland Pattern Repository. Various contributors dissected and expanded upon the ideas, some becoming fervent advocates, others becoming critics, and yet others picking out ideas (see agile software development).
Current State
Beck edited a series of books on XP, beginning with his own Extreme Programming Explained, spreading his ideas to a much larger yet very receptive audience. Authors in the series went through various aspects attending XP and its practices, even a book critical of the practices.
XP created quite a buzz in the late nineties and early two thousands, seeing adoption in a number of different milieus and environments radically different from its origins.
Future Directions
The high discipline required by the original practices often went by the wayside, causing certain practices to be deprecated or left undone on individual sites. Agile development practices have not stood still, and XP is still evolving, assimilating more lessons from experiences in the field. In the second edition of Extreme Programming Explained, Beck added more values and practices, and differentiated between primary and corollary practices.
Goal of XP
The main aim of XP is to lower the cost of change. In traditional system development methods (like SDM) the requirements for the system are determined at the beginning of the development project and often fixed from that point on. This means that the cost of changing the requirements at a later stage in the project—not unthinkable in a rapidly changing business—will be very high. This concept is illustrated by the graph below.
Missing image
Costofchange.jpg
Image:Costofchange.jpg
XP sets out to lower the cost of change by introducing basic values, principles and practices. By applying XP, a system development project should be more flexible with respect to changes. The graph below illustrates the intent to hem in the cost of change.
Missing image
Costofchangexp.jpg
Image:Costofchangexp.jpg
XP core practices
The core practices of eXtreme Programming, as described in the first edition of Extreme programming explained can be grouped into four areas (12 practices) as follows:
Fine scale feedback
- Test driven development
- Planning game
- Whole team (original name: Onsite customer)
- Pair programming
Continuous process rather than batch
- Continuous Integration
- Design Improvement (original name: Refactor)
- Small Releases
Shared understanding
- Simple design
- System metaphor
- Collective code ownership
- Coding standard or Coding conventions
Programmer welfare
- Sustainable pace (original name: Forty hour week)
In the second edition of Extreme programming explained a set of corollary practices are listed in addition to the primary practices.
The core practices are derived from generally accepted best practices, and are taken to extremes:
- Interaction between developers and customers is good. Therefore, an XP team is supposed to have a customer on site, who specifies and prioritizes work for the team, and who can answer questions as soon as they arise. (In practice, this role is sometimes fulfilled by a customer proxy.)
- If learning is good, take it to extremes: Reduce the length of development and feedback cycles. Test early.
- Simple code is more likely to work. Therefore, extreme programmers only write code to meet actual needs at the present time in a project, and go to some lengths to reduce complexity and duplication in their code.
- If simple code is good, re-write code when it becomes complex.
- Code reviews are good. Therefore XP programmers work in pairs, sharing one screen and keyboard (which also improves communication) so that all code is reviewed as it is written.
- Testing code is good. Therefore, in XP, tests are written before the code is written. The code is considered complete when it passes the tests (but then it needs refactoring to remove complexity). The system is periodically, or immediately tested using all pre-existing automated tests to assure that it works. See test-driven development.
It used to be thought that Extreme Programming could only work in small teams of fewer than 12 persons. However, XP has been used successfully on teams of over a hundred developers. It is not that XP doesn't scale, just that few people have tried to scale it, and proponents of XP refuse to speculate on this facet of the process.
XP values
Extreme Programming initially recognized just four values but a new value was added in the second edition of Extreme programming explained. The five values are:
- Communication
- Simplicity
- Feedback
- Courage
- Respect (the latest value)
Communication
A fundamental task of building software systems is communicating system requirements to the developers of the system. In formal software development methodologies, this task is accomplished through documentation.
Extreme Programming techniques can be viewed as methods for rapidly building and disseminating institutional knowledge among members of a development team. The goal is to give all developers a shared view of the system which matches the view held by the users of the system. To this end, Extreme Programming favors simple designs, metaphor, collaboration of users and programmers, frequent verbal communication and feedback.
Simplicity
eXtreme Programming encourages starting with the simplest solution and refactoring to better ones. The difference between this approach and more conventional system development methods is the focus is on designing and coding for the needs of today instead of those of tomorrow, next week or next month. Proponents of XP acknowledge the disadvantage that this can sometimes entail more effort tomorrow to change the system; their claim is that this is more than compensated for by the advantage of not investing in possible future requirements that may change before they become relevant. Coding and designing for uncertain future requirements implies the risk of spending resources on something that might not be needed. Related to the previous value, "communication", simplicity in design and coding should improve the (quality of) communication. A simple design with simple code can be easily understood by every programmer in the team.
Feedback
Within eXtreme Programming, feedback is related to different dimensions of the system development:
- Feedback from the system: by writing unit tests the programmers have direct feedback from of the state of the system after implementing changes.
- Feedback from the customer: The functional tests are written by the customer and the testers. They will get concrete feedback about the current state of their system. This review is planned once in every two or three weeks so the customer can easily steer the development.
- Feedback from the team: When customers come up with new requirements in the planning game the team directly gives an estimation of the time that it will take to implement.
Feedback is closely related to communication and simplicity. Flaws in the system are easily communicated by writing a unit test that proves a certain piece of code will break. The direct feedback from the system tells programmers to recode this part. A customer is able to test the system periodically according to the functional requirements (User Stories). To quote Kent Beck, Optimism is an occupational hazard of programming, feedback is the treatment.
Courage
The Extreme Programming doctrine of "Courage in system development" can be best explained by a couple of practices. One is the commandment to always design and code for today and not for tomorrow. This is an effort to avoid getting bogged down in design and requiring a lot of effort to implement anything else. Courage enables developers to feel comfortable with refactoring their code when necessary. This means reviewing the existing system and modifying it so that future changes can be implemented more easily. Another example of courage is knowing when to throw code away. Every programmer has experienced getting stuck on a complex problem in their own design and code after working on it all day, then coming back the next day with a clear and fresh view and rapidly solving the problem in half an hour.
Respect
Principles
The principles that form the basis of XP are based on the values just described and are intended to foster decisions in a system development project. The principles are intended to be more concrete than the values and more easily translated to guidance in a practical situation.
Rapid feedback
Feedback is most useful if it is done rapidly. The time between an action and its feedback is critical to learning and making changes. In eXtreme Programming, unlike traditional system development methods, contact with the customer occurs in small iterations. The customer has clear insight into the system that is being developed. He or she can give feedback on the progress and steer the development where is needed.
Unit tests also contribute to the rapid feedback principle. When writing code, the unit test provides direct feedback as to how the system reacts to the changes one has made. If, for instance, the changes affect a part of the system that is not in the scope of the programmer who made them, that programmer will not notice the flaw. There is a large chance that this bug will appear when the system is in production.
Assume simplicity
Assuming simplicity is about treating every problem as it can be solved "eXtremely simply". Traditional system development methods say to plan for the future and to code for reusability. Extreme programming rejects these ideas.
Incremental changes
The advocates of eXtreme Programming say that Rome wasn't built in a day: making big changes at once does not work. eXtreme Programming applies incremental changes: for example, a system in might have small releases every three weeks. By making lots of little steps the customer has more control over the development process and the system that is being developed.
Embracing change
It is always certain that there will be uncertainty. The principle of embracing change is about not working against changes but embrace them. For instance, if at one of the iterative meetings it appears that the customer's requirements have changed dramatically, programmers are to embrace this and plan the new requirements for the next iteration.
Activities
XP describes four basic activities that are performed within the software development process.
Coding
The advocates of XP argue that the only truly important product of the system development process is code (a concept to which they give a somewhat broader definition than might be given by others). Without coding you have nothing.
Coding can be drawing diagrams that will generate code, scripting a web-based system or programming an object-oriented C# program that needs to be compiled.
Coding can also be used to figure out the most suitable solution. For instance, XP would advocate that faced with several alternatives for a programming problem, one should simply code all solutions and determine with automated tests (discussed in the next section) what solution is most suitable.
Coding can also help to communicate thoughts about programming problems. A programmer dealing with a complex programming problem and finding it hard to explain the solution to fellow programmers might code it and use the code to demonstrate what he or she means. Code, say the exponents of this position, is always clear and concise and cannot be interpreted in more than one way. Other programmers can give feedback on this code by also coding their thoughts.
Testing
One cannot be certain of anything unless one has tested it. Testing is not a perceived, primary need for the customer. A lot of software is shipped without proper testing and still works (more or less). In software development, XP says this means that one cannot be certain that a function works unless one tests it. This raises the question of defining what one can be uncertain about.
- You can be uncertain whether what you coded is what you meant. To test this uncertainty, XP uses Unit Tests. These are automated tests that test the code. The programmer will try to write as many tests he or she can think of that might break the code he or she is writing; if all tests run successfully then the coding is complete.
- You can be uncertain whether what you meant is what you should have meant. To test this uncertainty, XP uses acceptance tests based on the requirements given by the customer in the exploration phase of release planning.
Listening
Programmers don't necessarily know anything about the business side of the system development. The function of the system is determined by the business side. For the programmers to find what the functionality of the system should be, they have to listen to business.
Programmers have to listen "in the large": they have to listen what the customer needs. Also, they have to try to understand the business problem, and to give the customer feedback about his or her problem, to improve the customer's own understanding of his or her problem.
Communication between the customer and programmer is further addressed in The Planning Game (see below).
Designing
From the point of view of simplicity, one could say that system development doesn't need more than coding, testing and listening. If those activities are performed well, the result should always be a system that works. In practice, that will not work. One can come a far way without designing but at a given time one will get stuck. The system becomes too complex and the dependencies within the system cease to be clear.
One can avoid this by creating a design structure that organizes the logic in the system. Good design will avoid lots of dependencies within a system; this means that changing one part of the system will not affect other parts of the system.
Practices
Planning Game
The main planning process within eXtreme Programming is called the planning game. This section will explain this process by using process models.
The planning process is divided into two parts:
- Release Planning: This is focused on determining what requirements are included in which release and when its going to be delivered. The customers and developers are both part of this. Release Planning consists of three phases:
- Exploration Phase: In this phase the customer will give all his requirements for the system. These will be written down on user story cards.
- Commitment Phase: Within the commitment phase business and development will commit themselves to the functionality that will be included and the date of the next release.
- Steering Phase: In the steering phase the plan can be adjusted, new requirements can be added and or existing requirements can be changed or removed.
- Iteration Planning: This plans the activities and tasks of the developers. In this process the customer is not involved. Iteration Planning also consists of three phases:
- Exploration Phase: Within this phase the requirement will be translated to different tasks. The tasks are recorded on task cards.
- Commitment Phase: The tasks will be assigned to the programmers and the time it takes to complete will be estimated.
- Steering Phase: The tasks are performed and the end result is matched with the original user story.
Missing image
Planninggame.gif
Image:Planninggame.gif
Exploration phase Release planning
This is an iterative process of gathering requirements and estimating the work impact of each of those requirements.
- Get Requirement from Customer: Business has come with a problem; during a meeting, development will try to define this problem and get requirements.
- Write a Story: Based on the business problem, a story has to be written. This is done by business, where they point out what they want a part of the system to do. It is important that development has no influence on this story for so far the functionality. The story is written on a so called user story card.
- Split a Story: If development isn't able to estimate the story (next item), it needs to be split up and written again. Again, this may not influence the business requirements.
- Estimate a Story: Development estimates how long it will take to implement the work implied by the story card.
When business cannot come up with any more requirements, one proceeds to the commitment phase.
Missing image
Expl_rp.gif
Image:expl_rp.gif
Commitment Phase Release planning
This phase involves the determination of costs, benefits, and schedule impact. It has four components:
- Sort by Value: Business sorts the user stories by business value.
- Sort by Risk: Development sorts the stories by risk.
- Set Velocity: Development determines at what speed they can perform the project.
- Choose scope: The user stories that will be finished in the next release will be picked. Based on the user stories the release date is determined.
Missing image
Com_rp.gif
Image:com_rp.gif
Sort by value
The business side sorts the user stories by business value. They will arrange them into three piles:
- Critical: stories without which the system cannot function or has no meaning.
- Significant Business Value: Non-critical user stories that have significant business value.
- Nice to have: User stories that do not have significant business value; an example can be an improvement in usability or presentation.
Missing image
Com_rp2.gif
Image:com_rp2.gif
Sort by risk
The developers sort the user stories by risk. They also categorize into three piles: low, medium and high risk user stories. The following is an example of an approach to this:
- Determine Risk Index: Give each user story an index from 0 to 2 on each of the following factors:
- Completeness (do we know all of the story details?)
- Complete (0)
- Incomplete (1)
- Unknown (2)
- Volatility (is it likely to change?)
- low (0)
- medium (1)
- high (2)
- Complexity (how hard is it to build?)
- simple (0)
- standard (1)
- complex (2)
- Completeness (do we know all of the story details?)
All indexes for a user story are added, assigning the user stories a risk index of low (0–1), medium (2–4), or high (5–6).
Missing image
Com_rp3.gif
Image:com_rp3.gif
Steering phase Release planning
Within the steering phase the programmers and business people can "steer" the process. That is to say, they can make changes. Individual user stories, or relative priorities of different user stories, might change; estimates might prove wrong. This is the chance to adjust the plan accordingly.
Exploration phase - Iteration planning
The exploration phase of the iteration planning is about creating tasks and estimating their implementation time.
- Gather User Stories: Gather and write all user stories for the next release
- Combine/Split task: If the programmer cannot estimate the task because it is too big or too small, the programmer will need to combine or split the task.
- Estimate task: Estimate the time it will take to implement the task.
Missing image
Expl_ip.gif
Image:expl_ip.gif
Commitment phase - Iteration planning
Within the commitment phase of the iteration planning programmers are assigned tasks that reference the different user stories.
- A programmer accepts a task: Each programmer picks a task for which he or she takes responsibility.
- Programmer estimates the task: Because the programmer is now responsible for the task, he or she should give the eventual estimation of the task.
- Set load factor: The load factor represents the ideal amount of hands-on development time per programmer within one iteration. For example, in a 40-hour week, with 5 hours dedicated to meetings, this would be no more than 35 hours.
- Balancing: When all programmers within the team have been assigned tasks, a comparison is made between the estimated time of the tasks and the load factor. Then the tasks are balanced out among the programmers. If a programmer is overcommitted, other programmers must take over some of his or her tasks and vice versa.
Missing image
Com_ip.gif
Image:com_ip.gif
Steering phase - Iteration planning
The implementation of the tasks is done during the steering phase of the iteration planning.
- Get a task card: The programmer gets the task card for one of the tasks to which he or she has committed.
- Find a Partner: The programmer will implement this task along with another programmer. This is further discussed in the practice Pair Programming.
- Design the task: If needed, the programmers will design the functionality of the task.
- Write unit test: Before the programmers start coding the functionality they first write automated tests. This is further discussed in the practice Unit Testing.
- Write code: The programmers start to code.
- Run test: The unit tests are run to test the code.
- Run Functional test: Functional tests (based on the requirements in the associated user story and task card) are run.
Missing image
Ste_ip.gif
Image:ste_ip.gif
Pair Programming
Pair Programming means that all code is produced by two people programming on one task on one workstation. One programmer has control over the workstation and is thinking mostly about the coding in detail. The other programmer is more focused on the big picture, and is continually reviewing the code that is being produced by the first programmer.
The pairs are not fixed: it's even recommended that programmers try to mix as much as possible, so that everyone knows what everyone is doing, and everybody can become familiar with the whole system. This way, pair programming also can enhance team-wide communication. (This also goes hand-in-hand with the concept of Collective Ownership).
Collective ownership
Collective ownership means that everybody is responsible for all the code; this, in turn, means that everybody is allowed to change any part of the code. Pair programming contributes to this practice: by working in different pairs, all the programmers get to see all the parts of the code. A major advantage claimed for collective ownership is that it speeds up the development process, because if an error occurs in the code any programmer may fix it.
By giving every programmer the right to change the code, there is risk of errors being introduced by programmers who think they know what they are doing, but do not foresee certain dependencies. Well enough defined unit tests address this problem: if unforeseen dependencies create errors, then when unit tests are run, they will show failures.
On-site customer
Within XP, the "customer" is not the one who pays the bill, but the one who really uses the system. XP says that the customer should be at hand at all times and available for questions. For instance, the team developing a financial administration system should include a financial administrator.
Unit testing
Unit tests are automated tests that test the functionality of pieces of the code (e.g. classes, methods). Within XP, unit tests are written before the eventual code is coded. This approach is intended to stimulate the programmer to think about conditions in which his or her code could fail. XP says that the programmer is finished with a certain piece of code when he or she cannot come up with any further condition on which the code may fail.
Refactoring
Because XP doctrine advocates programming only what is needed today, and implementing it as simply as possible, at times this may result in a system that is stuck. One of the symptoms of this is the need for dual (or multiple) maintenance: functional changes start requiring changes to multiple copies of the same (or similar) code. Another symptom is that changes in one part of the code affect lots of other parts. XP doctrine says that when this occurs, the system is telling you to refactor your code by changing the architecture, making it simpler and more generic.
Controversial aspects
Extreme Programming also has its share of controversial aspects:
- Detailed specifications are not created or preserved.
- Programmers are required to work in pairs - not all software developers expect to be asked to work this way.
- There is no Big Design Up Front. Most of the design activity takes place on the fly and incrementally, starting with "the simplest thing that could possibly work" and adding complexity only when it's required by failing tests. This could result in more re-design effort than only re-designing when requirements change.
- A customer representative is attached to the project. This role can become a single-point-of-failure for the project and some people have found it to be a source of stress.
In 2003, Matt Stephens and Doug Rosenberg published a book under Beck's XP series imprint called "Extreme Programming Refactored: The Case Against XP" which questioned the value of the XP process and suggested ways in which it could be improved (i.e. refactored). This triggered a lengthly debate in articles, internet newsgroups and web-site chat areas. The core argument of the book is that XP's practices are interdependent but that few practical organisations are willing/able to adopt all the practices; therefore the entire process fails. The book also makes other criticisms and it draws a likeness of XP's "collective ownership" model to socialism (the authors appear to regard this as undesirable).
Certain aspects of XP have changed since the book was published, in particular XP now accomodates modifications to the practices as long as the required objectives are still met. It also uses increasingly generic terms for processes. Some argue that these changes invalidate previous criticisms; others claim that this is simply watering the process down.
Recently, authors have attempted to reconcile XP with the older methods that XP sought to replace (such as the waterfall method) in order to offer a unified method. See http://www.lux-seattle.com/resources/whitepapers/waterfall.htm for an example.
See also
References
Pro-XP, generally considered authoritative
- Kent Beck: Extreme programming explained: Embrace change, Addison-Wesley, ISBN 0201616416
- Kent Beck and Martin Fowler: Planning Extreme Programming, Addison-Wesley, ISBN 0201710919
- Martin Fowler: Refactoring: Improving the Design of Existing Code, Addison-Wesley, ISBN 0201485672
- Ken Auer and Roy Miller: Extreme Programming Applied: Playing To Win, Addison-Wesley, ISBN 0201616408
- Ron Jeffries, Ann Anderson and Chet Hendrickson: Extreme Programming Installed, Addison-Wesley, ISBN 0201708426
- Kent Beck: Extreme programming explained: Embrace change, Second Edition, Addison-Wesley, ISBN 0321278658
Anti-XP
- Matt Stephens and Doug Rosenberg: Extreme Programming Refactored: The Case Against XP, Apress, ISBN 1590590961
Background reading
- Alistair Cockburn: Agile Software Development, Addison-Wesley, ISBN 0201699699
External links
- Ward Cunningham's Portland Pattern Repository wiki page on Extreme Programming
- Client-oriented introduction to XP
- Ron Jeffries web-mag XProgramming.com - an Extreme Programming Resource
- ExtremeProgramming.org
- Matt Stephens's satirical website, for an in-depth critique of Extreme Programming problems (also see this page for further links to XP critiques)
- Martin Fowler on XP
- Pair Programming, an Extreme Programming practice
- Manifesto for Agile Software Development
- Citations from CiteSeer
- Article XP Testing Without XP: Taking Advantage of Agile Testing Practices from the Methods & Tools magazine
- Article Will Pair Programming Really Improve Your Project? from the Methods & Tools magazine