Agile Development - Bertrand Meyer (edX)

Agile - The Good, The Hype, and the Ugly

This six week edX course covers the concepts, principles, practices, myths and misconceptions of Agile project development.  Lectures are done by Bertrand Meyer, and while not required, the recommended text is Meyer's book Agile - The Good, The Hype, and the Ugly.  According to Meyer, the intent of the class is to look at the Agile approach to software development in an objective way, identifying shortcomings and myths of Agile, as well as its strengths and merits.  The graded portion of the class consists of multiple choice quizzes, so all in all it's a very lightweight course.

Week 1 - Context

The Agile Manifesto

The Agile Manifesto is a document created in 2001 by a group of software engineers (mostly consultants) that laid out four values and 12 principles for a better way of developing software.  It was a reaction to the perceived shortcomings of traditional software engineering methodologies (such as waterfall).  It put a great deal of emphasis on customer interaction, short iterations, and continuously delivering working software.

Agile Methods

Four methodologies are covered

• Extreme Programming (XP) - created by Kent Beck.  Minimizes emphasis on management, puts programmers at the forefront.  A reaction against the "pointy haired bosses" of the software development management world.
• Lean - Tom and Mary Poppendieck.  Based on the concepts from Lean Manufacturing (Toyota).  Emphasizes the elimination of waste (undelivered documentation, work in process, etc.).
• Crystal - Alistair Cockburn.  Aims to combine best of agile with best of process methods.
• Scrum - Schwaber & Sutherland.  Most widespread methodology in practice.  Less technical, more of a management methodology.

Official agile principles

This segment looks at the 12 principles.  Meyer dissects the 12 principles and points out a number of problems with them.  The principles are a good starting point, but the have some deficiencies:

• Principles should be general, abstract, and prescriptive, but some of the "principals" are concrete and practical (practices), while other are statements about the world (assertions)
• Several of the Principles are redundant.  Delivering working software is mentioned in three different principles.  
• No mention of testing
• Beef with the notion that simplicity is attained by not working.  Rather, simplicity is attained by continually cleaning and polishing... i.e. working hard.

Agile values

Values are the general philosophy underlying the Agile approach.  These are not the same as the four values laid out in the agile manifesto:  individuals and interaction over process and tools, working software over documentation, customer collaboration over contract negotiation, and responding to change over following a plan.  The values Meyer lays out are:

• New, reduced role for manager - in particular, manager does not assign tasks
• No "Big Upfront" steps - architecture and design should be done "as you go"
• Iterative development - iteration are short (a few weeks), rather than many months 
• Limited, negotiated scope - only build what is needed
• Focus on quality, achieved through testing

Week 2 - Principles

No Big Upfront Anything

Before agile approaches came about, the software development lifecycle approach in widespread use was the "waterfall" method, described by Royce in 1970 (for the US Air Force).  Described a set of steps for developing a software system.  Problems with waterfall include:

• Late appearance of code
• Lack of support for requirement changes
• Lack of support for maintenance
• Division of labor hampers quality management
• Impedance mismatches - translating between requirements, implimentation, and testing can lead to miscommunication
• Highly synchronous model

Agile generally views upfront development of requirements (especially documentation) as wasteful and counterproductive, as these documents are not a deliverable and they rapidly become obsolete as requirements change over the life of the project.  Meyer cautions against rejecting any up front planing and design, though, arguing that having an understanding of the problem is necessary in order to build the right solution.  He notes that he's seen agile projects fail due to a refusal to do any up from planning or design.

Organizational Principles

• Put the customer at the center 
• Accept change
• Let the team self-organize
• Maintain a sustainable pace
• Produce minimal software:
• Produce minimal functionality
• Produce only the produce requested
• Develop only code and tests

Involving the customer throughout the process to ensure that you are not only building the system right, but building the right system.  Real customers (as opposed to proxies) are the best resource.  In XP this is the "embedded customer", in Scrum the "product owner".

The Agile Manifesto implores us to "welcome" change, which Meyer says is a bit of an exaggeration. Software systems should be designed to be extendable and easy to evolve, which object oriented programming is positioned to leverage.

In self organized agile teams, managers do not tell workers what to do.  Instead they work to encourage the team, remove impediments, and generally provide the needed support for the team.  There role is more one of facilitator rather than slave driver.

Maintaining a sustainable pace recognizes that knowledge workers are more productive when they are not over-stressed.  40 hour weeks (or at least infrequent overtime), frequently merging, testing, and refactoring of code, and collaboration can all help achieve this goal.

Producing minimal software reduces waste.  YAGNI (you ain't gonna need it) stems from XP and discourages working on anything but the story we currently have.  Meyer (and the wikipedia article) note that if improperly applied this principle can result in poor code.  Doing too much abstraction too early can result in a disproportionate increase in the complexity of code and make future development more difficult in a non-linear way.

Lean software development identifies several sources of waste in software projects:

• Extra/Unused features
• Partially developed work not released
• Intermediate/unused artifacts
• Seeking information (missing info on customer needs, algorithm, etc.)
• Escaped defects not caught by tests/reviews
• Waiting (including Customer Waiting)
• Handoffs (e.g. translating from UML to code)

Ward Cunningham implores us to minimize the amount of checks and generalization we do and aim for the simplest thing that could possibly work, a slogan often associated with XP.  Meyer thinks this statement goes too far, is luddite, and should be ignored (that is, there is value in looking toward the future).

Technical Principles

• Develop iteratively
• Produce frequent working iterations
• Freeze requirements during iterations (Closed-Window Rule)
• Treat tests as a key resource
• Do not start any new development until all tests pass
• Test first
• Express requirements through scenarios

Developing "iteratively", in the context of Agile, means that each iteration produces a working (if not feature complete) product, rather than each iteration focusing on only one layer of the application.  This means that concerns and systems such as database, network, business logic, and ui all get worked on in an iteration.  The Closed-Window Rule prescribes that during an iteration, no one may add functionality (no exceptions).  This rule keeps priorities predictable.  If something absolutely must be worked on immediately, then the current sprint is canceled.

Meyer contends that while Agile scorns working on infrastructure exclusively because it does not produce a working product, it is sometimes necessary as part of good engineering to ensure that the infrastructure and environment will support a working product, and thus is it acceptable to spend some time up front on these activities, keeping in mind not to spent an exorbitant amount of time on it.

Within practical constraints, the next iteration should not start until the test suite is passing.

Requirements are often written as "User Stories" in the form of "As a <user role>, I want <functionality>, so that <justification>".  Meyer points out that a weakness of this form of user story is that it is an example, which doesn't explicitly get to the underlying general abstraction.  They are still useful for validating more abstract requirements.

Method Specific Principles

Eliminate Waste:  central principle in Lean.  Everything that is not adding value is considered waste:  unnecessary or partially finished code and functionality, delays, unclear requirements, insufficient testing, avoidable repetition, slow communication, defects, and managerial overhead and bureaucracy.

Amplify Learning:  Another concept from Lean, speeding up learning gives us greater information.

• To prevent accumulation of defects, run tests as soon as code is written
• Instead of documenting and planning, experiment in code (try different solutions)
• Present UI to end users and get input
• Short iterations that include refactoring and integration testing
• Feedback sessions with customer

Delay decisions as long as possible, so decisions can be based on more facts rather than assumptions.  This necessitates building in capacity for change.  Just be careful not to procrastinate.

Focus on individual tasks rather than trying to "multitask", minimize interruptions, and avoid frequent project switching (stay on one project for multiple days or weeks)

Deliver as fast as possible (with caveats).  Try multiple designs in parallel (not used much because of cost).  Minimize dependencies (make user stories as independent as possible).  Highly dependent functionalities tend to multiply complexity.

Crystal emphasizes people and includes the notion of "Personal Safety", which encourages the free expression of ideas and restraining from ridicule in order to build trust.  XP recognizes that developers expect safety, accomplishment, belonging, growth, and intimacy.

Week 3 - Roles

Traditional Manager Roles

In a traditional setting, one of the primary tasks of a manager is defining goals and deadlines, assigning tasks, interfacing with customers and upper management, enforcing rules and deadlines, etc.  The success of a project is heavily influenced by the effectiveness of the manager.

Agile aims to shift many of these responsibilities away from the manager.

Three Scrum Roles

The (Self-Organizing) Team - cross functional team of 5-9 people that can do everything within guidelines to reach goal.  Selects goals for the iteration (what is worked on from the backlog), and demos work results to the product owner.

The Product Owner - responsible for ensuring the project meets the customer's needs (in practice this actually is the customer).  Defines features, release date and content, project ROI.  Prioritizes the backlog and has final say on whether features are "done".  Is not in charge of what specific feature a developer works on from day to day... indirectly influences this by prioritizing backlog, but team still decides how much work goes into a given sprint.

The Scrum Master - responsible for enforcing the methodology of scrum.  Shields team from external influences and removes impediments (secures needed resources, works with other teams, etc.).  Normally does not develop (in theory).

During Scrum meetings, only core participants (those "committed" to the outcome i.e. "pigs") are permitted to participate in discussion.  Other stakeholders (those who are "involved", i.c. the "chickens") may chime in if prompted to do so.  This prevents scrum meetings from getting off track.

In strict scrum there is not a "manager", but if there is a manager, the role should be focused on supporting the team, contributing wisdom and expertise (being a guru rather than a nanny).

Other Agile Roles

Expert user - (Crystal) expert knowledge in the problem area, can answer questions and suggest possible solutions.  Should be a bona-fide user and not just a tester.

Customer - (XP) must trust the developer's technical decision, while providing precise stories and prioritizing those stories.  Collaborates closely with team.

Developer - (XP, Scrum) Technical expert, chooses implementation for stories, identify risky features and point out to customer.  Must answer questions: "How will we implement it? How long will it take? What are the risks?".

Tracker - (XP, Scrum) keeps track of schedule, measures velocity.  Also may keep track of things like amount of overtime worked, changes in velocity, changes in # of passing tests.

Coach - (XP) Guides team, mentor, teach by doing.  Mostly done by Scrum Master in scrum.

Week 4 - Practices

Meetings

Most important meeting is the daily meeting (often called a "stand up" or "scrum").  Short meeting (usually 15 minutes strictly enforced).  Involves all "committed" team members.  Team answers three questions:  What did I do yesterday? What am I doing today? Am I facing any impediments?

Planning meeting (scrum) held at the beginning of every sprint and defines the work for that sprint.  8 hour time limit, split between time with product owner to prioritize product backlog, and time with team to produce sprint backlog, which defines the tasks for that sprint.

Retrospective meeting asks What when right? and What could be improved? during the past sprint.  The purpose of this meeting is to reflect on the process and identify process improvements. (3hr limit)

Review meeting reviews the completed and uncompleted work, demos completed work to stakeholders. (4hr limit)

Reflective improvement (Crystal) - developers should take break from development to look for ways to improve.

Development

Pair Programming is a technique from XP that has two programmers sitting together at one machine, one coding ("Driver") and the other paying close attention ("Observer").  Observer must be focused on task at hand.  This allows them to keep each other accountable and on task, brainstorm ideas, help each other get "unstuck".  Should not be confused with mentoring (pairing junior developer with senior developer).

Single code branch - XP discourages branching.

Shared code - reject "code ownership", or having bits of code that "belong" to any one developer.  Ownership creates dependencies and can lead to lack of team accountability (blame game)

Leave optimization until last - rather than guess what will need optimization, analyze actual results after the fact.  Meyer cautions that this isn't necessarily always a good idea, since complex algorithms may favor more up front analysis.

Simple design - "Do the simplest things that would work" and then refactor.

Design iteratively - rather than abstracting first, just do the simplest thing that will work.

Refactoring - restructuring existing code without affecting the external behavior.  The idea is the make the design more elegant, and the code cleaner, without breaking other code (or failing previously passed tests).  Meyer contends this technique is best applied to a good (but not perfect) design, and feels it is absurd to believe it can mold a bad design into a good one.

Release

Releases should be early and often.  Continuous Integration (CI) keeps the system fully integrated at all times by frequently releasing and rigorously testing.  CI involves building the system several times a day.  Because builds are so frequent, integration is easier because changes are smaller, and failures are caught earlier.

In XP, releases should be small.  Release running, tested software every iteration, and release to end users frequently as well (which may mean as often as daily for web based projects).

Functionality should be deployed gradually, since "big bang" deployment is risky.  This can even mean daily deployment.  "China Shop rules": you break it, you fix it.

Ten minute build - use automation to ensure that integration can be run often.  If it takes longer to build, break it into subprojects.

Weekly and Quarterly cycles in XP.  In XP, work is planned a week at a time, with the goal of having tests passing and deployable software at the end of the week.  Quarterly, review high level system structure and overall process.

Testing and Quality

Agile is focused on quality, with testing playing a central role in ensuring quality.

Coding standards unsure all members are coding with same conventions.

All code must have a unit test.  TDD has us write failing test before writing actual code.  In TDD, the code replaces the specification.  The expected benefits of this approach include catching bugs early, writing more tests, and improved productivity.  Another major benefit of TDD is that building up tests creates a collection of regression tests.

"A bug isn't an error, it is a test you forgot to write"

Root-cause analysis encourages digging deeper into the underlying reason for a defect so that future defects can be prevented.

Acceptance tests are black box tests representing expected results from the system.  Should be automated so they can run frequently, and the results published.

Management etc.

Scrum of scrums - addresses scaling up agile practices to larger projects.  Collection of scrum projects have a daily meeting, with similar questions as the daily scrum, but focused on the entire team and the teams interaction with other teams, rather than the individual developer.

Whole Team is an idea that teams should be cross functional and all sit together - programmers, testers, analysts, maybe even manager.

Planning poker is a technique for creating estimates for the amount of effort required for tasks on the backlog.  Often uses a Fibonacci scale to avoid quibbles over slight differences.  Meyer thinks this is more of a "folk lore" item an a widespread practice.

Open Workspace - organized around pairing stations, with whiteboard space.  Organize people around the discussions they should overhear.  Should provide room for personal effects and private conversations.  Crystal has a concept of "osmotic communication" that is encouraged by such an arrangement.  Making project management artifacts like burndown charts and backlogs should be part of the workspace.

Customer always available.

Week 5A - Artifacts

Two kinds of artifacts - "virtual" or intellectual artifacts, and tangible artifacts.  The tangible artifacts are typically there to support a more abstract intellectual idea.

User story: describes a unit of functionality. A given user story is divided into tasks, which are the individual pieces that must be implemented to complete the feature for the user story.

Use case: diagram type in UML.  Purpose is similar to user story, describes a single business goal through interactions between external actors and system.  May be more complex, and more precise, than a user story.

Product Backlog:  Prioritized list of user stories.  It is owned by the Product Owner, but is editable by the entire team.  Also includes estimates of business value and required effort for user stories.

Task Board (Story Board):  Used to visually track the state of tasks from "to do" to "in progress" to "done"

Velocity: measure of progress in project.  Measured in units of items delivered (user stories, tasks, story points).  Material counterpart is the burndown chart, which is a visual chart showing the remaining story points (or whatever unit the team uses) hopefully decreasing over time.

Bullpen:  Open work space.  The idea is it's a single open room that encourages communication between team members.

Week 5B - Assessment of Agile methods

The Ugly

These are ideas from Agile that actually harm the software development process.

• Rejection of upfront tasks
• Particularly: no upfront requirements
• Dismissal of a priori architecture work
• User stories as a replacement for abstract requirements
• Tests as a replacement for specification
• Feature-based development & ignorance of dependencies
• Method keeper (e.g. Scrum Master) as a separate role
• Test-driven development (but not the rest of agile's emphasis on tests)
• Dismissal of...
• ... traditional manger tasks
• ... auxiliary products and non-shippable artifacts
• ... a priori concern for extendibility
• ... a priori concern for reusability

The Hype

These ideas have been shown to be "indifferent", or agile "folk lore".  That is, they are held up as best practices by various agile methods, but in practice don't seem to have a big affect one way or the other.

• Pair programming
• Open-space working arrangements
• Self-organizing teams
• Maintaining a sustainable pace
• Producing minimal functionality
• Planning Poker
• Cross-functional teams
• Embedded customer

The Good

These are the ideas from agile that have a positive affect on software engineering practices.

• Acceptance of change
• Iterative development
• Emphasis on working code
• Tests as one of the key resources of the project
• Constant test regression analysis
• Notion of velocity
• No branching
• Product (but not user stories!) burndown chart
• Daily meeting

The Brilliant

These ideas really make a fundamental difference in the way we develop software:

• Short iterations
• Closed-window rule
• Refactoring (but not as a substitution as a design)
• Associating a test with every piece of functionality
• Continuous integration

Final Results

Piece o cake =D