Question: how to do acceptance testing in Scrum?

When starting to use Scrum for software development, many teams approach acceptance testing in a more or less traditional way. Developers focus on implementing the spring backlog items within the sprint. At the end of the sprint a new release of the system is delivered and declared ready for acceptance testing.

There are is a problem with this approach. When a Product Backlog Item is not tested for acceptance, is is not completely Done. When acceptance testing is done after the sprint, that is, during the next sprint, any corrections resulting from its feedback can only be planned in the sprint after that. Much too late! This approach tends to result in an ever increasing stack of corrections taking time that should be used to implement new Product Backlog Items, and thus slowing down development.

So, why don't we do acceptance testing within the sprint? In this way we receive earlier feedback and have the opportunity to repair any issues right away and have the Backlog Item really done by the end of the sprint. But this approach has its own problems. For instance, team members cannot do acceptance testing themselves, but someone outside has to do this. This means that the software must be made available to some acceptance testing environment, where outsiders can perform their tests. They need a reasonable amount of time for that, which means that near the end of the sprint developers have to stop developing because whatever they deliver cannot be tested anymore. That means loss of precious time.

Do you have experience with acceptance testing within Scrum sprints? By the way, this problem exists in every iterative style of development; not just in Scrum. What solutions work for you?


mdcroon said...

In our situation we are creating acceptance tests together with the product owner during the sprint. The acceptance tests are based directly on the acceptance criteria of a user story. During the first couple of days of the sprint the testers work together with the product owners in order to create the acceptance tests. We prefer to use the given-when-then-else format. This works out very good for us. It also gives a nice overview for the developers. Note that we are working with internal customers. I think this will make a difference, but on the other hand business should be closely involved during a sprint.
As soon as the acceptance tests are passed, from functionality perspective also the user story is done (apart from other items in the Definition of Done of course).


Eric Jan Malotaux said...

Thank you for your comment. Perhaps I should have mentioned that the small (3) development team is not colocated with the client, let alone the users, who are dispersed over the whole of the Netherlands. Given the short duration of the project, three sprints of three weeks each, the room for adjustments of the situation is limited.

Also the acceptance testers can only partially be part of the team, because they have other responsibilities as well.

For the current and last sprint, we decided on the following procedure: we will deliver each Sprint Backlog Item (SBI) as soon as possible to the acceptance test environment (ATE) using a semi-formal release procedure. With semi-formal I mean a tagged release with release notes. As soon as possible, but not more than once per day. In this way we hope to distribute the testing effort more evenly over the duration of the sprint and avoid running out of time for AT at the end of the sprint.