XP 2018

Given that most teams (Agile or not) report test automation rates around 30%, it’s not surprising that many development teams cite testing as one of their primary pain points. Adopting Agile, with its fast and frequent iterations, only exacerbates underlying problems with traditional approaches to test automation. Brittle UI tests need constant maintenance to keep pace with the evolving application, late-iteration testing is often the bottleneck for getting user stories “done done” on time, and slow test execution time is not compatible with CI and Agile’s emphasis on fast feedback. In fact, it’s not uncommon for regression suite execution time to exceed the total length of the sprint itself!
For teams to meet business expectations for more software—faster—without putting the business at risk, testing needs to shift perspectives. We used to ask “Are we done testing,” and answer it by checking whether new requirements/stories were covered and how many regression tests were executed. That’s no longer feasible when a releasable candidate must be delivered every two weeks, or sooner—and a software failure is now considered a business failure. To protect the brand and the user experience in the context of Agile release cycles, we need to ask “Does the release candidate have an acceptable level of risk?” This is where Continuous Testing comes in.