We all agreed that when GUI-level regression automation is developed in Release N of the software, most of the benefits are realized during the testing and development of Release N+1. I think that we were surprised to realize that we all shared this conclusion, because we are so used to hearing about (if not experiencing) the oh-so-fast time to payback for an investment in test automation.
Some benefits are realized in release N. For example:
o There’s a big payoff in automating a suite of aclearcase/" target="_blank" >cceptance-into-testing (also called "smoke") tests. You might run these 50 or 100 times during development of Release N. Even if it takes 10x as long to develop each test as to execute each test by hand, and another 10x cost for maintenance, this still creates a time saving equivalent to 30-80 manual executions of each test case.
o You can save time, reduce human error, and obtain good tracking of what was done by automating configuration and compatibility testing. In these cases, you are running the same tests against many devices or under many environments. If you test the program’s compatibility with 30 printers, you might recover the cost of automating this test in less than a week.
o Regression automation facilitates performance benchmarking across operating systems and across different development versions of the same program.
Take advantage of opportunities for near-term payback from automation, but be cautious when automating with the goal of short-term gains. Cost-justify each additional test case, or group of test cases.
If you are looking for longer term gains, across releases of the software, then you should seriously thinking about setting your goals for Version N as:
o providing efficient regression testing for Version N in a few specific areas (such as smoke tests and compatibility tests);