In the quality assurance field, the interest in automated surface tests (below still often referred to as automated tests / test suite) is currently as big as ever. The topic is thus not only becoming increasingly popular among customers – in terms of the in|sure Ecosphere as well, automated surface tests will help with testing cross-product end-2-end scenarios in the future.
Yet within product development teams as well, test automation is frequently utilized to improve quality assurance processes. This is because it has great potential and the basic idea is simple: Automated tests are faster, more cost-efficient and more reliable than manual tests. Sounds good, right? Let’s automate!
Automated tests are faster … right?It’s clear that computers work faster than humans. Let’s imagine a ready-to-use automated test suite that executes a non-changing sequence of application interactions, and in doing so always expects the same foreseeable result. This kind of test suite is not only faster, it also regularly checks the sequence as well as after every change, and across several browsers simultaneously. Even an experienced expert in the field of software testing cannot do that.
The implementation takes time
Throughout its duration, an incremental development project is still often characterized by requirements that are constantly changing. If one combines this with the almost typical absence of complete and clear application specifications, the ideal environment conditions for a successful test automation will often first be achieved with a certain degree of product maturity.
We now try to factor in automated surface tests early on in the development process, which makes for quite an interesting approach. Ultimately, it is the constant changes and adjustments to the software that make a precise regression plan so important. However, the project may turn out to be like a road trip across Canada with a 1.0 liter 3-cylinder Toyota Yaris: You can do it, but slowly, and you may stall. The reason for this is that every change to the application potentially involves an adjustment to the automated test. It is why it is sometimes extremely time consuming to design a complex test suite, and this initial extra effort and expenses is often underestimated. On the other hand, dedicated software testers are flexible and understand how to adapt the testing efforts to the constantly changing requirements and priorities in the shortest possible time.
A targeted approach is important
In order to thus not get on the wrong track during an early implementation of an automated test suite, it is absolutely necessary that you are clear about the initial scope of the test automation. Ideally, you’ll begin with the overlap from the most important processes and the processes that will likely be subject to the least adaptions and changes.
Automated tests are more cost-efficient … right?
In order to be able to the quantify an additional value achieved via an automated test suite, automated tests are often balanced against the costs for manual tests. However, you should always consider the fact that you’re dealing with completely different testing methods with varying conceptional bases. If we consider the expenses for test automation (if applicable), these are on the one hand the incurred license and direct operating costs for the infrastructure, but on the other hand, of course, they are primarily the overhead and personnel costs for the implementation. These can then be subdivided into apparent costs and the costs that tend to go by the board.
What is rarely considered
Another point that must be mentioned here are staff salaries. For the implementation of a comprehensive, automated test suite, full-time test engineers with basic programming experience are frequently in demand. Among (manual) software testers, in contrast, there are often individuals who lack much of a technical background, have switched careers, and are still figuring things out. A glance at the most commonly-used comparison portals shows that the average salary for test engineers due to their required, technical expertise is simply higher than that of software testers. This is something needs to be considered.
Automated tests are more reliable … right?
Once the first machine submits a sick note on its own, we’ll likely have a quite different set of problems. Subtle dystopian future aside … subtracting the human factor from a test is probably not a new approach and even includes some benefits. However, it is indeed the case that the test object, the test case and the test automatism itself has been described by humans, which is why the complete process of test automation is in itself error-prone (“false negatives”). Manual tests cannot exonerate themselves from this either. For this reason, it is absolutely necessary that the automated (and manual) suites themselves are subject to a review process, ideally on a regular basis (what was I saying about costs again?). Far too often, test automation efforts are operated according to a “fire and forget” principle, which means that their reliability and sustainability must be called into question.
Automated tests provide security
But what exactly are automated tests currently good at? As soon as an automated test has been run without having found an error, it will thus never be able to find a new one. That is, unless the underlying application code changes. Though in this case as well, you still need to verify whether you’re dealing with an error or an undesired change (“false positives,” with regard to costs). But that is exactly the point. Automated tests are like motion detectors that light the passable and apparent walkways. And at the moment in which something happens, a (red) light goes on. And by doing that, they give software testers the security and possibility to address the project’s hidden defects.
Automated tests are no replacement, but they are an important tool
In a development project in which both manual and automated tests are used, you can be sure that the large majority of discovered errors can be attributed to manual inputs. And that often despite the months and years of inputs that have flowed into test automation. But the goal shouldn’t be to weigh both stand-alone test methods against each other. That wouldn’t be successful anyway, as they each come with their own unique requirements, dynamics and objectives. As automated tests check the “happy path,” they give software testers the opportunity to focus on tracking down bugs and to propel the overall quality improvement process. The test methods therefore complement each other and form two important cornerstones of a balanced testing strategy.
The two most important findings (one for take away, please)
In principle, an automated test suite is important. Nevertheless, its good reputation often gets too far ahead of itself. So that it can still be a sensible addition as a part of a well thought out testing strategy, there are two things that should in my opinion be given special consideration.
Firstly: Carefully think through and document the test automation process (incl. tooling). Otherwise, the costs that may be incurred for constantly tweaking the test suite will swallow up any additional value. In doing so, make sure that the product development process or the product to be tested have reached a certain degree of maturity.
Secondly: The test automation should only be regarded as an expansion of the testing strategy and in combination with manual tests. Never as a replacement.
Test automation enjoys a good image. And deservedly so, too! On the road to a sustainable investment, however, there are often-overlooked potholes that you should drive around. I hope that with this critical reflection I was able to make you aware of some of these potholes.
The adesso blog has some interesting articles on the subject of test automation. Whether it’s regarding tooling or implemented projects, it’s worth checking out what’s there.
Would you like to learn more about how in|sure helps make insurance companies ready for the future? Then feel free to get in touch with Karsten Schmitt, head of business development.
