By Sarah Fittje
Why should you test your software products manually when test automation can do the job for you? Because manual tests can provide you with valuable insights where automated tests fail.
These days, test automation seems to be a hot topic within the QA community. New test frameworks are implemented constantly and praised as the ultimate solution to software testing. But test automation mostly only verifies that features work as expected while assuming that they are also being used as expected. If you rely exclusively on automated tests, certain bugs may fall through the cracks and you will also learn nothing about the quality of your UI design in the process.
Test automation is by no means inferior to manual testing or vice versa. Both have unique characteristics which are suitable for different cases of application. In fact, both test strategies complement one another.
Test Automation
Whenever tests consist of repetitive tasks and have to be executed numerous times, they are good candidates for test automation. The regression test is a classic example. These kinds of tests may expose crucial bugs and should definitely be applied prior to production deployment. Since setting up automated tests requires some initial effort, one should assess beforehand if it’s worthwhile to implement a dedicated test case automation. Frameworks enable you to execute test scripts with ease and offer useful test functions, but you’ll still have to write the test script yourself. If automated tests are advisable in order to ensure proper functionality of your product, you should start setting up test automation early on. Otherwise you might lose track of the big picture due to the amount of functionality already implemented. This will make it a lot more difficult to get started.
Test automation accelerates testing cycles and ensures constant and more consistent testing. But you shouldn’t blindly trust automated tests since test scripts only execute what they’re “told” to. The quality of automated tests still depends on the tester. And there are quite a number of things the tester might do wrong. Anything your test cases and corresponding scripts don’t cover might still end up containing bugs. Moreover, automated tests are not very helpful in uncovering flaws in your UI; they cannot check for usability or intuitive design.
Manual Testing
Test automation helps to ensure a high level of quality for your product’s functionality, but it can’t replace manual testing completely. Not uncommonly, the expected user behavior implemented in automation scripts can differ from reality. Users need to learn how to use your application on their own. Within this learning process, they might fail to complete steps, enter values you didn’t expect, or they even might not know what to do at all in certain situations. Users know far less about your product than you do, but they are the ones who will be using it in the end. For these reasons users are the most valuable indicator of the quality of your UI. At Project A, we recently established “The User Test Lab” with the purpose of realizing in-house user tests. We’ll dedicate a separate blog post to this topic soon.
But irrespectively of user tests you should consider manual testing strategies such as exploratory testing or usability tests whenever human behavior and intuition are critical factors.
Notably, exploratory testing tends to reveal rather unusual bugs no one even considered before. In exploratory testing, one draws on experience and intuition without applying strict test techniques and rules. Besides thoroughly documenting what you do in order to stay on top of things, exploratory testing is relatively open. Sure, it is partly based on specific use cases and user journeys, but it doesn’t dictate the whole test procedure. While testing a feature, the tester might creatively come up with ideas on how to continue testing. This might emulate rather unusual user journeys and thereby uncover critical bugs that haven’t been revealed by automated tests; the same goes for certain edge cases.
Manual approaches to software testing are closer to the experience a real user would have and thereby help to understand how users interact with certain features. By unveiling unusual bugs and supporting the evaluation of your product’s UX, manual tests cover exactly those parts within the QA process which are neglected by automated tests. The downside of manual testing strategies is that they’re quite time-consuming and require all of the tester’s attention.
Combined Testing
When taking into account the different strengths of test automation and manual testing respectively, it’s easy to see that both strategies complement one another. Automated tests don’t make manual testing obsolete, but they can take care of repetitive tasks, which would occupy lots of valuable working hours. In addition, they are way more accurate in the long run. Despite the fact that manual tests are quite time-consuming, they’re vital in order to ensure the highest quality for your product’s usability. By using both strategies in conjunction, it’s far more probable that your product will meet high standards with regard to functionality and UX.
To give you an example of a QA process which combines both approaches, I want to briefly outline the procedure we applied when we were testing new features for natue, an online shop for wellness and health products. Right after the successful deployment of new features with Jenkins on the test instance we triggered automated regression tests in CasperJS. While the automated tests were running, we simultaneously performed manual tests on those features while keeping an eye on the intermediate results of the regression test. If the automated or manual tests revealed any bugs, we fixed them and deployed anew, which in turn triggered another iteration of the regression test. If all tests were successful on the test instance, we created a release tag for the live deployment and basically applied the same testing procedure to the staging instance. In case these tests were successful too, we were ready for the live deployment.
By applying both automated and manual testing strategies, we were able to achieve a high level of quality in terms of functionality and usability of new features. Only when both approaches to software testing are utilized side by side can a high functional coverage be guaranteed. In addition, you might already detect potential flaws in your UI prior to performing any user tests. Always consider combining automated and manual tests to get the most out of your testing cycles.