Muhammad Saad wrote an interesting scenario on LinkedIn. I will represent it here in italics and comment on it…
Imagine yourself on the first day at your job as a tester. You are presented with an application to be tested. It’s an ERP application containing 100s of forms and thousands of reports. You begin your exploratory testing by opening a form that contains around 50 different fields.
You try to enter random data in this form which took around 20 minutes. Then you press submit. Voilà!! An error message is shown which looks like an unhandled exception. You become very happy. You proudly note down the steps and report the bug in your bug management system. Great effort, you feel really confident and energetic. You continue the testing until the day ends and find some more bugs. “Amazing first day”, you thought.
That would indeed be a good first day. The first days– and weeks– of your job as a tester are filled with learning. And a big part of that learning is working through the product under test. To test well you must have a strong mental model of the product. You can get that in lots of ways, but playful, open exploration with the product, a process I call “survey testing” in RST, probably will dominate over all other activities.
Now comes the next day, the developer has fixed the issue and released a new version of the build. You test the same form with the same steps and you found that the bug is fixed. You mark it fixed. Great effort. You have contributed to the quality of the product by identifying that bug and as this bug is fixed, the quality is improved.
Well, a lot of other things are happening in that day. You may not get around to verifying that bug fix. Maybe someone else will. Maybe it can wait. You have huge amount to learn and test. Still, it is plausible that you would verify a fix on the second day.
The more experience I have as a tester, the less interested I am in “contributing to quality.” My interest lies in excellent surveillance of the product, so that no problem can hide from me. I find quality itself to be a distracting idea. But I get it. A lot of people are happy about making the product better. So far, so good in Muhammad’s story.
Now comes the third day, and a developer has again released a newer version. Now you again have to test that form to make sure that no regression issue is found. Same 20 minutes. Now you feel a little bored.
Same 20 minutes? Bored? No and no. It’s not the same 20 minutes. I test a little differently every time I go through. I expand and change the data. I think about tools that might help me generate better data. I probably am talking to people who know the product better than I do (if this is a mature product then surely I am not the first to test it, and if it is not a mature product– how can it have hundreds of forms and reports, already?) It’s only the third day and I have about 200 more days of heavy learning ahead of me for a product as complex as Muhammad is describing.
I’m not bored. I’m engaged with the problem of finding ever cleverer ways of testing. I’m in the midst of learning the functional, operational, and technical details of this technology. People who get bored after such a short time are probably not people who love testing or know very much about it. Testing is not pressing buttons and checking output. Testing is an ongoing application of judgment as you collect evidence and make meaning from it.
If you constantly repeat the same shallow testing process you started with, then you will get bored pretty soon, I guess. But my testing grows deeper as it matures, until it levels out and then gets strategically shallow again as I learn exactly what and where to make the key observations.
Muhammad is probably simplifying this story to make his point clearer. Perhaps when he says “third day” we could take him to mean “day 20” and give him the benefit of the doubt.
Now imagine 1 month from now on, newer versions are constantly releasing, and on every release, you have to test this lengthy form plus 100 other forms like this, just to make sure that no regression is there.
Now you feel angry. You feel tired. You begin to skip steps. You fill around only 50% of the total fields. Your accuracy is not the same, your energy is not the same and definitely, your steps are not the same.
And one day, the client reports the same bug in the same form. You feel pathetic. You feel unconfident now. You think you are not competent enough. Managers are questioning your ability.
I have news for you; this is the story of 90% of the testers out there who don’t use code/tools in their testing.
Maybe Muhammad is projecting his own annoyance with the test process onto other testers. I think he’s right about some of those testers. Indeed, some testers may be angry and bored and tired. But I object strongly that his point holds for people who love to test, and study testing, and make it their career.
Regression testing is driven by regression risk. In the absence of regression risk, there is no need to retest. Regression testing is therefore a much lighter burden than he says it is. Unless the developers are completely out of control and reckless, regression risk is not going to be very high. Old bugs rarely re-open; stable areas of the product do not erupt in spontaneous unrest with bugs bursting out like rebels in a Roman province. It’s almost always more important to do the primary testing which uncovers bugs that have been there all along.
Also, retesting is not a process of repeating exactly what you did the first time. It’s better to introduce a lot of variation into your testing. This helps you find new bugs as well as those old bugs that you may have missed the first time around.
If you feel unconfident in yourself, that’s normal for a new tester. An experienced and trained tester knows that testing is hard, and keeps their expectations of progress to a reasonable level. Managers will question your ability even if you are brimming with ability, because those managers probably aren’t expert testers. They don’t have a clear idea how testing works. They may have some foggy idea that it should all be automated.
Except testing cannot BE automated. What can be automated are certain output checks. I agree with Muhammad that automation is important in most projects, although I generally find it’s done better by a toolsmith working under the advice of a tester, rather than the tester himself. You must always bear in mind what automation can and cannot do, and how experiential and interactive testing (what a lot of people call “manual” testing, but I think that word is insulting and inaccurate) is able to find bugs that automation never will.
Regression issues are the most painful issues. We are humans. And we cannot do the same thing with the same energy, speed, and accuracy every day. This is what machines do. This is what automation is required for, in order to repeat the same steps with the same speed, accuracy, and energy as they were repeated the first time.
Regression bugs are NOT the most painful bugs. The most painful bugs are the ones that hurt the users the most. Nothing about “regression” says that such a bug is especially severe. I want to find every bug that matters.
Muhammad is repeating a well worn myth, which I first attacked in print some 25 years ago in my article Test Automation Snake Oil. He seems to think that machine replicate what humans do. They don’t. He seems to think that automation is inexpensive to create and maintain. In fact, it’s rather expensive, especially relative to how few bugs it ever finds.
I repeat: I believe automation is important to consider in every project; and for some projects it’s absolutely critical. But it’s also quite a lot of work. It’s very easy to create a lot of low value checks that then have to be maintained as the product changes. Muhammad has painted a picture of a project that is severely understaffed, and has apparently gone a long time without a test team at all. Getting the testing sorted out is going to take time and management will have to be patient.
Automation should never be allowed to become an obsession in itself. I have a lot of personal experience obsessing over the tools I write (I started my career as a developer and I enjoy writing code to help me test). I know that I need people around me that help me maintain perspective.
As a professional tester, I work on test strategy, test documentation, performing tests, developing test data libraries, developing test tools, configuring my test platforms, reporting bugs, advocating for testability, learning the product, understanding the users. I’m doing a lot of interesting and challenging things! Automation is a small part of my problem.
Typo perhaps; did you mean “word” instead of “work”?
:
> (what a lot of people call “manual” testing, but I think that work is insulting and inaccurate)
[James’ Reply: Yes. Fixed. Thanks! Typos are hard for me. I look and I look and miss so many of them. Maybe my eyesite is going.]
Which would be worse for a tester:
Loss of eyesite, or websight?
[James’ Reply: Thank you for this incite.]
It is enough to excite 🙂
Lol 🙂
Do you mean eyesight? 🙂
[James’ Reply: Ugh. I’m not good at websights!]
I’ve often thought of software testing as being like the role of a test pilot. A test pilot does not get into a new aeroplane and run through a set of checks as to whether a switch turns on the landing lights and another engages the autopilot (we’ll discount the pre-flight checklist here, because that’s routine for every flight in every aeroplane, not related to the test pilot role). Rather, the test pilot’s objective is to fly the aeroplane and report back on its performance and the limits of that performance – how high it will fly, and how fast; how low, and how slow. If you remember the film “The Right Stuff”, test pilots speak of this process as ‘pushing the edge of the (flight) envelope’.
So in the example you’ve related, the tester’s job on the subsequent days of the project isn’t to repeat the same regression tests and get the same results; it’s to take the outcomes of the first day’s testing and then to go beyond them, to stretch the application’s performance envelope. And that’s how to find the limits of what the application can do and where it is likely to fail.
[James’ Reply: I once taught a class of Air Force pilots and flight instructors. They were going to help test combat flight planning software. I loved those guys. Smart and independent.]
Thank you James for this post. It is really insightful. I have some questions regarding some points you mentioned.
– You highlighted a contrast between a mature product and the fact that it might have hundreds of forms and reports. ERP system for example are quite complex and would always include such configuration. What is your point there?
[James’ Reply: Any system, ERP or otherwise, that has “hundreds of screens” and forms and business rules, etc. is the product of a lot of development work, no? This work happens over a long period of time. Therefore, if the first time you see a product it is already very complex, then either you are not the first tester to work on it, or for some reason the development process has not taken testing seriously up until now. Either way, it’s not your fault and it’s not your problem. You either share the testing burden with other people who must have already figured out a lot about the test strategy and tools, OR you do what you can do alone and don’t blame yourself for being on an understaffed project.]
– I 100% adhere to the idea that manual testing is not the right word to use, from now on I will use the words experimential and interactive testing.
[James’ Reply: The word was “experiential”, not experimental. All testing is experimental. Experiential testing means attempting to duplicate the kind of experience that a particular kind of user would have. You want to see what they would see and feel what they would feel. The opposite of experiential testing is instrumented testing, which means that there is something between you and the direct experience of the product, such as an automation tool, like Selenium. Instrumentation leads to a very different experience of using the product.]
– You mentioned that the regression should be lead by the risk of regression. How do you think that would be best put to practice when you have releases with multiple updated areas where it’s not an easy task to know where the risk is? In addition, there are some risks that are not easy to analyze or predict, like those that are not related to code change. Working on complex big applications might make it not an easy task to have an overall overview. What do you think about that?
[James’ Reply: Risk analysis is not easy, at all. We can never know for sure that we have identified the risks correctly. I recently did a risk analysis for a “smart lock.” Then I went out and used that very smart lock in my own home. To my surprise I found that the calibration process had major flaws. I hadn’t imagined it would be complicated at all to calibrate the lock, until I personally experienced it.
To analyze risk we apply various heuristics. An obvious one is “what, specifically, was changed? What could it have impacted? Avoid testing things that are probably not impacted.” I published a list of these heuristics that can be downloaded from this website. Along with a few articles about risk-based testing. Do a search on my downloads page. I also teach some of this in my classes.]
– how to gain testing maturity and upgrade own skills? How to go from the execution level to the more strategic tasks?
[James’ Reply: Well, you know… I do teach classes on this subject. But if you don’t want to spend money, try downloading some of the many documents I offer, here. Try visiting the Rapid Software Testing channel on YouTube.
Specific idea: search for “testopsy” on YouTube and watch one of my videos about that. A testopsy is an important learning practice.]
I totally agree on that.
Another tiny misconception from this Muhammad-like people:
The 20 minutes of the first day are not the same as on the second and third.
Once you know the the actions you need less time to execute them.
The first day is about finding out the exact scenario which triggers the bug. I also imply that he spent time thinking about the random data within this 20 minutes and not before. It also includes learning the product in general, how to use the function.
This does not needs to be done for the regression test as he already know the actions and the data to use.
When I repeat the same scenarios from day one I expect to execute them faster (aside bugs found) on the following days.
Because I already know them. No learning and exploration.
Also many people overlook that you need (at least) the first 20 minutes if you want automate actions and checks.
Somehow they believe that automation requires to first have learned to steps you want to automate.
This just as a side note to all the other points you mention.
Sebastian
[James’ Reply: Another thing about this… It seems to me that repetitive work is often pleasant. It can be comforting. To hear programmer’s talk, you’d think that having to type the same letter twice was like a bee sting. I bet we all have little repetitive rituals in our work. I bet that when certain people complain about the dreary nature of “manual” testing, the root cause is probably that they dislike testing– no matter whether it is repetive or different every time.]
I feel like I have been railing against the Regression boogie man for years. They pick up the idea that what they tested before, must still work, then assume that they must test exactly the same way and wonder why they find nothing else. The level of thinking in many organizations to lead to a testing practice like that is a headache. Do regressions happen? Absolutely, but they usually have a cause behind why they happen, or happen with frequency, and that is often tied back to the dev practices.
James you are right to call out these things. Thank you.
An approach that I have found useful is to guide regression testing with a Visual Coverage model, that expresses coverage targets for different areas, and different risks in human readable language.
We include information about the value delivered by our existing automated checks in these coverage models. Benefits and limitations are clear, and using human readable language prompts excellent questions from stakeholders.
I like the mix of repeatability and variability that occurs when we use this approach. Once you get stakeholder buy in, it can work really well
“It’s an ERP application containing 100s of forms and thousands of reports. You begin your exploratory testing by opening a form that contains around 50 different fields.
You try to enter random data in this form which took around 20 minutes. Then you press submit. Voilà!! An error message is shown which looks like an unhandled exception. ”
I have to ask, in the above example is the first day newbie tester choosing a form at random?
Or has someone with an understanding of business processes, given them a list containing a few forms which are a high priority, for that business, to be tested?
Because in most businesses in a situation like this there will be a lot of forms, but some of them are used every day and some are rarely used. Some, if things go wrong, are going to cause a lot of big problems, while others if they fail, are going to be a minor inconvenience for 1 or 2 non client people, at worst.
So for a new tester on their first day to start exploratory testing, seems to me not to be particularly useful, unless they are given some guidance as to what the project priority areas are to test (and why).
[James’ Reply: I think you have a unhelpful idea of exploratory testing. My view is that ALL testing is exploratory (to some degree). Some testing is highly exploratory, and other testing is more scripted, but it’s all at least somewhat exploratory. So when you say the phrase “exploratory testing” it’s very easy to be misunderstood. It sounds like your interpretation of term somehow precludes any guidance from other people. But you CAN test in a strongly exploratory fashion while listening to guidance from anyone. An exploratory process is one you control. It doesn’t mean you close your ears and your mind– it’s the opposite of that.
A new tester, on their first day, should learn, learn, learn about the product. Learning about the product is probably going to be an exploratory process, because each of us learns in our own way. However, there may be some scripted educational process available for a mature product.
I would probably have the tester work with someone who already knows about the product, but I might not. It really depends on how accessible the product is to someone who does not yet have much knowledge of it.
In any case, telling the tester to run a bunch of scripts is almost always a bad way of getting them up to speed as a thinking and enthusiastic tester.]
Otherwise it might be a good training exercise for a newbie (to the org/industry, not necessarily new to testing) but may not be the best use of their time in terms of doing the most important testing work to find high-risk defects.
[James’ Reply: I don’t expect a new tester to be directly useful as a bug hunter on their first day. I’m more thinking how can I get them up to speed and confident.]
Replying to myself, because I cannot reply directly to myself until my comment by myself that I am replying to is approved, lol)
Dont get me wrong, i love exploratory testing and some of my most fun bugs (though not for anyone if they got released!) have been found through exploratory testing and thinking “what if?”
But to do good exploratory testing to me is driven by context, I.e. what are the business processes that are being supported hair and what are the non-typical things that some of the end users might be doing?
And then thinking about the way the actual end users are using the product – which often is not the way that the developer of the product may have envisaged it being used – but is what actually works for the end user. And asking myself what sort of things are they going to be doing here and what sort of things are really going to break this big time? Or conversely what sort of things might not break this big time or even appear to be defects but which when done over, and over, and over, and over, every single day by an end user are going to add just a little bit of time each time they do it.
And it might be something that might be just a few extra keyboard taps, or a couple of extra copy and pastes which might not seem much. But when done hundreds or thousands of times during a day, actually costs a lot of money in terms of inefficiency, not to mention the frustration the end user is going to experience.
Unless I am exploratory testing something fairly basic and simple which has just been released to find a major issues asap, I feel like exploratory testing of large and complex applications needs to be done with some understanding of the business processes that those applications are supporting.
As every experienced tester will know, not all of the bugs we report are going to be fixed or even prioritised to fix later. So while you could pick something random and probably find lots of bugs, the question is, are those bugs actually going to be useful or valuable to the business to fix?
It is that sort of knowledge and understanding I like to be armed with, before I go into exploratory testing on anything bigger than a few pages of core functionality.
[James’ Reply: You are just talking about testing. Remove the word “exploratory” from the paragraphs above and I won’t have much problem with it. You are making a good point about professional testing versus amateur testing. But “exploratory” does not mean “amateurish.”]
I found this article and subsequent comments quite interesting. I primarily work as a software developer in test (SDET) and my focus in on building out automation for those repetitive tasks/tests that we run as part of regression testing. Exploratory testing is still done primarily on new features added to the app. What I’ve observed is that to get a new team member/new tester up to speed, we tend to have them go over existing test cases to familiarize themselves with the app. Another thing I see is that when we have a release, we tend to run the same “regression” tests in addition to the exploratory tests. The idea being that we want to give ourselves confidence that everything still works as expected before release. My experience is that 95% of the times, the regression tests don’t find any new bugs or serious bugs to stop a release. In this case, I would advocate that these of regression tests good candidates for automation so that testers can focus on other areas, especially if all the time folks are just following the steps in the test.
[James’ Reply: Repetitive tasks may be hard to automate or easy to automate. And some repetitive tasks, although repetitive, have substantial experiential or exploratory elements that make it dangerous to automate them. Therefore, the way I talk about this is just to say we apply automation where automation seems to make sense (I call that instrumented testing rather than automated testing). I don’t see much value in generalizing.
Although I suppose there are exceptions, I find that telling anyone to run test cases has little value and does little to get them up to speed. It has little value because the new testers don’t understand the test cases or the product. I much prefer having savvy, experienced testers run formal test procedures, if such procedures exist. I want new testers to use guided play. That gets them up to speed faster because it gets them more energized and builds a sense of agency. By “guided play” I mean I give them challenges and direct them to specific parts of the product.
The other thing is that I don’t even create test cases most of the time, because it seems like an expensive thing to do without much return on that investment. I go right from highly exploratory testing to instrumented testing (automated assay, flagging, or checking) without making “test cases.” I use a playbook-style of documentation (a mindmap or short document that describes the strategy, test data, etc.) instead of a bunch of step-by-step procedures.
Automation is helpful, but I never approach it in a formulaic way. It’s always bespoke and particular to the product and project.]
Thanks James for your insights. I found particularly interesting your point about test cases. In a way I feel it makes some testers “lazy” in the sense that they just follow the steps and not “explore” as much as they could. The other thing with test cases is that in my org, we rely on it a lot with reporting and we use test case management software. I like your idea of using a play-book style documentation and would have like to explore that in my org.
I also recognize that not all tasks/tests can be automated and some are difficult to automate and not worth the investment.