Most of the testing world is managed around artifacts: test cases, test documents, bug reports. If you look at any “test management” tool, you’ll see that the artifact-based approach permeates it. “Test” for many people is a noun.
For me test is a verb. Testing is something that I do, not so much something that I create. Testing is the act of exploration of an unknown territory. It is casting questions, like Molotov cocktails, into the darkness, where they splatter and burst into bright revealing fire.
How to Manage Such a Process?
My brother Jon and I created a way to control highly exploratory testing 10 years ago, called session-based test management (SBTM). I recently returned from an intense testing project in Israel, where I used SBTM. But I also experimented with a new idea: thread-based test management (TTM).
Like many of my new ideas, it’s not really new. It’s the christening (with words) and sharpening (with analysis) of something many of us already do. The idea is this: organize management around threads of activity rather than test sessions or artifacts.
Thread-based testing is a generalized form of session-based testing, in that sessions are a form of thread, but a thread is not necessarily a session. In SBTM, you test in uninterrupted blocks of time that each have a charter. A charter is a mission for that session; a light sort of commitment, or contract. A thread, on the other hand, may be interrupted, it may go on and on indefinitely, and does not imply a commitment. Session-based testing can be seen as an extension of thread-based testing for especially high accountability and more orderly situations.
I define a thread as a set of one or more activities intended to solve a problem or achieve an objective. You could think of a thread as a very small project within a project.
Why Thread-Based Test Management?
Because it can work under even the most chaotic and difficult conditions. The only formalism required for TBTM is a list of threads. I use this form of test management when I am dropped into a project with as little a day or two to get it done.
What Does Thread-Based Test Management Looks Like?
It’s simple. Thread-based test management looks like a todo list, except that we organize the todo items into an outline that matches the structure of the testing process. Here’s a mocked-up example:
Test Facilities
- Power meter calibration method
- Backup test jig validation
- Create standard test images
Test Strategy
- Accuracy Testing
- Sampling strategy
- Preliminary-testing
- Log file analysis program
- Transaction Flow Testing
- Essential Performance Testing
- Safety Testing
- warnings and errors FRS review
- tool for forcing errors
- Compliance Testing
- Test Protocol V1.0 doc.
Test Management
- Change protocol definition
- Build protocol definition
- Test cycle protocol definition
- Bug reporting protocol definition
- Bug triage
- Fix verifications
This outline describes the high level threads that comprise the test project. I typically use a mind-mapping program like MindManager to organize and present them.
So, you should be thinking, “Is that it? Todo lists?” right about now. Well, no. That’s not it. But that’s one face of it.
What Else Does Thread-Based Test Management Look Like?
It looks like testers gathered around a todo list, talking about what they are going to work on that afternoon. Then they split up and go to work. Several times day they might come together like that. If the team is not co-located, then this meeting is done over instant messaging, email, or perhaps through a wiki.
Is That All it Looks Like?
Well, there is also the status report. Whether written or spoken, the thread-based test management version of a status report lists the threads, who is working on the threads, and the outlook for each thread. It typically also includes an issues list.
Other documentation may be produced, of course. TBTM doesn’t tell you what documents to create. It simply tells you that threads are the organizing principle we use for managing the project.
Where Do Threads Come From?
Threads are first spawned from our model of the testing problem. The Satisfice Heuristic Test Strategy Model is an example of such a model. By working through those lists, we get an idea of the kinds of testing we might want to do: those are the first of the threads. After that, threads might be created in many ways, including splitting off of existing threads as we gain a deeper understanding of what testing needs to be done. Of course, in an Agile environment, each user story kicks off a new testing thread.
Which Threads Do We Work On?
Think priority and progress. We might frequently drop threads, switch threads, and pick them up again. In general, we work on the highest priority threads, but we also work on lower priority threads many times, when we see the possibility for quick and inexpensive progress. If I’m trying to finish a sanity check on the new build, I might interrupt that to discuss the status of a particular known bug if the developer happens to wander by.
Major ongoing threads often become attached to specific people. For instance “client testing” or “performance testing” often become full-time jobs. Testing itself, after all, can be thought of as a thread so challenging to do well, and so different from programming, that most companies have seen fit to hire dedicated testers.
How Do Threads End?
A thread ends either in a cut or knot. Cutting a thread means to cancel that task. A knot, however, is a milestone; an achievement of some kind. This is exactly the meaning of the phrase “tying up the loose ends” and marks either the end of the thread (or group of threads) or a good place to drop it for a while.
How Do We Estimate Work?
In thread-based test management, there is no special provision or method for estimating work, except that this is done on a thread-by-thread basis. Session-based test management may be overlaid onto TBTM in order to estimate work in terms of sessions.
How Do We Evaluate Progress?
In thread-based test management, there is no special provision or method for evaluating progress, either, except that this is done on a thread-by-thread basis, and status reports may be provided frequently, perhaps at the end of each day. Session-based test management is also helpful for that.
So What?
This form of management is actually quite common. But, to my knowledge, no one has yet named and codified it. Without a convenient way to talk about it, we have a hard time explaining and justifying it. Then when the “process improvement” freaks come along, they act like there’s no management happening at all. This form of management has been “illegible” up to now (meaning that it’s there but no one notices it) and my brother and I are going to push to make it fully legible and respectable in the testing arena.
From now on, when asked about my approach to test management, I can say “I practice Rapid Testing methodology, which I track in either a thread-based or session-based manner, depending on the stage of the project, and in a risk-based manner at all times.”
How is TBTM Any Different From Using a TO-DO List?
Michel Kraaij questions the substance of TBTM by wondering how it’s different from the age-old idea of a “to-do” list? See his post here.
This is a good question. Yes, TBTM is different than just using a to-do list, but even so, I don’t think I’ve ever read an article about to-do list based test management (TDBTM?). Most textbooks focus on artifacts, not the activity of testing. Thread-based test management is trying to capture the essence of managing with to-do lists, plus some other things in addition to that.
The main additional element, beyond just making a to-do list, is that a traditional to-do list contains items that can be “done”, whereas many threads might not ever be “done.” They might be cut (abandoned) or knotted (temporarily parked at some level of completion). Some threads maybe tied up with a bow and “done” like a normal task, but not the main ones that I’m thinking of. As I practice testing, for instance, I’m rarely “done” with test strategy. I tinker with the test strategy all the way through the project. That’s why it makes sense to call it a thread.
Once again: Thread-based management is not focused on getting things “done.” In this way it is different from KanBan, Scrum, ToDo lists, Session-based test management, etc., all of which are big into workflow management of definite units of work.
Another thing to recognize is that the main concern of TBTM is how to know what to put on your thread list. The answer to that invokes the entire framework of Rapid Software testing. So, yeah, it’s more than having an outline of threads, which does look very much like a to-do list– it’s the activity (and skills) of making the list and managing it. If you want to talk about to-do list based test management, then you would have to invent that lore as well. You couldn’t just say “make a to-do list” and claim to have communicated the methodology.
[You can find Jonathan’s take on TBTM here.] [I credit Sagi Krupetski, the test lead on my recent project, for helping me get this idea. His clockwork status reporting and regular morning question “Where are we on the project and what do you think you need to work on today?” caused me to see the thread structure of our project clearly for the first time. He’s back on the market now (Chicago area), in case you need a great tester or test manager.]
This is actually a great way of thinking about it. In my experiments with session-based testing, many people asked, “Well when does a charter end?” I didn’t know what to tell them. The answer could mean it’s an ongoing line of testing.
This also validates my thinking that the normal open/closed/delivered/etc. model of normal bug reporting systems is a horrible way to manage testing charters.
[James’ Reply: A session is like a bead on the thread. You might have many sessions on that thread.]
You have such a unique way of describing simple things to strengthen their credibility, well done.
I often model products in terms of their elements and quality criteria based on your Heuristic Test Strategy Model to come up with a high-level list of testing ideas. Collectively, these testing ideas form my test strategy and become the basis of one or more test sessions. Up until now I hadn’t thought of modelling all project related activities (threads) in that manner, thanks!
I can see TBTM and SBTM becoming complimentary approaches where you use TBTM to outline all high-level project activities and SBTM to actually do the work.
One of the things I like best about SBTM is that my sessions leave finger-prints of where I’ve been, and more importantly, what I tested in those sessions. If distracted for periods of time I can quickly review my sessions to determine where I left off last and what I was doing. If sessions weren’t used in TBTM due to time constraints, the fact threads have the potential to run for long periods of time, and the work is very dynamic in that you can drop threads, switch threads, and pick them up again – how would suggest testers evaluate and keep track of what testing they’ve done for a given thread? Notes scribbled in their pads? Comments in a Status Report? This is where SBTM becomes very powerful for me.
[James’ Reply: I would recommend using SBTM where possible, and thread-based when it’s just not feasible to stop the interruptions. For instance, on my recent project, we did two sessions on the first day of testing, but then we realized that straight-ahead testing had to be suspended, because what was more needed was for certain analysis and documentation tasks to be done first. Meanwhile, developers were coming in and out, asking questions, and I kept popping back to the product to do micro-sessions of 15 minutes or so, to answer those questions.
We found that doing a status report at the end of each day, and going over the mindmap of threads at the beginning of each day, helped us feel on track and allowed us to roughly gauge progress.]
I really like this idea. I’ve been lamenting of late, particularly at home at how many times my current ‘thread’ gets interrupted and focusing more on the annoyance then how things end up being knotted, weaved, and sewn together. I will try this as an exercise tomorrow, to look at them, not as interruptions, but new threads, or old threads being picked back up. This line of thinking, I think could totally change my perspective, and attitude toward such interruptions. We’ll see how it goes though.
I discussed thread based testing with Jon yesterday and one thing that was always true for me is that threads are a firm part of SBTM. They capture all the stuff outside of sessions (and some that transcends one session). I don’t think of it as being either thread or session.
Threads are much like a ToDo list thing. I can feel it in my bones that it will be a whole lot more too. I just can’t yet put my finger on it. What I’d like to see is someone beating me to the punch in coming up with simple methods and documentation ideas around threads that make it similarly easy as SBT is today.
I am writing a test strategy (or some call it test plan…so a high level testing document) for quite a large project atm. SBT already has a firm place in it and I will use the concept of a thread. The remaining question is if I will go as far as Jon suggested and llist ALL threads or just those that really pertain to testing and project.
So, keen to see where this goes. I’ll be “on” it for the next 3 weeks. By that time I will have to have nailed down how my project will test. I’ll keep an eye out for more info and post stuff if I find something interesting.
Thanks heaps to Jon and James for frying their neurons over this for the greater good!
Oliver
Excellent post ! It’s surprised me again your ability to turn some obvious testing process to new Test method.
Based on my hands on experience in this project, I’ll add my two cents.
You should add preparation phase for TTM method.
Preparation phase should contain:
1. Documentation Tread
Detailed system setup/preparation document, for example:
– Custom Simulators (include calibration process)
– Off the shelf equipment (verify calibration process)
– System setup
2. Test cycles
In current project i planned, as following:
– Sanity or Debugging cycle
– Test protocol design & Test protocol executing
– Formal Validation test cycle (at least one)
May be tomorrow i can add more.
[James’ Reply: Okay, folks, Sagi was the test lead on a recent project where I was test architect. It was his status reporting each day and review of priorities each morning that got me thinking about threads.
I didn’t intend to write out the real thread list from that project, man. I was just giving people a flavor.]
Hi James,
Thankyou.
I have been attempting to introduce something I feel is quite familiar that encompasses every task I would be expected to perform as part of a normal project. In my attempts to explain the meaning of my work to my manager, I identified this as similar to agile (in effect, a test sprint that runs as a project within the project), with milestones related to significant project dates – broken down in a very similar way to how you describe what TBTM looks like. This also includes status reports and a burndown chart of work. Thankyou again for explaining this in a way I can present and work towards much more easily.
James, very interesting post, thanks for sharing it.
I see SBTM and your thread example from two sides, one is from the tester perspective looking outside (how do I report to the test/project manager what I did) and the other is inside (how can I organise my testing so that I get the best coverage given the time and context of the project).
The thread approach now helps with the former with the question “Why does it take so long”? I think the thread approach helps the tester to organise their thoughts around the threads leading to better presentation of what has actually been done, not just the tiny part that people see as “testing”. This might help with PMs that are not that experienced with testing.
For the internal facing view of threads I can see that it helps me as a tester to be more organised in my thinking and play out the different approaches and heuristics that I can see useful to gain the best coverage in the context of the project.
Where I see problems is with the lack of estimating and, to a lesser degree, evaluation of progress. I’d argue that the thread approach is suited for projects not only with lots of interruptions but also with extreme time pressures where no in-depth testing can be carried out. I finished a project like that last week where I thought I’d use SBTM but ended up with a more thread based approach (even though I wasn’t thinking of it in this term) . It was both possible and useful to use the same session sheet for notes as for SBTM, however it was proportionally bigger.
[James’ Reply: There is no lack of estimating or evaluation in TBTM. All I was saying is that I don’t yet see how threads influence that process one way or another. You estimate and evaluate in whatever way you see fit, it’s just done thread-by-thread.
For session-based management, on the other hand, there is a special approach we use for estimation and evaluation.]
Where I’d disagree is that a thread doesn’t imply a commitment.
Say you have a web application where you can report a broken streetlight. You can also track your case(s) that you submitted, filter by date and see them on a map and table. I could have three threads, one for submitting, one for tracking, one for viewing. Or even more if you add in multiple browsers/Oss. Each thread could be tested by a single tester.
[James’ Reply: I might not track threads to that level of detail. But it’s up to you.]
What each tester says by picking up the thread is that they’re “committing” to test that thread to the best of their ability in context of the project. They may opt to create individual session out of it if they have the time, or they might not. But the commitment is there either way. A thread could be by functional area as described or by browser.
[James’ Reply: But you can also pick up a thread without committing to it. The mere existence of a thread does not imply that a commitment has been made to pursue it and tie it up.
You’re right that part of TBTM is to decide who will work on which threads, and that probably means commitment. I’m just saying that commitment isn’t built-into the thread itself, whereas in sessions there is commitment.]
In a similar project I opted for the “by browser” approach and changed functional areas multiple times as I learned something in one part that might have an impact on another area – that wouldn’t have easily been possible in SBTM.
From project managers point of view the how or the approach isn’t that important. What they care about is what information the tester can provide, if there are additional risks to the project or if we de-risked a part and what that part is. So the commitment might not be that a particular session is followed but the commitment is that the browser, functional area, performance, etc is investigated and information provided about what has been learned.
I really like the idea about knots and Jon’s description of splicing threads to sub-threads or recognising overarching threads. I can see that as helpful to gain a better understanding of the system when looking out for these possibilities.
Thanks for making me think.
Thomas
PS: Do you want to provide a link to Jon’s blog as well? It compliments this one very well.
[James’ Reply: Good point. I’ll add one.]
The reason I never adopted SBTM in my group was that my group work in an environment where the interruption and task switching (due to always changing priorities) is given. The TBTM quite accurately describes what we’ve been trying to do (with variable success).
Testing for a patch release is good example where TBTM view can be helpfull:
Bug Retest Thread:
– List of Bugs to retest, every bug retested represents mini-thread
Regression Thread:
– Regression tests you normally do plus extra regression based on the specific risks associated with the release are mini-threads here
Release Management Thread:
– If the group (like ours) is testing / coordinating installation packages and reviewing the release documentation lots of threads will happen here
Karol
Great post!
As a novice RST student I really liked the session idea as a way to log what you do when you ‘travel’ around.
Reading this I think it would enhances the ‘Long Leash’ heuristic. A distraction becomes a new thread, waiting to be “executed”, showing as a note in your session log.
[James’ Reply: Good point.]
I kind of live this way, and have struggled towards a formulation of it but never quite succeeded. If a taxonomy or process or whatever-you-want-to-call-it is laid out, it will help me, personally, a whole lot!
Some problems I struggle with, or have struggled with, are:
– How to explain to someone that there really is an overall thought process behind what might otherwise seem a little haphazard. (I don’t manage the testing process very tightly, half because I’m lazy and half because I really believe that if I try to manage it more tightly than I do, I will damage it somehow.)
[James’ Reply: By weaving your threads into a fabric. Making sense of your threads, in other words. A key part of thread-based test management is having a coherent mental model that explains what threads are needed and how they relate to each other. That’s why I have my Heuristic Test Strategy model. Threads must be woven into a compelling story of the product and the testing.]
– How to explain to an Agile organization that there is testing that can and should be done, that is nevertheless outside the scope of the dev-centric sprint heartbeat. (This is often entangled in the discussion of “what does ‘done’ really mean?” in that you try to do some testing in the sprint in order to allow people to call it “done”, but then everyone knows there’s more testing to be done after that.)
[James’ Reply: You must explain that there is no particular reason why testing effort should mirror development effort. If you decide to switch to a new set of libraries for doing charts, that transition may take a few minutes or a few hours to carry out. But testing may require several days or more. Just because the programmer says “I’m done” doesn’t mean we know the implications of that work.]
– How to manage an overall plan, in a way that doesn’t bury me in management-level artifacts (charts and tables and outlines and more charts and spreadsheets and aaaargh), but that gives me the ability to push back a little when Today’s (or This Week’s) Big Red Alert pops up and tries to steal people and resources that I’m using on a longer-term effort towards a longer-term payoff.
[James’ Reply: I prefer to use a one-page test strategy, in mind-map form.]
–JMike
James / Jon
I really loved the idea. Moreover for me this post has come at a very appropriate time as I am involved in an agile project.
I have been doing things in bits and pieces around similar approach as my day today activity using either an xls or MPP.
At a high level these have a well detailed WBS with all micro level activities nested at each level. The dependencies and pre-requisites are clearly called out. This helps me manage my day today tasks more efficiently and leaves me well planned.
In this post, specially liked the clarity that I got from section “How Do Threads End?”
It makes it so objective.
Regards
Meeta
I thought I should clarify my previous comment. In a traditional/scripted approach of testing, it is possible to create testcases and then wait for a pass/fail result (I realize the problems with that approach and am not advocating that). You can also have reviews/inspections of the testcases. Reviews provide an avenue for a manager/lead to provide input.
[James’ Reply: They aren’t reviewing the actual testing, just the descriptions of the testing. Management generally doesn’t perform such reviews, however. Or if they do, they generally do a poor job of it.]
Note that a similar approach can/was/is also used for development (at least in a traditional model).
Using SBTM I assume that leads/managers can be much more involved on an on-going basis in the actual testing strategy. Is that essential? If the testers are experienced, do the leads/managers get involved on an on-going basis?
Thanks,
– Nilanjan
[James’ Reply: That’s up to them. But the SBTM method that I created called for debriefings after every session.]
Hi James,
After reading the article, I could not differentiate btw the Thread based test management and Scrum activities.It sounds to me more like another name for scrum. Am I missing important details..?Plz let me know.
[James’ Reply: Really? How much do you know about Scrum? Describe Scrum for me, please.]
Thank you! I am usually fairly good at naming and describing, but I have never been able to articulate this, a process that I, and other testers I know, intuitively use. I need to think about my own processes more, definitely—sadly, I’ve minimal control at my current workplace to affect the actions of others—and that bit of recognition signals a very good place to start.
I am implementing a TBTM tool in github, would you like to give us some advice? Thanks.
https://github.com/aaronchen2k/ngtesting-platform
[James’s Reply: Okay. Feel free to email me or skype me (I am “satisfice”) to remind me about this.]