I just read this on another blog:
“Regression testing is usually seen as the poorer cousin of ‘proper’ domain-abstracted assertion-based testing. Often rightly so!”
In twenty years of doing testing, managing testing, attending many many conferences and reading many papers and books on testing, I have not heard of “domain-abstracted assertion-based testing”.
I don’t know what assertion-based testing is. So if it’s usually associated with regression testing, then that must be true of a community other than the ones I’ve encountered.
I don’t know what it means to be domain-abstracted. Again, perhaps some community of testers obscure to me uses that term.
Moreover, I find that when I imagine what those things might be– and I thought I was good at imagining things– the ideas I come up with don’t seem anything like regression testing.
I often exhort testing students that it’s okay to make up your own terminology. I am not in favor of a standardized way of speaking except within specific communities, companies, or projects. But this case reminds me to add this caveat: Remember friends, when you invent words and terms, it sure helps if you align them with some aspect of some corpus of existing vocabulary, so that the rest of us have a hope of figuring out what you mean.
Regression testing is a very widely used term in testing. I’m aware of no testing community that does not use that particular term. I googled “domain-abstracted testing” and found zero hits. However I googled “assertion-based testing” and found about 500 hits, some of them interesting.
I would have thought “assertion-based testing” meant embedding ASSERT calls in code. In other words, built-in tests. That would help detect regression-related bugs, but also bugs that aren’t regression-related, so I wouldn’t call it form of regression testing per se. As I google the term, it seems that it is used in various ways.
Some terms used commonly in testing I like to call “danger words”, by which I mean that using them without defining them will confuse people in most contexts that occur to me. It looks like “assertion” is an example of a danger word.
Still, I don’t really mind all this. I’d rather learn from how people baffle me with their words than attempt to stop all innovation and free thinking by legislating a lexicon.
I’m really glad you wrote this post, I read it too, and “domain-abstracted assertion-based testingâ€? had me doubting my level of knowledge. I thought I had missed something somewhere, but if you’ve never heard of it either, I feel reassured.
Steven
Hi James,
I had a different reaction than to set about trawling the net, trying to find who else might see regression testing as some kind of poor relation to “proper’ domain-abstracted assertion-based testing”, and what that actually meant.
When I read this phrase, I immediately thought it had an all too familiar ring to it – it sounded like bullshit.
Not necessarily the practice (whatever that is – as you discuss above), but the phrase itself.
The phrase sounds like it might have been cynically formed by deliberately throwing together software/test “power” words to sound “smart”. I’m sure it hasn’t, but my immediate reaction was that it was.
Every profession has forms of “restricted” language or code that exclude “outsiders”, but I think there is point where “over using” this vocabulary can “turn off” the very people you are trying to impress…good testing and a “suspicion of bullshit” don’t sit well together in my experience.
I, too, saw a set of buzzwords flanking an attempt at urbane misdirection (the passive voice generalization “…is usually seen…”). Then the second punch of “Often rightly so!”
Tailor made for putting down ignorant but well meaning people. “Oh, of course, I wouldn’t want to be unpopular…” Substitute other buzzwords and this copy reads like something from a teen girl magazine.
It’s an old human habit. Declare a consensus, using invented words, allow an out for the confused (“Whatever you thought before, it’s OK, just as long as you wise up from here on”). I just recently read Jurgen by James Branch Cabell. The protagonist frequently bulldozes adversaries by obliquely mentioning the works of nonexistent but august sounding scholars, most of whom agree with him; he includes an occasional fictitious one that disagrees, but they are always trumped by some other equally bogus but more eminent “source”. Jurgen is, by his own repeated assessment, “a monstrous clever fellow.” The desired response in the listener is “Oh, well, of course I’ve read {nonexistent source xyz}…” because not losing face is more important than stopping Jurgen from getting whatever he wants.
This is something like that.
“Assertion-based testing” (I too was guessing here) sounded to me like “find an assertion (in the documentation or from a stakeholder) and use it to frame one or more tests”. Any one of your students ought to be able to parse the limitations on that one, if I’m right. But I can make that fit into regression testing as generally voiced. Note, of course, that “regression testing” itself can mean at least three different things, though most people I’ve worked with seem to mean only “did this old bug we think we’ve fixed recur?”
[Note that I just used “of course” the way the person we’re commenting on used other slippery expressions of the “it goes without saying” type. In my case, it’s because I know you and I have had that conversation. Sorry, everyone else! 🙂 ]
“Domain-abstracted” is a real poser. If I abstract the domain (assuming I know what that is), am I not left with “assertions” that are nearly meaning-free?
“Stuff will behave in some way.” “Stuff won’t notwork.” “Somebody will like the outcome of this series of tests.”
Oh, wait, those are assertions, aren’t they? So perhaps domain-abstraction is… something… …else? Or not?
Pretty thin gruel. Does the author of the piece ever get around to defining those terms in situ?
[James’ Reply: I have skimmed the article he refers to. He seems to be another one of those Agile guys who conflates test automation with testing. Assertion testing in his mind seems to be testing using Assert statements. I didn’t find an explanation “domain-abstracted”, but I suspect it’s another way of saying that the tests are focused on the code, rather than the purpose for which the product was created.]
Perhaps the author is trying to say that regression tests are for people that don’t go back and enhance the test that should have found this so it would have never been a bug. In my experience if I had to find the most bugs in the shortest time I would run the regression tests. I think the poorer cousin is a harder worker.
I do like the idea of rolling regression tests back into the regular tests for organization reasons but it never seems to happen in practice. Keeping them separate is good if in fact they are more powerful and you can run them after your smoke tests to find bugs sooner.
Looking back on that post it could have been computer generated blabber.
Having just been to an Agile and agile conference [CITCON Sydney], I can safely say that the developers there used much more common terms like TDD, test code, developer tests (all in the context of automation), and no mentions of design-abstracted!. The original blog is on a site dedicated to TDD, which helps set the context (al be it implicitly). The testing reflections copy does not reflect this…. However, the original blog should have used keywords to set the context. [Anthony, can you set default keywords against some of the blogs from other sites like this one?)
The interesting thing is developers reinventing standard functional test automation. The main thrust of his post is creating checkpoint statements, recording the result during capture, then comparing during replay. We tried to fast forward this with one session at CITCON explaining what a lot of the pay-lots-of-money-for tools do, trying to motivate some developers to create open source equivalents. Some tools are getting very good, Selenium for example for web automation (but still at the GUI not the API). We also had a session on fragile tests, and it was also good to hear developers talking about including the negative tests like those that check boundaries values! way to go guys….!
From my notes taken at the Self Education for Testers Tutorial at CAST… I would have to say, “[I don’t know anything] .. by that name”, and “Please tell me what you mean by that”.
I was also interested by the use of ‘proper’. Sometimes it helps me understand better if someone explains how not to do it rather than how to do it. So maybe if there was some explanation about ‘improper’ domain-abstracted assertion-based testing? Or ‘things to avoid’ when doing domain-abstracted assertion-based testing.
Having now skimmed the article (which I probably should have done first….I see there are a number of ‘improper’ type things discussed. Unfortunately, I think lots of it went over my head. Will have to read again s l o w l y.
[James’ Reply: Hi Louise. One heuristic I often use is that if an article is difficult to read, perhaps it’s not worth my time to read it at all. It’s not so difficult to write plainly, even about complex subjects. It’s not necessarily your problem as the reader.]
I think the first clue is that the examples are in Ruby. When people talk about assertion-based testing in Ruby, they are usually talking about Behavior-Driven Development. (BDD) BDD is a “real” thing, invented by a “real” person, David Allstells. You can google it, or watch a video about it here:
http://video.google.com/videoplay?docid=8135690990081075324&q=behavior+driven+development&total=34&start=0&num=10&so=0&type=search&plindex=0
BDD is a developer-facing test technique. Software Testers who focus on the actual behavior of the software will be realy confused by it, because it uses terms we are used to in ways that we are not.
The basic idea behind BDD is that calling everything “test” effects our way of thinking or our cognitive process. It “limits” us to thinking of everything in a sort of black-box way. By changing the unit-test framework like jUnit, where every method has a corresponding Test_Methodname to an assertion framework, we can say:
Assert_thismethodDoesX and thus change our focus from testing to behavior.
If it sounds like I didn’t say much there, you’re right. The biggest innovation from these frameworks is probably changing the nomenclature from “test” to “assert”. Of course, we had the ok() function in perl ten years ago and assertions in C twenty years ago, but, whatever.
As to the domain-abstracted stuff, I have no idea what the guy is talking about, and this is my area of expertise.
We run into the same problem (developer-terminology v. tester terminology) all the time. For example, there is a developer-facing test technique called “Interaction Based Testing” that actually focuses on the behavior of objects in isolation, not the interaction of the objects. It’s kinda wierd.
I’m going to speak about this kind of stuff at GTAC next week. Wish me luck! 🙂
–heusser
[James’ Reply: Thanks Matt. That helps.]Â
Assertions are an integral part of JUnit, NUnit and other xUnit frameworks- examples are: assertTrue(someArg), assertFalse(someArg), assertEquals(baselineArg, someArg). These frameworks can also be used for some higher level component gui testing. Your mileage using them for thes purposes may vary. I have used assertions in Ruby-Watir. Note I am not a Ruby Watir power user and have to use a shrink wrapped gui tool for part of testing on my current test gig. hth
Mike Tierney
Points well taken! My reaction was:
1) Domain-Abstracted Assertion-Based Testing. Wow! I wonder what the heck that is??? I guess I should know, but I don’t. Must be some really important concept that is somehow “proper” in someone’s thinking. But I bet that thought-driven Exploratory regression testing will end up beating it hands-down!!!
2) Wonder what this Domain-Abstracted stuff really is?????
3) Oh – it’s a linguistic smokescreen!!! Got me!!! I BELIEVED IT!!!!!
4) Wow – this dude was really trying to fool people. Shame on him!!!
5) Shame on ME for buying it
6) But, as James pointed out… there are valuable Lessons to be Learned in pondering how people fool us with “Danger Words”. Something like thinking about magic tricks and pondering how we were fooled once we see how the trick was done. More Mind Expansion!!!!
7) So in the final analysis, folks who use strange and complicated terminology to fool us are helping us to learn, but perhaps not in exactly the ways they planned 🙂
I read the blog where the phrase was coined. I am doubtful of ever understanding what is meant by a compound phrase like “domain-abstracted assertion-based testing” when it’s coined by a person who uses the term “regression testing” to mean testing “against some previously approved copy.” I thought “regression testing” meant to look for newly introduced (or uncovered) errors, with an emphasis on re-testing a large system after changes are made. I can see where a “previously approved copy” might be extrapolated from the more common use of “regression testing,” but it’s misleading. Nevertheless, I agree with you, I’d rather struggle through ambiguously coined phrases than legislate a lexicon.