A person named Shiva contacted me on Skype back in May. Then he didn’t say anything for several months, until yesterday we had this exchange.
Shiva:
Hi James!
James:
Hi.
Shiva:
I wanted to set up some time with you to chat about an idea and interest you in it…what would be a good time to chat? Would 1130 AM PST Mon work?
James:
What is it about?
Shiva:
Well, I am on the advisory board of a company in China that does phenomenal work and they are very good in testing and SW dev. wanted to see if you wanted to take advantage of their low rates and high quality and make some money? Especially if you are planning to enter that business?
James:
I run a testing company.
Shiva:
I know.
James:
So… Why would I need a testing company? I already have one!
Shiva:
To improve your margins. Scale.
James:
I don’t see how that would be possible. I do a certain kind of testing that requires a high level of skill. I doubt that any other testing company in the world has that skill. Well, there are a couple, but they are expensive.
Shiva:
That is exactly the point I wanted to walk you through as this company has a phenomenal ability to learn, get resources due to their location and yet do high quality work. It is an idea. I am willing to explore it with you if you are interested. If not, I completely understand.
James:
I would have to see examples of the quality work you say that your company does. Is it posted online? Most companies that say they do quality work don’t do quality work at all. They do terrible work. So, I would need to see what you can do.
Also, I would need to know your training practices. I’d want to see them in writing. If you forward your training materials to me, I could review them.
Shiva:
Sure. Before I go forward, it would be good to have a three-way chat with their president/co-founder, myself and you and I am happy to arrange for samples for you to review – work samples, training practices.
James:
I won’t be interested in talking unless you can show me some basic evidence that we have anything to talk about.
Let me ask you just a few questions:
– Does your test lab document all of its testing in detail? Is every test procedure and action documented?
– Do you have complete expected results documented, too?
– Do you maintain statistics on passed/failed tests? Do you graph them?
– Are your testers ISTQB certified?
Your answers to these questions will allow me to quickly assess your capability. Otherwise, I worry you will waste your time.
Shiva:
James, all these are great questions and I have answers that you will like but I need to think about what you said first. In my opinion, business is not just some Q&A, it is also building cross-company relationships and getting to know each other. I need to think about whether we will be a fit that way at all with each other. Please don’t take this personally but I need to give this some careful thought, or else it may not even work even though we are able to deliver capabilities.
James:
You are right. Business is about relationships. I want a relationship with people who can answer basic questions. These are questions that I routinely answer for my clients.
I’m a testing company, too, right? You know that, right? I know what my clients demand of me, and I’m demanding that of you, too. If what you want is an uncritical client who is easily impressed, you came to the wrong guy.
I’m not taking it personally. I’m taking it as an indication that you are a bit over your head.
Shiva:
There is no need to be rude, James.
James:
I’m not being rude. I’m being honest. It’s not my problem if you can’t handle that.
Shiva:
(Skype indicates Shiva has gone offline.)
NOTICE TO TEST LABORATORIES THAT KEEP CALLING ME:
It is not rude for a potential customer to challenge your corporate capability. That’s normal due diligence. Your job is to speak honestly and forthrightly about what you can and can’t do. Don’t dodge questions.
The reason I’m skeptical is that almost no test lab actually knows what it is doing. And there’s no excuse. Test labs ought to know how to test, but mostly I see labs much better at faking testing than doing it.
Hi James,
Thanks for sharing this. I guess Shiva in question was not offering you a Business opportunity to scale but he was trying to do it for his company because he would have Satisfice or James Bach’s name associated with it (had he been successful, a terrible assumption I just made). Second thing, I am not sure if you are asked these 4 questions when someone approaches Satisfice for Testing Services? Whatever I could understand from reading your views in last 2 years, I think, your answer to all these questions would be “NO”. Am I right?
[James’ Reply: My answers would be “no” followed by a short lecture on testing. I framed them that way as a quick rhetorical trap for someone who assumes that “yes” is the right thing to say to a client.]
I liked the idea of showing evidences of one’s quality(?) work, irrespective of one is a company or a person. It builds credibility and allows people to witness “things” about the work. Quality is what values to someone and people will have different take on quality of the same thing.
Irony is, people who are capable of bringing change don’t read such posts and people who read will take time to bring in change.
Thanks again James.
James, on the software development side of the house I’m reluctant to tie capability to certification. Certification generally proves you can write the test, not that you can be an effective software developer. Do you see ISTQB as something different?
[James’ Reply: As I have said many times in public and private and on this blog, I think the commercial “professional” certifications such as the ISTQB are a fraud and a terrible blight on our industry. I urge everyone to reject them.
Each of the questions I asked Shiva was one I was hoping to hear a strong “No!” answer to.]
Appearing on your blog always has a credibility associated with it… meaning someone should have done something credible to have a mention on your blog. Sadly, Shiva appears in your blog for all the wrong reasons!!!
On the other side, “Shiva” is a Hindu God worshipped by Hindus. When I read the blog title, I really thought that he was the one who was annoyed with your questions :)!!!
Hi James,
I really dont understand why you are very much skeptical regarding ISTQB Certification. If one person will plan to do this certification, then he may learn a lot of things which is relevant and will helpful in applying in testing activities. I dont mean to say by doing ISTQB Certification, one will know each and everything but he will do the things in better way. I am also agree that by doing ISTQB, any individual’s thinking will change a lot but surely he can try to think differently and adapt it.
Let me know your thought and specific point why are you disagree.
[James’ Reply: I have already discussed that on this blog, many times. See my earlier posts.
The ISTQB syllabus is a collection of bad folklore from 20-40 years ago. It represents the equivalent of using a surgical manual from 1795 to train modern doctors. Their material on ET is lame. Their material on boundary testing is embarrassing. Etc., etc.
I and my colleagues strive to update and improve our thinking all the time, while the ISTQB proudly features the muddled ideas of our forefathers. That is not professional. That is a travesty.]
My answers to your questions would be:
– Does your test lab document all of its testing in detail? Is every test procedure and action documented?
At one time (3-4 years ago) I would be foolish enough to say “Yes” (though it was never the case). Now the answer is firm and solid “No”.
[James’ Reply: That’s good to hear, it is not only foolish but impossible to document every action, unless you automatically record yourself on video and with an activity logger. Even then, you won’t be recording all your thoughts. I would ask a follow-up: so, what DO you document?]
– Do you have complete expected results documented, too?
“No”, unless contractually bound to do so – I don’t have much control over that. Thankfully it’s been a while (year at least) since I have to provide anything like that. I can now better explain that this type of activity does not add much value to anyone.
[James’ Reply: Actually, contract or no contract, this is also impossible. The purpose of this question was to probe whether the “excellent” test lab is aware that all oracles are heuristic and incomplete.]
– Do you maintain statistics on passed/failed tests? Do you graph them?
“Sort of”. I have model which tries to answer:
“How far are we in the testing given the scope I would like to cover?”
“How many areas appears to be ok now, given the testing that we did?”
It can be greatly missused, but then I am in control of it and try to be aware of how this model can fool me and my stakeholders (and I try to explain that to them too). I try to update the model during the testing too (scope changes and coverage changes).
This is not the only type of reporting I do. I talk about problems in application under test too.
Any advice on how to better report testing progress?
[James’ Reply: Good answer. I was probing to see if he would create stupid and meaningless and misleading metrics if I asked for them.]
– Are your testers ISTQB certified?
Out of around 15 testers I have, I’ve got one with ISTQB recently (rest of the folks doesn’t have it and I would not advise them to take it). Let’s just say that I will not use that against him.
[James’ Reply: See, that wasn’t so hard. A test lab that gives me answers like this would encourage me to keep talking with them and examine their lab and practices in more detail.]
Just to play along this hypothetical test lab example:
what DO you document?
It really depends on what purpose we create our test documentation. In many cases it serve only as a starting point for internal test review / discussion (so it’s short around 1-2 pages per relatively big test area). In other cases it serves as a starting point for joint testing with client – then it’s little more detailed (but not “testcasified” unless we really have to). For regression testing we have (although still not very good ones) checklists that drives that type of testing. General assumption is that documentation will be used by someone who a) understand something about the applications under test b) knows something about testing.
On – Do you have complete expected results documented, too? I missed the “complete” in “complete expected results” so you got me there.
I pity the company who has employed such a sales person, and their gullible customers (if they have any).
I doubt if this has anything to do with any specific geographical region. Rather, it has got a lot to do with the organization work culture which gets exposed through its inept representatives
Hi James,
Great blog as always. I was recently asked to look into the ISTQB certifications courses to determine if the QAs at our company should be certified. Upon receiving the course overview pack, I was quite skeptical as a lot of material that’s being taught in these courses, I have gained from my work experience. Upon doing more research (such as reading your thoughts and your peers), I recommended to my company that we shouldn’t waste the company’s money and the QAs time in going on these courses.
Keep up on being a driving force in the QA community and as well as a inspiration to all QAs 🙂
James, I have seen the syllabi of ISTQB and decided not to take that certification. And now your opinion about ISTQB makes me feel I am right about ISTQB.
However, I want to know your opinion about CSTE and CMST certifications. Will you consider these certification exams are better? Or you dont consider any certification at all?
Please reply your thoughts.
[James’ reply: The ISTQB is especially bad, but all of the commercial certifications out there, including the one I worked on (the CSQE from ASQ) are worthless, in my opinion, and more than worthless: harmful. I have explained why in other posts.
What you need to is be a vigorous student of software testing. Be so good they can’t ignore you.]
That was mean and sneaky of you to pose those trick questions to trap that poor, defenseless sales drone. You’re a bad man.
(…this is me making a mental note to do the same thing.)
It is interesting to look at the propositions that shiva is trying to make – scale, margin etc. I have been with IT services companies quite a while now and fully understand Shiva’s language. To me, he appears like a ex-Director or VP from an IT services major and now incharge of a startup or a IT services company that wants ride on providing cheap testing services.
Given the community reputation and brand you hold – I am surprised Shiva choose to approach you instead of those 100’s of small to medium testing company heads and consultants who exactly seek the kind of “business relationships” that he is proposing. Why did not he go there? In any conference, it is easy to spot such people – simply talk about things like QA best practices, new automation framework or tool or methodology that promises huge cost savings, ROI, Test factory, Novel costing model for testing services. You will have the flocking around you.
It is still not clear to me what shiva was actually seeking in you. If he is an intellegent sales person, he would have done a real homework to know what kind of person are you and what is personal philisophy about testing. Reading few posts on your blog would be enough to check that out.
Shrini
Shrini wrote:
“Why did not he go there? In any conference, it is easy to spot such people – simply talk about things like QA best practices, new automation framework or tool or methodology that promises huge cost savings, ROI, Test factory, Novel costing model for testing services. You will have the flocking around you.”
If you will allow me to be a complete Judgemental Jerk, I’ll call these people freshmen:
http://www.youtube.com/watch?v=nu5AEw0Yafs
It’s ironic that they generally exist so high on the food chain that they are not exposed to the work itself, thus, they don’t get the opportunity to actually learn what’s really going on on the shop floor.
Nice post. After having communicated with James a bit, the questions made me smile, and then ‘…I have answers that you will like…’ – made me laugh
From a guy who works at a test lab (onshore), I get these partnering requests all the time and they end quickly.
They insist that the “cost savings” are too great for us to ignore, which prompts me to give quiz them about notions of value-over-cost including: “Would you rent the cheapest apartment in your city, no matter the condition or the neighborhood?”
As for the 4 questions, each of my answers is below:
– Does your test lab document all of its testing in detail? Is every test procedure and action documented?
What do you mean by “document”, “testing”, and “detail”? What is a “test procedure” to you? Definitions are important to me. We may not come to consensus on what a word means, but we should declare some assumptions. But let’s back up… could you tell me why you’re asking? What’s your concern about documentation?
– Do you have complete expected results documented, too?
Maybe, but we need to discuss “completeness”. That means many things to many people. Also, what kind (or detail) of documentation would solve what kind of problems for you? Can you give me examples or are you looking to me for recommendations?
– Do you maintain statistics on passed/failed tests? Do you graph them?
What’s the spirit of your question? Are you looking for trends, to know when to stop testing, to know when you might be ready to ship, to look at bug clustering, or some kind of risk assessment? Is there a standard you need to prove that you followed? Is being able to stand up to scrutiny important to you? If so, we need to talk about what constitutes a “test” and the criteria for how they are deemed a pass / fail. If I perceive the results of the experiments we do as “expected”, is there an easy way to know in the moment of assessment that that’s what you and your team expect? Do you have ideas on oracles or how we can know what our values are? I have some. As for graphing, are you concerned about reporting to people who do not understand testing? Can you send me an example of a good test results graph?
– Are your testers ISTQB certified?
None, but why do you ask? Is there something about that brand of certification that’s important to you? What if I said yes? If you knew our testers could recite everything the syllabus levels on demand, how would that give you confidence in our testing? What if we could recite them, but never took the exam? What if they were ISTQB certified at one time but aren’t anymore? What if they were “certified” by another context-driven or open method? What artifact could you see from them to give you confidence in their skill?
[James’ Reply: Good answers!]
“It is not rude for a potential customer to challenge your corporate capability.”
James,
Isn’t it at least possible to BOTH be rude and challenge someone’s corporate capability? The way I’ve always understood rudeness or politeness is that it has more to do with the manner of communication than with the specific topic discussed.
Jim
[James’ Reply: Of course. But he was only calling me rude BECAUSE I was questioning him.]
Hi James,
I have had test labs ringing me too and I always give them short shrift. I thought I would have a go at answering your questions:
– Does your test lab document all of its testing in detail? Is every test procedure and action documented?
It depends on the particular project. If the objective of testing is to prove compliance to a particular standard or set of requirements documentation would be more rigourous than on a project where that was not the case. 100% documentation would not be agreed to though. I agree with the stakeholders what the documentation requirements are in advance so that enough documentation is produced to enable business decisions to be made.
– Do you have complete expected results documented, too?
Not necessarily. It depends on how tightly scripted the tests are. Usually maximum benefit can be gained by allowing testers freedom of thought and as such results are couched in terms of “is there a problem with this?” I would never agree to document 100% of expected results as the results set would be too large to be of any use.
– Do you maintain statistics on passed/failed tests? Do you graph them?
No. This has been found to be relatively meaningless. I believe it is better to understand the story behind the tests and the test cycle that has just been undertaken. Graphs and ‘raw numbers’ can easily hide crucial information that can inform a decision.
– Are your testers ISTQB certified?
Yes. However the certificate only shows an ability to answer questions on a particular way of working at a particular point in time. This should not therefore be taken as evidence of competence (or incompetence). As evidence of my competence I prefer to submit my blog postings, recommendations on LinkedIn from peers and twitter postings. As you will see I engage in the wider testing community and seek to extend my education keeping up to date with the latest thinking in the field. This goes far beyond that advocated by ISTQB or ISEB.
Having had a go myself, I fail to see what is so difficult about answering these questions, since the very act of thinking about the answer is a learning exercise in itself.
Regards,
Stephen
[James’ Reply: Great job, Stephen!]
To Stephen…
Having had a go myself, I fail to see what is so difficult about answering these questions, since the very act of thinking about the answer is a learning exercise in itself.
For some people, that’s exactly what’s difficult about answering these questions.
—Michael B.
Jon,
Great questions. I, myself, often feel that way when filling out documents required by many larger companies who are prospects/yet to be clients. Often it is seems to be enough to mention exploratory testing and risk based testing as methodologies (along with some static analysis for the codebase), however it really doesn’t convey anything just name dropping ‘schools’/methodologies. It really makes me wonder about WHAT they want to know, and why. (besides having ammunition in case that there will be ‘politics’, most likely only the case if things go awry).
Do I document my tests – well.. I do have MindMaps that I may update during testing and that I write in the period before I take in a test task/user story – but, few people would really benefit from these, as they often only describe ideas and things to take into consideration.
1)I will ask some hindu god questions sent me the answers on my e-mail. 1)Hindu god shiva power is what? 2)Hindu god shiva create all things with one power what is that power name? 3)Desires should be done hindu god will use what power what is that power name?
Thanks James! I share the opinon with you about the ISTQB.
Have a look on the new syllabus. It is really a shit. A lot of mistakes.
On one hand we have the ISTQB syllabi on the other hand we have the people involved like Rex Black.
He is the typical person knowing everything better and in my opinion able to sell his mother for a couple of dollar.
Does anyone know that he just working more in Asia than in America or Europe. The Americans and Europeans got tired on him.
Since SQE kick him out, he is suffering. 😉
Rex is the bad face of the ISTQB. He was/ist involved in developing the syllabi, he develops teh material for his traning and probably also the exam questions. The ISTQB and Rex is all about money!
We should open an istqbleaks.org side to get the things done….
Folks,
Lets come back to the original topic 😉
Well what you can interpret if a sales guy talking with professional testing person and business man? Can you?????
Note: Let’s not diverse the initial topic. After reading i could easily see that ppl mostly are diverting to somewhere where we are actually not require to discuss 🙂 😉 🙂 -:)
Hello,
James thanks for your questions (not only in this theme).
– Are your testers ISTQB certified?
It is a Perfect question. Really nice. I studied to theoretical physics before start the IT life. And the story below was the most important from our great teachers (I think better to provide the full story better, I’m sorry for big post). The testing is a science. If knowledge has a bounds – this is ordinary and low effective.
“Some time ago I received a call from a colleague. He was about to give a student a zero for his answer to a physics question, while the student claimed a perfect score. The instructor and the student agreed to an impartial arbiter, and I was selected.
I read the examination question: “Show how it is possible to determine the height of a tall building with the aid of a barometer.” The student had answered: “Take the barometer to the top of the building, attach a long rope to it, lower it to the street, and then bring it up, measuring the length of the rope. The length of the rope is the height of the building.”
The student really had a strong case for full credit since he had really answered the question completely and correctly! On the other hand, if full credit were given, it could well contribute to a high grade in his physics course and certify competence in physics, but the answer did not confirm this.
I suggested that the student have another try. I gave the student six minutes to answer the question with the warning that the answer should show some knowledge of physics. At the end of five minutes, he hadn’t written anything. I asked if he wished to give up, but he said he had many answers to this problem; he was just thinking of the best one. I excused myself for interrupting him and asked him to please go on.
In the next minute, he dashed off his answer, which read: “Take the barometer to the top of the building and lean over the edge of the roof. Drop the barometer, timing its fall with a stopwatch. Then, using the formula x=0.5*a*t^2, calculate the height of the building.” At this point, I asked my colleague if he would give up. He conceded, and gave the student almost full credit.
While leaving my colleague’s office, I recalled that the student had said that he had other answers to the problem, so I asked him what they were.
“Well,” said the student, “there are many ways of getting the height of a tall building with the aid of a barometer.
For example, you could take the barometer out on a sunny day and measure the height of the barometer, the length of its shadow, and the length of the shadow of the building, and by the use of simple proportion, determine the height of the building.”
“Fine,” I said, “and others?”
“Yes,” said the student, “there is a very basic measurement method you will like. In this method, you take the barometer and begin to walk up the stairs. As you climb the stairs, you mark off the length of the barometer along the wall. You then count the number of marks, and this will give you the height of the building in barometer units.” “A very direct method.”
“Of course. If you want a more sophisticated method, you can tie the barometer to the end of a string, swing it as a pendulum, and determine the value of g [gravity] at the street level and at the top of the building. From the difference between the two values of g, the height of the building, in principle, can be calculated.”
“On this same tack, you could take the barometer to the top of the building, attach a long rope to it, lower it to just above the street, and then swing it as a pendulum. You could then calculate the height of the building by the period of the precession”.
“Finally,” he concluded, “there are many other ways of solving the problem. Probably the best,” he said, “is to take the barometer to the basement and knock on the superintendent’s door. When the superintendent answers, you speak to him as follows: ‘Mr. Superintendent, here is a fine barometer. If you will tell me the height of the building, I will give you this barometer.”
At this point, I asked the student if he really did not know the conventional answer to this question. He admitted that he did, but said that he was fed up with high school and college instructors trying to teach him how to think.
The name of the student was Niels Bohr.” (1885-1962) Danish Physicist; Nobel Prize 1922; best known for proposing the first ‘model’ of the atom with protons & neutrons, and various energy state of the surrounding electrons — the familiar icon of the small nucleus circled by three elliptical orbits … but more significantly, an innovator in Quantum Theory. ”
http://www.cooperativeindividualism.org/bohr_storyontests.html
[James’ Reply: I could use the barometer to note the atmospheric pressure while Googling for the answer that Niels Bohr gave.]