Invisible gorillas in the mist.
September 9, 2014 3:16 PM Subscribe
A different kind of standardized testing. Why the debate over ISO 29119, the proposed international standard for software testing, might matter to you.
The International Standards Organization has begun releasing its multipart standard for software testing. The actual standard is behind a paywall ($300); you can read abstracts here.
The need for thorough software testing, and organizational accountability for that testing, is illustrated regularly by high-profile debacles like healthcare.gov’s go-live last year. ISO purports to be delivering an “internationally agreed-upon” standard that will help companies solve their seemingly intractable bug problems.
Problem is, the standard has not been widely agreed upon. Many in the testing community fear, with good cause, that 29119 will essentially make a tester’s job into a documentation project. Instead of exploring and thinking deeply about a product and its intended (and unintended) users, the documentation checklist will have to be completed as first priority - and critical bugs will get missed.
As anyone who’s taken the “Invisible Gorilla” test knows, selective attention bias can cause people to miss seemingly obvious information. Scripted test plans, which lead the tester down a very narrow path and do not encourage independent exploration, create ideal conditions for selective attention bias to flourish. This very type of scripted testing is a likely result of ISO 29119.
A good collection of links, some pro, many con, is here.
Some of the most eloquent cases against the standard have made by:
This software tester hopes that the debate will pique the interest of those beyond the testing community who believe that software testing is not just a rubber stamp, and that integrity and intellectual commitment are critical to software quality.
The International Standards Organization has begun releasing its multipart standard for software testing. The actual standard is behind a paywall ($300); you can read abstracts here.
The need for thorough software testing, and organizational accountability for that testing, is illustrated regularly by high-profile debacles like healthcare.gov’s go-live last year. ISO purports to be delivering an “internationally agreed-upon” standard that will help companies solve their seemingly intractable bug problems.
Problem is, the standard has not been widely agreed upon. Many in the testing community fear, with good cause, that 29119 will essentially make a tester’s job into a documentation project. Instead of exploring and thinking deeply about a product and its intended (and unintended) users, the documentation checklist will have to be completed as first priority - and critical bugs will get missed.
As anyone who’s taken the “Invisible Gorilla” test knows, selective attention bias can cause people to miss seemingly obvious information. Scripted test plans, which lead the tester down a very narrow path and do not encourage independent exploration, create ideal conditions for selective attention bias to flourish. This very type of scripted testing is a likely result of ISO 29119.
A good collection of links, some pro, many con, is here.
Some of the most eloquent cases against the standard have made by:
- Michael Bolton
- James Marcus Bach
- Dr. Cem Kaner
- Ben Simo, on Twitter as @QualityFrog. Mr. Simo made headlines last year when he brought urgent-severity security and performance defects in healthcare.gov to public attention.
This software tester hopes that the debate will pique the interest of those beyond the testing community who believe that software testing is not just a rubber stamp, and that integrity and intellectual commitment are critical to software quality.
Some further context (heh), most of the opposition seems to be from the context driven testing school of thought.
And for those that aren't familiar the "cases against" are written by very high-profile folks in the testing world ('household names' if you will).
posted by el io at 3:39 PM on September 9, 2014
And for those that aren't familiar the "cases against" are written by very high-profile folks in the testing world ('household names' if you will).
posted by el io at 3:39 PM on September 9, 2014
I'm actually looking to become a trainee tester soon!
posted by ACair at 3:50 PM on September 9, 2014
posted by ACair at 3:50 PM on September 9, 2014
fukken laffo at anybody protesting this -- like you've got certified levels of programming skill in your shop in the first place.
posted by boo_radley at 3:53 PM on September 9, 2014
posted by boo_radley at 3:53 PM on September 9, 2014
This is a really weird debate even to be having.
First, it's an ISO standard. Some really, really big companies – the kind who use IBM for their IT services because it's the only vendor approved on their list – will adopt it and insist on its use. Most of the software world will ignore it, like they ignore a lot of such standards.
Second, the people objecting to it seem to be saying that documentation is inherently wasteful. Fuck that noise forever. As a professional who works in software development, I frequently have to read shitty undocumented code and evaluate systems that no one ever tested beyond seeing if clicking the button did something. Increasing the amount of documentation available is not useless, it's absolutely critical for much of the software out there.
Third, it seems that it's mostly protested against by the context-driven testing people, which while it sounds reasonable basically makes software testing an impenetrable "art" when it needs to be more rigorous and methodologically sound. I agree with this article objecting to the anti-standard petition:
posted by graymouser at 4:11 PM on September 9, 2014 [4 favorites]
First, it's an ISO standard. Some really, really big companies – the kind who use IBM for their IT services because it's the only vendor approved on their list – will adopt it and insist on its use. Most of the software world will ignore it, like they ignore a lot of such standards.
Second, the people objecting to it seem to be saying that documentation is inherently wasteful. Fuck that noise forever. As a professional who works in software development, I frequently have to read shitty undocumented code and evaluate systems that no one ever tested beyond seeing if clicking the button did something. Increasing the amount of documentation available is not useless, it's absolutely critical for much of the software out there.
Third, it seems that it's mostly protested against by the context-driven testing people, which while it sounds reasonable basically makes software testing an impenetrable "art" when it needs to be more rigorous and methodologically sound. I agree with this article objecting to the anti-standard petition:
The real reason the book burners want to suppress it is that they don’t want there to be any standards at all. Effective, generic, documented systematic testing processes and methods impact their ability to depict testing as a mystic art and themselves as its gurus.The idea – expressed in the OP and in many of the protests – that testing can't be done objectively is obscurantist. Unfortunately, like most of the ideas in the world of software, despite some pretension to science there is no objective proof. It's less like even a "soft" science and more like the business world, driven by personalities and anecdotes instead of any kind of rigor.
posted by graymouser at 4:11 PM on September 9, 2014 [4 favorites]
Some really, really big companies – the kind who use IBM for their IT services because it's the only vendor approved on their list – will adopt it and insist on its use.
For example, governments requiring it's use, and government vendors, and the military-industrial-complex? Unless the effectiveness/wisdom of this standard can be demonstrated, this standard (if adopted by the above parties) will add to the cost and time of projects without any evidence that it will produce better software. I say that as the burden of proof is on those proposing the standard - they have beta tested the standard in organizations and then measured its effectiveness, right? right?
Second, the people objecting to it seem to be saying that documentation is inherently wasteful.
I would strongly imagine that most of the people objecting spend more time documenting their work that the vast majority of developers. There is a good chance they spend most of their time as a tester doing non-testing activities (documentation, meetings, process). Probably more so than developers (in my experience, anyways).
I actually haven't seen that claim anywhere (that documentation is inherently wasteful). I think the concern is that instead of creating reams of documentation that might be useful (test plans, test cases, results of test cases, communication to the development team, etc, etc) that they will be creating documentation for the sake of compliance rather than being driven by an actual business (or engineering) purpose.
Unfortunately, like most of the ideas in the world of software, despite some pretension to science there is no objective proof.
Actually, I think you summed up the position of the people against this standard quite succinctly; they posit there is no evidence that this is a good idea (and indeed runs counter to modern software testing methodologies).
graymouser: what ISO development standards to you use when developing code?
posted by el io at 4:23 PM on September 9, 2014 [4 favorites]
For example, governments requiring it's use, and government vendors, and the military-industrial-complex? Unless the effectiveness/wisdom of this standard can be demonstrated, this standard (if adopted by the above parties) will add to the cost and time of projects without any evidence that it will produce better software. I say that as the burden of proof is on those proposing the standard - they have beta tested the standard in organizations and then measured its effectiveness, right? right?
Second, the people objecting to it seem to be saying that documentation is inherently wasteful.
I would strongly imagine that most of the people objecting spend more time documenting their work that the vast majority of developers. There is a good chance they spend most of their time as a tester doing non-testing activities (documentation, meetings, process). Probably more so than developers (in my experience, anyways).
I actually haven't seen that claim anywhere (that documentation is inherently wasteful). I think the concern is that instead of creating reams of documentation that might be useful (test plans, test cases, results of test cases, communication to the development team, etc, etc) that they will be creating documentation for the sake of compliance rather than being driven by an actual business (or engineering) purpose.
Unfortunately, like most of the ideas in the world of software, despite some pretension to science there is no objective proof.
Actually, I think you summed up the position of the people against this standard quite succinctly; they posit there is no evidence that this is a good idea (and indeed runs counter to modern software testing methodologies).
graymouser: what ISO development standards to you use when developing code?
posted by el io at 4:23 PM on September 9, 2014 [4 favorites]
ISO C/C++ are two very important standards (if there isn't an ISO standard for the language you write everyday as a programmer, there probably is for one of the top three implementation languages of that language). ISO 8859 used to be (character encodings). ISO/IEC/IEEE 60559 (floating-point arithmetic). There are some good, relevant standards out of ISO, and some less-good, less-relevant ones. I wouldn't know which the latter are.
posted by jepler at 4:29 PM on September 9, 2014
posted by jepler at 4:29 PM on September 9, 2014
If it's really focusing on testers and QA people writing documentation, that does seem like crap to me. From the link above the fold:
If anything, a standard should be specifying the software developers writing documentation and specifying other particulars of the pre-integration-testing development process. There's lots of documentation and other stuff it would be fine to mandate for testing/QA personnel, it just ought to all be subsidiary to and derive directly from things that are happening further back in the development process.
posted by XMLicious at 4:44 PM on September 9, 2014 [1 favorite]
I can't see many of the development teams I work with dropping everything to achieve ISO 29119 compliance, especially when it totally missed the idea of combining testing practices with development practices in an agile context.When I was testing (albeit in a startup, early on in my career) it was an enormous pain in the ass to figure out how things were supposed to work and what behavior should be considered acceptable, because the developers and project managers wouldn't bother to think through or talk through amongst themselves how things were supposed to work in any but the highest-level detail: the developers would just sit down and start slinging out code. Once, while pressing for details like this, I actually got told "The code is the documentation!"
If anything, a standard should be specifying the software developers writing documentation and specifying other particulars of the pre-integration-testing development process. There's lots of documentation and other stuff it would be fine to mandate for testing/QA personnel, it just ought to all be subsidiary to and derive directly from things that are happening further back in the development process.
posted by XMLicious at 4:44 PM on September 9, 2014 [1 favorite]
what ISO development standards to you use when developing code?
Is that supposed to be a gotcha? While it doesn't fit into it precisely, our development lifecycle is based on ISO/IEC 12207, and it was helpful to have an actual standard to design that from. Our QA group does a lot of what is described for 29119, and again a standard would be useful.
posted by graymouser at 4:45 PM on September 9, 2014
Is that supposed to be a gotcha? While it doesn't fit into it precisely, our development lifecycle is based on ISO/IEC 12207, and it was helpful to have an actual standard to design that from. Our QA group does a lot of what is described for 29119, and again a standard would be useful.
posted by graymouser at 4:45 PM on September 9, 2014
My favorite ISO standard is ISO/IEC 5218. It says to code 0 for not known, 1 for male, 2 for female, and 9 for not applicable. If you want to get the ISO standard for this, it will cost you about $95. Charging for standards is complete BS. Standard like this should be freely available on the Internet.
Any ISO standard on software testing will be obsolete in ten years as the industry and technology changes, even if it was okay now.
posted by Xoc at 4:46 PM on September 9, 2014 [6 favorites]
Any ISO standard on software testing will be obsolete in ten years as the industry and technology changes, even if it was okay now.
posted by Xoc at 4:46 PM on September 9, 2014 [6 favorites]
graymouser: Not exactly as a gotcha. But I was curious if you as a deveveloper that seemed to be in favor of ISO testing standards already engaged in ISO processes as a developer.
I read a bunch of those links looking to find supporters of the ISO standard, and didn't really find much there. I thought that the piece you quoted was engaging more in name-calling than an actual critique of the position of those opposed to the ISO standard ("book-burners", "themselves as its gurus").
posted by el io at 5:11 PM on September 9, 2014 [2 favorites]
I read a bunch of those links looking to find supporters of the ISO standard, and didn't really find much there. I thought that the piece you quoted was engaging more in name-calling than an actual critique of the position of those opposed to the ISO standard ("book-burners", "themselves as its gurus").
posted by el io at 5:11 PM on September 9, 2014 [2 favorites]
hey i support anything that will embarrass microsoft.
posted by quonsar II: smock fishpants and the temple of foon at 5:29 PM on September 9, 2014
posted by quonsar II: smock fishpants and the temple of foon at 5:29 PM on September 9, 2014
I heard what was possibly an urban legend that at Google they decided to apply machine learning to their interview process. They ran the resumes of anyone they had ever interviewed through a program that attempted to find a correlation between keywords and the people they had hired or rejected.
As the story tells, there was no keyword that had a positive correlation with being hired.
But there was one keyword that had a negative correlation: "Certified".
posted by A dead Quaker at 5:31 PM on September 9, 2014 [1 favorite]
As the story tells, there was no keyword that had a positive correlation with being hired.
But there was one keyword that had a negative correlation: "Certified".
posted by A dead Quaker at 5:31 PM on September 9, 2014 [1 favorite]
Second, the people objecting to it seem to be saying that documentation is inherently wasteful. Fuck that noise forever. As a professional who works in software development, I frequently have to read shitty undocumented code and evaluate systems that no one ever tested beyond seeing if clicking the button did something. Increasing the amount of documentation available is not useless, it's absolutely critical for much of the software out there.
I wouldn't say that documentation is inherently wasteful, but it's definitely not inherently useful. The same teams that write shitty undocumented code will write shitty, incorrect, out-of-date documentation if you give them a pile of documents to fill out. In fact, a lot of teams that write great code will write bad documentation, or will write great documentation before any of the code is written and never update it to reflect reality afterwards. At the end of the day, the number of people who care that the code works will always outnumber the people who care that the documentation is correct or complete, so the documentation will always tend to suck.
posted by burnmp3s at 6:22 PM on September 9, 2014 [1 favorite]
I wouldn't say that documentation is inherently wasteful, but it's definitely not inherently useful. The same teams that write shitty undocumented code will write shitty, incorrect, out-of-date documentation if you give them a pile of documents to fill out. In fact, a lot of teams that write great code will write bad documentation, or will write great documentation before any of the code is written and never update it to reflect reality afterwards. At the end of the day, the number of people who care that the code works will always outnumber the people who care that the documentation is correct or complete, so the documentation will always tend to suck.
posted by burnmp3s at 6:22 PM on September 9, 2014 [1 favorite]
I worked for a big pharma company programming manufacturing process control systems. I did a lot of testing, all of which was rigidly governed by FDA GMP and several layers of corporate CSQ. It didn't matter if the test plan was designed from the start to find as few problems as possible. As long as everything was properly documented, we were in compliance with the law. Having correct paperwork to satisfy an FDA inspector was much more important to them than finding and correcting bugs.
posted by double block and bleed at 6:24 PM on September 9, 2014 [1 favorite]
posted by double block and bleed at 6:24 PM on September 9, 2014 [1 favorite]
quonsar II: smock fishpants and the temple of foon: "hey i support anything that will embarrass microsoft."
"and that's why I run Windows 8.0."
posted by boo_radley at 6:34 PM on September 9, 2014 [1 favorite]
"and that's why I run Windows 8.0."
posted by boo_radley at 6:34 PM on September 9, 2014 [1 favorite]
Michael Bolton
He would be against a testing standard - he's always forgetting where to put the decimal point.
posted by Pogo_Fuzzybutt at 7:33 PM on September 9, 2014 [2 favorites]
He would be against a testing standard - he's always forgetting where to put the decimal point.
posted by Pogo_Fuzzybutt at 7:33 PM on September 9, 2014 [2 favorites]
I'm still waiting for my Six Sigma Green Belt to arrive in the mail.
posted by RobotVoodooPower at 9:33 PM on September 9, 2014 [1 favorite]
posted by RobotVoodooPower at 9:33 PM on September 9, 2014 [1 favorite]
Hey, this is relevant to my interests!
Been a tester since 1998 and what worries me about this, as it worries me everytime the ISO organisation moves from encoding standards for things (unicode frex) to standards for processes or people is that they'll try to fit a messy reality into a one size fits all approach.
The whole emphasis on software testing does not bode well in this regard, because there's no such thing. There's functional testing, user acceptance testing, accessibility testing, security testing, black and white box testing, usability, system test, unit test, undsoweiter. What makes sense in one context does not necessarily make sense in another...
And yes, ISO does have the reek of waterfall on it, so I understand the worry that it'll make less sense in agile methods, though lord knows agile's often used for sloppy practises and could use some standards.
Also, the idea that better testing could've prevented all those all too familiar stories of wased, bloated government IT implementations (don't think it's better in business; there it can just be hidden better): No. Just No.
Testing isn't a panacea for bad design and bad goals.
In short, it looks like the worst excesses of TMAP and we already have TMAP.
posted by MartinWisse at 10:43 PM on September 9, 2014 [5 favorites]
Been a tester since 1998 and what worries me about this, as it worries me everytime the ISO organisation moves from encoding standards for things (unicode frex) to standards for processes or people is that they'll try to fit a messy reality into a one size fits all approach.
The whole emphasis on software testing does not bode well in this regard, because there's no such thing. There's functional testing, user acceptance testing, accessibility testing, security testing, black and white box testing, usability, system test, unit test, undsoweiter. What makes sense in one context does not necessarily make sense in another...
And yes, ISO does have the reek of waterfall on it, so I understand the worry that it'll make less sense in agile methods, though lord knows agile's often used for sloppy practises and could use some standards.
Also, the idea that better testing could've prevented all those all too familiar stories of wased, bloated government IT implementations (don't think it's better in business; there it can just be hidden better): No. Just No.
Testing isn't a panacea for bad design and bad goals.
In short, it looks like the worst excesses of TMAP and we already have TMAP.
posted by MartinWisse at 10:43 PM on September 9, 2014 [5 favorites]
The real reason the book burners want to suppress it is that they don’t want there to be any standards at all. Effective, generic, documented systematic testing processes and methods impact their ability to depict testing as a mystic art and themselves as its gurus.This is of course not an argument; this is at best an assertion and a dubious one at that.
posted by MartinWisse at 10:43 PM on September 9, 2014 [1 favorite]
Testing, is but one small part of the Quality Assurance role, and not even the primary function in that role. Risk assessment is what we are truly there for.
This ISO standard seems to miss that.
Ideally, senior QA would be in the room with the product manager, taking the stakeholder business requirements and transforming them into test cases. Once the code passes those test cases it is complete and ready to ship. The test cases then, ARE the documentation - for developers, for users, for support, everyone.
Any other sort of documentation is just a bunch lies waiting to happen, since changes to the code are documented by dint of delegation and "you touched it last" types of behaviors, tending to be an unwinnable game of catchup.
Standardizing testing seems to be an attempt to further polish the turd of ill-thought development practices.
posted by drfu at 11:13 PM on September 9, 2014 [2 favorites]
This ISO standard seems to miss that.
Ideally, senior QA would be in the room with the product manager, taking the stakeholder business requirements and transforming them into test cases. Once the code passes those test cases it is complete and ready to ship. The test cases then, ARE the documentation - for developers, for users, for support, everyone.
Any other sort of documentation is just a bunch lies waiting to happen, since changes to the code are documented by dint of delegation and "you touched it last" types of behaviors, tending to be an unwinnable game of catchup.
Standardizing testing seems to be an attempt to further polish the turd of ill-thought development practices.
posted by drfu at 11:13 PM on September 9, 2014 [2 favorites]
Also, the idea that better testing could've prevented all those all too familiar stories of wased, bloated government IT implementations (don't think it's better in business; there it can just be hidden better): No. Just No.
Testing isn't a panacea for bad design and bad goals.
This is a very important point. If your project is a horrible mess, and you find out during testing, you are doomed already. Relying on testing alone to ensure the quality of your project is like driving with a blindfold on and expecting to figure out where the other cars are by gently bumping into them with your fenders.
posted by Dr Dracator at 12:59 AM on September 10, 2014 [1 favorite]
Testing isn't a panacea for bad design and bad goals.
This is a very important point. If your project is a horrible mess, and you find out during testing, you are doomed already. Relying on testing alone to ensure the quality of your project is like driving with a blindfold on and expecting to figure out where the other cars are by gently bumping into them with your fenders.
posted by Dr Dracator at 12:59 AM on September 10, 2014 [1 favorite]
MartinWisse: "Also, the idea that better testing could've prevented all those all too familiar stories of wased, bloated government IT implementations (don't think it's better in business; there it can just be hidden better): No. Just No."
This. When I was reading Ben Simo's account of the bugs he found in healthcare.gov last year, it looked to me like there had been just about complete nonfeasance in terms of technical (i.e. design/development) leadership. You don't solve that enormous systemic problem by testing the bugs out of it. You can't, because there will just be too many bugs to count.
Apparently the state of Oregon is up in arms enough about Oracle's "delivery" of their healthcare system platform that they've sued them. The linked article contains a link to the complaint itself, which is well worth reading. (I'm not sure if parallel federal and Massachusetts lawsuits have been filed. Sadly, I doubt it.)
posted by Sheydem-tants at 3:33 AM on September 10, 2014 [1 favorite]
This. When I was reading Ben Simo's account of the bugs he found in healthcare.gov last year, it looked to me like there had been just about complete nonfeasance in terms of technical (i.e. design/development) leadership. You don't solve that enormous systemic problem by testing the bugs out of it. You can't, because there will just be too many bugs to count.
Apparently the state of Oregon is up in arms enough about Oracle's "delivery" of their healthcare system platform that they've sued them. The linked article contains a link to the complaint itself, which is well worth reading. (I'm not sure if parallel federal and Massachusetts lawsuits have been filed. Sadly, I doubt it.)
posted by Sheydem-tants at 3:33 AM on September 10, 2014 [1 favorite]
Third, it seems that it's mostly protested against by the context-driven testing people, which while it sounds reasonable basically makes software testing an impenetrable "art" when it needs to be more rigorous and methodologically sound.
I think that's a very unkind reading of the context-based testing movement, given that they have some of the best, most methodological resources available for testers who are operating in contemporary contexts. To be frank, a huge number of classic testing resources that came into being before context-based testing have their roots in the 1970s and low-level programming languages. Many people testing today are testing products built on frameworks that were written in high-level languages, and even though some of the techniques are the same (how to define test cases, how to find boundaries and 'seams' in the code, how to question assumptions), the focus of testing is going to be necessarily different.
When you add web into the mix, it gets worse. Hell, one of the 'standard' QA books for before context-driven testing recently added a chapter about mobile development. Amazing! Wonderful! Except it presumes that you must, of course, be talking about native apps and not web apps so suddenly it becomes an issue of integration with this ever-growing pile of hardware combinations, instead of treating the browser as an abstraction layer that separates you from the phone's/computer's guts (which is how, necessarily, web QA has to be done, while checking for platform-based browser eccentricities).
tl;dr context-driven testing actually gave me the tools to do my job. The non-CDT resources, not so much.
posted by flibbertigibbet at 4:09 AM on September 10, 2014 [1 favorite]
I think that's a very unkind reading of the context-based testing movement, given that they have some of the best, most methodological resources available for testers who are operating in contemporary contexts. To be frank, a huge number of classic testing resources that came into being before context-based testing have their roots in the 1970s and low-level programming languages. Many people testing today are testing products built on frameworks that were written in high-level languages, and even though some of the techniques are the same (how to define test cases, how to find boundaries and 'seams' in the code, how to question assumptions), the focus of testing is going to be necessarily different.
When you add web into the mix, it gets worse. Hell, one of the 'standard' QA books for before context-driven testing recently added a chapter about mobile development. Amazing! Wonderful! Except it presumes that you must, of course, be talking about native apps and not web apps so suddenly it becomes an issue of integration with this ever-growing pile of hardware combinations, instead of treating the browser as an abstraction layer that separates you from the phone's/computer's guts (which is how, necessarily, web QA has to be done, while checking for platform-based browser eccentricities).
tl;dr context-driven testing actually gave me the tools to do my job. The non-CDT resources, not so much.
posted by flibbertigibbet at 4:09 AM on September 10, 2014 [1 favorite]
I think that's a very unkind reading of the context-based testing movement
Indeed. It's one of the few movements that I approve of wholeheartedly, because they understand that testing is a cost, and your testing aegis needs to fit what the product is and how it is developed. Flight Control software is *not* a web app, and you have to test them in very different ways. For a web app, you can't be slow. For Flight Control, you cannot afford a lack of rigor, it's a literally life-or-death situation.
Unlike most movements, Context Based Testing pretty much sings my song. Best Practices are meaningless unless the context matches. There is no one true answer, there is simply a large and growing toolbox, and you need smart people who can select the right tools *for that particular job*. If you don't know the environment in which a product is being created, then you cannot develop a good testing regimen for that product, except by sheer luck.
I think the simple fix to ISO 29119 is simply to define contexts in which it would be valid. I have little doubt, actually, that it would be a very good thing -- in the right context. I also have little doubt that it would be a fucking nightmare in the wrong context.
posted by eriko at 9:09 AM on September 10, 2014 [4 favorites]
Indeed. It's one of the few movements that I approve of wholeheartedly, because they understand that testing is a cost, and your testing aegis needs to fit what the product is and how it is developed. Flight Control software is *not* a web app, and you have to test them in very different ways. For a web app, you can't be slow. For Flight Control, you cannot afford a lack of rigor, it's a literally life-or-death situation.
Unlike most movements, Context Based Testing pretty much sings my song. Best Practices are meaningless unless the context matches. There is no one true answer, there is simply a large and growing toolbox, and you need smart people who can select the right tools *for that particular job*. If you don't know the environment in which a product is being created, then you cannot develop a good testing regimen for that product, except by sheer luck.
I think the simple fix to ISO 29119 is simply to define contexts in which it would be valid. I have little doubt, actually, that it would be a very good thing -- in the right context. I also have little doubt that it would be a fucking nightmare in the wrong context.
posted by eriko at 9:09 AM on September 10, 2014 [4 favorites]
« Older A Modern Pandora's Box | It seems this genet is making a habit of riding... Newer »
This thread has been archived and is closed to new comments
I took a look at the abstract, and it seems like there are some good techniques in the tool box, but absolutely no help in choosing or applying them well.
posted by miyabo at 3:33 PM on September 9, 2014