• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Satisfice, Inc.

Software Testing for Serious People

  • Home
  • About
    • Privacy Policy
  • Methodology
    • Exploratory Testing
    • Reasons to Repeat Tests
  • Consulting
    • Ways to Engage Our Services
  • Classes
    • James Bach’s Public Test Consulting
    • Testimonials
    • RST Courses Offered
    • Rapid Software Testing Explored
    • Rapid Software Testing Applied
    • Rapid Software Testing Managed
    • Rapid Software Testing Coached
    • Rapid Software Testing Focused: Risk
    • Rapid Software Testing Focused: Strategy
  • Schedule
  • Blog
  • Contact
  • Resources
    • Downloads
    • Bibliography: Exploratory Process
    • Bibliography: Risk Analysis
    • Bibliography: Coaching
    • Bibliography: Usability
    • Bibliography: My Stuff From IEEE Computer and IEEE Software Magazines
    • Bibliography: The Sociology of Harry Collins

Could the Military Be Waking Up?

Ever since I got into the testing field, almost 20 years ago, it’s been a truism that military software development is moribund. It’s not that they love process, it’s that they love bad process. Documentation? Bad documentation. Who can look upon 2167A or Mil-Std-499 without dismay? I’ll tell you who: paper mills and people paid to arrange the ink on all that paper. It’s just a scam and a travesty.

I was asked, in 1998, to analyze two military software software test plans, each for a different major weapons system. I told them up front that I was neither interested nor qualified to assess the test plans against DoD standards, such as Mil-Std-499. I was told, no problem, assess them against best commercial practice. Interpreting “best commercial practice” as what I would generally recommend doing for a life-critical or otherwise high stakes project in the commercial sector, I created a model for analyzing test plans (now published on my website and also in the appendices of Lessons Learned in Software Testing). I then applied the model to the test plans I was given. What was immediately apparent is that the military test documentation had very little information density. It looked like the 75-page documents had been automatically generated by a set of macros operating on a short bullet list.

I made a bunch of suggestions, including showing how the same information could productively be packaged in 5 pages or so. That way, at least we could wage war on something other than trees. They replied, after two months of silence, that my analysis was not useful to them. Why? Because my ideas about test plan documentation were not consistent with Mil-Std-499. That was the only feedback I received about that work. Way to squander taxpayer money, guys! Hoowaa!

A New Hope

The defense department may be waking up to the problem, at long last. See the NDIA Top Issues Report. Notice that two issues I like to harp about are in the top five challenges: skills and testing. Notice that the military is now officially concerned about wasteful documentation and low-skilled workers.

Maybe not coincidentally, I recently taught my class at Eglin Air Force base, with F-15s thundering regularly overhead. I was surprised that they would invite me to teach there. I was a bit more surprised that they were quite receptive to the concept of skilled and agile software testing, wherein our documentation is produce only because and only to the extent that it actually serves a useful purpose.

Filed Under: Context-Driven Testing, Exploratory Testing, Skills, Test Documentation, Testing Culture

Reader Interactions

Comments

  1. Mike Doel says

    October 10, 2006 at 9:05 am

    James,

    It was with great interest I read your experiences of the military plannng process. It seems to me that whenever you get involved with large governent based institutions, (such as the NHS – National Health Service in the UK) that you are often faced with reams of beurocratic process and adherence to convoluted, overly complex and out of date procedures no matter how insane. (I wonder when “Mil-std-499” was written.) This in some sense counterpoints my experiences with most commercial businesses who seem to have problems enforcing an adhereing to good practice and process. Maybe its because these processes have been given to them rather than developed and matured from within.

    What do you think, any ideas on why we seem have such extremes of bad practice.

    Perhaps the process of change for government institutions is just much slower than a commercially driven business. Perhaps the commercail world has been driven to run before it can walk ,where as the other doesnt want to walk unless it has too.

    Leading on from that, does rigoursly applied process narrow the mind and just give people an excuse for not thinking outside the box.

    Mike Doel

    [James’ Reply: Any sufficiently large institution, like a sufficiently large landform, creates its own weather. People working in very large organizations, especially if there is little turnover, tend to see their own experiences as defining the very limits of the world. Add money and ambition to this equation, and process work becomes something akin to a medieval landscape. Fiefdoms are established. Feudal lords reign. Learning, innovation, and high achievement take a back seat to bombast and ritual.

    To counteract this effect takes a strong personality– someone like Kelly Johnson, who created Lockheed’s Skunkworks in order to escape the stultifying organization that would otherwise have prevented his crew from creating such masterpieces as the U2, the SR-71, and the stealth fighter.]

    Reply
  2. insectivorous says

    October 17, 2006 at 3:48 pm

    The problem seems to be growing, and not correcting, at least insofar as the civilian government goes. I’m not sure who started it, but the FAA gets the blame for the system that was going to redefine civil air traffic control in the US, and instead redefined the actual worth of the CMM as per James’ article. Then it spread to the Department of Justice, who used standards and paperwork to create the FBI’s new case file management system. It’s quite rare to do that, to be able to produce a system which, years later, contains absolutely nothing of any use at all. It has spread to the Department of Energy, and should preclude any useful automation happening there. But I’m sure it will all meet the latest government standards as it fails, runs over budget and over schedule.

    Reply
  3. Erwin Van Trier says

    October 20, 2006 at 8:42 am

    The department of Defense also seems to have a study (I am trying to get a link to that report) that concludes that the waterfall method in as a SW development methdology had 0 (zero!) efficiency for their projects.

    I might be making too many assumptions now, but I always thought that the Capability Maturity Model (CMM) was originally created by/for the Department of Defense. It seems that they never used the model to come to the conclusions they are making now.

    On a seperate note, the FDA seems to start asking some questions. During one of the conferences with FDA participation, attended by my peers, the FDA people indicated that they wonder why medical devices have so many problems.
    My answer to the FDA would be that their own guidance would be a direct cause of the issue the are investigating. As far as I know the FDA guidance does not address skill nor training. I believe that both elements should be a key for any guidance document.

    During that same conference FDA also revealed the issue about document approvals
    There seems to be a systemic problem of too many reviewers and approvers on documentation, and that too many lessens the constructive criticism from the review process. An informal study by an FDA inspector inferred that the quality of a document was inversely proportional to the number of signatures when that number was greater then three.

    [James’ Reply: The SEI, which created the CMM, was funded by the Department of Defense for years. I don’t know whether they still are. The great traction that the CMM received has nothing to do with its merit. Adoption of it is limited almost exclusively to companies who gain a specific marketing benefit from it, or who seek preferred treatment on military contracts, which lamely require CMM “maturity”.

    I have seen people who seem to have fire in their eyes about the CMM despite no obvious economic benefit to themselves due to CMM PR, but they are few and far between.]

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Search

Categories

  • About Me (18)
  • Agile Methodology (14)
  • Automation (19)
  • Bug Investigation and Reporting (9)
  • Buggy Products (22)
  • Certification (10)
  • Context-Driven Testing (44)
  • Critique (41)
  • Ethics (19)
  • Exploratory Testing (32)
  • FAQ (5)
  • For Newbies (25)
  • Heuristics (28)
  • Important! (20)
  • Language (35)
  • Management (20)
  • Metrics (3)
  • Process Dynamics (27)
  • Quality (7)
  • Rapid Software Testing Methodology (22)
  • Risk Analysis (12)
  • RST (3)
  • Scientific Method (3)
  • Skills (30)
  • Test Coverage (7)
  • Test Documentation (8)
  • Test Oracles (4)
  • Test Reporting (11)
  • Test Strategy (26)
  • Testability (3)
  • Testing Culture (90)
  • Testing vs. Checking (17)
  • Uncategorized (10)
  • Working with Non-Testers (6)

Blog Archives

Footer

  • About James Bach
  • Satisfice Blog
  • Bibliography: Bach on IEEE
  • Contact James
  • Consulting
  • Engage Our Services
  • Privacy Policy
  • RST Courses
  • RST Explored
  • RST Applied
  • RST Managed
  • RST Coached
  • RST Focused: Risk
  • RST Focused: Strategy
  • RST Methodology
  • Exploratory Testing
  • Testing Training
  • Resources
  • Bibliography: Exploratory
  • Bibliography: Risk Analysis
  • Bibliography: Coaching
  • Bibliography: Usability
  • Bibliography: The Sociology of Harry Collins
  • Schedule
  • Upcoming Public Classes
  • Upcoming Online Classes
  • Public Events
  • Tester MeetUps

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in