• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Satisfice, Inc.

Software Testing for Serious People

  • Home
  • About
    • Privacy Policy
  • Methodology
    • Exploratory Testing
    • Reasons to Repeat Tests
  • Consulting
    • Ways to Engage Our Services
  • Classes
    • James Bach’s Public Test Consulting
    • Testimonials
    • RST Courses Offered
    • Rapid Software Testing Explored
    • Rapid Software Testing Applied
    • Rapid Software Testing Managed
    • Rapid Software Testing Coached
    • Rapid Software Testing Focused: Risk
    • Rapid Software Testing Focused: Strategy
  • Schedule
  • Blog
  • Contact
  • Resources
    • Downloads
    • Bibliography: Exploratory Process
    • Bibliography: Risk Analysis
    • Bibliography: Coaching
    • Bibliography: Usability
    • Bibliography: My Stuff From IEEE Computer and IEEE Software Magazines
    • Bibliography: The Sociology of Harry Collins

Agile Testing Heuristic: The Power of Looking

Today I broke my fast with a testing exercise from a colleague. (Note: I better not tell you what it is or even who gave it to me, because after you read this it will be spoiled for you, whereas if you read this and at a later time stumble into that challenge, not knowing that’s the one I was talking about, it won’t be spoiled.)

The exercise involved a short spec and an EXE. The challenge was how to test it.

The first thing I checked is if it had a text interface that I could interact with programmatically. It did. So I wrote a program to flood it with “positive” and “negative” input. The results were collected in a log file. I programmatically checked the output and it was correct.

So far this is a perfectly ordinary Agile testing situation. It is consistent with any API testing or systematic domain testing of units you have heard of. The program I wrote performs a check, and the check is produced by my testing thought process and its output analyzed by a similar thought process. That human element qualifies this as testing and not merely naked checking. If I were to hand my automated check to someone else who did not think like a tester, it would not be testing anymore, although the checks would still have some value, probably.

Here’s my public service announcement: Kids! Remember to look at what is happening.

The Power of Looking

One aspect of my strategy I haven’t described yet is that I carefully watched the check as it was running. I do this not as a bored, offhanded, or incidental matter. It’s absolutely vital. I must observe all the output I can observe, rather than just the “pass/fail” status of my checks. I will comb through log files, watch the results in real-time, try things through the GUI, whatever CAN be seen, I want to see it.

As I watched the output flow by in this particular example, I noticed that it was much slower than I expected. Moreover, the speed of the output was variable. It seemed to vary semi-randomly. Since there was nothing in the nature of the program (as I understood it) that would explain slowness or variable timing, this became an instant focus of investigation. Either there’s a bug here or something I need to learn. (Note: that is known as the Explainability Oracle Heuristic.)

It’s possible that I could have anticipated and explicitly checked for performance issues, of course, but my point is that the Power of Looking is a heuristic for discovering lots of things you did NOT anticipate. The models in your mind generate expectations, automatically, that you may not even be aware of until they are violated.

This is important for all testing, but it’s especially important for tool-happy Agile testers, bless their hearts, some of whom consider automation to be next to godliness… Come to think of it, if God has automated his tests for human qualities, that would explain a lot…

 

 

Filed Under: Agile Methodology, Heuristics, Skills, Testing vs. Checking

Reader Interactions

Comments

  1. @palhed says

    January 21, 2014 at 5:41 am

    If I’m allowed to speculate I suspect that the varying output you saw was impacted by the nature of your input a a double trick? Something along that way. Anyway, sound like a fun test challenge to crack.

    [James’ Reply: That would be a devious and educational trick. Indeed, the tools we use may well influence our systems in ways that obscure the truth.]

    Reply
  2. Claire says

    January 22, 2014 at 8:34 pm

    I’ve definitely found that watching my automation execute helps me when I’m testing. Glad it’s a strategy that makes sense to you. 🙂

    Reply
  3. Monirul says

    January 23, 2014 at 2:14 am

    How much practical it is to watch execution every time it runs? You know many scripts run over night, hour after hour.

    [James’ Reply: Is this a trick question? It’s obviously not practical. Also, that’s not the heuristic. The heuristic is the “power of looking,” not “the requirement to look at everything at all times.” Omniscience is not available to us.]

    But yes, I guess many people do this just after writing the scipts when they run first time. Maybe the aspects are not same.

    [James’ Reply: I recommend that you periodically return and look again. My deeper point is that looking is not part of the automation. You can’t automate that.]

    Reply
  4. Andrei says

    March 14, 2014 at 1:38 am

    Looking is indispensable, as is judgment.

    As complexity increases and ominpresence is no longer an option, I like to think in layers.
    Testware has layers, just like ogres and onions.
    Each one has a distinct purpose and it grows in time as the experience from looking gets coded.

    Automation is not a panacea. It is more like a fine blade: it must be used with care and responsibility. That is unless you want clip your own ear off.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Search

Categories

  • About Me (18)
  • Agile Methodology (14)
  • Automation (19)
  • Bug Investigation and Reporting (9)
  • Buggy Products (22)
  • Certification (10)
  • Context-Driven Testing (44)
  • Critique (41)
  • Ethics (19)
  • Exploratory Testing (32)
  • FAQ (5)
  • For Newbies (25)
  • Heuristics (28)
  • Important! (20)
  • Language (35)
  • Management (20)
  • Metrics (3)
  • Process Dynamics (27)
  • Quality (7)
  • Rapid Software Testing Methodology (22)
  • Risk Analysis (12)
  • RST (3)
  • Scientific Method (3)
  • Skills (30)
  • Test Coverage (7)
  • Test Documentation (8)
  • Test Oracles (4)
  • Test Reporting (11)
  • Test Strategy (26)
  • Testability (3)
  • Testing Culture (90)
  • Testing vs. Checking (17)
  • Uncategorized (10)
  • Working with Non-Testers (6)

Blog Archives

Footer

  • About James Bach
  • Satisfice Blog
  • Bibliography: Bach on IEEE
  • Contact James
  • Consulting
  • Engage Our Services
  • Privacy Policy
  • RST Courses
  • RST Explored
  • RST Applied
  • RST Managed
  • RST Coached
  • RST Focused: Risk
  • RST Focused: Strategy
  • RST Methodology
  • Exploratory Testing
  • Testing Training
  • Resources
  • Bibliography: Exploratory
  • Bibliography: Risk Analysis
  • Bibliography: Coaching
  • Bibliography: Usability
  • Bibliography: The Sociology of Harry Collins
  • Schedule
  • Upcoming Public Classes
  • Upcoming Online Classes
  • Public Events
  • Tester MeetUps

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in