Satisfice Website Quality Assurance

I am a tester. I like being a tester. I once was a developer but I would rather be a tester. However, when it comes to this website I have had to go into developer mode. Since I’m also the CEO of Satisfice, that means I am actually doing what I tell students we don’t normally get to do: quality assurance.

This is true, honest-to-applejacks QA. I have been improving this site constantly (well, today I also went to the dentist) since we went live on Saturday. I am trying to assure that it is a good site. During this development process I have re-experienced some of the classic dynamics that pervade quality and testing, and I want to share them with you.

Note: There are three of us on the development team. I’m in charge of content for the site, with the help of Michael Bolton. Michael’s wife, Mary Alton, does the visual design and WordPress configuration.

• The iron triangle lives! Time, resources and quality have an interactive relationship. I wanted to go live with the site by the end of December. I thought four weeks should be enough time to get it together. Then my dog died, my father got very sick, and I hosted a peer conference on Orcas Island. The crises led to temporary depression that sapped my ability to concentrate and destroyed my motivation. Now, if I had been rich I could have spent my way out of trouble. If I had less self-respect I could have released a bad site. That left time as the flexible variable. This process has taken about four times as much work as I thought it would, too. All in all, the January 1st date for launch slipped by almost four months.
• My wife, brother and colleagues helped me get into mental shape. I find that when I can’t engage in work, one therapy is to ask my wife to help me organize my office. She’s an expert at that. She should have a show like Marie Kondo. It took us three days to clear these decks. After that, I sought out collaborators. I can be in a mood where I’m just playing computer games for most of the day and binging BBC shows, but when I have a student on Skype come to me for coaching, or when Michael and I get on Skype video chat, my mind juices begin to flow. Then my brother stepped in, with his program manager chops, honed at eBay, and walked through the web-site to help me create a long punch list for going live.
• This was a social process, and not a completely rational one. We experienced conflict and worked through it. I would write something, and think it was pretty good, then Michael would argue with me about every sentence. Mary felt that that we should scrap our marketing language and start again. Our long experience with the dynamics of conflict and friendship helped us maintain our patience and make the occasional apologies.
• Unforeseen dependencies cropped up. The biggest one was our realization that I couldn’t release my site until Michael and I updated the Rapid Software Testing site. This in turn presented us with a special challenge, since he and I run two different and essentially competing companies. We aren’t used to coordinating so closely, yet now we have to share exactly the same class descriptions.
• The WordPress ecosystem can be frustrating to navigate. We had to test out various plug-ins and themes until we found ones that played well together. Then we had to resolve glitch after glitch. WordPress is the sort of system that you can get 90% working in a few minutes– then the last 10% takes weeks of tinkering. We once spent two hours just trying to change the look of a form field generated by PHP. We had to learn about .po files, regenerate .mo files, and change the site language to British English just to change one single word on a page. Fortunately, all three of us are pretty tenacious. We are used to confusion. We are willing to try many different solutions. That’s technical work, folks.
• We couldn’t create a test environment that was perfectly like production. Because we couldn’t build the system at the same site that it would eventually occupy, we could neither test with SSL, nor be sure that all the links would be correct when we went live. In fact, we got dozen or so links wrong because they referred to our sandbox and we didn’t notice.
• We tested our own work, knowing we were not testing very well. We are just too familiar with it, for one thing. For another, we were looking at it through the lens of “do we really have to fix that before release?” This is the problem of critical distance. Knowing that we were too close to easily see certain easy-to-see bugs, we planned to get our friends to help us test the site both immediately before and immediately after we released.
• The release quality standard was lower than the immediately-after-release quality standard. In retrospect it makes perfect sense, but I hadn’t experience this before as a developer. The dynamics works like this: before I released, I was thinking about what was absolutely necessary to go live. But the very moment we went live, a new dynamic took over: does it look clean and perfect? Does it have everything I want it to have? Suddenly problems that had seemed small felt big; bugs that we shrugged about two days ago are keeping me up tonight. Michael and I are on a furious pace of cleaning and polishing the site.
• Watching the product being used is helping shape my work. I’m monitoring every 404 error and putting in redirects to better connect the content from my old site to the content in the new site. I’m also watching what the popular links are and realizing where I probably should improve the content. Some of the content I thought I would abandon from my old site I’ve now decided to revamp and restore.
• We asked our friends to impress us with the bugs they can find in our work, and they did! I want to thank my brother Jon, Santhosh Tuppad, Alexey Kotomin, and Roman Podolyan for reporting many problems that had eluded the developers.
• The testing we needed to do was not “automated.” It was interactive and exploratory. It required human judgment. The only automated support we used was a link checker, which flagged a lot of things that weren’t really problems.