The short answer is: you never know for sure that a test has ended.
Case in point. The license plate on my car is “tester.” It looks like this:
On December 20th, I received this notice in the mail:
As you see, it seems that the city of Everett, which is located between Orcas Island (where I live) and Seattle (where I occasionally visit) felt that I owed them for a parking violation. This is strange because I have never before parked in Everett, much less received a ticket there. A second reason this is strange is that the case number, apparently, is “111111111”.
At first I thought this was a hoax, but the phone number and address is real. The envelope was sent from Livonia, Michigan, and that turns out to be where Alliance One Receivables Management, Inc. is based. They collect money on behalf of many local governments, so that makes sense. It all looked legitimate, except that I’m not guilty, and the case number is weird.
Then it occurred to me that this may have been a TEST! Imagine a tester checking out the system. He might type “tester” for a license plate, not realizing (or not caring) that someone in Washington actually has that plate. He keys in a fake case number of “111111111” because that’s easy to type, and then he forgets to remove that test data from the database.
Praise the Humans
I called the county clerk’s office to ask about this. At first I was worried, because they used an automated phone service. But I quickly got through to a competent human female. What can humans do? Troubleshoot. She told me that there indeed was a record in their system that I owed them money, but that the case number did not refer to a real case. In fact, she said that the number was incorrectly formatted: all their case number start with a “10.”
“This can’t be right,” she said.
“Could it be test data? Are you just starting to use Alliance One?” I asked.
“We’ve been using Alliance One for years. Oh, but we’re just starting to use their electronic ticketing system.”
She told me I was probably right about it being a test, but that she would investigate and get back to me.
A few days later I received this notice:
So, there you have it. Someone ran a test on November 9th that did not conclude until December 23rd when it is stopped via a court order! Thank you, Judge Timothy B. Odell.
I’m sure this will appear on an episode of Law and Order: Clerical Intent one of these days.
Just imagine if this hadn’t been a parking ticket program, but rather something that told the FBI to go and break down my door…
Morals of the Story
- Beware of testing on the production system.
- Always give the humans a way to correct the automation when it goes out of control. (Hear that, Skynet?)
- You never know when your test is over.
- If your name is “tester” or “test” or “testing”, eventually you will show up as test data in somebody’s project. Beware also if your name is “12345”, “asdf”, “qwerty”, “foobar”, or “999999999999999999999999.”
Nice. 🙂 That’s a great story with a great lesson and it made me laugh. Reminds me of little Bobby Tables http://xkcd.com/327/
Excellent story and morals!
Are you going to change your license plate? Or are you waiting for the next application to be tested? 🙂
Alex
[James’ Reply: Change? Never!]
Splendid story!
I think this also reveals something about the tester and his imagination. We might hope that they don’t have a tester that prefers using obscene names as test data, because then the proctologist using A**man on the license plate might get a ticket (taken from a joke in Seinfeld). 🙂
Two weeks ago I blogged about another incident when testing on a production system http://thetesteye.com/blog/2010/12/the-helpful-model/ (when thinking of it, I think I told you this story over a dinner at SWET1).
Cheers,
Henrik Emilsson
James,
I take another lesson from this too: take care when you are copying to the live environment; check that your scripts to copy everything live do not include the live data tables – the schema maybe but not the actual data.
As soon as we deploy to the live environment I do a quick test to make sure my test data has not inadvertently gone across too (before any scripts have run that do any data processing).
Test data in the production environment, as we have seen, can cause a lot of wasted time and money and loss of reputation.
Thanks James – a great story to tell in the future!
Regards,
Stephen
Wouldn’t want to be called “John Do” in the US – I wonder how many bugs like that they get…
I had also “interfered” with the tests on the production environment at least once.
I created a hosting account at Inmotionhosting.com with username “testalways”. I was waiting to get the account activated, but it didn’t. After I called support they informed me that the accounts with username beginning with “test” are not being activated.
They had good support that quickly fixed the issue.
–Sebi
If your name is “tester” or “test” or “testing”…
This name was stuck in my memory for a while: Ralph Tester.
Ralph P. Tester was an administrator at Bletchley Park, the British codebreaking station during World War II. He founded and supervised a section named the Testery for breaking TUNNY (a Fish cipher). […] TUNNY was used to encipher top-secret messages between German Army H.Q. in Berlin and the top generals and field-marshals on all fronts…
Also, while looking up for the article in Wikipedia, I found reference to The River Test in Hampshire, England.
James,
What a truly hilarious yes disturbing coincidence that you of all people would get the ticket from the “testing”. I was literally laughing so hard I can still barely breathe. I couldn’t agree more, in the value of humans for troubleshooting, and not just humans but intelligent humans that fluently speak english.
Great story. It makes me wonder how many tests have resulted in parking tickets that actually got paid. Watch out if your license plate starts with “asdf”.
That reminds me I made that mistake in my first testing job. I worked for a mortgage broker that hired some Foxpro experts to create mortgage processing software. She told me to test it because I had experience in processing mortgages. But I had no “instructions” and definitely no training. So I started creating accounts with all kinds of nasty data (outrageous numbers, etc., not “explicit”). Next thing I know their reports are all fronked. As a result, we created a test database. Oh the lessons learned from no guidance.
Dave
I often shudder to think of what turns up in the inbox of test@test.com
Hilarious!
At a recent I laughed at some of the test data. “Ali Mente” which would translate to “allmony”, besides that there might be someone out there called Ali Mente. You’ll never know.
This did make me laugh! Wonderful story James & very good food for thought for everyone.
I’ve seen a few mishaps like this in our workplace. Thankfully nothing so serious, just a few funnies that didn’t get removed.
Notice, too, the officer’s badge number: 01111
I find it ironic that the code was “Missing/Unlawful License Plate.”
We don’t break software, we reveal *laws* about software that can be broken.
What a great anecdote to share in future classes – and a relief it was so easily resolved. Hooray for competent humans! Sorting it out could have been quite a headache.
Not quite as cool as the NOPLATE/NOTAG bug (http://www.snopes.com/autos/law/noplate.asp), but nice work anyway.
Terrific story! Thanks, James.
Reminds me of the story of the woman from New England named None (no-nee) who had a vanity plate with her name. When she visited NYC for the first time, she was arrested and jailed as the biggest scofflaw ever–though she’d never even been in NY before. Turns out, tickets on all unlicensed cars had the license # filled in as NONE. I don’t remember the fines, but they were in the tens of thousands of dollars.
I think she eventually got out of jail and didn’t have to pay. I hope so, anyway.
I’m wondering if the culprit reads this blog? If so say hi 🙂
(or perhaps a sorry would be more appropriate?!?!)
Dang it! Now you’ve got me second guessing if I should name my kid “Tester”. Probably not a good idea I guess. The poor kid will probably be cursed with verbal corrections: “Did you say Chester?”, “NO, TESTER”, “Did you say Hester?”, “NO TESTER”.
Excellent story James. And it made me laugh so much that you of all people would be the victim of this test.
What I dont understand though, is what this eTicket system was supposed to do? Issue an eTicket that after lying around for 30 days is sent to court? You didnt receive the eTicket in any way before?
[James’ Reply: I suspect they tested with that offline. What probably happened is that some data export table was not cleared out, and then they went live and the data was automatically sent on to the next system.]
In this case, the defendant reads “Unknown Vehicle Violator”. I am unsure if the term “violator” can be used before judgement is pronounced.
Secondly, maybe they need US Laws to prosecute “test cases” 🙂 !!!
After dancing around separation of live data, auditors and anyone involved in testing for the last five years, I shudder that in my industry this would have be prosecutable, in no uncertain terms.
Me, I’d be decorating my bumper next. 😀
http://kyon.pl/img/15596,prev.html
My father had no middle name. He was often told to enter “none” on forms in lieu of a middle name. Humans recognized this convention.
Enter computerized records, and he started getting mail with “None” as his middle name. The computer programs not only weren’t designed to recognize the convention but they fixed the entry by capitalizing it.
I am just trying to visualize ….wondering if you were not a tester what would have happenned next…..
Would you have just made the payment or still tried to reason, logic and get to the root of the problem ?
🙂
Also makes me realize … State of ordinary human being is tough everywhere as they struggle to survive !!
[James’ Reply: My wife is not a tester. She received the letter and thought it was a fake. She wanted me to discover if the address and phone numbers were real. When I found that they were, our next idea was to call the court clerk. Lenore is always alert to the possibility of being sent fake or mistaken bills.]
Interesting story; ……We need good testers !
Hey,
This is awesome and wondering that how a tester is given a prod access, rather he/she must be provided with the dump of prod in QA env. Well now are you going to change your plate? Or you gonna waiting for the next application to be tested and next letter would on your way?
Check ur letter Box “You gotta mail” haa 🙂
Wonder if culprit reads this blog by any chance -:)
[James’ Reply: I’m a tester, dammit. I’m keeping my plate. Let the tests fall where they may.]
I’ve made similar mistake once when I was still a rookie. I was testing a banking system in live enviroment for post-live testing to make sure everything still worked there. We had a issue due to some small configuration difference (it wasn’t easy to fix) between live and test enviroments which caused some wierd problems in live. Thus I had to do final testing in live enviroment. After finishing work I thought I had deleted all of used testdata, but when 2 days later letter arrived to the company address about loan in debt… well, I had some explaining to do to the project manager about working late hours and tired mind. I learnt 3 things about it_
1) If testing in live enviroments (or whereever some data transfer might take place) then double (and triple if it’s late) check if you have deleted all the data. I missed about 10% of used testdata.
2) Know what system (and people using the system) does/is supposed to do with data. I had to explain why and how the letter came. If I hadn’t known the system from inside out, then it would have been way too much assuming.
3) Always use as real as possible and known data. What if the letter had gone to someone else who didn’t know about the product?
This story reminds an incident I had with a vendor registry system of a major software provider. I received their invitation to register our company and product at their system. For some reason I could not get through the last step of their online registration system. I kept getting a generic error saying something like your request did not go through.
I emailed them back about the problem but they did not see any error in their system. So I was asked to try it again as they were thinking it’s got to be a user error.
After many going back and forth, they finally found out that it was my company name that was causing the problem: it has “test” in it and the developer apparently put in some special logic in the code for his/her testing purpose during development which got deployed to production environment. Luckily the developer did offer a way to bypass this logic and allowed their helpdesk to complete the registration for me.
A wonderfull reaL llife example of what happens when testers test on live machines!
Very good of you to just call!
A reality of the testing strategy indeed. Once i used to test by using the email address ‘xyz@email.com’ . But few days back one of my colleagues showed me that there is a domain ‘www.email.com’. So, if there be a user using this email address, how many emails he/she got from my system.
Happened to me a few years ago. I received a notice that I owed Bloomington, IN $72.00 for a parking violation.
My vanity tag was Test 1.
I had not been in Bloomington for more than ten years.
Took a lot of phone calls but finally the fact that Test 1 was used in their test scenario was identified and the fine revoked.
Jim Dougherty
[James’ Reply: Wow. I wonder how many of us are out there? Victims of lazy test data selection…]
Good story & Good lesson! 🙂
this is an awesome example and a dangerous one also..what if it were a layman (not tester) at the receiver’s end and yes what if the issuers were e.g. FBI…..people still feel testing is simple and can be done by Hoi Polloi kind of people but that’s not true…This post should find its way to new york times so that it can be read by masses and inorder to make them aware of whats happening…
Hey James,
Thats awesome story and it made my day.:)
Even we are also facing same problem after making our system live. During testing we usually enter those things which are convenient for us but sometimes it creates such issues.
I think, thats why live data is most preferable during testing even on testing server.
In one of my the past project, when I used live data during registration of the system, Client understood those data is real and praised our hard work lots. We laughed at that time also.;)
But though thats preferable. 🙂