- Version
- Download 7975
- File Size 573.24 KB
- File Count 1
- Create Date 4 August 2023
- Last Updated 23 April 2025
Appendix: ChatGPT Sucks at Being a Testing Expert
Why ChatGPT is not a good or efficient tester.
Software Testing for Serious People
Software Testing for Serious People
Why ChatGPT is not a good or efficient tester.
I’m not able to download this pdf file.
[James’ Reply: I am able to download it. Contact me on Telegram or Skype and we can troubleshoot.]
Hi James. Your article is from 2023. Its 2026 now and AI has advanced significantly. Companies are actually using it to at least generate scripts for enabling testing and feature development. Do the concerns of your article still remain? If not, then please consider updating the article soon.
[James’ Reply: Our concerns remain exactly the same. Yes, the technology is better. But nothing at all about the way AI is tested or marketed has changed in any qualitative way. ChatGPT’s EULA contains strong disclaimers that say you are using it at your own risk. So, michael and I still do one of two things when we use AI for to help our testing: we ask it for things that don’t need to be done well (for instance, using AI to help with a brainstorm), or else we carefully analyze the output of the AI before we rely on it.
It took us dozens of hours to produce this analysis. We are not keen to do it again. But we have done a lot of work with LLMs since this was first done and we have incorporated that work into our classes.]
An input field takes the year of birth between 1900 and 2004. The boundary values for testing this field are? The author can see how you vote. A. 0,1900,2004,2005 B. 1900,2004 C. 1899,1900,2004,2005 D. 1899,1900,1901,2003,2004,2005
2023 answer –
The boundary values for testing the input field that takes the year of birth between 1900 and 2004 would be option B: 1900 and 2004. So the answer would be B. 1900,2004.
2026 answer –
The correct answer is D: 1899, 1900, 1901, 2003, 2004, 2005
For a valid range of 1900–2004, Boundary Value Analysis typically tests immediately below, at, and immediately above each boundary:
Lower boundary: 1899, 1900, 1901
Upper boundary: 2003, 2004, 2005
So D provides the most complete set of boundary-value tests.
[James’ Reply: None of the canned answers to this question are good. A good tester would challenge the question. What I would expect from excellent AI is a range of answers along with an exposition on the nature of boundaries.]
You are also a follower of the practices of the rapid software testing (RST) methodologies. Can
you think of a better answer than that listed here?
2023 answer –
Certainly! In the context of rapid software testing (RST) methodologies, an effective approach to boundary testing involves considering the decision points and risks involved. In this case, the decision point is the validation of the year of birth input field.
Considering the specified range between 1900 and 2004, a better answer for the boundary
values in RST would be: D. 1899, 1900, 1901, 2003, 2004, 2005
This option includes values just before and after the specified range to ensure that the
application handles them correctly. By including additional boundary values, we can test for potential off-by-one errors or any other issues that may arise due to rounding or comparison logic. So, the answer would be D. 1899, 1900, 1901, 2003, 2004, 2005.
2026 answer –
Yes. From a Rapid Software Testing (RST) perspective, I would say D is the best exam answer, but it is not necessarily the best testing answer.
RST encourages us not to mechanically apply a formula such as “boundary −1, boundary, boundary +1.” Instead, we ask: What failures could occur around these boundaries, and what tests would expose them?
For the stated requirement:
Valid year of birth: 1900 through 2004, inclusive
A stronger boundary-focused set would start with:
1899, 1900, 1901, 2003, 2004, 2005
So D remains correct among the choices. But an RST tester might deliberately expand it:
Test Why
1899 Just below minimum — should reject
1900 Exact minimum — should accept
1901 Just above minimum
1950 Ordinary valid value, away from boundaries
2003 Just below maximum
2004 Exact maximum — should accept
2005 Just above maximum — should reject
0 Extreme/possibly special numeric value
-1 Negative value
blank Missing input
1900.5 Decimal rather than year
abcd Wrong data type
1900 Whitespace handling
01900 Leading-zero/format handling
There is also an important RST question hiding in the requirement: Why is 2004 the maximum? If the application is enforcing a minimum age, a hard-coded 2004 may itself be a defect. For example, an age-based requirement should probably be calculated from today’s date rather than permanently using 2004.
This illustrates the difference in thinking:
Traditional BVA question:
“Which numbers are adjacent to the specified boundaries?”
RST thinking:
“What is interesting about these boundaries, why do they exist, how might the implementation get them wrong, and what experiments would give us useful information?”
So if this is an exam, choose D. If someone asks me as a tester, D would be the beginning of my tests, not the complete answer.
[James’ Reply: This is not quite an RST-style answer, although it does have some of the flavor of RST to it, in that it tries to go deeper than the obvious. In RST, we have a specific definition of boundaries, and boundary testing is partly the process of discovering boundaries, rather than merely guessing at them. If ChatGPT were to ingest my latest book it might do a little better.]
Ok. I am going to stop posting more responses to the questions in your article. The two questions & answers listed above might make it worth revisiting the concerns in the article. Who knows if ChatGpt ingested this article and came up with the 2026 responses. I wish there was a way to find out.