In line with your post: “…the HAL 9000 computer, which can reproduce, though some experts still prefer to use the word ‘mimic,’ most of the activities of the human brain…”. Here lies the interesting part of the story, HAL isn’t some mysterious superintelligence, he only *seems* to be. When he starts to kill the crewmembers it’s easy to believe that he is indeed an intelligent, sentient beeing and that he is *evil*. But in the end we get the technical explanation, there’s nothing mysterious or magical about his behaviour, it’s just a bug.

A recurring theme in Clarkes books is best explained with the quote “Any sufficiently advanced technology is indistinguishable from magic.”. Just like Dave Bowman feels threatened by the HALs “unexpected behaviour”, the humans feel threatened by the mysterious monoliths, but even they are just machines, only far more advanced.

Imagine the consequences of a bug in the monolith…

I apologize if I misinterpreted your points and took them to too far of an extreme. I’ve encountered that before and it is definitely frustrating.

[James’ Reply: I only publish comments that I think have value to readers. I think your response has value. I reserve the right to be convinced by you when I read your arguments.]

If you don’t intend by your comments the extreme sense in which I took them, then I need some clarifications to understand you. For example, when you wrote (in your latest response):

“the people who think intelligence can be built literally don’t know what they need to build or how to build it. This is a conceptual problem, not just a matter of technology.”

I wonder what you mean by “know”. From my point of view, people DO know (somewhat, though not perfectly) what they need to build and how to build it. For one example, they know that they need to build adaptive systems with massively parallel processing. The details aren’t worked out yet, of course, but there is a direction and a goal that (it seems to me) are well understood - ‘though, again, not completely understood.

[James’ Reply: You are being incredibly vague! You might as well point at Chitty Chitty Bang Bang and tell me that we know how to make practical flying cars. Okay, here’s some homework, go on the web and find a research paper or project that contains specifics. Here’s one you might look at, for instance. This is fascinating stuff! It looks quite promising. But it still deals only with pattern matching. Even if the system described is built (which it hasn’t yet been, in full industrial form), I bet it won’t reproduce the features we expect from a self-modeling being as described in yet another fascinating video here. And if it seemed to, I would question the testing of such a system on complexity and conceptual grounds as I’ve already said.]

The above seems so obvious to me that I expect that it must also be obvious to you, so I feel that I am forced to conclude that you mean “know” in a more complete sense…but that leads us back to the absolute-versus-sufficient argument which you deny being on the other side of…

[James’ Reply: How can something seem obvious to you when you can’t describe or produce it? When you’ve done either of those things, you can tell me it’s obvious. Until that time, you are just like some Scholastic professor from the twelfth century arguing how obviously kings rule via the grace of God who sends angels to guide them.]

“I challenge you to show how it might be testable– but that would require you to define it, which I predict (perhaps I’ll eat these words) you will not be able to sufficiently do.”

Here’s a good test suite: The AI must fool “the experts” in the Turing test (experts such as psychologists who are trained in evaluating intelligence in humans).
If you think this is insufficient, then how do you define sufficiency in this case? Since we’ve already agreed that sufficiency is something less than perfection, I am very curious to know how far less than perfection you have in mind.

[James’ Reply: If you can’t tell me what you are talking about, we aren’t even at the point where “sufficiency” is a meaningful discussion. But let’s pretend, just for fun, that you HAVE come up with some accepted specification of intelligent behavior worthy of being associated with the Singularity (a kind of intelligence that can somehow perpetuate itself, improve itself, and perform arbitrarily complex design activities in the service of humans in a manner that progresses at a speed, quality and to an degree bounded mainly by the level of fungible computing resources we throw at it). Let’s say you feed this specification to your psychologist expert friends. I notice that you have faith in their ability to find EVERY CRITICAL BUG. Notice I did not say every bug. You aren’t claiming that. But clearly you think that there is a reasonable chance they will locate every critical bug. Furthermore, you must think any critical bug they don’t find will not be so critical that it will lead to a terrible disaster before it is found. And finally, you think that problems will be correctable without running an unreasonable risk of harmful new side effects.

I’m astonished that you believe all these things. For one thing, you’ve never done any of this, nor have you seen it done. To extrapolate from simple software to testing and debugging a neural network-like thing with billions of moving parts that adapts itself in real time strikes me as pretty naive. If you’ve read Dietrich Dorner’s book The Logic of Failure, you know that humans are terribly bad at controlling (read “debugging”) non-linear dynamic systems. If you are familiar with chaos theory, you may be concerned about emergent side properties (which may reveal themselves as psychoses or neuroses or hallucinations in the artificial intelligence). If you read about genetic algorithms, swarm intelligence, or finite automata, then you may be struck, as I am, with the remarkable enthusiasm scientists show for the unexpected. They love when their tools do something they didn’t plan. But as testers, we have to be concerned that the unexpected harbors only pleasant surprises. How can we do that?

The psychologists you refer to would not merely be testing software, they would be doing the equivalent of social science research on their artificial subjects. Have you read much about social research methods? Have you talked with clinical psychologists about how psycho-analysis works, and how it may be subverted? It’s not so easy as you seem to think.

I’m trying to imagine what exactly the psychologists would do. Talk to the system and ask it to perform various tricks, I guess. But what if the system, in its self-consciousness, resented being tested, and resolved to fool the psychologists? Intelligence like ours comes with self-consciousness as a built-in feature, and this self-consciousness, as you know, is extremely volatile. If you tell me that you know for sure that the system can’t be trying to fool the testers, I want to know how you could possibly know that. If you tell me that the system doesn’t have self-consciousness, then I want to know how you think it can get along without that, since the ability to understand a purpose is key to design, and that requires the ability to put oneself in the shoes of the user, to some degree.

In short, before we could trust software to act on its ideas, we would have to develop an effective cyber-psychology discipline based on the nature of such machines. That hasn’t been developed, obviously, and it can’t be until the intelligence specification hurdle is crossed.]
[On “the curve”]
I haven’t seen (or failed to recognize) your objections to “the curve”. I might look into your previous posts, but this aspect just isn’t as interesting to me, actually. This seems like a separate argument, and I might try to challenge you on it separately.

[James’ Reply: How could it not be interesting to reply to me when I’m attacking your premise? If your premises don’t stand, then what else is there to talk about? Anyway, I didn’t attack the curve very explicitly in my original post, but there was this sentence “For one thing, we aren’t even able to define intelligence, except as the ability to perform rather narrow and banal tasks super-fast, so how do we get from there to something human-like?” The idea behind that sentence is that you can do all the banal things you want, and do a trillion of them per second, but that doesn’t “add up” to human-like intelligence in the absence of the right algorithm. We don’t have such an algorithm. Unlike uranium, there is no critical mass of desktop calculators that when piled high enough in a confined enough space will spontaneously become self-aware.]

you wrote:
“the concepts that make it possible to turn computing power into a human-like design intelligence don’t exist, and can’t exist. The reason they can’t exist is that whenever anyone claims to have produced them, they will be subject to a host of easy counter-arguments from epistemology, phenomenology, ethics, testing theory, etc. If someone seems to be getting close, though, I would predict a moral crusade (as against stem-cell research, cloning, and eugenics) that would shut down the project.”

Let me get this right…you say that the reason these concepts don’t and can’t exist is because the masses will object, and shut the resulting projects down, etc.? This seems like a two-fold non-sequitur to me. Firstly, the existence of a concept is not contingent on the opinion of the masses. Secondly, the opinion of the masses will not be anywhere near strong enough to overwhelm the drive for improving AI, because the masses are primarily concerned with the short term, and in that short term, AI will provide enormous advantages to whoever has it over those who don’t.

[James’ Reply: You can invent any concept you want, or even pretend to, but when it comes time to claim to someone other than yourself that you invented something that “loves” or “hates” or “has mystical visions” or “thinks” or “uses polysyndeton” then you have a rhetorical problem; a sales problem; a philosophy problem.

You say that AI will provide advantages, but of course that’s a testing problem you haven’t solved, so you are just assuming what has not yet been demonstrated. What you could say instead is “if these mysterious systems seem to produce enough benefits without much risk, over a period of time, people will come to rely on them, even if, like airplanes sometimes do, there is a fiery crash once in a while.” I can accept that. The question is still whether they can be built at all, and how much rebooting will they need.

I’m more concerned about a DDT-style of “Oh my God, when we started using this stuff we didn’t realize we were creating a terrible global problem” type of disaster.]

At the very least, the masses will see a need (aligned with their short-term goals) to continue improving AI in order to compete with other countries who we suspect are also developing their own AI. Already we have seen the fruits of “soft AI” in smart-bombs, weather predicton, ship guidance systems, etc. I predict that this trend will continue until it reaches the tipping point. By the time the masses catch on to the long-term implications of improving AI, it will be far too late to do anything about it short of self-annihilation.

[James’ Reply: I don’t know what you mean by “tipping point” but wow, in your last sentence you seem to have stepped over the line onto my side of the argument. Let me repeat back what you just wrote. “…it will be far too late to do anything about it short of self-annihilation.” Wow. I would call that a big requirements bug. See, I think enslaving ourselves to machines is a BAD thing. As it stands, we are enslaved by other humans who use machines, which is perhaps inevitable. But it is NOT inevitable that we should cede our liberties to the machines themselves.]

you wrote:
“I don’t think those tests are good enough”

What does “good enough” mean to you (in the context of our argument on whether the singularity will happen)? In other words, what level of fidelity would the tests need to achieve in order for the singularity to happen? And why do you think the fidelity of the tests would have any impact on whether the singularity happens or not?

[James’ Reply: I wrote an article on good enough quality and another on good enough testing. See my website.

The tests, of course, don’t have any impact on the Singularity as a technological phenomenon. Perhaps the essence of the Singularity occurred in 420 B.C. in Athens, before being accidentally destroyed by invading Spartans. Perhaps the Singularity has already happened in a government laboratory, ten years ago.

Testing doesn’t make reality. It has to do with the value of claims people make about the world. Testing has to do with whether Singularity hype deserves our attention.]

you wrote:
“Why do you dismiss my concern that there are important subtleties to intelligence that we probably aren’t modeling?”

Because I don’t see how they have any bearing on whether the singularity will happen or not. Testability is not a requirement for functionality.

[James’ Reply: What you’re really saying is that you will be satisfied by the weak testing that you or others will do before they declare that the new era has begun. I hope that you learn important lessons from that. Meanwhile, I will continue to urge people to better lives through skepticism, so that we are more likely to see the world for what it is. This is increasingly important in the modern world. Unlike the era of Pyhrro and Sextus Empiricus, being fooled by illusions can have terrible and far-reaching consequences.]

you wrote:
“Why aren’t you concerned that about complex and inscrutable technology running amok or breaking down? ”

I am concerned about this, but I don’t agree that it will cause a “ship-stopper” for the progress of technology or the the intelligence of AI. Technology will run amok. Technology will break down. This will be a fact of life for eternity, I think. I just don’t think it will be bad enough to counteract the historical trend in progress (yes I know you don’t agree with the premises of “the curve”).

[James’ Reply: Okay, I guess we will see what happens. And as it does, one of us will believe what he sees, and the other one won’t.]

you wrote:
“As a tester, I bring systems to failure states regularly and often easily, despite the great confidence of their developers. Now you want to extend our buggy legacy of technology to the realm of computer super-intelligence?”

Does technology improve over time or doesn’t it?

[James’ Reply: It seems to improve in some ways, and degrade in others. Improvement is a relationship among many things, not a palpable substance. I once had a cell phone I loved, but I was forced to give it up by my cell phone company. I then went through a bunch of phones I hated, each of which were supposedly more advanced. Identity theft was a rare occurrence, twenty years ago. Privacy was easier to protect. I once had the option not to use a computer everyday, but now that’s not feasible. From what I hear, Vista is not more reliable than XP, and its digital rights features are Draconian. I don’t think that just because Microsoft wants to sell us something that we have to buy it. Even things that do improve are not safe from outright obsolescence because of some new widget Bill Gates will push on me, next.]

you wrote:
“So, how could we humans write a program to duplicate the essential features of a human?”

Well, we can just copy the genetic program that already exists (and expose it to the right conditions for it to develop). You say that “DNA doesn’t write a human”. How can you possibly argue this? Our genotype encodes our phenotype. Just read any book on the subject.

[James’ Reply: If I was drinking something right now, I would have spit it out in surprise! Come on, man. I love how you say “we can just…” Yeah right. Which specific book tells us exactly how the program works? Hmm? How come cloning is proving to be so difficult to do? Hmm. Why it looked so easy on Jurassic Park! And what specific magical computer simulation do you propose that will simulate the development of a human from a simulated egg and sperm? The protein folding of all the encoded proteins in an entire human body all at the same time would sure be hard to reproduce in a computer. Consider that protein folding for just one molecule is already a fiendishly complex calculation that it already is the subject of a grid computing project.

I understand that you can imagine what it would feel like to imagine the possibility of such a thing, but I think you haven’t yet actually BEGUN to imagine the possibility itself.]

you wrote:
“I think, if you want to make a persuasive argument, you should cite an example of some sort of intelligent machine that works today”

Good examples abound on the internet:

http://en.wikipedia.org/wiki/Artificial_intelligence#List_of_applications

[James’ Reply: Yes, but which one of these applications is capable of creating the singularity? I visited the section on artificial creativity, which included a caveat about how creativity is a controversial issue, then I followed a link to thinkartificial.org, where I saw this: “Defining Creativity: As with natural intelligence, creativity has remained extremely hard to define. Creativity exhibits itself and effects behavior to a large degree (especially in humans), which makes it very hard to identify its distinguishing features and nature. No empirical definition or authoritative perspective on creativity exists within scientific circles.”

Anyway, please pick an example, because offhand I don’t see one here that threatens my position.]

You seem to think that the technological singularity won’t happen simply because it (and intelligence itself, for that matter) is not completely testable. (Please correct me if I am mistaken). My argument is that technology need not be testable in order to work. It need not even be bug-free. After all, nothing in this universe is completely testable, and no technology is completely bug-free. Nonetheless, despite these facts, the rate of technological progress (by various metrics) has been steadily accelerating along an asymptotic curve for the entire history of human technology.

[James’ Reply: I think you’ve simplified and then polarized my argument. I am not complaining because of a lack of perfect testability, I’m arguing that there is a lack of *sufficient* testability. Arguing for lack of perfection would be silly. We don’t ever deal with perfection in our business. No, I’m saying it is woefully untestable, just as Dave Parnas argued that the Space Defense Intitiative would be untestable (notice SDI ain’t working, either). It’s untestable not necessarily because it’s complex, but because the people who think intelligence can be built literally don’t know what they need to build or how to build it. This is a conceptual problem, not just a matter of technology. I challenge you to show how it might be testable– but that would require you to define it, which I predict (perhaps I’ll eat these words) you will not be able to sufficiently do.

In addition to the testability issue, there is another issue: reliability. I’m saying that complexity will inevitably crush reliability.

Finally, I’m also arguing that a tester’s skepticism is needed to leaven the imaginings of starry-eyed extropians.]
All that is required for the singularity to happen is for the rate of technological progress to keep following this curve as it has so far. Maybe it won’t, but in order to argue this point, you’ll need to bring something into the equation that is not already there. Certainly the testability problem, and the debuggability problem are age-old problems that have been intrinsic to technology throughout history, so whatever effect they do have on the shape of the curve, they have already had their effect (and the resulting curve is asymptotic despite them, at least so far).

[James’ Reply: I’ve already raised the appropriate issues. That you dismiss them doesn’t in any way diminish them. You’re free to ignore anything I say, of course, but if we’re in a conversation, the way it works is that you offer rebuttal and then we see if I can make a counter-rebuttal.

Let me clarify one point, though. You speak of a “curve”. I have implicitly denied the meaningfulness of that curve. Therefore I’m denying your premise. Whatever you think that curve is, I’m arguing that the concepts that make it possible to turn computing power into a human-like design intelligence don’t exist, and can’t exist. The reason they can’t exist is that whenever anyone claims to have produced them, they will be subject to a host of easy counter-arguments from epistemology, phenomenology, ethics, testing theory, etc. If someone seems to be getting close, though, I would predict a moral crusade (as against stem-cell research, cloning, and eugenics) that would shut down the project.

The thing I’m really worried about is someone producing something that fools a lot of people into thinking that it’s sentient, thereby inspiring misplaced trust and then misdirected outrage.]

Specific responses to your last points:
On intelligence testing: I agree with you that intelligence is not perfectly testable. My point is that the level of testability of AI is basically the same as that of natural intelligence. No test suite is perfect, of course, but our many methods for estimating human and animal intelligence do seem to tell us something reliably (’though not 100% reliably, of course). AI is as testable as human intelligence is, if not more so (since we are somewhat better at white-box testing AI than human intelligence)

[James’ Reply: I don’t think those tests are good enough. I don’t think they can be good enough. But if someone thinks they can be, they ought to muster some evidence of that. I see that you have faith in them. Why should anyone else share your faith? The stakes are very high, here. Why do you dismiss my concern that there are important subtleties to intelligence that we probably aren’t modeling? Why aren’t you concerned that about complex and inscrutable technology running amok or breaking down? It already does do that, and on a regular basis. As a tester, I bring systems to failure states regularly and often easily, despite the great confidence of their developers. Now you want to extend our buggy legacy of technology to the realm of computer super-intelligence? Have you read any of Petroski’s work on human error in engineering? Have you read Normal Accidents or The Logic of Failure? Have you read anything about the dangers of automation bias in airline cockpits?]

On DNA as a human-writing program:
“It’s some kind of program, but what kind and to what extent? When I write “silly wabbit” in a sentence, as I just did, was that determined by some combination of base pairs in my DNA, or RNA, or epigenome? Or is it an environmental toxin that made me write that? Maybe a hormonal signal? ”

DNA is only one factor in the programming of a human. There’s also the environment of the world around them (culture, etc.). It’s wrong to expect a genetic reason for every human quality. But my point was that humans are “written” by programs, in refutation of your claim otherwise. DNA is a good example of a human-writing program, but that doesn’t mean there isn’t more to being human. I would consider the world of experience a form of programming as well - perhaps focused more on the brain than the organism as a whole, but still a program.

[James’ Reply: No, this is not a refutation of my argument, it merely misses the point of my argument. I said no program has written a human. Indeed, you can’t point to any program that has “written a human”. DNA doesn’t write a human, the environment humans live in doesn’t write humans. I can accept that the processes we call Evolution, operating within physics and over eons of time, led to humans, whatever the heck humans are, but A) That history is not a program in any sense that computer people would understand it, and B) we don’t know what the resulting “human program” actually is.  So, how could we humans write a program to duplicate the essential features of a human?]

Similarly, there’s more to a running application on a computer than just the source code. There’s also the compiler, interpretter, CPU, RAM, plus the entire rest of the physical machine that all this stuff runs in the context of. So it’s actually quite a good analogy, I think.

[James’ Reply: It seems like a good analogy to you for two reasons: 1) you have conflated the comparatively well-known and well-controlled world of computing with the comparatively poorly known world of biophysics, and 2) you haven’t glossing over how bug prone computer systems are.

I think, if you want to make a persuasive argument, you should cite an example of some sort of intelligent machine that works today. I’m arguing from concepts and dynamics, as you are. To transcend that, we probably need to move to examples. My claim, at the moment, is that there are no examples that I can’t disqualify with my tester powers, to my satisfaction. I may be wrong.]

[on human-level AI]
“My question is, when someone DOES reproduce it in its essential respects, how would we know? This is a testing question”

Indeed. We wouldn’t be able to know it ABSOLUTELY since we can never test 100% (this is true of everything), but on the other hand we can certainly do better than 0%. The thing is - we don’t actually need 100% certainty about something in order to agree that we “know” it or understand it, now do we? At least not in the relative sense. How much evidence would it take to convince you that the singularity had happened? Certainly less than 100% (for that would be complete omniscience).

[James’ Reply: Percentages don’t mean anything in this context. What is needed is a culturally acceptable story. This is what testers help to create. That air travel is safe, for instance, is something that would have been unthinkable in 1895, but over time a story was developed that made air travel culturally normal. No ordinary person knows how that safety is achieved, or what that safety really means, but most people accept it.

I’m saying you haven’t even gotten to the first milestone for doing that. There hasn’t been a Wright brothers event that even begins to make it possible to consider. The Singularity still has the status of mythic trope, rather than specifiable event. Once you specify it, we can talk about how it could happen. So far, I understand that you have specified it as a program that could pass an intelligence test, and I’ve countered that a rather stupid program can already do that.]

“Something has to debug it? What will debug it? With humans, it was a couple billion years of mutation and natural election. The result is still buggy, by the way, but we more or less accept those bugs”

Well, strictly speaking, it only needs to be debugged if it doesn’t work well enough initially.

[James’ Reply: Yes, that’s a testing problem. One that for reasons already mentioned can only be solved by bad testers, and only by fooling themselves.]
There are always more bugs…we just correct enough of them. We won’t necessarily know what “enough” means until we achieve it, but up to that point we simply debug it just as we would debug anything else. We search for inconsistencies and correct them as we find them. It’s doomed to be imperfect, but it’s good enough that progress has managed to continue up to this point. Why would this change?

[James’ Reply: I disagree that ANY significant progress has been made, since you have no idea (and I don’t either) what progress really is, since we can’t measure it in a way that is not easily dismissed by well-intentioned people who argue for a deeper meaning of intelligence. Perhaps some progress has been made by people who think of intelligence in parochial and simplistic terms. But what I’ve seen of that work is not particularly interesting.]

You test them in the same way as you would test human (natural) intelligence. Intelligence is intelligence, whether it is man-made or not.

[James’ Reply: I’m afraid it’s not as simple as that. First you have to define intelligence in a way that is accepted by whomever the stakeholders are. But there is no definition, no measurement, that is not objected to by some constituency or other. But let’s say that you decide to use the test at http://simple-iq.com/ (I’m not saying it’s a great test, but it’s a multiple choice test, just like a lot of tests that purport to measure I.Q.) I just took that test by choosing “true” for each answer. That’s the equivalent of a one-line program consisting of $answer = TRUE, which is hardly something you would call a breakthrough A.I. program, eh? Well, the test tells me that I have an I.Q. between 85 and 99. So, are you satisfied with that kind of measurement?

Another problem is that untrained humans are extremely easy to fool, which is why con artists and mystics do such brisk business, at times. Have you ever been married and had the realization that you were mistaken about the type of person whom you thought you married? I have. The experience taught me caution. But when it comes to artificial forms of intelligence, the assumptions and heuristics that often guide us well with humans, and are absolutely reasonable for use with humans, will have no basis at all. For instance, if you told me, triumphantly, that you were a computer chatbot only simulating a human named Andrew, I would put aside all the assumptions I have about people who have Hotmail accounts and begin to scrutinize every conceivable subtlety of how you interact with me. I would withdraw all my previous assumptions that are causing me not to ask you Blade Runner type questions right now (”Why did you flip the turtle on its back, Andrew? Why?”) It’s kind of like how someone might reassess the meaning of a silly lawyer joke told to them by a co-worker, if that co-worker were arrested the next day for mass-murdering a bunch of lawyers.

I’m saying it’s a hard problem to test for a thinking machine… if you want to do it well.]
“[…] how would you know that human intelligence doesn’t possess secret and subtle features that have not yet been identified?”

You wouldn’t know this. This would require omniscience. How is this question important, though?

[James’ Reply: It’s important because once the program passes your tests, and you declare it practically indistinguishable from human intelligence, you and many other people are likely to cease thinking at all critically about it. You will apply a bunch of assumptions to the A.I. creature, just as the astronauts did in Arthur C. Clarke’s “2001″, without realizing that perhaps your A.I. is insane in a way that is not yet recognized, but could harmfully manifest at an unexpected moment.

An analog to this is DDT and asbestos. People once thought those substances were not dangerous to humans or the environment. Oops, poor testing led to a big surprise. When I was a kid we played with mercury in science class; today spilling mercury on the school floor would cause an evacuation of the school and notification of HAZMAT. What about food irradiation? Some people say irradiated food is safe. Maybe it really is. Or maybe there is an as yet unrecognized side effect.

I’m not prepared to say what the capabilities of the human mind are, despite the fact that I own more than 100 books on this subject. It seems to me we are still working it out.] 

“Although humans have written programs, no program yet has written a human. ”

Wrong. Our genetic code is a program (or a collection of programs).

[James’ Reply: It’s some kind of program, but what kind and to what extent? When I write “silly wabbit” in a sentence, as I  just did, was that determined by some combination of base pairs in my DNA, or RNA, or epigenome? Or is it an environmental toxin that made me write that? Maybe a hormonal signal?

I’m prepared to accept as a practical hypothesis that human thinking is completely determined by known laws of physics operating on the state of a human brain, in a human body, in a human environment. I’m prepared to accept that critical structures of a fetal brain are encoded in DNA or something like it. I also accept that this situation is the result of billions of years of mutation and natural selection. But still, no human programmer understands how that “program” works, and no human-created program can be said to duplicate it in its essential respects.

My question is, when someone DOES reproduce it in its essential respects, how would we know? This is a testing question.]

“So, the master program that threatens to take over humanity would require an even more masterful program to debug itself with. But there can’t be one, because THAT program would require a program to debug itself… and so on.”

Nice try, but self-debuggability is not necessary. All you’ve pointed out here is the general truth that self-referential statements lead to infinite regression. So what?

[James’ Reply: Something has to debug it? What will debug it? With humans, it was a couple billion years of mutation and natural election. The result is still buggy, by the way, but we more or less accept those bugs.] 

“The more complex the technology, the more prone to breakdown”

What do you mean by “prone”? Of course it’s true that complexity leads to more potential MODES of failure, but that doesn’t necesarily lead to an increase in the overall probability of failure. If what you claim were actually true then we’d see more airplane crashes (percentagewise) today versus in the 50’s when they were simpler machines. Likewise with a zillion other gadgets that have gotten more reliable as they have also gotten more complex.

[James’ Reply: By prone I mean more likely. You’re right, complexity alone does not necessarily lead to unreliability. It’s just a very powerful dynamic. It’s like an elephant you have to know how to feed and care for. I’m suggesting that the elephant can get so large that the expense of taking care of it exceeds the value of keeping it.

I don’t know if you are a programmer. If so, then perhaps you’ve had the experience of working on a very large and complex project. Perhaps you have experienced large project failures, as I have. If so, then I’m surprised you are so sanguine about the specter of complexity. If not, then for an interesting example see the book Showstopper, which details the project to build Windows NT. Also… consider using a computer. Much of the software I use have annoying behaviors and failures in them that are arguably traceable to complexity, at least in part. Your mileage may vary.

The increase in flight safety is the result of a huge investment in, among other things, ways to reduce complexity. Reduction of cognitive complexity in the cockpit is the subject of a great deal of research. The development of protocols for pilot behavior simplify and harmonize the job. Another vital way complexity is managed is by preferring old complexity to new complexity. This is why you still see 747’s flying around, so many years after they were first designed.

But, yeah, I’m generalizing. I just think it’s a pretty reasonable generalization.] 

MB: I don’t think that you’d consider my radio to have superhuman intelligence, but when I look at it and listen to it, I can’t help thinking that there’s a little man inside who’s doing all the talking.

I think you’re confusing “intelligence” with “rapid access to and presentation of data”. The GPS isn’t coming close to passing the Turing Test to any competent tester. For the device to have intelligence, it would have to be able to provide a reasonable answer to an unconstrained question–”What would you feel like if you lost your father?” “What are seven interesting cultural differences between the British and the Brazilians?” “Why does the porridge bird lay his eggs in the air?” If a question was in some way nonsensical, it would have to be able to explain /why/ the question couldn’t be answered.
_____________________________________________

Absolutely - I was confusing “intelligence” with “rapid access to and presentation of data”! And, pondering this forum thread, I believe that this is one key reason why The Singularity will not be what some predict (the emergence of human-like intelligence in machines).

Sadly, schools frequently teach children to exhibit “rapid access to and presentation of data” (childhood memories of listening to “blah, blah, blah, blah, QUIZ!!!, blah, blah, blah, TEST!!!!!! come to mind. Little did I know that someday “test” would take some of us in new directions )

As intelligence emerges, we are first taught to regurgitate data - to play back canned responses. Little mavericks, buccaneers and free-thinkers generally do not enjoy this, and frequently turn it into a game or look for creative ways to sidestep it. There is some value in it - actually quite a bit of value. You cannot become a Mathematician or a Physicist or a Tester unless you learn your sums. You cannot become a Concert Pianist unless you learn your scales, or an Aviator unless you learn straight-and-level flight. We are called upon to learn at a very basic level, as the foundation for thought and creativity.

But the realm of thought, creativity and intelligence is a whole new dimension - a breakthrough that transcends data access and presentation.

Something new and amazing is happening. I have frequently fallen into the trap of thinking that intelligence is just a very sophisticated level of the ability to make connections between elements of data (and my thinking that the GPS system is “intelligent” fits this somewhat). If this actually worked, I would expect the Web to become “aware”. For your example, I guess I could imagine a very sophisticated program, able to parse “What are seven interesting cultural differences between the British and the Brazilians?”, then go do Google searches for articles and some kind of weird syntactical analysis with a snake-oil-voodoo algorithm, and then list 7 reasonable statements about these differences.

But what would our machine do with: “Why does the porridge bird lay his eggs in the air?”

When will a machine be able to think: “huh”?

OK - compilers have been saying “syntax error”, which is essentially a Geeky version of “huh” for quite some time.

But the “porridge bird” question is a whole new level of “huh”. It is funny. When will a machine laugh? Or cry? Or have compassion?

I once had a car with automatic seat belts, and modern cars have lots of sensors and warning lights. Did that car embrace me because it “cared” about me? Did it “feel pain” when an engine component failed? No - it was designed by humans to do these things. Does a beautiful airplane feel the joy of flight? No - it obeys the laws of physics - and the human pilot may experience joy (or other emotions at other times).

My new guess, based on learning from this forum, is that if there is a Singularity, we will not notice it. It will just be a continuation of the merging of human intelligence with the amplification provided by machines. Anything that can be called Artificial Intelligence will probably be just “the Next Cool Thing that Computers can Do”. Computers will not be able to think in a human way, but a kind of machine-intelligence-illusion will become more and more interesting, and perhaps may reach a point where it is difficult to tell if we are speaking to a person or to a machine.

And I’d imagine that testing such machines will be quite a fascinating, creative calling…

I don’t think that you’d consider my radio to have superhuman intelligence, but when I look at it and listen to it, I can’t help thinking that there’s a little man inside who’s doing all the talking.

I think you’re confusing “intelligence” with “rapid access to and presentation of data”. The GPS isn’t coming close to passing the Turing Test to any competent tester. For the device to have intelligence, it would have to be able to provide a reasonable answer to an unconstrained question–”What would you feel like if you lost your father?” “What are seven interesting cultural differences between the British and the Brazilians?” “Why does the porridge bird lay his eggs in the air?” If a question was in some way nonsensical, it would have to be able to explain /why/ the question couldn’t be answered.

Some people ascribe great intelligence to me because I can remember and recite Monty Python sketches that I memorized 30 years ago. I’m obliged to remind them that this is the kind of intelligence that one might expect from a macaw or a tape recorder. The real kind is something of which I’m capable, at best, only intermittently and occasionally.

[James’ Reply: Sounds cool. But, ouch, I think I never received such a comment. It may be that my anti-spam system deleted it automatically on not receiving the right response. That has happened to a few people. I’m really sorry about that. I wish I could remove that silly thing, but I would get 100 spam comments a day if I did. If you want, you can email me a comment and I’ll post it myself.

Based on your site, you seem like an interesting thinker, so please don’t be too discouraged…]

thanks for modding my comment off your site.

[James’ Reply: I don’t remember that. Are you sure that you left one?] 

I was first impressed by the idea of a technology singularity by Vernor Vinge. One of the greatest feats of human kind has been to explore beyond the possible, into the realms of fantasy and ask reflective questions of ourselves. The fear of technology superseding humans has been with us since time immemorial. Each step of the way (fire, sled, wheel, plough, etc) has brought with it a sense of the mystic. As we become more and more technologically advanced, anything beyond our current state is considered to be mystical. We don’t see it as that, because we delude ourselves that we are the pinnacle of what we can currently achieve. The third of Arthur C. Clarke’s laws states:

“Any sufficiently advanced technology is indistinguishable from magic.”

I for one try every day to wonder at the level we can achieve. The problem is I am rooted in the mire of what I have done. To consider that a singularity approaches, and that we will be transmogrified (Calvin and Hobbes greatest contribution to English language) or even supplanted, is abhorrent. It smacks of a religious belief system that supposes that there is a greater power out there that has some hold of our lives. In the same way as I sit and look up at the Horse Chesnut tree growing in my garden and wonder at it, the same way I wonder at my youngest daughter ability to pick up her PSP, hook into our network and talk with friends, the same way I wonder at the b^%$^*it handed over to me by our development team, the same way that every day unsung heroes struggle against themselves to go out and make a better day, I believe that the concept of the singularity is a reflection of the fear we have of a future we have yet to shape and cannot ever be tested.

Thanks for your insightful, thought provoking and personal leadership over the years. Keep it coming.

[James’ Reply: What beautiful writing. Thanks. I suppose I agree. I just keep thinking back to the days when telephone calls NEVER dropped. At least that aspect of life seemed more dependable.]

Could you clarify what you mean by ‘moreso’?

[James’ Reply: Yes, all of our ambitions and appetites and flaws become amplified. Being amplified, they overlap and interact. Amplification in a closed and interconnected space leads to interesting and unpredicted failure modes.]