"By 2029, computers will be able to understand our language, learn from experience and outsmart even the most intelligent humans, according to Google's director of engineering Ray Kurzweil.
Computers are on the threshold of reading and understanding the semantic content of a language, but not quite at human levels. But since they can read a million times more material than humans they can make up for that with quantity. So IBM's Watson is a pretty weak reader on each page, but it read the 200m pages of Wikipedia. And basically what I'm doing at Google is to try to go beyond what Watson could do. To do it at Google scale. Which is to say to have the computer read tens of billions of pages. Watson doesn't understand the implications of what it's reading. It's doing a sort of pattern matching. It doesn't understand that if John sold his red Volvo to Mary that involves a transaction or possession and ownership being transferred. It doesn't understand that kind of information and so we are going to actually encode that, really try to teach it to understand the meaning of what these documents are saying.
You could make a pretty good argument that human history is mostly a study of the failure of inheritance based leadership. Given that good leaders/designers historically have generally produced garbage, and most of the population has produced garbage, and no one really seems to have come up with an effective society aka utopia, its most likely that the children of humanity will continue to disappoint and entropy will continue to override civilization.
A world of AI might look like the Wehrmacht marching down the streets of Paris; it's statistically more likely to resemble the worst parts of rural Appalachia or a hippie commune or an Amish community.
I'm not quite so down on Humanity. I don't think that our biological drive to protect our lineage is necessarily a bad thing (although it has, as so many other things, led us to disastrous extremes). The desire to protect ourselves, our family and ensure a future for our descendants has taken us from a small tribe of savannah-wandering apes to becoming the apex predator of the entire world (at least on the macroscopic level). And despite this innate wariness of "the other" we have learned to work together not only within our own species but with entirely other species (albeit ensuring we remain the dominant member of those partnerships.
But there is a drive built into all of us Earthlings that makes us - wisely, as far as billions of years of evolution go - put our interests ahead of any other species. This tendency will not change just because the next "species" we meet may be as intelligent as us or because we had a hand in their creation; ultimately, our genes will work to ensure that their survival takes per-eminence. Intellectually, we can work to overcome these drives to some degree but in our gut we cannot help but look askance at the Other; our ability to work in equal partnership with another sentience (or even to accept them as sentients) would be as much a celebration of our minds as the creation of those sentients themselves.
More likely we will achieve an uneasy peace between our two factions, hopefully aided by the fact that we have so little in common as to minimize reasons for conflict.
Actually, it's not quite as simple. With humans, something new has entered the evolution: We are now able to pass not only genetic information to our offspring, but also to pass information from one brain to another. That is, humans not only have genes, they also have memes. And the memes are no less selfish than the genes. When people are willing to die for their ideals, and taking themselves out of the gene pool by doing so, it's a case of the memes having won over the genes.
Therefore whether we will accept the AI depends very much on how similar it is to our mind, that is, how well it will be able to carry and pass on our memes. Of course we will not think that way. We will notice that the AI understands us and we understand the AI. We will be able to relate to the AI, to be friends with it, to share thoughts with it. We will be able to accept the AI as long as we have the impression that it is "just like us". Maybe more intelligent, and of course not having certain experiences (and having certain others that we don't have), but basically not entirely different from us.
If we manage to build such an AI, I guess over time it will get widespread acceptance, and I can even imagine that many people would accept the idea that they will eventually replace us (as long as that replacement doesn't happen in a violent way). However if the thinking of the AI remains alien to us, then it certainly won't get our sympathy, and we will always consider it something fundamentally different and potentially dangerous that we have to protect ourselves against.
"Maybe more intelligent, and of course not having certain experiences (and having certain others that we don't have), but basically not entirely different from us."
Picture 4chan /b/ distilled down to a hive mind, then imagine something a billion times weirder than /b/. Something that makes /b/ look like a bunch of conformist suburban neocon soccer moms in comparison.
"However if the thinking of the AI remains alien to us, then it certainly won't get our sympathy, and we will always consider it something fundamentally different and potentially dangerous that we have to protect ourselves against."
Told you so. Totally 4chan /b/ in a nutshell. Again, imagine something a billion times weirder yet.
By analogy we don't even have to leave computers and the internet to find the "other", now imagine something not even based on the same biological hardware, not even the same species.
I don't think there is any inherent reason to conclude there will be any cultural common ground, at all. Maybe the golden rule, maybe, but not much more.
I think the inherent reason will be that we built it, and we did so explicitly trying to build something "like us".
A world of AI [...]; it's statistically more likely to resemble [...] an Amish community.
I really cannot imagine an AI without modern technology. Indeed, I'm pretty sure it could not exist.
I'm writing about their self imposed limit. I probably will fail to do them justice, but I gather its something like "its irreverent to use post 1800 technology". Combined with a lifestyle and social order most would consider nostalgic.
I could totally imagine an AI with peculiar beliefs such that it finds numerical integration of equations to be a moral and ethical and dare I suggest ... religious? abomination. Or it finds a society based on source route bridging to be socially superior to BGP, again for its own peculiar self designed and imposed moral/ethical/religious viewpoints.