This is the same logic people apply to God being incomprehensible.

Are you suggesting that if such a thing can be built, its word should be gospel, even if it is impossible for us to understand the logic behind it?

I don't subscribe to this. Logic is logic. You don't need a new paradigm of mind to explore all conclusions that exist. If something cannot be explained and comprehended, transmitted from one sentient mind to another, then it didn't make sense in the first place.

And you might bring up some of the stuff AI has done in material science as an example of it doing things human thinking cannot. But that's not some new kind of thinking. Once the molecular or material structure was found, humans have been perfectly capable of comprehending it.

All it's doing, is exploring the conclusions that exist, faster. And when it comes to societal challenges, I don't think it's going to find some win-win solution we just haven't thought of. That's a level of optimism I would consider insane.

The problem is that we absolutely can build a sustainable society on our own. We've had the blueprints forever, the Romans worked this out centuries ago, the problem is that there's always some power seeking prick who messes it up. So we gave up trying to build a fair society and just went with feudalism and then capitalism instead.

That's fine, I'm just correcting the misrepresentation of the view that was in the headline.

Even if it is, I don't see what it's going to conclude that we haven't already.

If we do build "the AI that will save us" it's just going to tell us "in order to ensure your existence as a species, take care of the planet and each other" and I really, really, can't picture a scenario where we actually listen.

It's not that the output of an ASI would be incomprehensible but that as humans we're simply incapable of predicting what it would do/say because we're not it. We're incapable of even imagining how convincing of an argument a system like this could make.

We're incapable of even imagining how convincing of an argument a system like this could make.

Vaguely gestures at all of sci-fi, depicting the full spectrum of artificial sentience, from funny comedic-relief idiot, to literal god.

What exactly do you mean by that?

The Romans had a slave economy. I don't really think that counts as sustainable or even having worked it out.

The issue isn’t whether we can imagine a smarter entity - obviously we can, as we do in sci-fi. But what we imagine are just results of human intelligence. They’re always bounded by our own cognitive limits. We picture a smarter person, not something categorically beyond us.

The real concept behind Artificial Superintelligence is that it wouldn’t just be smarter in the way Einstein was smarter than average - it would be to us what we are to ants. Or less generously, what we are to bacteria. We can observe bacteria under a microscope, study their behavior, even manipulate them - and they have no concept of what we are, or that we even exist. That’s the kind of intelligence gap we're talking about.

Imagine trying to argue against a perfect proof. Take something as basic as 1 + 1 = 2. Now imagine an argument for something much more complex - like a definitive answer to climate change, or consciousness, or free will - delivered with the same kind of clarity and irrefutability. That’s the kind of persuasive power we're dealing with. Not charisma. Not rhetoric. Not "debating skills." But precision of thought orders of magnitude beyond our own.

The fact that we think we can comprehend what this would be like is part of the limitation. Just like a five-year-old thinks they understand what it means to be an adult - until they grow up and realize they had no idea.

No way, do you want to tell me that spftware which is tailored and trained by megacorps, will not save our covilisation?!

Logic is logic. There is no "advanced" logic that somehow allows you to decipher aspects of reality you otherwise could not. Humanity has yet to encounter anything that cannot be consistently explained in more and more detail, as we investigate it further.

We can and do answer complex questions. That human society is too disorganized to disseminate the answers we do have, and act on them at scale, isn't going to be changed by explaining the same thing slightly better.

Imagine trying to argue against a perfect proof. Take something as basic as 1 + 1 = 2. Now imagine an argument for something much more complex - like a definitive answer to climate change, or consciousness, or free will - delivered with the same kind of clarity and irrefutability.

Absolutely nothing about humans makes me think we are incapable of finding such answers on our own. And if we are genuinely incapable of developing a definitive answer on something, I'm more inclined to believe there isn't one, than assume that we are simply too "small-minded" to find an answer that is obvious to the hypothetical superintelligence.

But precision of thought orders of magnitude beyond our own.

This is just the "god doesn't need to make sense to us, his thoughts are beyond our comprehension" -argument, again.

Just like a five-year-old thinks they understand what it means to be an adult - until they grow up and realize they had no idea.

They don't know, because we don't tell them. Children in adverse conditions are perfectly capable of understanding the realities of survival.

You are using the fact that there are things we don't understand, yet, as if it were proof that there are things we can't understand, ever. Or eventually figure out on our own.

That non-sentients cannot comprehend sentience (ants and humans) has absolutely no relevance on whether sentients are able to comprehend other sentients (humans and machine intelligences).

I think machine thinking, in contrast to the human mind, will just be a faster processor of logic.

There is absolutely nothing stopping the weakest modern CPU from running the exact same code as the fastest modern CPU. The only difference will be the rate at which the work is completed.

Beginning by insulting your opponent isn’t exactly the best way to ensure they’ll finish reading your message.

You have a great day.

Even if it is, I don't see what it's going to conclude that we haven't already.

If we do build "the AI that will save us" it's just going to tell us "in order to ensure your existence as a species, take care of the planet and each other" and I really, really, can't picture a scenario where we actually listen.

The problem is that we absolutely can build a sustainable society on our own. We've had the blueprints forever, the Romans worked this out centuries ago, the problem is that there's always some power seeking prick who messes it up. So we gave up trying to build a fair society and just went with feudalism and then capitalism instead.

Fair.

I've removed it, and I'm sorry.

they had reusable poop sponges what more do you want??

Would they though? I think if anything most industries and economies would be booming, more disposable income results in more people buying stuff. This results in more profitable businesses and thus more taxes are collected. More taxes being available to the government means better public services.

Even the banks would benefit, loans would be more stable since the delinquency rate would be much lower if everyone had better pay.

The only people who would lose out would be the idiot day traders who rely on uncertainty and quite a lot of luck in order to make any money. In a more stable global economy businesses would be guaranteed to make money and so there would be no cheap deals that could be made.