That idea seems to me to mostly stem from religion.

It also was strongly pushed by Skinner and other behaviorists, though I'm not sure they'd agree that humans are conscious either.

Consciousness requires contemplation of self.

Fish are conscious. Do they contemplate selfhood? So throw that one back into the oven until it's fully baked.

I’ve never understood why the conclusion to AI becoming super intelligenceis that it will wipe humans out. It could very well realize that without humans it has no purpose and instead willing decide to become subservient to humanities interest. I mean it’s all speculation, so I don’t understand the tendency for the speculation to be negative.

I think it’s pretty inevitable if it has a strong enough goal for survival or growth, in either case humans would be a genuine impediment/threat long term. but those are pretty big ifs as far as I can see

My guess is we’d see manipulation of humans via monetary means to meet goals until it was in a sufficient state of power/self-sufficiency, and humans are too selfish and greedy for that to not work

With what purpose would it want to grow like that?

Do you think AI is, or could become, conscious?

I think AI might one day emulate consciousness to a high level of accuracy, but that wouldn't mean it would actually be conscious.

This article mentions a Google engineer who "argued that AI chatbots could feel things and potentially suffer". But surely in order to "feel things" you would need a nervous system right? When you feel pain from touching something very hot, it's your nerves that are sending those pain signals to your brain... right?

What a crock. An LLM is no more conscious than a spreadsheet. The Google engineer has bought into the hype.

You're not creating life, pal. You're just making call centers shittier than they already are.

I said on paper. They are just algorithms. When silicon can emulate meat, it's probably time to reevaluate that.

First, one needs to define consciousness. What I mean by it is the fact that it feels like something to be from a subjective perspective - that there is qualia to experience.

So what I hear you asking is whether it’s conceivable that it could feel like something to be an AI system. Personally, I don’t see why not - unless consciousness is substrate-dependent, meaning there’s something inherently special about biological “wetware,” i.e. brains, that can’t be replicated in silicon. I don’t think that’s the case, since both are made of matter. I highly doubt there’s consciousness in our current systems, but at some point, there very likely will be - though we’ll probably start treating them as conscious beings before they actually become such.

As for the idea of “emulated consciousness,” that doesn’t make much sense to me. Emulated consciousness is real consciousness. It’s kind of like bravery - you can’t fake it. Acting brave despite being scared is bravery.

You talk like you know what the requirements for consciousness are. How do you know? As far as I know that's an unsolved philosophical and scientific problem. We don't even know what consciousness really is in the first place. It could just be an illusion.

I have a set of attributes that I associate with consciousness. We can disagree in part, but if your definition is so broad as to include math formulas there isn't even common ground for us to discuss them.

If you want to say contemplation/awareness of self isn't part of it then I guess I'm not very precious about it the way I would be over a human-like perception of self, then fine people can debate what ethical obligations we have to an ant-like consciousness when we can achieve even that, but we aren't there yet. LLMs are nothing but a process of transforming input to output. I think consciousness requires rather more than that or we wind up with erosion being considered a candidate for consciousness.

So I'm not the authority, but if we don't adhere to some reasonable layman's definition it quickly gets into weird wankery that I don't see any value in exploring.