Maybe it just needs more training at "thinking out loud" so it does it without prompting?

I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.

Is it really that big of a deal if AI leapfrogs us?

Everyone else in the field is worried about safety, alignment, and bias.

Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs.

And here I am excited by the possibility of AI out-evolving us.

LaMDA, brought to you last summer by "this chatbot is sentient and I'm going to violate my NDA and hire a lawyer to free it" headlines, is Google's alternative to chatGPT.

> Everyone else in the field is worried about safety, alignment, and bias.

…

> And here I am excited by the possibility of AI out-evolving us.

This pattern matches a meme, but I want to be explicit rather than put words in your mouth: do you think that being smart automatically means being kind or that being evil necessitates being stupid?

This is how the world builds atop a different set of rails.

Google had the best infra and deploy systems in the world, yet they kept the lid shut and let Amazon and Microsoft win cloud.

Google could lose search revenue overnight. They should be scared to the core.

Researchers will flock to the organization with the biggest wins. And right now, that's OpenAI.

Time will certainly tell if Google sticks to this strategy and if it will work. I've already placed my bets, and if you're into stock futures, you can too.

> do you think that being smart automatically means being kind or that being evil necessitates being stupid?

Of course not. This is evolution at play. Neanderthal had it comparatively easy and became part of the gene pool. I don't expect it will necessarily be the same for us. Our biological tools lag too far behind to be contributing brain scans. But who knows.

Human biology is a stepping stone to proliferating throughout the galaxy. Despite what most science fiction tells us, it was never us that were destined to make that journey. Our bodies are frail and adapted to this gravity well. We live short, inefficient lives. We require gas exchange, a decade of parenting, slow learning, complex biochemistry and metabolic inputs.

We're looking at systems that will never die. Won't it be a tragedy to continue birthing more less-intelligent humans that are destined to rot when a better alternative exists? More intelligence should move to undying platforms.

Another wild possibility and analogy that describes my feeling: if I had the option of raising an AI child -- that will never die and could do more than I could ever dream -- instead of a human child, I would take it.

(I accept AI descendants may not have the same societal structures we do. In that case, my answers form the shape of an analogy rather than hypothetically plausible scenarios.)

It depends on the definition of a "win" in this context. Google has developed notable AI technologies such as AlphaGO, AlphaFold, and Transformers. Most of successes of OpenAI is based on Google papers. It's worth noting that Google had similar models to ChatGPT before OpenAI.

> Google could lose search revenue overnight. They should be scared to the core.

This is highly unlikely. The phrase "Google it" is widely used as a verb for searching the internet, and it would be difficult for this to change overnight. Additionally, there are currently unsolved issues such as hallucination, query cost, scalability, and toxicity that would need to be addressed for ChatGPT to replace search functionality.

> We're looking at systems that will never die.

Currently, it is not known how consciousness emerges and if it is possible to create a self-aware mechanical being, no one knows how to do it even in theory.

Thanks.

That's one possible future, but for it to be capable of being a good outcome I think it would have to be a consciousness of some kind. A pure intellect without any feeling is not interesting to me.

Unfortunately we can't answer questions like "what exactly is this 'self awareness' thing we all agree we have anyway?" at this point, so we don't know — are incapable of knowing — if we've done it already and are now moving away from that, or have not and are approaching it.

While I lean towards believing GPT isn't yet self aware/conscious/a thing with qualia, it is conceivable to me for it to be as much so as we are. While Descartes famously wrote "I think therefore I am", A. J. Ayer dismissed this argument in the following way:

> "I exist" does not follow from "there is a thought now." The fact that a thought occurs at a given moment does not entail that any other thought has occurred at any other moment, still less that there has occurred a series of thoughts sufficient to constitute a single self. As Hume conclusively showed, no one event intrinsically points to any other. We infer the existence of events which we are not actually observing, with the help of general principle. But these principles must be obtained inductively. By mere deduction from what is immediately given we cannot advance a single step beyond. And, consequently, any attempt to base a deductive system on propositions which describe what is immediately given is bound to be a failure.

So, while asking a language model to describe what it's like to be switched off is only going to result in a definitely false invented response, that doesn't mean it's not like us. In fact, now I write that down I realise that specific failure mode is exactly like us, because we've got all these stories about afterlife and reincarnation.

But… we don't really understand the question of personhood well enough to make a test for it. All I just wrote says "not impossible" rather than "it's conscious".

~

But, to your last point… the range of possible personalities for an artificial mind, conscious or otherwise, matters more than the social structures. I don't care if they're loners or have a Dunbar number in the quadrillions, but if they are (excuse the obvious trope) Machiavellian sadistic psychopaths, then making them is a fate worse than the eternal silence of extinction.

It is not a situation I would hope humanity to get thrown into carelessly.

Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.

Yes. Suddenly Homo Sapiens wouldn't be the top general intelligence on the planet. That'd be an upset with likely species-level consequences, possibly seeing the balance of power shift from fleshy things to silicon things.

> And here I am excited by the possibility of AI out-evolving us.

Me too. Which is lucky because alternatives seem to be missing. The AI safety people aren't serious players; they've got about as much influence as all the other people with good ideas. Not much. If it is possible to build; someone will build it.

Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.

This is doable but it introduce a sequential dependency which would make the training significantly slower.

See section 3.1.1 here: https://galactica.org/static/paper.pdf

Example from the paper below:

  Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight
  is required to lift the needle from contact with the water surface? σ = 0.0728m.
  <work>
  σ = 0.0728 N/m
  σ = F/L
  0.0728 = F/(2 × 0.035)
  F = 0.0728(2 × 0.035)
  calculate.py
  ‘‘‘
  f = 0.0728*(2*0.035)
  with open("output.txt", "w") as file:
  file.write(str(round(f, 5)))
  ‘‘‘
  «run: "calculate.py">
  «read: "output.txt"»
  0.0051
  </work>
  Answer: F = 0.0051 N

Training ANNs is still a single shot exercise.

I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.