> There are far far more dollars available to people that are on the "AI Safety" bandwagon than to those pushing back against it.

> The idea that the Upton Sinclair effect is the source of pushback against AI Safety zealotry, is getting things largely backwards AFAICT.

> Folks that are stressing the importance of studying the impact of concentrated corporate power, or the risk of profit-driven AI deployment, and so forth are receiving very little financial support.

IMO your comment doesn't substantively address michael_nielsen's comment, but I might be wrong. The following is how I understand your exchange with michael_nielsen.

The two of you are talking about three sets of people:

  Let A be AI notkilleveryoneism people.
  Let B be AI capabilities developers/supporters.
  Let C be people concerned with regulatory capture and centralization by AI firms.

  A and B are disjoint.
  A and C have some overlap.
  B and C have considerable overlap.

michael_nielsen is suggesting that the people of B are refusing to take AI risk seriously because they are excited about profiting from AI capabilities and its funding. (eg, a senior research engineer at OpenAI who makes $350k/year might be inclined to ignore AIXR and the same with a VC who has a portfolio full of AI companies)

And then you are pointing out that people of C are getting less money to investigate AI centralization than people of A are getting to investigate/propagandize AI notkilleveryoneism.

So, your claim is probably true, but it doesn't rebut what michael_nielsen suggested.

And I believe it's also critical to keep in mind that the actual funding is like this:

capabilities development >>>>>>>>>> ai notkilleveryoneism > ai centralization investigation

I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction. So I don't think it's a good argument. And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless.

On your last point, I do think it's important to note, and reflect carefully on, the extremely high overlap between those funding ai notkilleveryoneism and those funding capabilities development.

(this discussion is quite nuanced so I apologize in advance for any uncharitable interpretations that I may make.)

> I'm not really trying to rebut Michael's argument -- I think it's true, to an extent, some of the time. But I think it's more true more of the time in the reverse direction.

I understand you to be saying:

Michael: Pro AI capabilities people are ignoring AIXR ideas because they are very excited about benefiting from (the funding of) future AI systems.

Reverse Direction: ainotkilleveryoneism people are ignoring AIXR ideas because they are very excited about benefiting from the funding of AI safety organizations.

And that (RD) is more frequently true than (M).

IMO both (RD) and (M) are true in many cases. IME it seems like (M) is true more often. But I haven't tried to gather any data and I wouldn't be surprised if it turned out to actually be the other way.

> So I don't think it's a good argument.

I might be misunderstanding you here because I don't see Michael making an argument at all. I just see him making the assertion (M).

> And more importantly, I think it fails to properly grapple with the ideas, instead using an ad hominem approach to discarding them somewhat thoughtless.

I am ambivalent toward this point. On one hand Michael is just making a straightforward (possibly false) empirical claim about the minds of certain people (specifically, a claim of the form: these people are doing X because of Y). It might really be the case that people are failing to grapple with AIXR ideas because they are so excited about benefiting from future AI tech, and if it were, then it seems like the sort of thing that it would be good to point out.

But OTOH he doesn't produce an argument against the claim "AIXR is just marketing hype." which is unfair to someone who has genuinely come to that conclusion via careful deliberation.

> On your last point, I do think it's important to note, and reflect carefully on, the extremely high overlap between those funding ai notkilleveryoneism and those funding capabilities development.

Thanks for pointing this out. Indeed, why are people who profess that AI has a not insignificant chance of killing everyone also starting companies that do AI capabilities development? Maybe they don't believe what they say and are just trying to get exclusive control of future AI technology. IMO there is a significant chance that some parties are doing just that. But even if that is true, then it might still be the case that ASI is an XR.

I mostly agree with this. Certainly the last line!

I've been reflecting on Jeremy's comments, though, and agree on many things with him. It's unfortunately hard to tease apart the hard corporate push for open source AI (most notably from Meta, but also many other companies) from more principled thinking about it, which he is doing. I agree with many of his conclusions, and disagree with some, but appreciate that he's thinking carefully, and that, of course, he may well be right, and I may be wrong.

Thank you Michael. I'm not even sure I disagree with you on many things -- I think things are very complicated and nuanced and am skeptical of people that hold overly strong opinions about such things, so I try not to be such a person myself!

When I see one side of an AI safety argument being (IMO) straw-manned, I tend to push back against it. That doesn't mean however that I disagree.

FWIW, on AI/bio, my current view is that it's probably easier to harden the facilities and resources required for bio-weapon development, compared to hardening the compute capability and information availability. (My wife is studying virology at the moment so I'm very aware of how accessible this information is.)

Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing.

> We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions!

If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused.

> xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case".

By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth?

> All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning.

Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development.

Premise 1. I value my own survival over just about anything else.

Premise 2. If an existential catastrophe occurs, then I will die.

Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.

Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.

Premise 3 is where the problem is, of course.

We have no idea how to build AGI. We know LLMs won't be it.

Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.

Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.

Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.

This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".

Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.

LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.

Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.

Thank you for the thoughtful reply.

> Premise 3 is where the problem is, of course.

I don't believe premise 3 is a problem exactly, but I do believe that it is a non-trivial challenge to determine whether or not it is true.

> We have no idea how to build AGI. We know LLMs won't be it.

> Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence.

> Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from.

> This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered".

It looks to me that you are making the following argument:

  Premise G1. Humans do not currently know how to build AGI.
  Premise G2. It might be impossible to build ASI.
  Premise G3. It is unclear how likely an ASI is to cause an existential catastrophe.
  Conclusion. There is not a significant chance of catastrophe from ASI.

I believe that argument is about an important point (chance of AI catastrophe) and that it is a pretty good argument. But the original premise 3 says, "If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.", so AFAICT your argument doesn't substantively address it. (ie, your argument's conclusion doesn't tell me anything about whether or not premise 3 is true)

I apologize if I have misunderstood your point.

> Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI.

We may be using the word "alignment" slightly differently. By "alignment" I just meant getting the algorithmic system to have precisely the goal that its human programmers want it to have. I would call, for example, RLHF a "tool" for trying to achieve alignment.

How do you want to use the terms "alignment" and "alignment tool" going forward in the discussion?

> Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is.

I agree that there are other plausible sources of catastrophe for humans, to name a few others: asteroids, supervolcanoes and population collapse.

I understand you to be making a new point now, but I just want to state that I do not believe the existence of other plausible existential threats to be a rebuttal of premise 3.

> LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop.

I agree that LLMs may plausibly cause significant harm in the short term via disinformation and unemployment.

And again, I understand you to be making a new point, but I just want to state that I do not believe the plausibility of such LLM harms is a rebuttal against premise 3.

> Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.

I'm not sure who you mean by "we" here, so I'm not sure if your claim about them is true or not.

> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything.

This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work.

Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

> Neural networks are not new

I agree. The McCullough-Pitts paper was published in 1943.

> they're just mathematical systems.

What do you mean by "mathematical system"? AFAIK the GPT4 model is literally a computer program.

> LLMs don't think. At all.

This is the same assertion that OP made and I'm still confused as to how anyone could be certain of its truth given that no one actually knows what is going on inside of the GPT4 program.

> They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

Is that an argument for the claim "LLMs don't think."? It doesn't seem like it to me, but maybe I'm mistaken.

Not new, but we don't understand how they work at the large scale.

I don't think reductionistic arguments hold much water. Sure, neural networks are just matrix multiplication. In the same way that a brain is just a bunch of cells. Understanding the basic building blocks doesn't mean understanding the whole.

We can always say that LLMs don't think if we define "think" as using a biological brain, but the fact is that they generate outputs that from the human perspective, can only plausibly be generated via reasoning. So they, at the very least, have processes that can functionally achieve the same goal as reasoning. The "stochastic parrot" metaphor, while apt in its day, has proven obsolete with pretty much all the examples of things that LLMs "could not do" in early papers being actually doable with the likes of GPT-4; so arguments against the possibility of LLMs reasoning look like constant moving of the goalposts.

> and they're just mathematical systems

Obvious question: can Prolog do reasoning?

If your definition of reasoning excludes Prolog, then... I'm not sure what to say!

> With AI, there still seems to be a lot of hand-waving between where we are now and "AGI".

> I am more than prepared to admit that I may not be seeing (for various reasons) the evidence that this is near/possible - but I would also claim that nobody is convincingly showing any either.

If I understand you correctly, then (1) you doubt that AGI systems are possible and (2) even if they are possible, you believe that humans are still very far away from developing one.

The following is an argument for the possibility of AGI systems.

  Premise 1: Human brains are generally intelligent.
  Premise 2: If humans brains are generally intelligent, then software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.
  Conclusion: Software simulations of human brains at the level of inter-neuron dynamics are generally intelligent.

(fyi I believe there is an ~82% chance humans will develop an AGI within the next 30 years.)

For info: I don't believe (1), I do believe (2) although not that strongly - it's more likely to be a leap than a gradient, I suspect - I simply don't see anything right now that convinces me it's just over the next hill.

Your conclusion... maybe, yes - I don't think we're anywhere near a simulation approach with sufficient fidelity however. Also 82% is very specific!

> For info: I don't believe (1), I do believe (2) although not that strongly

Thanks for clarifying. Do you believe there is a better than 20% chance that humans will develop AGI in the next 30 years?

> I simply don't see anything right now that convinces me it's just over the next hill.

These are the reasons that I believe we are close to developing an AGI system.

  (1) Many smart people are working on capabilities.
  (2) Many investment dollars will flow into AI development in the near future.
  (3) Many impressive AI systems have recently been developed: Meta's CICERO, OpenAI's GPT4, DeepMind's AlphaGo.
  (4) Hardware will continue to improve.
  (5) LLM performance significantly improved as data volume and training time increased.
  (6) Humans have built other complex artefacts without good theories of the artefact, including: operating systems, airplanes, beer.

These achievements are impressive, but I'd rather not overhype it.

* GPT-4 still hallucinates as hell, can't do math, fails as basic logic, can't handle really big contexts, hard to update, easy to jailbreak etc.

* AlphaGo was defeated by a Go amateur with a help of another AI.

* AlphaStar basically failed to achieve real goals, was trivial to cheese even after defeating high-ranked players sometimes.

All these problems are architectural, you can't just throw more money and GPUs at it like GPT-2-3-4.

It's hard to predict at this point. We may get to AGI anywhere from 5 years to 100 years.

I don't think these reasons are very persuasive, as everything but 5 has been true at different times in the past. Obviously it's much more people, more dollars, and more impressive systems (but slower hardware progress), but I hope you see what I'm getting at.

And of course there's differences in what someone considers to be soon. Many AI x-risk believers think there's a ~50% chance of AGI before 2031 (https://www.metaculus.com/questions/5121/date-of-artificial-...) (I've heard this prediction site's userbase tends towards futurists/techno-optimists/AI x-riskers). I would consider that soon, I wouldn't consider 2054 soon.

Also (3) that AGI in practice will necessarily pose any danger to humans is doubtful. After all Earth has billions of human level intelligence and nearly all of them are useless and if they are even mildly dangerous it's rather due to their numbers and disgusting biology than intelligence.