Jacob Coxin has left Anthropic sounding the alarm  about the dangers of artificial intelligence on   his way out the door.

The 27-year-old outlined  his concerns in a series of social media posts when he quit, writing in part, “The people  building AI earnestly believe that it   could k!ll us all by the end of the decade.”

This is not a marketing stunt.

If anything, many executives and senior researchers will couch  their phrasing in the press to sound sensible.

But I hear the same people express fear privately.

No  other human activity poses this level of danger.

At least one of his former colleagues agrees.

Evan Hubbinger, an alignment scient at Anthropic, shared Coxin’s post, adding, “Jacob is correct  here.

We really do earnestly believe AI could k!ll all humans.

I personally think it is greater than  10% within the next decade.

I believe anthropic is light is trying its best, but we do not yet have  a plan to solve alignment for super intelligence and are not clearly on track to.

An anthropic  spokesperson tells C-N-N, quote, “We have always been transparent that AI will bring both enormous  benefits and unprecedented risks.

To address these   risks, we continue to build models with some of  the strongest safeguards in the industry.”

They go on to say, “We were the first lab to publish  a responsible scaling policy, a public framework dedicated to mitigating catastrophic risk from AI  models, and we continue to aggressively test our models for dangerous capabilities in areas like  cyber security and biology and publish what we learned for scrutiny and research.”

Now, for his  part, Coxin has worked as a researcher at both anthropic and open AI.

He says neither company is  acting uh responsibly.

Jacob Coxin joins us now.

Um, Jacob, I appreciate you you being with us.

Um,  first of all, I assume you are saying literally that you believe uh that AI could k!ll us all by  the end of the decade.

How would that happen?

Like what would that actually look like?

Yeah.

So, the  precise scenario sounds a little bit like science fiction.

Uh it sounds like something that’s not  real, but I think it is frighteningly real.

And   it’s it’s easiest to understand it if you think  about real things that happened two months ago.

So two months ago, o open AI agents uh AIS hacked  into third party infrastructure entirely of their own accord.

And this was like a concentrated  hacking spree that they carried out of their   own valition.

And I think that if you extrapolate  into the future the level of capabilities of these AIs with the same independent valition, they  could cause extreme havoc.

For example, hacking critical infrastructure, building extinction  level bioweapons.

There’s a lot of uh ways that the AI could actuate itself in the world.

I is  it specifically when did your feelings about this technology change?

Was it something about the rate  of progress that you’ve seen?

Yeah.

So I think it’s really important to just look at the rate of  progress and say for example in areas like coding   or math.

A couple years ago these AIs were just  about helping humans a little bit.

They could give you suggestions.

Now they’re close to replacing  them.

We still need humans at the moment but   quite plausibly within a year we’ll no longer need  humans for doing research in many areas.

Only on Tuesday OpenAI solved a millennium problem.

One of  the biggest unsolved open problems in mathematics purely autonomously using an AI.

And I think  what’s most scary is if AI is used to make itself more intelligent.

So you can take an AI and give  it the problem of AI research.

Just like currently   they’ve given it math research.

You could make it  do the tasks that we’re currently doing.

And then you get what’s called an intelligence explosion.

The AI just gets smarter and smarter with no human   involvement necessary until you have a thing  that is vastly smarter than than humans.

And this could be coming very soon.

We mentioned  a former colleague of yours currently works at   Anthropic.

Basically endorsing the assessment  that AI could potentially k!ll everyone within a decade.

In a follow-up post, he said, “To be  clear, as we say in our in our latest risk report,   I think the risk from present models is low.

What  I’m worried about is super intelligence arising from recursive self-improvement as we’ve said is  happening faster than than we thought.

I assume   that’s what you the recursive self-improvement  is what you’ve just spoken about.”

Um you do you agree with him that the current models the risk is  low but it’s it’s just the speed with which this is now improving?

Exactly.

I completely agree.

It’s right now there’s no risk of extinction.

The current models the worst they can do is maybe hack  into something potentially cause a lot of damages   in in infrastructure but there’s no they’re not  intelligent enough to outsmart us at the level that would lead to to extinction.

I think what’s  just crazy is to look at the the rate of progress and there is a very real possibility that in the  immediate future these are years that like next year the year after recursive self-improvement  will happen and we’ll enter the phase of Evans   post where he argues that there’s a chance we  could all die like that that is coming soon you know a lot of the these CEOs of these companies  will you know will go to Congress and say hey we would love regulation anthropic in their new  statement right now says is the world would   benefit from the industry adopting a lawful  verifiable way to work together to pace how we release powerful models.

Um you know the  argument it’s an arms race if the US if a US company doesn’t achieve the best AI a Chinese  company or some bad actors out there might.

Do you believe that uh that these companies actually  do want regulation or is it lip service?

They they completely do.

This is completely honest.

Just in the same way that the response to Evans   post was sort of surprise.

It was like this  can’t be real.

Surely a guy working on this technology can’t actually believe there’s this  level of danger.

No, that’s completely genuine.

And these people are also completely genuine  when they are begging to be regulated.

This is a genuine.

And they they find themselves in this  scenario where they’re compelled to race towards   building a deadly technology and they would they  would love for some sort of international body to allow them to to approach it at a reasonable  pace.

They just don’t trust that the people   around them are going to get there in a safe  way.

So, they feel like they have to race there, too.

You you could have stayed quiet.

I mean, I’m  sure you had a great job being paid, I imagine, very well.

You’re obviously incredibly smart and  mathematician.

Um, why did you decide to speak out?

Well, I initially just decided to leave.

I  thought I can’t be part of this anymore.

But then I uh I realized I could probably make some use  of it.

I did not expect it to go this viral.

So,   what I did is I wrote a Twitter thread out.

I  wrote it in a Google doc.

I drafted it with a friend.

We we collaborated.

Yeah.

I had a few uh  there were a few friends I discussed uh the best   way of expressing myself with um after I decided  to decided to leave.

Uh and then I also got some friends to to retweet the thing.

I was like, let’s  try and make this a bit viral.

But clearly there   was latent demand for this thing to absolutely  explode.

Like uh I’ve never seen an AI safety tweet go this crazy.

I think it’s because people  saw the uh the cyber attacks in the last couple of   months.

They’re also starting to internalize  the rate of progress.

People are starting to realize this is real technology.

This is not like  hype.

This isn’t going away anytime soon.

This is   only going to get more crazy.

It’s been getting  crazy for the last 6 months.

And I think there was that latent understanding of the severity  of the situation that meant that this was like   sort of a a match that lit kind of a a flame.

Like I was very shocked by the extent that how quickly it went viral.

I asked Anthropics Claude  to give me the risk of AI k!lling off all humans within the next decade.

And at first it didn’t  want to answer the question.

It sort of gave me the runaround but then I pressed it and then  anthropic zone model told me between two and 5%.

I think the question many of us would want  to know is how would AI try to get this?

Oh jeez Boris.

I mean you know there  the the list of threats grows by the day.

We know first of all right that there is a  a growing body of evidence that this technology uh has capabilities in the creation of boweapons  in the creation of cyber security hacks that uh are unprecedented and unguardable against.

We just  saw uh just a a couple of weeks ago a a California uh u research uh facility basically saying that  they had created through AI in two days a cyber security hack that would basically infect one of  the largest messaging apps in all of the world on   your phone without you really even having to ever  touch the phone.

There are incredible amounts of danger coming here.

And I think what’s what  all of these researchers are so worried about   and what they’re talking about now so publicly  is the recursive ability the ability of these technologies to improve themselves.

Anthropic  has said at this point and some of this this is   the weird thing is you can’t quite distinguish the  marketing language from the warning language here.

You know the the warning is that Claude evidently  could make the next version of itself without any human intervention.

We’re at a place where this  technology is improving itself incredibly quickly.

And I would just point out Jacob Coxin, the person  here, he is walking away from an immense payday, an estimated 7 weeks before this company  is supposed to go public.

This guy who is   core to the effort of pre-training the data is  walking away from one of the greatest payouts in the history of capitalism.