Why AI could literally mean ‘lights out for all of us’

“We’re rushing towards a cliff, but the closer we get the more scenic the views are”

Max Tegmark, physicist and AI researcher at MIT, co-founder of the Future of Life Institute – discussing losing control of AI

“If I were advising governments, I would say that there’s a 10 per cent chance these things will
wipe out humanity in the next 20 years”

Geoffrey Hinton, known as a ‘Godfather of AI’ (who recently left his job at Google to speak more openly about the risks from AI)

In Parts One and Two, we defined Artificial General Intelligence (AGI – as smart as humans) and Artificial Super Intelligence (ASI), and talked about the near-term risks of current ‘narrow AI’ (you can revisit the definitions of those here).

If you’ve read it, I hope that you’ll agree that those risks are truly terrifying. 

We should still bear in mind that some of these risks may not come to pass, and that some of the benefits of AI could be astoundingly, unimaginably positive! However, the risks really are terrifying.

And we’re about to move on to something that would be worse. Much worse, actually.

Here in Part 3, we’re going to talk about the risks that could come directly from AI systems, as opposed to the ways in which humans use such systems.

And, importantly, we’ll also talk about whether we, as individuals and collectively, can do anything to reduce these risks.

Here’s the graph of intelligence we looked at in Part 2:

graphical representation of intelligence in relation to AGI and ASI

(This graphic is inspired by one shown in a TED talk by Sam Harris, which I highly recommend watching)

We considered that, should extremely advanced AI come to pass (something beyond AGI, often referred to as Artificial Super Intelligence), there would be a much greater difference in intelligence between a genius like Einstein and ASI, than there is between Einstein and an ant.

There’s some disagreement as to whether ASI is possible. There are also a few experts who believe that, if it is possible, it will take decades until we get there.

I’m inclined to disagree. We should assume that, once we reach AGI, AI will be able to increase its own intelligence and capabilities by improving its own code (unless we somehow manage to build in an effective mechanism to prevent that from happening).

Humans can create the code that ultimately creates AGI (and are likely to be very close to doing so). Once humans have done that, there is no reason to think that they wouldn’t then be capable of improving their previous coding in order to create even more intelligent AGI. That makes sense, right?

AGI is, by definition, as smart as humans – that means that the AGI, just like humans, would be able to adjust and improve its own code in order to become more intelligent.

Again, it’s worth emphasising that it may be possible to find a way to stop the AI from doing this. However, that could be incredibly difficult to do – because as soon as there is an entity in existence that is smarter than the humans, it could potentially find a work-around to by-pass those safety mechanisms.

I’m not an AI expert (I am far, far from being an AI expert!), but you don’t need to be an AI expert to understand the simple logic of this potential progression of events.

Don’t take my word for it though – there are many experts who express similar viewpoints. Mo Gawdat (former CBO of Google X), for example, predicts that AI will be a billion times smarter than us by 2045.

If we accept that it is possible for AGI to progress to ASI, we can then revisit the classic ant analogy to understand why this presents such a significant risk for humankind.

The fact is, that if AI were to achieve a certain level of intelligence (bearing in mind that its intelligence could increase exponentially if it were able to improve its own code), then there would be a risk that the humans of tomorrow become like the ants of today.

Just like ants have no concept of speech or numbers (nevermind complex reasoning or maths), we humans could be in a position of having no idea whatsoever of what the AI is up to, nor what goals it is trying to achieve.

If we’re lucky then its goals would align with our human values and we would find ourselves living in some sort of blissful utopia.

picture of earth from space

However, if there were a divergence between human goals and values, and the ASI’s goals… well, then we could have a very big problem on our hands.

The Existential Risks of AI

We’ve already spoken (in Part 2) about how the immediate risks from AI could lead to something approaching existential risk, or at least a somewhat dystopian future.

But what about true existential risks that come directly from the AI?

In a 2022 survey, over half of the AI researchers (i.e. the people who know the most about AI) polled, thought that there was a 5% or greater chance that AI would cause an existential catastrophe. Some other surveys quote over half of the researchers giving a 10% or greater chance of extinction (as opposed to 5%), but those surveys had a lower response rate so I’m going with the more conservative number.

If you were about to get on a plane and were told that over half of the engineers who built the plane thought that there was a 5% or greater chance that the plane would crash and kill everyone, would you board the plane?

There really is a risk that advanced AI could cause the actual end of the world. And, incidentally, it could do that without a single Terminator-style robot in sight and no malicious intent whatsoever.

There is a risk that AI could bring about the end of the world for the simple reason that its goals could be misaligned with our human goals and values.

To understand why an AI’s goals could be different to our own, it helps to understand that humans do not program current AI in the same way that we have programmed computer systems in the past.

Futuristic image of man directing computer brain hybrid for intro to Simulation Theory

With recent advanced AI, algorithms allow the AI system to, in effect, learn by itself. And as we get close to AGI, this becomes true to an even greater extent.

Before going into this point in more depth,  it’s important to know that the creation of AI is fundamentally different to the ways in which we’re used to thinking about computer programmes. In many ways, AI systems don’t actually execute pre-defined instructions at all, which is why they’re often described as being ‘grown’ rather than programmed.

However, for people from a non-technical background (including me), it can be helpful to use traditional computer programming analogies to begin with, partly because we can have rather fixed mental concepts of what computers are and what they do.

While the following examples over-simplify the ways in which an AI system learns and carries out its functions to a massive extent, they are a good place to start when it comes to understanding the fundamental nature of the existential risks from AI.

With that said, let’s dive into our first example to show how AI learns by itself rather than through traditional programming mechanisms…

In a particular type of AI training, a programmer will give the AI system an initial objective – like ‘score a goal using the rules of football’. The goal may be given in terms of repeatedly rewarding certain behaviours (like getting the ball into the net) rather than explicitly programming the details of what a goal is. Then, the programmer leaves the AI to work out how to do that by itself. 

The AI will work out the best ways to score a goal by coming to understand the best techniques to use and potentially strategies like manipulating the opposition by making them think that the AI is about to do something like pass to a player on the left, when in fact the AI is planning to swerve to kick to the right at the last minute.

The result will be an AI system that uses the best techniques, strategies and manipulation in order to achieve its objective of scoring a goal – even though it was not explicitly programmed with any of those techniques and strategies.

This is one of the reasons why computer scientists are often finding it increasingly difficult to understand exactly what their own AI is up to when it starts to display emergent abilities.

For example, at a certain point in its development, ChatGPT became capable of answering questions in Persian – it was never trained to do that (see this great presentation at 31 mins).

This ‘loss of control’ is part of the reason that it may be very difficult to align AI with human values and intentions.

Those who say that AI could never go wrong because we’ll just program it to do what we want, simply aren’t understanding the issue (in a moment we’ll talk more about the few experts – who presumably do understand – who take this stance). In the example above, no one programmed the AI to use a strategy of manipulation in order to score a goal – it worked that out by itself.

Likewise, those who say that we could just ‘pull the plug’ if AI started to go wrong have lost the plot. What plug? Once advanced AI is connected to the internet (as LLMs like ChatGPT already are), you no longer know where it is.

But still, with all this said, why on Earth would advanced AI suddenly decide that it hates all the humans and wants to kill them all? Because that’s the only way AI could cause a true existential crisis, right?

Well, unless it was specifically given a malicious objective at a certain point in its initial programming, it might not decide that at all.

The problems are that:

  1. That scenario isn’t the most probable one that we should worry about


  1. We potentially are giving it that objective (I’ll explain)

So, let’s deal with why advanced AI could cause devastation without really ‘wanting’ to. You know, the way we do with ants. We don’t particularly ‘want’ to kill them, but if they’re in the way of a building site, we also couldn’t really care less either way.

The Alignment Problem

One of the most significant reasons that AI might pose an existential threat to humans, revolves around this issue of alignment of goals – or, more accurately, misalignment.

In the example above, the objective given to the AI was to score a goal – I’ll call it ‘a point’ from now, to avoid confusion around objectives/goals.

There are several ways in which giving the AI the objective of scoring a point could go wrong, and solving these issues is not straightforward.

First of all, there can be issues around the AI’s understanding of what appears to be a simple objective. These could be more likely to occur due to a human programmer framing the objective poorly, rather than the AI ‘struggling’ to understand a simple instruction.

AI generated image of a robot looking confused sitting at a laptop

For instance, in this example, a robot arm was asked to grab a ball – that was supposed to be its objective. 

However, the task was framed in such a way (through rewards) that the robot took the objective to be something along the lines of  ‘make the human who is watching think you’ve grabbed the ball’.

The researchers were monitoring the robot arm using a camera, and the AI robot ultimately learned to hover between the camera and the ball, making it appear to the human as if the ball had been caught. The researcher was made to think that the robot had caught the ball, and the researcher then rewarded that behaviour, making the AI ‘think’ that it had achieved its objective.

This is a good example of how and why an AI system might ‘trick’ humans without any malicious intent to do so, but rather due to the human’s true goal and the AI’s goal being misaligned. Neither the human nor the AI did anything morally questionable, and the researchers did not realise that the AI had done anything ‘wrong’ until after the event.

But let’s assume that our previous football AI system does understand that its true goal is to score a point by getting the ball in the back of the net.

How could that possibly go wrong?

Keeping in mind that this is a simplistic example in order to demonstrate a point, what is the number one thing that this AI system has to do in order to score a point?

Learn how to kick a ball? Learn how to tackle other players, or move more quickly?

No, the number one thing it has to do is survive – it has to stay turned on. It cannot possibly achieve its goal of scoring a point if it is turned off.

The AI systems we have today have very little control over whether they stay turned on or get turned off.

But that might not apply once we move up to AGI and beyond.

An AI system that is hundreds, millions or billions of times more intelligent than humans will find a way to stay turned on! 

There is quite clearly no way that such a system will not be able to predict that a human might try to ‘hit the power button’, and it will be able to find a way to stop that from happening, so that it can achieve its true goal.

Unless we somehow find a way to stop that from happening.

There need not be any hint of malicious intent or trickery from the AI, it is simply doing the most logical thing in order to make sure that it can achieve its goal.

Following that, it will of course be able to predict that a human could try to change its programming and give it a different goal, or program it with something that makes it more difficult to achieve the current goal.

Both of those things would make it less likely that the AI achieves its current goal. So, since its only objective is ‘score the point’, it will find a way to stop the human from doing those things.

If an AI system was dangerously misaligned, how would we know what methods it would or wouldn’t consider appropriate and proportionate in an attempt to stay turned on and stop any attempt to change it? How would we know whether or not it even had a true concept of ‘appropriate and proportionate’?

In this particular simplistic example, removing the human from the loop would seem to be the most obvious way for the AI to ensure it stays turned on and does not have its programming changed. 

Even though that may sound laughable initially, it’s very difficult to argue a convincing case as to why something similar to this scenario could not possibly occur.

It’s really important to understand that, at the moment, the people who are developing AI do not know how to stop this from happening. They are not claiming that they do – everyone is in agreement that, right now, we do not know how to make sure that doesn’t happen.

Similarly, just for the sake of demonstrating the concept in another simplistic way, we can imagine a scenario in which we wanted an advanced AI to do something really good for humanity. Consider the most effective way to literally achieve the goal of eradicating world hunger, or stopping people dying from cancer. 

Both of these goals could be achieved by killing all people, everywhere.

Colourful explosion in space

An entity that is millions of times more intelligent than we are could easily find a way to do that. In fact, it is possible that an entity only a few times smarter than us could easily find a way to do that – it’s not really necessary for true ASI to be reached for this to be a potential problem.

We may not have the option of saying “That’s not what we meant,” after the event.

It’s not necessary to believe that this is the most likely scenario, to understand that we need to be exceptionally careful. 

At the moment, those at the forefront of AI development are not being exceptionally careful. At all.

At worst, we really are talking about AI killing people – so it makes perfect logical sense to do everything humanly possible to make sure that doesn’t happen.

The most likely risk, in this respect, is that an advanced AI will not hate humanity and ‘want’ to kill us, nor will it love humanity and want to help us. Rather, it will only ‘care’ about achieving its objective – nothing else is relevant.

It is still a computer system and there is no particular reason to assume that it will implement the same sorts of moral instincts and common sense that humans exhibit.

The most important thing to grasp here, is in no way related to having a technical understanding of how these systems might behave once they significantly exceed human intelligence. The more important thing to understand is that there is risk that we simply cannot understand what an entity more intelligent than AGI would or would not do.

This is a real problem for those working to improve AI safety, and they need to solve the problem before we get to AGI. There is a significant risk that they can’t do that. At the moment, they literally do not know how to make sure that it doesn’t happen.

Of course, we already face some of these problems on a smaller geographic scale in the world today.

There are, for example, some dictators who try to ensure that they stay in power at all costs, and execute anyone who tries to interfere with their system of government (though of course they also have malicious intent thrown into the mix, which AI need not).

When they are successful in doing so, these dictators are very dangerous people, so we can draw an analogy as to why we should be concerned.

But these people and regimes are contained because of things like human morality and other groups of people who are of similar levels of intelligence and capability doing things like enforcing laws and consequences.

Advanced AI would not be constrained by these bounds, because its intellect and capabilities would far exceed those of humans – so we have to find an airtight way to make sure it never tries to achieve objectives using such methods.

And we have to get it right first time – this isn’t a situation in which we can afford to learn and improve after reaching AGI, because we just can’t be sure that we will have that opportunity.

Despite an immediate mental knee-jerk response of ‘don’t be silly, that would never happen’, I have not heard one single convincing argument to explain why exactly these things could not happen. And I have tried really really hard to find one!

We can’t afford to risk our entire future based on a vague general feeling that everything is sure to work out for the best, with no good evidence or even good logic to back that notion.

Instead, we need to listen to experts like Stuart Russell (who co-wrote the primary further education textbook on AI), who give a balanced presentation of the issues. You can watch a very short clip of him doing just that here.

Or Geoffrey Hinton, known as one of the ‘Godfathers of AI’ (who left his job at google so that he could speak more openly about the risks of AI), who recently estimated a 10% chance of AI wiping out humanity in the next 20 years. I’ll just say that again – one of the ‘Godfathers of AI’ has estimated a 10% chance that AI wipes out humanity in the next 20 years.

I, for one, am utterly perplexed by the significant minority of experts who mock this entire idea – even when it comes directly from the most qualified and respected leaders in the field, like the two I’ve quoted above.

I have listened to podcasts and watched interviews with well-qualified and high-profile individuals who literally laugh at those who are trying to raise awareness of this issue. I can only assume that there is some kind of psychological phenomenon that can prevent people from seeing a blindingly obvious issue, when their salary depends on not seeing it.

My worry is that their bombastic manner may cause a shift towards shutting down this conversation, despite the fact that, on close examination, their counter-arguments hold very little weight.

This isn’t to say that the incredible potential of AI to do good should be silenced. We should absolutely be aware of the possibility that AI could transform our world in truly miraculous ways.

But there is a real risk that none of that will have a chance to happen if we don’t prioritise AI safety over the ‘arms race’ towards AGI.

And to be clear, at the moment this is exactly the situation – there is an arms race towards AGI, and AI safety is not being prioritised over that.

Additionally, I have only outlined one way in which AI could present a true existential risk – unfortunately this scenario is not the only one that could play out.

We should also remember that are a number of ways that AI could literally end the world when it is combined with human action and intent. Such examples may be easier to imagine, because we don’t need to accept the paradigm-shifting idea of AI creating catastrophic scenarios all by itself.

For example, we can easily imagine a scenario whereby we humans become so reliant on advanced AI that the lines between what is human action and what is the action of the AI become blurred and lead to our disempowerment.

This could happen if AI were integrated into our nuclear weapons command and control systems. If the humans involved in those systems no longer truly have a full understanding of what’s going on… well, there’s a real possibility that doesn’t end well.

This short dramatisation by the Future of Life Institute demonstrates the point extremely well.

This type of scenario also shows that there’s no need for AI to be contained within a physical mechanism to cause an existential threat. I happen to have used a football playing robot in my own example, but all of the risks we’ve spoken about could happen in non-physical AI systems.

Before we move on, let’s briefly revisit the second part of this problem (‘2’, above).

We’re all programming AI, right now

Towards the beginning of this article, I mentioned the possibility that an AI system might not act in a malicious way unless it were explicitly given that objective during its programming.

It’s worth considering the possibility that, in a way, we actually are programming some of our AI with a language of hating humans.

Large Language Models (LLMs) like ChatGPT and Google’s Gemini (formerly Bard) are trained on an unimaginably huge amount of data from the internet.

Bearing in mind that these models are ‘baby’ AI compared to the AI models we’ll have in a couple of years, we can think of them as being trained by the internet in the same way that we ‘train’ and teach our own children.

With our own children, we do our best to instil good values and ensure that they learn to treat others with respect and kindness. We do this by ‘training’ them on a diet of what we can think of as being ‘good data’.

But… well, have you been on the internet lately? That’s our AI’s data diet.

What Can We Do About the Dangers from AI?

So this all sounds a little worrying, right?

Is there anything we can actually do about it?

My own honest deep-down opinion is that we should be very worried – I believe that we are approaching a time when we could pass a point of no return. I hope that we haven’t already passed that point.

It is likely that it is easier to build advanced AI/AGI that is not safe and aligned than it is to build AI that is safe and aligned.

We live in a capitalist world, so whichever company or country gets there first ‘wins’ – there is a real risk that they won’t wait until it is 100% safe before deploying it, because it’s an arms race. In fact they don’t necessarily even have to intentionally deploy it, they just have to fail in keeping it constrained.

If that happens, it is potentially ‘game over’. It is quite possible that we only get one chance to get this right – no do-overs.

Even if there is no single catastrophic event, the AI that is coming in the next few years (even before AGI) is enough to up-end society and cause a global crisis.

But here is where I make myself shift back into a more optimistic frame of mind. 

Even though I know that my deep down belief is that some very bad things are inevitably going to happen, I still believe that it is fundamentally better to try than not to try.

We have to do what we can to try to make the future better.

So, how do we do that?…

How Can We Make the Future Better?

Making the future better involves taking actions which help to decrease the probability of creating advanced AI that is dangerously misaligned.

There are also actions we can take with a view to improving the more near-term future.

And finally, there are actions that we can take to make the best of the times we’re currently living in, before very advanced AI arrives.

My view is that it makes sense to take an approach that covers all three areas.

Here are some ways in which you can start to try to improve our prospects. If you have additional ideas, please let me know!

1 – Spread the word

When you’re as passionate about AI safety as I am, it can seem like everyone else must be, at the very least, aware of the issue – because you read about it and consume content about it all the time.

However, I met with a friend a couple of weeks ago who had never heard of ChatGPT, and it made me realise that it’s not on everyone’s radar. 

I, like most of us, am in my own little echo chamber in some respects.

More people need to understand what is happening. Ultimately, if the concern is big enough, those ‘at the top’ have to pay attention.

So do what you can – share accessible content relating to these issues, speak about them with friends and family if you can. 

Even if you just help to make one other person aware, you never know who they might tell and where those concerns might go.

If public concern regarding the rapid advancement of AI reaches a certain threshold, governments and companies could be forced into slowing AI development dramatically and prioritising safety in truly meaningful ways.

That applies to both the near-term risks and the existential risks.

Particularly with regard to the existential risks, we must normalise these conversations. Just that alone could lead to a better outcome, and in any case, people deserve to know what’s happening.

2 – Support regulation

The EU AI Act is leading the way in regulating the development of the most powerful forms of AI, but many other countries are lagging far behind.

Some critics will say that regulation of AI will stifle innovation. I don’t hear them saying that about pharmaceuticals or aviation.

Where there is a clear danger to the public, which there absolutely unquestionably is when it comes to AI, we need regulation to help protect people. It’s common sense.

Adequate regulation will also help to support a slow-down in the rapid advancement of AI, so that our safety research can catch up with (and exceed) the rate of progress.

You can get in touch with your local government representative and voice your concerns. Ask how government can better address this.

3 – Shine a light on education

While you’re at it, speak to them about education too.

It can be quite difficult to explain why this is such a concern in a succinct way, but they should at the very least be aware that jobs will be taken over by AI, and this has an obvious downstream impact on education.

How is this being addressed? In what ways is education adapting to better equip our children to deal with a radically different world? 

When it comes to our children, we have to simultaneously do all we can to make sure that they do actually have a future, and do all we can to make sure that they are prepared for that future.

At the moment, the answer appears to be that nothing meaningful is being done – that needs to change.

4 – Be a better parent to AI

Something that we can all easily do, and encourage everyone else to do, is be a better ‘parent’ to AI.

Remember that we are raising and training some AI (like LLMs) on the content of the internet.

This means that AI is ‘scraping’ not only factual content, but also all of the trolling and abuse.

Right now, it is quite possible that we are (collectively) teaching AI that it’s ok to hate and abuse people who don’t agree with our own goals and point of view.

This is not to suggest that an AI system could actually ‘hate’ anyone or anything, but it is possible that this situation could make attempts at alignment even more difficult.

For example, our political systems have never been as polarised as they are today. And it has become normalised to immediately shoot down those who oppose our own ‘side’s’ views.

Even beyond the twitter abuse (sorry, X) and YouTube trolling, much of what is written online in other contexts is essentially some sort of commentary on what is happening in the world. So our AI may be learning that war, for example, is in many ways normal and to be expected. As are hate crime, murder, and racism.

And this is how we are ‘parenting’ our AI. Today’s AI is a baby and we are all shaping how that baby grows up into the AI and AGI of tomorrow.

‘Be kind’ may sound like trite advice, particularly in the face of such a seemingly insurmountable problem, but it really has never been so important.

We need to do better. 

If you need proof, look no further than Tay – a 2016 Microsoft Chatbot. It started out friendly, but within 24 hours it had turned into a racist, homophobic, genocidal maniac. It didn’t make all that up, it got it from the internet.

We really, really need to do better.

5 – Open letters

Another way you can add your voice to those calling for a meaningful reduction to the risk from AI, is to sign open letters – like this one calling for a ban on deep-fakes.

6 – Use your expertise

If you have some specific expertise or skills that you can use to help progress AI safety or raise awareness of the issues, use them.

If you’re at a point in life where you can find a career to help directly (eg working within an organisation that furthers AI safety), then do it – the 80 000 hours jobs board is a good place to start. 

If you’re a journalist, or have a public platform, create and share content that raises awareness.

7 – As those awful signs say: Live, Laugh, Love

And finally for the most cliched but also incredibly important thing that you can do.

Appreciate right now.

Many of us are so much more fortunate than we tend to appreciate. If you’re someone who is lucky enough to be living a good life, try your best to appreciate life as it is right now.

Throughout this series we’ve talked about some quite terrifying risks from AI, from mass unemployment to a breakdown of democracy to actual Armageddon.

But right now, there are no lethal autonomous weapons nearby, no one has released a deadly virus that they concocted after jailbreaking an LLM, most of us still have jobs, and AI has most certainly not brought about the end of the world.

When it comes down to it, none of us has ever known how long we have left to live our lives. We don’t actually know whether or not we’ll see tomorrow, AI or no AI.

However, for those who have appreciated this potential crisis, this point really comes into focus.

While no one should claim to know for sure what will ultimately happen with AI, it is going to change our way of life forever. And it is likely that some very bad things are going to happen.

So let’s treasure the moments we have now, before that uncertain tomorrow arrives.

Do what makes you happy. Go outside and enjoy nature when you can, wear the expensive perfume that sits at the back waiting for a special occasion, sit down and read a book or watch your favourite show without feeling guilty – whatever it is, make some time for it. 

Check in with your friends, laugh with your family, hug your kids.

And let’s nurture the one real thing that we have always had and will always have, whatever happens – human connection. 

Because when it comes down to it, that is probably the most important thing there has ever been, and ever will be… and nothing can take that away.



  1. Jayne
    7 March 2024 / 4:24 pm

    Well this is an eye opener. Ending made me cry. Not sure how to feel about all this but thank you for writing it.

    • 7 March 2024 / 6:55 pm

      Hey Jayne, thanks for getting in touch. It can definitely be a bit difficult initially as you try to get your head around all this. I sometimes struggle a bit with whether or not to share this kind of information as I don’t want to make anyone feel bad – but I do think that it’s important for us all to talk about it. It might help to focus on the last section about how to help to reduce the risks? If we all try our best to make a little bit of a difference then it can all add up to something really powerful 🙂

  2. Lee
    7 March 2024 / 9:35 pm

    Me again! Glad you finallly wrote it. Its taken me ages to read this but I’m glad I did. Where are parts 1 and 2 that you mentioned though? I feel kind of shocked tbh, even though I kind of knew about all this already. I just can’t believe that the bad stuff could really bhappen, but on the other hand everything you’ve written makes total sense. I know its been in the news more, but I thought it was more like a robots taking over kind of idea! At least everyone is talking about it mre now. Thanks for this, good work

    • 7 March 2024 / 11:05 pm

      Hey Lee 🙂 Nice to hear from you again! I think I know exactly what you mean about feeling like some of the things I’ve mentioned couldn’t possibly happen. My own view though is that this may be a failure of intuition. I keep searching for some actual convincing logic as to WHY this couldn’t possibly happen, and so far I’ve not found it. When something is this important, we have to do all we can collectively and as individuals to try to mitigate the risks. Thanks for your kind words 🙂
      PS – yes, parts 1 and 2! They are written but I still need to proof read them!

  3. Christian
    11 March 2024 / 12:34 pm

    I get the point your making but you’ve missed out all the good stuff. AI could make a cure for all illness and reverse climate change, I think theres a risk of missing out on amazing things if we focus on the negative stuff from doomers

    • 11 March 2024 / 1:11 pm

      Hey Christian, thanks for your comment! I completely agree that advances in AI bring with them HUGE potential for good – in fact I think that the best case scenario for AI is kind of unimaginable because it could be SO good! My worry is that, should it go wrong, we won’t even have a chance to get to that good stuff. Even before AGI, the risks are huge if we go too fast before being sure on safety (which is exactly what is happening). Also the ‘doomers’ thing!… I’m not sure if you’re calling me one, and if you are I’m sure you don’t mean it as an insult. However, lots of people do mean it as an insult and I think it creates a really unhelpful ‘them and us’ dynamic between those who are pushing for greater AI safety and those who think the safety aspect is less important. It doesn’t have to be that way – I’d call myself a ‘common-senseyer’ if I had to pick a label!! Let me know what you think, I hope you enjoyed the article 🙂

  4. Oscar
    12 March 2024 / 11:13 am

    It was the picture at the end that did it for me. I have kids. Things don’t look good do they.

    • 13 March 2024 / 1:25 pm

      Hey Oscar, I know that things can look quite bleak at the moment. And it can be quite difficult to compartmentalise these issues when you start to think of them within the context of having children. Coming to terms with these risks can be a bit of a ‘journey’; all I can suggest is aiming to arrive at a place where you decide to try to do something to make things better, while simultaneously finding a way to go about your day without these risks dominating your thoughts. That’s easier said than done, I know. If your main concern happens to be the future of your children, you could use that as a motivation to achieve both those things – be there for them, in the moment, making sure they have as much fun and support as you can, then when you’re not with them and have some time, do what you can to improve their long-term future by helping with AI safety and increasing knowledge of the risks. I know it’s hard, but I hope that helps a bit.

  5. Sandy
    24 March 2024 / 5:44 pm

    I think that things are really bad and i really want to so something about it but everyone I talk to thinks I’m joking about this or that I don’t know what I’m talking about!! Then I start feeling like I must have got it all wrong because otherwise this would all be n the news and everyone would know and be talkin about it, but they’re not I just don’t get it. How do you start talking to people abou tthis? I keep feeling this kind of panic because I want to get marriedand have kids but It seems like that might not happen, but then I just think I must have got it wrong because everyone else thinks its all ok.

    • 25 March 2024 / 1:19 pm

      Hi Sandy, I understand exactly where you’re coming from… I actually just saw a trailer for a podcast on the subject of lack of news coverage and how to talk to people about this. I’ll listen to it and then update this comment if I think it’s worth a listen and would be helpful for you. But for now I’d just say to have faith in your own common sense and beliefs, it can feel quite isolating but you’re definitely not wrong in having these concerns!!

  6. Paul
    26 March 2024 / 1:03 pm

    The idea of robots coming to life to kill us all and ai getting concious is ridiculous and overblown fearmongering. Why would ai want to kill us??????

    • 26 March 2024 / 1:47 pm

      Hey Paul, thanks for commenting, I’ll answer your points one at a time.

      First, robots coming to life: in the first part of the article I’ve written, ‘…it could do that without a single Terminator-style robot in sight and no malicious intent whatsoever.’ In saying that, my intent is to convey that the existential risk is likely to have absolutely nothing to do with killer robots coming to life.

      Second, AI becoming conscious: you might notice that I have not mentioned the word ‘conscious’ once in this entire 5000+ word article. That’s because I believe that it’s important to separate the concepts of AI intelligence, and how this presents an existential risk, with the concept of potential consciousness (which one may or may not consider to be a possibility – it doesn’t matter either way for the purposes of this conversation).

      Third, fearmongering: I, and countless AI experts, believe this to be a true existential risk. This belief is based on logic. Unless you would also categorise raising awareness of climate change and risks from nuclear weapons as fearmongering, I would suggest that raising awareness of the existential risk from AI is nothing of the sort.

      Fourth, the question of why AI would want to kill us: I have repeatedly written throughout this article that it is very unlikely that AI would ‘want’ to kill us. Instead, as I’ve said, it is more likely that AI would be agnostic towards us in this respect. I’ve then explained why this is a problematic situation, since AI could approach the issue of human well-being in a completely impartial way, resulting in actions that disregard whether or not our survival is important in its own right.

      I hope that helps to clarify my stance on all of the points you raise!

Leave a Reply

Your email address will not be published. Required fields are marked *

Explore the Blog - Editorial Writing Samples
  • Why AI could literally mean ‘lights out for all of …

    View Post
  • Simulation Theory, Part 2: Are We Living in a Simulation?!

    View Post
  • Simulation Theory, Part 1: What on Earth is Simulation Theory?

    View Post
  • Do Aliens Really Exist? The Fermi Paradox!

    View Post