Two years after
It's been two years since the infamous situational awareness. I completely agree with what Leopold said, "You can see the future first in San Francisco,". I wanted to find out where humanity is going in the next 20 years and I can't think of any other places to look for it other than San Francisco. Important enough that I paused whatever idea I was working on to stop the humming noise in my ear and flew in to San Francisco from London.
The first ultraintelligent machine is arriving. If you are in San Francisco you can probably feel it, even engineers who used to work on low-level engineering tasks no longer need to write code. Two years ago my friends and I would complain that it (coding agents) couldn't handle low-level networking tasks. As far as today goes, there are still small edges you need to drive it towards, but for the most part it can competently do the majority of the infrastructure work as long as the specs are clear, and handle small parts of eigendecomposition, convex optimisation, change detection and more in pure math.
We forgot the order of magnitudes (here and after OOM). We only saw what's now but forgot to relate now to the past and the speed it has been, the order of magnitude. What was soon going to happen was kept among several hundred people, most of them in San Francisco. Two years after, the world has realised what's going on, roughly, but it's still within a small subset of people. Not hundreds, but tens of thousands of engineers and people who work closely with the frontiers and help enterprises transform their businesses. The large majority is enjoying, or suffering from, the "AI boom", but this is not the end of the story.
I witnessed a lot of my friends treating every new generation of large release like a pet - "Why is it not responding to me?", "How can I make it work smarter?", "They cancelled the release? I am down today. I don't want to code." An invisible emotional bond with a junior employee who you have trained for the past couple of years. Once a good model gets retracted, they comment, "There was nothing like it. I don't want to use anything else now."
If the doomsday machine is meant to be made, we will need every power we have on this planet to survive our own curiosity. Perhaps Elon's SpaceX can help a very small group of us to sustain our lifeelsewhere.
I will reference a lot from Leopold's situational awareness (appreciation and gratitude to him who wrote it), combined with what I have heard and read, to fact-check what has changed. This was an article I read again and again, wish I have read, and anyone of my friends in London have read in 2024.
Let's see where we are at after two years.
Outline
Do we have enough land for this?
If the prediction is true, we will be needing 10-100+GW-powered data centers in every country. Looking at the global energy supply and data centers that have been built or are on track to be delivered...are we on track? What are the actual margins behind monster rounds? Mining precious metals, producing energy and building energy from space.
Generative models' lifecycle on racks
Just like programming languages have their own compilers and runtimes, and every compiler has its lifecycle and hooks. Models have the same. But much much larger. Large enough that it's sharded in order for it to fit in GPU forests.
Your brain lives somewhere else
The boom of workflows, VMs and storage. A small server can fit thousands of VMs. Fixed cost. Much better margin than models and GPUs (at least from the sound of it). Every generation of storage systems is truncated, collapsed into one word "memory". Agent orchestration in applications.
When the robots can build factories, produce themselves, there will soon be enough robots. Enough for every household and cornershop to have a complementary robot butler or waiter.
The software factories of rewriting businesses
Slurm. K8S. Transform electron from crypto? Or to token? The tech stack from each hyperscaler to the latest AI companies that are redefining RPAs. More than half of the FT500 will be shopping or falling.
Lock down every households and turn off electricity
Shutdown the electricity. Don't go out. Nothing can prevent it now. What if sovereign AI and state espionage is not enough/too much? DeepSeek stop hiring Chinese studied abroad. Inter-country collaboration for national safety.
We don't even know the boundaries
So far what we have been doing is giving it instructions to prevent it from going off the rails. What if we don't even know where the boundaries of the rails are? You don't want to put up guardrails, inject and modify assets when the assets are already sealed.
We all know it will get out of control. But when? How soon? How fast? We don't know yet.
...There was a pregnant moment in which Lawrence and Caroline saw the numbers flip to point nine nine nine. Then all Hell broke loose. The house disappeared. The island was barren; the palm trees were gone. In the sky, the GAT display had begun to seethe and boil....