Recently, I watched a movie on one of my frequent flights. The choice was Ex Machina, about a programmer who has been chosen to be the human component in a Turing test of Ava, a beautiful AI powered robot, at his CEO’s estate. However, Ava is far more self-aware and deceptive than either man imagined.
The capabilities of the robot in the movie were pretty far-fetched for the time. Even today, robotics has a long way to go to emulate that level of human-like behavior. But watching the movie, I couldn’t help but think that the level of interaction with the robot depicted, the conversations, are much, much closer to reality today.
This movie was released in 2014, four years even before Open AI’s first generative pre-trained transformer model (GPT) was released. AI has progressed rapidly since then, to the point the capabilities of the robot in the movie don’t seem so far-fetched anymore.
So what’s driving this rapid advance of AI?
The Evolution of Compute Power
Over the past two decades, the fields of big data, machine learning (ML), and now generative artificial intelligence (AI) have evolved at a breathtaking pace. From the early 2000s, when data storage and computational capabilities were limited and expensive, to today, where we have nearly unlimited access to high-performance computing and cloud storage, the transformation has been profound.
At the heart of these advancements lies the exponential improvement in computing power and storage technology. These improvements have enabled organizations, researchers, and developers to analyze massive datasets, train sophisticated AI models, and unlock new capabilities in generative AI.
Some of the key developments in compute that have enabled breakthroughs in AI include:
GPUs and TPUs: The AI Revolution
In the early 2000s, CPUs were the dominant processors for general computing tasks, but they were not optimized for the parallel processing requirements of machine learning algorithms. The introduction of Graphics Processing Units (GPUs) changed the game. NVIDIA, a company originally focused on gaming graphics, released CUDA in 2006, allowing developers to use GPUs for general-purpose computation. This was a milestone for deep learning, as researchers could train neural networks much faster than on traditional CPUs.
Google further accelerated AI research with the introduction of Tensor Processing Units (TPUs) in 2015. TPUs were specifically designed for neural network computations, significantly reducing the time and energy required to train large models. These specialized processors enabled the development of massive AI models such as BERT (2018) and GPT-3 (2020), which would have been impractical on traditional hardware.
Cloud Computing and Distributed Systems
Before the rise of cloud computing, AI researchers were constrained by the computational resources available to them locally. The emergence of cloud platforms like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure revolutionized access to high-performance computing. Now, researchers and companies could train large-scale models on powerful cloud infrastructure without needing to invest in their own expensive hardware.
One of the major breakthroughs facilitated by cloud computing was the training of AlphaGo, the AI system developed by DeepMind that defeated human world champions in the board game Go in 2016. AlphaGo’s success was partly due to its ability to train on Google’s vast cloud infrastructure, which allowed it to process massive datasets and refine its deep learning algorithms efficiently.
The Storage Revolution
As computational power has improved, so too has storage technology. The ability to store and retrieve vast amounts of data efficiently has been a key driver of AI and big data advancements.
From Hard Drives to SSDs and Beyond
In the early 2000s, data storage was dominated by traditional hard disk drives (HDDs). These drives, though reliable, were slow and had limited scalability. The introduction of solid-state drives (SSDs) in the 2010s dramatically improved data access speeds, reducing bottlenecks in data processing workflows. SSDs enabled faster training of machine learning models by ensuring rapid read/write speeds, allowing data to be processed with minimal latency.
Furthermore, innovations in distributed storage systems, such as Hadoop’s HDFS (Hadoop Distributed File System) and Apache Spark, allowed organizations to manage and analyze petabytes of data efficiently. These technologies became foundational for big data analytics, enabling businesses to extract meaningful insights from large datasets.
The Emergence of Object Storage and Data Lakes
The need to store unstructured data efficiently led to the development of object storage systems like Amazon S3 (launched in 2006). Object storage provided a scalable, cost-effective way to manage massive datasets without the constraints of traditional relational databases. Data lakes, which store raw data in its native format, became a crucial element of big data pipelines, enabling advanced analytics and AI-driven insights.
One key example of how data storage advancements have enabled AI progress is OpenAI’s training of GPT-4. The model required access to massive datasets, including books, articles, and code repositories, which were efficiently managed using modern data storage architectures.
The Rise of Generative AI
Generative AI encompasses models that can create text, images, music, and more. It has been one of the most exciting developments in recent years. These models require enormous amounts of data and computational resources, made possible by the improvements in compute and storage.
Breakthroughs in Large Language Models (LLMs)
Large language models like OpenAI’s GPT series and Google’s Bard have redefined what AI can do. The ability to train these models on trillions of words would not have been feasible without the advancements in GPU clusters, TPUs, and distributed training techniques.
In 2020, OpenAI released GPT-3, a 175-billion parameter model that could generate human-like text, summarize information, and even write code. The computational cost of training GPT-3 was estimated to be in the tens of millions of dollars, showcasing the reliance on cutting-edge hardware and cloud computing resources.
The Explosion of Generative Art and Creativity
Generative AI is not limited to text. Models like DALL·E (2021) and Stable Diffusion (2022) have demonstrated AI’s ability to create stunning visual art based on textual prompts. These breakthroughs were made possible by the combination of high-performance compute and efficient deep learning architectures. As well as vast amounts of image data stored in scalable cloud infrastructure.
The Future: Quantum Computing and Beyond
As we look ahead, the next frontier in computing could be quantum computing. Quantam computing promises to exponentially accelerate AI training and optimization tasks. Companies like IBM, Google, and startups in the quantum computing space are exploring how quantum processors could handle AI workloads that are currently infeasible even on the most powerful supercomputers.
Moreover, innovations in neuromorphic computing, which mimics the human brain’s neural architecture, could further push the boundaries of AI efficiency and performance. These developments will likely shape the next generation of AI models, making them even more powerful and accessible.
Ava is coming
The significant improvements in compute power and storage over the past two decades have been the backbone of the rapid progress in generative AI. From the early days of CPUs and HDDs, to the modern era of GPUs, TPUs, SSDs, and cloud-based AI, each leap in technology has enabled new breakthroughs that were previously thought impossible.
The concept of an autonomous, human-like, AI-powered robot as depicted by Ava in Ex Machina may not quite be a reality yet, but it is coming. And probably faster than we realize. Hopefully, it’s just not as dangerous as the robot depicted in the movie.
Evolving technologies like quantum and neuromorphic computing will pave the way for AI systems that are faster, smarter, and more capable than ever before. Further advancements in computing and storage will be critical to unlocking the next wave of innovation, pushing the boundaries of what machines can achieve.
