Parameter Count
“Deep learning” is a concept that essentially goes back to the beginning of AI research in the 1950s.
The first neural network was created in the 50s, and modern neural networks are just “deeper”,
meaning, they contain more layers – they’re much, much bigger and trained on lots more data.
Most of the major techniques used in AI today are rooted in basic 1950s research, combined with a
few minor engineering solutions like “backpropogation” and “transformer models”. The overall point is
that AI research hasn’t fundamentally changed in 70 years. So, there’s only two real reasons for the
recent explosion of AI capabilities: size and data.
A growing number of people in the field are beginning to believe we’ve had the technical details of
AGI solved for many decades, but merely didn’t have enough computing power and data to build it
until the 21st century. Obviously, 21st century computers are vastly more powerful than 1950s
computers. And of course, the internet is where all the data came from.
So, what is a parameter? You may already know, but to give a brief digestible summary, it’s analogous
to a synapse in a biological brain, which is a connection between neurons. Each neuron in a
biological brain has roughly 1000 connections to other neurons. Obviously, digital neural networks are
conceptually analogous to biological brains.
...
