Wednesday, June 3, 2015

Why Deep Learning Is a Hindrance to Progress Toward True AI

Supervised vs Unsupervised Learning

In a recent article titled Emtech Digital: Where Is AI Taking Us?, MIT Technology Review editor Will Night writes:
However, [Quoc] Le said that the biggest obstacle to developing more truly intelligent computers is finding a way for them to learn without requiring labeled training data—an approach called “unsupervised learning.”
This is interesting because we hear so much buzz lately about how revolutionary and powerful deep learning is and about how truly intelligent machines are just around the corner because of it. And yet, if one digs deeper, one quickly realizes that all this success is happening thanks to a machine learning model that will soon have to be abandoned. Why? Because, as Google Brain research scientist Quoc Le says, it is based on supervised learning.

No True AI Is Coming from the Mainstream AI Community Anytime Soon

I have reasons to believe that true AI is right around the corner but I don't see it coming from the mainstream AI community. Right now, they are all having a feeding frenzy over a soon to be obsolete technology. There is no question that deep learning is a powerful and useful machine learning technique but it works in a narrow domain: the classification of labeled data. The state of the art in unsupervised learning (no labels) has so far been a joke. The accuracy of current unsupervised deep neural networks, such as Google's cat recognition program, is truly abysmal (15% or less) and there is no clear path to success.

Time: The Universal Bottom-up Critic

One of the reasons that the performance of unsupervised machine learning is so pathetic, in my opinion, is that researchers continue to use what I call static data such as pictures to train their networks. Temporal information is simply ignored, which is a bummer since time is the key to the AI kingdom. And even when time is taken into consideration, as in recurrent neural networks, it is not part of a fundamental mechanism that builds a causal understanding of the sensory space. It is merely used to classify labeled sequences.

Designing an effective unsupervised learning machine requires that we look for a natural replacement for the top-down labels. As we all know, supervised or not, every learning system must have a critic. Thus the way forward is to abandon the top-down critic (i.e., label-based backpropagation) and adopt a universal bottom-up critic. It turns out that the brain can only work with temporal correlations of which there are two kinds: sensory signals are either concurrent or sequential. In other words, time should be the learning supervisor, the bottom-up critic. This way, memory is constructed from the bottom up and not top-down, which is as it should be.

The Deep Learning Killer Nobody Is Talking About

Other than being supervised, the biggest problem with deep neural networks is that, unlike the neocortex, they are completely blind to patterns they have never seen before. The brain, by contrast, can instantly model a new pattern. It is obvious that the brain uses a knowledge representation architecture that is instantly malleable and shaped by the environment. As far as I know, nobody in mainstream AI is working on this amazing capability of the brain. I am not even sure they are aware of it.

Conclusion: Be Careful

Sensory learning is all about patterns and sequences of patterns, something that mavericks like Jeff Hawkins have been saying for years now. The trick is to know how to use patterns and sequences to design the correct (there is only one, in my opinion) knowledge representation architecture. Hawkins is a smart guy, probably the smartest guy in AI right now, but I believe a few of his fundamental assumptions are wrong, not the least of which is his continued commitment to a probabilistic approach. As Judea Pearl put it recently, we are not probability thinkers but cause-effect thinkers. And this is coming from someone who has championed the probabilistic approach to AI throughout his career.

In conclusion, I will reiterate that the future of AI is both temporal and non-probabilistic. It may be alright to invest in deep learning technologies for now but be careful. Deep learning will become an obsolete technology much sooner than most people in the business believe.

See Also:

Unsupervised Machine Learning: What Will Replace Backpropagation?
Fast Unsupervised Pattern Learning Using Spike Timing
In Spite of the Successes, Mainstream AI is Still Stuck in a Rut
No, a Deep Learning Machine Did Not Solve the Cocktail Party Problem
Mark Zuckerberg Understands the Problem with DeepMind's Brand of AI
The World Is its Own Model or Why Hubert Dreyfus Is Still Right About AI