Part I, II, III, IV, V
In Part I, II and III of this five-part article, I wrote that the computer industry is in a panic because there is no easy way to program multicore processors. I also went over the advantages and disadvantages of MIMD, SIMD and heterogeneous multicore architectures. In my opinion, what is needed is a new programming model/processor architecture that combines the strengths of both SIMD and MIMD while eliminating their weaknesses. I am proposing a universal parallel computing model that uses fine-grain, deterministic parallelism in an MIMD configuration to handle anything you can throw at it. I will describe what I have in mind in Part V. In this post, I want to go over the reasons that I think the computer industry’s current multicore strategy will turn out to be a colossal failure.
High Stakes Gamble on the Future of Parallel Computing
The Chief Software Evangelist for Intel, James Reinders, believes that the key to successful parallel programming centers around scaling, debugging and future proofing (source: Dr. Dobb’s Journal). To that list I would add automatic load balancing and ease of programming. Of course, being an Intel evangelist and the author of the new book “Intel Threading Building Blocks”, Reinders is necessarily biased. When he says, “think parallel”, what he means is, “think threads”. Reinders strongly advises programmers to use threaded libraries or, better yet, Intel’s own threading building blocks. He is pushing the use of threads because Intel’s multicore processors are pretty much useless for non-threaded applications. The same goes for AMD and other multicore vendors. These guys are so desperate they’re giving all their code away, free.
Let’s say you decided to listen to the industry pundits and you painstakingly rewrote your entire legacy code to use multiple threads and got it stable enough to be useful. That would make Intel and AMD very happy and your code would indeed run faster on their multicore systems. But what if they are wrong about the future of parallel programming? What if (horror of horrors) the future is not multithreaded? Would all your time and effort have been in vain? The answer is yes, of course. All right, I am not trying to be an alarmist for the hell of it. I am trying to drive home an important point, which is this: Intel and AMD have already made the fateful decision to go the multithreading route and they are now irreversibly committed to it. If their gamble does not win out, it will undoubtedly mean tens of billions of dollars in losses for them and their customers. That would be a disaster of enormous proportions and they know it. This is what Reinders really means by ‘future proofing’. He is more concerned about future-proofing Intel's multicore CPUs than anything else. So if you listen to evangelists like Reinders (or AMD's Chuck Moore), you do so at your own risk because the sad reality (see below) is that the industry does not have a clue as to the real future of parallel computing. There’s something sadly pathetic about the blind leading the blind and both falling into the same ditch.
The parallel programming problem is not a new one. It has been around for decades. In a GCN article titled “The Multicore Challenge”, Cray J. Henry, the director of the U.S. Defense Department’s High Performance Computing Modernization Program, wrote:
The challenge of writing parallel software has been the key issue for the computational science and supercomputing community for the last 20 years. There is no easy answer; creating parallel software applications is difficult and time consuming.
This is rather telling. For two decades, some of the smartest people in the computer research community have been using threads to program super high-performance parallel computers. Even after spending untold zillions of dollars and man-hours on the parallel programming problem, they still have no answer. It is still “difficult and time consuming.” What is wrong with this picture? The answer should be obvious to anybody who is not blinded by reality: the academic research community has no idea what the solution might be. They are not any closer to a solution than they were when they started. They have spent twenty long years (an eternity in this business) trying to fit a square peg into a round hole! And, incredibly, they are still at it.
So what makes either Intel or AMD so confident that threads are the way to go? What is the theoretical basis of their multicore strategy? Well, the truth is that they are not confident at all. They have no leg to stand on, really. If they had solved the problem, they would not continue to pour hundreds of millions of dollars into research labs around the globe to find a solution. They are not pushing threads because they think it is the right way to do things. They are pushing threads because they have no alternative. And they have no alternative because they are following in the footsteps of academia, the same people who, in my opinion, got everybody into this mess in the first place. It is one more example of the blind leading the blind.
The Hidden Nature of Computing
The way I see it, if computer scientists had started out with the correct computing model (yes, there is such a thing, even with a single-core processor), there would be no crisis at all and you would not be reading this article. Adding more cores to a processor would have been a relatively painless evolution of computer technology, a mere engineering problem. Unfortunately, the entire computer industry still has the same conception of what a computer should be that Charles Babbage and Lady Ada Lovelace had one hundred and fifty years ago! Is it any wonder that we are in a crisis? To solve the parallel programming problem, the industry first needs to understand the true nature of computing and then reinvent the computer accordingly, both software and hardware. There are no two ways about it. And the longer they wait to wake up and realize their folly, the worse the problem is going to get.
The true nature of computing has nothing to do with universal Turing machines or the Turing computability model or any of the other stuff that academics have intoxicated themselves with. I have already written plenty about this subject elsewhere and it does not make sense to repeat myself here. Suffice it to say that, as soon as one comes to grips with the true nature of computing, it becomes immediately clear that the multithreaded approach to parallel computing is a complete joke, an abomination even. If you are wise, you would take heed not to listen to the likes of Mr. Reinders or to anybody who goes around selling what I call the threaded snake oil. As the native americans used to say, they speak with a forked tongue. :-) Threads are not part of the future of computing. Using threads for so-called future-proofing is a disaster in the making, wishful thinking notwithstanding. Reality can be cruel that way.
There is a way to build self-balancing, MIMD, multicore computers to implement fine-grain, reactive, deterministic parallelism that will not only solve the parallel programming problem but the reliability problem as well. I’ll go over this in my next post. Stay tuned.