Saturday, March 1, 2008

Nightmare on Core Street, Part I

The Parallel Programming Crisis

Part I, II, III, IV, V

Panic in Multicore Land

There is widespread disagreement among experts on how best to design and program multicore processors. Some, like senior AMD fellow, Chuck Moore, believe that the industry should move to a new model based on a multiplicity of cores optimized for various tasks. Others (e.g., Anant Agarwal, CTO of Tilera Corporation) disagree on the grounds that heterogeneous processors would be too hard to program. Some see multithreading as the best way for most applications to take advantage of parallel hardware. Others (Agarwal) consider threads to be evil. The only emerging consensus seems to be that multicore computing is facing a major crisis. Here’s a short excerpt from an interview conducted by Dr. Dobb’s Journal with Ryan Schneider, CTO and co-founder of Acceleware:

DDJ: It seems like a system running, say a NVIDIA GPU and a many-core CPU could get pretty complicated to program for. If so, what's a developer to do?

RS: Hide. Look for a different job. Day trade on the stock market... Personally, I find that the fetal position helps. :) In all seriousness though, this is a nasty problem. Your question really describes a heterogeneous system, and most ISVs etc. are probably having enough trouble squeezing decent performance/multiples out of a quad-core CPU without adding another, different beast into the mix.
Schneider is not very optimistic about the future of parallel programming. He goes on to say, “Ultimately there is no easy solution to developing parallel systems.” He’s not alone in his pessimism. In a recent EETIMES article titled “Multicore puts screws to parallel-programming models”, AMD’s Moore is reported to have said that “the industry is in a little bit of a panic about how to program multicore processors, especially heterogeneous ones.”

Incompatible Beasts and Hideous Monsters

The main problem with multicore processors is that they are hard to program. In addition, there is a huge legacy of software applications that cannot automatically take advantage of parallel processing. That being said, what is remarkable, in my view, is that there is currently no single architecture and/or programming model that can be used universally across all types of applications. Multicore processors come in essentially two incompatible flavors, MIMD (multiple instructions, multiple data) and SIMD (single instruction, multiple data). Neither flavor is optimal for every situation. Logic dictates that universality should be the primary objective of multicore research. Yet, amazingly, industry leaders like Intel and AMD are now actively pushing the field toward a hybrid (i.e., heterogeneous) type of parallel processor, a truly hideous monster that mixes both MIMD and SIMD cores on a single die. It is obvious, at least from my perspective, why the industry is in a crisis: they don’t seem to have a clue as to the real nature of the problem.

In Part II of this five-part article, I will go over the pros and cons of the MIMD parallel programming model as it is currently used in multicore CPUs. In the meantime, please read How to Solve the Parallel Programming Crisis to get an idea of where I am going with this.

[This article is part of my downloadable e-book on the parallel programming crisis.]

See also:

Why Parallel Programming Is So Hard
How to Solve the Parallel Programming Crisis

No comments: