Hardware

How supercomputers still influence programming

Justin James makes the case that the supercomputing's legacy is playing a greater role in the way programmers write software today.

 

Earlier this year, I did a lot of research into the history of computing. I learned about mainframes, microcomputers (what we now call a PC), clusters, grids, and minicomputers. I also learned about the fabled class of computers known as supercomputers.

During the Labor Day weekend, my family and I went to the National Air and Space Museum in Washington DC. We saw one of the Cray computers (the Cray-1) that NASA used. While the Cray computers have changed, and supercomputers as a class have lost a bit of their mythological cachet, the supercomputing's legacy is playing a greater role in the way we write software today. (On a fun side note, check out a CNET TV video that shows how Star Trek influenced development of the Cray supercomputers.)

Cray-1What separates the supercomputer class from other multiprocessor "Big iron" machines is the focus on execution speed. Mainframes are largely devoted to batch processing, such as filing insurance claims or handling financial transactions. Supercomputers, on the other hand, are charged with things such as weather prediction and chess playing. While both may have dozens, hundreds, or thousands of CPUs, mainframes are more oriented towards reliability and fail safety, while supercomputers care more about speed. A good example of this can be seen in my picture of the Cray-1. All of that blue in the middle are wires; there are miles upon miles of wires (none are more than a few feet long), and the system has the cylinder shape to reduce the distance that any signal needs to travel. These systems are the computer equivalent of jet fuel funny cars -- insanely fast but useless on a normal road.

Supercomputers have changed a lot over the years. The class often used specialized processors not used for other purposes; today, the x86/x64 family of CPUs is the overwhelming choice. Their OSs have varied from obscure "one off" systems to various Linux distros, and Windows even has a "High Performance Computing" variant that is now offered on Crays and other supercomputers.

The design of these systems is often decades ahead of what 99.99% of us are using today, but at the same time, our systems tomorrow will often use the same technology or ideas. Some of the revolutions occurring in servers rooms, such as clustering and grid computing, were first used in these systems. Programmers of supercomputers pioneered techniques that mainstream developers are just now starting to use, thanks to the adoption of multi-core CPUs in the server room and on the desktop. Ideas such as the N-tiered architecture that is so common today has roots in various message passing systems used in supercomputing. Systems like the NVIDIA CUDA are designed to turn a graphics card into a massive array of processors for intense number crunching, which can cheaply and easily turn a $1,000 PC into a supercomputer.

I recently went on the record as stating that parallel processing techniques do not have many applications for the typical developer today. At some point in the future, I think we will get past the types of applications that we are doing today and enter an era in which computer analysis becomes more common in relation to computer accounting than it is today; when that happens, these types of use cases will become "typical business applications." In other words, if you want to know what you may be doing 20 years from now, take a look at what supercomputers are doing today.

J.Ja

Disclosure of Justin's industry affiliations: Justin James has a working arrangement with Microsoft to write an article for MSDN Magazine. He also has a contract with Spiceworks to write product buying guides.

---------------------------------------------------------------------------------------------------------------

Get weekly development tips in your inbox Keep your developer skills sharp by signing up for TechRepublic's free Web Developer newsletter, delivered each Tuesday. Automatically subscribe today!

About

Justin James is the Lead Architect for Conigent.

23 comments
jk2001
jk2001

It seems like the multi-core model is built for parallelism. I've been reading up on things like google's map/reduce paper, learning Haskell, remembering some stuff from Scheme (and counterparts in Perl), and more difficult SQL.

Justin James
Justin James

Ironic, I was reading a long interview with one of the original authors of Haskell last night... What you find is that when the discussion turns to parallel processing, functional languages come up a *lot*. For one thing, their design makes automated parallism (especially lazy evaluation languages like Haskell, O'Caml, F#, etc.) more accurate and possible. In addition, those languages are often well suited for the kind of problems that parallelism is often used to address. J.Ja

oz penguin
oz penguin

> Windows even has a "High Performance Computing" finally, there is a machine that Vista will be able to run on ;)

pgit
pgit

Interesting article, good links. Thanks. I am a bit of a dinosaur, in that I had to learn the details of how the x86/x64 architecture works. (I taught an A+ course for a time) I find this very useful, and more frequently than I ever imagined it would come in handy. It helps with troubleshooting both hardware and software problems based on software 'symptoms.' I remember when Norton 2002 came out it broke win98 systems. Seeing where things fell apart in boot logs I realized where the problem was right off. The "fix" I recommended at the time was to dump Norton! It was introducing what I considered a security breach. (long story) I try to teach people a little about how computers tick, but nobody cares. How many drivers really know how the car they are driving works? How many want to know?

Justin James
Justin James

I agree that deep knowledge is critical to troubleshooting and well as good design, but as many in the industry acknowledge, it is cheaper to throw hardware (or even replace old hardware with new hardware instead of fixing the bug) at these problems than it is to educate people. :( J.Ja

pgit
pgit

Computers have just about become a commodity like dry wall and nails, price wise. But I still have a lot of "frugal" customers that whether they know it or not benefit from the knowledge I've accumulated.

dennis.jones
dennis.jones

One area where we have forgotten the past is in code efficiency. In the 70's and 80's I worked for ICL the UK IT company and we would have meetings with customers to work out how to speed up the (expensive) kit that we were working with and this would usually require us to know how the hardware operated. The result was that you could run an airline reservation system on something with less power than today's PCs. Nowadays most IT professionals haven't a clue what goes on inside a PC and care even less, they don't program , they just call procedures and code blocks written by others. The result is summed up by the fact that my PC applications are no faster now than they were 15 years ago on slower hardware.

chris.j.ireland
chris.j.ireland

Back in the 80s, when I was working on mainframe systems for a major airline, there were two distinctly different types of mainframe programming. The first called for transaction processing to be a fast as was humanly possible and one of my colleagues was given a bonus payment for reducing one common routine by two machine cycles per iteration. The common routine in question was only 12 lines of assembler coding but, over a normal day of usage in the scores of programs it was linked into, the time saving was significant. Conversely, if the system went down, it had to be brought back as soon as possible and any transactions in progress when the system crashed were 'sacrificed' as not being important enough to reconstruct from transaction trails. This was typical of a reservation system where the number of bookings, and hence revenue, which could be lost during an outage far outweighed the cost of losing a couple of bookings which were being processed at the time of failure. It only made itself felt at check-in time when a passenger turned up and their booking could not be 'found'. Of course there were procedures to rectify this... The second called for transaction processing to be as accurate as possible with NO loss of data. Before the system was restarted, all transactions running at the time of failure were fully backed out of the databases, report logs and reports were written, customers informed, and a post-mortem was held at the daily operations meeting. This was typical of systems dealing with accounts and aircraft components and hence aircraft airworthiness. In this instance you did not want to lose the records of a component which could cost up to $750,000 (inertial navigation computers or Main undercarriage assemblies on a Boeing 747, say) so the downtime, in some cases up to an hour, was acceptable from a business point of view as a 'missing' component was much more of a business problem than an irate customer. Neither situation was ideal but some situations were preferable. Writing code was a challenge to get it right first time, before release, rather than an exercise in getting it out as quickly as possible and then spending months fixing it.... if you are allowed the luxury of perfecting it. I'm not convinced that, in the modern era of constant change, we apply the same rigour to the majority of modern software unless it is in an industry deemed to be 'critical' (Nuclear reactor control is, I would suggest, one industry where accuracy is paramount to the success of an operation... and there are others...). I view with dismay the current practice of outsourcing code to people/companies who have no feel for the industry for which the software is developed and the idea that you can purchase 'best of breed' and apply it in conjunction with other 'best of breed' products and everything will be 'hunky-dory'. It won't be.... if only because the various manufacturers enter into a blamestorm revolving around "It's their problem, not ours!" resulting in the only winners being the accountants, not the clients or the customers. The other casualty of this trend is the IT department and its employees who want to do the best job possible. When the last battery in the world is used up and the calculators fail, who will be left to do anything more complicated than basic addition and subtraction? Where does skill-atrophy end? End of rant...

tuomo
tuomo

Back in 70's (and even 60's already) airlines did a lot of nice things with computers and global networks - at that time mostly Univac. And yes, as today, there were two types of transactions. One you really don't care so much, a risk is a risk, but another you just can't lose for monetary or maybe legal reasons. About "supercomputing", there is this one fallacy worse today - cheap hardware can replace good design and coding. Totally false but unfortunately at first companies try it, are successful a while, but sometimes incredible difficult and expensive to fix later (been there, seen that). I have been developing financial, manufacturing, control, scientific, etc transaction and non-transaction systems in distributed (sometimes global, sometimes just local clusters, etc) infrastructures over 35 years and have never seen faster hardware alone to solve the problem in long run! Sometimes even has caused other problems - unbalanced systems can be much worse than underpowered! "Simple" (well, everything is relative, isn't it?) mathematical problems can often be solved faster with faster hw until someone finds a better algorithm! Coding skills - what coding skills? I'm sorry but for reliability, maintainability, efficiency, expandability, etc - the skills started go down with PCs and it is very difficult to find persons who can both design and code for critical systems. Web pages - who cares, you just retry or forget and maybe add a couple of more load balancing servers. Doesn't work where money or safety is the concern! Or - at least shouldn't! Any good software designer / developer from 70's and 80's knows multi-threading, distributed, etc, it was everyday job, nothing special. Some even know how vector processors can be optimized - we did run benchmarks on those and between fast scalar and vector processors (a little biased, I loved Hitachi at that time), all governments, some universities and big companies did that. I think most development / coding problems are not because of developers but the management today? The management model has changed - earlier the developers were told what kind of systems they were supposed to build but today the management tries also to tell how? Earlier - every manager and even a good developer though about the whole business - no it's like "not my problem" - guess how efficient systems you can build that way? Giving the current management technical skills, learned from PC magazines or in dog and pony shows - it is asking disasters! Maybe I'm a little too skeptical but lately (like last 10+ years!) talking with development managers in largest sw companies hasn't been very encouraging! The lack of computer / systems knowledge on that level is unbelievable, a lot of vendor fud and marketing knowledge, acronyms, abbreviations, blah, blah - no idea what they are and how they really work - if they do?

chris.j.ireland
chris.j.ireland

Couldn't agree with you more.... Of course as every CEO worth his salt knows, or implies by their behaviour, a Business Degree now equips graduates to run businesses, doesn't it? If you can manage a grocery store then you can manage a software house, sotware department or code generation unit (used to be called programmers but that's way too descriptive a job title for a modern business giant, or even the Human Resources Department to comprehend). It's easy, there's no difference between bits of food and bits of data! Additionally every graduate with one of the above mentioned bits of paper can run anything, can't they? Of course they can. They've been told so for several of their formative years by respected? academics. Oh dear, I think all is left for me to do is to toddle off and slash my wrists with a core dump... if the paper saving brigade allow me to print one off or maybe I'll just beat myself insensible with a plasma screen until I see the error of my ways... I really can't see a way out of the intellectual fudge that most IT departments have been allowed to fall into. If people much more worthy? than ourselves can't see the wood for the trees, we appear to be destined to fight a losing battle in the few remaining forests.

Justin James
Justin James

Yeah, that was a *little* but before my time, but I've read a lot about that kind of thing (not just that story about the optimization of where the code was stored on the drum, either). Software used to be VERY machine specific in some cases. Store it in such-and-such register, it's physically closer to the register you will be using it with... while on the one hand, I am grateful that we don't need to work like that today, I think some of the "magic" in development is gone as a result. J.Ja

mattohare
mattohare

I think the advantage of not having to program for speed in everything is the ability to program for good reusablity. Instead of global procedures, make them static members of the proper class. Instead of global variables, use a class object that you can send around. These class objects can have events that bring about their own self-destruction when they're no longer needed. Move a declaration inside a code block to limit its scope, avoid its accidental use later. And don't get me started on inheritance. I'm doing things now that I could only dream of back in the DEC-Pascal and QuickC days. Mind, we still need to code for performance on occasion. (Well sound programmers do.) That said, I really feel like I'm back in the dark ages when I'm writing T-SQL code. What I wouldn't give for a simple SWITCH/Select Case type of construct!

PJfromOttawa
PJfromOttawa

I'm trying to decide whether to use Ext-JS, Yahoo User Interface, Scriptaculous, Prototype, (i.e. Framework-of-the-week) etc. Or how to nudge the CSS so granny can still see what's on screen. The registry-efficiency issues are the domain of the guy who ports the compiler to the CPU platform. The sorting algorithm efficiency is for the guy who writes the canned sorting module. The way the Yahoo User Interface bits interact with each other is for the YUI guys. The way the mouse events trickle to YUI's "tree" widget is for YUI guys to worry about. How we fill the tree is what concerns many of us "working" programmers now, as it should, right? What's the definition of "High Level Programming" if not assumptions of the lower-level stuff working as expected (ya, we all know about "ass, you me" -- don't bother). When the assumptions fail, in the software world, we generally find out pretty quickly. It's the old "Standing on the shoulders of giants" thing. And that's good, I think. I'm old enough to know that certain coding techniques are more cycle-expensive than others but I'm not writing medical/nuclear bomb systems. As Matt stated, the magic is in different places now. So Nicky-the-N00b's inefficient loop takes 0.0000013s instead of 0.0000009s but he can read it? Fine. Yes, over time it would add up. Let's move on, shall we?

mattohare
mattohare

When it comes to programming, that's true with all the philosophies and general ideas. Reading about Data Structures (Almost all of my Programming education is self-taught, but from Uni texts) really opened my eyes to a lot of things. Same with Inheritance. I wish I could take you on board as a junior programmer. I need to get my firm to the point where it can pay a salary first, alas.

microP
microP

Being positive is another magic in this era and well done you! Though dont still understand what supercomputer I will be in 20 years time but thank you for talking from my language, made me feel that it will be worth studying OOP after all.

dennis.jones
dennis.jones

I don't go back as far as machine code but programmed in Fortan, Cobol, Algol68 (forerunner of most modern languages). The amazing thing to me is that when the big mainframes crashed (which they did at least once a day) they would output a core dump, basically pages and pages of hexadecimal. Even senior managers seemed to be able to go through this and work out what was wrong and perhaps suggest a code fix. Another interesting story from the time in the late 70's, when I had a spell in sales, one day a mathematician from a local University was invited to talk to our sales meeting about algorithms for the efficient sorting of lists of numbers (it's actually a difficult problem)so that we could talk knowledgably to our customers! Can you image in that now! In those days the computer users were mainly scientists and eggheads anyway.

Justin James
Justin James

When I worked at NCR, we had huge manuals laying around with instructions on debigging core dumps. I was stunned to see my manager's name in one of them as the owner of the book. I had not idea that she was at that level of technical knowledge, and I saw her in a different light from there on out. In college, I had a class that discussed things like O numbers and such, since it was a data structures class, so tree walking, sorting linked lists, etc. was important. Interesting stuff, but not to me *at that time*. I wish I had paid more attention then, as it is now more interesting to me. But like most, I didn't want to hear boring theory, I wanted to get my hands on the keyboard ASAP and start to code. Now, I've been trying to compensate for that lack of formal education, and it is a real uphill battle. J.Ja

Justin James
Justin James

Are there ideas from supercomputing in how you do things today? If so, in what way? J.Ja

IAMheretohelp
IAMheretohelp

Nice article, does anyone know of a site that would give you comparison of various super computers with today's computers. Aside from just pure curiosity it would be interesting to see if there is any correlation between supercomputers and PC - similar to Mores law.

Justin James
Justin James

That's a great question, I have no idea. You can make your own "eyeball" comparison from the FLOP chart in the Wikipedia article on supercomputers, and comparing it to any of the common FLOP charts for PCs over the years. J.Ja

rogerbalakrishnan
rogerbalakrishnan

There is a site top500.com that compares the top 500 supercomputers. I dont know of one that compares them against a pc.

bdanilovski
bdanilovski

The name of the site is top500.org, not top500.com

Mark A. Lewis
Mark A. Lewis

The site looks like a French site. Is it the correct site name? If so, where are the comparisons found?