Open Source

Skill sets and company size


I am seeing a growing gap between the technology used by mega

corporations and by small/medium/large companies. The latest evidence of this

is Port 80's recent survey showing Microsoft IIS

in complete domination of the Fortune 1000's public Web sites. My initial

reaction was to be quite surprised; after all, it is a well known fact that

Apache runs the majority of Web sites out there. I was suspicious because

Port80 is a Microsoft partner, and surveys and studies funded by Microsoft

require rigorous inspection. The first thing I did was to check Port80's

methodology. Their measurement technique was simple, direct, and effective:

they simply pulled a Web site by the main site URL and checked the HTTP

headers. While it is always possible that some servers have masked or modified

their headers, it is doubtful that it would not account for the disparity between

IIS and Apache.

Of course, a large part of Apache's apparent market domination is the

number of free/cheap Web hosts that have thousands of domains on one server,

none of which get more than a few visitors a month, and to reduce costs they

run Apache on a free *Nix. But IIS having over 50% market share in the F1000

for over three years now shows the differences between the corporate IT

mentality and the smaller company's IT approach.

It is always interesting to me to see this difference when looking at

job ads; the huge companies (think insurance companies, banks, pharmaceutical

companies, etc.) look for skills like .Net, IIS, Windows 2003 Server, Oracle,

DB2, Solaris (not until recently was Solaris Open Source), and other

proprietary, closed source pieces of software that you cannot learn without

paying money and cannot use without paying money. Smaller companies seem to ask

for Linux, Apache, PHP, and other FOSS solutions.

At the end of the day, what this means is that there is a gulf between

the skills you need in the big company and the skills you need for the smaller

company. Gone are the days when HTML and CGI/Perl could get you a Web

development job at any company out there, or C++ could get you in the door as

an application developer. Now, your skill set needs to be tailored to what

environment you want to work in.

Sure, there are plenty of larger companies out there running *Nix or

the LAMP stack, or FOSS in general. Similarly, there are plenty of small

companies using .Net, IIS, Windows server, Oracle, and other proprietary

solutions. But by and large, when I encounter a large company using a FOSS

solution, it is frequently a company that was a startup only a few years ago.

The "old line" companies tend to be closed source, VAR supplied, consultant

installed, maintenance contract paying shops. It is baked into their culture,

and it is hard to change it.

As a developer for some time now, I have watched my skill set change

over time as I move from job to job working with different customers. When I was

working with startups and dot-coms and other companies that were looking to

save money or had a lot of employees that loved technology for technology's

sake, I was using Perl, the Apache, PHP, MySQL, and other FOSS solutions. At my

current job (we perform contract coding for the pharmaceutical industry), it

nearly 100% non-FOSS software. Even our customers using *Nix are using Oracle

on Solaris or similar combinations of "big iron" with "big iron OS's" and "big

iron databases" and other pricey solutions. I spend my days with Microsoft

technologies because they all (except for one, who we do not write any

applications for) use IIS.

So as a developer, you really need to tie your choice of skill sets to

your choice in work environments. The next time I am on a job search, I am sure

that all of my recent experience with .Net, Oracle, SQL Server, Windows Server

2003, and other "big bucks" systems will win me attention from the big boys. At

the same time, I make sure that I have a *Nix (FreeBSD, in this case, I like

the BAMP stack) server in the mix, and work hard with it, so that my skills

with FOSS solutions are kept tip-top. I am lucky; as the sole IT person, I get

to call the shots. Not many HR people are going to care if you read a lot of

MSDN Magazine if you have never touched Visual Studio, or that you have an old

laptop at home running Linux in a desktop environment if you are unable to

install a piece of code that did not code in an RPM package. They want real

world, hands on experience. That means that your choice in skills today affects

where you work tomorrow, and where you work today will affect what you learn.

Think carefully before you take a job, because you can end up locked into an

industry or career path that you are not happy about, and may end up taking a

pay cut or demotion to work in an environment you prefer with the tools you

like best.

J.Ja

About

Justin James is the Lead Architect for Conigent.

23 comments
rclark
rclark

But that isn't really true anymore. Microsoft is a terrible partner to wish on anyone, but the alternative is script kiddies. And you don't trust a fortune to script kiddies unless you want a dot com bust. Linux may be on the verge of getting it's feet into corporate America with it's support from IBM, but like all things IBM it will probably be a branded version. Right now they have bet the farm on JAVA and it's portability. Jury is still out on that one. Until you can get a stable version of unix that is installable on big iron, you won't get a material difference in the stats. I was in one of Oracles early classes on SQL in 1994. Larry said if you were not programming in SQL in 5 years you wouldn't be programming. At the time it too had performance issues. I challenged them then to speed it up or get out of the way. We didn't have time to wait on their bolt on database language to give us the answers. One sweet young lady who had sat through the class soaking it in got up and left at that point. I stopped her and asked why. She told me that this was great technology, but she had 120 data centers that she needed to poll for 1M+ transactions a piece, each and every night, and their little pc's couldn't even call that many, much less transfer and summarize the data. Larry was wrong, it took longer than five years, but most of us use something like SQL nowdays. He missed his bet because those tools were not pushed out to the big iron first. They were developed on the bleeding edge and are just now getting to be boiler plate technology. I helped to develop a system in 1980 that used interactive control language, the DOD precursor for SQL. It had a lot of features that SQL still lacks. But the point is that tech for tech sake, doesn't get the job done. You have to move transactions through databases. You have to summarize data. And you have to report on it. Do this right and you keep your job for another two weeks. One of my favorite commercials is talking about a bank that has the largest clearance of checks without errors. When asked how they clear that many without any errors, the boss said the secret was to get the process down so that a single check processes correctly every time, then duplicate the process 10 million times a day. That is big iron. These little unix boxes of all flavors, just do not have the price points to push mainframe transactions reliably. Never have. With Moores Law, that should have changed a few years ago, but corporate bean counters capitalize over five or six years and it will probably be a couple more before the majority replace the big iron with newer boxes/os/db. And while the hardware is subject to Moores Law, the software has just not kept up. That is where IBM shines. It's OS was doing all of this stuff 20 years ago, and they haven't stood still since then either, and IBM hardware is subject to Moores Law as well. We in the AS400 world laughed when everyone talks about going to 32/64/128 (you choose a number) bit CPU's, quad cores, etc. We have been living there for ages. Now the hot thing is Virtual Machines, (been there done that), parrallel processing (live there), clustering, you name it IBM has led the way for years. And every IBM mainframe is either plug replaceable or in place upgradeable. Shoot they even have some processor clusters that you can throw a software switch and turn additional processors on and off (You only pay for what you use). I don't know what the current limit on processors are, either 24 or 32, may be higher than that, but I'm pretty sure there are real world 16 way processors out there, and not little PC processors either, these are the big boys. We are talking about systems that you can pull a card and upgrade to a new machine in an afternoon. You pay for it, but it is doable. And if you can stay on the leading edge without getting to the bleeding edge, it's all a budgetable expense. So you can acurately predict what it will cost you. That is invaluable for corporate bean counters. The one area that IBM has let us down is in the graphics arena. They have everything everyone else does, but using it is just not as easy as the rest of their systems. And the websphere stuff is just not cost effective for most intranets. Speed is also a problem with large transaction sets that are washed through websites. I'm sure they have something out there with a commerce flavor that will do the job, but nothing as cheap as a windows server with odbc drivers on it. As for skill sets, the bread and butter skill sets are programming in whatever main language the corporation uses. Mine is RPG-ILE currently. That pays the bills, but most of my days are spent running SQL against massive databases and data mining for answers for management. Or writing interfaces for data transfer and transformation. And yes, the training is very expensive. Usually around two thousand dollars per class. So it's probably accurate to say an IBM education will cost you about 150K to get really good and productive. But you can't handle that much education all at once anyway. It takes time to become an expert. Take a bunch of classes on the front end to get on the box, then a couple a year to acquire new skills. Over a lifetime of programming, it adds up. But once you are, you have it. It doesn't go out like Microsoft Certifications. We still have applications running that were written on System3-15D's back in 1982. They are occasionally updated, but any of our people can work on them. IBM makes things that last, and are upgradeable. I also spend a lot of time updating content on the intranet. We use it to store departmental policies and things like that. I picked up most of the DotNet/HTML/ASP/IIS/CGI stuff on the side. It is way to complicated, but it does look nice. I have a few driver programs, a few communication programs, and a few interfaces written in the PC side, but most of that stuff is just facing for mainframe data or control of hardware devices. Marketing took over the external web site last year, and farmed it out to some script kiddies using php and unix servers. BIG SIGH OF RELIEF. They said it was to allow increased functionality, but we haven't seen any yet. Just a different look and feel. Maybe someday the functionality promised will be there. Right now we are just thankful that the artists over in marketing are telling someone else what is wrong with the way the site looks.....

Mark Miller
Mark Miller

Re: Think carefully before you take a job, because you can end up locked into an industry or career path that you are not happy about, and may end up taking a pay cut or demotion to work in an environment you prefer with the tools you like best. That's why I got out of the Unix development world some years back. I worked in it for a while and got bored. I moved over to the Microsoft stuff and had more fun. I didn't take a pay cut at first when I made the move. In fact I got a raise in pay. I may have paid for it later, though it's hard to tell, since that's when the bottom fell out of IT 5 years ago. I'm now moving into FOSS more. I have some ASP.Net (1.1) experience under my belt. Recently I've been finding the FOSS stuff, particularly Ruby on Rails, and Seaside more exciting. I'm not sure what direction I'll go in yet. I feel a little more comfortable working on projects for big businesses. They're better organized, but I've had one good experience doing work for a small business so far. The budgets are smaller, but what they want is more small scale. Sometimes that's really interesting stuff. I agree that .Net probably is more for the large businesses, though I have done one .Net project for a small business. My experience with big business systems sounds like yours. Back when I was working on client/server software for consulting companies, almost universally our customers used Windows on the client, and either Unix (many different flavors) or Windows NT on the server. They used SQL Server or Oracle. Linux was just beginning to get on the radar for corporate IT then, and our clients hadn't started using it. Now it's kind of rare to hear about a .Net deployment that doesn't also use SQL Server. I recently heard a reason for this, and it made sense: Visual Studio. Visual Studio knows how to work with SQL Server like a hand in a glove. You can bring up a list of database tables inside VS, drag a table onto a design surface, and it'll generate the code necessary to access that table. It only does this with SQL Server. Now with ASP.Net 2.0, it uses database polling in its caching. So you can load up tables in a cache and have them auto-update when the corresponding database tables change. Again, I think this is only available with SQL Server (could be wrong). You can use other databases with .Net, like Oracle, or MySQL, but VS doesn't auto-generate the data access code for them. The developers have to do that manually, or use a 3rd party code gen tool, like Codesmith. Microsoft may want to think about opening this up more. Ruby on Rails dynamically generates the data access model, and it does it with just about any database.

Justin James
Justin James

"You can use other databases with .Net, like Oracle, or MySQL, but VS doesn't auto-generate the data access code for them. The developers have to do that manually, or use a 3rd party code gen tool, like Codesmith." Visual Studio 2005 works very well with MySQL in addition to SQL Server. Luckily, I have not tried working with ORacle in Visual Studio 2005. Visual Studio 2005 is so far beyond Visual Studio 2003. They finally started cleaning up a lot of the annoyances, like instead of putting ASP.Net projects in C:\Inetpub\WWWROOT, they now go in the Visual Studio directory in My Documents, so if My Documents is on a different drive (especially a network drive), it puts the ASP.Net projects there too; this also makes moving files around a lot easier. In a nutshell, VS2005 doesn't look much different when you're staring at the screen, but as you work with it, things you used to hate doing (like making a copy of a project to take to another PC or distribute as a package), or working with non-SQL Server databases have gotten much, much easier. J.Ja

Mark Miller
Mark Miller

So VS2005 will generate data access code with MySQL? I must admit I haven't tried VS2005 yet. I only brought up the point because a few months ago I talked with the leader of a local .Net user group. He said that he went to visit with a group of developers working for a company. They did a demo of VS working with SQL Server, auto-generating the data access code. He said it would not do that with Oracle. They'd have to hand code all of it if that's what they used. Maybe he was talking about VS2003, but I kind of doubt it. All of the presentations they've been having this year have been about working in VS2005 exclusively.

Justin James
Justin James

Mark - VS2005 is pretty good at automatically generating basic data access code for MySQL and Oracle, in my experience. It will not be anything really spectacular, just basic SELECT, DELETE, INSERT, and UPDATE statements. You will always need to tweak the non-SELECT statements to have a key, of course. I have not used VS2005 on any complicated SQL Server databases, so I don't know if it does something with SQL Server above and beyond what it can do with other data sources. Overall, VS2005 is much, much better than VS2003 with a lot of the under the hood stuff. My favorite improvement is not putting ASP.Net projects in C:\Inetpub\WWWROOT and using a local IIS process, but doing its own hosting out of the projects directory. It really makes it so much easier to deal with! Add in generic Lists such as: Dim Verbs as List(Of String) And VS2005 + .Net 2.0 is an extremely worthwhile upgrade. J.Ja

JFPSF
JFPSF

Another interesting split I have found in web development is that internal IT development (Intranets, reporting) tends to be overwhelmingly Microsoft, whereas externally facing websites tend more to be Java or LAMP on Unix.

Justin James
Justin James

Not always Microsoft on the inside & LAMP or Java on the outside. It all seems to depends on what they are doing. But you are right, there is often a split. The odd thing about the split, is that it means that the two systems often have a hard time working together, and it creates duplication of efforts, data, etc. It's a mess, and it comes from stupid IT concepts. One of those concepts is that the external Web site belongs to the Marketing department while the Intranet is owned by IT. Indeed, many times, I see the opposite split. Internal IT prefers the FOSS, Java, or *Nix approach, and the Marketing department goes with Microsoft. J.Ja

Tony Hopkinson
Tony Hopkinson

so they got out their copy of frontpage and amade a really good web site without having to didturb one of us obnoxious techy types. LOL After all any fool can craete a professional looking site with with FP, it says so in teh help. I've always been a big fan of IIS, especially ISAPI, which is a wonderful piece of kit. Knocks the crap out of CGI and scripting quality wise. Course it's not for someone who thinks FP designs quality web sites.

shardeth-15902278
shardeth-15902278

It seems that in large companies, which seem to be primarily financially driven (that and regulatory), that IT percieves their role as cops, rather than enablers. Their job is to fence in stupid users, rather then to leverage technology to help their co-workers perform their specific jobs more effectively.

Justin James
Justin James

... there are a million and one reasons why people hate involving IT, starting with "change requests that take 30 days to fulfill even though it is a five minute change" to "bad attitudes and 'RTFM'". I worked at one company (think top 5 global pharmaceutical companies) where the Facilities department (the lights and toilet paper people) actually made their own IT department under corporate radar, with their own servers and staff just because corporate IT was such a royal pain to work with! I think that is one reason why we see the split, too. J.Ja

JamesRL
JamesRL

My billion dollar division of a multi billion dollar corporation primarily uses Linux servers - both in onsite servers for customer and for our ASP implementation. We do use Windows for smaller implementations, primarily because we bought other vendors who had a windows platform. We use Linux for scalability. We can simply handle more users on the Linux side with the same amount of hardware. James

ARon99
ARon99

I have some asp script on a few aspx forms and a .vb class that goes along with it. I use mostly .net objects (text box listbox...). We have an inhouse webserver being set up with a mysql server behind it ALL linux. I have suggested using a microsoft box to serve the page (we are a 501c3 so liscensing is only 800 for unlimited) that way I can use my asp without porting to Php. 1.Is this right? is a Linux box unable to properly serve .net web forms with vb classes? 2. How hard is it to set up a microsoft box for web serving? Do I need a certain OS or some special software? I can write the pages but I've never had to set up the server...

apotheon
apotheon

They did know something you didn't, unless you figured it out in those language comparison classes. What they figured out, as you rightly identified, is that the purpose of a computer is to help you do your work. It's not the god to which you must supplicate and from which you must beg favors. The hacker languages are the best for actually getting work done. Everywhere from OCaml (latter-day ML descendant) for nearly bare-metal coding to Ruby for speed-metal coding (ultra high level programming), the languages made by hackers for hackers tend to make hackers much, much more productive. There's a language for pretty much every purpose and, failing that, you can use Lisp to make one on the fly to fill in a blank. One of my more recent obsessions is UCBLogo, the Berkely implementation of the Logo language. It bridges the gap between Logo and Lisp by adding support for macros to the language. Lisp was originally conceived as using an M-expression syntax that would be easy for the common programmer to read and write. The M-expression syntax would be translated to S-expression syntax, which would in turn be translated into something the computer can use (since S-expressions are essentially parse trees). When Lisp was first implemented, it got as far as the S-expression parser. The M-expression parser would be added later -- except it never happened. Lispers were perfectly happy with the S-expression syntax, so it stuck. The M-expression parser actually did, happen, though. It happened to Logo. Logo's syntax is (drumroll please) basically just a modern implementation of an M-expression syntax. Logo -- and particularly UCBLogo, with macro support -- is Lisp without (mandatory) parentheses. Of course, if you really want to write Logo using S-expression syntax, you can still stick parentheses around everything. I've been kicking around the idea of creating an M-expression interactive shell, inspired by UCBLogo. It could be fun.

Mark Miller
Mark Miller

I assume you were responding to my post. :) Marketing is definitely a part of it, but I can't help but think that there have been other factors I don't fully understand yet. For example, back in the 1980s Lisp was used in the U.S. for various projects. Symbolics made Lisp machines. That died by the time the 1990s came around. Symbolics disappeared from the scene by the mid-90s, I think. The one project that proved everyone wrong was ViaWeb, created by Paul Graham and his associates. ViaWeb is now Yahoo Store. The original implementation was almost 100% Lisp. Since Yahoo got it they've translated most of it into C++ and Perl. The reason? There aren't enough Lisp developers around to maintain it. Graham's only consolation was that even the parts they translated into C++ had to be kind of Lisp-like, which fits with Greenspun's 10th rule: ?Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.? Lisp continues to be used. It sounds like the only real world applications where it's become a major player since its heyday is with airline reservation systems. Just about all of them use Lisp to some extent. The most prominent being Orbitz. It's used to search flight schedules to form itineraries. With the discount sites, it tries to find the lowest priced itineraries as well. Smalltalk was in use for real world projects by the late 80s, but died out in the early 1990s. It's still around and kicking in the U.S. There are the occasional, rare want ads for people with Smalltalk skills, but it's nothing like it was 16 years ago. I know, because I remember when it was common to see want ads looking for people with this skill. I read about this recently. According to one guy who's followed Smalltalk's progress for a long time, the main problem was the Smalltalk community itself. They couldn't unite around a common way forward. The vision was lost, everyone went their own way. And so companies stopped using it. Apparently this has happened a few times in the Smalltalk community. It's showed signs of it even recently. I read a bit of satire online a while back about what was going on with the Lisp community. It sounded like it was a similar thing. The community could agree on some things, but not on some other very important matters, and so it remains in minority status, at least in the U.S. Smalltalk is apparently doing well in Europe, and is growing in use in Canada. It sounds like its rise in Canada is due to a web framework for Smalltalk called Seaside. One of the co-authors of Seaside is Canadian, and he used it to create DabbleDB (www.dabbledb.com), a free-form online cross between a spreadsheet and a database, for all those "Excel database" users out there. Seaside has some promising features, the main one being that you have to deal with session state a LOT less. For simple apps. you don't have to deal with it at all. What a boon! It has AJAX capabilities as well, so you can just deal with the API, without writing a line of Javascript. I'm guessing there's a similar library for RoR.

Mark Miller
Mark Miller

[i]Do you have the time to write libraries? I'd say that, in most cases, the answer is "yes", particularly if you already know Smalltalk well enough to do so with some reasonable skill (rather than needing to learn too much Smalltalk as you go). See, two weeks of glue code ("application logic") and a month and a half of writing libraries is a likely scenario when compared with two months spent writing the glue code in Java and using libraries that already exist. Furthermore, you end up with a set of libraries that will help shorten the development time on future projects.[/i] I can see what you're saying. So, one needs to approach time estimates a bit differently. I've written software libraries before, and I've enjoyed it. They've always been in the static languages. I've had the experience of painting myself into a corner with them, though. Maybe that wouldn't have happened if I had used a dynamic language. It's hard to say at this point for me. For example, one of the libraries I worked on years ago was a database wrapper that worked with API-based database engines. If we wanted to use a different library (engine), we could just change the wrapper, and not have to rewrite our app. code. This was nice, until we got into more sophisticated projects with it. We eventually realized we'd be more productive if we could do joins, which neither our wrapper nor the API-based engines knew how to do. It's conceivable we could've emulated joins inside the wrapper, but it would've taken a rewrite to do it. It wasn't anticipated in the design. The solution I thought was to use pre-built libraries and components wherever possible. Typically these had advanced features that I wouldn't need to use at first, but I could "grow into" using them in the future. Taking that approach was an improvement, and I got projects done faster because of it. Lately I've come to realize that some of the languages and frameworks I've been using have been limiting me. For example I think the way Ruby on Rails handles the database interface is great. In order to achieve the same thing in .Net or Java, they have you generate (one way or another) all of this plumbing code, and compile that into your app. I haven't gotten too far into it, but I understand that with a production build in RoR, you actually compile your Ruby code, as opposed to running it through an interpreter. So perhaps it's vulnerable to the same issues at that point, but at least during development, where all the code is interpreted, it's nice because the dynamism works for you.

apotheon
apotheon

"[i]The one weakness that languages like Lisp and Smalltalk probably have is lack of external support. It's just easier to accomplish some tasks with more mainstream languages, because it's easy to find libraries online, or ones you can buy, that support them and do just what you need, whereas in the minority languages you might have to write the code for it yourself. This would probably be fun to do, if you have the time for it (that's a big 'if').[/i]" You're right, that the major weakness of languages like Lisp and Smalltalk is lack of support for many everyday tasks where you're used to using other languages. They tend to have support (which other languages lack) more in the realms where they're more commonly used -- such as AI development for Lisp. The matter of having time to write libraries and the like for a language like Lisp or Smalltalk is kind of a tricky one, though. The problem isn't really how much time you have, so much as how much of the time spent is on reinventing wheels. For instance, in Java, you have libraries galore. You churn out what amounts to glue code for libraries, and voila, you have a working application. It might take you two months of hard coding to do this, while Smalltalk might have accomplished the same task in two weeks if it had the libraries you needed. Do you have the time to write libraries? I'd say that, in most cases, the answer is "yes", particularly if you already know Smalltalk well enough to do so with some reasonable skill (rather than needing to learn too much Smalltalk as you go). See, two weeks of glue code ("application logic") and a month and a half of writing libraries is a likely scenario when compared with two months spent writing the glue code in Java and using libraries that already exist. Furthermore, you end up with a set of libraries that will help shorten the development time on future projects. Corporate project managers, on the other hand, tend to look at the likelihood of time spent and think about the fact that most of your coding time will be spent on libraries. Somehow, managerial types usually seem to be in desperate need of being disabused of the notion that all languages are created equally: they think, somehow, that experience that says writing libraries for the app in Java would take three times as long as the application logic itself, and would thus take six months on that otherwise two month project, means that writing libraries in Smalltalk would also take six months. Now, obviously, I may be using a little hyperbole to demonstrate a point -- the differences in development time are likely to be less grandiose than I've described -- but most of the reason that my numbers are likely to be unrealistic is tied up more in the fact that all your programmers will know Java and none will know Smalltalk starting out than in any poor guesswork on my part about the development advantages of Smalltalk (all else being equal). It's the "all else being equal" clause that keeps getting in the way: all else is NOT equal. You have more libraries with Java than with Smalltalk, and your people will know Java better than Smalltalk, in almost all cases. If you could magically solve either problem, however, the development benefits of the better language would likely make up for the detriments that remain -- for instance, people skilled in Smalltalk could leverage its power to overcome the lack of library support, while strong library support for Smalltalk would allow your programmers to get as much work done in the same time thanks to Smalltalk's power despite the fact that they're learning the language as they go. Paul Graham, ESR, Joel Spolsky, and I all have pretty much the same solution to this conundrum of Smalltalk-inexperienced programmers and lack of library support. See, what you really need to do is hire talented hackers who play with languages like Lisp, Haskell, Smalltalk, Eiffel, and Ruby on their free time for the pure joy of it. With such people, even if they don't already know the language best suited to the task, they'll learn it so quickly it'll make your head spin. These are people who tend to learn wide ranges of interesting, flexible, powerful languages that leverage wildly varying programming paradigms to good effect and, as a result, adapt to new high-power languages very quickly and thoroughly. You will probably have to pay them a bit more than daycoder legions of mediocre code monkeys regurgitated in droves by university Java-mills, but their productivity tends to be great enough that you can hire fewer of them for the same effect. In fact, when you start using excellent programmers of the love-to-learn sort, you'll find that there's a sweet spot of a half dozen or so programmers on a give project at most, whereas simply throwing ever-increasing numbers of daycoders at a problem seems like the surest solution to over-schedule nightmares when using mediocre code monkeys. "[i]The popular OSS languages like PHP, Perl, and Ruby probably have sufficient support to where this isn't a problem. It seems that way with Ruby, anyway.[/i]" Ruby does okay for itself in this area. It's nowhere near the pervasive library support of Java, but it has some decent library support, and that support is growing every day. PHP has library support to rival Java's within certain very limited problem domains, but the libraries mostly suck -- its real strength is in small projects that need to be put together quickly and potentially maintained by less than stellar coders. Perl has library support, in the form of CPAN, that is the envy of pretty much every other language on the planet: it's really a wonder to behold. "[i]Another potential weakness is it's easy to write "write-only" code. In other words, code that made sense at the time, but if you come back later and try to modify it...good luck! I think this comes from the fact that these languages are so dynamic. They use dynamic typing, and it's possible to generate code dynamically, a common tactic in Lisp.[/i]" There are good coding practices that can increase the maintainability of such clever code significantly, bringing it on par with more mainstream languages (for coders of the same skill level with the language at hand). The trick is in realizing that those practices are to some degree tailored to the language. C/C++ practices for increased readability do not translate as well to Lisp practices as one might expect. After all, the C/C++ practices translate reasonably well for other mainstream languages like Java, and even Perl to a limited extent. The problem is that C/C++, Java, and Perl are all largely descended from a single extended family of programming languages. Lisp is its own, pretty much entirely separate, family. Smalltalk is as well -- and the degree to which Java's requirements diverge from C/C++ is influenced to some extent by how much Java was inspired by Smalltalk in its early conception. Haskell is pretty much another entirely separate language family all its own. Et cetera. When considering how one must learn the practices that make code more readable and maintainable, keep in mind that while switching from C/C++ to Java you're just switching languages, while between these and something like Lisp you're actually switching language families. It's also worth noting that, when you hire excellent programmers who delight in powerful languages, they tend to be better at reading truly elegant code that a common daycoder may not understand at all. To a mediocre code monkey, Lisp's parentheses look like toenail clippings: to a Lisp hacker, they look like a succinct closing notation for something akin to footnotes, neatly enclosing code whose syntactic rules are so regularized and clean that they simply can't grasp why so many people shy away from prefix functional notation. "[i]Dynamic typing is a strength and a potential weakness. On the one hand it makes code more expressive, because you can express your intent without getting into what type something is. The problem can come in (I've had a bit of this experience) when you come back later and try to add to what you wrote before. It's easy to forget what types were assigned to which variables, and therefore how to use them in new code.[/i]" Actually, this is where those previously mentioned language-family "good practices" come into play. Each major shift in linguistic design you encounter will include a different, but always (in a language that isn't complete crap, like BASIC) consistent, typing scheme. The major problem with adjusting to this difference. To take a mild example, for instance, consider people having issues with the dynamic properties of Perl's type system when approaching it from experience with C/C++: they look at Perl's scalars and wonder how the hell Perlists can keep track of what their code is doing when they don't know whether a scalar holds an int, a char, a string, or a float. What they haven't yet realized, and to what they still need to adjust, is the fact that Perl has a very strong type system: it just uses an entirely different type scheme. Perl types are scalar, array, and hash. When you write your code in a manner consistent with this, you don't have to know whether what $foo contains is an int, char, float, or string. All you need to know is that it's a scalar, where it gets its value, and where its value is used later -- the same stuff you'd need to know in C/C++, except then you need to know that it's a (for instance) float, where it gets its value, and where its value is used later. Either way, good coding practice involves testing for the contents of your variables and the data you're feeding into them and extracting from them, but what those tests are and what their results mean to your program depends on the type system of the language you're using if you are doing it right. With Lisp it gets weirder (to the experience of the old C/C++ programmer), because you essentially define your own type systems. There are good code conventions that are applicable to how you go about doing such things, however, and those practices provide clear direction to the skilled Lisp hacker that instruct at a glance on how to deal with the types generated in the course of your code. "[i]Lastly, I've heard recently that with dynamic languages it *really* helps to use TDD with your code. You could probably get away with not doing this in past projects using other languages, but apparently you're just asking for trouble if you don't do this with dynamic languages. Maybe it has to do with what I talked about above.[/i]" You're right, it does. TDD isn't a silver bullet -- it's just something to keep in mind as a means of improving your coding practice with dynamic languages. It's not extra work you have to do, though: you pay this price in exchange for some work infrastructure that is necessary in more static languages, but not in the dynamic languages. For instance, thirty lines of type declaration for ten lines of program logic in a more static language goes away entirely, and so does all the extra stuff you have to do inside your code as the project grows that repeatedly checks data being sent to an int or double variable. It's also worth keeping in mind, of course, that when a forty-line segment of code looks more readable and maintainable in Java than a forty-line segment of code in Perl, the two examples are not otherwise equal. In Java, that forty-line segment is probably ten percent of the application, not counting libraries. In Perl, it's probably [b]the whole application[/b]. Now, suddenly, you're considering the fact that you should be comparing the readability of forty lines of Perl with that of four hundred lines of Java, and that's assuming you implicitly trust all of your Java libraries and your knowledge of how those libraries will behave under edge-case conditions. For an example, consider the fact that I'd rather try to read the weird punctuation mash of [b]$total =~ s/$foo/$_$\//;[/b] than the line upon line of Java code necessary to accomplish the same effect. A hint for understanding that code: [b]perldoc perlvar[/b] is a wonderful thing. Just as object oriented programming literally requires a paradigm shift for programmers coming to it from COBOL and C, so too does programming with dynamic languages require a different way of thinking for programmers coming to it from static language programming. Like object oriented programming, there are a great many benefits to be gained. Unlike object oriented programming, these benefits are not gained at the expense of having to write more scaffolding code. . . . and, in fact, Ruby has finally fixed that problem to some extent by showing that dynamic languages can leverage their flexibility and power to great effect in eliminating much of the OOP scaffolding requirements imposed by languages like Java and C++.

Mark Miller
Mark Miller

That's something I think I still need to learn. Typically I haven't realized a language's weaknesses until I tried to do something in it where it does just an OK job. Sometimes practicality intrudes as well. Last year I was working on trying to improve the install process for a Win32 app. It would've been much easier for me to do this using a .Net DLL. The API for what I needed to do would've been very simple. The problem was the client didn't have the .Net Framework installed on their machines, and I knew there was no way they were going to install it just for the install process. So I "dropped down" into native C code to do the same thing. I created a nice compact DLL. I think it was around 30K. Barely noticeable. It was harder to write than the .Net equivalent, but it got the job done effectively and without the customer having to change their system configuration. The reason I got into this at all was due to some articles I read on Lisp. Paul Graham wrote an essay on his web site (www.paulgraham.com), called "Beating the Averages" that really impressed me. It's the first article I think I've ever seen where a programmer/entreprenuer attributed his competitive advantage and success to a *programming language*, Lisp. Graham is probably biased, but he said that he considers Lisp to be the most powerful programming language out there. Up to that point, I knew there were some languages that were ill-suited to a task and wouldn't work out, and there were ones that could slow you down some if used for certain things, but I had never heard of a language being a critical factor in a business's success. Further, Paul G., Eric Raymond (someone who I hardly ever read), and I think Joel Spolsky as well said that Lisp is a good, enriching language to learn, even if you never use it on a real world project. Basically they said that by using it, you'll learn new ways of programming that you did not know were possible. It'll make you a better programmer. You can bring those skills, even without Lisp, into your projects. Plus, I was reading good things (quotes, anyway) about this language from all sorts of sources, like Dr. Edsgar Dijkstra (who rarely endorsed a programming language), and Phil Greenspun. So I thought it was worth diving back into it and giving it another chance. The one weakness that languages like Lisp and Smalltalk probably have is lack of external support. It's just easier to accomplish some tasks with more mainstream languages, because it's easy to find libraries online, or ones you can buy, that support them and do just what you need, whereas in the minority languages you might have to write the code for it yourself. This would probably be fun to do, if you have the time for it (that's a big "if"). The popular OSS languages like PHP, Perl, and Ruby probably have sufficient support to where this isn't a problem. It seems that way with Ruby, anyway. Another potential weakness is it's easy to write "write-only" code. In other words, code that made sense at the time, but if you come back later and try to modify it...good luck! I think this comes from the fact that these languages are so dynamic. They use dynamic typing, and it's possible to generate code dynamically, a common tactic in Lisp. Dynamic typing is a strength and a potential weakness. On the one hand it makes code more expressive, because you can express your intent without getting into what type something is. The problem can come in (I've had a bit of this experience) when you come back later and try to add to what you wrote before. It's easy to forget what types were assigned to which variables, and therefore how to use them in new code. So I consider variable documentation as essential in this situation. Dynamic code generation (macros in Lisp) is a way to solve some thorny problems, I'm sure, but I think even Paul Graham acknowledged that it can get to the point where it's extremely difficult to decipher how those things operate, especially for a new programmer being added to a team. Again, it may have made sense at the time they were put in. So overall, documentation becomes even more important than it used to be. Lastly, I've heard recently that with dynamic languages it *really* helps to use TDD with your code. You could probably get away with not doing this in past projects using other languages, but apparently you're just asking for trouble if you don't do this with dynamic languages. Maybe it has to do with what I talked about above.

rclark
rclark

I know it is out dated, but it produces very tight code and it runs the same on everything from Dos to XP. And the kicker is it does not depend on any version of DLL's so it won't suddenly quit working two years after it's installed. So if I need something to just work, without muss and fuss, and it has a simple job to do, I sometimes code it in QB45. For all the serious user interfaces I use DotNet, MiniMainframe languages, or ASP. The point is, don't use a language because it's sexy, use it because it is the right solution for the right problem at the right time. Of course to do that, you also have to have the right programmer and the right designer.

Mark Miller
Mark Miller

Re: "I think perhaps one of the factors that contributes to a language's commercial success, believe it or not, is its difficulty of use. The more difficult a language is to use, the more likely it is to be viewed as a 'serious' programming language. Smalltalk suffers from the problem of being far too easy to use." I think you're on to something, but I don't know if it's the whole story. BASIC suffered from that for years, though the joke was usually on the nerds. It was said for a decade or more that BASIC was the most popular programming language out there. Yet it was looked down upon by professionals. Part of that was because it was easy, but also I'm sure partly due to it lacking programming power. That's changed. If you look at VB.Net today, it's closer to Delphi (probably) in terms of language power. It still has some holdovers of its old days in its keywords, but it keeps evolving. Incidentally, Smalltalk was designed so it would be easy even for children to use, though I understand it's used today for high-end systems. So I think its image in people's minds (those who know about it) is different. Alan Kay, the language's author, studied for a while with Seymor Pappert back in the late 60s. It really influenced where he went with Smalltalk and the concept of the personal computer. I watched a video recently where Kay talked about his experience with Logo and Pappert. At the time he thought it was revolutionary: a programming language for kids. Back then there was no such thing. The languages most knew about were assembly, Fortran, Cobol, and PL/1. Too complicated for kids. I haven't heard anyone refer to Smalltalk as weird, but then I haven't met too many people who know about it. I think the ones most people who are familiar with the spectrum of languages think are weird are Lisp and Scheme, 1) because of all the parentheses, and 2) because they do everything in prefix notation. Interestingly, as I was reading through some newsgroups on Ruby someone commented that people are discovering Smalltalk, *because* of Ruby. They share some characteristics with each other, and there is some cross-pollenation that can happen in transferring one's knowledge between the two. I fell into that category as well, though I was somewhat familiar with Smalltalk from my college experience. Until you mentioned UCBLogo, I also had the impression of Logo as being just an educational language for kids. I programmed in Logo when I was in junior high school. It was fun, but what I saw of it wouldn't have caused me to see it as a language I could use to write applications with. What we learned was all about writing procedures (it was my first procedural programming experience) and drawing graphics. So maybe I'll check that out someday.

apotheon
apotheon

That was in answer to you. There was a period of a couple hours on the 25th where everything I posted ended up being posted in the wrong place. At first, I thought I'd made a stupid mistake, but it very quickly became clear that TR was having some kind of fit. Somehow, I imagine George Ou typing "shutdown -h now" in the wrong terminal window -- except, of course, that option doesn't exist with Windows, and I'm sure he wouldn't be caught dead using bash. I think perhaps one of the factors that contributes to a language's commercial success, believe it or not, is its difficulty of use. The more difficult a language is to use, the more likely it is to be viewed as a "serious" programming language. Smalltalk suffers from the problem of being far too easy to use. Things have been creeping gradually closer and closer to being easy to use with new languages, but anything that departs significantly from the average value of ease in contemporary languages will be viewed with suspicion. Think about it: if you code in assembly language, you're viewed as some kind of programming genius, while if you use Smalltalk you're just weird. Even worse is Logo. As I've mentioned, UCBLogo is basically just an easier to use Lisp. It's a victim of its own success as an instructional language, however -- even more so than Scheme. Both are regarded as not being "serious" languages in part because they have been so successful for teaching programming concepts in an educational setting. It's worse for Logo because of the fact that it's easy enough to be used to introduce actual children to programming. It's so easy to use that "real programmers" scoff at it. "Isn't that the thing with the turtle?" Sure. It's also the example language for a trilogy of advanced computer science textbooks used at UC Berkeley. It's also Lisp, complete with macros, if you get the right implementation of Logo. . . . but it's far too easy to use for anyone to take it seriously. C'est la vie.

Justin James
Justin James

It's interesting that you bring up scalability. I would take a Windows server with .Net over a Linux server with Java, but I would take a Linux server with Perl over a Windows server with .Net. For plain HTML, I think Apache beats IIS, and the platform does not make too much of a difference. Indeed, I have seen studies that show that IIS is slightly more scalable than Apache for big, plain HTML sites. It all depends on what you are doing. J.Ja

apotheon
apotheon

The .NET framework on Windows is kicking Java's butt on Linux -- not because .NET is actually any better, particularly, but because Sun's closed-source Java VM and surrounding software infrastructure offerings are not really optimized for Linux. In Sun's mind, Linux is an afterthought for Java support. Java is mostly targeted at Solaris and big iron for "enterprise" web development support. Mono, the open source implementation of .NET, is a little laggy in some areas as well -- which isn't surprising. It's up to par in others, however, so you really need to know what you're doing when you choose to consider .NET vs. Mono. Apache is, as you note, more scalable for the complex stuff, but probably just lags a bit on static HTML unless you're running something like memcached or squid proxies. That's how the Wikimedia Foundation handles high-load web servers: Apache servers with an array of squid proxies. If you just want straight-up static HTML webserver performance, however, there's basically nothing out there that beats the TUX webserver for Linux. It was designed specifically with that in mind, and I don't think it can even do anything else at all. I haven't looked into it too closely, so I can't be sure of details, but the first time I saw the performance numbers I thought it was a misprint. Of course, there are ways to cheat in a free unix environment for scalability. For instance, PostgreSQL database clusters scale better than a monolithic database engine like Oracle ever could, even though on a one-for-one implementation comparison Oracle handles the very highest end slightly better than PostgreSQL. Similarly, IIS only runs on a Windows server, while Apache can be run on a high availability cluster system (different sort of "cluster" from the DB cluster mentioned above), using eight machines for a single Apache instance and putting any pack of eight load-balanced IIS systems to shame for pretty much anything. You're right to bring up Perl for high-end scalability, despite the "common wisdom". The problem is that the common so-called wisdom thinks (Perl == CGI). Run mod_perl with an appropriate toolkit/framework on Apache, and you'll get excellent performance and scalability to make your head spin. The problem is finding programmers that aren't either .NET obsessed proprietary elitists or J2EE "Kingdom of Nouns" devotees who will turn their noses up at Perl saying "I only use [b]real[/b] programming languages," not noticing the C/C++ guys turning up their noses at Java and C#. Then, of course, there's some of the most scalable stuff of all: take two highly motivated young hackers (in the old sense), give them six months to develop something grand, promise them excessive riches, and provide them all the technology and caffeine they want for the task. They could give you an industry-revolutionizing solution built from scratch, from the ground up, on a modified DragonflyBSD kernel, with most of the system (including webserver software, all scripting, and even the configuration files) written in something like Scheme or OCaml. It'll be the hottest web application system on the market for as long as you keep giving them piles of money to stay at work, and as long as they find the task interesting, then they'll leave to do something else more interesting than maintenance. At that point, you'll discover that all the schmucks crawling out of the dregs of college Java 101 classes not only don't know Scheme or OCaml but don't [b]want[/b] to know, and you'll end up rewriting the whole thing in PHP and Java. Of course, you could have fired 90% of your half-baked coders, replaced them with about a tenth as many guys as long as you picked from the "enthusiastic hacker" crowd rather than the ranks of the "daycoders", and kept the system as-is, but since the people who find themselves making these decisions are all pointy-haired bosses they'll want "industry standard" for both coders and code, and as such they'll end up with their formerly industry dominating unique killer app turning into an "industry standard" also-ran. Such is the game.

Mark Miller
Mark Miller

This year I've been rediscovering some of the "hacker languages", like Lisp (predecessor to Scheme) and Smalltalk, and liking it. I was introduced to these languages through my CS education in college, along with Prolog, and SML (predecessor to OCaml). Unfortunately they were all in "comparative language" courses. We'd do a couple of assignments in each, then move on to the next language or subject. Back in the day we did most of our work in Pascal or C. After I graduated, the major language used at my alma mater was C++ (for a few years). Now it's Java. Looking back on it, what I find surprising is that most/all engineering students (ME, EE, etc.) at my university at the time were using Scheme for ALL of their programming/simulation assignments. Makes me wonder if they knew something the CS department didn't. My guess as to why the difference in priorities is that CS at the time was partly seen as a discipline that focused on understanding how computers operated. It got into algorithms, and data structures as well. They didn't really buy into the Dijkstra school of computer science, which says that computers should be used to execute the programmer's ideas. What we got was more like: "Here's how a computer really works. Here's how to *get it* to execute your idea," rather than having a runtime or VM figure that out. I liken it to the "food processor" approach to cooking: First you understand how a food processor works. Then you use the food processor for preparing all of your ingredients, no matter what recipe you're using--even if the processor ruins the recipe if you're not extra careful with it.