Data Management

The changing roles of developers and DBAs

The development industry is changing rapidly, and a lot of job roles are being affected. Justin James says DBAs, developers, and PMs are at the top of that list.

In the last couple of years, there has been a radical series of changes around how developers work with databases. One of the results has been to change the responsibilities of DBAs and developers with regards to the database. Let's examine how these changes affect the development process.

Where we came from

During the client/server years, the role of DBA was a hybrid of systems administrator (specializing in supporting databases) and developer (creating various views, stored procedures, functions, etc. for consumption by applications). The DBA had to know how to optimize hardware and OS configurations for the best performance. DBAs also needed to have a bag of tricks such as table partitioning, careful index creation and tweaking, and so on to get the best performance out of a system. In addition, the DBA would handle security concerns; one of the most common chores in a DBA's day was to create stored procedures that provided extremely limited access to a database and expose that stored procedure to the application as needed, and keep the underlying table locked down.

Developers, on the other hand, were generally at the mercy of the DBA. The DBA had full access to the database, and only granted permission to applications and users where needed. The last thing the DBA wanted was to allow the developer the ability to model a database, since developers are notorious for making some less-than-stellar decisions about database design. In addition, DBAs tried to limit developers' ability to run custom SQL against the data, since many times developers' SQL code would have poor performance because they were not as well versed in databases as the DBA.

However, in many organizations, these distinctions and role divisions never existed. Some IT departments let the developers have full access to the database systems, and in other cases, the developers basically were the DBAs, and the server folks handled the rest. But in larger companies, this division of labor has been very common and expected for quite some time.

What's changed

There has been a huge explosion in development frameworks and systems that make it much easier for developers to run decent code against the databases. Various systems came out that finally let developers run properly parameterized code against a database (Visual Studio 2005 comes to mind) instead of gluing SQL statements together as strings, exposing the system to SQL injection attacks. At the same time, a variety of systems came out (like reporting/data warehousing/BI/etc.) that would create their own SQL code; these systems were often as good as or better than the average developer could put together. Hand tweaking by a skilled human could often improve the generated code, but for most purposes, it is good enough.

We have had major advances in object-relational mapping (ORM) systems like Hibernate and the .NET Entity Framework that have added an additional abstraction layer on top of the database and further obscured it. The ORMs are much easier to work with if the developer has full access to the database and does not need to worry about working through limited access stored procedures. LINQ to SQL in .NET and Rails's AREL are additional examples of systems that make it much easier for a developer to work directly with the database than to work through a stored procedure.

The biggest change of all is the advent of various Agile development techniques. Project specs (and therefore, database models) can now change with every call with a client, and applications are worked on in two weeks sprints as opposed to the six month timelines of the past. Waiting for a DBA to update the data model and change the stored procedures, views, etc., and then have the developers get to work is too many moving parts in a stripped down Agile team. In these kinds of environments, developers are often creating and packaging the databases themselves, and leaving it to an automated deployment process to push the changes to the database.

What's ahead

Does this mean that the traditional DBA role is dead? I don't think so. DBAs are still needed, but they are definitely losing their status as "keepers of the keys to the kingdom" in a number of organizations and probably many more down the road.

Databases still have a very unusual set of performance characteristics, and it takes a trained and experienced professional to plan and tweak systems for the best performance. Beyond that, the existing installation base of legacy applications is huge. Plus, for environments where many applications access a shared database, a DBA is needed to coordinate things (and often to write stored procedures with the proper transactional safeguards) to ensure that no applications trample each other. It's one thing for an application to be loose with data constraints in its own data; it's another thing when other applications need to access that data as well, for example. Constraints are a great example of the kinds of things developers tend to overlook but DBAs don't.

The world of development is changing rapidly, and a lot of job roles are being affected. DBAs, developers, and PMs are at the top of that list.

J.Ja

Disclosure of Justin's industry affiliations: Justin James has a contract with Spiceworks to write product buying guides; he has a contract with OpenAmplify, which is owned by Hapax, to write a series of blogs, tutorials, and articles; and he has a contract with OutSystems to write articles, sample code, etc.

About

Justin James is the Lead Architect for Conigent.

15 comments
oldfield
oldfield

Every situation is different and it there is a balance between how critical is the data (think bank) and what do you want to do. We hit one extreme where the DBA made the data so secure - not even our services could access it; which begs the question of why do we have the data in the first place. We run multiple databases which are not big (few 100 G) but very complex with 1000's attributes/tables and cross references. Designing things requires a collaborative effort between developers, web-staff, project managers and DBA staff. DBA staff in an isolated tower does not work for us, and employing inexperienced developers does not work either. Good procedure and talking to each other does.

Tony Hopkinson
Tony Hopkinson

OODBMS, coupled with the tools is taking the relational knowledge out of the job. Hierarchical is very different and far simpler approach. It's also very inflexible, change which is a given in our profession will cost big style. Not changing will leave horrendous inefficiencies. Short of trivial if you are doing relational, you need someone who doesn't understand table to be a 2D array of cells... Optimising, even steering clear of Oracle, there are so many options it's a minefield of design and implementation compromises. No the real problem was, is and probably will be , very few know enough of both to realise that the DB schema and application do in fact have an impact on each other. No they really do, I can prove it if you want!. This is just another dumb down approach so business can employ more cheap glorified clerks instead of qualified professionals. The tools translate your objects in to tables, are your objects normalised? In a word no, in fact hell no. Tony up to date developer and out of date DBA. If yiou want evidence about hiow generally crap developers are about DBMSs Look no further. http://techrepublic.com.com/5208-7343-0.html?forumID=102&threadID=336429&messageID=3363554&tag=content;leftCol Poor fellow is far from alone.

GuidoCavaliere
GuidoCavaliere

AS a DBA i feel a bit stirred up. There are planty of things that Developers will have to master first in order to fully absorb the "DBA". property. Only "Keepers of the Keys"? I dont think so. DBAs do not only execute queries, grant access, or are in charge of performance tuning, they also develop databases, plan dbs, analize them, deploy Packages, Rebuild indexes, etc. Notice also that the PT part depends on what DB you are working on. For Example ORACLE has a HUGE PT part, its even a module (DBA 4). Sure Java can take care, but still a proffessional dba will be needed. Conclusion: If the field of development is going to guzzle all of that load, then it will have to be carefull, because if not, it may divide itself without concience. But time will decide, after all technology does advance. Oh well, might i start preparing to disembarck in CCNA :)

oldfield
oldfield

1) We employ developers at various levels which means we hire inexperienced programmers and software-engineers and pay them less than the experienced ones. What people are allowed to do depends on their experience - simple as that. The level of guidence/project design/task or project work depends on a persons experience. 2) I can spot someone who lies on their resume very easily - honest - that is part of my job.

Tony Hopkinson
Tony Hopkinson

anywhere, it's also quite widespread. Have to admit the worst case I've did involve Oracle DBAs as well though. :p And it was a property bag approach as well... As an experienced developer I told them they were doing it wrong, but they ignored me anyway. You can't do an effective application without an input of expertise from both so the correct constraints and design compromises are chosen. Demarkation is always work against not with... Inexperienced developers? That's what business wants, they are cheap. The result of that constraint are all too apparent to anyone who looks at the bigger picture.

apotheon
apotheon

I always wonder how people find their "experienced developers" to hire when there is basically nowhere for "inexperienced developers" to get a job and gain experience. I have my suspicions. Among them is that many places hire people with basically zero development experience without realizing it -- because they lie on their resumes so they can get jobs, then spend exactly the same amount of time training up their new developers as they would have if they had just explicitly hired "inexperienced developers". They probably think they'd be much worse off if they hired someone who admitted (s)he was inexperienced, never realizing the only difference is actually the honesty of the new hire.

Realvdude
Realvdude

... in the pathetic, won't lift a pinky to solve their own problem kind of way. I guess that is the difference between wisdom and knowledge; with wisdom you can gain knowledge, but the opposite is not always true. My wisdom is knowing that I am not a dba, I only play one at my job. I continue to expand my knowledge as I learn from those who share and from RTFM.

Tony Hopkinson
Tony Hopkinson

Course when I put 5 years experience with Windows SLQ Server 2008, I was trying to get past HR. :p While you may do the job right, there are quite obviously some who don't.. Got 9 candiadates from the pimps for experienced client server developer. Only two of them knew waht a transaction was. None of them knew what transisolation was. Five of them didn't know how to catch an exception. No lies on the resume as such, just bollocks.

jck
jck

I can't remember how many places I've worked that they try to "streamline" by pushing more roles/responsibility on folks then cut staff and expect others to pick up the slack. Then when things start getting flaky, they blame staff rather than poor management choices. Plus, when they try to stick fresh programmers in a new system with 150,000 lines of code and expect them to just be able to fix it as fast as the guys they pushed out who worked on it for 10+ years? Gotta love business, don't ya? lol

Justin James
Justin James

That is one of the hidden truths of this industry. We've wrecked the mentoring systems, the employer/employee relationship and loyalty is dead so no one is willing to invest in employees and employees know that career growth only comes from moving to another company. On top of that, much of the work at the "entry level" is going overseas. The only way to get to an "intermediate" or "experienced" level of developer for many people is to either "fake it until you make it" (lying on a resuming and BS'ing your way through a job and switching jobs before anyone catches on, until you are mediocre enough to not get fired), working for yourself, or working with open source. I'm becoming rather down on advising people to try breaking into the industry right now, because it is so hard to get in without experience. The economy doesn't help, since there are enough actual experienced developers out of work and willing to work for near entry-level pay. J.Ja

Tony Hopkinson
Tony Hopkinson

to set a constraint of can't create a table for solving a problem like this simply proves academia has lost all contact with reality. An they are teaching these new "developer"s to be "DBAs" and vice versa. Good job I can't afford to retire, I'd miss out on a lot more work cleaning up after these people. :p

jck
jck

I've been applying for positions for which I'm fully qualified. Yet, I still only get the form letters thanking me for applying and saying someone will contact me later if I am chosen for an interview. I have 15 years programming experience, and ten years of that is MS development tool Windows client development experience. Plus, I've done web, mainframe, etc., worked with more (R)DBMSes than I can remember, et. al. And it's not like I've applied for entry level stuff where I am over-qualified. I applied for even jobs where I am not fully qualified just to see if they were looking for someone a little short on skills so that they could lower their pay. I've even thought that maybe it had to do with me being out of state from the jobs I'm applying for, so I have gone so far as to personally talk to the HR department an write them that in my cover letters that I am willing to pay for my own relocation. I guess I need to learn to BS better, because being honest and factual about my experience is not getting me anywhere.

Justin James
Justin James

"One can't even work one's way up from the tech equivalent of the "mailroom" any longer." The current corporate mentality (not just for IT) is now, "if you are good in the mailroom, you need to stay in the mailroom because we can't be bothered to find someone to replace you if we move you out." My little birdie network tells me that the attitude used to be, "if you are good in the mailroom, we'll move you out because we need smart, hard working people in [job above the mailroom], and we'd rather roll the dice replacing you in a low value job like the mailroom, then taking our chances with an unknown entity in [job above the mailroom]." The joke is, the current outlook supposedly mitigates risk, but it creates *more* risk because you are bringing unknown outsiders into the high value jobs instead of the low value ones. Whatever they are teaching at business school should be illegal and have a street value. J.Ja

apotheon
apotheon

The only way to really "break into the industry" these days is to never try. Do something else; write code for the love of writing code in your free time. If you're dedicated enough to it, enjoy it enough, and spend enough time on it, you can eventually either end up with your own business (probably almost by accident) based on code you've written for your own reasons, or end up with a strong open source development background that provides the experience needed to get a job at some company that isn't too badly hamstrung by its own ridiculous notions of bureaucratic suitability for employment. One can't even work one's way up from the tech equivalent of the "mailroom" any longer. All you can do, if you want to make a concerted, intentional effort to get into software development as a traditional career path starting from square one, is get a degree (any degree will do, really), become an open source or otherwise independent rockstar programmer with a fanbase of some sort, and schmooze your way into a job that way. That, or lie, but there's no way I'm going to recommend that to anyone. People who get jobs by lying about their experience actually make the problem worse for everybody else, including the other liars. They can go screw themselves.

Editor's Picks