General discussion


Unnecessary job - need opinion

By MirrorMirror ·
I have a dev sql server for a data warehouse that uses a well known application to present and report on the data. So that the developers can work with current data, I made a job that sync's all of the dimension tables and select fact tables from production down to dev every day when no-one is using the app. This pumps several GB of data across our network and takes about an hour to run. This way the data in dev is recent and there is no reason to unnecessarily do the daily data load ETL process. The data in the dev environment is one day behind the data in production. When I made the sync job this was discussed and agreed this was sufficient for the developers to work with.

After figuring out recently that the developers were still doing some development on our production app (that means making new reports that hit the production database, no database structure changes), I adamently requested that they immediately move all development to the development server. (I'm such a mean old DBA!) So, they scheduled a cut over date that has already been missed but have scheduled another one. I hope they actually make this date. While discussing how to sync up DTS packages and other jobs, the developers mentioned that the normal daily data load ETL job for the data warehouse is working in development. This means that we don't need the job I made to sync dev to production. No big deal, I thought, have one or the other job run but not both.

The developers want both to run. The reason that they state for this is that if the daily load ETL breaks then the sync job will still be running. I don't think there is any guarantee of this, but that is what they say. Since this is a dev box, I don't REALLY care but I do think that it is silly to have both jobs running. I am kinda funny about wasteful things when it comes to my database servers. Actually, if I had not spent the time making the sync job, I doubt that I would care at all.

So, should I let both jobs continue to run or only allow one job to run?

This conversation is currently closed to new comments.

Thread display: Collapse - | Expand +

All Comments

Collapse -

I've got a dumb question

by CharlieSpencer In reply to Unnecessary job - need op ...

If the dev data is representative of the type of data normally processed on a daily basis, why sync it daily by EITHER method? Why chew up an hour of bandwidth every day? Wouldn't monthly or quarterly be sufficient? When they've got something that passes testing against the last sync, then do another sync and run a final test.

Collapse -

Not a stupid question

by MirrorMirror In reply to I've got a dumb question

I asked that same question when presented with the need to have current data in dev. I was told that this will also allow them to develop and answer questions. Since they are fielding questions on daily dollar and order amounts, they are more comfortable with VERY up-to-date info in dev. Our data warehouse is in its infancy. So, instead of reporting on trends they are reporting on to-the-penny balances, orders, etc. I was originally shooting for a once a week sync, but that got shot down.

Collapse -

Something completely different

by stress junkie In reply to Not a stupid question

All right so you answered your own initial question when you responded to Palmetto. As long as the nightly sync doesn't cause production jobs to fail or to run too long then it should be okay.

Now here is an idea about something else. You say that the developers were using the production machine without your knowledge. Getting them to use the development machine is difficult. How about if you just disable their accounts on the production machines? Then when they start to cry you can copy over whatever files that they say that they need to do their work. Get your manager's consent first, but sometimes you have to be firm when people are breaking the rules AND THEREBY putting the production data at risk. Believe me. I've seen times that developers were using a production machine when a development machine was available. Nothing could go wrong. The things that they were doing were harmless. Then one day an IRS report with a deadline is messed up because some developer moved a buggy bit of code into the production code area. Feathers get ruffled and YOU are asked why YOU allowed the developers to use the production system. I've had that happen so I know it could happen again.

Collapse -

I'm with ya!

by MirrorMirror In reply to Something completely diff ...

This place has never had a DBA or for that matter anyone to ask them to be careful with thier production boxes. I have had to very slowly move them in this direction. When I got here, just about everyone and everything connected to the production databases as "sa". Not to mention, they had development and production databases in the same instance. It has been slow, but they are moving away from the "free for all" method of operation. I'm just trying to pick my battles. As you can tell, I am leaning towards letting the jobs run. I wanted to see if anyone thought this was worth fighting for or if anyone had a similar experience.

Collapse -

You've got big problems

by amcol In reply to I'm with ya!

Developers should NEVER be allowed in a production box. Period. End of sentence.

To say what they're doing is not affecting production is to deny reality, or at least potential reality. Perhaps they haven't executed the query from **** yet, but when they do...AND THEY WILL...and your production box crashes who's going to explain it to your customers?

Why do they need real time full production data to test? Every data warehouse I've ever built had a statistically representative sample data mart attached to it. A 5% sample of the data in architectural identity to the data warehouse located on the dev box is all the developers need.

Where's your CIO? Or maybe a QA department? No standards where you work? The CIO, or more correctly the CTO, should be laying down the law here because it's that person who the business customers are going to come looking for when the prod box crashes.

Doesn't sound to me like anything's going right in your shop. No standards, no management, no accountability, no risk assessment, no architecture, and the inmates are running the asylum. Good luck.

Collapse -

How did you know??

by MirrorMirror In reply to You've got big problems

"No standards, no management, no accountability, no risk assessment, no architecture, and the inmates are running the asylum." Boy, do you have that right! In their defense, this has been and still is primarily a mainframe shop. They still call any thing not mainframe "open systems" and "new". I did manage to get them to separate production and development. I have started implementing standards. I have gotten them to start going through a change control process. There are other things that have changed for the better but it has been slow.

"Where's your CIO?" Even though he has a hard time turning on his computer, he thinks that he knows all there is to know about all things tech. Even though our admin team has asked for controls and adequate resources our CIO has constantly shot our requests done.

We do have problems, and are working to fix them. People are fighting me every step of the way, but I am going to drag them into some semblance of order. They do admit that they need to change. They are just slow. If I get too radical, then I will alienate all of them and no one will work with me. I would sincerely LOVE to pull the plug on them.

When they were starting the data warehouse, I begged them to make sure to implement dev, test/QA and production environments but they only wanted dev and production. The other environments seemed unnecessary and cost too much. I argued and presented but had no luck. They also did not want to pay to have a dev environment that was relatively close in configuration and fire power as production. That is why they still want to use production to do their development, because dev is slower than production. I could just scream. Now they are wondering what they are going to do to train people on the data warehouse. All I can do now is sit back and show the documentation where I told them the reasons for the number of environments. <sigh!!>

Thanks for the good luck wish...I'll need it.

Collapse -

I agree with your approach

by stress junkie In reply to How did you know??

I think you've got a good handle on your situation. You're right about having to pick your battles. You're right about documenting your efforts. I would only add that you should keep talking frequently about what needs to be done with everyone involved. You should talk about these things so much that when people see you coming they will say "Oh here comes DL05. Let me guess what he'll talk about today. Maybe Q/A for software? Maybe keeping development off of production machines?..." This way you will develop a reputation for wanting to see these things done. When problems arise everyone will know that you have talked about preventing problems because that's all you ever talk about. Well, it's food for thought anyway.

The upside of your situation is that it can be very rewarding. You know that these people need you and your ideas. As your ideas are adopted you will know that the business is benefitting from your efforts. That's what gets me out of bed every the morning.

Collapse -


by illilli In reply to You've got big problems

It's been my experience that CIO's don't always know the best way to run IT production shops. Otherwise, our friend would have final say on all production database issues (and final responsibility for it's failure). It sounds like the DL05 has a CIO similar to our CIO. My company's CIO couldn't spell development, but he sure has a spiffy network for e-mail. I don't have the heart to tell him that there are other tools besides e-mail that use the network...(laugh), and the computers for that matter. I wonder what he thinks those computers are used for?

Collapse -

Not too hard to understand

by amcol In reply to CIOmyGOD

CIO's aren't supposed to know how to run production shops...that's the job of the CTO. If you work in an organization where the CIO and CTO are the same person and your C-level (whatever the title) is completely non-technical then you have a legitimate gripe. However, you have to understand the distinction between the functions and responsibilities of the two positions, which are quite different.

Bear in mind also that even in a small shop with only a CIO that person can't know everything and must cherry-pick which issues need attention. You talk derisively about the e-mail system, but that's what the rest of the organization principally sees...including the folks who pay the CIO's salary, the same folks who understand far less about technology than your developmentally challenged CIO. What do you think the CIO is most likely to put at the top of his/her list?

Collapse -

Picking your battles

by CharlieSpencer In reply to I'm with ya!

Back to your original question. Compared to the other problems you're fighting, this one isn't worth it. Sync the dev anyway they want, be grateful you're doing it only twice a day, and save your ammo to change something with more potential payback.

Related Discussions

Related Forums