Emerging Tech

Human error the cause of WGA meltdown


The facts are in. The weekend snafus involving WGA that resulted in validation and activation problems for some 12,000 Windows Vista systems was caused by human error.

According to Microsoft, the problem began when preproduction code was sent mistakenly to production servers. A rollback resolved activation problems within 30 minutes. However, it failed to solve the problem for the validation portion of the system, which suffered a downtime of some 20 hours.

Excerpt from eWeek:

Microsoft, based in Redmond, Wash., confirmed Aug, 27 that the problems with processing validations started at about 3:30 p.m. PDT on Friday, Aug. 24, and continued until around 11:15 a.m. Pacific Time on Saturday.

Excerpt from the the WGA Blog:

Nothing more than human error started it all. Preproduction code was sent to production servers. The production servers had not yet been upgraded with a recent change to enable stronger encryption/decryption of product keys during the activation and validation processes

While the response to the activation issue was quick (less than thirty minutes) the effect on our validation service continued even after the rollback took place. We expected the rollback to fix both issues at the same time but we now realize that we didn't have the right monitoring in place to be sure the fixes had the intended effect.

Microsoft is keen to stress that its WGA system is designed to default to genuine if the server is disrupted or unavailable.

In its own words: "If our servers are down, your system will pass validation every time. This event was not the same as an outage because in this case the trusted source of validations itself responded incorrectly."

Does this report cause you to regain some of your confidence with WGA?

--------------------------------------------------------------------------------

Stay on top of the latest tech news

Get this news story and many more by subscribing to our free IT News Digest newsletter, delivered each weekday. Automatically sign up today!

About

Paul Mah is a writer and blogger who lives in Singapore, where he has worked for a number of years in various capacities within the IT industry. Paul enjoys tinkering with tech gadgets, smartphones, and networking devices.

35 comments
TG2
TG2

Human Error? ... Maybe microsoft should build a better human.... That couldn't be any worse.. could it?

TechExec2
TechExec2

. *** Satire *** TheFakeNews has obtained a copy of a confidential e-mail containing the first draft of Microsoft's blog post covering the WGA meltdown. --------------------------------------------- Hi FakeAlex, Great blog post! You KNOW I agree with your views expressed here. But, remember that we're trying to dial it back a notch or two. Customers are getting really upset. Customers are getting screwed. We know it. They know it. But, we shouldn't be overly provocative just because it feels so good! :-) Please revise this so it isn't quite so harsh. Thanks, FakeBillG P.S. You're doing a great job, Lexie!! Expect to see some more stock grants on your next statement! Keep up the good work! ----- Original Message ----- From: Fake Alex Kochis To: Fake Bill Gates Sent: Tuesday, August 28, 2007 11:19 AM Subject: Blog post covering the WGA meltdown (first draft) First Draft (unapproved) [b]So what happened?[/b] Now that we have had time to dive into the details more on what happened that caused the temporary problems last weekend, I want to share what we found so far. We still have more work to do, but we now understand what occurred and why. We've also implemented several changes to prevent this kind of thing from happening again so we can keep the whiners under control. [b]What exactly happened?[/b] It wasn't much, really. An underpaid foreign intern (who SHOULD have an H-1B visa) working out of the new Microsoft office in Vancouver, Canada, screwed up! That's it. End of story. Except, of course, that he is now hanging from his thumbs for a while now that is. :^0 If the whiny U.S Congress would get off their duffs and give us all of the H-1B visas we paid them for, nobody else will have to get hurt. Some people STILL don't understand who is running things. [b]Why did it take so long to fix?[/b] It didn't, you pathetic whiners! It was over almost before it started. Only 12,000 people were affected! Do you realize how insignificant that number is?! We have hundreds of millions of customers! A lot of you are stealing our stuff anyway. You know it. We know it. You should count yourselves lucky that we don't do it again on purpose NEXT Friday, and take [u]all three days of the long weekend to fix it[/u]. Don't bite the hand that feeds you. Think about it. [b]If the servers are down, why don't you just assume the systems are genuine?[/b] Are you friggin' kidding me? We assume EVERYONE is not genuine. You have to prove you're genuine to us, over and over and over, as many times as we demand it. Windows is OUR friggin' operating system. You're lucky we let you run it at all. [b]What changes have you made?[/b] Aside from the intern hanging from his thumbs, not much. [b]Why were some customers told that this problem might continue for days?[/b] It was a joke! Get it? My bad. :^0 [b]What were customers experiencing?[/b] Panic in the streets! We were laughing so friggin' hard! :^0 To watch people run around like chickens with their heads cut off that they might lose contact with "their" Windows! It's just a reminder that we've got you by the short hairs. I also want everyone to know that I am personally very disappointed that this event didn't last a few days longer. As an organization we've come a long way since this program began and it's should be clear to all of you that we mean business! Stop using Linux you pathetic whiners! IBM thinks they invented the "IBM PC", but they were wrong back then. Now you think it is your PC, and you are wrong now. It is OUR PC. And, don't you forget it. If you don't do as we say, we'll shut you down faster than you can say "Windows Genuine Advantage". Say it! Right now! All of you! As always, please send your feedback to me through the blog (you can use the email link in the upper left hand corner of the page) or post comments. But, don't expect a reply. Thanks, FakeAlex

TechExec2
TechExec2

. [b]Microsoft says:[/b] [i]"...If our servers are down, your system will pass validation every time. This event was not the same as an outage because in this case the trusted source of validations itself responded incorrectly..."[/i] Microsoft's lack of veracity here is very disturbing. If WGA verification really worked this way, all a pirate would have to do is make it [u]appear[/u] that the WGA servers were down and the pirate's system would "pass validation every time". Only someone who is begging for any reason to believe Microsoft is a good company would believe a lie like that. WGA does not work that way. WGA would be worthless if it did. In fact, if it did work that way, I might run some Windows and block access to the damn WGA servers! :^0 Secondly, playing semantic games with what the word "outage" means is insulting. If the system is not working, its an outage. Period. There was a 20 hour outage. And Microsoft wants to be an online service company now? [b][i]Does this report cause you to regain some of your confidence with WGA?[/i][/b] Absolutely not. I have had ZERO confidence in the new aggressive WGA since I first heard about it back in February. This incident, and the disturbing way that Microsoft has handled it, has reaffirmed my February decision to permanently leave Windows for Linux. I very happily run Kubuntu Linux today. As far as I'm concerned, Windows is a dead operating system.

paulmah
paulmah

Does this report cause you to regain some of your confidence with WGA?

sundarhp
sundarhp

Oh dear, one more filthy attack on H1Bs. Grow up fella. In the time you spent typing such a lenghty fake email, had you taken a minute to visit google.com, IBM.com, HP.com and even Microsoft.com, you will know majority of the top "fellows" and scientists of these companies are foreign born, speicifically from India, brought on H1B.

brian.mills
brian.mills

That's pretty funny, yet sad at the same time, since that seems to be pretty much the atitued that M$ has taken in dealing with their customers. I wonder if they compare notes with the RIAA and the MPAA on how to screw their customers... --Edited because my smiley wasn't showing up right, so I just deleted it.

Applicant_2000
Applicant_2000

It proves that Microsoft's Live Servers ARE, no doubt as shown, that WGA could be an inaccurate or useless piece of trash, trashing other people's computers.

Chuckmeo
Chuckmeo

Windows remains what it has always been--toy software made by monkeys. Why people tolerate the endless succession of security flaws, piratical pricing and monopolistic behaviour of this vendor year after year after year is beyond me.

Johnj4269
Johnj4269

Error, error, error, (failed to think) error, error, error, error

boxfiddler
boxfiddler

to have some confidence in the first place in order to regain it in the second? Didn't have it in the first place, not me.

HAL 9000
HAL 9000

It's good for M$ and Lousy for M$ customers who get screwed yet again. Col

perfesser9
perfesser9

For Microsoft to allow this is a complete confidence loser. What "errors" will occur next? "Pre-production" driver upgrades that cause unrecoverable BSODs? If M$ can't control the software they install in their own computers, why am I supposed to feel better about what I'm downloading to mine?

Locrian_Lyric
Locrian_Lyric

It's gone from less than zero confidence to zero confidence.

TechExec2
TechExec2

. [Update: I later found out to my surprise that "sundharp" was correct and I was wrong. The H-1B "non-immigrant temporary work visa" explicitly allows the visa holder (and his family) to live and work in the United States for as long is it takes to process his application for permanent residency, not just for the 6-year duration of the visa. So, it's official! The Federal Government has a "War on Programmer Salaries". Hundreds of thousands of programmers (and their families) have used the H-1B "non-immigrant" visa to immigrate to the USA. See later posts in this section of the discussion for more information.] [b][i]"...In the time you spent typing such a lenghty fake email, had you taken a minute to visit google.com, IBM.com, HP.com and even Microsoft.com, you will know majority of the top "fellows" and scientists of these companies are foreign born, speicifically from India, brought on H1B..."[/i][/b] If what you say is correct, then you are providing evidence that H-1B visa program is corrupt and being abused to obtain cheaper workers. The H-1B visa is a [u]temporary 3-year work visa[/u] that can be renewed ONCE for a maximum of 6 years. The H-1B visa is supposed to help resolve the [i]~claimed~[/i] temporary worker "shortages". There are very strict requirements that the hiring company must demonstrate they have searched high and low and "cannot find" an American worker to do the job. They are not supposed to be used for the permanent immigration of cheaper foreign computer programmers who become [i]"the majority of the top fellows and scientists"[/i] in major USA corporations. This is the tip of the iceberg http://www.youtube.com/watch?v=TCbFEgFajGU You don't know what you're talking about...fella.

brian.mills
brian.mills

What I think is sad is the fact that the US has to import these types of top people from India rather than promoting our own people to these positions. Have we really gotten so behind the rest of the world in business sense that we don't have anyone qualified for the top positions without looking overseas? Don't get me wrong, I'm all for immigrants succeeding and making significant contributions to society they've adopted as their own, but I'm also for people succeeding in the society they are born in. Enough ranting, though. It's time for a holiday weekend.

Neon Samurai
Neon Samurai

Those seem to be the big reasons from my experience. Lock In: If you choose one MS product, your stuck choosing the rest of the product line (ie. Exchange means using Win Server as a host with AD and Outlook on the workstations which means choosing Win Workstation.) If you choose MS Office, your data formats are locked into MS product or at best, work semi broken with any other software offering. (empowering the end user by using indistry standards would just not be in the interests of MS shareholders; screw what's in the interests of the end user) The early stage lucky strike: MS got in when the computer industry was very young with a couple of very strategic choices by Mr Gates. Until they got lucky, MS was an obscure software company near bankruptcy. Even then, it was the strength of business strategy rather than product quality that made the difference. Marketing: We can't forget marketing. MS has successfully sold CEOs and endless lists of very intelligent people a polished, lipstick wearing steaming turd. And still, people run out to say "here, take my hard earned money for your well marketed below average quality product" every time the gross profit tells MS it's time to release a new version. As for why people continue to throw money at Redmond; who knows but as long as they do, I'll have to know the products well enough to support them.

TechExec2
TechExec2

I later found out that sundharp was right and I was wrong. I just updated my previous posts to reflect that. Thanks.

Genera-nation
Genera-nation

More Information at the above.... A US H1B visa is typically valid for up to six (6) years and entitles your spouse (husband/wife) and children to accompany you and 'live' in America. One of the main advantages of the H1B visa (US work permit) is that it is a 'dual intent' visa which means that you can apply for a Green Card (Legal Permanent Residency). To obtain an H1B visa to work in the USA.... * the 1st step is that you Must find an H1B Job with an H1B sponsor company. * Individuals can Not sponsor/apply for your own H1B visa - only your new employer can.

Tony Hopkinson
Tony Hopkinson

This is redefining illegal immigrants as temporarily legal. Makes you want to puke doesn't it?

TechExec2
TechExec2

. [b][i]"...Can an employer 'change their mind' about whether they sponsor the guy for citizenship? That would be nuts in the vice big time wouldn't it?...[/i][/b] Yes they can (unless I'm missing something else). They can also fire him. If fired, if he isn't picked up by another employer who qualifies for an H-1B visa job, he must return to his home country. This is part of why this whole thing is so wrong. This is why these foreign workers are so desirable to USA corporations. This is as close to "indentured servitude" as it gets in a modern western society with the rule of law.

Tony Hopkinson
Tony Hopkinson

about whether they sponsor the guy for citizenship? That would be nuts in the vice big time wouldn't it?

TechExec2
TechExec2

. Well, I have a red face now. This proves I did not read the entire law myself. What I did read did not cover this. [b]H-1B Visas provide for "dual intent"[/b] You are correct. The H-1B visa law (1) specifically provides for "dual intent" (2). The H-1B visa is a non-immigrant temporary work visa. But, H-1B visa holders are permitted obtain sponsorship from their USA employers and to apply for permanent residency. And, if their applications take longer than the maximum 6 year duration of the H-1B visa (and this is typical), they are permitted to stay and work [u]as long as it takes for them to get their permanent residency[/u]. They don't move to the front of the immigration line. They just get to stay and work in the USA indefinitely on an expired temporary non-immigrant work visa. My God those lawyers in Washington D.C. are clever! That is simply brilliant! [b]What does this mean?[/b] It means things are worse than I thought before I spoke with you. This means the H-1B visa program is officially and explicitly a conduit for USA corporations to obtain cheap (3) foreign high-tech workers that displace Americans. These people do what they are told when they are told because they want to immigrate to the USA and will put up with abuse that American citizens will not. ------------------------------------------ (1) H-1B Visa http://en.wikipedia.org/wiki/H1B_visa (2) Dual Intent http://en.wikipedia.org/wiki/Dual_intent (3) About "cheap" My use of the word "cheap" is not intended to impugn the foreign workers in any way. In the absence of the foreign workers, the pay rates and working conditions of computer programmers working in the USA is driven by market forces (competition, supply, and demand). Employers compete to get the best workers by raising pay and working conditions. By injecting large numbers of foreign workers who want to immigrate to the USA, that competition is greatly diminished and all USA workers lose for it. USA corporations [u]prefer[/u] foreign computer programmers who want to immigrate to the USA for obvious reasons. The H-1B visa program is fraudulent and corrupt. Human to human? I'm happy for those who benefit from it. I'm very sad for the Americans, and their families, who are hurt by it. There are large numbers of them. If you own an "average" home in the Silicon Valley of California (cost: $700,000), you cannot compete economically with a cheap foreign worker on an H-1B visa. You MUST be paid more than that or you will be driven into bankruptcy. [b][i]"...I cant believe I am doing this..."[/i][/b] I don't know why you say this. A proper discussion involves substantiation when there is disagreement. Otherwise we just have people throwing around their uninformed opinions and there is nothing gained by anyone. I have learned something from this exchange with you, as have some of those who read it. Thank you for substantiating your position. I expect myself to do the same and I do. But, occasionally I am embarrassingly wrong as I was in this case.

M_Ski
M_Ski

You stated: "H1B is meant to be a dual intent visa, with potential for immigration." Then quoted as evidence: http://www.uscis.gov/propub/ProPubVAP.jsp?dockey=d1e25ad63f1da92f551bcd3e2a5c69d9 So a rule that prevents an INS agent from dening an "application for adjustment" based upon the applicant's current non-immigrant status, or the fact that an H-1b holder left the country without following proper procedures, is really the whole purpose of the H-1b. (Note: the "Dual-Intent" seems to be on behalf of the H-1b holder who is also applying for citizenship, not of the H-1b process itself.) One must be in prime physical condition to make that leap...

Tony Hopkinson
Tony Hopkinson

Mix government and IT and you get something permanently temporary.

sundarhp
sundarhp

but if it clears any doubt http://www.uscis.gov/propub/ProPubVAP.jsp?dockey=d1e25ad63f1da92f551bcd3e2a5c69d9 (A) H and L Nonimmigrants with ?Dual Intent? . Under 8 CFR 214.2(h)(16)(i), 8 CFR 214.2(l)(16) and 245.2(a)(4)(ii)(C), H-1 and L-1 nonimmigrants, and their dependent family members, are able to maintain and to extend their nonimmigrant status while their applications for permanent residence under section 245 of the Act remain pending. This rule also addressed the issue of the eligibility of these aliens to travel outside the Untied States without abandoning their applications for adjustment of status.

TechExec2
TechExec2

. [Update: I later found out to my surprise that "sundharp" was correct and I was wrong. The H-1B "non-immigrant temporary work visa" explicitly allows the visa holder (and his family) to live and work in the United States for as long is it takes to process his application for permanent residency, not just for the 6-year duration of the visa. So, it's official! The Federal Government has a "War on Programmer Salaries". Hundreds of thousands of programmers (and their families) have used the H-1B "non-immigrant" visa to immigrate to the USA. See later posts in this section of the discussion for more information.] [b][i]"...H1B is meant to be a dual intent visa, with potential for immigration..."[/i][/b] This is completely false. Cite the H-1B visa law text that supports this claim of yours or retract it!

sundarhp
sundarhp

H1B is meant to be a dual intent visa, with potential for immigration. So, just because these guys happened to stay back beyond 6 yrs doesn't mean the system is corrupt. 1) Highest number of patent holder for IBM - Ravi 2) VP for HP's cash-cow printer business 3) Pepsi CEO 4) http://www.google.com/intl/en/corporate/execs.html and 100s more. If you think these guys come for cheap, I cannot begin to tell you how naive it is...

jdclyde
jdclyde

Someone from friggen Jersey complaining about others not speaking english clearly..... :0 :p

HAL 9000
HAL 9000

[i]By the way, to all Microsoft's bashers out there, nobody is perfect, you don't like Microsoft then stop using it and spare us your comments. All products have problems. By the time a piece of software is perfect, it is already outdated.[/i] While I part way agree with this does it have to be so savage? I can live with WGA shutting down 1 Machine that isn't a problem but when it shuts down every machine in one place because they are on a Volume License with the Same Key that is a different story and makes things unworkable. If you think that this is acceptable just how long do you think that it takes to do a Repair Install of 2,500 Desktop units so that you can have a working OS again and then block the WGA Server to stop the same thing happening again? Of course M$ didn't get blamed for this but I did and M$ even accepted a Piracy Report on the software that they sold to this place but accepted the wrong information that I sold it. Perhaps you would consider it acceptable to be accused of this but I can assure you that it's no fun and made getting a replacement Product Key much longer than things needed to be. Granted this was with XP Pro, Vista which currently is not usable to business would be considerably worse. Col

Locrian_Lyric
Locrian_Lyric

I HATE getting people on the phone who can't even speak English clearly. I'm half deaf and can't understand a damn word they sya.

jdclyde
jdclyde

just like the outsourcing of the help desk, it was not done to improve performance, it was done to lower costs. Find a country with a devastated economy and they will feel lucky to make 1/2 of what techs elsewhere make.

Locrian_Lyric
Locrian_Lyric

It's just that Americans don't think too highly of working in IT for 10 dollars an hour.

MadestroITSolutions
MadestroITSolutions

I am an immigrant and I understand your frustration. I totally agree with you. Unfortunately that's just the way it is. This country's education system needs a major overhaul. I went to a University in New Jersey, and the stuff they were teaching me in my 4th year I had already learned in 12th grade of high school back in my country. The Math "aptitude test" I took to join College could be done by my 13 year old cousin. The classrooms had 20 people at the most and a bunch of empty seats. That shows you how much interest american teenagers have in getting an education. Hell, this country gives tons of money away in grants for education and they still can't get them to attend. The biggest issue I think is the living standard. You can just get a mini van and do deliveries and make a decent living, so who needs college?... Not that there is anything wrong with being a driver. Granted, there are great professionals out there I have had the pleasure of working with, but overall, education level is poor. In all the places I have worked in the last 7 years, at least 90% of the developers I have met are immigrants. I also sense a lack of moral and cultural value teaching in our education system. I don't say all these things to bash our education system or say my country's is better. I say it out of frustration as well as this is the country I live in that has given me so much and this is the country where my children will live in all their lives and it makes me wonder where we will end up at this rate. By the way, to all Microsoft's bashers out there, nobody is perfect, you don't like Microsoft then stop using it and spare us your comments. All products have problems. By the time a piece of software is perfect, it is already outdated.

TechExec2
TechExec2

. [b][i]"...polished, lipstick wearing steaming turd..."[/i][/b] ROTFLMAO!! :^0 :^0 :^0 Very funny! Thanks!