Behind the scenes at Redmond as Microsoft gets ready to declare the code for the Windows update finished.
By Ina Fried
Staff Writer, CNET News.com
REDMOND, Wash.—The last stop for Vista is a windowless conference room in Building 26, on Microsoft's sprawling campus in the Seattle suburbs.
Each day, members of the Windows team gather inside this "shiproom" to go over the bugs that remain, and to debate which of these can still be fixed in the days left until the product is declared finished, a milestone that is expected any time now.
The intense "end game," as these final weeks are known, is a well-worn tradition inside the shiproom, which is on the third floor of the Windows development building. The small room, with its dated, dark wood conference table has been the war room for every Windows release since Windows 2000.
On the wall are knick-knacks from past projects, as well as clocks showing the minutes ticking away in a dozen time zones. The clocks serve as a reminder that Microsoft has a deadline to meet. The company has scheduled a November 30 press event in New York to announce the availability to businesses of Windows Vista, while computer makers need to get the final code in order to finish their testing and get Vista on PCs in time for a broad launch in January.
The once-daily shiproom meetings have become twice-a-day events as the product has neared completion. Projected onto a screen is a list of unresolved issues that need to be addressed before Vista can leave. There were about five dozen such issues at a meeting last Wednesday morning.
Sven Hallauer, who heads up the process, moved quickly through the list as about 40 programmers, nearly all with a laptop in tow, worked to keep up. At each sticking-point, the person responsible for tracking the issue gave a short, one-sentence report on where things were at.
In one case, there was a bug in the Slovenian release of Vista. It was quickly tabled as not pressing, given that Slovenian is not in the first or even second wave of localized versions of Vista. Other reports came in—this software program has a hitch, this particular laptop has trouble waking up from sleep.
Some of the glitches were already known. Many were things that have already been fixed, and a few were too new and need investigating. None appeared to be a show-stopper.
Hallauer had predicted that the morning's meeting would be fairly short—maybe a half-hour. After 20 minutes, the group decided that things seemed pretty good. Perhaps they wouldn't need to revise the code again.
At the afternoon meeting, though, the team was forced to revisit that decision. It turned out that there was an issue within Vista's new diagnostics: if a piece of software failed to install properly, the system would nonetheless get a report that it had been successful. Hallauer and team decided to spin one more build.
It's all part of the process. Several times, Hallauer and others have thought that they had the final version of Vista done, only to find something that meant the team had to put in another fix.
Two weeks ago, Microsoft thought it had something that promised to be the final version. But within a couple of days, two new glitches had surfaced. The issues were arcane, but significant enough. In one case, there was a potential problem with burning DVDs. If a Vista machine attempted to burn information to a blank DVD directly from a network drive, there was a chance that data could be lost, if the network was slow. The other problem had to do with offline folders: Under certain circumstances, applications weren't being notified if the cache was full.
"That could end up with users losing their data or having a really bad experience," Hallauer said.
While it seems natural to go ahead and fix such bugs, changing the code at this point is a big deal. There isn't time for the full regression testing, which investigates whether a fix in one area has some hidden impact somewhere else in the system. Instead, teams must create solutions that only touch a part of the code and count on their ability to not break something elsewhere.
And not everyone agrees which things need to be tackled. The battles inside the shiproom can get testy sometimes. These days, there are certainly folks who feel Vista is ready to send on its way. Others keep lobbying for particular fixes, including some requests made late last week.
Hallauer said he doesn't see his job as just saying "no"—but at this point, it is certainly about only saying "yes" to the right things. "Through most of the product cycle, the teams are fairly independent," he said. "Now that we are at the end of the release cycle, it is more (about) taking stronger reins."
Sharks and limpets
While Vista is not glitch-free, the product is finally coming together. When Microsoft does find a bug, it gets classified into one of two categories.
One is "sharks"—bugs that everyone agrees need to be fixed before the product ships. And then there are "limpets," which are issues that can be fixed, but where the need is less critical. In those cases, the fixes are developed, but don't get implemented unless a shark comes along that they can use to float into the code.
Retiring Windows chief Jim Allchin doesn't like the shark and limpet analogy. To him, nearly every bug is a shark worth fixing.
"(If) there's a fix, I want to put it in," Allchin said. "It should be clear that date means not much to me, that quality is much more important."
But Allchin is finding plenty of resistance these days. Microsoft is under a fair bit of pressure to get Vista out the door.
The latest shark, though, means that he can get in one of the changes he wanted. For months, the company has been struggling with an issue in the Vista set-up process. As the operating system was loading, the screen would appear to freeze up, with no indication that the installation was still progressing—although it was.
Developers put that problem right. But as a result, a dialog box that asked users to identify the type of network they have was popping up twice.
To Hallauer, it was an issue that might or might not have justified a new build. Allchin was convinced it did.
"When I heard about it, I thought, there's no way...(We've got) to fix this," Allchin said.
The unrelated software-installing problem let Allchin win the day.
As the daily debate over whether to spin another build goes on in the shiproom, other teams spend their days putting the latest builds through their paces.
Until recently, Microsoft has issued a new internal release of Vista every day. That's a grueling process. Typically, its servers start cranking away at the raw code around 7 p.m., compiling it through the night, with the goal of releasing the new build by early afternoon the next day.
Down the hall from shiproom, Windows unit employees can pick up the latest builds. About 500 people pick up a DVD with new code in person each day, with many more getting the code over the network, and some even bringing their home machines into the office.
That list includes rank-and-file Windows employees, as well as some of the company's top brass. Allchin and his technical assistant, for example, are still trying to find bugs that the servers and development teams have missed.
"I'm doing video calls with my mom in Boston," Allchin said. "I'm doing remote assistance to jump into a machine, and then I 'remote desktop' from that machine to another machine."
Elsewhere, Allchin is testing a multimonitor set-up with four displays, including some in portrait mode. Paul Donnelly, who manages part of Microsoft's Vista test operation, has been doing the same thing for some time. As the finalization deadline has neared, he has added more systems to his office. As of last week, he had nine machines crammed into his office. He is among those who nearly always picks up the daily build.
"Pretty much every day since at least May 2005, I think I have installed on some machine," he said.
Donnelly, who tinkers with old cars and classic pinball machines in his spare time, said that he tries to do the opposite of what an IT manager would recommend.
He changes all the default settings, for instance. And instead of testing a clean installation on a new machine, he'll try to upgrade an older model.
"You find bugs," he said, "You absolutely find bugs that way." Luckily, he said, it is getting harder and harder to find issues that aren't already on the radar screen and being addressed.
"We're on watch right now...keeping an eye on things to make sure that we haven't missed anything," he said. "I haven't had any 'heart attack' issues arrive in the last few months."
But Vista's fortitude does not depend solely on the watchful eyes of Windows veterans like Allchin and Donnelly.
With each day's build, Microsoft is running a battery of automated tests against around 1,000 of the leading software programs. It has written 750,000 lines of code just to create the test patterns, which take 355 servers the better part of the day to run.
"Our job is to try and break the apps and find the bugs," said Mike Kirby, who runs the automated test lab. These days, though, the team is just hoping that each day's build doesn't bring up any new bugs.
In another building, individual software and device makers have their own private offices, where they can work on their own Vista-related issues. One of the key areas for Microsoft, beyond finishing its code, is getting hardware and software makers to get their products ready for the new operating system. To make that as attractive as possible, it has created a building on its campus just for them: the Platform Adoption Center, better known inside the company as the "high-touch" lab. The building, one of the hippest on Microsoft's largely bland campus, offers an inviting atmosphere with private offices, a lounge with a Xbox 360 game console and plenty of munchies.
"We try to keep them well-fed and well-caffeinated," said Dave Wascha, who helps lead Microsoft's effort to make sure other software makers have their products ready to go when Vista ships.
The companies that come also get their own rooms that lock with a code combination that only they know. They can use PCs from Microsoft, or bring their own machines. Either way, the computers can connect directly to the Internet without going through Microsoft's network.
"Essentially, this is their office," Wascha said.
The center has been home to 16,000 people since 2004, and is booked solid every week. It has been home to Microsoft's traditional partners as well as some of its fiercest rivals, many of whom did not want to be named.
One rival that has been public about the hand it received from Microsoft is the Mozilla Foundation, creator of the Firefox browser. In August, the open-source software maker an offer of help from Microsoft.
Another rival that credits Microsoft for helping it get Vista-ready is AOL. The Internet services provider went through the Redmond lab in July, while Microsoft engineers traveled to AOL's campus in Dulles, Va., in August and September.
"We worked through a ton of issues," said Julie McCool, the AOL vice president who manages the team that handled the Vista work. One of the many efforts the two companies worked together on was coming up with a way to let customers get a Vista-friendly version of AOL's software when they stick an older CD into a new PC. In the end, the companies figured out a way to alert people that the CD they pulled off a two-year-old magazine doesn't have the Vista version and to get that software from the Web instead.
McCool said that AOL has continued to meet weekly with Microsoft. Initially, the company had plenty of bugs it was working through, but in recent weeks it has been smooth sailing. "I don't think we had any big surprises in the past week," McCool said.
Eating their own dogfood
Microsoft's own work force is a key arbiter of whether a release is ready to go out the door. About 60,000 machines inside Microsoft are running Vista as part of the company's "dogfooding" process.
CIO Ron Markezich signed off last week, saying that Vista had met his goals—a step that has to happen before a product can leave Redmond.
"It's totally ready to go," Markezich said.
Microsoft is trying to do a better job with the final testing of Vista than it has in past versions of Windows. "We have to learn from ourselves," Donnelly said. "We don't have the ability to go to somebody bigger than us and go, 'What were the problems with your last release?'"
Donnelly, who has been at the company since Windows NT 3.5, said he remembers an early NT release over a Labor Day that was particularly hairy. "I just remember the pizza boxes stacking up in the kitchenettes," he said.
There's urgency, but no panic this time, he said: "You just don't see people running around like crazy."