Thursday, April 02, 2009

Hey! You! Get off of that cloud!

Or Microsoft won't support you.

We've had an interesting several days dealing with Microsoft at work. BigFix recently signed an Enterprise Agreement with Microsoft, where we committed to X licenses of the workstation OS, and put a number of other things under Select Agreement, including Server OSes, Exchange, SQL Server, MSDN subscriptions, and so on. This came with a few free support calls.

Our OS X and iPhone users (in particular, our CEO) have been anxious to get on Exchange 2007 for the rumored improved Entourage and iPhone support around calendars. So when our CFO wrote the large (for us) check to Microsoft, the IT Team committed to implementing Exchange 2007 in an aggressive time frame. Currently, we're using Exchange and OWA 2003.

Doing some research, it looked like the best option was to build new Exchange and OWA machines and migrate mailboxes. It also looks like the best OS choice is Windows 2003 Server Enterprise 64-bit. We read some documents that indicate Exchange 2007 isn't fully supported on 32-bit Server, and has only just been qualified on Server 2008.

We put Exchange itself on physical hardware for performance reasons. It's probably not really necessary, but we're being conservative. We used a Dell 2850 with about 1TB of disk and 32GB of RAM that was a VMWare ESX server until I replaced it with an even larger Dell R900. It's running Windows Server 2003 Enterprise R2 64-bit. The OWA machine doesn't need any particular performance characteristics though, so we decided to put it on a VM. It's on the same OS. No problems running 64-bit guests, by the way. We do it all the time.

Like many companies, we're trying to virtualize a lot of our infrastructure. We've made a fairly large investment in VMWare's enterprise products for a company our size, especially in our Engineering organization. I won't get into the benefits here, but for us they are substantial, and our entire disaster recovery plan is tied to VMWare ESX.

Things were on schedule with the Exchange 2007 configuration. In the interest of time, we had made one support call to Microsoft for install problems on the physical hardware. It burned the equivalent of $299, but for our schedule, it was easily worth it. Exchange was working.

We ran into a second issue with OWA 2007. My sysadmin was having trouble getting Outlook Anywhere to work correctly with Outlook 2003 and Entourage. He called again. This time, while the Microsoft support engineer was remote into our OWA server, he saw VMWare Tools in the Add/Remove Programs list. He asked, and we said yeah, it's a VM.

He said he could not support us, closed the ticket, and advised us to rebuild on physical hardware and call back. The support engineer also said that if we had had Premier Support, that he could "Look into it." He cited this article:

I'll summarize it: Microsoft only supports virtualized Windows and MS apps if you use Microsoft virtualization software.

That had never occurred to any of us in the IT department. That policy is so ridiculous as to defy belief.

I complained into the air on Twitter. I got two categories of response: Lie to Microsoft Support, and No, they do support it. It's called the SVVP.

Sure, we're willing to lie to support. We just didn't know it was necessary, and we got caught this time.

By the way, I'm going to jump ahead in the story for a moment and say that yes, we did rebuild OWA on physical hardware and call back. And it turns out that the problem was on the Exchange server, NOT the OWA server. So no, it's not possible that VMWare was a factor, and yes, we did waste days and slipped our schedule for no good reason. I say this mostly to save you the trouble of trying to fix my technical problem, it's already done.

And of course, that's not the real issue.

During these several days while my sysadmin gave up and build a physical box to appease Microsoft Support, the rest of us were complaining bitterly to our Microsoft sales rep. We still could not believe that they really intended to have that as a policy. He insists that they did. He knows, because he has had "lots of customers complain about it."

What about the SVVP, I asked my sales rep? Both a Microsoft employee and a VMWare employee pointed out to me on Twitter that ESX IS supported. Nope, my sales rep says that's only for the Windows OS itself.

But wait, the VMWare guy pointed out to me that Exchange on VM is specifically covered under the SVVP. Surely this means I'm good, right? This is just a case of Microsoft Support not being up on the latest Microsoft policies?

Nope. That article only covers Exchange 2007 SP1 (good) on SVVP virtualization software (good) on Windows Server 2008 (bad, I'm using Server 2003.)

So yes, they STILL turned me down for support on ESX. But they would support all of it if I was using Hyper-V.

This is far worse than my little problem not being handled. This would seem to indicate that Microsoft intends to qualify every single app they produce as being covered on VMWare or not. And only the versions that they feel like. And only if it's on a Windows version they want to cover.

So the latest set of articles on how to tune SQL Server 2005 on ESX? Forget it. It's not supported.

It's really hard to not immediately leap to accusing Microsoft of more anticompetitive behavior and vendor lock-in for their own virtualization technology.

Does Microsoft qualify every individual app on the hardware in the supported hardware list? Of course not. If the OS works, the apps should work. That is the basic job of the OS, yes? To abstract the hardware for the apps? So if Microsoft has qualified Windows 2003 on ESX, why should they decline to support OWA on it?

Is there an Exchange 2007 SP1 supported hardware list somewhere I'm not aware of?

At my most generous, I can assume that Microsoft Support is just not aware of Microsoft's own policies on this topic. And Microsoft Sales isn't either. My rep still says he can't help me. I can even see wanting to qualify Microsoft OSes on ESX "hardware", just like they would on Dell or HP. (Though when is the last time Microsoft Support even ASKED you what hardware you're running on?)

But to try and take a policy that every app needs to be qualified individually, down to the service pack level? Unless you're on Microsoft's virtualization technology?

That's just quite possibly criminal.


Domdingelom said...

I remember when I was trying to build in an extra CPU into a server and it broke (one of the pins got bended). I called support and told them the story. The guy on the other end of the line said that he was sorry but it was obviously a human mistake and he could not help me. He also said that it would have been a different story if I would've said that it had come broken out of the box. That's when I said "OK, let's start over, I opened the box and I saw that the CPU was broken" ... he registered my call and an engineer came on site the day after to replace the CPU.

All this to say, you have to understand where support is coming from. They obviously have instructions. There are some issues re. Exchange/OWA/VMWare, if you find the solution, I'm interested to learn what it was.

thanasisk said...

Stating the obvious:
Microsoft is really a conglomerate of smaller companies. It is only natural that sync errors (SVVP vs Support) errors to occur. To continue say that they will not support you unless you are using an MSFT virtualization technology is quite dangerous for them (think "Monopoly" :-) )

Scott Pinzon said...

Microsoft must be telling a LOT of callers they are not covered. F5 conducted surveys at major virtualization conferences in the US in Q4, 2008, and found that 75% used VMWare ESX, while only 12% used Microsoft Hyper-V. (F5 has a whole white paper coming out on their survey, soon.) I am very interested to learn whether this tactic of trying to bully users into Hyper-V succeeds.

justpaul said...

Re: 32 vs. 64, i thi=ought only the technical preview ran on 32, the retail Exchange 2007 onoly runs on 64. I am curious why enterprise vs. standard 64. Is it the amount of cores? memory? something I have no idea? There's a decent spread between standard and enterprise. I'd love to know why you guys went ent. We are facing the same decision (though implementation most likely in 2010.)

Anonymous said...

Hi Ryan,

Posted via twitter thought I'd get more verbose here.

The issue you're running into is that you think that by licensing MS software you get to decide how you use it. The reality is, it's not yours. And being a software pirate, or lying to software providers doesn't fix the problem of the licensing issue that's in front of your company.

If you need the software, use it. If you can live without it, or use an alternative that meshes with your infrastructure architecture better, then do a POC and deploy it on VMware. There are alternatives out there, and maybe if more management chains realized how inflexible this particular software vendor is, and also that there are options out there, that work with your platform of choice, that mesh with your DR plan, things would change. By not playing by the rules, you don't change the game.

Do the right thing. Keep it in a supported configuration, you don't want to cause grief for your company if things break a month from now. Then give your management the information that MS in it's current configuration can't fit in your Disaster Recovery solution.

Then, and only then, will your licensing issues be addressed properly. Even if all it takes is an upgrade to Server 2008.


Ryan Russell said...

@justpaul Exchange 2007 pretty much requires 64-bit. The box we happen to be putting it on has 32GB of RAM and 8 cores. This is the box that's not/not going to be a VM. I didn't check myself, but my sysadmin said that we wanted Enterprise for the hardware.

Ryan Russell said...


re: I get to decide how to use it. You bet I think that. I'm the consumer. Unreasonable EULA clauses don't get upheld in court. A separate question is whether you will be supported. I also think consumers have a right to demand what we want there.

re: Lying and pirating. Who is pirating? You're not accusing me of that, are you? But no, lying won't change the policy. Complaining might. I'm complaining.

This wasn't covered here, but let me volunteer that we have hundreds of Windows VMs, mostly in our dev/test environment. They work fine. It's not a question of whether it works or doing a POC, it's a question of Microsoft supporting their products.

And finally there's the big elephant in the room when it comes to "use it like they say or don't" arguments, Microsoft is a monopoly.

Ryan Russell said...


Thanks for the numbers, I have suspected it was something like that. Just further emphasizes how silly it is to try and claim they won't support it.

Anonymous said...

Re: re: I get to decide how to use it. You bet I think that. I'm the consumer.

You comment is valid. You are a consumer. However, you didn't buy it, you licensed it. There is a huge difference. THAT is the elephant in the room. The difference between buying a car, and catching a cab.

Unreasonable EULA clauses don't get upheld in court.

If you're already saying that you won't abide by the EULA,and you live in the US, it's as good as piracy. I'd rather avoid dragging my company into a courtroom to attempt to prove a point. My company tends not to like those sorts of things.

A separate question is whether you will be supported. I also think consumers have a right to demand what we want there.

You have the right to demand! I completely agree! I can demand that my car company build an espresso machine into the dash, that doesn't mean they're going to do it.

re: Lying and pirating. Who is pirating? You're not accusing me of that, are you?

No. You're saying you aren't going to abide by the EULA. That puts all your companies MS agreements at risk.

But no, lying won't change the policy. Complaining might. I'm complaining.

Complaining doesn't matter, not giving them money matters. If you complain and still license their product, do you think they care?

This wasn't covered here, but let me volunteer that we have hundreds of Windows VMs, mostly in our dev/test environment. They work fine. It's not a question of whether it works or doing a POC, it's a question of Microsoft supporting their products.

No, it's a matter of you using their products in a way that you've agreed to use their products, AND MS saying how they will support their software. There's even documentation out there that shows Exchange running BETTER in a VMware environment. Doesn't mean that MS will support it.

And finally there's the big elephant in the room when it comes to "use it like they say or don't" arguments, Microsoft is a monopoly.

I disagree with the elephant. I disagree with the room. Last comment I agree with, but they aren't the only "groupware" provider out there, just the biggest.

justpaul said...

I forgot where I read it, but MS has every intention of supporting all of the Exchange roles in a VM environment and they already support 4 of the 5 roles. The problem is that for the mailbox role, MS licensed the real-time engine from a third-party. That third party is not supporting their engine in a VM environment, so MS can't.

Ryan Russell said...


They DO support it, on VMWare on Windows Server 2008. They DO support it, on Windows Server 2003 & 2008 if you use Hyper-V.

I don't think that's the problem.

Anonymous said...

Just hide the Vmware tools icon so support can't see it. You don't need to lie, just don't volunteer the info that it is a VM.
You can bet that even if you are running the SVVP fully supported config, MS support will support you but I bet they will be less 'motivated' if they know its a VM.

Mark said...

Hi Ryan,

Here's what I found on the SVVP website. My interpretation is that Windows Server 2003 SP2 should be supported, just like Windows Server 2008.

"SVVP is open to any vendor that delivers a machine virtualization solution that hosts Windows Server 2008, Windows 2000 Server Service Pack 4 (SP4)and Windows Server 2003 Service Pack 2 (SP2) and subsequent service packs."

John said...

I recently scrapped Exchange in place of Kerio Mail Server. The cost of this was less than the CAL licenses that would have been needed. It can also email remote domain addresses without it bouncing back if the users aren't on the server CALs. It runs on a Dell in Suse Linux, using OpenSuse in a Novell XEN virtualisation. It has Homeworkers OWA, Iphones, Macs, Nokia's and Samrt phones all working great. No problems upto now, haven't rebooted for 3months.

Anonymous said...

I have been surprisingly impressed by Kerio myself.

I wanted to dislike it, but works great for more than a year now.

Anonymous said...

I have implemented Exchange 2007 - twice. Once with Windows 2003, Exchange 2007 RTM, second time with Exchange 2007 sp1, Windows 2007. In my entire IT career, I had called MS support once to fix a problem (I go way back to Exchange 5.4 and every version to present). Once I started working with Exchange 2007, I have called 5 times in less than a year. This sucker is hard to install, the learning curve very steep, and frankly, pretty fragile - especially IIS 7 to support the web side of things. If I had it to do all over again, I probably would skip 2007 completely, although the new Entourage 2008 for Exchange Server might make me change my mind.

I ran into the support issue on our old Exchange 2003 system, since the front end was virtualized on ESX.

I do hope your luck is better than mine with the fragility issues. I also assume you are never planning to install SP1 adn use it with Windows 2008 either, since that won't be an upgrade possibility.

Anonymous said...

Exchange 2007 only has a test version in 32 bit. They flat out will not support it on anything other than 64 bit OS and the 32 bit test version runs awful.

Ergo why SBS 2008 is only shipping in 64bit because there is truly only a 64bit OS.

As far as support they do support the paid VMware, but the free VMwareESX we get push back as well in the smb space.

HyperV is what they prefer.

Microsoft Exchange Server
Microsoft Exchange Server 2007 Service Pack 1 and later versions are supported. For more information about the support for Exchange Server, visit the following Microsoft Web site: (

Support policy for Microsoft software running in non-Microsoft hardware virtualization software:

I think you are getting the push back here because of the 2k3 sp2 platform. They are making all of their virtualization support policies for the 2k8 era software, not the 2k3 stuff.

site said...
This comment has been removed by a blog administrator.