Or Microsoft won't support you.
We've had an interesting several days dealing with Microsoft at work. BigFix recently signed an Enterprise Agreement with Microsoft, where we committed to X licenses of the workstation OS, and put a number of other things under Select Agreement, including Server OSes, Exchange, SQL Server, MSDN subscriptions, and so on. This came with a few free support calls.
Our OS X and iPhone users (in particular, our CEO) have been anxious to get on Exchange 2007 for the rumored improved Entourage and iPhone support around calendars. So when our CFO wrote the large (for us) check to Microsoft, the IT Team committed to implementing Exchange 2007 in an aggressive time frame. Currently, we're using Exchange and OWA 2003.
Doing some research, it looked like the best option was to build new Exchange and OWA machines and migrate mailboxes. It also looks like the best OS choice is Windows 2003 Server Enterprise 64-bit. We read some documents that indicate Exchange 2007 isn't fully supported on 32-bit Server, and has only just been qualified on Server 2008.
We put Exchange itself on physical hardware for performance reasons. It's probably not really necessary, but we're being conservative. We used a Dell 2850 with about 1TB of disk and 32GB of RAM that was a VMWare ESX server until I replaced it with an even larger Dell R900. It's running Windows Server 2003 Enterprise R2 64-bit. The OWA machine doesn't need any particular performance characteristics though, so we decided to put it on a VM. It's on the same OS. No problems running 64-bit guests, by the way. We do it all the time.
Like many companies, we're trying to virtualize a lot of our infrastructure. We've made a fairly large investment in VMWare's enterprise products for a company our size, especially in our Engineering organization. I won't get into the benefits here, but for us they are substantial, and our entire disaster recovery plan is tied to VMWare ESX.
Things were on schedule with the Exchange 2007 configuration. In the interest of time, we had made one support call to Microsoft for install problems on the physical hardware. It burned the equivalent of $299, but for our schedule, it was easily worth it. Exchange was working.
We ran into a second issue with OWA 2007. My sysadmin was having trouble getting Outlook Anywhere to work correctly with Outlook 2003 and Entourage. He called again. This time, while the Microsoft support engineer was remote into our OWA server, he saw VMWare Tools in the Add/Remove Programs list. He asked, and we said yeah, it's a VM.
He said he could not support us, closed the ticket, and advised us to rebuild on physical hardware and call back. The support engineer also said that if we had had Premier Support, that he could "Look into it." He cited this article: http://support.microsoft.com/kb/897615
I'll summarize it: Microsoft only supports virtualized Windows and MS apps if you use Microsoft virtualization software.
That had never occurred to any of us in the IT department. That policy is so ridiculous as to defy belief.
I complained into the air on Twitter. I got two categories of response: Lie to Microsoft Support, and No, they do support it. It's called the SVVP.
Sure, we're willing to lie to support. We just didn't know it was necessary, and we got caught this time.
By the way, I'm going to jump ahead in the story for a moment and say that yes, we did rebuild OWA on physical hardware and call back. And it turns out that the problem was on the Exchange server, NOT the OWA server. So no, it's not possible that VMWare was a factor, and yes, we did waste days and slipped our schedule for no good reason. I say this mostly to save you the trouble of trying to fix my technical problem, it's already done.
And of course, that's not the real issue.
During these several days while my sysadmin gave up and build a physical box to appease Microsoft Support, the rest of us were complaining bitterly to our Microsoft sales rep. We still could not believe that they really intended to have that as a policy. He insists that they did. He knows, because he has had "lots of customers complain about it."
What about the SVVP, I asked my sales rep? Both a Microsoft employee and a VMWare employee pointed out to me on Twitter that ESX IS supported. Nope, my sales rep says that's only for the Windows OS itself.
But wait, the VMWare guy pointed out to me that Exchange on VM is specifically covered under the SVVP. Surely this means I'm good, right? This is just a case of Microsoft Support not being up on the latest Microsoft policies?
Nope. That article only covers Exchange 2007 SP1 (good) on SVVP virtualization software (good) on Windows Server 2008 (bad, I'm using Server 2003.)
So yes, they STILL turned me down for support on ESX. But they would support all of it if I was using Hyper-V.
This is far worse than my little problem not being handled. This would seem to indicate that Microsoft intends to qualify every single app they produce as being covered on VMWare or not. And only the versions that they feel like. And only if it's on a Windows version they want to cover.
So the latest set of articles on how to tune SQL Server 2005 on ESX? Forget it. It's not supported.
It's really hard to not immediately leap to accusing Microsoft of more anticompetitive behavior and vendor lock-in for their own virtualization technology.
Does Microsoft qualify every individual app on the hardware in the supported hardware list? Of course not. If the OS works, the apps should work. That is the basic job of the OS, yes? To abstract the hardware for the apps? So if Microsoft has qualified Windows 2003 on ESX, why should they decline to support OWA on it?
Is there an Exchange 2007 SP1 supported hardware list somewhere I'm not aware of?
At my most generous, I can assume that Microsoft Support is just not aware of Microsoft's own policies on this topic. And Microsoft Sales isn't either. My rep still says he can't help me. I can even see wanting to qualify Microsoft OSes on ESX "hardware", just like they would on Dell or HP. (Though when is the last time Microsoft Support even ASKED you what hardware you're running on?)
But to try and take a policy that every app needs to be qualified individually, down to the service pack level? Unless you're on Microsoft's virtualization technology?
That's just quite possibly criminal.
Thursday, April 02, 2009
Saturday, March 22, 2008
Arr! VMWare is driving me nuts.
Several random VMWare things I want to throw out there that bother me.
At this point, I have used and continue to use most of VMWare's products. This started with Workstation back to 3.x.
Oh, at let me get my biases out of the way; I run a QA department, and we use VMWare for everything we can. Nothing better than being able to restore to a know state or save off a machine exactly where it is when exhibiting a problem. BigFix, where I work, also makes an agent that runs inside the management partition on ESX 3.x boxes.
VMWare Workstation - Great product, great price point. You can run multiple machines (a few), manage whole snapshot trees. Only really useful if you're in front of the box Workstation is running on. Gets the bleeding-edge features. VMs running under Workstation don't perform great, but are adequate if you give them enough physical RAM. Pretty much exactly matches expectations, but then it's the first product and is the one the others vary from. So in a very real way, this is what sets my expectations for the other products.
VMWare Server - The first larger VMWare purchase I made was GSX Server, somewhere around $3,000US for the software, and a $6,000 Dell 2U running Windows to put it on (BigFix's money, not my personal budget). Not bad, performance is still not great, slightly worse than Workstation. Might be because of remote access latency. Shareable, remote access built-in, which is key. Only one snapshot though, which is an immediate problem. I can manually backup machines at the expense of 30 minutes instead of 60 seconds, and disk space per copy is the same as the original rather than a fraction like a snapshot. But I found I could have a library of 30 machines, and run around 15 simultaneously, depending.
I originally assumed they had just left it out of GSX so far... or maybe, that was their hook to get people to go to ESX? I hadn't looked into ESX yet at the time. It's not a casual evaluation. That's about when VMWare made Server free. Hey, great right? No. There go my hopes of ever getting multiple snapshots on Server. Because VMWare would be insane to put that feature in the free product. For someone in my position, multiple snapshots are probably 40% of the advantage of ESX over Server. And I use ESX now, so why do I care? Because I can't give up Server! I have to keep using this intentionally crippled product. I'll get to why in a sec.
VMWare ESX Server (family) - At this point BigFix has standardized on ESX for as many QA machines as possible. (We have stuff that runs on Mac, Solaris SPARC, AIX PPC, HP-UX PA-RISC and Itanium, Windows Itanium, Windows Mobile on ARM. The x86 virtualization doesn't help much on those. It could with Mac, but Apple only just recently allowed OS X Server on VMs. When I'm trying to qualify our product on OS X, I can't go the hackintosh route. Also, I have a DLP product and some Wake-on-LAN functions I need real machines for. Oh, and I have an agent that runs IN ESX. I can't run ESX in ESX....)
But back to what I LIKE about ESX for a sec. It's the fastest of the bunch, scales better, has better remote access, better machine cloning, migration between physical ESX hosts and drives, and has MULTIPLE SNAPSHOTS. I put my team on ESX, and some of the install matrix stuff instantly takes half the time because of the snapshot feature alone. There's also a almost real infrastructure management. For my purposes, this means I get all my VMs in one window with one login. If you have more than one Server, then you log into each one separately (as far as I know. More on that in a sec, too.) I have as many as 30-40 machines running simultaneously per physical ESX box, out of a library coming up on 100, and it does a fantastic job at resource sharing the 8 cores and 16GB of RAM per physical box. It loves it some disk space, but that sort of thing happens when you build a hundred VMs averaging around 10GB each.
Sure, it's a little pricey. I think I'm paying $3000-4000 per ESX box, plus something for Virtual Center, and I'm not sure what else. I'm buying $9,000 Dell 2Us now, because ESX can actually make us of the resources. And I'm in for an external Dell SATA drive array, 15 400GB drives RAIDed, giving my 1TB on one ESX box, and 1.4TB on the other ESX box. I think we paid $15,000-$20,000 for that. I get less clear on the costs at this point, because I can now just budget for more capacity, and my IT department is buying it. We're in the process of picking on a 40TB SAN for the big cutover, where I bring some other groups into production on ESX who have been suffering with Workstation and piles of external 500GB USB hard drives. We have a tiny bit of production virtualization that VMWare constantly touts, but 90% of my ESX use falls under QA-style use.
Great, right? So one day, I grab the VMWare Converter tool (awesome tool!) to convert the last of my Server images over to ESX... and it balks. OK, no big deal.. I can make them again, they're just a few Win9x boxes, some Solaris x86 10... Hey, the Win9x OSes are missing from the list of standard OSes in the UI. I do some digging, and...
Windows 9x is not supported on ESX.
What? That can't be right... do some investigation... supported on Workstation... supported on Server. Not supported on ESX.
The Solaris x86 10 doesn't seem to work so well on ESX either, though support is claimed. But only starting at a particular patch level. Uh, I kinda need to test compatibility all the way back to no patches, guys. But I haven't finished my heroic effort getting it running on ESX yet. (Not that I should have to work that hard, of course.)
So in one shot, ESX has now forced me to maintain some number of Server machines. Sure, I already had to have piles of physical boxes for the random non-x86 unices. But I was so close on the Win9x. It should work. VMWare just doesn't want to. Can I have multiple snapshots on Server? No. Can I have Win9x on ESX? No. And I can't pay them for it, they don't want to.
While I'm complaining, there's one more thing I don't like about ESX (besides the usualy incremental stuff). I have no idea what the various ESX pieces do, or if I have them, or if I want them, or what kind of setup I need for them. I know I have ESX, Converter, and Virtual Center. I think I want VMotion. I think it does cool stuff with automatically balancing loads and migrating machines. I think I need a SAN for that. I sure hope my IT guy who spec'd that and the SAN out has it straight. I think there are bundles that have some of what I want. And I don't know what else I'm missing.
Like, I have Virtual Center. Does that help with my requirement for Server still? I don't think it does. I could be wrong. There's some ACE authentication product or something too, right? Why would I want that? What does it do?
Why did you buy Determina?
Now, if you actually know what you're doing with VMWare, you are assuming I haven't done my homework and haven't been to training and haven't been reading the docs and whitepapers. And you're right. But I'm the customer. I have entitlement issues. I define good products as ones that I can figure out without much work, that don't make me read the docs. I've been doing this for 25 years now, I like it this way. If I have to read your docs, then I lose for some reason. So when I can't figure out your product line a differentiation, that's ultimately our fault and you have made me bitter and/or sold me less. Make it simpler.
And then when I HAVE figured out your product differentiation when you didn't really want me to (i.e. your artificial limitations), that's not so hot either.
OK, I feel better thanks. And yes, for those of you who actually know the VMWare stuff in depth, PLEASE correct me.
BTW, what brought on the rant? I've got a presentation next week on malware analysis. I need Windows for that, and I'm carrying around a MacBook Pro with Leopard lately. So I bought a copy of VMWare Fusion straight from VMWare for about $70 yesterday. That's about half the cost of Workstation (Windows/Linux host only.)
It only does single snapshots.
Could I give you the extra $50 for multiple snapshots, PLEASE?! I only need this on my laptop when I'm traveling. I will use just as much ESX when I'm at work, I promise.
At this point, I have used and continue to use most of VMWare's products. This started with Workstation back to 3.x.
Oh, at let me get my biases out of the way; I run a QA department, and we use VMWare for everything we can. Nothing better than being able to restore to a know state or save off a machine exactly where it is when exhibiting a problem. BigFix, where I work, also makes an agent that runs inside the management partition on ESX 3.x boxes.
VMWare Workstation - Great product, great price point. You can run multiple machines (a few), manage whole snapshot trees. Only really useful if you're in front of the box Workstation is running on. Gets the bleeding-edge features. VMs running under Workstation don't perform great, but are adequate if you give them enough physical RAM. Pretty much exactly matches expectations, but then it's the first product and is the one the others vary from. So in a very real way, this is what sets my expectations for the other products.
VMWare Server - The first larger VMWare purchase I made was GSX Server, somewhere around $3,000US for the software, and a $6,000 Dell 2U running Windows to put it on (BigFix's money, not my personal budget). Not bad, performance is still not great, slightly worse than Workstation. Might be because of remote access latency. Shareable, remote access built-in, which is key. Only one snapshot though, which is an immediate problem. I can manually backup machines at the expense of 30 minutes instead of 60 seconds, and disk space per copy is the same as the original rather than a fraction like a snapshot. But I found I could have a library of 30 machines, and run around 15 simultaneously, depending.
I originally assumed they had just left it out of GSX so far... or maybe, that was their hook to get people to go to ESX? I hadn't looked into ESX yet at the time. It's not a casual evaluation. That's about when VMWare made Server free. Hey, great right? No. There go my hopes of ever getting multiple snapshots on Server. Because VMWare would be insane to put that feature in the free product. For someone in my position, multiple snapshots are probably 40% of the advantage of ESX over Server. And I use ESX now, so why do I care? Because I can't give up Server! I have to keep using this intentionally crippled product. I'll get to why in a sec.
VMWare ESX Server (family) - At this point BigFix has standardized on ESX for as many QA machines as possible. (We have stuff that runs on Mac, Solaris SPARC, AIX PPC, HP-UX PA-RISC and Itanium, Windows Itanium, Windows Mobile on ARM. The x86 virtualization doesn't help much on those. It could with Mac, but Apple only just recently allowed OS X Server on VMs. When I'm trying to qualify our product on OS X, I can't go the hackintosh route. Also, I have a DLP product and some Wake-on-LAN functions I need real machines for. Oh, and I have an agent that runs IN ESX. I can't run ESX in ESX....)
But back to what I LIKE about ESX for a sec. It's the fastest of the bunch, scales better, has better remote access, better machine cloning, migration between physical ESX hosts and drives, and has MULTIPLE SNAPSHOTS. I put my team on ESX, and some of the install matrix stuff instantly takes half the time because of the snapshot feature alone. There's also a almost real infrastructure management. For my purposes, this means I get all my VMs in one window with one login. If you have more than one Server, then you log into each one separately (as far as I know. More on that in a sec, too.) I have as many as 30-40 machines running simultaneously per physical ESX box, out of a library coming up on 100, and it does a fantastic job at resource sharing the 8 cores and 16GB of RAM per physical box. It loves it some disk space, but that sort of thing happens when you build a hundred VMs averaging around 10GB each.
Sure, it's a little pricey. I think I'm paying $3000-4000 per ESX box, plus something for Virtual Center, and I'm not sure what else. I'm buying $9,000 Dell 2Us now, because ESX can actually make us of the resources. And I'm in for an external Dell SATA drive array, 15 400GB drives RAIDed, giving my 1TB on one ESX box, and 1.4TB on the other ESX box. I think we paid $15,000-$20,000 for that. I get less clear on the costs at this point, because I can now just budget for more capacity, and my IT department is buying it. We're in the process of picking on a 40TB SAN for the big cutover, where I bring some other groups into production on ESX who have been suffering with Workstation and piles of external 500GB USB hard drives. We have a tiny bit of production virtualization that VMWare constantly touts, but 90% of my ESX use falls under QA-style use.
Great, right? So one day, I grab the VMWare Converter tool (awesome tool!) to convert the last of my Server images over to ESX... and it balks. OK, no big deal.. I can make them again, they're just a few Win9x boxes, some Solaris x86 10... Hey, the Win9x OSes are missing from the list of standard OSes in the UI. I do some digging, and...
Windows 9x is not supported on ESX.
What? That can't be right... do some investigation... supported on Workstation... supported on Server. Not supported on ESX.
The Solaris x86 10 doesn't seem to work so well on ESX either, though support is claimed. But only starting at a particular patch level. Uh, I kinda need to test compatibility all the way back to no patches, guys. But I haven't finished my heroic effort getting it running on ESX yet. (Not that I should have to work that hard, of course.)
So in one shot, ESX has now forced me to maintain some number of Server machines. Sure, I already had to have piles of physical boxes for the random non-x86 unices. But I was so close on the Win9x. It should work. VMWare just doesn't want to. Can I have multiple snapshots on Server? No. Can I have Win9x on ESX? No. And I can't pay them for it, they don't want to.
While I'm complaining, there's one more thing I don't like about ESX (besides the usualy incremental stuff). I have no idea what the various ESX pieces do, or if I have them, or if I want them, or what kind of setup I need for them. I know I have ESX, Converter, and Virtual Center. I think I want VMotion. I think it does cool stuff with automatically balancing loads and migrating machines. I think I need a SAN for that. I sure hope my IT guy who spec'd that and the SAN out has it straight. I think there are bundles that have some of what I want. And I don't know what else I'm missing.
Like, I have Virtual Center. Does that help with my requirement for Server still? I don't think it does. I could be wrong. There's some ACE authentication product or something too, right? Why would I want that? What does it do?
Why did you buy Determina?
Now, if you actually know what you're doing with VMWare, you are assuming I haven't done my homework and haven't been to training and haven't been reading the docs and whitepapers. And you're right. But I'm the customer. I have entitlement issues. I define good products as ones that I can figure out without much work, that don't make me read the docs. I've been doing this for 25 years now, I like it this way. If I have to read your docs, then I lose for some reason. So when I can't figure out your product line a differentiation, that's ultimately our fault and you have made me bitter and/or sold me less. Make it simpler.
And then when I HAVE figured out your product differentiation when you didn't really want me to (i.e. your artificial limitations), that's not so hot either.
OK, I feel better thanks. And yes, for those of you who actually know the VMWare stuff in depth, PLEASE correct me.
BTW, what brought on the rant? I've got a presentation next week on malware analysis. I need Windows for that, and I'm carrying around a MacBook Pro with Leopard lately. So I bought a copy of VMWare Fusion straight from VMWare for about $70 yesterday. That's about half the cost of Workstation (Windows/Linux host only.)
It only does single snapshots.
Could I give you the extra $50 for multiple snapshots, PLEASE?! I only need this on my laptop when I'm traveling. I will use just as much ESX when I'm at work, I promise.
Full RSS Feed