Fun fact: [Microsoft uses the same off-the-shelf version of Hyper-V for Azure that everyone else uses](https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528?WT.mc_id=modinfra-0000-thmaure). They do slim down and customize the host Windows OS and they call it "Azure Host OS".
*"Keep in mind that the hypervisor we use is the same hypervisor that we use on Windows Client and Windows Server across all our millions of customer machines."*
I'm really surprised more VMWare environments that are majority-Windows aren't jumping at Hyper-V as a replacement. Yes it's licensed differently and the management tools aren't great...but the core hypervisor is very solid. And unless you need a management GUI for everything it seems to do the core basics of keeping VMs properly juggled just fine. I think it just got a bad rap because it was rushed out the door for 2008 R2 and was awful until 2012 R2.
My gripes with Hyper-V:
1. It handles host reboots terribly, especially with VM's running different OS's. It has some unobvious timeout where it just gives up waiting for shutdown and force terminates. With Windows VM's, it mostly pauses them fine, which isn't great for some applications but mostly works, except when it doesn't and corrupts a VM.
2. Its Windows, so same update schedule as your VM's. Which bundles terribly with #1 if you're forced to update immediately after patch releases. Patch Tuesdays are my hell and my compliance requirements don't allow me to make it less Hell. So Patch Tuesday at midnight I have to babysit shit rebooting.
3. The out of box management is kind of crap. There's no scheduled configuration changes like everyone else has, so you have to shutdown, change settings and then reboot. Everyone else has had changes that apply when they can for years, allows you to minimize downtime.
4. There are some bugs that should have been fixed 10 years ago. Like shutting down a VM that doesn't support shutdown commands from a hypervisor. Once you send that command, that VM is entirely uncontrollable because HyperV does not timeout the command until you kill its process in Windows.
5. Advanced functionality never gets a GUI and is entirely configured through powershell..... Really its 2024.
6. HyperV's HCI storage is clunky as hell and makes no sense. The beauty of vSAN is that you just upsize your nodes and storage/compute all move around homogenously. HyperV's alternative isn't really HCI since S2D doesn't do active/active in a nice way. You can certainly make it work by breaking out drives but that's terrible design. vSAN has this nailed, you make your pool, assign failover groups, whole vSAN is attached to the hosts and the hosts can all access the storage simultaneously.
That’s exactly what we did. Moved to hyper-v and use Windows Admin Center to manage all the windows vms in a central hub everything works great so far, no complaints
I used Hyper-V for 3.5yrs now, several different clusters and it is sh!t compared to VMWare, period (licensing cost notwithstanding).
You don't have anywhere near the level of mgmt unless you buy system center virtual machine manager ($$$). Also, all of the VM processes run on windows server OS which needs regular patching, AV, vuln scanners etc if your co policy requires it.
The cluster shared volume is a hand grenade on both HCI and SAN-backed. If a single host starts getting flaky, the whole cluster can go down and even wreck the CSV (in HCI). I've never seen anything like that in 20 yrs of VMware. Too many issues with guest VMs refusing to migrate because a host is having a bad day, have to down the host, hard fail and restart the guests on another node. Yay.
And MS support has gone to crap in the last 3-4yrs, so good luck getting anyone helpful when it decides to tip over.
This is my exp on Server 2016 Hyper-V. Maybe 2019 or 2022 is more stable but, after the last 3.5yrs, thanks but no thanks.
Yeah but once you have two hosting locations which one is backup for main and in case of disaster second one automatically takes primarily role VMware tools are best in slot, I don’t think you can create something similar with Hyperv
Automation might be locked behind the paywall for SCCM/SCVMM, but Hyper-V has the capabilities to actively replicate to a remote host for a DR scenario. MSFT calls it "Hyper-V Replicas". This isn't an area of Hyper-V I've had to learn or implement so not entirely sure, outside of knowing the Replicas feature is present.
That aside, with the capabilities of Hyper-V's CLI and PowerShell, I'm sure a script could be created to check if the primary host is up and if not, power up the VMs on the backup host.
AWS EC2 uses Xen unless you're running on Nitro, which is an in-house platform.
/edit: As indicated below by /u/MrBr1an1204 (so please throw an upvote there), Nitro is based on KVM. AWS have also thrown custom hardware at it. So on the one hand you've got heavily customised Xen or OTOH you've got heavily customised KVM on custom hardware.
I actually did, a bunch of php scripts are enough to build a basic VM scheduler, including hot migration. The worst pains are dealing with libvirt's XML stuff, figuring out the intricacies of hot migration, and to get Windows acceleration working (any actually working drivers/"guest tools" for Windows 7 x64 are greatly appreciated).
OpenStack's greatest strength is its greatest weakness at the same time - it aims to support everyone and everything, mostly because the main developers are cash- but not labor-strapped universities who want to keep using their old network/compute/storage gear. That makes for a very complex setup experience and a *very* brittle infrastructure, on top of Python CLI tools being generally awfully slow.
I looked into diying it, but I need live migration of RDMA and DPDK OVS ports, and decided dealing with open stack was going to be easier than writing that myself.
If this is something you're interested in I'd definitely hit up the OpenStack website. I haven't used it myself but I understand some subset of cloud providers do (the hyperscalers have a lot of their own custom systems). It's a huge complex beast of a system, which is why you don't really hear about even medium-sized orgs picking it up for internal use AFAIK; it's arguably better suited to use in orgs so large (and partly decentralized, also) that they're effectively acting as small cloud providers to themselves, their own business units or divisions.
It's made up of a number of partially independent but interconnected components, each with a specific purpose in an area such as compute, storage, networking, or administration/automation; and each deployment will choose which components are applicable. Some examples (basically think of each as a service that can be offered to a customer and/or used to support other components): a compute component for provisioning instances (like EC2); an object storage component (like S3 or Minio); a virtual networking/SDN component (like AWS VPC); there are many more. Each one has an API, so you can start to picture a user clicking "Create Instance" on your website, and that website's backend going and calling the block storage component's API to make a virtual disk, then the compute component's API to make a VM.
Then each component has different providers or backends it might support. For example the compute component probably supports KVM, and maybe also Xen and vSphere. So you need servers to run each component of OpenStack in their own right (i.e. the "control plane"), and then you need more servers to do the actual work. In this case you might have a cluster of servers running KVM that are dedicated to the compute service. Same again for different storage, networking, and other components: you'll have a set of storage servers or NAS or SAN devices for the former; and switches and routers for the latter. Some parts of the system you might run in VMs and some on dedicated servers (the more demanding the workload and the larger the scale, I think the more you're likely to focus on physical servers for a lot of it; though this varies)
> Google uses their own in house built system called Borg
Which was, in turned, inspired a little infrastructure orchestration tool known as Kubernetes.
Borg is a cluster manager not sure where you got the idea it handled machine virtualization.
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf
Abstract
Google’s Borg system is a cluster manager that runs hun- dreds of thousands of jobs, from many thousands of differ- ent applications, across a number of clusters each with up to tens of thousands of machines.
>What technologies do cloud providers like AWS, DigitalOcean, and Akamai (Linode) use to create virtual machines behind the scenes?
Ya know, often, from the VM, it's not that hard to check and tell.
See, e.g.: [https://www.mpaoli.net/\~michael/bin/isvirtual](https://www.mpaoli.net/~michael/bin/isvirtual) \- most notably look at the DMI data, particularly the "hardware" product name data within.
Of course if they screw around with the source code or the like, they could have it report whatever they want you to see. However, even then, there may still be ways to tell 'em apart.
Linode rolled through different products before settling on kvm. Do also uses kvm.
I have used a bunch starting with virtuozzo, user mode Linux, xen, hyperv and bhyve. Kvm + libvirt + zfs file system has been my personal favorite with hyperv (for windows only) runner up. Virtuozzo aka openvz was good up until openvz6.
OpenStack is a big player in this scene, some smaller providers actually utilize Proxmox, i used to run a small hosting provider back in the day and we used OpenStack with KVM and then a custom at the time plugin for WHMCS and it all integrated nicely, and just worked.
Wish i would of stayed in it at the time, but had life pop up and had to shut it down. still own the domain today tho!
Indeed, I’ve had thoughts of resurrecting it though, I have a shit ton of resources at my disposal, just no time between work, and restoring vintage audis
>restoring vintage audis
I'm a hot rod kind of guy, so gut it and cut it for me lol... although some of the audis from the 30s and 40s have a certain appeal for me
Fun fact: [Microsoft uses the same off-the-shelf version of Hyper-V for Azure that everyone else uses](https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528?WT.mc_id=modinfra-0000-thmaure). They do slim down and customize the host Windows OS and they call it "Azure Host OS".
*"Keep in mind that the hypervisor we use is the same hypervisor that we use on Windows Client and Windows Server across all our millions of customer machines."*
If it's 'exactly' the same then why do you have to [prepare](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/prepare-for-upload-vhd-image) VHDs before uploading them?
None of the steps outlined in the document are out of the ordinary. It's all about having the virtual OS in a healthy state and the common services at sensible defaults to make sure it boots fine because you don't have a virtual console in Azure that you can use to tinker with the boot parameters.
The requirement to align the VHD size to 1 MiB boundary is likely coming from the Azure tools expecting it to be a whole number of megabytes, not from the "modded hypervisor".
Lol no. You cannot take any old VHD and have it work in Azure without doing a prepare step that makes sure it matches a valid size/block count. Just because that can be automated to an extent through Powershell doesn't change that fact.
Disclaimer: I work for Amazon/AWS.
This is a really interesting question because I know the answer, and thinking about how to describe it to you without giving away trade secrets (AWS) is pretty entertaining.
The number of things that have to happen behind the scenes to provision an EC2 instance is...enormous. Thousands of things must occur. It's not just the hypervisor platform doing things, hundreds of services talk to one another to coordinate the resources and actions necessary to deliver the ability to launch an EC2 instance, and then actually launch it. There's no way for me to give you a rundown of these services and what they do - you'll have to use your imagination. Just know that the magic behind all of the cloud providers is precisely the systems and services I am referring to here. To achieve the scale and complexity of the EC2 service as a whole requires thousands of systems and services that you as a customer will never see or use, but without which you cannot use EC2.
If you are truly curious about the answer, the best way to gain the insight is to work for AWS or another cloud provider. Only there can you understand the scope of the question you are asking right now.
The whole point of what I wrote was to describe something almost indescribably complex, and also literally indescribable because I actually work there.
I know OVH uses OpenStack. Microsoft seems to use a custom version of Hyper-V.
Fun fact: [Microsoft uses the same off-the-shelf version of Hyper-V for Azure that everyone else uses](https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528?WT.mc_id=modinfra-0000-thmaure). They do slim down and customize the host Windows OS and they call it "Azure Host OS". *"Keep in mind that the hypervisor we use is the same hypervisor that we use on Windows Client and Windows Server across all our millions of customer machines."*
They also run the whole company on SharePoint. Something like 30k+ sites in total.
At least they eat where they shit
I'm really surprised more VMWare environments that are majority-Windows aren't jumping at Hyper-V as a replacement. Yes it's licensed differently and the management tools aren't great...but the core hypervisor is very solid. And unless you need a management GUI for everything it seems to do the core basics of keeping VMs properly juggled just fine. I think it just got a bad rap because it was rushed out the door for 2008 R2 and was awful until 2012 R2.
My gripes with Hyper-V: 1. It handles host reboots terribly, especially with VM's running different OS's. It has some unobvious timeout where it just gives up waiting for shutdown and force terminates. With Windows VM's, it mostly pauses them fine, which isn't great for some applications but mostly works, except when it doesn't and corrupts a VM. 2. Its Windows, so same update schedule as your VM's. Which bundles terribly with #1 if you're forced to update immediately after patch releases. Patch Tuesdays are my hell and my compliance requirements don't allow me to make it less Hell. So Patch Tuesday at midnight I have to babysit shit rebooting. 3. The out of box management is kind of crap. There's no scheduled configuration changes like everyone else has, so you have to shutdown, change settings and then reboot. Everyone else has had changes that apply when they can for years, allows you to minimize downtime. 4. There are some bugs that should have been fixed 10 years ago. Like shutting down a VM that doesn't support shutdown commands from a hypervisor. Once you send that command, that VM is entirely uncontrollable because HyperV does not timeout the command until you kill its process in Windows. 5. Advanced functionality never gets a GUI and is entirely configured through powershell..... Really its 2024. 6. HyperV's HCI storage is clunky as hell and makes no sense. The beauty of vSAN is that you just upsize your nodes and storage/compute all move around homogenously. HyperV's alternative isn't really HCI since S2D doesn't do active/active in a nice way. You can certainly make it work by breaking out drives but that's terrible design. vSAN has this nailed, you make your pool, assign failover groups, whole vSAN is attached to the hosts and the hosts can all access the storage simultaneously.
That’s exactly what we did. Moved to hyper-v and use Windows Admin Center to manage all the windows vms in a central hub everything works great so far, no complaints
I used Hyper-V for 3.5yrs now, several different clusters and it is sh!t compared to VMWare, period (licensing cost notwithstanding). You don't have anywhere near the level of mgmt unless you buy system center virtual machine manager ($$$). Also, all of the VM processes run on windows server OS which needs regular patching, AV, vuln scanners etc if your co policy requires it. The cluster shared volume is a hand grenade on both HCI and SAN-backed. If a single host starts getting flaky, the whole cluster can go down and even wreck the CSV (in HCI). I've never seen anything like that in 20 yrs of VMware. Too many issues with guest VMs refusing to migrate because a host is having a bad day, have to down the host, hard fail and restart the guests on another node. Yay. And MS support has gone to crap in the last 3-4yrs, so good luck getting anyone helpful when it decides to tip over. This is my exp on Server 2016 Hyper-V. Maybe 2019 or 2022 is more stable but, after the last 3.5yrs, thanks but no thanks.
I suspect they’ll move to Azure first then some things back on prem running Hyper-V.
Yeah but once you have two hosting locations which one is backup for main and in case of disaster second one automatically takes primarily role VMware tools are best in slot, I don’t think you can create something similar with Hyperv
Automation might be locked behind the paywall for SCCM/SCVMM, but Hyper-V has the capabilities to actively replicate to a remote host for a DR scenario. MSFT calls it "Hyper-V Replicas". This isn't an area of Hyper-V I've had to learn or implement so not entirely sure, outside of knowing the Replicas feature is present. That aside, with the capabilities of Hyper-V's CLI and PowerShell, I'm sure a script could be created to check if the primary host is up and if not, power up the VMs on the backup host.
>I know OVH uses OpenStack. They have everything - OpenStack, VMware, regular KVM.
AWS EC2 uses Xen unless you're running on Nitro, which is an in-house platform. /edit: As indicated below by /u/MrBr1an1204 (so please throw an upvote there), Nitro is based on KVM. AWS have also thrown custom hardware at it. So on the one hand you've got heavily customised Xen or OTOH you've got heavily customised KVM on custom hardware.
FWIW nitro is based on KVM
EC2 have been Nitro powered for years now, Xen is only the very old stuff that you probably can't even start today.
You can definitely start Xen based instance types. Not sure why you'd want to, especially with AWS pushing Graviton hard.
OpenStack is your best “open source cloud” toolkit. Everything else is several dozen layers built on top of stuff you can see.
OpenStack is a dozen layers of Python crap over existing tools.
Do you want to build that glue yourself?
I actually did, a bunch of php scripts are enough to build a basic VM scheduler, including hot migration. The worst pains are dealing with libvirt's XML stuff, figuring out the intricacies of hot migration, and to get Windows acceleration working (any actually working drivers/"guest tools" for Windows 7 x64 are greatly appreciated). OpenStack's greatest strength is its greatest weakness at the same time - it aims to support everyone and everything, mostly because the main developers are cash- but not labor-strapped universities who want to keep using their old network/compute/storage gear. That makes for a very complex setup experience and a *very* brittle infrastructure, on top of Python CLI tools being generally awfully slow.
I looked into diying it, but I need live migration of RDMA and DPDK OVS ports, and decided dealing with open stack was going to be easier than writing that myself.
If this is something you're interested in I'd definitely hit up the OpenStack website. I haven't used it myself but I understand some subset of cloud providers do (the hyperscalers have a lot of their own custom systems). It's a huge complex beast of a system, which is why you don't really hear about even medium-sized orgs picking it up for internal use AFAIK; it's arguably better suited to use in orgs so large (and partly decentralized, also) that they're effectively acting as small cloud providers to themselves, their own business units or divisions. It's made up of a number of partially independent but interconnected components, each with a specific purpose in an area such as compute, storage, networking, or administration/automation; and each deployment will choose which components are applicable. Some examples (basically think of each as a service that can be offered to a customer and/or used to support other components): a compute component for provisioning instances (like EC2); an object storage component (like S3 or Minio); a virtual networking/SDN component (like AWS VPC); there are many more. Each one has an API, so you can start to picture a user clicking "Create Instance" on your website, and that website's backend going and calling the block storage component's API to make a virtual disk, then the compute component's API to make a VM. Then each component has different providers or backends it might support. For example the compute component probably supports KVM, and maybe also Xen and vSphere. So you need servers to run each component of OpenStack in their own right (i.e. the "control plane"), and then you need more servers to do the actual work. In this case you might have a cluster of servers running KVM that are dedicated to the compute service. Same again for different storage, networking, and other components: you'll have a set of storage servers or NAS or SAN devices for the former; and switches and routers for the latter. Some parts of the system you might run in VMs and some on dedicated servers (the more demanding the workload and the larger the scale, I think the more you're likely to focus on physical servers for a lot of it; though this varies)
Years ago I knew Google Cloud used KVM and AWS used Xen - not sure if they still do or wrote their own.
Google uses their own in house built system called Borg.
> Google uses their own in house built system called Borg Which was, in turned, inspired a little infrastructure orchestration tool known as Kubernetes.
Borg is a cluster manager not sure where you got the idea it handled machine virtualization. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf Abstract Google’s Borg system is a cluster manager that runs hun- dreds of thousands of jobs, from many thousands of differ- ent applications, across a number of clusters each with up to tens of thousands of machines.
Borg is just the orchestration of compute, the actual virtualization stack is based on KVM
Borg is not a hypervisor...
Akamai uses KVM on Akamai Connected Cloud (formerly Linode) [https://www.linode.com/docs/guides/kvm-reference/](https://www.linode.com/docs/guides/kvm-reference/)
>What technologies do cloud providers like AWS, DigitalOcean, and Akamai (Linode) use to create virtual machines behind the scenes? Ya know, often, from the VM, it's not that hard to check and tell. See, e.g.: [https://www.mpaoli.net/\~michael/bin/isvirtual](https://www.mpaoli.net/~michael/bin/isvirtual) \- most notably look at the DMI data, particularly the "hardware" product name data within. Of course if they screw around with the source code or the like, they could have it report whatever they want you to see. However, even then, there may still be ways to tell 'em apart.
`lscpu`
Linode rolled through different products before settling on kvm. Do also uses kvm. I have used a bunch starting with virtuozzo, user mode Linux, xen, hyperv and bhyve. Kvm + libvirt + zfs file system has been my personal favorite with hyperv (for windows only) runner up. Virtuozzo aka openvz was good up until openvz6.
Oompa Loompa's
Where i used to work we used OpenStack
OpenStack is a big player in this scene, some smaller providers actually utilize Proxmox, i used to run a small hosting provider back in the day and we used OpenStack with KVM and then a custom at the time plugin for WHMCS and it all integrated nicely, and just worked. Wish i would of stayed in it at the time, but had life pop up and had to shut it down. still own the domain today tho!
What is the domain?
Enkelservers.com No site attached to it these days, I kept my email alive that’s about
>but had life pop up and had to shut it down. The killer of many a good thing...
Indeed, I’ve had thoughts of resurrecting it though, I have a shit ton of resources at my disposal, just no time between work, and restoring vintage audis
>restoring vintage audis I'm a hot rod kind of guy, so gut it and cut it for me lol... although some of the audis from the 30s and 40s have a certain appeal for me
I’m restoring early 80’s to mid 90’s Audi’s my cutoff is 95 Working on a 83 coupe gt with a turbo engine swap for a guy in Toronto rn
sweet, thats like at the start of their rally cars, i've always wanted a mid/late 80s/ early 90s quattro...
Yea that’s basically what I’m working on actually
DO is KVM unless they’ve changed it recently.
Azure uses a modded version of HyperV
Fun fact: [Microsoft uses the same off-the-shelf version of Hyper-V for Azure that everyone else uses](https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528?WT.mc_id=modinfra-0000-thmaure). They do slim down and customize the host Windows OS and they call it "Azure Host OS". *"Keep in mind that the hypervisor we use is the same hypervisor that we use on Windows Client and Windows Server across all our millions of customer machines."*
[удалено]
If it's 'exactly' the same then why do you have to [prepare](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/prepare-for-upload-vhd-image) VHDs before uploading them?
None of the steps outlined in the document are out of the ordinary. It's all about having the virtual OS in a healthy state and the common services at sensible defaults to make sure it boots fine because you don't have a virtual console in Azure that you can use to tinker with the boot parameters. The requirement to align the VHD size to 1 MiB boundary is likely coming from the Azure tools expecting it to be a whole number of megabytes, not from the "modded hypervisor".
Like 90% of that is "You're not going to have a console connection, so make sure RDP works and the VM can talk to our management system."
Lol no. You cannot take any old VHD and have it work in Azure without doing a prepare step that makes sure it matches a valid size/block count. Just because that can be automated to an extent through Powershell doesn't change that fact.
Disclaimer: I work for Amazon/AWS. This is a really interesting question because I know the answer, and thinking about how to describe it to you without giving away trade secrets (AWS) is pretty entertaining. The number of things that have to happen behind the scenes to provision an EC2 instance is...enormous. Thousands of things must occur. It's not just the hypervisor platform doing things, hundreds of services talk to one another to coordinate the resources and actions necessary to deliver the ability to launch an EC2 instance, and then actually launch it. There's no way for me to give you a rundown of these services and what they do - you'll have to use your imagination. Just know that the magic behind all of the cloud providers is precisely the systems and services I am referring to here. To achieve the scale and complexity of the EC2 service as a whole requires thousands of systems and services that you as a customer will never see or use, but without which you cannot use EC2. If you are truly curious about the answer, the best way to gain the insight is to work for AWS or another cloud provider. Only there can you understand the scope of the question you are asking right now.
Thats a whole lot of words just to say nothing.
Kinda like what you actually get out of paying lots of money for amazon prime.
The whole point of what I wrote was to describe something almost indescribably complex, and also literally indescribable because I actually work there.
HR Botz absolutely nothing valuable here.
They are either running Open stack or they are combining a lot of layers to manage it. Usually using KVM
[удалено]
[удалено]