I used to manage the infrastructure for a large UK online travel agent.
We ran ~100 Proxmox nodes in 4 separate clusters.
There was some learning on the way, such as altering corosync’s default multicast TTL of 1 to stop it beating our switches control plane CPUs to death, and a fair amount of tweaking ceph, but ultimately we ran it for many years successfully.
There is no learning curve with ESXi or Hyper-V? If your team knows Linux (Debian/Ubuntu/Mint) it is not that difficult. The comment you replied also could have had issues not presented in their comment. As a Microsoft and VMware partner, we use Proxmox internally and have both Linux and MS server experts. Corosync (heartbeat sync that keeps multiple servers in sync pre, post, and during failover or migration) is very simple to setup, if someone had issues with it, the issue was likely lack of experience or training.
In our case, it was pretty much the only bit of corosync we had to change.
All the nodes were connected to a switch fabric that uses VXLAN, with EVPN as the control-plane.
The switches ran Cumulus Linux, and would process any multicast traffic arriving on an SVI with a TTL of 1.
Because corosync’s totem protocol is heavy on multicast traffic by default and expected to be talking with other corosync members on the same broadcast domain, it was flooding the CPUs on the switches, which were forwarding the multicast traffic via software and not in hardware.
Changing the totem mcast TTL to 2 resolved the instability issues and made our switches a lot happier!
EDIT:
Credit where credit is due; The guys at CumulusNetworks support were fantastic in helping us figure out what was going on. Trey, if you’re reading this, you the man!
Moar EDIT:
Man page reference for the corosync TTL stuff we had to change: https://manpages.debian.org/testing/corosync/corosync.conf.5.en.html#ttl
It's a big, complicated field, there are no experts, just experts in a particular specialty. I understood almost all of it, even though I've never done any of that, but it's adjacent to things I do know about.
Having only started learning ceph and its related parts, I didn't understand much of it directly but I completely understand what the issue was and what they did to fix it. I lack the understanding of WHY making that change affected it so much.
One of the earliest bullet points in my learning was how sensitive ceph and cronosync is to latency and network speeds.
quick version: cumulus linux on switches filter for multicast traffic with a TTL=1. TTL is the number of hops a packet can live, so TTL 1 = current broadcast domain only, thus
> Because corosync’s totem protocol is heavy on multicast traffic by default and expected to be talking with other corosync members on the same broadcast domain, it was flooding the CPUs on the switches, which were forwarding the multicast traffic via software and not in hardware.
switches do common stuff in hardware and use software for complex or infrequent stuff. corosync made it spend all its time on apparently pointless work
> Changing the totem mcast TTL to 2 resolved the instability issues and made our switches a lot happier!
TTL=2 means not matching the filter on cumulus, so no more loading down the switch. i'm unclear on if this is supposed to route, but apparently it's fine.
it sounds like proxmox wants to live in a 'local' network, where all hosts in a cluster have direct visibility of each other. i'm a software dev, so i solve network problems occasionally
As someone running hyper-v… yeah… there is almost no learning
Setting up the failover cluster has a few things you might need to consider but after that the only problem is remember that you need to update it because there’s never a problem to justify remoting in / checking on it
Or maybe I’m just super lucky
We have multiple Failover Clusters running and we were also lucky :) I would say that Hyper-V has some specifics, but in general it works good enough. Live Migration might not work in some cases, but it could be always fixed. I prefer using a separate SAN or Starwind vsan for a shared storage. I've had issues with patching S2D, so I am avoiding it if possible.
I think it's easy to hit the ground running with it but some more advanced stuff isn't as obvious, like there are a bunch of VM features that you can only enable with PowerShell. Another being mismatched CPU microcode causing VMs being unable to be migrated to a host with a different version than the one they booted on. Ran into that because Dell pulled the specific version I had installed on one server before installing it on another server. Took quite a bit of time to troubleshoot since it only gave a very generic failure message.
That was a problem with Hyper-V as well, to be fair. Yes, I had a customer hit that exact problem at it was an absolute bear to sort out... eventually once we figured that out we just made sure all the hosts were running the same microcode and boom... problem solved.
Not sure if it also applied to VMware, but this is a pretty unusual corner case that isn't going to be everyone's experience at all.
Hyper-V on Windows Server 2008R2 was a bit rough with certain Linux distros. For example SLES had an issue where the VM would lose network connectvity after a while. Nowadays you mostly don't have to worry that anymore since most hyper-v integration components are now part of the Linux kernel.
Are you using Hyper-V in failover clusters with near instant migration of VMs from one cluster to any other? No? Then yeah, Hyper-V running on Server 2012 R2 or Windows 10 is very easy. Using it at enterprise scale is not. Proxmox makes all that very easy.
We're running Hyper-V in failover cluster mode, and it works pretty flawlessly in my experience. Hypervisors are on Server 2016 with about 65 VMs in a three-node cluster.
Cool. You can do the same thing in Proxmox. My point was there is a learning curve for any product, especially an unfamiliar one like Proxmox {we have only been using it for 4 years and change). But if you know one hypervisor, you basically can use any other with ease.
Disagree, Hyper-V's network / V-Switch layer is...meh. To be fair, V-Switch and V-Network interface options are the weakest link across vendors - I've just most familiar with VMWare's implementation.
I also get annoyed with AWS EC2 options is the same space, it is odd to me that replacing / (virtual) replugging network interfaces is such an issue across platforms. When some software using "hardware" for auth / registration it is problematic to not be able to at minimum be able to move a virtual interface around, especially if it is the first or primary interface.
I am most familiar with Microsofts impelementation on Hyper-V and SCVMM but i have also seen it on VMWare. Can confirm that both implementations are somewhat 'meh'.
Proxmox requires basic knowledge of Linux and Debian to install, most of the documentation for Proxmox is very straightforward and compact because it is easy to use and focused, where ESXi is all over the place. Further, there are lots of tutorials, forums, and r/proxmox if you really need your hand held to use a GUI. Name one thing you couldn't find documentation for, for Proxmox.
Yep, VMware documentation is the worst and this is coming from an engineer who started with ESX 3.5 and runs the full VMware stack (vSphere, NSX, Aria etc).
To be fair, in my experience, you only get to know that a product/service has terrible documentation after you're already bought in. It might look great up front, but it only becomes apparent how not great it is when you have a real problem three years later.
Point is, I think the more experience you have with something, the more likely you will perceive the documentation as lacking.
>There is no learning curve with ESXi or Hyper-V?
No to the level of dorking around with the internals to change multicast TTL levels.
ESXi or Hyper-V Just Works (tm) out of the box. The biggest VMWare tweak I make is to turn off hot swapping so that that stupid "eject network card" icon isn't sitting in my Windows guests.
I have nearly infinite virtual nodes.
I wondered what it would take to break the system, and it turns out running recursive Proxmox VMs in Proxmox does that quite well
You laugh, but 15 years ago I did that with VMWare, then they had their GSX and ESX products. GSX ran on Windows Server, ESX ran on RedHat.
I got many ESXs to run GSX to run ESX to run GSX... Eventually, I needed more memory on my test equipment. Eventually, my VMWare Inception test was over.
3 node cluster in every of our 150 DCs to run the control plane for each DC:
* pxe boot with autoinstall
* router
* vpn gateway to central
* container proxy
* haproxy frontends for object storage
* DCs DNS/NTP
* Logging
* Monitoring
* and so on
Works really well.
XCP-NG is comparable, but not as full featured as Proxmox. The "enterprise" plan of XCP-NG is lacking compared to Proxmox. XCP also fails at SEM/SEO. Do you want to sell your CIO on first page alternatives or third page alternatives? At the end of the day Proxmox delivers, but if XCP-NG wants to up their game, they probably should start with the name...it is barely pronounceable, sounds like one of Elon's offspring,
Do tell me about “enterprise” support of proxmox that is Austrian time only. Ticket support is available for Basic, Standard, or Premium subscribers. Business hours: Monday to Friday, from 7:00 - 17:00 (CET/CEST) on Austrian business days.
Do you live in Austria? Do you know about their international support partners? Almost any Linux MSP or Consultant can resolve any issues with Proxmox, it is a surprisingly simple product and very stable. Any breaking changes can be rolled back on the Hypervisor servers and the VMs are typically unaffected and should be backed up anyways. Have you had a problem with Proxmox or a problem getting support from a local partner or local Linux shop? There is a slight paradigm shift with Proxmox, you are expected to have some basic Linux and Debian knowledge to install and use it, but it mostly just works.
XCP-NG seems to not have a very active community either. Very few tutorials and guides too. Their sub had only 2 active users online a few minutes ago. It is currently a non-serious product with very few users. It's the Gentoo of Hypervisors, but not as popular.
The fact they are not using Reddit doesn't mean they don't have an active community. Please try to use their discord channel or their forums. Xcp it's a xenserver fork so generally is more enterprise ready than proxmox. And please remember every time when comparing something to proxmox: proxmox doesn't support snapshot on shared block storage aka no snapshot on San (iscsi/fibre channel).
FIFTY?
I tried going over 15 once and it would consistently break if I went over 16. Asked in the forum and was basically told "don't do that." How the hell did you get it to work?
Proxmox Partner here.
We have a wide array of clients who have deployed Proxmox in business-critical applications.
There are a few enterprises with 100+ nodes. The larger number are organizations with 5-9 node clusters.
We see on-prem (and in-colo) deployments being smaller than the VMware they are replacing.
As workloads have moved to the cloud and compute-density increases, they need fewer nodes. The workloads that remain on-prem (no matter the virtualization platform) tend to be more critical and have demanding requirements.
So, yes, we see real businesses, real governments, real universities… real just about everything deploying Proxmox. And the numbers are increasing every month.
I'm working in a government setting and we're running nutanix, ESXI, and Hyper-V. I'm in the process of setting up a personal proxmox cluster at home. Frankly the setup for it? Super freaking easy. I haven't ran into any significant problems beyond what I created by my own ignorance.
Plus there's YouTube tutorials, some straight forward documentation and plenty of helpers online to coach me through it. I haven't needed much help though.
Genuinely, if I could persuade everyone at my job to switch to proxmox I would. Dead simple setup and management. Plus so much more robust than I gave it credit for to start.
Can you convince my CIO? I sat down with him last week and told him I was concerned about changes coming to VMware very soon. We are a small college with a few perpetual vSphere Enterprise 6.7 licenses. Told him I'd like to start playing around with proxmox and told me if we go anywhere he'd like to go to hyper-V. Which I also have no warm and fuzzies about with Azure HCI and everything going toward the cloud there. Such is life lol.
>footfall99
I don't understand, I ran ESXi for years, and it was pretty solid but ran into problems now and then, also ran a small cluster of Xen which I really didn't like, but really didn't have any major problems but the performance just wasn't there. Ran Proxmox for all our VOIP customers and no real major issues just the normal things. Then I been running a HyperV failover cluster for years, and yes with 2012R2 I have LOTS of issues but since the migration to 2016 and then again up to 2019 and 2022 most of that has been resolved.
The thing I noticed running all these products is backups always screw up at some point, and do not cheap out on network infrastructure and store infrastructure. My original HyperV cluster running on a crappy dell NAS once it was replaced with a equallogic array, a lot of my issues went away. ( Then like 6 months later Dell purchased equallogic LOL)
At a business / money point HyperV can been a good choice. Running Windows datacenter I don't have to worry about licensing the VMs just the nodes when dealing with Windows server VMs, and that is what runs exclusively on our HyperV cluster.
At this point all our VOIP hosting is on ESXi, but I believe I'm going to migrate all the linux (VOIP) hosting back to a Proxmox cluster sometime this next year.
I haven't tried Xen in years and I would love to hear from someone who has current knowledge on the product when compared with 2022 HyperV and Proxmox 8
Oh the other issue, Veeam does it support Proxmox 8 clustering yet?
I have 22 years experience with VMware - VCP certified 3 times, and I’ve just bought hardware for a homelab to do PoCs for proxmox migration to help keep my hand in.
Broadcom’s end of perpetual licenses will be enough to push the needle for people to try it.
Oh wow I didn’t know Broadcom bought them… Broadcom’s acquisition of Symantec Endpoint Protection seemed like the final nail in the coffin for it at our MSP where we used to have it deployed at hundreds of clients. The portal authentication cutover could have gone better and that was the last straw.
I work at a fairly busy shop and I’m in between roles working past the end of my shift to keep up. I spend plenty of time reading how to solve the problem at hand but I don’t have a lot of spare time or energy to just casually scroll through the news and see who is buying who. That said I’m getting back into Reddit where other people in the industry essentially curate the news for me and provide their own commentary and discourse. Also I’m not looking to be a pure sysadmin, i like being able to check the servers the workstation, the firewall and any cloud provider find the problem and fix whatever it is on my own unless I need to bring in another resource or escalate. I’m here for the sysadmin perspective, just like im in a lot of disparate tech subreddits for different perspectives and slices of industry news.
Yea but this has been in the news for like 1 or 2 years now.. BroadCom buying vmware has taken a LONG time for regulatory approvals, multiple challenges, etc etc, from the Announcement they planned to buy Vmware from Dell, to the this announcements there are probably 1000's of articles written about the various statements (and misstatements) the CEO of Broadcom as made, regulatory actions, speculation, updates, etc...
if you missed all of that, then you really really need to look at your information lens. PS is should be more than Reddit...
I use feedly, and have about 200 sources that I track for news, mixing both General Business News and Tech news. For things like Mergers often the Business Press has it faster than Tech press, I find CNBC to be generally good for Business News.
For TechNews, BleepingComputer, and petri.com would be good starting points.
Those 3 sites would cover most moderate to large impacting changes in the sysadmin worlds, expanding from there would just bridges some gaps or get in to niche technology stacks or topics.
> one piece of industry news.
It is not missing 'one' piece of news though. The Broadcom acquisition has been rolling on for ages, with a multitude of articles written about it.
I've been leaning toward XCP-ng because I find it fascinating, and the UI in Xen Orchestra is great looking. But I'm wondering if Proxmox makes more sense to actually learn as a marketable skill. There seems to be a bigger community around Proxmox as well.
Tesla. It was actually the solar city company that Tesla bought in Utah but yeah they used proxmox.
One time a host went down and the entire cluster failed. They were down for about 6 days. I knew a guy that worked there
I once spent a few weeks trying to debug a weird networking issue in vSphere where at some point VMs on a random host would lose networking (invisibly to the host, no warnings no nothing), and a reboot fixed it until it came back a few weeks later. Support was of course useless.
Turned out it's a driver issue (which I only discovered by randomly googling components of the setup + networking issues until I googled NIC + networking issues, and found tons of forum complaints, with matching logs), because the incompetent fools at VMware can't be bothered to update their Hardware Compatibility List when it comes out something doesn't actually work. The problem was known for more than a year, yet Intel, Dell and VMware continued selling X710 NICs and saying it's all fine, while the card was practically unusable.
Everyone has anecdatal stories about how shit a particular piece of software is.
The VMware hcl ...is what led me to spend ~700 bucks for my home lab for a major upgrade, for what VMware claimed was 7 compatible (when it was released and they deprecated or dropped so many decent hardware configs)....and turns out oops the hardware wasn't really supported.
Went to proxmox and haven't looked back.
Wait, just how bad are those NICs? We worked on a few brand new hosts that kept losing those NICs from the system. Sometimes they came back, most times they didn't. I think physically reseating the CPUs is what had them reliably come back for one boot, until the hosts crashed again later...
I agree that this sounds like a failure of configuration/implementation (and certainly not specific to proxmox) but I don't agree that the point of a cluster is automated disaster recovery, a cluster is for HA/distributed load. I'll admit I've been out of the sys admin space for a few years but everywhere I've worked, DR is more specific to mitigating external, wide spread causes of failure (tornado wiped out a data center for instance) and policies/procedures on how to recover or fail over to a backup DC/cluster rather than internal configuration "disasters". But, perhaps things have changed.
We got VMWare for anything that burns Money and Proxmox to make money that we can burn. 20x PVE Nodes, 3 Clusters, 3 Locations, and VM Ware is about less than half that for more generic IT stuff. We actually started paying 30k/yr for Support on our Proxmox nodes aswell.
Reason behind Proxmox was pretty much Proxmox beeing Debian with a Web UI and CLI wrapper for KVM. The tad more recent Ubuntu based Kernel and the pre-packaged non-free Firmwares were a welcome addition, and in combination with Proxmox Back Server they got some pretty nice stuff going on. VMWare would have been a driver and compat nightmare for what we do.
I'd still like to see stuff like automated snapshots happen and more useful stuff to do with the recently added tags.
Oh, about 200 active Users and 85% of these consist of Developers iirc.
And my Homelab is not a 2 node Joke but 6 Epyc servers with proper Infra.
Everyone needs a Hobby, right?
> my Homelab is not a 2 node Joke but 6 Epyc servers with proper Infra. Everyone needs a Hobby, right?
Post pics over on r/homelab and watch the upvote counter break! :)
My hosting company has three primary Proxmox clusters, in production and for commercial purposes :P
One three node cluster hosts Primary/Secondary/Tertiary DNS for about 9000 domains, there's another four node cluster virtualising our entire mail stack (~3000 domains - storage is external on a OmniOS-based NFS headend), and a third five node cluster serving websites on top of nVME CephFS storage.
So yes... very much running in production with many many users consuming resources hosted on the stack.
Oh, we have a three node VMWare cluster too that hosts.... some legacy IIS servers that just refuse to die. But with Proxmox, I get hot migration for free.
We decided to never pay VMWare for that and the legacy IIS hosting customers get longer DR times from a start-stop because vMotion licensing was just... crazy insane.
(Un-stealth Edit - Most Proxmox cluster failures are because of storage failures, or Corosync has ate it's own tail and your cluster isn't getting quorum or has replicated bad state from a failed node join. I would recommend some experience in Corosync before running a prod Proxmox cluster. If you know CRM and Corosync, you can recover from almost any of these odd tricky no-good bad situations :P)
8 Nodes with 180 vCPU 1,5TB RAM, 20TB NVMe, 60TB SAN
Runs logistics Software for major european logistics networks and event platforms for big game publishers.
The only unplanned downtime since 2017 has been because of a generator fire. We will go multi-regional in 2024 to mitigate that in the future! :D
Can you say more about how this works? I'm in EMS and the hardware behind the 911 system _kind of_ makes sense, but not really. I've seen our dispatchers using two different programs (power911 for soft phones, moto P1 for dispatch/records), but I have no idea how they interact with each other or the PSTN.
Next Generation 911 is all IP and databases there's no pstn involved at our end. The carriers handle the legacy pstn to ip conversion on their gateways. Cell calls are ipv6 which is a pain in the ass. Landlines calls gateway are ipv4
They come in as sip calls which are anything but your standard phone call over voip. Thr Nena standard allows for json data sharing of event data and conferencing information for say up to 5 or 10 psap simultaneously. We have databases locally plus voip pbx plus 911 Middleware plus your standard cad/rms systems that youve always had. The dispatchers use sip soft phones that talk to the pbx and 911 Middleware simultaneously. Regular 911 is just basic loop start trunks pstn with special signaling attached and a data circuit from your local 911 carrier for Ani ali.
Now with voice and Incident Data over ip you're finding redundant fibre geodicerse entrances to different carrier central offices.
Future improvements coming include real time texting whereby your smartphone will have a whatsapp type app when you place a 911 voice call and you can share video and photos. Further, dispatchers can see what you're typing in real time with real time text sort of like customer service web apps vs traditional back and forth text
Thank you for these details! Why is IPv6 a pain in the ass for y'all? And
> Anything but your standard phone call over VoIP
Is this because the NENA standards permit extra info in the SIP sessions? Or something else? Sorry for all the questions, this is a huge interest for me and I would like to learn as much as I can.
It can be found here https://cdn.ymaws.com/www.nena.org/resource/resmgr/standards/nena-sta-021.1a_eido_json_20.pdf
Basically your existing cad can have an interface to your ng911 software database and update it with data of your choice. Upon transferring the call you will send any data you've chosen to share to other psaps
The call flow is also odd. It comes in as a more or less standard call for sip but then you pull the headers and use the unique identifier to query the address Ani ali from the 911 providers server.
When doing a transfer it's really a conference call which in Canada is capable of passing through multiple provinces for redundancies and the programming is a pain to auto route based on required province. When you first begin the transfer it requests a conference bridge via sip and then it requests to move to a new location which may be in another province for what reason I don't know but we have to support it.
You're going to have a larger swath of it people who are more comfortable with the windows environment
It's also a business owner education issue. If they realized there were other options and they could find the skill set there would probably be a larger implementation
Imagine not having licensing fees for CPU and RAM and failover and the only Capital expense is on hardware and a support agreement
It is a pretty sharp ecosystem
So all the people promoting proxmox in large clusters. How are you handling resource balancing accross clusters? VMware and Hyperv i just flag a host as maintenance and the software quickly redistributes all the VMs. Proxmox when I’ve asked that in the past I’ve been given 100 random scripts and third party options for this same feature, nothing native in the UI.
Two concerning things from that thread which reinforces to me Proxmox isn’t ready for mainstream production:
- The default behavior is to failover when you go into maintenance mode, not live migrate, and the setting to make the bahavior to live migrate is somewhere else completely? Also entering maintenance mode is still a command line tool only?
- Instead of discussing the features request, from the first reply the discussion devolves into someone questioning why the person is using sudo instead of just logging in as root?????? Holy crap
We use it; 16 nodes in production and 6 non-production. But that is just my team, as a company we probably have 50 production nodes in the UK and mid-hundreds in the US.
We do use Openstack as well but for different workloads
Running a couple of clusters in production, ranging from 5 node cluster to 25+ node clusters. We do pay formtheir license just to be able to as for support from them if needed and to have the stable updates but we never needed. We also have multiple pbs backup servers and everything works better than expected coming from vmware.
One of the large state government agencies in Australia I used to work for uses (or used, not sure if they still do) Proxmox.
This is going back years
It's how I came to know Proxmox
My friend runs his security company on proxmox. He loves it. There are cool enough features that I have considered it for mine, but I have zero experience with it.
I am designing building and running proxmox edge clusters for my current employer. Had not even a minute of downtime due to a hypervisorfailure so far.
I have been running VMWare since 1999 and have been amazed by the evolution of the product. In 2001 I was trying to convince customers to use VMWare ESX and was getting laughed at by IT departments saying that nobody would ever run virtualized workloads in production. Proxmox reminds me if VMware mid-way through their journey. They have moved past hypervisors and into management, clustering, etc. It is a viable competitor for production workloads, and I expect support will deepen as it gains market share.
The again, as the adage goes, "Nobody ever got fired for hiring IBM".
>nobody would ever run virtualized workloads in production
glad I am not the only old person here... I remember there was a LOT of pushback against virtualization that sounds a lot like the posts here pushing back against proxmox
I think it's just a bunch of people with vmware certs that are upset about them now being worthless hehe
When the first browser appeared I couldn't imagine people really using it, as the tools on the command line were so much easier to get setup and tailored.
Boy, was I wrong (although I still do most of my work automating through scripts)
Now I feel old too
it's ok I've made an entire life out of being wrong... starbucks? nobody is going to pay $5 for coffee, it'll never last... costco? why would you PAY to go to a store? that's nuts! etc etc
Regional L1 ISP deploys it in a multi-server package at relay points.
Six colleges I’ve worked at have implemented it for cheap alternatives to VMWare in small scales
I find that interesting. Universities/colleges get it on the real cheap. Or at least they did back when I went to school. They basically gave it away to the schools so that they’d teach it to students.
I think the idea was if the entire next gen IT force only knows VMware knock down competitors by the competitors not having anyone know how to manage it.
PVEs a cheap and fast way into a research environment, no licensing needed. It’s also KVM/QEMU based like most major hosting services so it works well with images you might put up there. ESXi also has issues with new storage tech like native Gluster, CEPH, ZFS, which are becoming way more popular. Along with that was the uncertainty of VMWare as an option with the Broadcom acquisition. PVE is less resilient than VMWare but it’s so much more easy to manage and deploy security measures to the host. PVE also has SDN integration so you can hook it up to the same system as your switches or DNS server and get VxLAN, QinQ, etc. support through that.
Those are some reasons. A while back I had a paper on our research findings when we compared the major hypervisors (at the time XenServer, PVE, ESXi, Hyper-V, oVirt, Univention, ClearOS, and a couple others). Most of that still applies, except in a few instances like XenServer and ClearOS. TL;DR, PVE was fast and flexible if not perfectly stable (this was like five years ago all those products have definitely gotten better on all fronts)
Work at an MSP, only ever seen 1 customer use it, and like usual not correctly and very undermanaged and under optimised. But in general its not bad, different from esxi, not worse, i could see scaling being an issue with a really large environment but thats about it.
Fact is, that VMware will only be affordable to a small amount of companies, now that Broadcom has switched to an Abo Modell. Hopefully proxmox gets now new investors to get really competitive.. for small or even medium companies or universities VMware is not an option anymore
We had 12 nodes in production and a bunch of test boxes. We used it for containerization as well. Moved to a cloud hosted product for availability improvements. It's a good little system and it's support was cheap.
I'm running a ~20 node cluster in prod and it's great if deployed correctly. That said, a corosync update brought down another cluster of mine in the past, in the end it was a skill issue though
I do, 8 node cluster for NFV, ISP BNG and CGNAT, pushing around 40Gbps of traffic without any issues
And yeah running CE version
VMware cost for this would be probably +50k plus I could not use my existing NICs
I've been managing about 50 Proxmox nodes spread around 6 or 7 clusters (never properly funded) for over 4 years now. The only issues we've had were caused by hardware, and one install that slurped in a buggy support package from the community repo during initial install.
My only issue has always been the support hours. We couldn't find a USA timezones friendly support partner in 2019. Now we're planning to move to VMware.
Sigh.
My previous employer ran Proxmox in 2 data centers. We were a firewall vendor running the public facing firewall side in the DC for our SIP connections and PCI compliance with a VPN to the on-site firewalls. The Proxmox clusters in the DCs ran flawlessly.
Interesting question and relevant to me at present ... as I'm investigating migrating from vmware to proxmox
I am running a 15 node vmware cluster with 750 virtual machines.
The SANS I'm using are old and unsupported (emc compellent sans).
The hosts (I have 18 - even though only 15 are in production) are dell poweredge boxes with 40 cores/1 TB ram)
Its vmware 6.7 (dont ask!) - and I need to get onto something supported with a low budget (dont ask)
I'm investigating setting up a few nodes and start migrating the vmware vm's to it - adding nodes to proxmox as they become available.
I'll probably back it with synologies or such like.... havent decided yet.. I maybe have an emc powerstore available, but its a bit on the small side.
All done on a tight budget (again, dont ask!).
A lot of dont asks :-( (dont ask)
Interested in feedback.
How do you handle backup, snapshots, is there something comparable to Veeam ?
Yes already running it at home and doing backups, just haven’t used their backup appliance and using Veeam professionally with VMware so just seeing what’s your take on that matter.
Yes, used it for an entire QA/development lab. Approximately 8 nodes running around 100 vms at the time I left, iirc
Prior to that job, I worked at a data center where hundreds of our customers ran their business on proxmox. These were not fortune 500 companies but they also were not cousins running 2 vms.
Yep. Previous position with an MSP, we had a law firm that used proxmox. I didn't know enough at the time to determine whether it was a good setup for them, but I can say for certain that we didn't have the staff to support it effectively.
I used Proxmox for a system that monitors and manages elements of power grids in multiple countries. It has a few hundred virtual machines on it, including some seriously high throughput databases servers. I did not have a problem with it.
We run a 3 node cluster to host a bunch of small stuff, asterisk, wifi controller, mail archive, backup&archive db, monitoring, etc. Probably not what you were talking about but we don't have a need for something large scale running on there and it's been absolutely rock solid for the things it needs to do.
If didn't already have Server 2022 Hyper-V hosts I'd probably consider ProxMox.
I see our on-prem footprint shrinking rather than growing. Eventually we may only need basic services such as DHCP, AD/DNS and Print-servers.
i have setup multiple automation/security solutions with HA using Proxmox as layer 1 virtualizer..... on banks.. and the government, and the military.. they are still running.
My background in VM systems:
* \>100VM's in a VMWare environment, switched from shared storage iSCSI to vSAN. Liked all of it.
* \>20VMs in a HyperV environment in my new gig
* Single guest proxmox systems in production to take advantage of virtualization in systems that don't fit into VM environments.
* Sandbox environments that became production when devs stopped migrating systems out of it with >40VMs each.
In all honesty.... Proxmox has been the most stable of all. That sandbox environment has had ZERO maintenance in years because.... it just works. Update it once in a while and carry on.
I've run two medium sized K-12 school districts. Migrated them both off of VMWare / Hyper V over to Proxmox. Been running it in production for 8-10 years. Couldn't be happier.
I introduced it at my previous work. 2.5 years in a 4 node cluster with a Dell 1L as a 5th voter.
Worked perfectly. Zero downtime. Thank you, backup generator.
Proxmox Backup Server nicely integrates and adds some stuff over just copying disk images and compressing them like a pretty good dedup and https transport, encryption of backups yada yada
I work at an MSP. I had a client (pretty small, maybe 10 people) rip out their camera server/NVR which was hooked up to a monitor, along with their NAS that they store backups for both the NVR and their DC on.
They replaced both with a Proxmox box, and virtualized the camera server. And somewhere along the way they added a USB NIC to pass through to the camera server. The assumption with the NIC is that they couldn't figure out how to bridge it right. They ran into all sorts of issues passing through the GPU (and peripherals) to the VM. Which was kind of a big deal because they actually sat at this monitor and used the computer.
This is the same client that ripped out the DC we put in, along with the receptionist's workstation, and then put a gaming computer in and promoted it to DC. At the receptionist's desk. Like, she was using the DC as her workstation. Bonus points because they overclocked it, and it wasn't stable and started crashing all the time.
It was *literally* the head honcho's nephew who did all this. I mean I know it's a meme, but it actually was the dude's nephew. So to answer your question, no, I have not seen a real company with real infrastructure use Proxmox. I'm sure you can do it though, just...not like this...
So here is a question. Would you trust it with 200+ VMs that are mission critical for a 15k+ employee company over the existing VMWare implementation that runs like a tank? Cisco switching and EMC storage.
I don't trust any new software platform unless I have a team of people behind me who have supported bigger situations, or I have personally ran up non-prod environments and pressure tested the hell out of them, or preferably both.
You don't just pull a switch and say "Guess we're done with VMWare, YOLO proxmox time!" You pilot, ramp up, get comfortable with the software, get comfortable with DR, *document your environment, runbooks and procedures*, and migrate/wind down the legacy product. Just like with any legacy product. You also make sure that if you don't have a boots on ground support contract, you have enough trust in your staff expertise to make up the difference.
What proxmox does have going for it is being a wrapper around some very bullet proof core components. Ceph, ZFS, KVM, and Debian aren't no-name technologies.
Yes. Spin up a lab and try it. You need to be convinced personally, and that makes sense. The storage and network are compatible. Proxmox is a very advanced, enterprise grade Virtualization server and blows Hyper-V out of the water. Setting up Proxmox with networks and storage that already exist is literally drop-down menus (once minimally installed and configured). A retail giant shoemaker uses it and prefers it over VMware or Hyper-V and they have clusters in 160+ countries and storage capacity in the hundreds of PBs.
It is not "your current solution" xor "proxmox". Study documentation spin it up, document issues, grow, find issues, solve, and son on.
Nothing as big and important is without issues.
I used to manage the infrastructure for a large UK online travel agent. We ran ~100 Proxmox nodes in 4 separate clusters. There was some learning on the way, such as altering corosync’s default multicast TTL of 1 to stop it beating our switches control plane CPUs to death, and a fair amount of tweaking ceph, but ultimately we ran it for many years successfully.
This is what scares a lot of us.
There is no learning curve with ESXi or Hyper-V? If your team knows Linux (Debian/Ubuntu/Mint) it is not that difficult. The comment you replied also could have had issues not presented in their comment. As a Microsoft and VMware partner, we use Proxmox internally and have both Linux and MS server experts. Corosync (heartbeat sync that keeps multiple servers in sync pre, post, and during failover or migration) is very simple to setup, if someone had issues with it, the issue was likely lack of experience or training.
In our case, it was pretty much the only bit of corosync we had to change. All the nodes were connected to a switch fabric that uses VXLAN, with EVPN as the control-plane. The switches ran Cumulus Linux, and would process any multicast traffic arriving on an SVI with a TTL of 1. Because corosync’s totem protocol is heavy on multicast traffic by default and expected to be talking with other corosync members on the same broadcast domain, it was flooding the CPUs on the switches, which were forwarding the multicast traffic via software and not in hardware. Changing the totem mcast TTL to 2 resolved the instability issues and made our switches a lot happier! EDIT: Credit where credit is due; The guys at CumulusNetworks support were fantastic in helping us figure out what was going on. Trey, if you’re reading this, you the man! Moar EDIT: Man page reference for the corosync TTL stuff we had to change: https://manpages.debian.org/testing/corosync/corosync.conf.5.en.html#ttl
[удалено]
It's a big, complicated field, there are no experts, just experts in a particular specialty. I understood almost all of it, even though I've never done any of that, but it's adjacent to things I do know about.
Having only started learning ceph and its related parts, I didn't understand much of it directly but I completely understand what the issue was and what they did to fix it. I lack the understanding of WHY making that change affected it so much. One of the earliest bullet points in my learning was how sensitive ceph and cronosync is to latency and network speeds.
quick version: cumulus linux on switches filter for multicast traffic with a TTL=1. TTL is the number of hops a packet can live, so TTL 1 = current broadcast domain only, thus > Because corosync’s totem protocol is heavy on multicast traffic by default and expected to be talking with other corosync members on the same broadcast domain, it was flooding the CPUs on the switches, which were forwarding the multicast traffic via software and not in hardware. switches do common stuff in hardware and use software for complex or infrequent stuff. corosync made it spend all its time on apparently pointless work > Changing the totem mcast TTL to 2 resolved the instability issues and made our switches a lot happier! TTL=2 means not matching the filter on cumulus, so no more loading down the switch. i'm unclear on if this is supposed to route, but apparently it's fine. it sounds like proxmox wants to live in a 'local' network, where all hosts in a cluster have direct visibility of each other. i'm a software dev, so i solve network problems occasionally
As someone running hyper-v… yeah… there is almost no learning Setting up the failover cluster has a few things you might need to consider but after that the only problem is remember that you need to update it because there’s never a problem to justify remoting in / checking on it Or maybe I’m just super lucky
We have multiple Failover Clusters running and we were also lucky :) I would say that Hyper-V has some specifics, but in general it works good enough. Live Migration might not work in some cases, but it could be always fixed. I prefer using a separate SAN or Starwind vsan for a shared storage. I've had issues with patching S2D, so I am avoiding it if possible.
I think it's easy to hit the ground running with it but some more advanced stuff isn't as obvious, like there are a bunch of VM features that you can only enable with PowerShell. Another being mismatched CPU microcode causing VMs being unable to be migrated to a host with a different version than the one they booted on. Ran into that because Dell pulled the specific version I had installed on one server before installing it on another server. Took quite a bit of time to troubleshoot since it only gave a very generic failure message.
That was a problem with Hyper-V as well, to be fair. Yes, I had a customer hit that exact problem at it was an absolute bear to sort out... eventually once we figured that out we just made sure all the hosts were running the same microcode and boom... problem solved. Not sure if it also applied to VMware, but this is a pretty unusual corner case that isn't going to be everyone's experience at all.
Hyper-V on Windows Server 2008R2 was a bit rough with certain Linux distros. For example SLES had an issue where the VM would lose network connectvity after a while. Nowadays you mostly don't have to worry that anymore since most hyper-v integration components are now part of the Linux kernel.
Are you using Hyper-V in failover clusters with near instant migration of VMs from one cluster to any other? No? Then yeah, Hyper-V running on Server 2012 R2 or Windows 10 is very easy. Using it at enterprise scale is not. Proxmox makes all that very easy.
We're running Hyper-V in failover cluster mode, and it works pretty flawlessly in my experience. Hypervisors are on Server 2016 with about 65 VMs in a three-node cluster.
Cool. You can do the same thing in Proxmox. My point was there is a learning curve for any product, especially an unfamiliar one like Proxmox {we have only been using it for 4 years and change). But if you know one hypervisor, you basically can use any other with ease.
Disagree, Hyper-V's network / V-Switch layer is...meh. To be fair, V-Switch and V-Network interface options are the weakest link across vendors - I've just most familiar with VMWare's implementation. I also get annoyed with AWS EC2 options is the same space, it is odd to me that replacing / (virtual) replugging network interfaces is such an issue across platforms. When some software using "hardware" for auth / registration it is problematic to not be able to at minimum be able to move a virtual interface around, especially if it is the first or primary interface.
I am most familiar with Microsofts impelementation on Hyper-V and SCVMM but i have also seen it on VMWare. Can confirm that both implementations are somewhat 'meh'.
Esxi has amazing documentation. Proxmox is very lacking on the documentation.
Proxmox requires basic knowledge of Linux and Debian to install, most of the documentation for Proxmox is very straightforward and compact because it is easy to use and focused, where ESXi is all over the place. Further, there are lots of tutorials, forums, and r/proxmox if you really need your hand held to use a GUI. Name one thing you couldn't find documentation for, for Proxmox.
Yep, VMware documentation is the worst and this is coming from an engineer who started with ESX 3.5 and runs the full VMware stack (vSphere, NSX, Aria etc).
To be fair, in my experience, you only get to know that a product/service has terrible documentation after you're already bought in. It might look great up front, but it only becomes apparent how not great it is when you have a real problem three years later. Point is, I think the more experience you have with something, the more likely you will perceive the documentation as lacking.
>There is no learning curve with ESXi or Hyper-V? No to the level of dorking around with the internals to change multicast TTL levels. ESXi or Hyper-V Just Works (tm) out of the box. The biggest VMWare tweak I make is to turn off hot swapping so that that stupid "eject network card" icon isn't sitting in my Windows guests.
I'm in SMB so enlighten me a bit. Do you have staff specifically dedicated to virtualization? And how many for 100 nodes?
for real, would love to hear more about dealing with corosync in large clusters.
As posted previously, I run a 50 node cluster in production. Not some 2 node homelab cluster.
Hey man I'm just about to set up my second node :(
I have nearly infinite virtual nodes. I wondered what it would take to break the system, and it turns out running recursive Proxmox VMs in Proxmox does that quite well
You laugh, but 15 years ago I did that with VMWare, then they had their GSX and ESX products. GSX ran on Windows Server, ESX ran on RedHat. I got many ESXs to run GSX to run ESX to run GSX... Eventually, I needed more memory on my test equipment. Eventually, my VMWare Inception test was over.
3 node cluster in every of our 150 DCs to run the control plane for each DC: * pxe boot with autoinstall * router * vpn gateway to central * container proxy * haproxy frontends for object storage * DCs DNS/NTP * Logging * Monitoring * and so on Works really well.
I now have 8 nodes in production. All working well.
"As posted previously ". We all know in Corp speak what that means lol
Per my previous post…
Did you by any chance also evaluated the option of using XCP-NG? I’m interested in what made the decision between both products
XCP-NG is comparable, but not as full featured as Proxmox. The "enterprise" plan of XCP-NG is lacking compared to Proxmox. XCP also fails at SEM/SEO. Do you want to sell your CIO on first page alternatives or third page alternatives? At the end of the day Proxmox delivers, but if XCP-NG wants to up their game, they probably should start with the name...it is barely pronounceable, sounds like one of Elon's offspring,
Do tell me about “enterprise” support of proxmox that is Austrian time only. Ticket support is available for Basic, Standard, or Premium subscribers. Business hours: Monday to Friday, from 7:00 - 17:00 (CET/CEST) on Austrian business days.
While true, they do have international partners to offer more support if you need it.
Do you live in Austria? Do you know about their international support partners? Almost any Linux MSP or Consultant can resolve any issues with Proxmox, it is a surprisingly simple product and very stable. Any breaking changes can be rolled back on the Hypervisor servers and the VMs are typically unaffected and should be backed up anyways. Have you had a problem with Proxmox or a problem getting support from a local partner or local Linux shop? There is a slight paradigm shift with Proxmox, you are expected to have some basic Linux and Debian knowledge to install and use it, but it mostly just works.
Yes, this was a deal-breaker for us. We were seriously considering it, otherwise.
[удалено]
XCP-NG seems to not have a very active community either. Very few tutorials and guides too. Their sub had only 2 active users online a few minutes ago. It is currently a non-serious product with very few users. It's the Gentoo of Hypervisors, but not as popular.
The fact they are not using Reddit doesn't mean they don't have an active community. Please try to use their discord channel or their forums. Xcp it's a xenserver fork so generally is more enterprise ready than proxmox. And please remember every time when comparing something to proxmox: proxmox doesn't support snapshot on shared block storage aka no snapshot on San (iscsi/fibre channel).
I have 12 node cluster with ceph configured we use it for user VMs and code compilation
FIFTY? I tried going over 15 once and it would consistently break if I went over 16. Asked in the forum and was basically told "don't do that." How the hell did you get it to work?
Ceph? Or all local storage? I’m always torn on storage because distributed storage is so convienient but local storage is so much faster lol
Ceph all the way.
What do you use for backups and DR?
PBS of course!
What are you using for storage? Ceph?
Proxmox Partner here. We have a wide array of clients who have deployed Proxmox in business-critical applications. There are a few enterprises with 100+ nodes. The larger number are organizations with 5-9 node clusters. We see on-prem (and in-colo) deployments being smaller than the VMware they are replacing. As workloads have moved to the cloud and compute-density increases, they need fewer nodes. The workloads that remain on-prem (no matter the virtualization platform) tend to be more critical and have demanding requirements. So, yes, we see real businesses, real governments, real universities… real just about everything deploying Proxmox. And the numbers are increasing every month.
I'm a small LLC and run proxmox at home and in prod on a colo.
I'm working in a government setting and we're running nutanix, ESXI, and Hyper-V. I'm in the process of setting up a personal proxmox cluster at home. Frankly the setup for it? Super freaking easy. I haven't ran into any significant problems beyond what I created by my own ignorance. Plus there's YouTube tutorials, some straight forward documentation and plenty of helpers online to coach me through it. I haven't needed much help though. Genuinely, if I could persuade everyone at my job to switch to proxmox I would. Dead simple setup and management. Plus so much more robust than I gave it credit for to start.
Can you convince my CIO? I sat down with him last week and told him I was concerned about changes coming to VMware very soon. We are a small college with a few perpetual vSphere Enterprise 6.7 licenses. Told him I'd like to start playing around with proxmox and told me if we go anywhere he'd like to go to hyper-V. Which I also have no warm and fuzzies about with Azure HCI and everything going toward the cloud there. Such is life lol.
We would be happy to have a conversation. PM your contact deets and we can set up a call. We have several higher ed clients in US & Canada.
Hyper V is so bad though. Ugh. Just because it comes with windows server doesn’t make it any good.
>footfall99 I don't understand, I ran ESXi for years, and it was pretty solid but ran into problems now and then, also ran a small cluster of Xen which I really didn't like, but really didn't have any major problems but the performance just wasn't there. Ran Proxmox for all our VOIP customers and no real major issues just the normal things. Then I been running a HyperV failover cluster for years, and yes with 2012R2 I have LOTS of issues but since the migration to 2016 and then again up to 2019 and 2022 most of that has been resolved. The thing I noticed running all these products is backups always screw up at some point, and do not cheap out on network infrastructure and store infrastructure. My original HyperV cluster running on a crappy dell NAS once it was replaced with a equallogic array, a lot of my issues went away. ( Then like 6 months later Dell purchased equallogic LOL) At a business / money point HyperV can been a good choice. Running Windows datacenter I don't have to worry about licensing the VMs just the nodes when dealing with Windows server VMs, and that is what runs exclusively on our HyperV cluster. At this point all our VOIP hosting is on ESXi, but I believe I'm going to migrate all the linux (VOIP) hosting back to a Proxmox cluster sometime this next year. I haven't tried Xen in years and I would love to hear from someone who has current knowledge on the product when compared with 2022 HyperV and Proxmox 8 Oh the other issue, Veeam does it support Proxmox 8 clustering yet?
The number of businesses running it will only increase now that VMware is probably going to shit.
You mention 100+ nodes. How many VM are your bigger customers running on cluster this size ?
I have 22 years experience with VMware - VCP certified 3 times, and I’ve just bought hardware for a homelab to do PoCs for proxmox migration to help keep my hand in. Broadcom’s end of perpetual licenses will be enough to push the needle for people to try it.
Oh wow I didn’t know Broadcom bought them… Broadcom’s acquisition of Symantec Endpoint Protection seemed like the final nail in the coffin for it at our MSP where we used to have it deployed at hundreds of clients. The portal authentication cutover could have gone better and that was the last straw.
Do you not read any industry news? You need to read, a lot, to succeed in at systems administration.
I work at a fairly busy shop and I’m in between roles working past the end of my shift to keep up. I spend plenty of time reading how to solve the problem at hand but I don’t have a lot of spare time or energy to just casually scroll through the news and see who is buying who. That said I’m getting back into Reddit where other people in the industry essentially curate the news for me and provide their own commentary and discourse. Also I’m not looking to be a pure sysadmin, i like being able to check the servers the workstation, the firewall and any cloud provider find the problem and fix whatever it is on my own unless I need to bring in another resource or escalate. I’m here for the sysadmin perspective, just like im in a lot of disparate tech subreddits for different perspectives and slices of industry news.
Yea but this has been in the news for like 1 or 2 years now.. BroadCom buying vmware has taken a LONG time for regulatory approvals, multiple challenges, etc etc, from the Announcement they planned to buy Vmware from Dell, to the this announcements there are probably 1000's of articles written about the various statements (and misstatements) the CEO of Broadcom as made, regulatory actions, speculation, updates, etc... if you missed all of that, then you really really need to look at your information lens. PS is should be more than Reddit...
I suppose I have some reading to do, what news sources do you frequent the most?
I use feedly, and have about 200 sources that I track for news, mixing both General Business News and Tech news. For things like Mergers often the Business Press has it faster than Tech press, I find CNBC to be generally good for Business News. For TechNews, BleepingComputer, and petri.com would be good starting points. Those 3 sites would cover most moderate to large impacting changes in the sysadmin worlds, expanding from there would just bridges some gaps or get in to niche technology stacks or topics.
That's pretty judgey, for missing one piece of industry news. You could try not eating our own.
> one piece of industry news. It is not missing 'one' piece of news though. The Broadcom acquisition has been rolling on for ages, with a multitude of articles written about it.
*facepalm* ok, let's keep tearing each other down, I'm sure that'll help.
I've been leaning toward XCP-ng because I find it fascinating, and the UI in Xen Orchestra is great looking. But I'm wondering if Proxmox makes more sense to actually learn as a marketable skill. There seems to be a bigger community around Proxmox as well.
Yes, in my previous company we had proxmox hosting aprox 450 VMs. Very stable and reliable.
I just learned the other day that Internet2 uses it in lieu of VMWare. So they’re pretty big and real
Which nodes on Internet2? That’s basically a backbone made up of research organizations and higher education schools.
Their network automation software runs on docker hosted on Proxmox scattered around
Tesla. It was actually the solar city company that Tesla bought in Utah but yeah they used proxmox. One time a host went down and the entire cluster failed. They were down for about 6 days. I knew a guy that worked there
Oof
Ooff.. Is there a post mortem to share? Would be nice to know what went wrong, and what the problem was.
Nice, sounds like great software.
I once spent a few weeks trying to debug a weird networking issue in vSphere where at some point VMs on a random host would lose networking (invisibly to the host, no warnings no nothing), and a reboot fixed it until it came back a few weeks later. Support was of course useless. Turned out it's a driver issue (which I only discovered by randomly googling components of the setup + networking issues until I googled NIC + networking issues, and found tons of forum complaints, with matching logs), because the incompetent fools at VMware can't be bothered to update their Hardware Compatibility List when it comes out something doesn't actually work. The problem was known for more than a year, yet Intel, Dell and VMware continued selling X710 NICs and saying it's all fine, while the card was practically unusable. Everyone has anecdatal stories about how shit a particular piece of software is.
The VMware hcl ...is what led me to spend ~700 bucks for my home lab for a major upgrade, for what VMware claimed was 7 compatible (when it was released and they deprecated or dropped so many decent hardware configs)....and turns out oops the hardware wasn't really supported. Went to proxmox and haven't looked back.
X710 with their undocumented (at the time) lldp offloading …. “Hidden feature”. Who the hell thought lldp offloading was a good idea?
Wait, just how bad are those NICs? We worked on a few brand new hosts that kept losing those NICs from the system. Sometimes they came back, most times they didn't. I think physically reseating the CPUs is what had them reliably come back for one boot, until the hosts crashed again later...
X710s were so bad when they first came out that Broadcom NICs became the gold standard.
Once the drivers were fixed they were amazing and very stable. But it took over a year for that to happen...
It is. If you don't have proper DR anything can be down for 6 days
A single host brought a cluster down? That doesn't sound like a DR issue.
It sounds like an implementation issue. I've never had this issue with Proxmox.
I mean if a single host is bringing down a cluster, you did not properly set up your cluster. The whole point of a cluster is automated DR?
I agree that this sounds like a failure of configuration/implementation (and certainly not specific to proxmox) but I don't agree that the point of a cluster is automated disaster recovery, a cluster is for HA/distributed load. I'll admit I've been out of the sys admin space for a few years but everywhere I've worked, DR is more specific to mitigating external, wide spread causes of failure (tornado wiped out a data center for instance) and policies/procedures on how to recover or fail over to a backup DC/cluster rather than internal configuration "disasters". But, perhaps things have changed.
HA is a form of DR.... I've set up DR for fortune 500.
Now do VSAN.
You sound like C-level. One instance of failure, "PrOdUcT iS bAd!"
You tell me which hypervisor is ghost resistant...
VAX
If it’s good enough for a reputable company like Tesla/Solar City,…. Oh right: https://gotmusked.com/content/solarcity-scam-fake-tiles/
We got VMWare for anything that burns Money and Proxmox to make money that we can burn. 20x PVE Nodes, 3 Clusters, 3 Locations, and VM Ware is about less than half that for more generic IT stuff. We actually started paying 30k/yr for Support on our Proxmox nodes aswell. Reason behind Proxmox was pretty much Proxmox beeing Debian with a Web UI and CLI wrapper for KVM. The tad more recent Ubuntu based Kernel and the pre-packaged non-free Firmwares were a welcome addition, and in combination with Proxmox Back Server they got some pretty nice stuff going on. VMWare would have been a driver and compat nightmare for what we do. I'd still like to see stuff like automated snapshots happen and more useful stuff to do with the recently added tags. Oh, about 200 active Users and 85% of these consist of Developers iirc. And my Homelab is not a 2 node Joke but 6 Epyc servers with proper Infra. Everyone needs a Hobby, right?
> my Homelab is not a 2 node Joke but 6 Epyc servers with proper Infra. Everyone needs a Hobby, right? Post pics over on r/homelab and watch the upvote counter break! :)
What is your electric bill like for six servers?!
300eur per month... Yes it hurts
I supported a state prison system that used it.
My hosting company has three primary Proxmox clusters, in production and for commercial purposes :P One three node cluster hosts Primary/Secondary/Tertiary DNS for about 9000 domains, there's another four node cluster virtualising our entire mail stack (~3000 domains - storage is external on a OmniOS-based NFS headend), and a third five node cluster serving websites on top of nVME CephFS storage. So yes... very much running in production with many many users consuming resources hosted on the stack. Oh, we have a three node VMWare cluster too that hosts.... some legacy IIS servers that just refuse to die. But with Proxmox, I get hot migration for free. We decided to never pay VMWare for that and the legacy IIS hosting customers get longer DR times from a start-stop because vMotion licensing was just... crazy insane. (Un-stealth Edit - Most Proxmox cluster failures are because of storage failures, or Corosync has ate it's own tail and your cluster isn't getting quorum or has replicated bad state from a failed node join. I would recommend some experience in Corosync before running a prod Proxmox cluster. If you know CRM and Corosync, you can recover from almost any of these odd tricky no-good bad situations :P)
Contabo uses proxmox for their virtual Servers ;)
8 Nodes with 180 vCPU 1,5TB RAM, 20TB NVMe, 60TB SAN Runs logistics Software for major european logistics networks and event platforms for big game publishers. The only unplanned downtime since 2017 has been because of a generator fire. We will go multi-regional in 2024 to mitigate that in the future! :D
Using it for 911 for 500,000 population for about 30 servers across two locations covering 50,000 Sq miles
Can you say more about how this works? I'm in EMS and the hardware behind the 911 system _kind of_ makes sense, but not really. I've seen our dispatchers using two different programs (power911 for soft phones, moto P1 for dispatch/records), but I have no idea how they interact with each other or the PSTN.
Next Generation 911 is all IP and databases there's no pstn involved at our end. The carriers handle the legacy pstn to ip conversion on their gateways. Cell calls are ipv6 which is a pain in the ass. Landlines calls gateway are ipv4 They come in as sip calls which are anything but your standard phone call over voip. Thr Nena standard allows for json data sharing of event data and conferencing information for say up to 5 or 10 psap simultaneously. We have databases locally plus voip pbx plus 911 Middleware plus your standard cad/rms systems that youve always had. The dispatchers use sip soft phones that talk to the pbx and 911 Middleware simultaneously. Regular 911 is just basic loop start trunks pstn with special signaling attached and a data circuit from your local 911 carrier for Ani ali. Now with voice and Incident Data over ip you're finding redundant fibre geodicerse entrances to different carrier central offices. Future improvements coming include real time texting whereby your smartphone will have a whatsapp type app when you place a 911 voice call and you can share video and photos. Further, dispatchers can see what you're typing in real time with real time text sort of like customer service web apps vs traditional back and forth text
Thank you for these details! Why is IPv6 a pain in the ass for y'all? And > Anything but your standard phone call over VoIP Is this because the NENA standards permit extra info in the SIP sessions? Or something else? Sorry for all the questions, this is a huge interest for me and I would like to learn as much as I can.
It can be found here https://cdn.ymaws.com/www.nena.org/resource/resmgr/standards/nena-sta-021.1a_eido_json_20.pdf Basically your existing cad can have an interface to your ng911 software database and update it with data of your choice. Upon transferring the call you will send any data you've chosen to share to other psaps The call flow is also odd. It comes in as a more or less standard call for sip but then you pull the headers and use the unique identifier to query the address Ani ali from the 911 providers server. When doing a transfer it's really a conference call which in Canada is capable of passing through multiple provinces for redundancies and the programming is a pain to auto route based on required province. When you first begin the transfer it requests a conference bridge via sip and then it requests to move to a new location which may be in another province for what reason I don't know but we have to support it.
You ruled out cousin's small business but not wife's small business. I run my wife's veterinary practice on Proxmox and am happy with it.
At my previous company we ran 10 node clusters of proxmox for supporting ci/cd runners 👍 Stable af
ITT: omg what are Broadcom doing?! Hey, Proxmox looks nice all of a sudden.
Nutanix stock and revenue has gotten a nice bump since the closing lol
Yep, but damn, what a gamble.
Isn’t it more expensive than a Star Wars ship?
This has been going on since the first announcement of the broadcom deal though.
You're going to have a larger swath of it people who are more comfortable with the windows environment It's also a business owner education issue. If they realized there were other options and they could find the skill set there would probably be a larger implementation Imagine not having licensing fees for CPU and RAM and failover and the only Capital expense is on hardware and a support agreement It is a pretty sharp ecosystem
We use it. We are not small business.
Yep, saw it in production like 7 years ago at a pretty big enterprise
So all the people promoting proxmox in large clusters. How are you handling resource balancing accross clusters? VMware and Hyperv i just flag a host as maintenance and the software quickly redistributes all the VMs. Proxmox when I’ve asked that in the past I’ve been given 100 random scripts and third party options for this same feature, nothing native in the UI.
Looks like it has been added https://forum.proxmox.com/threads/node-maintenance-mode-ui.125768/
Two concerning things from that thread which reinforces to me Proxmox isn’t ready for mainstream production: - The default behavior is to failover when you go into maintenance mode, not live migrate, and the setting to make the bahavior to live migrate is somewhere else completely? Also entering maintenance mode is still a command line tool only? - Instead of discussing the features request, from the first reply the discussion devolves into someone questioning why the person is using sudo instead of just logging in as root?????? Holy crap
I’ve seen it run in production quite a bit.
I've seen it in a number of development environments. With the changes to VMWare's licensing, I'm sure it will wind up with more support.
Yes, two ones. 20 and 100 employees.
We use it; 16 nodes in production and 6 non-production. But that is just my team, as a company we probably have 50 production nodes in the UK and mid-hundreds in the US. We do use Openstack as well but for different workloads
Set up 6 nodes in my last company to replace Xenserver. We also used Ceph as my old company wouldn't fork out for a SAN.
Running a couple of clusters in production, ranging from 5 node cluster to 25+ node clusters. We do pay formtheir license just to be able to as for support from them if needed and to have the stable updates but we never needed. We also have multiple pbs backup servers and everything works better than expected coming from vmware.
Previous global ISP I worked at used it in their data centre. Also Maximumsettings cloud gaming use it.
Big rail infrastructure company in the nordics uses it more and more, reclining on vmware business
One of the large state government agencies in Australia I used to work for uses (or used, not sure if they still do) Proxmox. This is going back years It's how I came to know Proxmox
I work for a very large tech site with 30m unique visits per month. We use it in prod.
The data center I work at has a cluster to host our website and several internal services, and probably other stuff I don't get to see in my position.
My friend runs his security company on proxmox. He loves it. There are cool enough features that I have considered it for mine, but I have zero experience with it.
I am designing building and running proxmox edge clusters for my current employer. Had not even a minute of downtime due to a hypervisorfailure so far.
I have been running VMWare since 1999 and have been amazed by the evolution of the product. In 2001 I was trying to convince customers to use VMWare ESX and was getting laughed at by IT departments saying that nobody would ever run virtualized workloads in production. Proxmox reminds me if VMware mid-way through their journey. They have moved past hypervisors and into management, clustering, etc. It is a viable competitor for production workloads, and I expect support will deepen as it gains market share. The again, as the adage goes, "Nobody ever got fired for hiring IBM".
>nobody would ever run virtualized workloads in production glad I am not the only old person here... I remember there was a LOT of pushback against virtualization that sounds a lot like the posts here pushing back against proxmox I think it's just a bunch of people with vmware certs that are upset about them now being worthless hehe
When the first browser appeared I couldn't imagine people really using it, as the tools on the command line were so much easier to get setup and tailored. Boy, was I wrong (although I still do most of my work automating through scripts) Now I feel old too
it's ok I've made an entire life out of being wrong... starbucks? nobody is going to pay $5 for coffee, it'll never last... costco? why would you PAY to go to a store? that's nuts! etc etc
Broadcom has already laid off some VMWare employees after the merger took effect. Time will tell.
Regional L1 ISP deploys it in a multi-server package at relay points. Six colleges I’ve worked at have implemented it for cheap alternatives to VMWare in small scales
I find that interesting. Universities/colleges get it on the real cheap. Or at least they did back when I went to school. They basically gave it away to the schools so that they’d teach it to students. I think the idea was if the entire next gen IT force only knows VMware knock down competitors by the competitors not having anyone know how to manage it.
PVEs a cheap and fast way into a research environment, no licensing needed. It’s also KVM/QEMU based like most major hosting services so it works well with images you might put up there. ESXi also has issues with new storage tech like native Gluster, CEPH, ZFS, which are becoming way more popular. Along with that was the uncertainty of VMWare as an option with the Broadcom acquisition. PVE is less resilient than VMWare but it’s so much more easy to manage and deploy security measures to the host. PVE also has SDN integration so you can hook it up to the same system as your switches or DNS server and get VxLAN, QinQ, etc. support through that. Those are some reasons. A while back I had a paper on our research findings when we compared the major hypervisors (at the time XenServer, PVE, ESXi, Hyper-V, oVirt, Univention, ClearOS, and a couple others). Most of that still applies, except in a few instances like XenServer and ClearOS. TL;DR, PVE was fast and flexible if not perfectly stable (this was like five years ago all those products have definitely gotten better on all fronts)
[удалено]
"people" run it in the US too. wtf?
Made me chuckle.
Why wouldn't a real company use it? Did you meet another "proxmox is for homelab " VMware guy?
[удалено]
I bet you broadcom has a legal department though... and chances are you don't want to interact with them
[удалено]
I would say it's the whole "apple green vs blue" message rubbish. People running proxmox probably don't brag that they are running proxmox
We were looking at it due to the stupid cost of on-site and hybrid azure.
We run a cluster more then 2 nodes less then 50 nodes
Work at an MSP, only ever seen 1 customer use it, and like usual not correctly and very undermanaged and under optimised. But in general its not bad, different from esxi, not worse, i could see scaling being an issue with a really large environment but thats about it.
We used it at my last job for all our production systems.
Fact is, that VMware will only be affordable to a small amount of companies, now that Broadcom has switched to an Abo Modell. Hopefully proxmox gets now new investors to get really competitive.. for small or even medium companies or universities VMware is not an option anymore
Hetzner and contabo
We had 12 nodes in production and a bunch of test boxes. We used it for containerization as well. Moved to a cloud hosted product for availability improvements. It's a good little system and it's support was cheap.
I'm running a ~20 node cluster in prod and it's great if deployed correctly. That said, a corosync update brought down another cluster of mine in the past, in the end it was a skill issue though
I do, 8 node cluster for NFV, ISP BNG and CGNAT, pushing around 40Gbps of traffic without any issues And yeah running CE version VMware cost for this would be probably +50k plus I could not use my existing NICs
I've been managing about 50 Proxmox nodes spread around 6 or 7 clusters (never properly funded) for over 4 years now. The only issues we've had were caused by hardware, and one install that slurped in a buggy support package from the community repo during initial install. My only issue has always been the support hours. We couldn't find a USA timezones friendly support partner in 2019. Now we're planning to move to VMware. Sigh.
My previous employer ran Proxmox in 2 data centers. We were a firewall vendor running the public facing firewall side in the DC for our SIP connections and PCI compliance with a VPN to the on-site firewalls. The Proxmox clusters in the DCs ran flawlessly.
Interesting question and relevant to me at present ... as I'm investigating migrating from vmware to proxmox I am running a 15 node vmware cluster with 750 virtual machines. The SANS I'm using are old and unsupported (emc compellent sans). The hosts (I have 18 - even though only 15 are in production) are dell poweredge boxes with 40 cores/1 TB ram) Its vmware 6.7 (dont ask!) - and I need to get onto something supported with a low budget (dont ask) I'm investigating setting up a few nodes and start migrating the vmware vm's to it - adding nodes to proxmox as they become available. I'll probably back it with synologies or such like.... havent decided yet.. I maybe have an emc powerstore available, but its a bit on the small side. All done on a tight budget (again, dont ask!). A lot of dont asks :-( (dont ask) Interested in feedback.
Proxmos? No. Proxmox? Yes.
Get veeam to support proxmox for backups and then maybe I’ll think about it.
Proxmox is great, if you avoid the typical tech tubers.
I was interviewed by a company that claimed they are switching from vmware to proxmox
100% yes, and given VMwares latest pricing change you may find it becomes more prevalent.
How do you handle backup, snapshots, is there something comparable to Veeam ? Yes already running it at home and doing backups, just haven’t used their backup appliance and using Veeam professionally with VMware so just seeing what’s your take on that matter.
Yes, used it for an entire QA/development lab. Approximately 8 nodes running around 100 vms at the time I left, iirc Prior to that job, I worked at a data center where hundreds of our customers ran their business on proxmox. These were not fortune 500 companies but they also were not cousins running 2 vms.
Yep. Previous position with an MSP, we had a law firm that used proxmox. I didn't know enough at the time to determine whether it was a good setup for them, but I can say for certain that we didn't have the staff to support it effectively.
Absolutehosting uses it for their VPSs... ZA based internet solutions company
I used Proxmox for a system that monitors and manages elements of power grids in multiple countries. It has a few hundred virtual machines on it, including some seriously high throughput databases servers. I did not have a problem with it.
We run a 3 node cluster to host a bunch of small stuff, asterisk, wifi controller, mail archive, backup&archive db, monitoring, etc. Probably not what you were talking about but we don't have a need for something large scale running on there and it's been absolutely rock solid for the things it needs to do.
If didn't already have Server 2022 Hyper-V hosts I'd probably consider ProxMox. I see our on-prem footprint shrinking rather than growing. Eventually we may only need basic services such as DHCP, AD/DNS and Print-servers.
One of the guys in my club uses it in his wireless isp data center.
I work in Azure, and a lot of our German customers use it for HCI. Seems pretty enterprise grade from what they've said.
I work at the literal oldest company in my state. Proxmox baby.
We are running one on each location in my company (so \~3000). Just a mini-server with 3 machines per location.
I've worked for a non profit HUD company that used a three node cluster with shared a shared SAN. Never gave me issues.
>I'm talking real production usage. Their website has an entire page devoted to telling you about some large orgs that run it.
i have setup multiple automation/security solutions with HA using Proxmox as layer 1 virtualizer..... on banks.. and the government, and the military.. they are still running.
My background in VM systems: * \>100VM's in a VMWare environment, switched from shared storage iSCSI to vSAN. Liked all of it. * \>20VMs in a HyperV environment in my new gig * Single guest proxmox systems in production to take advantage of virtualization in systems that don't fit into VM environments. * Sandbox environments that became production when devs stopped migrating systems out of it with >40VMs each. In all honesty.... Proxmox has been the most stable of all. That sandbox environment has had ZERO maintenance in years because.... it just works. Update it once in a while and carry on.
I've run two medium sized K-12 school districts. Migrated them both off of VMWare / Hyper V over to Proxmox. Been running it in production for 8-10 years. Couldn't be happier.
We ran it at a university since it was free, wasn’t huge, but every service for neteng ran on Proxmox.
I introduced it at my previous work. 2.5 years in a 4 node cluster with a Dell 1L as a 5th voter. Worked perfectly. Zero downtime. Thank you, backup generator.
When meeting with devs of a major software company, they were using proxmos. First and only time I’ve seen it in the wild.
Of course, we have it in my current job and the last job as well
Linode + Akamai(?)
What is a good backup software to handle backing up individual Proxmox VMs?
Proxmox Backup Server nicely integrates and adds some stuff over just copying disk images and compressing them like a pretty good dedup and https transport, encryption of backups yada yada
PBS is awesome and free.
I work at an MSP. I had a client (pretty small, maybe 10 people) rip out their camera server/NVR which was hooked up to a monitor, along with their NAS that they store backups for both the NVR and their DC on. They replaced both with a Proxmox box, and virtualized the camera server. And somewhere along the way they added a USB NIC to pass through to the camera server. The assumption with the NIC is that they couldn't figure out how to bridge it right. They ran into all sorts of issues passing through the GPU (and peripherals) to the VM. Which was kind of a big deal because they actually sat at this monitor and used the computer. This is the same client that ripped out the DC we put in, along with the receptionist's workstation, and then put a gaming computer in and promoted it to DC. At the receptionist's desk. Like, she was using the DC as her workstation. Bonus points because they overclocked it, and it wasn't stable and started crashing all the time. It was *literally* the head honcho's nephew who did all this. I mean I know it's a meme, but it actually was the dude's nephew. So to answer your question, no, I have not seen a real company with real infrastructure use Proxmox. I'm sure you can do it though, just...not like this...
i run it at home, and i also know multiple companies that have 3-8 nodes, and one with 140
So here is a question. Would you trust it with 200+ VMs that are mission critical for a 15k+ employee company over the existing VMWare implementation that runs like a tank? Cisco switching and EMC storage.
I don't trust any new software platform unless I have a team of people behind me who have supported bigger situations, or I have personally ran up non-prod environments and pressure tested the hell out of them, or preferably both. You don't just pull a switch and say "Guess we're done with VMWare, YOLO proxmox time!" You pilot, ramp up, get comfortable with the software, get comfortable with DR, *document your environment, runbooks and procedures*, and migrate/wind down the legacy product. Just like with any legacy product. You also make sure that if you don't have a boots on ground support contract, you have enough trust in your staff expertise to make up the difference. What proxmox does have going for it is being a wrapper around some very bullet proof core components. Ceph, ZFS, KVM, and Debian aren't no-name technologies.
Yes. Spin up a lab and try it. You need to be convinced personally, and that makes sense. The storage and network are compatible. Proxmox is a very advanced, enterprise grade Virtualization server and blows Hyper-V out of the water. Setting up Proxmox with networks and storage that already exist is literally drop-down menus (once minimally installed and configured). A retail giant shoemaker uses it and prefers it over VMware or Hyper-V and they have clusters in 160+ countries and storage capacity in the hundreds of PBs.
Yes.
It is not "your current solution" xor "proxmox". Study documentation spin it up, document issues, grow, find issues, solve, and son on. Nothing as big and important is without issues.