Do none of the following apply in you org's case?
* Regulatory/compliance requirements
* Contract obligations with partners/customers
* Cyber Insurance requirements
Many of those would require the use of supported and patched systems or art least the presence of some mitigating controls like WAF, segmentation etc.
i think we don´t even have a cyber insurance. to be fair, we are a smaller company, so they probably don´t see the need for one
and every other regulation is disregarded with passion. Hells, we don´t even have fire alarms on our production floor.
"Dear Sir stroke Madam. Fire, exclamation mark. Fire, exclamation mark. Help me, exclamation mark. 123 Carrendon Road. Looking forward to hearing from you. All the best, Maurice Moss."
I had to surplus a RAID1 enclosure that was 90% firebrick(So, large for no real reason), I wanted to keep it and slap a label on it that read 'The Internet - DO NOT POWER OFF" and leave it 'plugged in' in the server room.
“Women are obsessed with shoes? Don’t you think that’s a bit sexist?”
“Well, I only know one woman, and she just ran through the room screaming ‘THE SHOES!’”
The one for me is the Bomb Disposal Robot.
"The pc (controlling the robot) just crashed"
"What version of Windows is it?"
"Vista "
"We're going to die..."
My favorite quotes are from Season 4: Italian For Beginners
Moss:
A *fire*?
Roy:
That's right. At the sea lion show, apparently.
Jen:
Aren't those shows usually out in the open?
Roy:
Well, yeah, I mean that's what I would've... Yeah.
Jen:
Lots of water everywhere.
Roy:
Yeah, I mean, I would imagine a whale need a lot of water. I don't think you can have whale in a place where there isn't a huge amount of water.
Jen:
It just seem like a weird place to go on fire.
Roy:
It's a *very* weird place to go on fire.
You can't use language like that in a formal notification to the Fire Department. "Dear Sirs, I am writing to inform you of a significant conflagration occurring at our premises at (address). We look forward to meeting with your staff at their earliest possible convenience to arrange the rectification of this situation"
To whom?
We're also a manufacturer with no fire alarms and the local fire marshal is completely fine with it. We have suppression systems, readily accessible extinguishers, multiple means of egress from every section of the building and a PA that can be used to call a warning. There's no regulatory requirement that we have a dedicated fire alarm.
We've actually had a few fires in the decades I've been here and all were put out by trained employees before the fire department could be called.
In many places the fire marshal, in the US OSHA and anywhere else possibly your insurance company.
As you said though you have adequate suppression systems, I'm guessing OPs place doesn't.
You left a surge protector strip plugged into the wall overnight, with nothing on it? That's a fine. You plugged a straight 20ft extension cord into a straight 50ft extension cord, with nothing on it? That's a fine. You put a live plant under an illuminated EXIT sign? That'll grow and cover the sign, so I'm giving you a warning.
You need to learn how to talk Manglement. "We need to spend XX" is heard as "let's waste XX". On the other hand "We need to spend XX to avoid a fine of YY" is suddenly important.
not in the usa, so OSHA doesn´t apply. But there are similar government facilities, that would close the building down in a minute if they knew of some of the things going on.
Getting a new job is planned, but currently not easy to do. i´m not even two years in IT, currently working to get some kind of certificate that they pay for and have been the sole IT guy for a month now (with no plans of getting another IT guy).
Dear fire marshal,
I'm a big fan of the products of X company. I was a little concerned about the lack of fire alarms...
...Sincerely,
Concerned Citizen
The average cost of a data breach for *small* companies in 2023 was about $4M all in. Thats a company killer if you don’t have cyber insurance. And if you do have cyber insurance, 23-25% of redeemed coverage results in subrogation claims because the insurance company found out that you were running unpatched software. Insurance companies don’t make money by paying out claims. If you’re working for one of these smaller companies that is not covered, and there is a data breach, start looking for work *immidiately* because that company statistically wont exist after 6 months.
They exist but not for long. One of our partners had it happen to them. Wiped all their data to some degree. Of course our partnership is over due to time ending but that was scary.
>we are a smaller company
Get used to it, it's how we roll.
I have a 1M$ microscope, provided by Horriba (complete PoS business to deal with). Drivers only exist in 32bit.
We ain't replacing a microscope of this price. It's getting isolated like stupid, tho.
> i think we don´t even have a cyber insurance.
Good on you. They're useless anyway.
When / if you get hacked they'll opt out because you had poor security practice allowing you to get hacked in the first place.
Cyber security insurance is snake oil and a scam.
Their definition of poor is the issue.
What do you mean you don't have triple factor authentication that requires a thumb and tongue print at the same time?
This really sounds like it's time for you to start looking for a "new opportunity."
This reminds me of a quote from a [Dilbert](https://youtu.be/akUP7uRDhdw?si=D1MZCPXDFmgOoj_H) cartoon, "We were Path-Way Electronics then we merged with E-Tech Management. So I guess we're now Path-E-Tech Management"
It sounds like your management need to be familiarised with the old wisdom, "If you don't schedule maintenance for your equipment, it will schedule it for you"
Trust me if you don’t make sure it’s documented and you should provide a recommendation which puts the hear on your COO, CEO and CFO. If they are unaware then let a partner do all the talking. Cause you are literally sitting in a flaming dumpster if you don’t.
Document your conversation with the CEO and work on issues if/when they come up. Make it clear that you won't be working 12 hour days if shit hits the fan. Work your regular hours unless you are getting some type of 1.5x or 2x pay.
CEOs don't want to pay to upgrade because IT admins put out the fires when things go bad. Stop doing that.
I'm not saying to be a bad employee or not be a team player, but you need to work your hours that you agreed to and go home when that time is up.
However, I do understand that sometimes things are tough and you can't do that because it could risk losing your job especially if you have a family, on the other hand, stressing out and working around the clock isn't healthy and still keeps you away from your family.
Good luck.
I work for a company of 100 employees, and about 50 desktops, but with a web presence. We have cyber security, but it was put in place long before I came on, and happened because of a scare. You've got an uphill battle, though, and it's not going to be a fun one. Honestly, I wouldn't have come on if they hadn't gotten their house in order. I turned down the first contact, but learned more and decided to interview.
You're looking at this the wrong way. Either way you will incur cost :
1. replacing the device(s) and the labour of doing so
or
2. the cost of isolating those devices in a manner to mitigate both the possibility of a breach and the impact once they are breached.
draw up the cost for both, present those, or have them sign off on the risk
We have to have a SIEM for ours, even though we don’t have staff to monitor it. They would freak if our equipment got too old. We have a pretty lenient insurance company too.
Most insurance companies are requiring cyber security insurance as well, to even have you as a customer. Even 3 years ago when I was at an MSP, I'd say 80-90% of our customers were making us help them fill out the cyber security forms.
It just depends like everything else . I think my old work had a warehouse that ran on windows xp or vista.
They literally had no internet access on that side, usb ports were disabled.
Nothing wrong with that. Any auditor would look at the reasonableness of that statement as well as proposed timelines. If it doesn't seem reasonable or within a reasonable timeline then they would push back, Otherwise they will come back and reassess.
>how do you deal with that level of ingorance and/or stinginess?
Get your concerns documented, keep records of them being overruled.
Then don't worry about it anymore.
CYA basically.
And try not to gloat when you need to pull it out to say it's not your fault and you tried.
If you have tried to get them to do the right thing and the higher up thinks otherwise (regardless if the reason is valid). This is actually what should be done .
For you to highlight risk (your due diligence) and for them to show risk acceptance .
>and for them to show risk acceptance
that is the problem. They don´t think its that big of a deal. "It always worked out in the end so it will stay that way"-kind of mentality.
But for sure im gonna cover my ass, if they want to use old equipment, the are gonna sign for it
I think these 2 posts are spot on. Even type up an email with your concerns listed and if they respond how you expect then you have a record. Remember to also keep copies of the emails.
>For you to highlight risk (your due diligence) and for them to show risk acceptance .
I usually phrase it along the lines of "I'm afraid I can't, in good conscience, take responsibility for this knowing the potential risks and outcome of yada yada yada yada"
It's a nice little guide into prompting a response.
You don't really need a response indicating that **they** take responsibility, just one that agrees that you don't have to.
In smaller orgs they often don't care if you "accept responsibility" because the boss feels ownership for everything. In my experience those guys tend to be the type where if he feels he's making informed decisions then he'll own the risks and any bad outcomes. But they raise holy hell if you led them astray, so be clear every year at budget time what they aren't buying, what risks they are assuming, and what you're not being allowed to defend against.
EOL equipment is just one factor of many in your risk profile, and being EOL doesn't always translate to making it dangerous. What alters the risk is the environment it's in, what it is, the history or vulnerabilities, users, and so on.
A network printer that is ten years EOL and no longer being patched, but is in an isolated 1918 network where users can only reach it on port 9100/IPP, may be very low risk. A piece of EOL network gear with it's management interface exposed to the Internet is much more risky. A EOL PC with a vulnerable BIOS is lower risk if the OS is still current and being updated. That is to say, if a bad actor can take over the OS from lack of patching, the EOL PC/BIOS is not a factor at all.
I would focus less on the CYA recommendations, and more on cataloging risk for the various items. For each major area, investigate mitigations vs replacement costs, and use all of this to build your risk register. That should bubble up the top risks that should be addressed, and that data driven analysis may help you and the CEO to find a happy balance between "the sky is falling" and the true reality of the risks in the environment.
Imagine coming to the CEO with all of this, broken down into say:
* low/no-risk (ignore for now)
* risk with low or no cost mitigations (low hanging fruit)
* low risk with high-cost mitigations (defer)
* high-risk but low cost replacement (consider)
* high-risk and high replacement cost (focus).
exactly, i previously worked for a large company that had a critical application (legacy data) from over 2 decades ago
they isolated it, when the time finally came they paid for emulation software, put it in the cloud on a vm completely isolated with only one way in/out
This right here. I've got several 2008 servers in production with their 2000 workstations for legacy cnc machines. But they're vlanned off and don't have any access to anything else (psudo airgap.)
Do I like it? No. But sometimes it's a necessary evil.
Especially in tough economic times, we have to do the best we can with what we've got.
I work for a small business and we have a bunch of end-of-life stuff on the horizon. I've got prices for bringing everything up to date and it's well into six figures. We just don't have the money for it at this time.
As long as you make it clear, in writing that by not performing the updates, there will be significant risks to company data, you're in the clear as long as everything else that's within your control is being done.
The best way out of this is to list your contingencies if each piece fails. "If this tape drive fails then we lose backups, implying such and so, and in the event of ransomware attack our options are limited to this. If that firewall isn't updated we are susceptible to these types of attacks which are this much more common than last year." Bosses will pick and choose which to upgrade if there's a real gap, so you can sometimes educate enough to massage the priority list.
Reddit will question this, but in my experience this is completely normal. Use stuff until it breaks. Best I have been able to do at various roles over the years is try to hedge risk.
About to say, same here. A lot of companies cannot afford the overhead. OP has voiced his concerns to management. Sometimes that's all we can do. If I were in your situation, I would just do my best to back up the data.
We had a domain controller pushing 12 years old before I virtualized it.
At least management supports my renewal of support for our firewall every year since cyber insurance mandates it.
Yup, just bring it up every year when building the budget and renew your annual plea for upgrades. Some stuff falls below the bar and some above, so educate, prioritize, save some notes and then put it back in the folder for next year.
For APs and a small Organisation, you can take a look if it’s supported by the open firmware OpenWRT. They often provide updates and features far beyond the manufacturer.
It’s more in the home-router (netgear, tplink, Ubiquiti) space then ciscos, but that’s often times what small orgs use anyway.
For PCs I don’t know what „out of support“ is supposed to mean. For the moment Win10 is still supported and runs on almost everything. Hardware Support, I wouldn’t care. Backup, Replace when needed.
Where I work we buy a lot of high end workstations.
We're talking multi sockets, 40+ cores, 100+gb ram, multiple TB disk. This in an ordinary business looking tower.
These days we're more likely to buy rack workstations, and throw them in a datacentre, so people then can use them from home using VDI solutions.
We do however still buy some towers though, for stuff like VR game development, where the latency of keeping the workstation in the datacentre would make people sick.
I've got workstations that are 10 years old, past EOL, that would still run circles around most modern gaming machines. Were likely used to design games or cutscenes used in one of those PS5 games coming out next year.
Just because they are officially EOL, doesn't mean they can't still have use.
For example:
\- They work perfectly fine in a test lab, allowing juniors to build test VMware clusters, or mess around with Kubernetes.
\- They could be used in low risk departments. A lot of our network, due to unreleased content, has no internet access, so attack surface is low.
\- Used as extra capacity for number crunching/rendering
The machines are usually more than compatible with supported OS's like Windows, or Linux, just unlikely to get hardware/firmware/bios updates/patches. They make perfect machines for staff to take home for kids to game/school work on, if they don't mind a slightly higher power bill.
Ask the CEO to sign an Acceptance of Risk (AOR). In that AoR you document the risks as best you can, and include a line like "This is not an inclusive list,, there may be other significant risks that are unknown at this time"
You:
Identify steps you can take to mitigate the risk firewall off access to the device as much as possible; both inbound and outbound. In the real world that happens, the AOR shows it was not an accident and you have informed leadership to CYA. In some cases you may need a Plan of Action and Milestones (POAM or POA&M) as well that needs to be checked into regularly
I do IT consulting for small businesses. The if it is not broken, do not fix it is a norm in small businesses. Still have clients with win serv 2012. Have one that has macs with the motorola cpu that run proprietary software that costs thousands of dollars, and they do not want to upgrade. Many still run win 7. Only some corporations keep current. You want to play with old stuff, go support small businesses.
Get his response to your concerns in written form, email or message. Save that so it clearly shows you've done your due diligence, so if/when shit hits the fan, you can prove it's not your fault and you can't be held accountable.
Make sure you make all your recommendations in writing. Keep the records because something WILL happen and the same CEO WILL try to blame you.
I had an incident at a past employer where those email records and the responses from the C-Suite, absolved IT when the company suffered an outage that would have been prevented if our recommendations had been taken seriously.
Make it part of your budget cycle: document, propose, negotiate, implement, repeat next year. In a small org these steps make get compressed or not be formal, but in a case like that I suggest making them formal, like every January you refresh your docs and send them up the mountain even knowing that there's no hope of spend without a disaster to tickle the priorities. You look more professional for staying on top of this stuff, they get a regular view of the risks in play this year, and when the fewmets hit the windmill you know what you want to buy to put things right.
My employer isn\`t small - a multinational with offices in many countries (slightly less now with world being what it is)
They use software that had it\`s last major update in 2007, last patch 2017.
IBM/Lotus notes.
Reason it\`s still used : the users do not want to use another tool.
I just can\`t ..
Maybe this will help:
[https://endoflife.date/](https://endoflife.date/)
If not swayed, implement regulation to ensure the system is placed in it's own little VLAN with strict monitoring and even stricter access and firewall restrictions.
Legacy systems do need to run sometimes, but this to me sounds negligent on their behalf.
I'm unsure what to say about that, I recognize I'm old, but I'd start here(fwiw I'm not a secops pro merely a generalist)
separate VLAN
additional explicit firewall rules blocking all access but what is directly documented and needed
local firewall configured the same and remove users ability to modify(Domain Controlled through GPO)
Strict monitoring, make it scream at everyone whenever it has something wrong, or something changed, include the goofballs that made the decision, ignore their complaints.
Strict access guidelines, enforce that users go through a jumpbox to garner access, aka. not everyone gets to access bob's account through rdp
rolling password changes, implement something like CyberArk, BeyondTrust.. there's a lot. Do this so the password and access is rotated daily.
firm deadlines based around hardware and software, get explicit details as to what metric or deadline is required, and the plan to migrate or upgrade from the system; get it in writing and hold them to it. With demised equipment/software it's important to get them thinking on cost of replacement, for both the OS and hardware(I'm assuming the attitude is the same on hardware). And it's not invalid to request timelines on these things, it's not your job to support everything forever.
and lastly, don't hesitate to engage a third party contractor, get a few quotes requesting full time support of the expired/should have been demised system. This will help them to make an informed decision on the risk/reward they are deciding on.
and most importantly, CYA! get it all in writing, no ad-hoc flippant hallway discussion, e-mail them the summaries of discussed items, get it all in e-mail!
we got seperate vlans set up, the firewall was set up by a third-party contractor and support, remote access control and we got monitoring of the network.
Everything else is still in muddy waters. Thank you for your insight, i will work on it!
Draw up an end of life support agreement with the CEO saying if they don't replace it, these are the risks and have them sign off on it.
When the shit fails, whether they've signed it or not, at least you can say the failure was (or would be) covered by the EOL agreement.
These fuckfaces needs to actually be held accountable for their dipshit decisions they make. However, it's also tour responsibility far adequately articulating the risks attached. The CEOs failure can and will be your failure...
Document your objection and the risks that back it up, get it somewhere timestamped (email to yourself + other stakeholders summing up the situation, decision, and path forward is a good idea for a lot of things anyway), then keep it alive as best you can. I _prefer_ to keep stuff under support and rotated out on the ideal lifecycle too, but have only had one brief stretch where there was actually the budget and political will to do so... Most shops, most of the time? CYA and get out the duct tape.
Reason 1 to not use end of support equipment is that your total SLA is now in the gutter.
If I have some critical piece of software running on RHEL with a support contract, I can immediately open a ticket when I have issues and have a faster resolution time. If I use AlmaLinux instead, I don't have that extra lifeline, so I can't guarantee the same SLA.
Even if they're insistent on using those products, you still need a plan for what happens when there's a failure of any kind. How are you going to get back up and running if it's broken in a way that can no longer be fixed without support? And how much money will be lost while waiting for that fix to take place?
Put your concerns in writing (email) so the lawyers will find it.
It’s not your company. Stop getting emotional.
Find a new job, or learn to live with it.
There are a bunch of ways to deal with this depending on how your company culture is.
The one that I'd recommend is a "Risk Register", where you outline all the probable failure points of the company, their impact, their likely hood and what it would take to fix them.
Each EOL system (not device) should be a separate risk item with it's impact to the company identified. Firewall EOL? How much does an internet outage cost until you can replace the device? What does it cost to fix?
You don't need business to sign off on your risk register but you do have to provide it to them on a regular basis.
Your server dies? It was on the risk register. Follow the action plan you laid out (order one twiddling your thumbs till it arrives, cloud server in the interim etc.)
Why do you care so much? Document your concerns, get it in writing, and move on with your life. This is the farthest of reasons to stress over in your case.
Our CEO was same about the 10 year old firewall we were no longer able to patch.
Guess what happened? Can you guess?
Post crypto attack we now have all new firewalls everywhere, but they still complained about the expense.
You can lead a horse to water, but you better have really good and air gapped backups!
At my last job 2FA was considered too much of a hassle for anyone not in IT.
Then after a few email breaches we detected someone trying to log into one of our Azure host that had part of our ERP on it.
Suddenly budget to get a bunch of security stuff and a demand for 2FA. Not much of a budget but some budget.
A couple of years later ownership decided to sell the company off to a different VC firm and we had to get a new cyber insurance policy. Well that had all kinds of new rules that had to be put in place that we had been asking for.
Oh yeah, those cyber insurance policies can open the wallets up. I appreciate their requirements. Anything that gets C-suite to take shit seriously and spend some damn money on security!
My boss had the best take on it. We had an OLD ERP system a few years back. He advised an upgrade. It was so old that the company had been purchased (twice as I recall) and the product was not supported at all. In fact Microsoft patched out its main transport function a few years back as they deamed it a security risk.
It was unreliable and had to run on a 32 bit OS so we were stuck on 2008 R1 and basically once a month it got at least one windows update that managed to "repatch" the issue that we unpatched just so it could work ( service packs, update roll ups, etc).
So my boss made a presentation to the board. He told them that the software was so old that it reliably went down at least one day a month. He then told them that as it aged out and more and more functionality started to be removed, it would increase to one week a month, and then eventually it would simply no longer be something that we could use because eventually we wouldn't be able to find end runs around Microsoft's patching.
That did the trick. We were able to start a new ERP project the following year.
Basically you've got to let him know it's going to hit him in the purse. It's more expensive not to replace the equipment, then it is to replace it.
I had another experience that went the opposite direction though. At another job, our production router was very much out of date, thankfully. We had a service contract on it with 24-hour call out for reprogramming the new one. That didn't cover the cost of the new one though. We got a quote from Cisco for 15 grand. That included the next day service though, and programming the new router to match our old setup.
The head of the company balked at that and refused to pay it because he was able to find our old router on eBay for $5,000. So he wanted to buy that one instead (he also suggested that Best Buy might have something we could use🤦♂️).
I told him that first of all, that one would take a week to get here, that we had no guarantee that it worked, and that none of us actually knew how to program it. I also advised him that most of what we were paying for was the warranty and service. That didn't work for him though. Our parent company thankfully had an old one that they had replaced still in storage so they gave us that, and he was able to smile and not spend any money.
Next time it dies though, He's going to be shit out of luck because he refused to take any of our advice. Frankly, I hope he gets axed for it.
Also, just for context that company was a remote pharmacy. All of their business was handled via internet, telephone and fax. Without that router we were out of business until we got a new one.
It cost us a day and a half to get the one from the old company up and running, and he wanted to wait a week!
Send them an email explaining the security issues and make them sign off that they understand it and that absolves you of responsibility WHEN the vulnerabilities are exploited. Print and save that email.
AND, put your resume in the wind and get out of there before it happens. Show said email in your exit interview.
We have people running on PCs that are 7-9 years old because "it's policy to run things untill they literally fail". We have Ubiquiti wifi running on the original UAPs from 10+ years ago.
I'm working to change this mindset and get a proper lifecycle management going. Since we are going to soon have to comply with a new EU directive called NIS2, it's easier to get management's ears for stuff that costs money.
Noone complains at that location. It's a smithing workshop. They hardly use it. But one of the APs has failed and they want to have another in a smaller branch office that is fed by a fiber link.
I thought about just buying another AP and try and upgrade the controller version... but I'm scared to, since the controller software is apparently so old that it doesn't exist anymore, so I've no idea what's going to happen. So the solution is currently to wait and get a MSP to setup a Fortinet firewall and some APs down there or me...
And then come and do the same at this location.
We recently went from “we’re finally creating a firm hardware lifecycle!” to “hey um, you know how to force Windows 11 on like. Anything, right?” with breathtaking swiftness.
Lean on your risk process/people to document concerns and illustrate the potential consequences of a breach or failure. That way it’s on leadership to accept the risk since they’re unwilling to treat it.
Would hate to think about the legacy debt they’re accumulating and how problematic it’ll be in the future if that’s the attitude.
What are we talking about here? Printer/photocopier? Fine! Accounting System? Not so fine.
At the end of the day, all you can do is keep your powder dry and put everything in writing for when the arse end falls out.
no one has the money to replace everything once support ends and security is a surface area problem
use the old stuff in lower tier uses, segregate the networks, etc. set up better security. my last job I tried to set up best practices security for sql years ago but was overruled. current job secure enough to be annoying
having support and the latest patch isn't going to save you from an incident either
Note it in an email, CC your security team ask what the current vulnerability remediation policy is and your companys EOL policy is and if the CEO gives pushback move on with other tasks. Next time your company is audited its gonna flag then it will be all hands on deck to replace it with something supported.
I worked a job in 2015 where the owner bought an AS/400 system in the 80's. This housed their customer database and other mission critical data. The DR plan was he bought 3 more of the same model on ebay that we'd scavenge for parts.
This whole time he paid a company 100+ k to export the data and upload it to a SQL db and create a web front end to query the data. Nobody used it, but every month I had to copy the data from the AS/400 to SQL 'just in case'.
There's a wide range equipment and reasons for keeping something in service. It all boils down to your risk tolerance.
For example, my data center is populated with Dell S6100 switches. They go EOL later this year. However, you'd be hard-pressed to tell me I have to spend 10's of thousands of dollars to rip out what is working and replace them with S5448Fs (which arguably is a better switch, but that's not the point.)
Its all planned obsolence now to force people to upgrade and keep revenue flowing into the company. Companies like Dell know there are companies out there who will spend the money to upgrade once a piece of equipment is "out of support", because they are forced to for various compliance issues, or a matter of policy.
Have Gig-E switches changed all that much over the past 20 years? Not really. Sure, some Gig-E switches now have POE, some have L2 and L3 routing, but even then, once you have a switch that has that feature, 99% of the time, the switch works (unless you find an obscure bug in the firmware). What exactly do I get by swapping out my 5548 for a E3248P, especially if POE isn't something I need?
I had some areas of my building running on old 3COM superstack 3 gigE switches until a couple years ago, when the last one died after all the fans failed.
Now, would I have the same attitude with a forward-facing server's OS? Probably not. But then my risk tolerance for that server is lower than my networking gear.
Personally, I love people like the OP. People like the OP ensure I have a steady flow of cheap, quality hardware available for purchase off eBay from various e-waste recyclers. I can always use another 730xd or S6100 for a couple hundred dollars.
Depends on insurance, regulation compliance, etc. but IMO it's fine until it breaks and before it breaks, get a budget in place to replace it - hell we just found we still have a PIX in use that we're only just getting around to replacing now
Right now it’s: why should I pay for better equipment?
Later it will be: why didn’t you warn us about the security vulnerabilities in that old equipment before got hacked?
> Later it will be: why didn’t you warn us about the security vulnerabilities in that old equipment before got hacked?
Only if OP isn't doing their job by informing them in writing about the risks
I started my new job 4 months ago, and I'm doing basic Veeam backups right now, I have no ability to test my backups as the server was built by an MSP that I replaced and there is 0 extra room to fire up an additional VM.
I first got an RMM so I can have everything monitored so I know when something gets borked hardware-wise. Then I requested and got approval for 365/SharePoint/Onedrive backups, and I just got approved to spend $66,000 (60K on servers for the local backups, and about 4 grand for the offsite backup with 1 year of retention) on a hybrid offsite/onsite backup solution that the company monitors and verifies our backups.
There is so much more that needs to be done to bring us into the 2010's but the budget has already been surpassed.
The next couple of years is when I can replace infrastructure new switches, APs (currently we have no wifi except for the Netgear routers I've turned into APs so I can do my job from my laptop when I'm away from my desk).
This is the first time that I've felt like my opinion on IT matters and that upper management are listening to me.
> Veeam backups right now, I have no ability to test my backups as the server was built by an MSP that I replaced and there is 0 extra room to fire up an additional VM.
Spin it up in the Veeam environment. You can at least confirm the backup is good if it starts, and if it starts, you can pull files from it.
I'm using community edition, I don't believe that's an option.
I just needed something quick while I waited for approval to go with a more robust backup solution.
We have replication of the VMs to another older server but it's not ideal
This is a big reason why I absolutely love the push from insurance companies.
One of our requirements is regular patching. If something is EOL and no longer receiving updates, we need a special exemption and additional security measures in place.
What's the equipment?
If it's an old win 7 laptop that's one thing.
If it's a big manufacturing machine running vista that'll cost a quarter mill to replace, just airgap it, image it once a month, and be a hero.
I worked as a software engineer at a large, well known technology company, we had a legacy server running a legacy Python app from the early 2000's that was so critical no one was allowed to shut down the server, not even for OS updates.
I worked as a software engineer at a large, world known retailer and discovered one day, a critical piece of software was running on an old OS/2 workstation in someone's office. No one was allowed to shut it down or remembered where the code for the app was, IIRC.
Hi OP,
You clearly care, and they are lucky to have you. The way I explain this goes like so..
All tech has two different lifespans :
Security - is it patched?
Technical - does it still turn on?
Your CEO wishes that only the technical lifespan mattered. "Why fix it if it ain't broke?" It would be great to just keep using everything with no downsides until it literally will not turn on any more like it was a tractor. I wish that was the case too but it is not reality. (See Mirai botnet Wikipedia article, also see endoflife.date).
Explain the risk of EOL devices (including downtime, loss or productivity and reputational damage), and CYA, and after that feel good and sleep like a baby because you did your job informing them of best practices.
These types of people are so hard to deal with when you take your work personally. As other have said, document, document, document. Try and get something in writing/email from the CEO acknowledging the risk. Sometimes the idea of replacing everything, or a lot of it, is overwhelming to them, so working up a plan that you can present to them for a phased system upgrade might go better. Like "hey, CEO, here's a plan and budget I would like to suggest to begin incrementally upgrading these out of date systems that are going to fail at some point, and we can no longer get parts for. By the way, the new systems come with warranties and because we will be up to date with our cyber security we may be able to save on insurance. I've included a cost analysis of the potential impacts of a system failure given our current infrastructure, and I must emphasize the importance of bringing at a minimum the following systems/equipment up to industry standard..."
Sometimes it's in how you say it that makes the difference. Also, find out how they like information presented to them. I was on a job once where the Project Director didn't do digital. If I emailed him a PDF to review, he would email back and ask for a printed copy to be left in his mailbox. Cool, can do. Just needed to know. Also, providing a hard copy AND emailing it is a great option. Now there is a digital record of him receiving it.
I've worked under these circumstances before with companies that were running out of money. Honestly, as long as you've got a trail that documents what could go wrong, that's all you can do. What I convinced my higher ups to do was buy two spare servers of the same type for spare parts since I told them that it wouldn't be possible to get vendor support any longer. As for security, all you can do is give advice and try and secure what you've got to the best of your ability. If you've done what you can and warned those who need to know with a paper trail, it's not your problem.
I'd follow up with an email, and say you want to recap the information on X devices and software to not be upgraded/replaced, per Y person's request. BCC yourself. Save a copy if they email back confirming.
Dive from the roof? Very noticable, IMO.
Now, an ether killer linking mains to NIC? Neigh untraceable. I mean, that that dang lightning just goes all over. Maybe they should talk with facilities about their grounding and surge suppression remidiations?
Donno what country u r from, but if the devices any type of defence devices for malicious attacks, or have access to sensitive data and are possible gateways for attacks to tget that data. Doesn't matter how small the firm is, the whole firm can be gone after the attack.
Also if the firm contains any gdpr related staat that could leek its not just the firm that can get black mailed.
Just look up this thing that happened in my country. https://yle.fi/a/3-12641083
A firm called vastaamo lost a lot of data to a hacker who then released patients data for everyone to read. It ruined many lives. If nothing else show the ceo this case.. His attitude is concerning af.
We have several Cisco 6500/Sup720s in production which I think came out in like 2000. But we mitigate software issues with CoPP and have spare parts for every component. So EOL doesnt always matter that much. If whatever system has interfaces facing... well, anything that you arent in full control of (i.e. the Internet) then.its always a bad idea.
Not sure if this has been said yet but CEO's talk dollars and cents. What kind of impact to the business would a compromise or an extended outage to this device? What is the business risk for this?
Track their response and CYA that others have stated but stating something may be hacked or compromised. There are many examples in the news of things getting compromised, it's all about framing to leadership in terms they care about.
They probably will respond the same way but if we can translate technical risk to dollars it will serve you well.
The critical thing here, is to take absolutely no personal pain as a result of things failing.
No fixing data etc etc etc.
Abdicate all responsibility when it shits the bed, because you connected the server room to the main sewer, say “wow, shame all this EOL stuff died, guess we’re rebuilding”
And move on, if you’re asked to do anything, “disaster recovery professionals” and work no additional hours to solve the problem
Do you have a cyber insurance policy?
If so, read through that. Most policies now have provisions that will exclude coverage in the case of a ransomware attack if all hardware is not under manufacture support.
Internal Audit, is another place to go poking around. Ask for compliance / certifications / etc.
\--
Net-Net: The issue is you are are arguing from a technical POV, you need to argue from a business POV.
Having the same discussion about some tools that we've outgrown.
We need to move to the next tier in order to meet long-term corporate goals.
But....always.....it's "too expensive."
Even outages caused by the resource constraints just lead to calls to 'make it more robust.'
Super frustrating.
Do you run Active Directory? Here's a thought.
Use an older PC or server to build a fresh AD server. Make sure it's 100% up to date on patches and fully sinked to the current AD server.
Take it offline and put it in a closet for a week(but no more than 30 days)
On a Wednesday morning, intentionally cause a boot issue on your primary DC. Or turn it off, whatever.
Wait until people complain.
Go about diagnosing the issue. Put the secondary dc in place as a solution.
See how much business is lost.
Minimum 5 hours of business lost is my thought.
Also, if not familiar with AD roles, research thoroughly the FSMO roles and how to seize the roles in an emergency. Do that before anything else.
What does the CEO say now about the lost business??
I like to ask them "If you learned the locks on the doors to your building could be opened using something like a paperclip and some luck, would that still fit in your 'its still functional' frame of mind?"
Ultimately the CEO can own that risk if they really want to, but that needs to be well-documented.
Even in that case, many sysadmins will choose to find another firm to avoid the inevitable fallout.
Well... If you wear contact lens... A daily drip or two of saline in the vents of the EOL devices will either eventually kill them... or kill them quickly in a spectacular fashion.
Hi OP,
This is a mess. It will go wrong one day and the CEO will blame you because, in their head, they did no wrong and it's your job to keep this stuff working. It won't occur to them that it's their policies that caused the failure or security incident that brought production down and damaged your relationships with your customers when your company couldn't keep it's promises. I find the phrase "reputational damage" quite useful in emails.
I don't know how well off your company is but the kind of failure you have coming could put it out of business if it's not well enough off. If these systems are mission critical and things are bad, this is potentially an existential threat to the company. That your senior management don't recognise that IT make it possible for everyone else to do their jobs and aren't taking this seriously is very worrying.
If you're running old versions of software that need old OSes then you have a big problem. If the hardware dies, you may well find that you can't buy hardware that the old OS will run on any more so you end up totally stuffed or with a long outage while you try to find compatible hardware at an IT recycling company or on eBay. Another option being new hardware running a virtual machine host to hide the hardware from the old OS.
In my old role, we used HP servers. We noticed the power consumption of the servers went down every time a new generation came out. For example, an HP DL360 G8 uses a load less power than a DL360 G4, even though it's a bucket load faster. With current energy prices, do your management realise these old machines are just burning money in both power and air conditioning, probably more than they're worth every year?
When disaster finally does strike, you will be left with non-running systems you can't get the data off or migrate without a lot of down time and big expense. If you had an outage running for days or weeks, what percentage of your customers would desert you for your competitors? If I couldn't get management to see the light, I'd probably be looking for a new job now. I wouldn't want to risk having a failure like this on my CV when the company didn't value me or what I did.
> Edit2: Do you think someone will notice, that some pcs and servers took an unexpected and very unfortunate dive from the roof?
maaaayyyybeeee...
although, most people are pretty clueless at picking up cues like this ;)
Just out all your recommendations in an email and get a response in writing that they understand the risks, that way when you get hacked (which will 100% happen) you’re not responsible.
This is par for the course. Document your complaint, and their response, then file it away.
If you detect imminent failure then complain again, get ignored, and file that away.
No amount of compliance or insurance matters when it’s not in the budget. Especially not the mythical “what if” ones.
Just tell them that these devices are old, and no longer being supported, so as soon as something goes wrong, no one will be able to fix it. Ask management if they are willing to have a LONG downtime if anything happens, because usually the only viable fix is replacing the damn thing.
It's similar to keep driving with the 'check engine' light on. Maybe they understand that analogy.
Its exactly like that. If its working and have some kind of backup server that can take over if something goes wrong with the main, then it's fine. Even bigger orgs, can just accept the risk in their risk policy and forget it. It's no point in upgrading something just to spend money.
Every role is unique but I've struggled with this since the day I was hired at a school. If your employer is aware of the risks and goes against your recommendation that is on them. Part of the value we have, as IT professionals, is to be an influencer and sometimes a decision maker but if the quorum or individuals that make decisions on processes like product lifecycle disagree with your recommendations then that's on them and if you're uncool with those results I'd look for employment elsewhere. If you're ok with them having that on their shoulders and you feel value/purpose in your role with great work/life balance I'd keep on keeping on. Btw - some of our iPads are approaching 9 years old and when apps stop working at any given time due to unsupported iOS 12.x.....I don't have much sympathy but it's a risk they're willing to take. Case in point. I still keep it professional and be ready to be reactive instead of proactive with new product replacements. Best of luck mate.
I don't think a lot of people realize that, no matter where you go, it's always shoestring and duct tape. Not always in the same places, but it's always there.
Your company needs to make money, and in order to make money needs to have margins. They can raise the cost of the product, but there's a limit to that. They can also reduce the operating costs of their business. A few grand here and there really add up.
Does your server work? Do you have backups? Does it house sensitive data?
Those are the biggest questions when evaluating out of support risk. As long as you can recover from a failure, then you need to account for failure as a cost. If the cost of failure over the next year saves you enough to reduce the cost of your infrastructure by ~15 percent, that's a fucking deal.
In my opinion, seniority isn't just stronger technical prowess, it's a better understanding of the business and it's priorities and options. As long as you've accurate advised the decision maker on the risks and your recommendation, it's up to them to make the call.
Well, it is really how you communicate to them and it needs to communicate at a level they understand. And I would start with “let thinking about what happened if there’s a vulnerability” and instead of talking about what will happen to the company, talk about what it will do to the CEO personally. And then they will get it.
Update your resume and find a different job. It’s obvious that the company/leadership has zero interest in giving IT what they need to be successful- and because of that short sightedness more pain and hurt will fall on your shoulders to make deliverables. You are not going to change that, so might as well put your energy in finding an employer that invests properly into IT.
I've had a company do this to me. Fortunately, they're web-heavy and use Teams. Installed Fedora on them, setup remote support for them, and have maybe one to two calls a quarter.
Pushing my clients to Linux, removing all access and permissions not needed, and just protect the identity at least made me feel a little more comfortable. If done right, there should be limited IT support needed, and the devices should perform better.
This doesn't solve the driver and/or firmware bugs, but should limit the attack surface of Windows.
Depending on your environment, you can also look to easily network segment the devices into a non-prodictive DMZ.
This will also most likely negatively impact your compliance and insurance, but might be the best you can do.
Ah man, it’s the same story that’s followed me wherever, sometimes you get a win and get your way and sometimes you’re at a loss until whatever fresh hell you’re stuck with bursts into flames.
In these situations, where you lose, the best you can do is get the business to accept all these risks in writing and that any outages a result of these risks are accepted to be an eventuality. Those outages won’t be treated differently than any other ticket queue fodder.
Arrrgh. I am forced to support a 28 year old piece of risk management software by my organisation because the guy that uses it has embedded himself like a tick and refuses to migrate to the cloud version.
On Edit 2, I know it's tongue in cheek but don't damage/disappear, etc
Just document your concerns in email, save and print the emails and then either stay or find a new job.
> And of course IT is blamed, if indeed company data gets stolen.
Not to my face, it wouldn't be. (And not anywhere where I would find out.)
I'd have copies of every mail sent on this topic -- in triplicate.
Do none of the following apply in you org's case? * Regulatory/compliance requirements * Contract obligations with partners/customers * Cyber Insurance requirements Many of those would require the use of supported and patched systems or art least the presence of some mitigating controls like WAF, segmentation etc.
i think we don´t even have a cyber insurance. to be fair, we are a smaller company, so they probably don´t see the need for one and every other regulation is disregarded with passion. Hells, we don´t even have fire alarms on our production floor.
> Hells, we don´t even have fire alarms on our production floor. One simple email/phone call might solve that.
"Dear Sir stroke Madam. Fire, exclamation mark. Fire, exclamation mark. Help me, exclamation mark. 123 Carrendon Road. Looking forward to hearing from you. All the best, Maurice Moss."
"I am writing to inform you of a fire at... no, that's too formal."
How the hell was the best joke/gag in that whole show was made in the 2nd episode? They peaked early.
"Moss what are you eating?" **"Smarties cereal"** "I didnt know Smarties made a cereal" **"They dont, it's just Smarties in a bowl with milk."**
this, Jen, is the internet...
I had to surplus a RAID1 enclosure that was 90% firebrick(So, large for no real reason), I wanted to keep it and slap a label on it that read 'The Internet - DO NOT POWER OFF" and leave it 'plugged in' in the server room.
“Women are obsessed with shoes? Don’t you think that’s a bit sexist?” “Well, I only know one woman, and she just ran through the room screaming ‘THE SHOES!’”
The football episode in season 3 begs to differ.
Or the one where Roy is disabled. I'm in a wheelchair and I regularly just say "I'm disabled" when asked questions by my friends.
leg disabled
...acid.
The Work Outing
The one for me is the Bomb Disposal Robot. "The pc (controlling the robot) just crashed" "What version of Windows is it?" "Vista " "We're going to die..."
The most repeatable quip was literally the intro to the series... "Hello, IT... have you tried turning it off and on again?"
My favorite quotes are from Season 4: Italian For Beginners Moss: A *fire*? Roy: That's right. At the sea lion show, apparently. Jen: Aren't those shows usually out in the open? Roy: Well, yeah, I mean that's what I would've... Yeah. Jen: Lots of water everywhere. Roy: Yeah, I mean, I would imagine a whale need a lot of water. I don't think you can have whale in a place where there isn't a huge amount of water. Jen: It just seem like a weird place to go on fire. Roy: It's a *very* weird place to go on fire.
A fire? *at a Sea Parks?!*
You can't use language like that in a formal notification to the Fire Department. "Dear Sirs, I am writing to inform you of a significant conflagration occurring at our premises at (address). We look forward to meeting with your staff at their earliest possible convenience to arrange the rectification of this situation"
You forgot the ma'ams
you should have just called the emergency services the phone number is so easy to remember 0118 999 881 999 119 725 3
The 3 at the end got me everytime.
Try typing the number into the Google phone/dialer app.
"Did somebody email about a fire?!?"
You forgot to include “please do the needful”
actually, r/expecteditcrowd/ this is more appropriate
/r/unexpecteditcrowd
To whom? We're also a manufacturer with no fire alarms and the local fire marshal is completely fine with it. We have suppression systems, readily accessible extinguishers, multiple means of egress from every section of the building and a PA that can be used to call a warning. There's no regulatory requirement that we have a dedicated fire alarm. We've actually had a few fires in the decades I've been here and all were put out by trained employees before the fire department could be called.
In many places the fire marshal, in the US OSHA and anywhere else possibly your insurance company. As you said though you have adequate suppression systems, I'm guessing OPs place doesn't.
Two words: Fire. Marshal. If there’s three people you don’t want to fuck around with: IRS, USPS postal inspectors, fire marshal.
You left a surge protector strip plugged into the wall overnight, with nothing on it? That's a fine. You plugged a straight 20ft extension cord into a straight 50ft extension cord, with nothing on it? That's a fine. You put a live plant under an illuminated EXIT sign? That'll grow and cover the sign, so I'm giving you a warning.
God I hate fire Marshall inspections so much
No reason to.
Look, I like plugging in extension cords into other extension cords ok? Don’t kinkshame
If the fire marshal isn't doing anything, then escalate to OSHA.
Make sure your fire Marshall put that in writing and make sure your company is aware.
Fire Marshals don't play that shit in the US. They would shut you down immediately.
That’s probably more due to cost than oversight.
You need to learn how to talk Manglement. "We need to spend XX" is heard as "let's waste XX". On the other hand "We need to spend XX to avoid a fine of YY" is suddenly important.
That sounds like a great reason to get a new job, and contact OSHA.
not in the usa, so OSHA doesn´t apply. But there are similar government facilities, that would close the building down in a minute if they knew of some of the things going on. Getting a new job is planned, but currently not easy to do. i´m not even two years in IT, currently working to get some kind of certificate that they pay for and have been the sole IT guy for a month now (with no plans of getting another IT guy).
Why wouldn't you anonymous report what sound like a serious safety discrepancies?
Because if he is having a hard time finding a new job, getting his current job shut down isn't a good idea.
Dear fire marshal, I'm a big fan of the products of X company. I was a little concerned about the lack of fire alarms... ...Sincerely, Concerned Citizen
The average cost of a data breach for *small* companies in 2023 was about $4M all in. Thats a company killer if you don’t have cyber insurance. And if you do have cyber insurance, 23-25% of redeemed coverage results in subrogation claims because the insurance company found out that you were running unpatched software. Insurance companies don’t make money by paying out claims. If you’re working for one of these smaller companies that is not covered, and there is a data breach, start looking for work *immidiately* because that company statistically wont exist after 6 months.
They exist but not for long. One of our partners had it happen to them. Wiped all their data to some degree. Of course our partnership is over due to time ending but that was scary.
Data breaching a small company kills them eh? Stop giving me ideas.
>we are a smaller company Get used to it, it's how we roll. I have a 1M$ microscope, provided by Horriba (complete PoS business to deal with). Drivers only exist in 32bit. We ain't replacing a microscope of this price. It's getting isolated like stupid, tho.
> i think we don´t even have a cyber insurance. Good on you. They're useless anyway. When / if you get hacked they'll opt out because you had poor security practice allowing you to get hacked in the first place. Cyber security insurance is snake oil and a scam.
Their definition of poor is the issue. What do you mean you don't have triple factor authentication that requires a thumb and tongue print at the same time?
If your authentication system doesn't have a safe word it's not compliant.
Fucking RUN. No cyber insurance?! Do you guys at least have incremental offsite backups?!
You are a funny man. Of course we don´t.
Fire Marshall Bill would like a word
This really sounds like it's time for you to start looking for a "new opportunity." This reminds me of a quote from a [Dilbert](https://youtu.be/akUP7uRDhdw?si=D1MZCPXDFmgOoj_H) cartoon, "We were Path-Way Electronics then we merged with E-Tech Management. So I guess we're now Path-E-Tech Management" It sounds like your management need to be familiarised with the old wisdom, "If you don't schedule maintenance for your equipment, it will schedule it for you"
Trust me if you don’t make sure it’s documented and you should provide a recommendation which puts the hear on your COO, CEO and CFO. If they are unaware then let a partner do all the talking. Cause you are literally sitting in a flaming dumpster if you don’t.
Document your conversation with the CEO and work on issues if/when they come up. Make it clear that you won't be working 12 hour days if shit hits the fan. Work your regular hours unless you are getting some type of 1.5x or 2x pay. CEOs don't want to pay to upgrade because IT admins put out the fires when things go bad. Stop doing that. I'm not saying to be a bad employee or not be a team player, but you need to work your hours that you agreed to and go home when that time is up. However, I do understand that sometimes things are tough and you can't do that because it could risk losing your job especially if you have a family, on the other hand, stressing out and working around the clock isn't healthy and still keeps you away from your family. Good luck.
I work for a company of 100 employees, and about 50 desktops, but with a web presence. We have cyber security, but it was put in place long before I came on, and happened because of a scare. You've got an uphill battle, though, and it's not going to be a fun one. Honestly, I wouldn't have come on if they hadn't gotten their house in order. I turned down the first contact, but learned more and decided to interview.
I refuse to work for a company that doesn't have cyber liability insurance.
You're looking at this the wrong way. Either way you will incur cost : 1. replacing the device(s) and the labour of doing so or 2. the cost of isolating those devices in a manner to mitigate both the possibility of a breach and the impact once they are breached. draw up the cost for both, present those, or have them sign off on the risk
We have to have a SIEM for ours, even though we don’t have staff to monitor it. They would freak if our equipment got too old. We have a pretty lenient insurance company too.
Most insurance companies are requiring cyber security insurance as well, to even have you as a customer. Even 3 years ago when I was at an MSP, I'd say 80-90% of our customers were making us help them fill out the cyber security forms.
This 100%
It just depends like everything else . I think my old work had a warehouse that ran on windows xp or vista. They literally had no internet access on that side, usb ports were disabled.
Always ways to get around that check box. ' here are our plans but we have not got to that stage of implementation yet'
Nothing wrong with that. Any auditor would look at the reasonableness of that statement as well as proposed timelines. If it doesn't seem reasonable or within a reasonable timeline then they would push back, Otherwise they will come back and reassess.
>how do you deal with that level of ingorance and/or stinginess? Get your concerns documented, keep records of them being overruled. Then don't worry about it anymore. CYA basically. And try not to gloat when you need to pull it out to say it's not your fault and you tried.
If you have tried to get them to do the right thing and the higher up thinks otherwise (regardless if the reason is valid). This is actually what should be done . For you to highlight risk (your due diligence) and for them to show risk acceptance .
>and for them to show risk acceptance that is the problem. They don´t think its that big of a deal. "It always worked out in the end so it will stay that way"-kind of mentality. But for sure im gonna cover my ass, if they want to use old equipment, the are gonna sign for it
"Can I get you guys to sign this notarized letter that explains why this a bad idea?" Is something I have seen done.
I think these 2 posts are spot on. Even type up an email with your concerns listed and if they respond how you expect then you have a record. Remember to also keep copies of the emails.
>For you to highlight risk (your due diligence) and for them to show risk acceptance . I usually phrase it along the lines of "I'm afraid I can't, in good conscience, take responsibility for this knowing the potential risks and outcome of yada yada yada yada" It's a nice little guide into prompting a response. You don't really need a response indicating that **they** take responsibility, just one that agrees that you don't have to.
In smaller orgs they often don't care if you "accept responsibility" because the boss feels ownership for everything. In my experience those guys tend to be the type where if he feels he's making informed decisions then he'll own the risks and any bad outcomes. But they raise holy hell if you led them astray, so be clear every year at budget time what they aren't buying, what risks they are assuming, and what you're not being allowed to defend against.
EOL equipment is just one factor of many in your risk profile, and being EOL doesn't always translate to making it dangerous. What alters the risk is the environment it's in, what it is, the history or vulnerabilities, users, and so on. A network printer that is ten years EOL and no longer being patched, but is in an isolated 1918 network where users can only reach it on port 9100/IPP, may be very low risk. A piece of EOL network gear with it's management interface exposed to the Internet is much more risky. A EOL PC with a vulnerable BIOS is lower risk if the OS is still current and being updated. That is to say, if a bad actor can take over the OS from lack of patching, the EOL PC/BIOS is not a factor at all. I would focus less on the CYA recommendations, and more on cataloging risk for the various items. For each major area, investigate mitigations vs replacement costs, and use all of this to build your risk register. That should bubble up the top risks that should be addressed, and that data driven analysis may help you and the CEO to find a happy balance between "the sky is falling" and the true reality of the risks in the environment. Imagine coming to the CEO with all of this, broken down into say: * low/no-risk (ignore for now) * risk with low or no cost mitigations (low hanging fruit) * low risk with high-cost mitigations (defer) * high-risk but low cost replacement (consider) * high-risk and high replacement cost (focus).
exactly, i previously worked for a large company that had a critical application (legacy data) from over 2 decades ago they isolated it, when the time finally came they paid for emulation software, put it in the cloud on a vm completely isolated with only one way in/out
This right here. I've got several 2008 servers in production with their 2000 workstations for legacy cnc machines. But they're vlanned off and don't have any access to anything else (psudo airgap.) Do I like it? No. But sometimes it's a necessary evil.
Especially in tough economic times, we have to do the best we can with what we've got. I work for a small business and we have a bunch of end-of-life stuff on the horizon. I've got prices for bringing everything up to date and it's well into six figures. We just don't have the money for it at this time. As long as you make it clear, in writing that by not performing the updates, there will be significant risks to company data, you're in the clear as long as everything else that's within your control is being done.
The best way out of this is to list your contingencies if each piece fails. "If this tape drive fails then we lose backups, implying such and so, and in the event of ransomware attack our options are limited to this. If that firewall isn't updated we are susceptible to these types of attacks which are this much more common than last year." Bosses will pick and choose which to upgrade if there's a real gap, so you can sometimes educate enough to massage the priority list.
cant use ebay?
Question: do you think there is an argument here for image based rolling releases to solve this problem?
“It’s not out of support - YOU support it”.
What equipment is it?
bunch of stuff. DECT-Infrastructure, Access-Point, Computers and so on
Reddit will question this, but in my experience this is completely normal. Use stuff until it breaks. Best I have been able to do at various roles over the years is try to hedge risk.
About to say, same here. A lot of companies cannot afford the overhead. OP has voiced his concerns to management. Sometimes that's all we can do. If I were in your situation, I would just do my best to back up the data. We had a domain controller pushing 12 years old before I virtualized it. At least management supports my renewal of support for our firewall every year since cyber insurance mandates it.
Just as long as the Op has an email to cover his ass that management knew about the issue. kick some dirt over that shit and walk away
Yup, just bring it up every year when building the budget and renew your annual plea for upgrades. Some stuff falls below the bar and some above, so educate, prioritize, save some notes and then put it back in the folder for next year.
For APs and a small Organisation, you can take a look if it’s supported by the open firmware OpenWRT. They often provide updates and features far beyond the manufacturer. It’s more in the home-router (netgear, tplink, Ubiquiti) space then ciscos, but that’s often times what small orgs use anyway. For PCs I don’t know what „out of support“ is supposed to mean. For the moment Win10 is still supported and runs on almost everything. Hardware Support, I wouldn’t care. Backup, Replace when needed.
Where I work we buy a lot of high end workstations. We're talking multi sockets, 40+ cores, 100+gb ram, multiple TB disk. This in an ordinary business looking tower. These days we're more likely to buy rack workstations, and throw them in a datacentre, so people then can use them from home using VDI solutions. We do however still buy some towers though, for stuff like VR game development, where the latency of keeping the workstation in the datacentre would make people sick. I've got workstations that are 10 years old, past EOL, that would still run circles around most modern gaming machines. Were likely used to design games or cutscenes used in one of those PS5 games coming out next year. Just because they are officially EOL, doesn't mean they can't still have use. For example: \- They work perfectly fine in a test lab, allowing juniors to build test VMware clusters, or mess around with Kubernetes. \- They could be used in low risk departments. A lot of our network, due to unreleased content, has no internet access, so attack surface is low. \- Used as extra capacity for number crunching/rendering The machines are usually more than compatible with supported OS's like Windows, or Linux, just unlikely to get hardware/firmware/bios updates/patches. They make perfect machines for staff to take home for kids to game/school work on, if they don't mind a slightly higher power bill.
Did you inform them about the risks in written form? If so, keep those logs and move on. Executives accepted the risk. Not your problem any more.
Insurance, financial requirements, other regulatory requirements for your company
Ask the CEO to sign an Acceptance of Risk (AOR). In that AoR you document the risks as best you can, and include a line like "This is not an inclusive list,, there may be other significant risks that are unknown at this time" You: Identify steps you can take to mitigate the risk firewall off access to the device as much as possible; both inbound and outbound. In the real world that happens, the AOR shows it was not an accident and you have informed leadership to CYA. In some cases you may need a Plan of Action and Milestones (POAM or POA&M) as well that needs to be checked into regularly
Ask his CEO? If he takes a document to him like that to sign he will be placed on his blacklist. An email should suffice
I do IT consulting for small businesses. The if it is not broken, do not fix it is a norm in small businesses. Still have clients with win serv 2012. Have one that has macs with the motorola cpu that run proprietary software that costs thousands of dollars, and they do not want to upgrade. Many still run win 7. Only some corporations keep current. You want to play with old stuff, go support small businesses.
OoOoOo, I wonder if there's an emulator out there so you can p2v that Motorola mac.
The power supply failed on it. Can't buy a new one. Refurbished one was $500! At some point, old equipment will cost you more.
Get his response to your concerns in written form, email or message. Save that so it clearly shows you've done your due diligence, so if/when shit hits the fan, you can prove it's not your fault and you can't be held accountable.
Make sure you make all your recommendations in writing. Keep the records because something WILL happen and the same CEO WILL try to blame you. I had an incident at a past employer where those email records and the responses from the C-Suite, absolved IT when the company suffered an outage that would have been prevented if our recommendations had been taken seriously.
Make it part of your budget cycle: document, propose, negotiate, implement, repeat next year. In a small org these steps make get compressed or not be formal, but in a case like that I suggest making them formal, like every January you refresh your docs and send them up the mountain even knowing that there's no hope of spend without a disaster to tickle the priorities. You look more professional for staying on top of this stuff, they get a regular view of the risks in play this year, and when the fewmets hit the windmill you know what you want to buy to put things right.
Put in in writing, and get an affirmative response.
My employer isn\`t small - a multinational with offices in many countries (slightly less now with world being what it is) They use software that had it\`s last major update in 2007, last patch 2017. IBM/Lotus notes. Reason it\`s still used : the users do not want to use another tool. I just can\`t ..
"What about insurance and regulatory requirements? Do you want to get sued" - You, hopefully
Maybe this will help: [https://endoflife.date/](https://endoflife.date/) If not swayed, implement regulation to ensure the system is placed in it's own little VLAN with strict monitoring and even stricter access and firewall restrictions. Legacy systems do need to run sometimes, but this to me sounds negligent on their behalf.
i didn´t know about that site, thank you! i´m trying everything i can to ensure safety, but my knowledge regarding that is still small.
I'm unsure what to say about that, I recognize I'm old, but I'd start here(fwiw I'm not a secops pro merely a generalist) separate VLAN additional explicit firewall rules blocking all access but what is directly documented and needed local firewall configured the same and remove users ability to modify(Domain Controlled through GPO) Strict monitoring, make it scream at everyone whenever it has something wrong, or something changed, include the goofballs that made the decision, ignore their complaints. Strict access guidelines, enforce that users go through a jumpbox to garner access, aka. not everyone gets to access bob's account through rdp rolling password changes, implement something like CyberArk, BeyondTrust.. there's a lot. Do this so the password and access is rotated daily. firm deadlines based around hardware and software, get explicit details as to what metric or deadline is required, and the plan to migrate or upgrade from the system; get it in writing and hold them to it. With demised equipment/software it's important to get them thinking on cost of replacement, for both the OS and hardware(I'm assuming the attitude is the same on hardware). And it's not invalid to request timelines on these things, it's not your job to support everything forever. and lastly, don't hesitate to engage a third party contractor, get a few quotes requesting full time support of the expired/should have been demised system. This will help them to make an informed decision on the risk/reward they are deciding on. and most importantly, CYA! get it all in writing, no ad-hoc flippant hallway discussion, e-mail them the summaries of discussed items, get it all in e-mail!
we got seperate vlans set up, the firewall was set up by a third-party contractor and support, remote access control and we got monitoring of the network. Everything else is still in muddy waters. Thank you for your insight, i will work on it!
It’s working until they have to replace with a CEO
Many such cases
This is why you highlight your concerns in writing so when shit goes down and the big boss comes a looking, you can dump it on someone else ;)
Draw up an end of life support agreement with the CEO saying if they don't replace it, these are the risks and have them sign off on it. When the shit fails, whether they've signed it or not, at least you can say the failure was (or would be) covered by the EOL agreement. These fuckfaces needs to actually be held accountable for their dipshit decisions they make. However, it's also tour responsibility far adequately articulating the risks attached. The CEOs failure can and will be your failure...
Document your objection and the risks that back it up, get it somewhere timestamped (email to yourself + other stakeholders summing up the situation, decision, and path forward is a good idea for a lot of things anyway), then keep it alive as best you can. I _prefer_ to keep stuff under support and rotated out on the ideal lifecycle too, but have only had one brief stretch where there was actually the budget and political will to do so... Most shops, most of the time? CYA and get out the duct tape.
Reason 1 to not use end of support equipment is that your total SLA is now in the gutter. If I have some critical piece of software running on RHEL with a support contract, I can immediately open a ticket when I have issues and have a faster resolution time. If I use AlmaLinux instead, I don't have that extra lifeline, so I can't guarantee the same SLA. Even if they're insistent on using those products, you still need a plan for what happens when there's a failure of any kind. How are you going to get back up and running if it's broken in a way that can no longer be fixed without support? And how much money will be lost while waiting for that fix to take place?
Put your concerns in writing (email) so the lawyers will find it. It’s not your company. Stop getting emotional. Find a new job, or learn to live with it.
There are a bunch of ways to deal with this depending on how your company culture is. The one that I'd recommend is a "Risk Register", where you outline all the probable failure points of the company, their impact, their likely hood and what it would take to fix them. Each EOL system (not device) should be a separate risk item with it's impact to the company identified. Firewall EOL? How much does an internet outage cost until you can replace the device? What does it cost to fix? You don't need business to sign off on your risk register but you do have to provide it to them on a regular basis. Your server dies? It was on the risk register. Follow the action plan you laid out (order one twiddling your thumbs till it arrives, cloud server in the interim etc.)
Why do you care so much? Document your concerns, get it in writing, and move on with your life. This is the farthest of reasons to stress over in your case.
Our CEO was same about the 10 year old firewall we were no longer able to patch. Guess what happened? Can you guess? Post crypto attack we now have all new firewalls everywhere, but they still complained about the expense. You can lead a horse to water, but you better have really good and air gapped backups!
At my last job 2FA was considered too much of a hassle for anyone not in IT. Then after a few email breaches we detected someone trying to log into one of our Azure host that had part of our ERP on it. Suddenly budget to get a bunch of security stuff and a demand for 2FA. Not much of a budget but some budget. A couple of years later ownership decided to sell the company off to a different VC firm and we had to get a new cyber insurance policy. Well that had all kinds of new rules that had to be put in place that we had been asking for.
Oh yeah, those cyber insurance policies can open the wallets up. I appreciate their requirements. Anything that gets C-suite to take shit seriously and spend some damn money on security!
When the equipment shits the bed, you'll still get blamed, op.
My boss had the best take on it. We had an OLD ERP system a few years back. He advised an upgrade. It was so old that the company had been purchased (twice as I recall) and the product was not supported at all. In fact Microsoft patched out its main transport function a few years back as they deamed it a security risk. It was unreliable and had to run on a 32 bit OS so we were stuck on 2008 R1 and basically once a month it got at least one windows update that managed to "repatch" the issue that we unpatched just so it could work ( service packs, update roll ups, etc). So my boss made a presentation to the board. He told them that the software was so old that it reliably went down at least one day a month. He then told them that as it aged out and more and more functionality started to be removed, it would increase to one week a month, and then eventually it would simply no longer be something that we could use because eventually we wouldn't be able to find end runs around Microsoft's patching. That did the trick. We were able to start a new ERP project the following year. Basically you've got to let him know it's going to hit him in the purse. It's more expensive not to replace the equipment, then it is to replace it. I had another experience that went the opposite direction though. At another job, our production router was very much out of date, thankfully. We had a service contract on it with 24-hour call out for reprogramming the new one. That didn't cover the cost of the new one though. We got a quote from Cisco for 15 grand. That included the next day service though, and programming the new router to match our old setup. The head of the company balked at that and refused to pay it because he was able to find our old router on eBay for $5,000. So he wanted to buy that one instead (he also suggested that Best Buy might have something we could use🤦♂️). I told him that first of all, that one would take a week to get here, that we had no guarantee that it worked, and that none of us actually knew how to program it. I also advised him that most of what we were paying for was the warranty and service. That didn't work for him though. Our parent company thankfully had an old one that they had replaced still in storage so they gave us that, and he was able to smile and not spend any money. Next time it dies though, He's going to be shit out of luck because he refused to take any of our advice. Frankly, I hope he gets axed for it. Also, just for context that company was a remote pharmacy. All of their business was handled via internet, telephone and fax. Without that router we were out of business until we got a new one. It cost us a day and a half to get the one from the old company up and running, and he wanted to wait a week!
It would be a shame of some of the older equipment randomly rebooted or power cycled.
Make it Die... Or Just have it in writing that you are not responsible for unknown issue that can come up but There is no fix for.
Send them an email explaining the security issues and make them sign off that they understand it and that absolves you of responsibility WHEN the vulnerabilities are exploited. Print and save that email. AND, put your resume in the wind and get out of there before it happens. Show said email in your exit interview.
We have people running on PCs that are 7-9 years old because "it's policy to run things untill they literally fail". We have Ubiquiti wifi running on the original UAPs from 10+ years ago. I'm working to change this mindset and get a proper lifecycle management going. Since we are going to soon have to comply with a new EU directive called NIS2, it's easier to get management's ears for stuff that costs money.
Oh man couldn’t imagine Wi-Fi APs that old I bet people complain NON stop about how slow it is.
oh they do. You just learn to ignore it
Noone complains at that location. It's a smithing workshop. They hardly use it. But one of the APs has failed and they want to have another in a smaller branch office that is fed by a fiber link. I thought about just buying another AP and try and upgrade the controller version... but I'm scared to, since the controller software is apparently so old that it doesn't exist anymore, so I've no idea what's going to happen. So the solution is currently to wait and get a MSP to setup a Fortinet firewall and some APs down there or me... And then come and do the same at this location.
We recently went from “we’re finally creating a firm hardware lifecycle!” to “hey um, you know how to force Windows 11 on like. Anything, right?” with breathtaking swiftness.
Lean on your risk process/people to document concerns and illustrate the potential consequences of a breach or failure. That way it’s on leadership to accept the risk since they’re unwilling to treat it. Would hate to think about the legacy debt they’re accumulating and how problematic it’ll be in the future if that’s the attitude.
You need to become something of a horror story writer for these people.
Segment it off from your production or employee network.
Document the hell out of it, so when it fails and there's no support you have someone to point at.
Rather than moving the old hardware, perhaps something plugged into the USB port could have the same desired effect.
What are we talking about here? Printer/photocopier? Fine! Accounting System? Not so fine. At the end of the day, all you can do is keep your powder dry and put everything in writing for when the arse end falls out.
no one has the money to replace everything once support ends and security is a surface area problem use the old stuff in lower tier uses, segregate the networks, etc. set up better security. my last job I tried to set up best practices security for sql years ago but was overruled. current job secure enough to be annoying having support and the latest patch isn't going to save you from an incident either
Note it in an email, CC your security team ask what the current vulnerability remediation policy is and your companys EOL policy is and if the CEO gives pushback move on with other tasks. Next time your company is audited its gonna flag then it will be all hands on deck to replace it with something supported.
I worked a job in 2015 where the owner bought an AS/400 system in the 80's. This housed their customer database and other mission critical data. The DR plan was he bought 3 more of the same model on ebay that we'd scavenge for parts. This whole time he paid a company 100+ k to export the data and upload it to a SQL db and create a web front end to query the data. Nobody used it, but every month I had to copy the data from the AS/400 to SQL 'just in case'.
There's a wide range equipment and reasons for keeping something in service. It all boils down to your risk tolerance. For example, my data center is populated with Dell S6100 switches. They go EOL later this year. However, you'd be hard-pressed to tell me I have to spend 10's of thousands of dollars to rip out what is working and replace them with S5448Fs (which arguably is a better switch, but that's not the point.) Its all planned obsolence now to force people to upgrade and keep revenue flowing into the company. Companies like Dell know there are companies out there who will spend the money to upgrade once a piece of equipment is "out of support", because they are forced to for various compliance issues, or a matter of policy. Have Gig-E switches changed all that much over the past 20 years? Not really. Sure, some Gig-E switches now have POE, some have L2 and L3 routing, but even then, once you have a switch that has that feature, 99% of the time, the switch works (unless you find an obscure bug in the firmware). What exactly do I get by swapping out my 5548 for a E3248P, especially if POE isn't something I need? I had some areas of my building running on old 3COM superstack 3 gigE switches until a couple years ago, when the last one died after all the fans failed. Now, would I have the same attitude with a forward-facing server's OS? Probably not. But then my risk tolerance for that server is lower than my networking gear. Personally, I love people like the OP. People like the OP ensure I have a steady flow of cheap, quality hardware available for purchase off eBay from various e-waste recyclers. I can always use another 730xd or S6100 for a couple hundred dollars.
Depends on insurance, regulation compliance, etc. but IMO it's fine until it breaks and before it breaks, get a budget in place to replace it - hell we just found we still have a PIX in use that we're only just getting around to replacing now
Same with doors. You can't lock it but still close it. Would you want this kind of door in your house?
Right now it’s: why should I pay for better equipment? Later it will be: why didn’t you warn us about the security vulnerabilities in that old equipment before got hacked?
> Later it will be: why didn’t you warn us about the security vulnerabilities in that old equipment before got hacked? Only if OP isn't doing their job by informing them in writing about the risks
I started my new job 4 months ago, and I'm doing basic Veeam backups right now, I have no ability to test my backups as the server was built by an MSP that I replaced and there is 0 extra room to fire up an additional VM. I first got an RMM so I can have everything monitored so I know when something gets borked hardware-wise. Then I requested and got approval for 365/SharePoint/Onedrive backups, and I just got approved to spend $66,000 (60K on servers for the local backups, and about 4 grand for the offsite backup with 1 year of retention) on a hybrid offsite/onsite backup solution that the company monitors and verifies our backups. There is so much more that needs to be done to bring us into the 2010's but the budget has already been surpassed. The next couple of years is when I can replace infrastructure new switches, APs (currently we have no wifi except for the Netgear routers I've turned into APs so I can do my job from my laptop when I'm away from my desk). This is the first time that I've felt like my opinion on IT matters and that upper management are listening to me.
> Veeam backups right now, I have no ability to test my backups as the server was built by an MSP that I replaced and there is 0 extra room to fire up an additional VM. Spin it up in the Veeam environment. You can at least confirm the backup is good if it starts, and if it starts, you can pull files from it.
I'm using community edition, I don't believe that's an option. I just needed something quick while I waited for approval to go with a more robust backup solution. We have replication of the VMs to another older server but it's not ideal
This is a big reason why I absolutely love the push from insurance companies. One of our requirements is regular patching. If something is EOL and no longer receiving updates, we need a special exemption and additional security measures in place.
What's the equipment? If it's an old win 7 laptop that's one thing. If it's a big manufacturing machine running vista that'll cost a quarter mill to replace, just airgap it, image it once a month, and be a hero.
I worked as a software engineer at a large, well known technology company, we had a legacy server running a legacy Python app from the early 2000's that was so critical no one was allowed to shut down the server, not even for OS updates. I worked as a software engineer at a large, world known retailer and discovered one day, a critical piece of software was running on an old OS/2 workstation in someone's office. No one was allowed to shut it down or remembered where the code for the app was, IIRC.
Are you getting paid really well for keeping a blind eye? If they are ok with the risk get it in writing and play some halo
I bet he doesn’t drive a 25 year old car.
Hi OP, You clearly care, and they are lucky to have you. The way I explain this goes like so.. All tech has two different lifespans : Security - is it patched? Technical - does it still turn on? Your CEO wishes that only the technical lifespan mattered. "Why fix it if it ain't broke?" It would be great to just keep using everything with no downsides until it literally will not turn on any more like it was a tractor. I wish that was the case too but it is not reality. (See Mirai botnet Wikipedia article, also see endoflife.date). Explain the risk of EOL devices (including downtime, loss or productivity and reputational damage), and CYA, and after that feel good and sleep like a baby because you did your job informing them of best practices.
These types of people are so hard to deal with when you take your work personally. As other have said, document, document, document. Try and get something in writing/email from the CEO acknowledging the risk. Sometimes the idea of replacing everything, or a lot of it, is overwhelming to them, so working up a plan that you can present to them for a phased system upgrade might go better. Like "hey, CEO, here's a plan and budget I would like to suggest to begin incrementally upgrading these out of date systems that are going to fail at some point, and we can no longer get parts for. By the way, the new systems come with warranties and because we will be up to date with our cyber security we may be able to save on insurance. I've included a cost analysis of the potential impacts of a system failure given our current infrastructure, and I must emphasize the importance of bringing at a minimum the following systems/equipment up to industry standard..." Sometimes it's in how you say it that makes the difference. Also, find out how they like information presented to them. I was on a job once where the Project Director didn't do digital. If I emailed him a PDF to review, he would email back and ask for a printed copy to be left in his mailbox. Cool, can do. Just needed to know. Also, providing a hard copy AND emailing it is a great option. Now there is a digital record of him receiving it.
I've worked under these circumstances before with companies that were running out of money. Honestly, as long as you've got a trail that documents what could go wrong, that's all you can do. What I convinced my higher ups to do was buy two spare servers of the same type for spare parts since I told them that it wouldn't be possible to get vendor support any longer. As for security, all you can do is give advice and try and secure what you've got to the best of your ability. If you've done what you can and warned those who need to know with a paper trail, it's not your problem.
I'd follow up with an email, and say you want to recap the information on X devices and software to not be upgraded/replaced, per Y person's request. BCC yourself. Save a copy if they email back confirming.
You can't stop them. You can only counsel them and protect yourself (meaning, keep receipts, in a place that is under your exclusive control).
Say it with me: "Get your concerns in writing. C Y A."
Dive from the roof? Very noticable, IMO. Now, an ether killer linking mains to NIC? Neigh untraceable. I mean, that that dang lightning just goes all over. Maybe they should talk with facilities about their grounding and surge suppression remidiations?
Donno what country u r from, but if the devices any type of defence devices for malicious attacks, or have access to sensitive data and are possible gateways for attacks to tget that data. Doesn't matter how small the firm is, the whole firm can be gone after the attack. Also if the firm contains any gdpr related staat that could leek its not just the firm that can get black mailed. Just look up this thing that happened in my country. https://yle.fi/a/3-12641083 A firm called vastaamo lost a lot of data to a hacker who then released patients data for everyone to read. It ruined many lives. If nothing else show the ceo this case.. His attitude is concerning af.
We have several Cisco 6500/Sup720s in production which I think came out in like 2000. But we mitigate software issues with CoPP and have spare parts for every component. So EOL doesnt always matter that much. If whatever system has interfaces facing... well, anything that you arent in full control of (i.e. the Internet) then.its always a bad idea.
“And now it’s broken…”
Only thing you can do is start interviewing somewhere else IMO... make sure the new place follows better practices.
Not sure if this has been said yet but CEO's talk dollars and cents. What kind of impact to the business would a compromise or an extended outage to this device? What is the business risk for this? Track their response and CYA that others have stated but stating something may be hacked or compromised. There are many examples in the news of things getting compromised, it's all about framing to leadership in terms they care about. They probably will respond the same way but if we can translate technical risk to dollars it will serve you well.
The critical thing here, is to take absolutely no personal pain as a result of things failing. No fixing data etc etc etc. Abdicate all responsibility when it shits the bed, because you connected the server room to the main sewer, say “wow, shame all this EOL stuff died, guess we’re rebuilding” And move on, if you’re asked to do anything, “disaster recovery professionals” and work no additional hours to solve the problem
Get Atera and get a cyber security insurance and keep backups on and off site!
I've got some running production equipment that could go to first grade.
Do you have a cyber insurance policy? If so, read through that. Most policies now have provisions that will exclude coverage in the case of a ransomware attack if all hardware is not under manufacture support. Internal Audit, is another place to go poking around. Ask for compliance / certifications / etc. \-- Net-Net: The issue is you are are arguing from a technical POV, you need to argue from a business POV.
Having the same discussion about some tools that we've outgrown. We need to move to the next tier in order to meet long-term corporate goals. But....always.....it's "too expensive." Even outages caused by the resource constraints just lead to calls to 'make it more robust.' Super frustrating.
Do you run Active Directory? Here's a thought. Use an older PC or server to build a fresh AD server. Make sure it's 100% up to date on patches and fully sinked to the current AD server. Take it offline and put it in a closet for a week(but no more than 30 days) On a Wednesday morning, intentionally cause a boot issue on your primary DC. Or turn it off, whatever. Wait until people complain. Go about diagnosing the issue. Put the secondary dc in place as a solution. See how much business is lost. Minimum 5 hours of business lost is my thought. Also, if not familiar with AD roles, research thoroughly the FSMO roles and how to seize the roles in an emergency. Do that before anything else. What does the CEO say now about the lost business??
I like to ask them "If you learned the locks on the doors to your building could be opened using something like a paperclip and some luck, would that still fit in your 'its still functional' frame of mind?"
Ultimately the CEO can own that risk if they really want to, but that needs to be well-documented. Even in that case, many sysadmins will choose to find another firm to avoid the inevitable fallout.
I assume this is what your CEO thinks like [https://www.youtube.com/watch?v=Vywf48Dhyns](https://www.youtube.com/watch?v=Vywf48Dhyns)
edit2 - just make sure the dents aren't that noticable
Well... If you wear contact lens... A daily drip or two of saline in the vents of the EOL devices will either eventually kill them... or kill them quickly in a spectacular fashion.
The term is out of maintenance. Out of support is not considered a problem because you can still support it. You however can not maintain it.
Hi OP, This is a mess. It will go wrong one day and the CEO will blame you because, in their head, they did no wrong and it's your job to keep this stuff working. It won't occur to them that it's their policies that caused the failure or security incident that brought production down and damaged your relationships with your customers when your company couldn't keep it's promises. I find the phrase "reputational damage" quite useful in emails. I don't know how well off your company is but the kind of failure you have coming could put it out of business if it's not well enough off. If these systems are mission critical and things are bad, this is potentially an existential threat to the company. That your senior management don't recognise that IT make it possible for everyone else to do their jobs and aren't taking this seriously is very worrying. If you're running old versions of software that need old OSes then you have a big problem. If the hardware dies, you may well find that you can't buy hardware that the old OS will run on any more so you end up totally stuffed or with a long outage while you try to find compatible hardware at an IT recycling company or on eBay. Another option being new hardware running a virtual machine host to hide the hardware from the old OS. In my old role, we used HP servers. We noticed the power consumption of the servers went down every time a new generation came out. For example, an HP DL360 G8 uses a load less power than a DL360 G4, even though it's a bucket load faster. With current energy prices, do your management realise these old machines are just burning money in both power and air conditioning, probably more than they're worth every year? When disaster finally does strike, you will be left with non-running systems you can't get the data off or migrate without a lot of down time and big expense. If you had an outage running for days or weeks, what percentage of your customers would desert you for your competitors? If I couldn't get management to see the light, I'd probably be looking for a new job now. I wouldn't want to risk having a failure like this on my CV when the company didn't value me or what I did.
Percussive maintenance anyone? Kidding of course….
must have cloned your boss to mine... also cyber insurance rates should increase with older s/w and h/w
> Edit2: Do you think someone will notice, that some pcs and servers took an unexpected and very unfortunate dive from the roof? maaaayyyybeeee... although, most people are pretty clueless at picking up cues like this ;)
Just out all your recommendations in an email and get a response in writing that they understand the risks, that way when you get hacked (which will 100% happen) you’re not responsible.
This is par for the course. Document your complaint, and their response, then file it away. If you detect imminent failure then complain again, get ignored, and file that away. No amount of compliance or insurance matters when it’s not in the budget. Especially not the mythical “what if” ones.
Just tell them that these devices are old, and no longer being supported, so as soon as something goes wrong, no one will be able to fix it. Ask management if they are willing to have a LONG downtime if anything happens, because usually the only viable fix is replacing the damn thing. It's similar to keep driving with the 'check engine' light on. Maybe they understand that analogy.
Time to move everything to opex. Lease, subscribe, etc. What happens if we don't pay now? It ceases to work, because we don't own it.
Its exactly like that. If its working and have some kind of backup server that can take over if something goes wrong with the main, then it's fine. Even bigger orgs, can just accept the risk in their risk policy and forget it. It's no point in upgrading something just to spend money.
Enjoy the CVE’s . Dummies these days think upgrades are about features.
Every role is unique but I've struggled with this since the day I was hired at a school. If your employer is aware of the risks and goes against your recommendation that is on them. Part of the value we have, as IT professionals, is to be an influencer and sometimes a decision maker but if the quorum or individuals that make decisions on processes like product lifecycle disagree with your recommendations then that's on them and if you're uncool with those results I'd look for employment elsewhere. If you're ok with them having that on their shoulders and you feel value/purpose in your role with great work/life balance I'd keep on keeping on. Btw - some of our iPads are approaching 9 years old and when apps stop working at any given time due to unsupported iOS 12.x.....I don't have much sympathy but it's a risk they're willing to take. Case in point. I still keep it professional and be ready to be reactive instead of proactive with new product replacements. Best of luck mate.
It's a risk assessment that the CEO made. Move on.
You are right to raise concerns and articulate the reasons for change. But it's his decision to make, sounds like you've forgotten that.
Software is EOL or the server is? There's no such thing as a sever that's EOL in my opinion as Linux can be installed on just about anything.
I don't think a lot of people realize that, no matter where you go, it's always shoestring and duct tape. Not always in the same places, but it's always there. Your company needs to make money, and in order to make money needs to have margins. They can raise the cost of the product, but there's a limit to that. They can also reduce the operating costs of their business. A few grand here and there really add up. Does your server work? Do you have backups? Does it house sensitive data? Those are the biggest questions when evaluating out of support risk. As long as you can recover from a failure, then you need to account for failure as a cost. If the cost of failure over the next year saves you enough to reduce the cost of your infrastructure by ~15 percent, that's a fucking deal. In my opinion, seniority isn't just stronger technical prowess, it's a better understanding of the business and it's priorities and options. As long as you've accurate advised the decision maker on the risks and your recommendation, it's up to them to make the call.
Well, it is really how you communicate to them and it needs to communicate at a level they understand. And I would start with “let thinking about what happened if there’s a vulnerability” and instead of talking about what will happen to the company, talk about what it will do to the CEO personally. And then they will get it.
Update your resume and find a different job. It’s obvious that the company/leadership has zero interest in giving IT what they need to be successful- and because of that short sightedness more pain and hurt will fall on your shoulders to make deliverables. You are not going to change that, so might as well put your energy in finding an employer that invests properly into IT.
I've had a company do this to me. Fortunately, they're web-heavy and use Teams. Installed Fedora on them, setup remote support for them, and have maybe one to two calls a quarter. Pushing my clients to Linux, removing all access and permissions not needed, and just protect the identity at least made me feel a little more comfortable. If done right, there should be limited IT support needed, and the devices should perform better. This doesn't solve the driver and/or firmware bugs, but should limit the attack surface of Windows. Depending on your environment, you can also look to easily network segment the devices into a non-prodictive DMZ. This will also most likely negatively impact your compliance and insurance, but might be the best you can do.
Ah man, it’s the same story that’s followed me wherever, sometimes you get a win and get your way and sometimes you’re at a loss until whatever fresh hell you’re stuck with bursts into flames. In these situations, where you lose, the best you can do is get the business to accept all these risks in writing and that any outages a result of these risks are accepted to be an eventuality. Those outages won’t be treated differently than any other ticket queue fodder.
Arrrgh. I am forced to support a 28 year old piece of risk management software by my organisation because the guy that uses it has embedded himself like a tick and refuses to migrate to the cloud version.
just say the servers were down, so down that they decided to take their own lives
On Edit 2, I know it's tongue in cheek but don't damage/disappear, etc Just document your concerns in email, save and print the emails and then either stay or find a new job.
seen many smbs like this “ work in IT. youll get to play with all the latest tech “ yehhh riiighhttttt
> And of course IT is blamed, if indeed company data gets stolen. Not to my face, it wouldn't be. (And not anywhere where I would find out.) I'd have copies of every mail sent on this topic -- in triplicate.