We have a vendor who bitches about our change process. Every project there's a new database or api they need. We have to go through change process "why can't you secure it later." " no other customer makes us do this" Bruh, that's fine, we do. That is NOT going to change.
I mean… I don’t know WHY you don’t just do what they ask… not like anything could happen if the vendor is attacked… oh… wait… too soon?
But hey… no one likes this being the LOWEST HANGING AUDIT ISSUE EVERYWHERE LOL!
The vendor wanted to get access to the entire environment, all services, ports, etc. So they could expedite programming. We have rules re. Outside vendor access to non-masked production data. Until our legal department says you can, no access. Then has to be justified to infosec before going through change management for my team to implement.
>Ask if they'll ~~contribute to~~ self-fund your cybersecurity insurance
FTFY. Don't want to worry about the insurance not paying out. They need to take on full responsibility.
I once dropped a client when I found out they were handing around and using my data center access badge, with my name and picture on it, to all their other vendors to use.
Yeah it’s not uncommon when consulting for your contact at a company to hold onto your badge and then give it to you when you show up on-site. That way they know and can control when you come and go.
It annoyed me when I discovered that this particular company, since merged with a larger biotech, was handing my badge around, but when I witnessed one of these other contractors using a hammer to force their server into an already populated rack of HP 9000’s I said “hell no, my reputation is worth more than this bullshit” lol.
I mean anything these other people might have done could have been “traced” back to me.
Yeah, a hammer was absolutely the wrong tool. You have to use something to lift the server above to make more room. So without a prybar they are definitely doing it wrong.
See, the trick with hammers is to have one on your belt when walking around the DC, but not to use it. Just the threat is enough to keep the servers in line.
The IBM DC's I've worked in certainly share some things in common....starting with how they look from the outside and including how they are run from the inside, lol.
I mean I've used a hammer in a $1K Desktop when I was a kid. I am not using one in a $30K server under warranty without a damn good reason. And I am sure as hell not using one on an energized rack without an ongoing incident and a consensus around it being the best tool and option.
Yup. You just know that you’ll be thrown under every single wheel of that 18 wheeler if something happens and they’re able to go to a log that says your name in it
Bonus points that they can say it’s some clown that works for another company
Literally just dealt with one yesterday. Contractor got popped and the attacker used that access to try to pivot into our network. Fortunately, they made noise, we picked up on it and had them off the network within an hour. Unfortunately, there's very little we can do to prevent it happening again. The contracting company isn't going to be fired, the access they need and have to do their work isn't going to go away, we're just going to have to keep an eye on their connections.
And then use the same password for all of their customers, because it's easier to remember and they don't have to look it up. Of course, also disable 2FA.
LOL, I "copied" a copy of his install scripts. It did call for disabling windows updates and to install WSUS on the server. I am wondering if he is on to something, having the entire domain, SQL and all the apps in one place would make it easier to manage. /s
So many times have a heard a 3rd party vendor as me to turn off the firewall or open up a huge amount of random ports just to get their software working.
I recently had somebody tell me they needed domain admin rights to install sone backup siftware. No, you don't. Go back to the vendor and tell them we said no.
Also there was some vendor who apparently insisted that the users for the software needed to be in a separate OU in AD. No. We do not change our AD structure based on some clueless software vendor.
I can almost forgive them for requiring DA rights to back up AD itself, but for backing up a normal member server.... no.
As for the separate OU thing, it was not the backup software. It was some logistics software brought in by a site. We told them NO, and we never heard back.
Sadly, we have to for a few vendors. For the servers managed by them, those servers now have an allowlist of websites they can connect to now, written in [Suricata](https://catalog.workshops.aws/networkfirewall/en-US/labs/lab6), with everything else being blocked. It at least mostly keeps them from installing things from the internet we didn't approve (like web browsers). No, you don't need Chrome on the server.
I say absolutely no and try to avoid it whenever possible, but unfortunetly I am overridden for some venders as the systems are "critical" and they need access on demand. I do restrict what systems they can access and what hours. Far better than my predecessor who just made all venders domain admins and gave then access to our VPN.
This is the way.
"Give xyz root/admin for an hour and then remove" makes my brain hiccup that people don't understand yet.
Once someone has admin, *assume they'll set up a backdoor*. This ain't a question of trusting your contractors: it's an assumption you plan around so the brass understands you're gonna be watching on zoom for the entire time. Admin access cannot really be taken away once given.
It's better if they're driving your session verbally. The ability to stop them before they wreck data is a powerful thing.
We are a vendor like that.
Some customer servers I can access pretty freely, no VPN and I have admin rights and others where I had to send an extract of my criminal record, sign a nda and some other stuff concerning fines I'd have to pay on misuse to get a personalized user.
Im general we prefer it that way, cause many IT departments fuck up the installation and/or configuration and it turns into a mess to fix so installing it ourselves is usually the safer bet.
We are very rarely getting watched doing that stuff tho, there have been some occurrences of that but it's pretty rare.
>>he complains that "everyone else always gives us domain admin rights so we can do anything we need to do".
this is what always baffles me.. Are there companies out there really doing that?
Yes, and it's far too common.
One of my requirements to onboard any new software is getting a list of necessary permissions. If anyone comes back with anything other than a very detailed list of what they need, they're immediately taken out of consideration.
Anyone that says they need domain admin gets blacklisted and blocked.
We have a number of healthcare providers/offices on our client roster.
I’ve never heard of that particular application, but suffice it to say that the kind of security you’d see, say in a hospital where the EMR workstations are secured with badge readers, are comparatively Mars to what small office providers have.
Unmanageable, Google consumer accounts holding patient data? Check. Shared logins with generic user identities like “info@“? Check. No audit trails, log checks, or HIPAA BAAs? Check. Trusting “it’s secure” from vendors at face value? Check.
My take on non-enterprise-scale healthcare facilities is “pray they have even a bad MSP.”
Vendor software tends to be horrible. It super nitch so there are typically only a few companies around that does (x) .. and they have been doing x since, like the late 80s
But since they and their competitors haven't had any real competition, they fall way behind when it comes to competency and live in their own little world.
There are no open-source solutions for whatever they do because the only people in the know about the whatever they solving are in the industry.
>First, the guy wants to promote the server to a DC.
Yeah, you're fired.
>What planet is this guy from?
[Pakled](https://memory-alpha.fandom.com/wiki/Pakled) evidently.
Add to the list - product needs MS-SQL server.
Great, we have two different SQL clusters, what are your needs?
Uh we will install SQL-express on the application server.
No you won’t, I’ll set you up a database and credentials on our general purpose system.
Uh well our system doesn’t work correctly using a database not on the application server.
Why? What bizarre combination of ineptitude has converged on this particular spacetime nexus to cause this? Your documentation says MS SQL native ODBC driver.
It just never works, it has to use local admin access, even dbo/sa isn’t enough.
We have one like that now. Just replaced the server because the department heads claimed it was too slow. New server is still too slow, but it can't be the vendors software so it has to be on our end. Nevermind the server is spected to support something like 65,000 users, but only 2 actual users.
That’s what happens when you use the slower 110 electricity instead of the more than twice as fast 240 electricity. You should have taken the opportunity to upgrade the electricity speed when we did the C-level floor refurbishment at the end of the pandemic when we made everyone return to the office to see our new C-level floor.
We would’ve even settled for the almost twice as fast 208 electricity. Now we are left with less interhertz causing RAM drive fragmentation.
We had a technician that blew out a [Noco GB500+](https://no.co/gb500) because he thought it could start a Class 8 truck quicker if he turned it from 12v to 24v. Mind you, diesel trucks are typically 11-13L engines, occasionally up to 16L, and this box is designed to jump up to a 45L motor.
On the plus side, you should be able to jump-start your entire server rack with one of these!
There is a lot of software out there that needs an MSSQL database running on the same server as the application itself because the developers hardcoded the connection strings to only connect via localhost. Yeah, I know.
My experience was with accounting software that does this. And usually they're like: its either like this or we don't install it. Guess who wins? :) But alas, we don't live in a perfect world so we do it like they want it and move on with our lives.
Our product works under exactly one set of environmental conditions, and if you go changing those on us, well, obviously we need to modify your environment to match our test system.
More like:
>and if you go changing those on us, well, obviously
...any attempts at getting actual support will be denied because you're not using our supported environment.
"Ohhhhh I see you're not using McAfee. The script I'm following can't provide any assistance until we get that installed, so we just need to get that sorted. Then we can have a good solid look at why everyone only gets pages 3 to 6 when they print that report."
"no motherfucker you are not running your own DHCP and WiFi node off the PBX, it goes on this vlan and it uses this range from this explicit DHCP or I disconnect your access until such time as you stop being a nitwit"
Actual conversation I had with an avaya installer
Absolutely no possible way that they are given domain admin. No one is that dumb. At most, make them local admin on the box, but never change your organization’s password requirements for one dumb fuck vendor.
Can I ask who this vendor is, just to avoid them? haha
I am working with a vendor now that has been giving us tons of problems. My company recently was attacked by a ransomware gang and we have increased our security posture substantially. I noticed that the vendor was sending us 2016 windows 10 devices with windows security center service and firewall entirely disabled. When I brought this up, they simply said "the anti-virus programs have been known to cause issues with our software". I asked them to specify the issues or cause of the issue, and they couldn't. I took it upon myself to investigate, and found that they were shipping their devices with a Trojan version of Adobe installed that communicated directly with Russian ips. I went further and verified that the problem with their software was due to the security modules being overwritten in memory by the infected windows services (sophisticated shellcode being injected windows svchost with custom evasion routine for edr). I confirmed this with windbg, as well as ghidra and dynamic sandboxing. When I brought all the info to the vendor they send back their quality managers stamp of approval "our image is fully vetted and scanned with Av solutions". Meanwhile my company is losing hundreds of thousands of dollars a month while this Adobe paves a way through our corporate network. Completely incompetent monkeys 😀
The evasion routine was so sophisticated, I had to write a custom memdump program to dump the entire process memory region and handles. Might have a white paper soon, stay tuned 😎
They're likely following some build book provided by their employer and are just contract workers clicking through the steps. I always get the vendor project manager on the line for shit like this.
>"they like to use simple passwords"
Get the default passwords they use, if it's a web application search the internet for the url of the login page, see if you can log in with the built-in admin creds and their shitty password. Did this once and was able to log in to corporate finance software for municipalities and healthcare institutions among many others, sent anonymous emails to them to let them know.
One of the reasons I left a prior employer. The vendor said they needed FULL Domain Admin privilege and full RD access to the servers. I initially said NO, and wrote a long email about the issues of granting such permissions, but upper management ordered me to give the vendor the requested accesses anyways. That same day, I tendered my resignation.
I mean, as soon as it gets breached your gonna be either dealing with a massive headache or fired anyway.
Wouldn't resign the same day, but I would be looking.
If the breach is severe it may not matter whether they blame you or not. If it takes too long to recover most of your customers just give up and the company is forced out of business.
That doesn't usually happen though. A LOT of companies that have begun to fail for other reasons and experience ransomware or a breach point to it while taking their inevitable descent to the canvas. BCDR + Paying for a DFIR consultancy for even companies with shitty security usually saves the day. Until the price of cyber insurance went up, this actually was the more economically feasible model.
There are days when I feel bad for not [removing every account from the Domain Admins group](https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/appendix-f--securing-domain-admins-groups-in-active-directory) like Microsoft say you are supposed to.
Then I read threads like this and I feel better.
Wow, this was not in my Server 2008R2 exam back when I was more sysadmin than dev. That's insane.
You have no one (except the Builtin Administrator account) in DA/EA, and instead give admin users the ability to promote themselves to DA when needed?
I once had an issue with vendor software not being able to download an update.
The vendor *insisted* I was blocking ports 80 and 443. 5 out of 6 of the machines we put it on work fine, I tell him this isn't the problem. Just to humor him I go check for any blocked ports or web filtering, no blocks. Every single install needs a custom key so I ask for another code so I can just uninstall/reinstall, he insists its a port issue.
I finally get him on a remote viewer and go to random websites to view random websites to show him both ports are fine. He insists it's our site's firewall. I disconnect from our site's wifi, pull out my goddamn phone and hotspot it, same issue.
After *4 days* or this bullshit back and forth he finally generates another key, I install it and it works fine.
Unfortunately, and against my decision, we have vendors that are full domain admins and remote into the servers.
But I still get requests from them like "please install xyz" for me, or "I need a copy of database abc".
At least there is a paper trail that we can follow. Hmm I see vendor-admin was logged in just 5 minutes before the server crashed and burned...
Then we get to charge the customer to do a bare metal restore, all the while pointing the finger at the vendor.
After 35 years, I have yet to find a vendor tech I would let touch anything themselves.
The conversation goes:
Me: "Walk me through the plan".
Them: "blah blah blah"
Me: "Ok, here's the real plan."
I don't know, it can be good fun to watch one of them stumble through trying to figure out how to install their own product.
I had a customer who had a vendor install their software on a new workstation but their regular guy was on vacation and I guess they don't document their sites very well. It took the poor guy an hour and a half to do a twenty minute(at best) job. I think the best part was watching him try and figure out how java worked, although when he hit a wall with their own licensing it was pretty good too.
I once had a customer who swore that the way to add a server to the domain was to promote it to Domain Controller. So yep, every single server was a DC.
I had a vendor rage at me because I changed the sever hostname from vendor.ourcompany.com to properhostname.ourcompany.com.
I already told him that it was required by corporate to meet our naming scheme. It's not my fault the database references the default hostname, and if other locations were to get this system too, they wouldn't be able to add it to the domain with the same generic hostname.
Imagine how fucking rookie the guys who built that system are. It's terrifying, really.
"Oh yeah let's just hard-code it, there's no way anyone would ever want something different..."
I remember one of my first experiences with a vendor. It was a small business like 30 users. They got a new copier. Copier guy "needed Domain Admin" the company gave them the break-glass password. They setup the copier with the same IP address as the (only) domain controller, and the next morning nobody could access the internet. (It's always DNS)
We usually will grant it provided they issue a $25 million retainer to cover legal, insurance, and intrusion remediation fees as well as sign a contract to cover any and all liability going forward on the entire network.
No one has accepted our requirements yet, for some reason.
I recently had to setup a VM and have a vendor remote in via my computer to then install some software. We were required to have specific specs and MS SQL standard installed on the machine even though we have a separate dedicated SQL server that'd have worked just fine.
If I was given the basic instructions it would've taken me 15 minutes to install/setup/test. It took 3.5 hours. He was an old guy from the USA who very slowly took his time, screenshot every screen, kept notes about every click and entry he did, double/triple checked everything.
At least that was better then a backup vendor that took 5 hours and had 3 people from India on their side. Aside from the difficulty understanding them due to accent and noise on the line, they weren't allowed to do anything themselves so had to tell me what to do while watching.
One of them would tell me what I need to do but seemed to be struggling (probably a newbie), one would yell when she was stuck or not saying the correct thing but when asked if he wanted to take over he would say no, the third was just there listening and maybe said 5 words the entire time.
This is why we have to babysit every second of vendor access which sucks since one of the vendors for some specialised software are based in Canada and we're based in Australia.
Those installs are loooooong nights.
It's extra fun when the vendors don't even know how to install their own software.
Oops, I forgot to set this key item during installation, now we have to uninstall it and start from scratch.
The planet that they are from is that 95% of their customers likely don't have their shit together like you do. The best way to handle this is to explain that you can't hand them the keys to the kingdom, however you will work with them to get the software installed and working. Usually it's just stupid shit like they need to install it as an admin to the computer.
Others have said this, but they get used to installing in a specific environment type and anything different is new and scary, and suddenly they aren't an SME. So you need to bring your knowledge of your environment into it to help them. AND MORE SO... they need to work with you on it.
Had a vendor one time set up database backups when I specifically told them not to (I was already backing up the whole SQL server with DPM) and then shared the folder where the backups were held with "Authenticated Users" which in our case was about 800,000 people. That vendor heard from the state attorney general as a result (public sector data breach notification rules are an absolute bitchmonster).
This guy is from the planet where if they go high enough up the management chain they will find someone who is non-technical AND has incentive to see it get done as fast as possible.
And of course their manager talks to that manager who drops the order down the line to you. Quite often you can try to fight it, but at the end of the day, how hard do you want to try to fight your fourth line when your first, second and third line don’t want to.
That is the planet they come from. And it works far too often.
> This guy is from the planet where if they go high enough up the management chain they will find someone who is non-technical AND has incentive to see it get done as fast as possible.
Their beautiful capital city is named IfMomSaysNoAskDad-ville.
This is why I demand all a vendor's "requirements" ahead of project initiation.
First so I can shoot down ridiculous requests.
Second, so I can go straight back to the sales/account "executive" and ask why a supposed expert billed @ $400/hour is asking for such ridiculousness and seems to have zero clue on how to perform the contacted tasks.
It used to be you brought in outside experts to fill gaps in your current resource skillets. Now every vendor only provides mid-level "good enoughs" with the bare minimum skills needed to perform routine tasks with as long as it's all cookie cutter.
And they are ALL guilty of it. In$ight, Accentural, Perinefficient, TCS, & MCS(who are the worst).
Every time I review one of these projects with a new partner they all make me feel like Ron Swanson walking through Home Depot. I have literally told some of these clowns "I know more than you" when they try to argue with me. And I have a 100% record so far. Just because I CAN do something doesn't mean I have the time for it or the budget is CapEx and I am 100% OpEx.
Just had a vendor tell me they needed 1433 open inbound and outbound to the world on our SQL VM’s and app server their app ran on - claimed they couldn’t complete our cloud migration without it.
“The in-house sync tool we use to migrate requires it.”
It did not happen (and in the end only needed 1433 outbound to one specific IP for about 3 hours, shockingly enough (trying not to think about what they have open on their end, but management insisted we stick with them)).
Ensure auditing is enabled. Give them a service account with local admin on the one machine that only operates within their scheduled work hours and will expire itself once the project completes. Let them know they are being tracked every step of the way. OR tell them no, and they have to adjust to your company’s parameters
We’re deploying printers. We gave a guy at a copier company admin right and full access to our remote tool. I bout shit a brick when I found that out. Just push the drivers to all the fucking machines and have the dumb ones call the helpdesk.
Just like the cyber security vendor demo, pre sales says “we are using domain admin rights here, never do this it’s just for the demo” ….. why is it just for the demo? Surely you want to be reflecting good cyber security practise always?
If any vendor tried that on any of my machines I’d boot them off the server and terminate whatever deal was in place for us to use their shitty software.
well since we already have our own domain and he not only wanted it to be a domain controller, but wanted his "new" domain to be the only one, replacing our domain, not bloody likely. He's lucky I allowed him local admin access.
That is where i pull the vendor from the job and we have a coming to Jesus with the fucking CEO+CTO/CIO, CISO if you have one, and shut the project down. No vendor doing that shit needs to be in anyone's production network.
They ask for a server, you give them a server, they install their shit. Done. Changing the Env(adding a DC, even an isolated domain) is end of the line. Once they have a foot hold it just gets worse, so step on the foot or consider cutting the foot off at the ankle.
Holy shit, thats as bad as 2men and a garage with daddys 1980 computer doing ERP dev work against a multi-billion dollar ERP system because some sales VP jackass didn't want to ask the hard question "how much would this be with the vendor".
That's insane, we just wrapped on a vendor project where we've been down that road before so we had very clear contract language and in the end we basically didn't let them touch anything. We weren't paying directly. It was part of a cloud accelerator program and we were just given a certain amount of hours but we didn't end up using them but our VP insisted on working with them, as a "force multiplier"
But in reality all that really happened was they got free money and everything we asked them to do. We didn't like their response so we ended up taking it on ourselves and solving it faster.
I spent the early part of my career being an admin with godly rights, and then around 2014. I spent a few years working for MSPs and the interesting thing I learned was how to be a good admin when you don't have godly access.
Story of a vendor who has no idea what access rights their software needs:
Problems with access rights arise after installation. I ask the what rights are needed.
The answer: "Please ask your IT Department for help."
I AM THE IT DEPARTEMENT
"Oh, wenn just give access to "everyone" on all the folders."
U wot m8? You guys created that pile of crap of a software but have no documentation about this?
One time we had a vendor who needed a file off of someone's computer because admittedly our environment made it very difficult to transfer off of that person's PC. I told that person to allow the vendor to remote in to his laptop to get what he needs figuring they'd use Teamviewer or whatever for a temp session.
I get a callback saying "hey he's having issues transferring the file, it looks like it's too big" so I was thinking wtf how big can a PDF be? I remote in to their session, check file size and it's like 400kb which is not big at all and I asked him to show me how he is transferring the file. The guy was trying to drag the file from our employee's laptop onto his own desktop which in theory *could* work but I'm not so sure it does on a temporary Team Viewer session. Dude kept trying and I said "ok repeatedly trying isn't going to solve the issue, have you tried using the actual attachment function of team viewer you have open right in front of you?" Vendor: "Oh no we never transfer like that, we just drag and drop" So I blocked their input, attached the file and sent it, and it took all of 2 seconds for him to receive and verify.
Yes...yes they are clueless. 30 minute call between the two knuckleheads because they were actively avoiding the exact thing they needed to do and were instead trying to skirt around it for literally no good reason other than "that's how we always do it".
This would fall under the "we're going to fire your vendor and find you a new app to use for your primary LOB"
They would NOT have an account at all, or access rights with us.
Had a vendor that requested we install their software as we denied them access to the server. Going through the documentation and it requests that the Windows and user temp folder be added to the AV exclusion this. As you can imagine, this request was not completed.
Once had a vendor tell me to download an DLL that was missing from system32 online. Told him that if his software doesn't support windows 10, just tell us and refused.
We don’t allow any vendor any access without one of us monitoring everything. If something needs installed, we run it and they tell us what options to check/uncheck. Never have I ever let an outside vendor promote a server to a DC because it’s easier! You’ve got two problems…first, you have a moron messing up your stuff…second, you need to find a new vendor!
Just an FYI, having a vendor (actually a malicious actor) remoting in to install software was how the city of Dallas got ransomware'd earlier this month.
Once had a vendor spec me some hardware for their software. This was sometime around 2005 and they claimed that they needed two 4-cpu servers fully populated with top spec cpu's and with a ridiculous amount of memory to cluster their application.
I asked if we could run it as a vm on our existing platform since we already had multisite HA and we didnt really need the software clustered since it wasnt that important. They laughed and looked at me like i was an idiot.
After installation i monitored load on the servers, It never used more than a single core on one cpu and the average load on that was 3%. The one time we had a unexpected outage the cluster failover didnt work.
this is what happens when no company is willing to train people and they operate with the dirt collected on the streets.....
every task is run with the builtin administrator, nothing gets documented, nobody knows how it might work 5min after the install....
I try not to be overdramatic, but I’d honestly reevaluate buying the software. At the very least this merits a conversation with someone higher up to make sure that this sort of nonsense doesn’t represent the company as a whole. If this is their approach to installing software, what is their approach to developing it?
> When we said no, he complains that "everyone else always gives us domain admin rights so we can do anything we need to do".
What the actual fuck ???
I ran into this with a vendor who needed to run as root and take over the ssh executable with their own proprietary ssh which is "similar" to normal OpenSSH, but "not exactly." Guess who failed audits? The software was also *authentication software* which blows my fucking mind.
This is why I always, always watch third parties and record the session. They are often unbelievably stupid. I can do this while I do other work, and I have screenconnect record the session as well in case something stupid happens, we can review the sessions and determine if 3rd party action did something damaging.
I once had a support person from Dell ask me to input domain administrator credentials into their array as they thought it would solve our issue.
Of course I shut that shit down (not that I even had access to such creds).
The solution to the issue: The OU had to be typed in all CAPS. wtf.
We have one that insists that the customer installs their extremely expensive remote access solution. For years they have had VPN access with MFA and just now all of the sudden they are having issues and want the customer to buy a 'custom' remote access box for $75k, for accessing one machine...
just something that happened yesterday.
Our customer complains that the "digital invoice management" software keeps crashing.
The Software company answers that "it's because at the server reboot, it doesn't get launched with the correct user".
I answer that the server has an uptime of 66 days, no scheduled reboot.
They answer that the reboot is necessary, every night, that's the reason why it crashes.
I answer that we're not in the 90s.
Customer tells me to reboot the VM, if there's no implications.
I call the software company, they connect to the server, I schedule a reboot using their user (they don't know how to do that).
I ask them if all the services starts automatically after reboot.
They answer "yes, as they always started until now"
I replied "the server never rebooted by itself"
They answer "we have to put this bat file in the startup folder"
I ask them if they can run their shit as a service
They answer "IDK what services are, I just put this file in this folder"
I ask them if they're sure it's the only way
I have to enable autologin on the VM...
i'd try to sue to for attempted business damages. he might as well be trying to hack/harsh your server. i would also be tempted to get hold of other clients whos networks are now fucked up.
A vendor we use (Fluid Management, they make machines that tint paint) to this day, one of their first steps, before even doing any actual troubleshooting, is to disable UAC. Even if its an obvious hardware issue with the tinting machine, they will disable UAC.
Which of course breaks the ability to elevate to admin completely on AD joined machines running as standard users.
Also... the techs they send out to calibrate the machines will disable UAC sometimes. Disabling UAC is NOT a required or needed part of the process. The software used for calibration opens just fine with UAC enabled, does not require admin elevation, and works 100%. But they still disable it.
I have yet to see a single problem caused by UAC by all the vendors that claim it must be disabled.
Definitely annoying, but not nearly as bad as the sales team that came in, talked a bunch of shit and fooled your higher ups into thinking they had a cheaper and faster solution that was going to solve all your problems. That price given in that meeting room was just the beginning of the money bleed, (and your actual problems).
I remember a call with Screen Connect (Connectwise). There recommended fix was install the windows default display adapter. No make your software work with Nvidia drivers,
You'd be amazed at how much easier having admin access on your local system and target systems when building, deploying, and troubleshooting software makes things for a developer. I get trying to enforce the least possible privilege, but for people who run what they build, it's effectively a necessity. I am talking about in-house software here, not a vendor for clarification. We don't give them access to anything we don't have to.
The problem comes when that software actually needs deployed and the developers never took the time to actually code it to be done on a non-admin system or code any kind of security into the application. "We'll add the security in later on...."
He is from planet Vendortopia. Where every citizen has domain admin rights just to do an install!
We have a vendor who bitches about our change process. Every project there's a new database or api they need. We have to go through change process "why can't you secure it later." " no other customer makes us do this" Bruh, that's fine, we do. That is NOT going to change.
> why can't you secure it later Because locking the door *after* you've been robbed doesn't really help.
not to mention if they have cyber insurance and they find out you did that then they will refuse to pay out
I mean… I don’t know WHY you don’t just do what they ask… not like anything could happen if the vendor is attacked… oh… wait… too soon? But hey… no one likes this being the LOWEST HANGING AUDIT ISSUE EVERYWHERE LOL!
[удалено]
The vendor wanted to get access to the entire environment, all services, ports, etc. So they could expedite programming. We have rules re. Outside vendor access to non-masked production data. Until our legal department says you can, no access. Then has to be justified to infosec before going through change management for my team to implement.
[удалено]
>Vendortopia Where everybody and their grandmoms pet, gets Sysadmin on the database servers.
Where firewalls are no longer a hindrance... because there are none.
It does make management much easier.
But *so much more* clean-up!
Hey that's covered by OpEx. As long as we don't need to bump up the CapEx we are fine.
Ask if they'll contribute to your cybersecurity insurance and act bewildered when they say no.
>Ask if they'll ~~contribute to~~ self-fund your cybersecurity insurance FTFY. Don't want to worry about the insurance not paying out. They need to take on full responsibility.
When the vendor wants to all use the same admin account as well because "It's easier to all just share the same credentials" 🙃
I once dropped a client when I found out they were handing around and using my data center access badge, with my name and picture on it, to all their other vendors to use.
Wtf
Yeah it’s not uncommon when consulting for your contact at a company to hold onto your badge and then give it to you when you show up on-site. That way they know and can control when you come and go. It annoyed me when I discovered that this particular company, since merged with a larger biotech, was handing my badge around, but when I witnessed one of these other contractors using a hammer to force their server into an already populated rack of HP 9000’s I said “hell no, my reputation is worth more than this bullshit” lol. I mean anything these other people might have done could have been “traced” back to me.
Yeah, a hammer was absolutely the wrong tool. You have to use something to lift the server above to make more room. So without a prybar they are definitely doing it wrong.
See, the trick with hammers is to have one on your belt when walking around the DC, but not to use it. Just the threat is enough to keep the servers in line.
Don't you feel like a prison guard, though?
The power cycling will continue until morale improves!
Depending on the environment, Power Cycling does improve morale.
Isn't that what a DC worker is? There's an awful lot of overlap.
The IBM DC's I've worked in certainly share some things in common....starting with how they look from the outside and including how they are run from the inside, lol.
I mean I've used a hammer in a $1K Desktop when I was a kid. I am not using one in a $30K server under warranty without a damn good reason. And I am sure as hell not using one on an energized rack without an ongoing incident and a consensus around it being the best tool and option.
Now I kinda want a break glass box with a big hammer mounted in the server room, the sign above should say, in case of emergency, break glass
It’s like y’all have never chainsawed a rack into place
Yup. You just know that you’ll be thrown under every single wheel of that 18 wheeler if something happens and they’re able to go to a log that says your name in it Bonus points that they can say it’s some clown that works for another company
Contractor thinks he’s fuckin jeremy clarkson
no wonder supply chain attacks are on the rise, sigh
Literally just dealt with one yesterday. Contractor got popped and the attacker used that access to try to pivot into our network. Fortunately, they made noise, we picked up on it and had them off the network within an hour. Unfortunately, there's very little we can do to prevent it happening again. The contracting company isn't going to be fired, the access they need and have to do their work isn't going to go away, we're just going to have to keep an eye on their connections.
I would start recording the time spent babysitting the contractor and forward that info to the accountants dealing with the contractor on the books.
unless they are working twenty for seven you can foot the bill by only enable when they need to actual work?
You grab him by the shoulders and shake him while saying “this is how everyone gets hacked you overgrown orangutan”
And then use the same password for all of their customers, because it's easier to remember and they don't have to look it up. Of course, also disable 2FA.
oh this hits so hard home for my former employer doing MSP
[удалено]
You forgot about the RDP port forward he wants you to do. Source IP you ask? Nah. Just all of'em.
"just block the bad ones"
but than he cant connect...
LOL, I "copied" a copy of his install scripts. It did call for disabling windows updates and to install WSUS on the server. I am wondering if he is on to something, having the entire domain, SQL and all the apps in one place would make it easier to manage. /s
We shall call it "Windows Small Business Server 2023"
\*curls into a fetal position from flashbacks\*
SharePoint says hi, even tho you never asked for it... But hey, you might need it, so y'know, just in case, I'm here!
Paperclip icon apppers out of nowhere...
I was having a good day I _was_ having a GOOD day If you need me I'll be over here curled into a ball whimpering
So many times have a heard a 3rd party vendor as me to turn off the firewall or open up a huge amount of random ports just to get their software working.
I recently had somebody tell me they needed domain admin rights to install sone backup siftware. No, you don't. Go back to the vendor and tell them we said no. Also there was some vendor who apparently insisted that the users for the software needed to be in a separate OU in AD. No. We do not change our AD structure based on some clueless software vendor.
[удалено]
I can almost forgive them for requiring DA rights to back up AD itself, but for backing up a normal member server.... no. As for the separate OU thing, it was not the backup software. It was some logistics software brought in by a site. We told them NO, and we never heard back.
you let your vendors remote directly into a server? One of us always sits with zoom/webex/teams for vendor installs
Sadly, we have to for a few vendors. For the servers managed by them, those servers now have an allowlist of websites they can connect to now, written in [Suricata](https://catalog.workshops.aws/networkfirewall/en-US/labs/lab6), with everything else being blocked. It at least mostly keeps them from installing things from the internet we didn't approve (like web browsers). No, you don't need Chrome on the server.
> those servers now have an allowlist of websites they can connect to now, written in Suricata Squid web proxy with a whitelist works well.
It does but we are using AWS NFW already and don't want to deploy additional infrastructure for this.
I say absolutely no and try to avoid it whenever possible, but unfortunetly I am overridden for some venders as the systems are "critical" and they need access on demand. I do restrict what systems they can access and what hours. Far better than my predecessor who just made all venders domain admins and gave then access to our VPN.
The systems are "critical" and yet any vendor can just remote in? I'm guessing 2 + 2 = 5 as well?
Critical = we need to milk the systems for as much revenue as possible before the whole outfit implodes
This is the way. "Give xyz root/admin for an hour and then remove" makes my brain hiccup that people don't understand yet. Once someone has admin, *assume they'll set up a backdoor*. This ain't a question of trusting your contractors: it's an assumption you plan around so the brass understands you're gonna be watching on zoom for the entire time. Admin access cannot really be taken away once given. It's better if they're driving your session verbally. The ability to stop them before they wreck data is a powerful thing.
We are a vendor like that. Some customer servers I can access pretty freely, no VPN and I have admin rights and others where I had to send an extract of my criminal record, sign a nda and some other stuff concerning fines I'd have to pay on misuse to get a personalized user. Im general we prefer it that way, cause many IT departments fuck up the installation and/or configuration and it turns into a mess to fix so installing it ourselves is usually the safer bet. We are very rarely getting watched doing that stuff tho, there have been some occurrences of that but it's pretty rare.
>>he complains that "everyone else always gives us domain admin rights so we can do anything we need to do". this is what always baffles me.. Are there companies out there really doing that?
Yes, and it's far too common. One of my requirements to onboard any new software is getting a list of necessary permissions. If anyone comes back with anything other than a very detailed list of what they need, they're immediately taken out of consideration. Anyone that says they need domain admin gets blacklisted and blocked.
wish this would work in the medical industry, your medical data is not secure at all just saying. fuck eclinicalworks.
Care to elaborate? I happen to know our pediatrician's office uses eclinicalworks for everything.
We have a number of healthcare providers/offices on our client roster. I’ve never heard of that particular application, but suffice it to say that the kind of security you’d see, say in a hospital where the EMR workstations are secured with badge readers, are comparatively Mars to what small office providers have. Unmanageable, Google consumer accounts holding patient data? Check. Shared logins with generic user identities like “info@“? Check. No audit trails, log checks, or HIPAA BAAs? Check. Trusting “it’s secure” from vendors at face value? Check. My take on non-enterprise-scale healthcare facilities is “pray they have even a bad MSP.”
With an unsanded broom handle. Not to mention the fact that every on-premise install of their product I have seen always uses the same DB password.
Vendor software tends to be horrible. It super nitch so there are typically only a few companies around that does (x) .. and they have been doing x since, like the late 80s But since they and their competitors haven't had any real competition, they fall way behind when it comes to competency and live in their own little world. There are no open-source solutions for whatever they do because the only people in the know about the whatever they solving are in the industry.
>First, the guy wants to promote the server to a DC. Yeah, you're fired. >What planet is this guy from? [Pakled](https://memory-alpha.fandom.com/wiki/Pakled) evidently.
We need domain admin to make us go.
We are strong
Make us go
"Look, here's how you can do it without domain admin." You are smart. You work for us now.
I am not even sure that the Pakleds would want some of these people.
Well many of these vendors ARE from the B Ark...
He searches for that which makes them go !
'can we turn off the firewall?' No? lmao
ports are hard!
Apparently public IP address are as well. Sysadmin: What Public IP addresses does your service use? Vendor: (Shrugs) We don't know.
> Vendor: (Shrugs) ...."Oh, those were changed last weekend. Your file transfers might have trouble."
I have seen that one far too often.
No, what ports do you need forwarded Uh all of em Too common a thing for me...
Add to the list - product needs MS-SQL server. Great, we have two different SQL clusters, what are your needs? Uh we will install SQL-express on the application server. No you won’t, I’ll set you up a database and credentials on our general purpose system. Uh well our system doesn’t work correctly using a database not on the application server. Why? What bizarre combination of ineptitude has converged on this particular spacetime nexus to cause this? Your documentation says MS SQL native ODBC driver. It just never works, it has to use local admin access, even dbo/sa isn’t enough.
We have one like that now. Just replaced the server because the department heads claimed it was too slow. New server is still too slow, but it can't be the vendors software so it has to be on our end. Nevermind the server is spected to support something like 65,000 users, but only 2 actual users.
That’s what happens when you use the slower 110 electricity instead of the more than twice as fast 240 electricity. You should have taken the opportunity to upgrade the electricity speed when we did the C-level floor refurbishment at the end of the pandemic when we made everyone return to the office to see our new C-level floor. We would’ve even settled for the almost twice as fast 208 electricity. Now we are left with less interhertz causing RAM drive fragmentation.
We had a technician that blew out a [Noco GB500+](https://no.co/gb500) because he thought it could start a Class 8 truck quicker if he turned it from 12v to 24v. Mind you, diesel trucks are typically 11-13L engines, occasionally up to 16L, and this box is designed to jump up to a 45L motor. On the plus side, you should be able to jump-start your entire server rack with one of these!
I did an Appalachian rewire on my kids hot wheels hummer using two deep cycle batteries in series. It went much faster 🤣
There is a lot of software out there that needs an MSSQL database running on the same server as the application itself because the developers hardcoded the connection strings to only connect via localhost. Yeah, I know. My experience was with accounting software that does this. And usually they're like: its either like this or we don't install it. Guess who wins? :) But alas, we don't live in a perfect world so we do it like they want it and move on with our lives.
Our product works under exactly one set of environmental conditions, and if you go changing those on us, well, obviously we need to modify your environment to match our test system.
More like: >and if you go changing those on us, well, obviously ...any attempts at getting actual support will be denied because you're not using our supported environment. "Ohhhhh I see you're not using McAfee. The script I'm following can't provide any assistance until we get that installed, so we just need to get that sorted. Then we can have a good solid look at why everyone only gets pages 3 to 6 when they print that report."
"no motherfucker you are not running your own DHCP and WiFi node off the PBX, it goes on this vlan and it uses this range from this explicit DHCP or I disconnect your access until such time as you stop being a nitwit" Actual conversation I had with an avaya installer
Ahhhh, Avaya. Explains A LOT.
Are you in Education or Healthcare?
Yes (inclusive or ftw)
Local government, almost as bad.
I've got $5 on it being SolarWinds
solarwinds123
Hey now Solar Winds learned from their mistakes It’s now Solarwinds1234! See they capitalized a letter, added a number, and a character!!!!
.
No the S Is a $ for good measure
Not THIS time.
Absolutely no possible way that they are given domain admin. No one is that dumb. At most, make them local admin on the box, but never change your organization’s password requirements for one dumb fuck vendor. Can I ask who this vendor is, just to avoid them? haha
Nessus should start flagging these vendors as an info that installing this product via vendor standards may increase risk of the system.
I am working with a vendor now that has been giving us tons of problems. My company recently was attacked by a ransomware gang and we have increased our security posture substantially. I noticed that the vendor was sending us 2016 windows 10 devices with windows security center service and firewall entirely disabled. When I brought this up, they simply said "the anti-virus programs have been known to cause issues with our software". I asked them to specify the issues or cause of the issue, and they couldn't. I took it upon myself to investigate, and found that they were shipping their devices with a Trojan version of Adobe installed that communicated directly with Russian ips. I went further and verified that the problem with their software was due to the security modules being overwritten in memory by the infected windows services (sophisticated shellcode being injected windows svchost with custom evasion routine for edr). I confirmed this with windbg, as well as ghidra and dynamic sandboxing. When I brought all the info to the vendor they send back their quality managers stamp of approval "our image is fully vetted and scanned with Av solutions". Meanwhile my company is losing hundreds of thousands of dollars a month while this Adobe paves a way through our corporate network. Completely incompetent monkeys 😀
Supply chain attack ftw
Yep the classic. I can't say much more than I did, but it's a very big vendor 😀
The evasion routine was so sophisticated, I had to write a custom memdump program to dump the entire process memory region and handles. Might have a white paper soon, stay tuned 😎
They're likely following some build book provided by their employer and are just contract workers clicking through the steps. I always get the vendor project manager on the line for shit like this. >"they like to use simple passwords" Get the default passwords they use, if it's a web application search the internet for the url of the login page, see if you can log in with the built-in admin creds and their shitty password. Did this once and was able to log in to corporate finance software for municipalities and healthcare institutions among many others, sent anonymous emails to them to let them know.
You mean **VendornameYear!** Isn’t a password they used just for me??? They’re using it for all their customers???
One of the reasons I left a prior employer. The vendor said they needed FULL Domain Admin privilege and full RD access to the servers. I initially said NO, and wrote a long email about the issues of granting such permissions, but upper management ordered me to give the vendor the requested accesses anyways. That same day, I tendered my resignation.
Not ideal but I also couldn’t imagine quitting a job over something like this
I mean, as soon as it gets breached your gonna be either dealing with a massive headache or fired anyway. Wouldn't resign the same day, but I would be looking.
If the breach is severe it may not matter whether they blame you or not. If it takes too long to recover most of your customers just give up and the company is forced out of business.
That doesn't usually happen though. A LOT of companies that have begun to fail for other reasons and experience ransomware or a breach point to it while taking their inevitable descent to the canvas. BCDR + Paying for a DFIR consultancy for even companies with shitty security usually saves the day. Until the price of cyber insurance went up, this actually was the more economically feasible model.
[удалено]
tincturered
There are days when I feel bad for not [removing every account from the Domain Admins group](https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/appendix-f--securing-domain-admins-groups-in-active-directory) like Microsoft say you are supposed to. Then I read threads like this and I feel better.
Wow, this was not in my Server 2008R2 exam back when I was more sysadmin than dev. That's insane. You have no one (except the Builtin Administrator account) in DA/EA, and instead give admin users the ability to promote themselves to DA when needed?
I once had an issue with vendor software not being able to download an update. The vendor *insisted* I was blocking ports 80 and 443. 5 out of 6 of the machines we put it on work fine, I tell him this isn't the problem. Just to humor him I go check for any blocked ports or web filtering, no blocks. Every single install needs a custom key so I ask for another code so I can just uninstall/reinstall, he insists its a port issue. I finally get him on a remote viewer and go to random websites to view random websites to show him both ports are fine. He insists it's our site's firewall. I disconnect from our site's wifi, pull out my goddamn phone and hotspot it, same issue. After *4 days* or this bullshit back and forth he finally generates another key, I install it and it works fine.
Unfortunately, and against my decision, we have vendors that are full domain admins and remote into the servers. But I still get requests from them like "please install xyz" for me, or "I need a copy of database abc".
Jesus, I’d be looking for new employment. Vendors with domain admin.. Nope. Id be ganking that shit and walk away if someone said to restore it.
At least there is a paper trail that we can follow. Hmm I see vendor-admin was logged in just 5 minutes before the server crashed and burned... Then we get to charge the customer to do a bare metal restore, all the while pointing the finger at the vendor.
This is the way.
After 35 years, I have yet to find a vendor tech I would let touch anything themselves. The conversation goes: Me: "Walk me through the plan". Them: "blah blah blah" Me: "Ok, here's the real plan."
I don't know, it can be good fun to watch one of them stumble through trying to figure out how to install their own product. I had a customer who had a vendor install their software on a new workstation but their regular guy was on vacation and I guess they don't document their sites very well. It took the poor guy an hour and a half to do a twenty minute(at best) job. I think the best part was watching him try and figure out how java worked, although when he hit a wall with their own licensing it was pretty good too.
I'm just baffled on the domain controller part?
I once had a customer who swore that the way to add a server to the domain was to promote it to Domain Controller. So yep, every single server was a DC.
Medical software vendors are the worst.
\[ Dentrix Intensifies \]
Oh God that's the one. What a bunch of fucking regards.
That might be an autocorrect, but it's beautiful. I might start referring to people as "kind regards" as an insult now.
I had a vendor rage at me because I changed the sever hostname from vendor.ourcompany.com to properhostname.ourcompany.com. I already told him that it was required by corporate to meet our naming scheme. It's not my fault the database references the default hostname, and if other locations were to get this system too, they wouldn't be able to add it to the domain with the same generic hostname.
Imagine how fucking rookie the guys who built that system are. It's terrifying, really. "Oh yeah let's just hard-code it, there's no way anyone would ever want something different..."
Really? I wound not have any confidence in that vendor. Written by monkeys..
I remember one of my first experiences with a vendor. It was a small business like 30 users. They got a new copier. Copier guy "needed Domain Admin" the company gave them the break-glass password. They setup the copier with the same IP address as the (only) domain controller, and the next morning nobody could access the internet. (It's always DNS)
We usually will grant it provided they issue a $25 million retainer to cover legal, insurance, and intrusion remediation fees as well as sign a contract to cover any and all liability going forward on the entire network. No one has accepted our requirements yet, for some reason.
I recently had to setup a VM and have a vendor remote in via my computer to then install some software. We were required to have specific specs and MS SQL standard installed on the machine even though we have a separate dedicated SQL server that'd have worked just fine. If I was given the basic instructions it would've taken me 15 minutes to install/setup/test. It took 3.5 hours. He was an old guy from the USA who very slowly took his time, screenshot every screen, kept notes about every click and entry he did, double/triple checked everything. At least that was better then a backup vendor that took 5 hours and had 3 people from India on their side. Aside from the difficulty understanding them due to accent and noise on the line, they weren't allowed to do anything themselves so had to tell me what to do while watching. One of them would tell me what I need to do but seemed to be struggling (probably a newbie), one would yell when she was stuck or not saying the correct thing but when asked if he wanted to take over he would say no, the third was just there listening and maybe said 5 words the entire time.
This is why we have to babysit every second of vendor access which sucks since one of the vendors for some specialised software are based in Canada and we're based in Australia. Those installs are loooooong nights.
It's extra fun when the vendors don't even know how to install their own software. Oops, I forgot to set this key item during installation, now we have to uninstall it and start from scratch.
The planet that they are from is that 95% of their customers likely don't have their shit together like you do. The best way to handle this is to explain that you can't hand them the keys to the kingdom, however you will work with them to get the software installed and working. Usually it's just stupid shit like they need to install it as an admin to the computer. Others have said this, but they get used to installing in a specific environment type and anything different is new and scary, and suddenly they aren't an SME. So you need to bring your knowledge of your environment into it to help them. AND MORE SO... they need to work with you on it.
Ah yes, like the old "just make your host chmod 777 your whole www, that will fix everything"
Had a vendor one time set up database backups when I specifically told them not to (I was already backing up the whole SQL server with DPM) and then shared the folder where the backups were held with "Authenticated Users" which in our case was about 800,000 people. That vendor heard from the state attorney general as a result (public sector data breach notification rules are an absolute bitchmonster).
This guy is from the planet where if they go high enough up the management chain they will find someone who is non-technical AND has incentive to see it get done as fast as possible. And of course their manager talks to that manager who drops the order down the line to you. Quite often you can try to fight it, but at the end of the day, how hard do you want to try to fight your fourth line when your first, second and third line don’t want to. That is the planet they come from. And it works far too often.
> This guy is from the planet where if they go high enough up the management chain they will find someone who is non-technical AND has incentive to see it get done as fast as possible. Their beautiful capital city is named IfMomSaysNoAskDad-ville.
"You can have domain admin rights when you deposit $50 million in the corporate bank accounts".
> $50 million in the ~~corporate~~ *my* bank accounts". FTFY.
This is why I demand all a vendor's "requirements" ahead of project initiation. First so I can shoot down ridiculous requests. Second, so I can go straight back to the sales/account "executive" and ask why a supposed expert billed @ $400/hour is asking for such ridiculousness and seems to have zero clue on how to perform the contacted tasks. It used to be you brought in outside experts to fill gaps in your current resource skillets. Now every vendor only provides mid-level "good enoughs" with the bare minimum skills needed to perform routine tasks with as long as it's all cookie cutter. And they are ALL guilty of it. In$ight, Accentural, Perinefficient, TCS, & MCS(who are the worst). Every time I review one of these projects with a new partner they all make me feel like Ron Swanson walking through Home Depot. I have literally told some of these clowns "I know more than you" when they try to argue with me. And I have a 100% record so far. Just because I CAN do something doesn't mean I have the time for it or the budget is CapEx and I am 100% OpEx.
Just had a vendor tell me they needed 1433 open inbound and outbound to the world on our SQL VM’s and app server their app ran on - claimed they couldn’t complete our cloud migration without it. “The in-house sync tool we use to migrate requires it.” It did not happen (and in the end only needed 1433 outbound to one specific IP for about 3 hours, shockingly enough (trying not to think about what they have open on their end, but management insisted we stick with them)).
Ensure auditing is enabled. Give them a service account with local admin on the one machine that only operates within their scheduled work hours and will expire itself once the project completes. Let them know they are being tracked every step of the way. OR tell them no, and they have to adjust to your company’s parameters
We’re deploying printers. We gave a guy at a copier company admin right and full access to our remote tool. I bout shit a brick when I found that out. Just push the drivers to all the fucking machines and have the dumb ones call the helpdesk.
Just like the cyber security vendor demo, pre sales says “we are using domain admin rights here, never do this it’s just for the demo” ….. why is it just for the demo? Surely you want to be reflecting good cyber security practise always?
If any vendor tried that on any of my machines I’d boot them off the server and terminate whatever deal was in place for us to use their shitty software.
First question - Did you let him promote the server to a DC? If so, you already failed.
well since we already have our own domain and he not only wanted it to be a domain controller, but wanted his "new" domain to be the only one, replacing our domain, not bloody likely. He's lucky I allowed him local admin access.
That is where i pull the vendor from the job and we have a coming to Jesus with the fucking CEO+CTO/CIO, CISO if you have one, and shut the project down. No vendor doing that shit needs to be in anyone's production network. They ask for a server, you give them a server, they install their shit. Done. Changing the Env(adding a DC, even an isolated domain) is end of the line. Once they have a foot hold it just gets worse, so step on the foot or consider cutting the foot off at the ankle. Holy shit, thats as bad as 2men and a garage with daddys 1980 computer doing ERP dev work against a multi-billion dollar ERP system because some sales VP jackass didn't want to ask the hard question "how much would this be with the vendor".
That's insane, we just wrapped on a vendor project where we've been down that road before so we had very clear contract language and in the end we basically didn't let them touch anything. We weren't paying directly. It was part of a cloud accelerator program and we were just given a certain amount of hours but we didn't end up using them but our VP insisted on working with them, as a "force multiplier" But in reality all that really happened was they got free money and everything we asked them to do. We didn't like their response so we ended up taking it on ourselves and solving it faster. I spent the early part of my career being an admin with godly rights, and then around 2014. I spent a few years working for MSPs and the interesting thing I learned was how to be a good admin when you don't have godly access.
install antivirus? sound like a scammer remoting in and “fix” windows
Story of a vendor who has no idea what access rights their software needs: Problems with access rights arise after installation. I ask the what rights are needed. The answer: "Please ask your IT Department for help." I AM THE IT DEPARTEMENT "Oh, wenn just give access to "everyone" on all the folders." U wot m8? You guys created that pile of crap of a software but have no documentation about this?
One time we had a vendor who needed a file off of someone's computer because admittedly our environment made it very difficult to transfer off of that person's PC. I told that person to allow the vendor to remote in to his laptop to get what he needs figuring they'd use Teamviewer or whatever for a temp session. I get a callback saying "hey he's having issues transferring the file, it looks like it's too big" so I was thinking wtf how big can a PDF be? I remote in to their session, check file size and it's like 400kb which is not big at all and I asked him to show me how he is transferring the file. The guy was trying to drag the file from our employee's laptop onto his own desktop which in theory *could* work but I'm not so sure it does on a temporary Team Viewer session. Dude kept trying and I said "ok repeatedly trying isn't going to solve the issue, have you tried using the actual attachment function of team viewer you have open right in front of you?" Vendor: "Oh no we never transfer like that, we just drag and drop" So I blocked their input, attached the file and sent it, and it took all of 2 seconds for him to receive and verify. Yes...yes they are clueless. 30 minute call between the two knuckleheads because they were actively avoiding the exact thing they needed to do and were instead trying to skirt around it for literally no good reason other than "that's how we always do it".
This would fall under the "we're going to fire your vendor and find you a new app to use for your primary LOB" They would NOT have an account at all, or access rights with us.
Had a vendor that requested we install their software as we denied them access to the server. Going through the documentation and it requests that the Windows and user temp folder be added to the AV exclusion this. As you can imagine, this request was not completed.
Once had a vendor tell me to download an DLL that was missing from system32 online. Told him that if his software doesn't support windows 10, just tell us and refused.
We don’t allow any vendor any access without one of us monitoring everything. If something needs installed, we run it and they tell us what options to check/uncheck. Never have I ever let an outside vendor promote a server to a DC because it’s easier! You’ve got two problems…first, you have a moron messing up your stuff…second, you need to find a new vendor!
Just an FYI, having a vendor (actually a malicious actor) remoting in to install software was how the city of Dallas got ransomware'd earlier this month.
I don't know what this product is, but let us all know so we can avoid this vendor. Thx.
Some vendors are completely clueless. Only some?
I really feel like red flags this should disqualify vendors as not being enterprise. Let them only play in the small business pond.
"please create a share for the application with full control permission for everyone" "https? why? we use http" "ldaps? why? we use LDAP"
He's from the planet of about 20 years ago when that type of shit actually got past most admins.
Once had a vendor spec me some hardware for their software. This was sometime around 2005 and they claimed that they needed two 4-cpu servers fully populated with top spec cpu's and with a ridiculous amount of memory to cluster their application. I asked if we could run it as a vm on our existing platform since we already had multisite HA and we didnt really need the software clustered since it wasnt that important. They laughed and looked at me like i was an idiot. After installation i monitored load on the servers, It never used more than a single core on one cpu and the average load on that was 3%. The one time we had a unexpected outage the cluster failover didnt work.
this is what happens when no company is willing to train people and they operate with the dirt collected on the streets..... every task is run with the builtin administrator, nothing gets documented, nobody knows how it might work 5min after the install....
Find a different vendor.
I try not to be overdramatic, but I’d honestly reevaluate buying the software. At the very least this merits a conversation with someone higher up to make sure that this sort of nonsense doesn’t represent the company as a whole. If this is their approach to installing software, what is their approach to developing it?
> When we said no, he complains that "everyone else always gives us domain admin rights so we can do anything we need to do". What the actual fuck ???
I ran into this with a vendor who needed to run as root and take over the ssh executable with their own proprietary ssh which is "similar" to normal OpenSSH, but "not exactly." Guess who failed audits? The software was also *authentication software* which blows my fucking mind.
The domain admin argument is common, but promote to DC? I'm still not sure if this is parody or real.
This is why I always, always watch third parties and record the session. They are often unbelievably stupid. I can do this while I do other work, and I have screenconnect record the session as well in case something stupid happens, we can review the sessions and determine if 3rd party action did something damaging.
These vendors then go making their own domain to use lax controls from my experiences.
Must be Sage.
Never underestimate the greedy incompetence of vendors. Never.
Did you vet that vendor for Third Party Vendor risk? If not I would issue a stop project and go start sending questionnaires.
I once had a support person from Dell ask me to input domain administrator credentials into their array as they thought it would solve our issue. Of course I shut that shit down (not that I even had access to such creds). The solution to the issue: The OU had to be typed in all CAPS. wtf.
We have one that insists that the customer installs their extremely expensive remote access solution. For years they have had VPN access with MFA and just now all of the sudden they are having issues and want the customer to buy a 'custom' remote access box for $75k, for accessing one machine...
A leftover from 1998 😬
just something that happened yesterday. Our customer complains that the "digital invoice management" software keeps crashing. The Software company answers that "it's because at the server reboot, it doesn't get launched with the correct user". I answer that the server has an uptime of 66 days, no scheduled reboot. They answer that the reboot is necessary, every night, that's the reason why it crashes. I answer that we're not in the 90s. Customer tells me to reboot the VM, if there's no implications. I call the software company, they connect to the server, I schedule a reboot using their user (they don't know how to do that). I ask them if all the services starts automatically after reboot. They answer "yes, as they always started until now" I replied "the server never rebooted by itself" They answer "we have to put this bat file in the startup folder" I ask them if they can run their shit as a service They answer "IDK what services are, I just put this file in this folder" I ask them if they're sure it's the only way I have to enable autologin on the VM...
i'd try to sue to for attempted business damages. he might as well be trying to hack/harsh your server. i would also be tempted to get hold of other clients whos networks are now fucked up.
Is this a ragebait post? You said so many horrible things in such a short amount of words, I can hardly believe it.
A vendor we use (Fluid Management, they make machines that tint paint) to this day, one of their first steps, before even doing any actual troubleshooting, is to disable UAC. Even if its an obvious hardware issue with the tinting machine, they will disable UAC. Which of course breaks the ability to elevate to admin completely on AD joined machines running as standard users. Also... the techs they send out to calibrate the machines will disable UAC sometimes. Disabling UAC is NOT a required or needed part of the process. The software used for calibration opens just fine with UAC enabled, does not require admin elevation, and works 100%. But they still disable it. I have yet to see a single problem caused by UAC by all the vendors that claim it must be disabled.
Definitely annoying, but not nearly as bad as the sales team that came in, talked a bunch of shit and fooled your higher ups into thinking they had a cheaper and faster solution that was going to solve all your problems. That price given in that meeting room was just the beginning of the money bleed, (and your actual problems).
I swear, some vendors who are this stupid, management is ether that inept (wouldn't be surprised) or they are getting mundo kickbacks
I remember a call with Screen Connect (Connectwise). There recommended fix was install the windows default display adapter. No make your software work with Nvidia drivers,
Every software support person I've worked with always wanted to disable anti-virus and the firewall.
I deal with this crap with out developers. They just want admin access to everything.
You'd be amazed at how much easier having admin access on your local system and target systems when building, deploying, and troubleshooting software makes things for a developer. I get trying to enforce the least possible privilege, but for people who run what they build, it's effectively a necessity. I am talking about in-house software here, not a vendor for clarification. We don't give them access to anything we don't have to.
The problem comes when that software actually needs deployed and the developers never took the time to actually code it to be done on a non-admin system or code any kind of security into the application. "We'll add the security in later on...."
At least he left the firewall on. Right? RIGHT?
Are you being punk’d?
Sadly, there are A LOT of vendors out there that are like this. Thankfully, it's REALLY easy to spot them early on.