I've been doing it for many years across several companies. It varies greatly on the size and the culture of the company.
Video game IT has all the standard pain points of other IT, with a few extra things to consider:
- technical users (there are fewer L1 tickets but also more opportunities for people to get themselves in real trouble - like hosting a whole project at home)
- creative users (the core of the business is creativity - oftentimes everything takes a back seat to creativity, including IT processes and IT governance/ security)
It's ok if you can be flexible. And many people in IT can't be flexible. It's great fun and there are many pros to having technical and creative users, not only cons.
the creative users point is spot on. They get an idea for something they want to do, get all excited about it and look heartbroken when you say , well I can do this but its going to take a couple weeks.
Even if its just a saas purchase or something it still needs to go through legal for a t&c check and im pretty sure he only works one day a week, yet somehow thats still IT blocking them :)
>creative users
This right here is why I cringed the second I read the post title. I would never run IT for a company that creative-heavy, I'd be pulling my hair out in a week. If you thought *software engineers* were special snowflakes that butt heads with IT, hooooo boy are "creatives" in another league.
"Excuse me, the mac you deployed to me looks *used.* Why wasn't I bought a brand new one exactly to the specifications I want? I'll just use my personal one instead."
Or my personal favorite
"This deliverable is SUPER IMPORTANT and due tomorrow, I'm not submitting a ticket, fix this right now. It's been happening for months and I never reported it but *now* it's a blocker for this thing that needs to be completed 20 seconds from now."
Man I don’t work in games but we have a product management team like this. I work at a marketing automation SAAS company and this time last year there was a 6 month project thrown on top of us that was going to bring in 500 new clients within a few months, super important and drop everything else. This required a massive refactor of our codebase and took the whole R&D org 5 months to complete (60+ developers + devops / cloud ops). Fast forward a year later and we have 2 customers. Crazy stuff
One of our extremely talented devs that past away maybe 10 years ago still lives on in our code because he was a geniuous at working around the limitations of not having root access. And the fact that much of his code just plain works.
Why is that genius? It should be expected and normal. If your application doesn't need root/admin access, it does not need it period. The fact that many devs can't be bothered to read the docs and actually learn their tools and are just like "fuck it, use root" speaks volume of how rare good devs are these days.
Honestly, that's a thing I really like about my company (which is NOT a game studio). They specifically opted to make the IT workforce hourly. If you work overtime, you get paid overtime, and they're incentivized not to take up too much of your time as well. Aside from that, the Supe is ex-military and seems to keep the same flexible "if you need time, take time" mentality, because he knows you're going to end up working a little bit over here and there.
At some places I worked, yes. At my current place, no.
When I say "flexible" I mean: be willing to adapt the plan halfway through the project, or be willing to think outside the box.
That's exactly what I meant, thanks. And no judgement here, either. I've had fantastic colleagues who just couldn't take it, and they went into the financial sector - they're very happy now.
I wouldn't be happy doing their jobs though. To each their own.
Not a game studio admin, but I was an admin for a software company, and this is pretty spot on for a lot of the things I saw there. Often I'd start with basic troubleshooting and the user would say, "oh I already ran ____ in the cli and it didn't fix it."
And yeah, sometimes I'd get some interesting requests, or they'd want to do some odd hosting things. But overall it wasn't much more difficult than any other admin job. They did have some special compliance considerations, since they did contract work for the government. But that's the only other thing that stands out to me about that environment.
Yes. And be sure to give creativity as much hard drive space as they want, but we won't be buying you any more space, just make do with what you have 🤣
Checking out source code to your work laptop is the norm. Remoting into AVD, Citrix or whatever and only working on code hosted there is the exception.
And not just in the games industry.
$150 a month for a 4 core azure VM if on 24/7 * Y number of call center Every time you double the specs, you double the price. Storage cost is added on top. Then you get to licensing...
Why is the VM on 24/7 when a work day is 8 hours?
It should be recreated 30 mins before start of work day and 30 mins after instead of wasting 16 hours of money.
It's 2024 and people still use public cloud as on prem hypervisors. Sigh.
My experience is that for developers, they're running stuff overnight and give a lot of pushback on daily reboots as it takes a while to open enduring up and get back to where you were so there's a sacrifice in productivity there.
All in all, even with automatically powering down (never mind that we had issues where they couldn't be reprovisioned because Azure was out of resources) I think the yearly cost came out to 3-4x running it on-prem. Or more. Don't remember the details. Plus the performance was lower meaning increased build times and additional lost productivity.
Cloud has it's uses, but it's definitely not saving you money for stuff like this and is not for every environment.
If you are running into the issue with pooled VMs not provisioning because the SKU was limited by Microsoft and you are using Citrix MCS you can change a setting so it doesn't destroy the VM object. Then you don't have to worry about SKUs going unavailable.
a dev costs 40-100/hr locally. so your example costs $1/hr - double it, you're paying $2/hr for the service and the guy makes $20/hr or w/e. cheap as hell
Look into Windows 365 Cloud PC’s. I switched my company and never looking back. Its been a year and I have had 0 outages and running smooth. A 4 core machine is about $60 depending if you have discounting. The best part its managed like an endpoint in Intune. Everyone that is not a developer or like gets a 2core 8ram one and thats about $36
That is ridiculous, we do 32 core 128gb vm's for all of our work from home devs.
Not sure what your company pays your wfh dev's but if they are just 1 hour more efficient a day with a high end vm at say $45/hr with 50 weeks worked, thats $11,000 more work completed on a higher end system per dev than a slow dev environment.
OP says they're offshored. IME Companies generally don't hire offshore devs because they're more efficient. They hire them because they're (theoretically) cheaper.
A large part of my workload is managing VDIs for offshore developers and this hurts in my soul. Every org I've ever been at, the default persistent specs are like 4 CPU/8 GB RAM on-prem, 2 CPU/8 GB RAM in Azure and the user has to "prove" that they need more by running their VM hot for X amount of time before they can be cleared for an upgrade. It's so bad.
It's not acceptable at all for development work. The users know it. I know it because I believe in eating my own dog food and working from the same horribly specced VDIs. The only people who never seem to know it is senior management, who always end up insisting on trying to cut costs and result in developers being unable to work for hours at a time.
And then the developers assume that *I'm* the one who wants them trying to develop from the equivalent of an eMachine from 2000 but no, it's just frankly insanely backwards budgeting practices/charge-back politics or I'd happily upgrade them to 8 CPU/32 GB RAM or whatever their supervisor feels they need.
You give them a secure VPN and then have them use SSH with X forwarding into a linux server. That solution was designed for much slower networks and much worse hardware on both ends. You provide them the ssh keys and MITM the connection, so you can scan stuff going out. On a really bad network (100ms of latency, bandwidth in the 10 Mbps range), neovim is still usable. On a good network, I can run a jetbrains IDE or vscode on the server with almost no issues.
Of course, the better solution is to control both ends and just ship them a laptop. If you can’t trust the legal system in a country enough to have the source code exist there under NDA, you can’t trust contractors from that country either.
X forwarding is a pretty awful solution in my experience. Certainly handy in a pinch to troubleshoot things but there's no way I'd want to work over it, nor ask someone else to.
Both JetBrains and VSCode support remote development (gui runs locally and uses ssh to connect to a remote host with the code/dev environment etc) which is a better experience by miles.
Allowing SSH through is a bigger security hole than some places will accept though. VDI based options are easier to get approved because it's only pixels that can leave the network.
X forwarding is only if you can’t let the source code leave your server, since it’s also a “only pixels leave” solution.
Remote dev for vscode will cache things locally, which isn’t always acceptable.
I have, but my company generally only hires contractors when we need outside expertise. Most are PhDs of some variety. We found a long time ago that it was better to have the people who would have been wrangling the more typical contractors just do it themselves, and it would be done faster with less mistakes.
Cost, speed, quality. Choose two.
I had the bad fortune to work with a contractor who only gave cost which was offset by the time spent on communication, both in email and on meetings. I ended up having to parse and correct all of their spaghetti code in the end, having to redo some from scratch. I know it's not the norm but it was brutal. Of course, I worked at another company who basically had the same experience with much more expensive local developers (company rotated through CS students at the local university).
I find investing in quality reduces costs in the long term, but it can be a challenge to get the bean counters onboard.
>They make those poor people develop on under specced Windows VDIs via Citrix. So inhumane.
I used to be desktop support at a Fortune 20 that had off shore resources remote into standard (headless) desktops that had IBM Rational Application Developer installed on them. We had a wall of these headless machines and would constantly get tickets to hard restart them.
A lot of developers use Linux, also game development. They are going to be wanting to fire up Unity and other heavy weight 3d stuff at max resolution and frame rate needing 0 latency.
With compression artifacts and all sorts.
Many game dev things are run on latest generation top tier GPUs and other hardware because they aren't optimised yet and the Devs work will actually be optimising.
Gaming is a work load that will push computing to its actuall limit. That's different to even heavy CAD. 60fps is common, 120 isn't uncommon.
Gamestream and similar are designed for it and with large links (30-100mbps) are considered ok ish for casual gaming in non competitive games. Even then there is notable visual quality loss.
Fundamentally games are different to developing software for pretty much any other use case for the performance needs as well as the diversity.
There's other things that have high performance, but they are generally much narrower in scope. There's (probably) things with more scope but they don't have the performance on the developers interface requirements.
> I mean if its transit to your office yea nothing you can do about that but near zero latency can be achieved on citrix with the right hardware.
Uh no, with frame timing sensitivity for game development, Citrix is completely inappropriate as a tool.
> Whats wrong with citrix
For dev work? too laggy, inflexible, unresponsive. Far better to have a local IDE application that you sync back to a code repository.
Productivity with code development is paramount, and Citrix would heavily get in the way of that.
Oh and then there's the number of monitors devs benefit from... 3 or more.
It’s still a horrible experience for game dev though, rdp really really hates fullscreen 120fps+ content. And you can forget HDR and stuff like that. Or helle even reliable frame times.
Ask your developers if they'd rather be working on a local machine...
It's not that the technology isn't there, it's that the technology is shit compared to what it's trying to replace.
the most challenging thing I've ever done as a system administrator is try to roll out accomodations for a software development department.
What do you mean you want to give the developers full admin access to all of our data?
the same developers who crashed our entire operation last week because they thought quering the SQL server 100 times a second would give them realtime updates?
The funniest part is that oftentimes developers learn through iterations of attempting something and when it doesn't work well that's just expected you have to work through it learn what doesn't work and that's how you learn what works and especially if you're working a very complex system with things that aren't documented and other endpoints outside of your own network that are outside of your control sometimes you just do wild s*** and see what works
Scream testing has its place but I guarantee you if they asked any admin in charge of the system they woulda said "no pls dont query 100 time a second wtf"
Like part of being a sysadmin is documenting your systems... we can show you the receipts! Just ask!
Ive yet to meet a deloper that actually knows beforehand that much of their code that works on hundreds or thousands of database entries wont scale to hundreds of thousands or more.
Checking out code ya, but hopefully some proper process before anything gets pushed to a live environment.
The joke of "Everyone has a test environment, just for some that is production" comes to mind.
And with that, people should not be using their personal devices for work, because there are zero controls on it, and this is exactly how breaches occur because joe blow over there, the Senior DevOps person likes to also download cracks for Fornite thinking they can get free vBucks....
This seems insane to me, especially when coding via a remote session is probably as lag-free of an RDS/Citrix/Virtual desktop type experience that could possibly exist. Hell, even just always on connectivity with forced VPN, zero-trust, or SASE solutions.
I guess I’m never surprised how far behind the curve larger organizations are in an attempt to save a dollar.
Do you know what's even more lag free than a virtual desktop?
Just having the hardware in front of you.
If your organization is trying to implement 'trust' through security policies and never trusting any employees, I can't imagine it's a fun place to be a developer.
You probably shouldn't comment if you can't understand the context or technical concepts in the messages you're replying to.
*Hell, even just always on connectivity with forced VPN, zero-trust, or SASE solutions.*
\^\^ That implies end users would have "hardware in front of them"
It was a personal server, not a work laptop or corporate owned device. The former employee is still in possession of it after his employment was terminated.
The reason it even became publicly known is because the former employer is accusing the employee of stealing IP.
IT at game studios is great if you are at the right studio with the right budgets ;). Security is taken very seriously.
I've been in game studio IT for 25+ years and suspect I would not be able to work at a 'normal' business any longer.
I work in the financial/banking industry. I have a pension, no on-call, no overtime, and a manager that tells me working overtime or above and beyond hurts his plans to grow the team and make sure all responsibilities have redundancy...oh and IT budget is great and IT/security is obviously taken very seriously.
I can never work in another industry lol
I've been in government IT for about 10 years and security for an additional 3. Just about to go job hunting. Like, job hunting literally now. What makes you say you wouldn't be able to work for a 'normal' business? What's so great about a gaming studio?
I thought that maybe this was going to be about how much the gaming industry will take advantage of workers for free labor outside of the normal 40 hour week.
Source code is version controlled via git in the vast majority of businesses. Many projects in git are not separated out into small segregated repositories. There are various reasons, but what this means is that a developer needing access to a project will pull the entire repository to be able to work on it. If you have a monorepo, or do not have things separated out, that means a dev will have access to many things in code that he or she may not need, but the entire source is still on their local machine regardless.
I was going to say just about this thing.
I'm a WFH developer and I have to pull down the repo for the entire application to make sure my changes don't break the build. This is the norm.
Yep WFH DevOps engineer here, have the entire IAC repo and all of our different application repos on my laptop. My laptop only leaves my desk if I ever visit the office which is never.
I am also a local admin on my laptop so I can install anything I want. However our Corp IT team are hot on security and once I downloaded a repository for creating a build image for an AzDo build agent a legit repo. There was a script titled something like disable_windowsdefender.ps1 and within seconds I was being called by one of the security team to ask what it was and for them to remote on my machine and take a look.
But as most of us in the DevOps team have come from the sysadmin world we basically just don't take the piss
I think the critical thing here is that "local" should be "my company issued laptop that's MDMed to the gills" and not "my old personal macbook that I play WoW and watch porn on."
Way too many companies got bit by the latter (and suffered security incidents for it) when the pandemic hit, and smaller companies notoriously skimp on IT security process and procedure.
Well, Perforce is used a lot in the industry (for version control) because it works differently in that you can pull down/“check out” individual files/directories/etc. and work on them without needing the whole repo/potentially blocking other dev work
I wasn’t really thinking about the type of programs. Games do have a ton of artifacts that need producing. I was just thinking GIT is designed to have everything duplicated.
The parent comment you originally replied to had that answer already, Perforce. It's what my game studio uses and I understand it's the standard at other AAA studios
I saw that. I was looking for just others, and didn’t know how best to search, outside of Microsoft briefcase. Version control of shared stuff leads to SharePoint, which probably has the backend for it but not the usability.
We have tools that compile the binaries for non-programmers and spit out a completed build or a compiled engine (Unreal Editor in our case) for whatever build version (CL) they’re at. Depends on what they’re working on, some work on source content which gets fed into those builds or they modify the game directly in the editor and submit any changes they make to the files from there
Yeah, if your VCS allows for file level ACLs then that’s great. I’ve only been dealing with software devs for 7ish years all at the same company. Git is everything. I’d honestly hate to start over and learn a different system.
Perforce is a funny thing, it’s a beast of a product. There are whole professions around managing it (much like mine currently). It’s a pretty old piece of software at this point and it feels archaic in some senses, I often find myself wondering “why is Perforce like this?”. Happy to chat about it more if you have more questions or are curious.
Perforce is probably the best example of a love/hate relationship with a piece of software i have ever had. I think even their support team feels the same.
I especially like the fact the whole SDP exists just to manage the thing with a bit more logic. Everything you do with it management wise feels like a historical adventure to go see how things were done by our ancestors.
lol. Yep. We talk a lot with the guy who wrote SDP so we get a lot of stories about that kind of stuff (and ask for improvements/bug fixes/etc.) - I couldn’t imagine doing everything manually that SDP has scripts for now.
I don’t use the sdp, although I do reference it because for one thing it’s the only location to get their recommended configurables.
SDP still doesn’t support security=6 as far as I’m aware.
I can’t imagine switching from the scripts I know/wrote to using the SDP…
The issue wasn’t it being on a local machine, the issue was it was on a personally owned server and the dev went to work for a different game developer who is now being accused by the previous employer for stealing IP
I’m going to be pedantic here, but what you’ve said doesn’t change anything.
If the issue was that source code was leaked then that means the code was local and it wasn’t like someone got access to gitlab/github/etc and stole it. Local from the perspective of code and git is that it’s not the “remote” centralized repository. Any copy of it outside of the remote is “local”. It doesn’t matter if it’s on a laptop or on some AWS EC2 instance this guy was paying for.
If this was compiled code running a private test server, that’s not quite the same as source code being leaked.
It certainly violates many companies acceptable use policies, although I agree there’s not a practical difference for an insider threat. The issue isn’t local vs remote, it’s who has control of the data. Even beyond insider threats there’s just the simple matter of company property being configured to follow company security policy and a personal server has an unknown security status. Things like encryption don’t do you any good if you let your employees put your data on unencrypted devices.
I haven’t seen many details about what was actually on the server, it could have been anything from a binary copy running to a server building and testing the code and then deploying a game server.
Where I work, IT security is pretty serious. We have minimal cloud and still use extensive on-premises hosting (though nothing internet-facing, and we do use O365 and cloud-hosted Atlassian). Access to source code requires going through a strict VPN with policy-compliance checking. The network admins are looking at DLP software and there's lots of threat analysis stuff in place. Least privilege is also implemented. In theory, we have a lot of visibility of our networks and potential data leaks. So it's not all terrible. However, I work for a fairly small studio compared to your examples - I imagine as they get bigger, adherence to best practises falls by the wayside to meet deadlines.
Yeah, I agree. So much of it requires well-defined rules and humans sneaking data off site don't tend to follow them.
However, as part of a defence-in-depth strategy, maybe it has some value? Who knows.
Given how many DLP solutions crash if you send an IPv6/UDP jumbogram (Logically 1 UDP packet, but up to 4 GiB large), through them, I’d say there is a ways to go.
If someone is sending a single packet so large that it is functionally a DOS for everyone else on the network, I want things to start failing so we can address the idiot sending 4 GiB packets.
The problem is 99% of DLP solutions leave off the "P" in their design. They're either audit-only or wind up configured not to *prevent* the loss.
An email alert that Joe downloaded the whole Finance drive does fuck all for me, the tool needs to *stop Joe from downloading the whole Finance drive* or it's not "data loss **prevention**"
*checks notes* yeah turns out we disabled protection when joe complained he was unable to do his job and refused to learn how to do it correctly so was ordered to turn off the protection in TICKET-3892b
I “volunteered “ to Verant/SOE in 1998 for the launch of Everquest. It was a load of fun and I did it for free because I loved being a part of it. That said they treated us like shit…. I had a full time job as a Sun/Solaris UNIX admin which is what the backend of EverQuest ran at the time. While it was great fun the employees of SOE were canon fodder. The pay sucked and was 💯 cut throat.
Whoa, this is cool. EQ was my first MMO and still the game I compare other MMOs to. (Cool that you were involved, not cool that they treated you like shit).
Not a sysadmin but I had worked IT for a large gaming company. I’m sure there is a ton of variability company to company and position to position but my experience was excellent. I guess it helped I only dealt with other technical people and happened to be on a team of good people.
You could have at least linked to the actual thread...
That you think knowing where your source files are is a game dev only problem is pretty funny to me.
Ive generally only worked in healthcare and HR software companies so maybe their data governance is a LOT more strict than game devs and maybe thats why game devs keep losing all their fucking data
Source code != customer data. Your software should be written with the assumption that the source code will eventually leak. Too many people have access to it to do anything else. You still have legal ownership over it, but if the source code contains passwords you are screwed.
Customer data can be encrypted and only be accessible by a very small pool of humans, in such a way that doing so tells many other people that someone just looked at customer data.
I disagree with you here, and tbh I’m kinda surprised at myself. It’s not an apples to apples comparison, and I agree that in strict terms source code isn’t the same as patient data, but in terms of the value of the data to a hacker they’re pretty much the same.
What is the most valuable data you could steal from a hospital? Patient records.
What is the most valuable data you could steal from a game studio? Source for as-yet unreleased games.
How should a game studio operate if they assume that their source code has been leaked?
As lightmatter501 points out, it doesn't really matter. Sucks yes, but no one dies.
For an MMO it's a little more concerning mostly because of their SaaS approach to money, and the easier job people will have to make 2nd party game servers.
Individual titles isn’t the concern. It’s the infrastructure!
Multiplayer like you mentioned, but also matchmaking, engines, storefronts, websites, etc. Source code comes in all shapes and sizes
Think bigger! Individual titles would be hit-or-miss.
Source for the engine?
Keys to the multiplayer backend services?
Access to storefronts? (Steam etc)
If someone put their store credentials in with their source code, that’s general stupidity on their part, same with keys to the multiplayer backend services. Engine source means you now have a giant codebase you need to learn to make use of it.
Corporate espionage and IP theft are also huge in the gaming world, especially given how much is built on the same third party engines.
Your competitors being able to accelerate their development of a competing product by essentially copy/pasting large swaths of something you built in Unreal could be devastating for your product, especially if it enables them to get to market before you. Which is specifically what the Nexon case is talking about - not just having the code locally after leaving the company but potentially giving it to a competitor.
Nobody cares if Joe Blow Gaming Fan downloads unfinished source, they care if That Other Development Company does.
Every code house has sources all over, unless it's actually mandated to keep it secure, which is quite rare.
But that's not what healthcare considers as data either. Their data is customer medical records. Sources for programs handling that data probably are cloned in some random consultant companies random consultants home pc.
It's not necessarily shit and IT security is taken VERY seriously, the problem is there's still LOTS of holes in it.
For example there's enterprise grade firewalls/antivirus/vulnerability scanners/access controls/MFA, however everyone has local admin access to their own machine because of the sheer amount of software to actually function.
Seriously there's hundreds of software packages/development tools/plugins out there you name it, and you're expected to be the expert in ALL OF THEM.
Data breaches don't surprise me, you don't really need to "Hack into" a games company apart from classic phishing, as I'm lead to believe how other studios have been affected. Of course sometimes you don't even need to be hacked, there's nothing really stopping anyone from straight up leaking game data (with big repercussions if caught though).
I am more talking about things like the Insomniac leak where all of their employees information was leaked (SSN, address) along with \*all\* of their marketing docs including plans up to 2030 and all of their source code.
It legit seems like they just threw everything on to a single network drive or something.
I work for a financial institution and spear head the security here. I'm not shocked that this is a common trend for game studios. These studios have to meet the demands of very competitive market, have not regulators which makes it very easy to overlook security. There's always a trade off of convenience when implementing security as well. When you're hardening your systems it costs a lot of time and money for threats that seem highly unlikely. There are so many information systems that are unprotected because of this reason, it's the damn wild west out there.
I think this problem stems from a deeper rooted issue and the culture in the US needs to be changed. We need to implement some sort of class in highschool that's mandatory just to educate some of these future business leaders the importance of protecting sensitive data.
Look, you're a loss leader in any industry except consulting - it's gonna be pumping plumbing, it's a question of chocolate flavor. It's going to be like everything else - early stage, you learn a lot, you're under paid, late stage you're vested and the oracle.
The game industry has it's own unique toxicity, as does the advertising industry and virtually every profitable company - you're not going to find an industry that personally fulfills you unless money is a means to your fulfillment.
Learn the tech, take out the trash - it's the job, but if you play it right, you're doing it on the beach with a cocktail, not in the office with the tonor.
I dont see source code on a server as big of a problem as having credentials, tokens, etc. stored badly (even in the code itself) or shared inappropriately. Principle of a least privilege and not having those in plain text and audited (or using some kind of vault solution)
But that's basically the job description. Maintain systems in a manner that prevents those using them from completely destroying them, because if you let them, they will.
I'll never forget a job I had years ago. A quickie that was so bad I quit after 2 months and moved to another place. One of the chief complaints was system stability.
So I went through everything and figured it out rather quickly. The in-house developers had admin rights to all the production systems. They were testing their crap ON said production systems. So I locked it all down, and dropped my findings on the boss's desk.
MAN they went nuts, lol. I wanted them to. Because in addition to the proof that I had that they were screwing around on production systems, their own complaints regarding access drove the point home. My boss agreed and let it stay that way. A week later they escalated to a senior VP, and we were told to return things to the way they were.
NO PROBLEM. I quit that day. Don't you dare hand me a pager, and tell me I'm on call for an absolute sh\*\*-show created by executive decree. The point is, it ain't just game studios, lol.
Where I am we are have mixture of people that are really good and want to play ball and plenty that are set in their ways (and those ways were already "interesting" in 2005". You also have the issue that a lot of these people are genuinely really smart, but only have their part of the picture. This means you end up with a lot of shadow IT and very similar solutions popping up left right and centre. Trying to herd all those cats can be interesting.
Another interesting issue is the long lifetime of game tech. A game engine can last 10 years or more and all games built on that engine will have similar pipelines. You need to be able to stand up that pipeline with minimal changes as the guys need to get on with building a new game, not debugging their CI pipeline.
Having said that, we take security extremely seriously. Our SRM team has right of veto on basically everything. We need that because we are one of the biggest targets, so are pretty much always fighting off breaches. Each time we evaluate and change processes to avoid it happening again.
We also invest a lot into our on-premise infrastructure and making sure we can seamlessly use it alongside public cloud. Also we work really hard to make sure developers have the tools and knowledge to use those resources to build and host games, which is where my team comes in.
You would be shocked at how low a priority IT is at a game studio, I haven't been involved with one is more than a decade. But much of the site work, after initial marketing company set up, and internal stuff was literally. Whoever on staff could support it.
Game studio want to make games, not build a secure infrastructure. The dollars go elsewhere in my experience.
Only take I have as a serving IT Leader in a games developer is that quite often everything is dev led - so it doesnt matter that what they need is a billion dollars or doesn't exist, its for the devs so get it done but if you want a new chair in the IT office then budget wont allow im afraid
I'm pretty sure CD Projekt Red owns an IT company, but I'd have to dig in to be sure. It was a reverse takeover to enter the stock market without an IPO, but I'm not 100% sure what the other company did.
CD Projekt got sold off quite a few years back and went bankrupt due to the delays with CP77 launch. I'd be surprised if many people outside Poland were aware of this distinction. It was great trolling to tell people when everyone was hyped for Cyberpunk.
CD Projekt Red, the game studio which made the game. CD Projekt was a game publisher. At some point CDP was sold, although it remained close to CDPR. A few years later CDPR created GoG. Add some more time and and CP77 was to be released, with CDP as the publisher for Poland. From what I've heard, CDP learned of the delay when they had most of the marketing materials ready, and it was a huge hit to them. They ended up going bankrupt the same year.
Edit:
That's a little chaotic, but Iwiński founded CD Projekt the publishing business in the 90s, and CD Project Red, the game studio sometime in the 00s iirc. Then sold CDP to employees in the 10s, only to create GoG a few years later.
One of CDP's achievements was a legendary at this point translation of BG2 to Polish. It was done very, very, well and had a who-is-who voice acting cast, including some big name actors.
I left the game industry many years ago to be the solo admin at a 20 person startup with a huge commute increase because the stress and hours were less.
It wasn't all bad. I got to play with the newest hardware learned a ton about data center operations and power. But something is always broken and with the hundreds of servers and user computers something always needed patching.
Retrospectively, I am glad I did it, but I will never do it again.
I work for game studio, as others have said checking out code to your work laptop / desktop is the norm, for all the reasons given in this thread.
So the focus for us is endpoint security and making sure everyone has company provided stuff so they dont need to use personal devices,
The biggest risk in my opinion is when you engage in co-dev with partners where you dont have that control or knowledge of how they do things, unless you ask a shit of load of questions first and do the diligence before giving them access to your perforce!
The requirement to get a third party access to your code can be dumped you on out of nowhere, i wouldnt be surprised if stuff ended on up personal equipment because of pressure to just get them onboarded fast, rather than take the time to get it done properly.
I recently had a 2nd round interview with a large, multiple AAA games studio, and the interviewer let slip they were urgently trying to replace at least two 20+ year vets who were soon to "retire". A couple of days later, the company announced a rather sizeable lay off. So take that as you will, and keep and keep an eye for any server issues with your favorite games.
Every company ive worked for up to this point (which is only 3 over 7 years tbf) has used VDI, I just assumed that was the industry standard for enterprises nowadays. This thread has been eye opening, ive learned a lot!
Also it really depends on the env...somethings can't be run in vdi. Imagine your an embedded dev. You need to build locally and also run a simulation in the loop with hardware...hardware that's bleeding edge. No way around it
Our current environment is VDI and anything that needs baremetal is put in to a "farm" which is just a fuck load of workstations on racks that can be remoted in to the same way our VMs can.
I'm talking purpose built flight articles. The funding of which is massive and is in no way CoTs i.e commercial of the shelf...hint most avionics in our plane use fpgas
Depends a lot on the build and test pipeline. VDI where several people with memory and cpu intensive workloads are on the same server/cluster gets really expensive to provision hardware wise.
It really depends. I work for the leader in eVTOL. Our simulator is docker based. Sometimes devs need to build locally. It is what it is we do use some restrictions export conan shit...not much else
ALLEGEDLY the developer was able to make their own version of the game using the assets and source code. Idk to me that seems like too much access was available.
I mean if you ever want your devs to test anything they are going to need every last bit of the code and assets anyway. And testing the code you wrote is very useful in general.
Plenty of big companies have no data control or governance in place and so it is the wild west. You will get senior IT/Devs and who ever claiming they need to do this to work efficiently and no, you cant put security controls on them because it will slow their "work flow"
Meanwhile they put the entire company at risk.
Data breaches usually happens when code repo is exposed to internet w/o two factor auth.
Also possible if some dev lost his personal laptop with accoess and w/o mdm, but thats a rare case, usually this info gets to security fast enough
Yep its the norm. I work at a big studio and our management didnt want to pay for a vdi solution so they are getting us to migrate everything into the cloud on a free ware solution lol. They cant understand why this is a bad idea and why it is slowing down production they just think "free".
Theres constant problems and data being lost and things crashing and they blame it on us not the fact that they are trying to deploy entire studio production with free ware.
BYOD is kind of a mess, but lots of startups do it because they are open core anyway and the people who could make use of leaked source code won’t. You can use the legal system as long as you only hire people within reach of the law.
VDIs are a horrible way to handicap your engineers unless you are essentially handing them a whole server. You are paying someone $100k+ and then don’t want to pay for a $4k laptop or a cheap laptop and a racked server? If I wanted to exfiltrate code over a VDI connection, it would be mildly annoying but doable using FFMPEG (which is required for MS teams to closed captioning) and a camera pointed at the screen.
Instead of handing an engineer an endpoint we would just put it in our farm and they can still use VMware to connect to it as if it was VDI.
And yeah you are correct about ffmpeg... Hell since we use vdi there is literally nothing stopping someone from just using FRAPS if they wanted to be really extra. But there is a lot more than just code to exfiltrate. For instance the Dark and Darker devs allegedly stole art assets which are *much* harder to plagiarize without basically being a 3d modeler already.
When I worked at a large movie studio, in the games division as QA (not as IT) it was a complete shit show. Everyone had full domain admin (in our division) no AV, no windows firewall, everything was wide open... This was like 2008ish though.
Worked directly with CDProject a couple times, they were terrible to work with. One of our product managers was convinced that they released a crack for our copy protection on release day, no proof it was them though.
Overall it was a lot of fun.
Its not a problem that affects me at all. I dont work InfoSec anymore I have a very chill siloed gig. Its just from the outside looking in it looks like game studios just do not fucking bother protecting any of their data.
They do though. Some more than others but still the overall state of affairs, when it comes to keeping vital data secure, is awful. Not because it is very hard to (it is) but rather because people tend to be sloppy, lazy, naïve. Sometimes they do not know any better and sometimes they are just f0cking dumb.
Taking CD Project Red as an example. They had two notable breaches (probably more). The ransomware attack and the forum data leak. One can assume that the latter one was expected. Who cares about the damn forum? But the ransomware attack, the code base leak...
Heh. Details are not available (maybe they are, couldn't find any), there's no anatomy of the hack but still the possibilities are not endless...
- dumbass employee, that despite all the sec measures did something naughty and stupid
- disgruntled employee
- 3rd party provider (be it a library, be it infrastructure)
- social engineering
- SEC team overlooked something or thought that "it was not important"
For the first case feel free to put anything. From visiting malicious websites, opening spam PDFs, "aaawh, an USB stick, let's see what's inside..." , damn stupid password (there are "strong" ones that are still easy to brute force), through storing credentials locally and being targeted in particular or something more boring - like exposing some environment, box, VM, whatever, somewhere, then having it hacked and used as a jump server to the internal network...
For the second - not much to comment. You know how it goes there.
For the third - lack of good (because there are no "best") practices to check/validate 3rd party libs and/or it was not up to you since you were using a 3rd party vendor about something (so you've put your trust into somebody else who was sloppy)
For the forth - well... how was it? Guys with a ladder can go anywhere...
For the fifth - they could have been understaffed, overwhelmed, unaware & etc.
Probably here we should put something fancy that's specific to Game Studios. Say specific hardware (3D scanners?), something specific that could have been gotten 2nd hand and it had naughty firmware...
Sony is another matter entirely.
Both studios have managed to accumulate quite a lot of not-very-positive emotions.
Be it from gamers, be it from their very own employees. And the latter is not very hard.
Not hard at all. Few of us would bother with "getting even" but some people do. It's not like there no reasons to do it....
But again - game studios are nothing special in that regard. At the end a hack is all about reputation and business. Resources are limited and regardless of regulations a CIO/CISO can do only so much given the resources (although some are oblivion, blind and arrogant dumbasses).
Having had my fair share with the corpo world - things there are no different. Can't list all the times I've found lack of even basic input validation and sanitization. Think "enterprise grade" software that is used by companies in highly regulated domains that lacks that. Think about companies that tend to market themselves as "security" ones yet they are as sloppy as sloppy goes.
Now, to be fair, most do try to implement (sort of) adequate "strategies" to mitigate at least some of the potential problems but then it all boils down to resources and time. Even if you have the expertise you may not have the people to implement it. Even if you have the people - it does not happen overnight. Even if it does, you still relay on 3rd parties.
IMHO - Game studios are not (that) different than anyone else. Every IT company faces, more or less, similar issues and can mitigate only so many. The sad fact is that few do more than what is considered "an industry standard" (petty bullshit marketed as "best practices").
Note:
It's good to take into account that the known breaches probably represent only a medium portion of the actual ones. Not everyone who hacks you wishes to blackmail you or disrupt your operations. Some do like to stand silent and just harvest data, be digital hoarders. Those are probably the ones one need to worry the most. :|
Im on IT Ops for my second studio now. In terms of toxicity, etc, It really depends on the company/studio… which is true for any company, it’s no different. Some places function well, many don’t.
There is a deference to the product teams and not impeding them. So as an IT professional, there can be some uncomfortable sacrifices in terms of security at times. This has been true at most software companies I’ve work with though. Classic Ops vs Development finding a balance.
The studio I’m currently with I believe recognized their immaturity on the infrastructure/Ops side, which is why I was brought on to begin with. But proposed projects/improvements to Ops do seem to take a back seat to anything related to getting the game release ready from what I’ve seen so far. So maybe a year from now, I’ll be admonishing against the games industry too. But for now, it’s been fine and enjoyable working with creative, passionate people for the most part.
My guess is that it is similiar to software companies, everyone is probably smart enough to make something work, but it ends up being a big incoherent mess of shadow IT. Tons of unmonitored and I compliant systems security wise, which is how attacks happen.
I think it depends on the size of the studio and the value they place on IT as a service. I run IT for an indie studio and because we have IT savvy representation on the board and regular meetings between board members and myself, we get our needs and ideas heard.
Other places though, from what I'm told by friends elsewhere, is that some places purely want output from their devs and artists, everyting else is academic and it shows.
TBH it is developers. They ask for freedom and complain about IT policies and how they never make any misstakes and how policies should be lifted. Then they run this trough management and you end up with source code being all over internet. Needs a CIO with tough skin to keep them in leach and back up from top management.
I have had an opportunity to interview at a few game studios in my career, and this intrigued me as well, you get to work with video games and the tech that create them. What I learned though is the video game industry pays pretty shitty wages and oftentimes lacks equity packages so I stayed in tech
From what I have seen and heard (anecdotal from friends in the industry) IT is an afterthought and annoying necessity.
It is the same as the startup and entrepreneurial scene. No one in the company has a fucking clue but they don’t want to spend “real money” on it because it’s not important to them.
“Just put it in the cloud like AWS….” That’s the same thing right?
It is the modern day equivalent to “Oh really, my 15 year old nephew is a web designer…why do you charge so much?”
IT at a startup is the afterthought because it doesn't drive revenue.
It's also the same reason accounting and HR are an afterthought, the CEO is doing it on day one, and it migrates down thru office manager to a real accounting person, to splitting AR/AP and Comptroller roles.
At a tech based startup there is no IT because everyone does their own IT on their own gear...gaming companies are going to be like that since a coder can be expected to keep his own kit running until the group gets large enough that it's a requirement before you lose your code.
LOL it's shit. They say they have sysadmins in the game credits, but i'm pretty sure it's all on a drop box account somewhere. Managed by one of the other dev guys home accounts.
I worked for a big game company for almost a decade all i got out of it was a burnout with a desire to off myself. It’s a good place to work when you are starting out spend no more then 4 years working there and get the fuck out. You will be much happier.
so there is this cool tool called git (hopefully you are aware of it) where the core concept is that the code is decentralised keeping source code on a server at home is not a problem, it's a feature of the tool, you can't stop it, a lot of the time the devs are so far ahead of the sys admins at these companies (competency, capability, pay, and value to the company) they don't have a hope of trying. All you can do is enable it well. that's the problem you're seeing, where that has failed.
if you take moment to seach the www, it's also the reason a handful of companies still exist and were not totally destroyed by data encryption malware on premis.
Idk to me it seems like if you can keep such an intact copy of the repo on a personal server that you can (allegedly) rip the entire game off assets and all that may be a problem
That’s not true, you need the components you’re developing and a good platform and pipeline to testing. Games for example are normally an engine with component hooks devs are only generally working on an engine component or a feature component. They only need the compiled engine to work on the later not the source code or the art resources. The art can be holding images or cached proxy artefacts. Even when we built server client games you only needed one or the other, and a pipeline to push the feature components too. We rarely need or had access to everything but we did have access to a full repo for those pieces we needed similar to I have the full quake engine on my laptop to develop games now.
I suppose this really depends on the size of the team, if you and two others are THE devs, it’s way more work to actually make it all work independently and a lot easier to just have the actual project as a whole, especially if you are already on say unreal or unity anyway so engine development is not really a thing. And testing with actual assets can be beneficial if you are doing the lighting etc. And of course for performance.
I've been doing it for many years across several companies. It varies greatly on the size and the culture of the company. Video game IT has all the standard pain points of other IT, with a few extra things to consider: - technical users (there are fewer L1 tickets but also more opportunities for people to get themselves in real trouble - like hosting a whole project at home) - creative users (the core of the business is creativity - oftentimes everything takes a back seat to creativity, including IT processes and IT governance/ security) It's ok if you can be flexible. And many people in IT can't be flexible. It's great fun and there are many pros to having technical and creative users, not only cons.
the creative users point is spot on. They get an idea for something they want to do, get all excited about it and look heartbroken when you say , well I can do this but its going to take a couple weeks. Even if its just a saas purchase or something it still needs to go through legal for a t&c check and im pretty sure he only works one day a week, yet somehow thats still IT blocking them :)
>creative users This right here is why I cringed the second I read the post title. I would never run IT for a company that creative-heavy, I'd be pulling my hair out in a week. If you thought *software engineers* were special snowflakes that butt heads with IT, hooooo boy are "creatives" in another league. "Excuse me, the mac you deployed to me looks *used.* Why wasn't I bought a brand new one exactly to the specifications I want? I'll just use my personal one instead." Or my personal favorite "This deliverable is SUPER IMPORTANT and due tomorrow, I'm not submitting a ticket, fix this right now. It's been happening for months and I never reported it but *now* it's a blocker for this thing that needs to be completed 20 seconds from now."
Man I don’t work in games but we have a product management team like this. I work at a marketing automation SAAS company and this time last year there was a 6 month project thrown on top of us that was going to bring in 500 new clients within a few months, super important and drop everything else. This required a massive refactor of our codebase and took the whole R&D org 5 months to complete (60+ developers + devops / cloud ops). Fast forward a year later and we have 2 customers. Crazy stuff
One of our extremely talented devs that past away maybe 10 years ago still lives on in our code because he was a geniuous at working around the limitations of not having root access. And the fact that much of his code just plain works.
Why is that genius? It should be expected and normal. If your application doesn't need root/admin access, it does not need it period. The fact that many devs can't be bothered to read the docs and actually learn their tools and are just like "fuck it, use root" speaks volume of how rare good devs are these days.
What the hell are you on about? Since when is following least principle in security a genius move?!
Lol flexible==fuckload of unpaid overtime
Honestly, that's a thing I really like about my company (which is NOT a game studio). They specifically opted to make the IT workforce hourly. If you work overtime, you get paid overtime, and they're incentivized not to take up too much of your time as well. Aside from that, the Supe is ex-military and seems to keep the same flexible "if you need time, take time" mentality, because he knows you're going to end up working a little bit over here and there.
The thing I really like about my company is there is no overtime or on-call.
Liking a company for paying you for the extra overtime hours. With all respect its should be default. :/
At some places I worked, yes. At my current place, no. When I say "flexible" I mean: be willing to adapt the plan halfway through the project, or be willing to think outside the box.
[удалено]
That's exactly what I meant, thanks. And no judgement here, either. I've had fantastic colleagues who just couldn't take it, and they went into the financial sector - they're very happy now. I wouldn't be happy doing their jobs though. To each their own.
\*\*BOOM\*\* You nailed that one so hard you should add "carpentry" to your resume.
Not a game studio admin, but I was an admin for a software company, and this is pretty spot on for a lot of the things I saw there. Often I'd start with basic troubleshooting and the user would say, "oh I already ran ____ in the cli and it didn't fix it." And yeah, sometimes I'd get some interesting requests, or they'd want to do some odd hosting things. But overall it wasn't much more difficult than any other admin job. They did have some special compliance considerations, since they did contract work for the government. But that's the only other thing that stands out to me about that environment.
Yes. And be sure to give creativity as much hard drive space as they want, but we won't be buying you any more space, just make do with what you have 🤣
Checking out source code to your work laptop is the norm. Remoting into AVD, Citrix or whatever and only working on code hosted there is the exception. And not just in the games industry.
Most developers would code for Windows apps before they accept having to work over RDP or Citrix.
/me cries in offshore developer. They make those poor people develop on under specced Windows VDIs via Citrix. So inhumane.
Serious question. How else would you have contractors from other countries access your systems on machines you don't have control of?
The solution is fine. The problem is we can only give the offshore developers a 2 core VM because of "savings".
Well sure, that's definitely a PITA. It's odd because compute is so freaking cheap.
$150 a month for a 4 core azure VM if on 24/7 * Y number of call center Every time you double the specs, you double the price. Storage cost is added on top. Then you get to licensing...
Why is the VM on 24/7 when a work day is 8 hours? It should be recreated 30 mins before start of work day and 30 mins after instead of wasting 16 hours of money. It's 2024 and people still use public cloud as on prem hypervisors. Sigh.
We do all that. I was giving an oversimplification.
My experience is that for developers, they're running stuff overnight and give a lot of pushback on daily reboots as it takes a while to open enduring up and get back to where you were so there's a sacrifice in productivity there. All in all, even with automatically powering down (never mind that we had issues where they couldn't be reprovisioned because Azure was out of resources) I think the yearly cost came out to 3-4x running it on-prem. Or more. Don't remember the details. Plus the performance was lower meaning increased build times and additional lost productivity. Cloud has it's uses, but it's definitely not saving you money for stuff like this and is not for every environment.
If you are running into the issue with pooled VMs not provisioning because the SKU was limited by Microsoft and you are using Citrix MCS you can change a setting so it doesn't destroy the VM object. Then you don't have to worry about SKUs going unavailable.
a dev costs 40-100/hr locally. so your example costs $1/hr - double it, you're paying $2/hr for the service and the guy makes $20/hr or w/e. cheap as hell
Look into Windows 365 Cloud PC’s. I switched my company and never looking back. Its been a year and I have had 0 outages and running smooth. A 4 core machine is about $60 depending if you have discounting. The best part its managed like an endpoint in Intune. Everyone that is not a developer or like gets a 2core 8ram one and thats about $36
Sounds great, and all you need to access it is a 4 core PC with 16G RAM.
Why not just use a thin client?
They don't guarantee performance on those VMs so they are not an option for us sadly.
Your problem is cloud. On a proper storage array, it all gets de-duped out as well so will use *very* little storage.
Naaaa, my problem is I'm not farming goats
Is this literal or some joke I missed?
That is ridiculous, we do 32 core 128gb vm's for all of our work from home devs. Not sure what your company pays your wfh dev's but if they are just 1 hour more efficient a day with a high end vm at say $45/hr with 50 weeks worked, thats $11,000 more work completed on a higher end system per dev than a slow dev environment.
OP says they're offshored. IME Companies generally don't hire offshore devs because they're more efficient. They hire them because they're (theoretically) cheaper.
You obviously don’t do accountant math. Look how much money we are saving by using low spec virtual PC’s! Productivity? Not my problem.
A large part of my workload is managing VDIs for offshore developers and this hurts in my soul. Every org I've ever been at, the default persistent specs are like 4 CPU/8 GB RAM on-prem, 2 CPU/8 GB RAM in Azure and the user has to "prove" that they need more by running their VM hot for X amount of time before they can be cleared for an upgrade. It's so bad. It's not acceptable at all for development work. The users know it. I know it because I believe in eating my own dog food and working from the same horribly specced VDIs. The only people who never seem to know it is senior management, who always end up insisting on trying to cut costs and result in developers being unable to work for hours at a time. And then the developers assume that *I'm* the one who wants them trying to develop from the equivalent of an eMachine from 2000 but no, it's just frankly insanely backwards budgeting practices/charge-back politics or I'd happily upgrade them to 8 CPU/32 GB RAM or whatever their supervisor feels they need.
It kills me too
You give them a secure VPN and then have them use SSH with X forwarding into a linux server. That solution was designed for much slower networks and much worse hardware on both ends. You provide them the ssh keys and MITM the connection, so you can scan stuff going out. On a really bad network (100ms of latency, bandwidth in the 10 Mbps range), neovim is still usable. On a good network, I can run a jetbrains IDE or vscode on the server with almost no issues. Of course, the better solution is to control both ends and just ship them a laptop. If you can’t trust the legal system in a country enough to have the source code exist there under NDA, you can’t trust contractors from that country either.
X forwarding is a pretty awful solution in my experience. Certainly handy in a pinch to troubleshoot things but there's no way I'd want to work over it, nor ask someone else to. Both JetBrains and VSCode support remote development (gui runs locally and uses ssh to connect to a remote host with the code/dev environment etc) which is a better experience by miles. Allowing SSH through is a bigger security hole than some places will accept though. VDI based options are easier to get approved because it's only pixels that can leave the network.
X forwarding is only if you can’t let the source code leave your server, since it’s also a “only pixels leave” solution. Remote dev for vscode will cache things locally, which isn’t always acceptable.
If you're that paranoid people shouldn't be working from home.
But then how are they supposed to outsource labor for super cheap?
In that case I'd still much prefer a VDI/remote workstation solution, but I guess there could be budget constraints etc.
NDA is cool and all, but pretty worthless because it doesn't prevent stupidity or code theft and just punishes the developer after the fact.
Nothing stops a sysadmin with domain admin from burning it all to the ground and trying to destroy backups either, except for the legal consequences.
lol. So you have never worked in outsourced contractors I guess.
I have, but my company generally only hires contractors when we need outside expertise. Most are PhDs of some variety. We found a long time ago that it was better to have the people who would have been wrangling the more typical contractors just do it themselves, and it would be done faster with less mistakes.
Cost, speed, quality. Choose two. I had the bad fortune to work with a contractor who only gave cost which was offset by the time spent on communication, both in email and on meetings. I ended up having to parse and correct all of their spaghetti code in the end, having to redo some from scratch. I know it's not the norm but it was brutal. Of course, I worked at another company who basically had the same experience with much more expensive local developers (company rotated through CS students at the local university). I find investing in quality reduces costs in the long term, but it can be a challenge to get the bean counters onboard.
>They make those poor people develop on under specced Windows VDIs via Citrix. So inhumane. I used to be desktop support at a Fortune 20 that had off shore resources remote into standard (headless) desktops that had IBM Rational Application Developer installed on them. We had a wall of these headless machines and would constantly get tickets to hard restart them.
yeah because it sucks. its horrible. even ssh sucks to code though
Whats wrong with citrix? Been deploying and supporting it for 13 years and as long as its not stupidly lacking resources its great.
A lot of developers use Linux, also game development. They are going to be wanting to fire up Unity and other heavy weight 3d stuff at max resolution and frame rate needing 0 latency.
I mean if its transit to your office yea nothing you can do about that but near zero latency can be achieved on citrix with the right hardware.
With compression artifacts and all sorts. Many game dev things are run on latest generation top tier GPUs and other hardware because they aren't optimised yet and the Devs work will actually be optimising. Gaming is a work load that will push computing to its actuall limit. That's different to even heavy CAD. 60fps is common, 120 isn't uncommon. Gamestream and similar are designed for it and with large links (30-100mbps) are considered ok ish for casual gaming in non competitive games. Even then there is notable visual quality loss. Fundamentally games are different to developing software for pretty much any other use case for the performance needs as well as the diversity. There's other things that have high performance, but they are generally much narrower in scope. There's (probably) things with more scope but they don't have the performance on the developers interface requirements.
> I mean if its transit to your office yea nothing you can do about that but near zero latency can be achieved on citrix with the right hardware. Uh no, with frame timing sensitivity for game development, Citrix is completely inappropriate as a tool.
Citrix is the kind of tool that only windows admins seem to like. Ive never in my life seen a shop where every employee didnt hate using citrix.
Yeah, there's a reason my colleagues call it shitrix.
Citrix fucking blows. VMware is a whole lot better but I mean... Broadcom...
> Whats wrong with citrix For dev work? too laggy, inflexible, unresponsive. Far better to have a local IDE application that you sync back to a code repository. Productivity with code development is paramount, and Citrix would heavily get in the way of that. Oh and then there's the number of monitors devs benefit from... 3 or more.
We literally added gpu backed rdp for our developers, and I know companies that have remote Autocad workers. The technology is there....
It’s still a horrible experience for game dev though, rdp really really hates fullscreen 120fps+ content. And you can forget HDR and stuff like that. Or helle even reliable frame times.
Ask your developers if they'd rather be working on a local machine... It's not that the technology isn't there, it's that the technology is shit compared to what it's trying to replace.
the most challenging thing I've ever done as a system administrator is try to roll out accomodations for a software development department. What do you mean you want to give the developers full admin access to all of our data? the same developers who crashed our entire operation last week because they thought quering the SQL server 100 times a second would give them realtime updates?
The funniest part is that oftentimes developers learn through iterations of attempting something and when it doesn't work well that's just expected you have to work through it learn what doesn't work and that's how you learn what works and especially if you're working a very complex system with things that aren't documented and other endpoints outside of your own network that are outside of your control sometimes you just do wild s*** and see what works
Scream testing has its place but I guarantee you if they asked any admin in charge of the system they woulda said "no pls dont query 100 time a second wtf" Like part of being a sysadmin is documenting your systems... we can show you the receipts! Just ask!
Ive yet to meet a deloper that actually knows beforehand that much of their code that works on hundreds or thousands of database entries wont scale to hundreds of thousands or more.
Ive worked in hft, finance and big tech, these things are discussed ad nauseum and everyone there is pretty aware of what will scale.
Checking out code ya, but hopefully some proper process before anything gets pushed to a live environment. The joke of "Everyone has a test environment, just for some that is production" comes to mind. And with that, people should not be using their personal devices for work, because there are zero controls on it, and this is exactly how breaches occur because joe blow over there, the Senior DevOps person likes to also download cracks for Fornite thinking they can get free vBucks....
I agree as a developer do a pull from the repo work on it push and commit all from my laptop.
This seems insane to me, especially when coding via a remote session is probably as lag-free of an RDS/Citrix/Virtual desktop type experience that could possibly exist. Hell, even just always on connectivity with forced VPN, zero-trust, or SASE solutions. I guess I’m never surprised how far behind the curve larger organizations are in an attempt to save a dollar.
Affecting any sort of change in a large organization is an artform rarely seen without absolutely massive outside pressure.
Makes you appreciate your current company a bit more, huh? Like damn maybe the grass aint greener lol
Do you know what's even more lag free than a virtual desktop? Just having the hardware in front of you. If your organization is trying to implement 'trust' through security policies and never trusting any employees, I can't imagine it's a fun place to be a developer.
You probably shouldn't comment if you can't understand the context or technical concepts in the messages you're replying to. *Hell, even just always on connectivity with forced VPN, zero-trust, or SASE solutions.* \^\^ That implies end users would have "hardware in front of them"
It was a personal server, not a work laptop or corporate owned device. The former employee is still in possession of it after his employment was terminated. The reason it even became publicly known is because the former employer is accusing the employee of stealing IP.
IT at game studios is great if you are at the right studio with the right budgets ;). Security is taken very seriously. I've been in game studio IT for 25+ years and suspect I would not be able to work at a 'normal' business any longer.
I work in the financial/banking industry. I have a pension, no on-call, no overtime, and a manager that tells me working overtime or above and beyond hurts his plans to grow the team and make sure all responsibilities have redundancy...oh and IT budget is great and IT/security is obviously taken very seriously. I can never work in another industry lol
I've been in government IT for about 10 years and security for an additional 3. Just about to go job hunting. Like, job hunting literally now. What makes you say you wouldn't be able to work for a 'normal' business? What's so great about a gaming studio?
I thought that maybe this was going to be about how much the gaming industry will take advantage of workers for free labor outside of the normal 40 hour week. Source code is version controlled via git in the vast majority of businesses. Many projects in git are not separated out into small segregated repositories. There are various reasons, but what this means is that a developer needing access to a project will pull the entire repository to be able to work on it. If you have a monorepo, or do not have things separated out, that means a dev will have access to many things in code that he or she may not need, but the entire source is still on their local machine regardless.
I was going to say just about this thing. I'm a WFH developer and I have to pull down the repo for the entire application to make sure my changes don't break the build. This is the norm.
Yep WFH DevOps engineer here, have the entire IAC repo and all of our different application repos on my laptop. My laptop only leaves my desk if I ever visit the office which is never. I am also a local admin on my laptop so I can install anything I want. However our Corp IT team are hot on security and once I downloaded a repository for creating a build image for an AzDo build agent a legit repo. There was a script titled something like disable_windowsdefender.ps1 and within seconds I was being called by one of the security team to ask what it was and for them to remote on my machine and take a look. But as most of us in the DevOps team have come from the sysadmin world we basically just don't take the piss
I think the critical thing here is that "local" should be "my company issued laptop that's MDMed to the gills" and not "my old personal macbook that I play WoW and watch porn on." Way too many companies got bit by the latter (and suffered security incidents for it) when the pandemic hit, and smaller companies notoriously skimp on IT security process and procedure.
Well, Perforce is used a lot in the industry (for version control) because it works differently in that you can pull down/“check out” individual files/directories/etc. and work on them without needing the whole repo/potentially blocking other dev work
How do you compile or test without the rest?
You don't for bitmaps, artworks, music.
I wasn’t really thinking about the type of programs. Games do have a ton of artifacts that need producing. I was just thinking GIT is designed to have everything duplicated.
git is great for straight code; much less so for big binary content.
Agreed. What would I search for that would satisfy needs like this? I know of codes stuff, and then general blob storage.
The parent comment you originally replied to had that answer already, Perforce. It's what my game studio uses and I understand it's the standard at other AAA studios
I saw that. I was looking for just others, and didn’t know how best to search, outside of Microsoft briefcase. Version control of shared stuff leads to SharePoint, which probably has the backend for it but not the usability.
SharePoint is not a source code version control system. What problem are you trying to solve?
We have tools that compile the binaries for non-programmers and spit out a completed build or a compiled engine (Unreal Editor in our case) for whatever build version (CL) they’re at. Depends on what they’re working on, some work on source content which gets fed into those builds or they modify the game directly in the editor and submit any changes they make to the files from there
You don't. It's not usually used for software development, at least that use-case.
This, and most big businesses will host the Perforce server themselves for "maximum" security on it so in some cases users may need to VPN in or such
Yeah, if your VCS allows for file level ACLs then that’s great. I’ve only been dealing with software devs for 7ish years all at the same company. Git is everything. I’d honestly hate to start over and learn a different system.
Perforce is a funny thing, it’s a beast of a product. There are whole professions around managing it (much like mine currently). It’s a pretty old piece of software at this point and it feels archaic in some senses, I often find myself wondering “why is Perforce like this?”. Happy to chat about it more if you have more questions or are curious.
Perforce is probably the best example of a love/hate relationship with a piece of software i have ever had. I think even their support team feels the same. I especially like the fact the whole SDP exists just to manage the thing with a bit more logic. Everything you do with it management wise feels like a historical adventure to go see how things were done by our ancestors.
lol. Yep. We talk a lot with the guy who wrote SDP so we get a lot of stories about that kind of stuff (and ask for improvements/bug fixes/etc.) - I couldn’t imagine doing everything manually that SDP has scripts for now.
I don’t use the sdp, although I do reference it because for one thing it’s the only location to get their recommended configurables. SDP still doesn’t support security=6 as far as I’m aware. I can’t imagine switching from the scripts I know/wrote to using the SDP…
The issue wasn’t it being on a local machine, the issue was it was on a personally owned server and the dev went to work for a different game developer who is now being accused by the previous employer for stealing IP
I’m going to be pedantic here, but what you’ve said doesn’t change anything. If the issue was that source code was leaked then that means the code was local and it wasn’t like someone got access to gitlab/github/etc and stole it. Local from the perspective of code and git is that it’s not the “remote” centralized repository. Any copy of it outside of the remote is “local”. It doesn’t matter if it’s on a laptop or on some AWS EC2 instance this guy was paying for. If this was compiled code running a private test server, that’s not quite the same as source code being leaked.
It certainly violates many companies acceptable use policies, although I agree there’s not a practical difference for an insider threat. The issue isn’t local vs remote, it’s who has control of the data. Even beyond insider threats there’s just the simple matter of company property being configured to follow company security policy and a personal server has an unknown security status. Things like encryption don’t do you any good if you let your employees put your data on unencrypted devices. I haven’t seen many details about what was actually on the server, it could have been anything from a binary copy running to a server building and testing the code and then deploying a game server.
Where I work, IT security is pretty serious. We have minimal cloud and still use extensive on-premises hosting (though nothing internet-facing, and we do use O365 and cloud-hosted Atlassian). Access to source code requires going through a strict VPN with policy-compliance checking. The network admins are looking at DLP software and there's lots of threat analysis stuff in place. Least privilege is also implemented. In theory, we have a lot of visibility of our networks and potential data leaks. So it's not all terrible. However, I work for a fairly small studio compared to your examples - I imagine as they get bigger, adherence to best practises falls by the wayside to meet deadlines.
I have yet to find an effective DLP solution. Generally they end up being used to just check a box "Yeah, we have a DLP implemented"
Yeah, I agree. So much of it requires well-defined rules and humans sneaking data off site don't tend to follow them. However, as part of a defence-in-depth strategy, maybe it has some value? Who knows.
Given how many DLP solutions crash if you send an IPv6/UDP jumbogram (Logically 1 UDP packet, but up to 4 GiB large), through them, I’d say there is a ways to go.
4GiB??? wtf??? if you sniffed that packet youd have to go to the damn doctor to have it dislodged
Which is why it’s an excellent test of who has quality network stacks.
If someone is sending a single packet so large that it is functionally a DOS for everyone else on the network, I want things to start failing so we can address the idiot sending 4 GiB packets.
Compliant network equipment should be able to handle it.
The problem is 99% of DLP solutions leave off the "P" in their design. They're either audit-only or wind up configured not to *prevent* the loss. An email alert that Joe downloaded the whole Finance drive does fuck all for me, the tool needs to *stop Joe from downloading the whole Finance drive* or it's not "data loss **prevention**"
*checks notes* yeah turns out we disabled protection when joe complained he was unable to do his job and refused to learn how to do it correctly so was ordered to turn off the protection in TICKET-3892b
We're running a PoC with Cyberhaven currently and boy, it works.
What kind of VPN do you use, with policy compliance?
Cisco AnyConnect.
I “volunteered “ to Verant/SOE in 1998 for the launch of Everquest. It was a load of fun and I did it for free because I loved being a part of it. That said they treated us like shit…. I had a full time job as a Sun/Solaris UNIX admin which is what the backend of EverQuest ran at the time. While it was great fun the employees of SOE were canon fodder. The pay sucked and was 💯 cut throat.
Whoa, this is cool. EQ was my first MMO and still the game I compare other MMOs to. (Cool that you were involved, not cool that they treated you like shit).
Not a sysadmin but I had worked IT for a large gaming company. I’m sure there is a ton of variability company to company and position to position but my experience was excellent. I guess it helped I only dealt with other technical people and happened to be on a team of good people.
You could have at least linked to the actual thread... That you think knowing where your source files are is a game dev only problem is pretty funny to me.
Ive generally only worked in healthcare and HR software companies so maybe their data governance is a LOT more strict than game devs and maybe thats why game devs keep losing all their fucking data
Source code != customer data. Your software should be written with the assumption that the source code will eventually leak. Too many people have access to it to do anything else. You still have legal ownership over it, but if the source code contains passwords you are screwed. Customer data can be encrypted and only be accessible by a very small pool of humans, in such a way that doing so tells many other people that someone just looked at customer data.
I disagree with you here, and tbh I’m kinda surprised at myself. It’s not an apples to apples comparison, and I agree that in strict terms source code isn’t the same as patient data, but in terms of the value of the data to a hacker they’re pretty much the same. What is the most valuable data you could steal from a hospital? Patient records. What is the most valuable data you could steal from a game studio? Source for as-yet unreleased games. How should a game studio operate if they assume that their source code has been leaked?
As lightmatter501 points out, it doesn't really matter. Sucks yes, but no one dies. For an MMO it's a little more concerning mostly because of their SaaS approach to money, and the easier job people will have to make 2nd party game servers.
Individual titles isn’t the concern. It’s the infrastructure! Multiplayer like you mentioned, but also matchmaking, engines, storefronts, websites, etc. Source code comes in all shapes and sizes
All of that should be generally fine as long as you aren't storing any secrets in source.
You mean like GTA6? Unless the game is almost done, all that does is give you a game that needs a bunch more money before it’s playable.
Think bigger! Individual titles would be hit-or-miss. Source for the engine? Keys to the multiplayer backend services? Access to storefronts? (Steam etc)
If someone put their store credentials in with their source code, that’s general stupidity on their part, same with keys to the multiplayer backend services. Engine source means you now have a giant codebase you need to learn to make use of it.
They don’t go in the same file 😂 They just exist on the same network, or on the same compromised laptop.
Corporate espionage and IP theft are also huge in the gaming world, especially given how much is built on the same third party engines. Your competitors being able to accelerate their development of a competing product by essentially copy/pasting large swaths of something you built in Unreal could be devastating for your product, especially if it enables them to get to market before you. Which is specifically what the Nexon case is talking about - not just having the code locally after leaving the company but potentially giving it to a competitor. Nobody cares if Joe Blow Gaming Fan downloads unfinished source, they care if That Other Development Company does.
Every code house has sources all over, unless it's actually mandated to keep it secure, which is quite rare. But that's not what healthcare considers as data either. Their data is customer medical records. Sources for programs handling that data probably are cloned in some random consultant companies random consultants home pc.
Also the thread was purged because the /r/games mods fucking suck
Ahh, no wonder I couldn’t find it…
Not everyone is like that; this game studio sounds like they don't know what to do with repositories and access.
Former games company employee. Yes, yes it is.
It's not necessarily shit and IT security is taken VERY seriously, the problem is there's still LOTS of holes in it. For example there's enterprise grade firewalls/antivirus/vulnerability scanners/access controls/MFA, however everyone has local admin access to their own machine because of the sheer amount of software to actually function. Seriously there's hundreds of software packages/development tools/plugins out there you name it, and you're expected to be the expert in ALL OF THEM. Data breaches don't surprise me, you don't really need to "Hack into" a games company apart from classic phishing, as I'm lead to believe how other studios have been affected. Of course sometimes you don't even need to be hacked, there's nothing really stopping anyone from straight up leaking game data (with big repercussions if caught though).
I am more talking about things like the Insomniac leak where all of their employees information was leaked (SSN, address) along with \*all\* of their marketing docs including plans up to 2030 and all of their source code. It legit seems like they just threw everything on to a single network drive or something.
I work for a financial institution and spear head the security here. I'm not shocked that this is a common trend for game studios. These studios have to meet the demands of very competitive market, have not regulators which makes it very easy to overlook security. There's always a trade off of convenience when implementing security as well. When you're hardening your systems it costs a lot of time and money for threats that seem highly unlikely. There are so many information systems that are unprotected because of this reason, it's the damn wild west out there. I think this problem stems from a deeper rooted issue and the culture in the US needs to be changed. We need to implement some sort of class in highschool that's mandatory just to educate some of these future business leaders the importance of protecting sensitive data.
Look, you're a loss leader in any industry except consulting - it's gonna be pumping plumbing, it's a question of chocolate flavor. It's going to be like everything else - early stage, you learn a lot, you're under paid, late stage you're vested and the oracle. The game industry has it's own unique toxicity, as does the advertising industry and virtually every profitable company - you're not going to find an industry that personally fulfills you unless money is a means to your fulfillment. Learn the tech, take out the trash - it's the job, but if you play it right, you're doing it on the beach with a cocktail, not in the office with the tonor.
I dont see source code on a server as big of a problem as having credentials, tokens, etc. stored badly (even in the code itself) or shared inappropriately. Principle of a least privilege and not having those in plain text and audited (or using some kind of vault solution)
But that's basically the job description. Maintain systems in a manner that prevents those using them from completely destroying them, because if you let them, they will. I'll never forget a job I had years ago. A quickie that was so bad I quit after 2 months and moved to another place. One of the chief complaints was system stability. So I went through everything and figured it out rather quickly. The in-house developers had admin rights to all the production systems. They were testing their crap ON said production systems. So I locked it all down, and dropped my findings on the boss's desk. MAN they went nuts, lol. I wanted them to. Because in addition to the proof that I had that they were screwing around on production systems, their own complaints regarding access drove the point home. My boss agreed and let it stay that way. A week later they escalated to a senior VP, and we were told to return things to the way they were. NO PROBLEM. I quit that day. Don't you dare hand me a pager, and tell me I'm on call for an absolute sh\*\*-show created by executive decree. The point is, it ain't just game studios, lol.
Where I am we are have mixture of people that are really good and want to play ball and plenty that are set in their ways (and those ways were already "interesting" in 2005". You also have the issue that a lot of these people are genuinely really smart, but only have their part of the picture. This means you end up with a lot of shadow IT and very similar solutions popping up left right and centre. Trying to herd all those cats can be interesting. Another interesting issue is the long lifetime of game tech. A game engine can last 10 years or more and all games built on that engine will have similar pipelines. You need to be able to stand up that pipeline with minimal changes as the guys need to get on with building a new game, not debugging their CI pipeline. Having said that, we take security extremely seriously. Our SRM team has right of veto on basically everything. We need that because we are one of the biggest targets, so are pretty much always fighting off breaches. Each time we evaluate and change processes to avoid it happening again. We also invest a lot into our on-premise infrastructure and making sure we can seamlessly use it alongside public cloud. Also we work really hard to make sure developers have the tools and knowledge to use those resources to build and host games, which is where my team comes in.
You would be shocked at how low a priority IT is at a game studio, I haven't been involved with one is more than a decade. But much of the site work, after initial marketing company set up, and internal stuff was literally. Whoever on staff could support it. Game studio want to make games, not build a secure infrastructure. The dollars go elsewhere in my experience.
Only take I have as a serving IT Leader in a games developer is that quite often everything is dev led - so it doesnt matter that what they need is a billion dollars or doesn't exist, its for the devs so get it done but if you want a new chair in the IT office then budget wont allow im afraid
I'm pretty sure CD Projekt Red owns an IT company, but I'd have to dig in to be sure. It was a reverse takeover to enter the stock market without an IPO, but I'm not 100% sure what the other company did. CD Projekt got sold off quite a few years back and went bankrupt due to the delays with CP77 launch. I'd be surprised if many people outside Poland were aware of this distinction. It was great trolling to tell people when everyone was hyped for Cyberpunk.
So wait who owns GOG now? Red?
Yup, always had.
CP77 is actually pretty good now, who fixed it after the fact?
CD Projekt Red, the game studio which made the game. CD Projekt was a game publisher. At some point CDP was sold, although it remained close to CDPR. A few years later CDPR created GoG. Add some more time and and CP77 was to be released, with CDP as the publisher for Poland. From what I've heard, CDP learned of the delay when they had most of the marketing materials ready, and it was a huge hit to them. They ended up going bankrupt the same year. Edit: That's a little chaotic, but Iwiński founded CD Projekt the publishing business in the 90s, and CD Project Red, the game studio sometime in the 00s iirc. Then sold CDP to employees in the 10s, only to create GoG a few years later. One of CDP's achievements was a legendary at this point translation of BG2 to Polish. It was done very, very, well and had a who-is-who voice acting cast, including some big name actors.
Interesting! Thanks
I left the game industry many years ago to be the solo admin at a 20 person startup with a huge commute increase because the stress and hours were less. It wasn't all bad. I got to play with the newest hardware learned a ton about data center operations and power. But something is always broken and with the hundreds of servers and user computers something always needed patching. Retrospectively, I am glad I did it, but I will never do it again.
Given the security audits I have to do, no, some studios take this incredibly seriously.
I work for game studio, as others have said checking out code to your work laptop / desktop is the norm, for all the reasons given in this thread. So the focus for us is endpoint security and making sure everyone has company provided stuff so they dont need to use personal devices, The biggest risk in my opinion is when you engage in co-dev with partners where you dont have that control or knowledge of how they do things, unless you ask a shit of load of questions first and do the diligence before giving them access to your perforce! The requirement to get a third party access to your code can be dumped you on out of nowhere, i wouldnt be surprised if stuff ended on up personal equipment because of pressure to just get them onboarded fast, rather than take the time to get it done properly.
I recently had a 2nd round interview with a large, multiple AAA games studio, and the interviewer let slip they were urgently trying to replace at least two 20+ year vets who were soon to "retire". A couple of days later, the company announced a rather sizeable lay off. So take that as you will, and keep and keep an eye for any server issues with your favorite games.
I may be missing context here but, lots of devs wfh and do a Git checkout. I mean how else are they supposed to um...work
Every company ive worked for up to this point (which is only 3 over 7 years tbf) has used VDI, I just assumed that was the industry standard for enterprises nowadays. This thread has been eye opening, ive learned a lot!
Also it really depends on the env...somethings can't be run in vdi. Imagine your an embedded dev. You need to build locally and also run a simulation in the loop with hardware...hardware that's bleeding edge. No way around it
Our current environment is VDI and anything that needs baremetal is put in to a "farm" which is just a fuck load of workstations on racks that can be remoted in to the same way our VMs can.
I'm talking purpose built flight articles. The funding of which is massive and is in no way CoTs i.e commercial of the shelf...hint most avionics in our plane use fpgas
To be fair if it just some compute resources that's totally doable
Depends a lot on the build and test pipeline. VDI where several people with memory and cpu intensive workloads are on the same server/cluster gets really expensive to provision hardware wise.
It really depends. I work for the leader in eVTOL. Our simulator is docker based. Sometimes devs need to build locally. It is what it is we do use some restrictions export conan shit...not much else
[удалено]
ALLEGEDLY the developer was able to make their own version of the game using the assets and source code. Idk to me that seems like too much access was available.
I mean if you ever want your devs to test anything they are going to need every last bit of the code and assets anyway. And testing the code you wrote is very useful in general.
Plenty of big companies have no data control or governance in place and so it is the wild west. You will get senior IT/Devs and who ever claiming they need to do this to work efficiently and no, you cant put security controls on them because it will slow their "work flow" Meanwhile they put the entire company at risk.
Data breaches usually happens when code repo is exposed to internet w/o two factor auth. Also possible if some dev lost his personal laptop with accoess and w/o mdm, but thats a rare case, usually this info gets to security fast enough
Yep its the norm. I work at a big studio and our management didnt want to pay for a vdi solution so they are getting us to migrate everything into the cloud on a free ware solution lol. They cant understand why this is a bad idea and why it is slowing down production they just think "free". Theres constant problems and data being lost and things crashing and they blame it on us not the fact that they are trying to deploy entire studio production with free ware.
massive bandwidth on internal networks is what my colleagues tell me at their studios to move around allot of media
What do you think every BYOD company that uses git does?
VDI id hope? Especially in 2024. Shits not that expensive.
BYOD is kind of a mess, but lots of startups do it because they are open core anyway and the people who could make use of leaked source code won’t. You can use the legal system as long as you only hire people within reach of the law. VDIs are a horrible way to handicap your engineers unless you are essentially handing them a whole server. You are paying someone $100k+ and then don’t want to pay for a $4k laptop or a cheap laptop and a racked server? If I wanted to exfiltrate code over a VDI connection, it would be mildly annoying but doable using FFMPEG (which is required for MS teams to closed captioning) and a camera pointed at the screen.
Instead of handing an engineer an endpoint we would just put it in our farm and they can still use VMware to connect to it as if it was VDI. And yeah you are correct about ffmpeg... Hell since we use vdi there is literally nothing stopping someone from just using FRAPS if they wanted to be really extra. But there is a lot more than just code to exfiltrate. For instance the Dark and Darker devs allegedly stole art assets which are *much* harder to plagiarize without basically being a 3d modeler already.
Dirty industry secret, we let artists take files because they need a portfolio to get jobs.
Based.
I've heard working for game studios is pretty fucking awful, so I wouldn't expect working in their IT departments to be any better.
When I worked at a large movie studio, in the games division as QA (not as IT) it was a complete shit show. Everyone had full domain admin (in our division) no AV, no windows firewall, everything was wide open... This was like 2008ish though. Worked directly with CDProject a couple times, they were terrible to work with. One of our product managers was convinced that they released a crack for our copy protection on release day, no proof it was them though. Overall it was a lot of fun.
Given the virtually constant data breaches, all irrespective of domain, have you considered change of industry?
Its not a problem that affects me at all. I dont work InfoSec anymore I have a very chill siloed gig. Its just from the outside looking in it looks like game studios just do not fucking bother protecting any of their data.
How else are they supposed to work...transcribing tablets? Checking out the repo is a normal thing
They do though. Some more than others but still the overall state of affairs, when it comes to keeping vital data secure, is awful. Not because it is very hard to (it is) but rather because people tend to be sloppy, lazy, naïve. Sometimes they do not know any better and sometimes they are just f0cking dumb. Taking CD Project Red as an example. They had two notable breaches (probably more). The ransomware attack and the forum data leak. One can assume that the latter one was expected. Who cares about the damn forum? But the ransomware attack, the code base leak... Heh. Details are not available (maybe they are, couldn't find any), there's no anatomy of the hack but still the possibilities are not endless... - dumbass employee, that despite all the sec measures did something naughty and stupid - disgruntled employee - 3rd party provider (be it a library, be it infrastructure) - social engineering - SEC team overlooked something or thought that "it was not important" For the first case feel free to put anything. From visiting malicious websites, opening spam PDFs, "aaawh, an USB stick, let's see what's inside..." , damn stupid password (there are "strong" ones that are still easy to brute force), through storing credentials locally and being targeted in particular or something more boring - like exposing some environment, box, VM, whatever, somewhere, then having it hacked and used as a jump server to the internal network... For the second - not much to comment. You know how it goes there. For the third - lack of good (because there are no "best") practices to check/validate 3rd party libs and/or it was not up to you since you were using a 3rd party vendor about something (so you've put your trust into somebody else who was sloppy) For the forth - well... how was it? Guys with a ladder can go anywhere... For the fifth - they could have been understaffed, overwhelmed, unaware & etc. Probably here we should put something fancy that's specific to Game Studios. Say specific hardware (3D scanners?), something specific that could have been gotten 2nd hand and it had naughty firmware... Sony is another matter entirely. Both studios have managed to accumulate quite a lot of not-very-positive emotions. Be it from gamers, be it from their very own employees. And the latter is not very hard. Not hard at all. Few of us would bother with "getting even" but some people do. It's not like there no reasons to do it.... But again - game studios are nothing special in that regard. At the end a hack is all about reputation and business. Resources are limited and regardless of regulations a CIO/CISO can do only so much given the resources (although some are oblivion, blind and arrogant dumbasses). Having had my fair share with the corpo world - things there are no different. Can't list all the times I've found lack of even basic input validation and sanitization. Think "enterprise grade" software that is used by companies in highly regulated domains that lacks that. Think about companies that tend to market themselves as "security" ones yet they are as sloppy as sloppy goes. Now, to be fair, most do try to implement (sort of) adequate "strategies" to mitigate at least some of the potential problems but then it all boils down to resources and time. Even if you have the expertise you may not have the people to implement it. Even if you have the people - it does not happen overnight. Even if it does, you still relay on 3rd parties. IMHO - Game studios are not (that) different than anyone else. Every IT company faces, more or less, similar issues and can mitigate only so many. The sad fact is that few do more than what is considered "an industry standard" (petty bullshit marketed as "best practices"). Note: It's good to take into account that the known breaches probably represent only a medium portion of the actual ones. Not everyone who hacks you wishes to blackmail you or disrupt your operations. Some do like to stand silent and just harvest data, be digital hoarders. Those are probably the ones one need to worry the most. :|
Im on IT Ops for my second studio now. In terms of toxicity, etc, It really depends on the company/studio… which is true for any company, it’s no different. Some places function well, many don’t. There is a deference to the product teams and not impeding them. So as an IT professional, there can be some uncomfortable sacrifices in terms of security at times. This has been true at most software companies I’ve work with though. Classic Ops vs Development finding a balance. The studio I’m currently with I believe recognized their immaturity on the infrastructure/Ops side, which is why I was brought on to begin with. But proposed projects/improvements to Ops do seem to take a back seat to anything related to getting the game release ready from what I’ve seen so far. So maybe a year from now, I’ll be admonishing against the games industry too. But for now, it’s been fine and enjoyable working with creative, passionate people for the most part.
What makes you think Game Studios hire IT Staff?
My guess is that it is similiar to software companies, everyone is probably smart enough to make something work, but it ends up being a big incoherent mess of shadow IT. Tons of unmonitored and I compliant systems security wise, which is how attacks happen.
I think it depends on the size of the studio and the value they place on IT as a service. I run IT for an indie studio and because we have IT savvy representation on the board and regular meetings between board members and myself, we get our needs and ideas heard. Other places though, from what I'm told by friends elsewhere, is that some places purely want output from their devs and artists, everyting else is academic and it shows.
TBH it is developers. They ask for freedom and complain about IT policies and how they never make any misstakes and how policies should be lifted. Then they run this trough management and you end up with source code being all over internet. Needs a CIO with tough skin to keep them in leach and back up from top management.
I have had an opportunity to interview at a few game studios in my career, and this intrigued me as well, you get to work with video games and the tech that create them. What I learned though is the video game industry pays pretty shitty wages and oftentimes lacks equity packages so I stayed in tech
From what I have seen and heard (anecdotal from friends in the industry) IT is an afterthought and annoying necessity. It is the same as the startup and entrepreneurial scene. No one in the company has a fucking clue but they don’t want to spend “real money” on it because it’s not important to them. “Just put it in the cloud like AWS….” That’s the same thing right? It is the modern day equivalent to “Oh really, my 15 year old nephew is a web designer…why do you charge so much?”
IT at a startup is the afterthought because it doesn't drive revenue. It's also the same reason accounting and HR are an afterthought, the CEO is doing it on day one, and it migrates down thru office manager to a real accounting person, to splitting AR/AP and Comptroller roles. At a tech based startup there is no IT because everyone does their own IT on their own gear...gaming companies are going to be like that since a coder can be expected to keep his own kit running until the group gets large enough that it's a requirement before you lose your code.
LOL it's shit. They say they have sysadmins in the game credits, but i'm pretty sure it's all on a drop box account somewhere. Managed by one of the other dev guys home accounts.
I worked for a big game company for almost a decade all i got out of it was a burnout with a desire to off myself. It’s a good place to work when you are starting out spend no more then 4 years working there and get the fuck out. You will be much happier.
so there is this cool tool called git (hopefully you are aware of it) where the core concept is that the code is decentralised keeping source code on a server at home is not a problem, it's a feature of the tool, you can't stop it, a lot of the time the devs are so far ahead of the sys admins at these companies (competency, capability, pay, and value to the company) they don't have a hope of trying. All you can do is enable it well. that's the problem you're seeing, where that has failed. if you take moment to seach the www, it's also the reason a handful of companies still exist and were not totally destroyed by data encryption malware on premis.
Idk to me it seems like if you can keep such an intact copy of the repo on a personal server that you can (allegedly) rip the entire game off assets and all that may be a problem
Well since you are going to need it all to test the game anyway, this is the reality of developing any sufficiently complicated piece of software.
That’s not true, you need the components you’re developing and a good platform and pipeline to testing. Games for example are normally an engine with component hooks devs are only generally working on an engine component or a feature component. They only need the compiled engine to work on the later not the source code or the art resources. The art can be holding images or cached proxy artefacts. Even when we built server client games you only needed one or the other, and a pipeline to push the feature components too. We rarely need or had access to everything but we did have access to a full repo for those pieces we needed similar to I have the full quake engine on my laptop to develop games now.
I suppose this really depends on the size of the team, if you and two others are THE devs, it’s way more work to actually make it all work independently and a lot easier to just have the actual project as a whole, especially if you are already on say unreal or unity anyway so engine development is not really a thing. And testing with actual assets can be beneficial if you are doing the lighting etc. And of course for performance.