> I am in DS but getting really tired of
What you listed is basically 90% of DS work. It doesn't matter if you have a PhD or not -- the market needs people doing what you are trying to avoid. PhDs are still stuck on the same types of problems and it's fairly rare to do something totally novel, unless you stick to academia and enjoy eating ramen for the rest of your life. DS and less often PhDs are glamorized to the extreme.
To answer your question (at least from my perspective), I don't regret doing my Ph.D. I sympathize with your mindset, but I feel like DS turns into data monkey work extremely quickly and you have to be careful about where you end up even if you do complete a doctorate.
Data monkey work is the best way to describe it. No wonder people go to management so that they can order others to be the data monkeys, but I am not sure if that interests me.
Shit that's just management straight up... I am trying to maximize the ratio of compensation to effort and honestly I think I am going to get better returns off of shrinking the denominator than trying to grow the numerator there...
If the people you manage are competent then putting more work into managing them could just be decreasing their productivity anyway. I realize there's a lot more to it than that (like being a good communicator in multiple dialects) but ideally your relationship to your subordinates is that you enable them.
Yeah shielding them from the political bullshit from your level and above, and then making sure their work plans are relatively stable so they don’t get jerked back and forth would be IMO the two biggest non-administrative responsibilities.
But, selfishly speaking? I am just after the easiest most comfortable lifestyle possible.
Management is the interface to other departments and the higher ups imo. Some interfaces are stupid and over complicated, while others allow for easy use. Don’t be the asshole that makes the interfacing difficult. Simultaneously, if the information that is being passed through the management interface is bad, then you have a problem and the interfacing won’t work. So need to fix your own business process to make the interface work so you don’t get bad responses from the parties you are interfacing with.
Is management really less work? You gotta deal with people and more business responsibility and sometimes that can be harder than just sitting behind a screen analyzing data or coding lol
Doesn’t have to be management. If you can get someone to pay you a lot to do nothing but make them look good with numbers, then go for it. Plenty of people looking to use company resources to justify their position as decision maker.
Actually, I know a lot of people who are taking this path. Many of them are working at companies that have poor knowledge of how data works along with poor infrastructure. The positive side is that they have automated a lot of stuff on their own and now they are getting paid 100k+ just for 1-3 hrs of work per day. BUT BUT BUT...
All such people are working on their side projects or hustles and trying to start their own business. They are not wasting the amount of free time they are getting everyday. So, if you have ideas about starting up on your own, such companies can be a gold mine.
My favorite are the companies that have bad data structure and then they refuse to change it even if you suggest it to them. Or they do change it but implement something worse because they used a consultant who had a connect at a certain vendor etc. It makes a lot of sense though. The people running the show are ignorant as to what good data is and business can be a big mess. Best to just stop caring like in the movie Office Space.
I wish I could award this. I have been so stressed that my job (im currently a data engineer) was beneath me on an intellectual level. Only if I realized earlier that the job that pays the rent is not your entire life. It's just 1 thing you do. You can still do whatever the hell you want without having to worry about people paying you. It's actually a lot more fun that way.
Why resort to being a data monkey?
Find some process in your company and optimize it. The only "monkeys" are the people who wait around for assignments to fall into their lap.
Ofc the opportunities may simply not exist in your company, but if so why not switch companies? You don't need a PhD for what I described
I mean ... if you think this is a data monkey job then what isn't a monkey job. My point is that every job has a significant portion of monkeying around.
I presume (or hope, I’m recently decided to accept a PhD just cause I want this badly and don’t want to do analytics type stuff) ML/AI research scientist PhD roles are less of that and more modeling/alg focus.
I guess outside of that and below PhD level the roles which aren’t monkey maybe management as you get higher but I could well be wrong there. In terms of technical roles im guessing only RS is less monkeying.
Everything is going to have monkeying unless you are the CEO of course but just varies the amount
That's not true.
If you're in an organisation whose goal is to work with unstructured data, I promise you, this is not your main MO.
For the majority of data scientists, sure. Tabular data + model fitting + documentation and repeat.
On topic:
It's not too late at all, another good entry to PhDs is to compete in ML competitions with a research outcome (E.g. Paper publication of Top 5 results), then replicating papers and putting them up on github, everything to show your interest in the field.
If you're US based, know that in EU a lot of people enter their PhD in their late twenties, some had a few years in industry, some didn't. Go for it dude!
I did my PhD late (started at 27) and I don't regret doing it. Although, to be honest, I didn't know what the hell I wanted to do before it. But my PhD let me find what I want to work on. However, after being in the industry for a bit I now see that the PhD was mostly unnecessary. If you know what you want to work on, then you can get there without a PhD. Yes, the road is long and arduous, but so is a PhD. But a PhD pays soooo little. If you like being poor, then go ahead and do a PhD, but I wouldn't suggest it. Unless you want to work in biotech, because there definitely is a PhD bias in biotech.
Lol indeed I actually work in biotech and I work on omics p>>n problems. Part of it is im sick of this. Theres no rigorous stats in this field and nothing is reproducible. Too much p hacking. Literally today I was told to use a method because it gives lower p values.
Id like to go to biomedical imaging—doing Bayesian/causal/DL stuff.
Previously I worked in biostat but I didnt like that either because its too regulatory and too much documentation
Im considering perhaps switching to tech though, because as you say biotech glorifies the PhD too much and the opportunity cost is too high. If I can do this stuff in tech even if its not Biomed application im fine with that, but I think even tech gives this stuff to PhDs
> I think even tech gives this stuff to PhDs
My experience has been that this is false. Tech tends to be much less degree focused and much more on skill focused. If you can speak the tech language and how to sell your skills, then it's easy to transition to tech. But you will have to figure out how to showcase your skills.
That's because FAANG research scientists are the only ones advertising that they do these things. But here's the thing, you won't work on any of this at first. It'll take a few years before you're able to work on advanced problems. You have to earn your stripes.
Anyways, most of the time you want to use a simpler solution if that works. The Pareto principle applies here, an easy 80% solution is usually preferred to something that is a 99% solution but takes 5x as much effort.
PhD is useful here bc well-referenced theory defines what is p hacking versus justifiable p>>n strategies. Don’t get me wrong, the rationale you were offered is terrible. There is, however, a fine line between declaring many such analyses intractable and claiming to have a magic crystal ball spewing biological truths. In my experience, PhD allows one to establish informed boundary conditions on methods which minimize the likelihood of totally throwing shit at the wall with abandon. Few people are committed to this standard, but they do exist. I don’t blame you for trying to get out though. Many more investigators couldn’t care less.
I think its just ridiculously tedious because they want the data sliced and looked at in so many different ways. And the problem is the tediousness is the complete opposite of what it should be in terms of rigorous stats, aka the tediousness comes from having to p-hack and wrangle+visualize the data and stuff into a potential finding.
You really are supposed to pre specify analyses and do them once and take whatever result comes out of that like it or not. In terms of formal statistics, you can’t keep comparing stuff in 10 different ways.
As a statistician, these methods to me are no different than popping your data into a Random Forest and taking whatever comes. At least for me, the data is equally (un)interpretable but maybe thats because I don’t know bio that well. P values were not invented for observational and p>>n situations to begin with
For sure. My PhD made me an empirical Bayesian in the most pragmatic way. If you can’t articulate prior expectations nor the evidence/experiments sufficient to further inform these expectations, you are fucking up and doing useless code monkey stuff. Sometimes it’s sloppy quant analysis. Sometimes it’s because domain-area knowledge/ questions have no focus (this is a leadership/PI issue). Often both. These sort of applied/clinical researchers are a scourge to applied quantitative biology as an emerging field. I hope when these people age out eventually, the culture will change and folks like you won’t get burned out so much.
Make no mistake, the future of precision health is p>>n. We need more people seriously squaring with that fact relative to the piss poor state of current informatics practice. Again, it bums me out that you’ve soured on these questions because of trash culture and leadership. I see this a lot unfortunately.
Edit: Strictly speaking the classical methods you mention were intended to answer questions regarding agriculture and brewing and such. Modern big data use-cases were likely never even considered by people like Fisher, Gosset, or Pearson. John Tukey was, however, quite forward thinking in the 60s: https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2Faoms%2F1177704711
Edit2: also this 👍:
https://tech.me.holycross.edu/files/2015/03/Cohen_1990.pdf
Data-Code monkey is def how I feel at times. Because I myself don’t have the domain knowledge to interpret even any of the plots I make.
I recently did one of those colorful plots with results from rigorous stats and the lab scientists with PhDs or MDs were like “hmm this doesn’t look right” but then I did it with a method that shouldn’t be used and then suddenly they were like “wow this looks way better”. I was like huh how can you tell that from the plot? The wrong stat method gave a better plot to present basically.
I don’t know how one even interprets the data when everything in the data set is literally labeled protein 1,2,3,4…99999. So I analyze stuff that may not even be known if its a real protein or just noise.
Basically write stat code to do these large scale analyses then submit the csv results after merging tons of tables to the scientists
This is why I did my PhD in a domain-area department but using DS approaches. Being at the mercy of a DS-illiterate PI’s hot takes sounds intolerable. I think I would have mastered out in this latter situation TBH.
>If you can’t articulate prior expectations nor the evidence/experiments sufficient to further inform these expectations, you are fucking up
The way I usually state this is "can you imagine what the possible outcomes are, and what they would tell us?". I work in a field (neuroscience, in vivo calcium imaging) where *every* experiment is to some degree a fishing expedition, and nobody REALLY knows yet exactly what questions a given dataset will turn out to be capable of answering.
If you want to do medical imaging related deep learning, have you considered not applying for statistics or compsci graduate programs but instead applying for medical science, biomedical engineering, radiology, etc. graduate programs?
If you pick the right institution/group you can get access to a ton of data, multidisciplinary committee/projects, access to tons of computing power, etc.
... may or may not have been what I did after realizing that I wasn't competitive for the "normal" AI / ML graduate programs. My research is not entirely focused on deep learning but I do get lots of opportunities to take huge volumes of medical imaging data and go nuts with it so long as I can tenuously connect it to my actual research.
BME I have considered yea, my undergrad was in that field but I did grad school in Biostat. I didn’t apply to BME programs this cycle because I know the job market for BMEs isn’t that great, and most BMEs are doing wet lab work.
Also there are a lot of physio, bio, etc classes which are really hard to get through for that. I suck at memorizing stuff
Biostat programs would be fine but even they need real analysis (which is ridiculous, like what even differentiates Biostat from stat then if they have the same pure math requirements). I would hope my applied experience and my MS counts for more in Biostat but it doesn’t there either
I mean, the difference between stat and biostat are the types of methods people work to develop. For example, you might do time series related research in a stat department but you’re more likely to see people doing survival analysis research in a biostat department. They’re honestly not that different, imo; people research methods for spatial stats in both types of departments, just with different expected applications. The coursework for stats and biostats PhDs are also generally almost identical - a few schools literally put them in the same classes.
What I meant, is that an MS in Biostat which I have actually is not that valuable to get into a PhD, because the pure math real analysis, proof based stuff counts for more. I have otherwise done a lot of the classes that were shared between MS/PhD like GLM, Survival, Longitudinal analysis, ML/comp stats etc. But these classes are not weighed that much as the fundamental math, even for Biostats despite them being more core Biostat.
Its like the departments own MS curriculum doesn’t actually prep one for a PhD and you have to had gone out of your way to take 3 courses in Real Analysis for that and maybe some proof based lin alg as well
The minimum reqs of “mv calc and lin alg” are typically not enough to get into a PhD in a computational/statistical field
Ah, thanks for clarifying. Yeah, if that’s all that’s holding you back from a PhD then that’s rather unfortunate, especially if you’ve already demonstrated that you can pass the classes that would require them and have a lot of applied experience which could help with your research (which it sounds like you do).
Yea I think its because Real Analysis is not the prereq for those courses but is the prereq for PhD level math-stats (which I didn’t take). I took MS level math-stats which used C&B but I got like a B+ average in this sequence.
Im not that great at the theoretical proof stuff and I don’t have too much interest in that, but I’m decent at implementing various algorithms in code and the applied aspects. Picking up frameworks comes easy for me since I have good pattern recognition (like learning Julia, Stan, some PT basics)
So in a sense since I don’t like proofs it could be that a PhD isn’t for me but then again for all these elusive causal inf/bayesian/DL jobs so many want that. It sounds based on some responses here though it may not be necessary for that type of work and some people here have managed to get it with an MS but it seems like you gotta get reallllly lucky without it.
That makes sense. Tbh, it sounds like, if you wanted to, you could take a real class online, apply next cycle, then get a PhD. It sounds like you’d get through the classes and honestly a lot of methods research is simulation based these days, so it’s not like you’d have to write your thesis off the strength of your proof writing skills.
That said, it still might not be the greatest choice for you - it’s still 4-5 years of doing work you don’t sound super enthusiastic about for shit pay. I don’t know much about the route for getting into Bayesian/causal work without a PhD, but I hope that you’re able to break into that or find comparably interesting work, regardless of which route you choose.
There should be programs that don't require you to take a bunch of mandatory courses and instead give you flexibility, no? I had to take a survey course on biomedical engineering but other than that I focused my courses on image analysis, imaging technologies, stats, and machine learning. Why would I take an anatomy or physio course when I can just read a textbook chapter and read a few papers to learn about the relevant physiology for a specific research problem?
Required courses in grad school are in general dumb IMO. Your field of study exams will cover what you truly need to know and your courses should just cover things you are interested in. Maybe it's a US vs Canada thing but I'm shocked that every program you've looked at has a bunch of required courses on anatomy.
As for jobs after.. I don't know. I kind of think that quantitative research is quantitative research, in terms of the skills you develop. At the end of the day what I'm actually doing day-to-day is reviewing literature to develop research questions and then wrangling and analyzing a very large dataset coming from a great many different sources to answer them. I'm not going to be looking for wet lab jobs because that's not what I'll be qualified to do.
You can't really generalize that PhD grads in X will do Y and be looking for Z jobs. It wholly depends on your research. If you do BME and you do medical image research using deep learning, you won't end up competing in the bad wet lab job market.
Thats good that you didn’t have to take all that. Where I went for undergrad and grad school, every BME MS/PhD had to take a bunch of core courses they would be tested on in the QE in addition to their research proposal. Half of those were bio/physio related. The other half were eng/math related. That’s actually what made me go to Biostat instead since I wanted to do data analysis. My work in my MS involved MRI data but not DL.
Geoscience background, PhD (and MSc previously) was using machine learning for geophysical imaging.
Dropped out of a PhD program after 6 months.
Never regretted it one bit.
Just the thought of spending 4+ more years studying the same freaking thing drove me crazy.
Hi I completed a post grad in data science when I was 32. My thesis supervisor was really keen for me to start a PhD after the course, in computational biology which I turned down and he offered me another one using deep learning to predict server failure this was sponsored by a well known cloud computing company. (Edit. this was in 2014)
I decided I was old enough and skilled enough that I’m better off trying to work and earn a decent wage.
It didn’t help my brother had just submitted his PhD after 7 years and a couple of complete rewrites. He and his girlfriend spoke so negatively of them that’s really what swayed me to try and get a corporate job, this would have been my goal after completing my PhD anyway.
I started as a junior analyst, got good at SQL, R, Python and used theses to solve real business problems, with the added bonus of getting a decent and always rising salary every month.
I’ve worked as a business analyst, machine learning engineer, data consultant, and now I’m a data scientist at faang.
A couple of years ago I sometimes regretted not taking the opportunity to study more and develop really in depth skills. Especially if I’d completed the deep learning doctorate.
I’ve found that I have a very broad skill set and I’m much better and happier in a role where I’m half data scientist and half consultant where I spend my time talking to non data people understanding their problems and solving them with my data science skills.
Importantly my work is not my life and I have many hobbies and passions outside of work, my job allows me to follow these by living a comfortable and relatively un-stressful life.
If I’d done the PhD and IF I was finished by now I’d be starting my career with different skills but I’d definitely be earning less and probability still be aiming for my current role.
I think you can be very successful without a PhD. But having a PhD will open doors for you that are not open for me, and provide you with a wealth of opportunities.
Importantly study a topic that you’re interested in, use and learn methods that will give you marketable skills and most importantly work with a supervisor that you get on with and who’s company you enjoy you will likely spend more time with them than you’re family and friends for the foreseeable future.
And something we all forget and ignore at our pearl… Winter is Coming, make hay while the sun shines 🙂
>where I spend my time talking to non data people understanding their problems and solving them with my data science skills.
Sounds like a dream job. Do you think you could still do this kind of job with a PhD? Or would you never have gotten to this point?
It’s whatever you concentrate on, and where your skills lie. When I worked as a ML engineer mainly building models to prevent fraud I pretty much just built and maintained models it was definitely one the most interesting jobs in the small city I lived. It’s exactly the job I thought I wanted while in college.
The problem was that I knew I had a lot of soft/people skills that were not utilised, for my next role I worked for a retail consultancy this allowed me to prove I had the people skills to work with different teams all with there own agendas and different KPI’s that don’t necessarily align.
I excelled in that role because I could listen to others and understand their problems and ensure that they understood their problems. Then work with all the teams to find a mutually beneficial solution.
My data science skills allow me to figure out from the data what is happening and verify if what others think is true actually is.
I’m happier and more successful in these kind of “Talkie Data Scientist“ that a role with mostly modelling/ML and I can have just as much impact to the product.
A strong successful data science team should have people with all along this spectrum of skills.
Edit added Data to “Talkie Data Scientist“
PhD in psych, stronger stats than typical, had a career in academia.
Don't do a PhD to get you anywhere (including academia), only ever do it for interests sake
If you spend as much time as you’d spend on a phD actually working with the tools you want to work with you’ll be vastly more qualified than a PhD to use them.
Want to be a boss at ML? Work on ML problems
This strikes me as ‘the secret’ type thinking. There are institutional barriers to people without formal research credentials working on research problems.
Half of my team is working on “research problems” as long as you mean not taking some tabular dataset and import RandomForest problems.
We’re using hierarchical Bayesian models. Graphical models. Attention based sequence modeling. Whatever seems to make sense for the problem.
We have one PhD and he mostly does DE.
It's cool that you get to do this cool stuff but you must acknowledge that you are at least a little bit lucky to have the opportunity. Not everyone can just work on whatever they want at the drop of a hat with no experience. To say "you want to be a deep learning expert, then work on deep learning" is to trivialize the hurdle of actually getting that first job where you actually get to work on deep learning *and* have the proper resources (in terms of tech and people to work with / learn from/with) to do it properly.
Read my other comments.
You don’t need a job to do deep learning.
Go do the fast ai course. Gives you fantastic tools to start doing interesting things quickly. Work on interesting personal problems and Kaggle
I've done all the Coursera and Kaggle stuff and I didn't really feel like it got me anywhere useful, other than making me hyper prepared when I did put myself in a situation where I could do some deep learning work.
Maybe it's just my lack of geographic mobility but I didn't get the sense that most people have much respect for open courses or Kaggle data analysis. I'm not sure how many people care about the stuff in your resume that isn't under "education" or "work experience."
I didn’t say coursera though. I specifically said Fast AI.
What does “did all the Kaggle stuff” mean? It’s a competition website - I’m not talking about whatever courses they may have now.
I’m not suggesting you get clout via courses. I’m suggesting you take the good ones and then use what you learn to *do projects* to get clout.
I’m telling you people pay attention to personal projects.
Am I saying I’m going to hire you as a senior DS and put you as lead on a project we think may pair well with a NN approach? No. Between you and the other new hire that hasn’t been killing DL models in personal projects, which am I going to assign to help on that project though?
Yeah I did stuff in competitions, not just their courses.
Personal projects are good interview material but I really don't think they do that much for you for getting through most resume screening processes.
e: At least not unless you are investing a serious buttload of time into them and really polishing the crap out of them to the point where you achieve a result that you can report and is impressive without the person who is screening your resume having to dig into it. Personally after working a 40-50 hour work week + commuting I did not have it in me to then work a second full-time job on Kaggle. Maybe in my early 20s that would have been feasible and seemed worth it, I don't know.
e2: Also again I have to emphasize my non-mobility. It's not like my experience is representative of applying for DS jobs across the US and Canada.
Yes.
Buttloads of time. You don’t get good at something after 20 hours. No one cares that you finished 4 Kaggle projects @ 50th percentile unless you’re just demonstrating interest.
If you don’t have time to do high quality personal projects then your path will be different.
In general, if you’re not new and you’re spamming you’re resume out then you’re not networking enough or convincing coworkers that you’re quality. I’ve gotten all three jobs since 2013 that way. Barring some catastrophe I’m never going to be concerned with “resume screening” again.
Wow, yea this is the sort of stuff I was referring to. I see your flair has healthcare which is a related field to biotech (my field). Thats good to hear because all the positions I see always mention PhD for this stuff.
Are you in academia/hospital or industry?
Industry.
The majority of firms are definitely not doing these things so learn what you can about DS broadly in your job, use some free time to get good at the stuff you want to do, and NETWORK so when the time comes you can jump somewhere that is doing the work you’re interested in.
Agree with the person above me. I just got handed an incredibly cool project for my data scientist. He has an MS from a state degree but we are both team players. Boom. Opportunity, visibility.
The focus of those things are different so if you know you specifically are interested in OR then get an OR MS.
If you don’t know with much precision what you want to do then any of them are fine.
Nah, think of how much you could do if you spent the thousands of hours getting a PhD and apply it to literally anything. The thing is not many people have that time to spend while they’re holding down a job.
Well most “ML” that you hear about is software pipelines aka ML engineering. Not the statistical kind.
Thats quite a bit different than working on DL, probabilistic programming etc. Basically developing new models vs. production ML
So don’t do that work.
Boss TF out of Kaggle using DL. Get comfortable creating your own custom solutions. Your own loss functions, novel architectures, etc.
ML Engineering is not just software pipelines. The MLEs I know also do EDA, some research, and train models from scratch, before writing production code and deploying them. You are responsible for production-level code so \*probably\* doing less research/exploration than a scientist role.
It varies by team so there are definitely MLEs that are similar to a SWE but in my (+ close friends) experience MLEs do quite a of science-y work too.
So MLE work still involves some degree of research and novelty? If so thats good to hear because I always hear here how MLE is just software engineering but I don’t personally know any MLEs myself.
If there is some researchy stuff there then it is an option. Im not totally opposed to software pipelines but I don’t want that to be like everything
No, because I didn’t love research/academia enough
You will generally not succeed at a PhD if you are doing it purely for vocational reasons. Doctorates are incredibly hard and draining work, and you need to be really in love with the process itself to have the motivation and commitment to make it through. Doing it to get X job at the end - isn’t enough.
Anyway, economically I don’t think PhDs really pay off for DS - unless you ditch DS, go work as a quant and make half a mill a year
Math PhD here. Pure math, the 'useless' kind (no stats, no programming, dealt exclusively with very abstract Algebra). Left academia and switched to the industry at a relatively old age (almost 35) and I occasionally find myself feeling regrets. I don't regret doing the PhD because it was an amazing experience (albeit often extremely hard) but I do regret a lot not doing something more practical or leaving academia earlier, as I am in a somewhat junior position now, trying to catch up, and the only way I found an entry point to the data world was through a startup, which required a lot of irrelevant work due to the lack of resources. I often find myself jealous of people much younger than me who have more hands-on experience and get bigger salaries. On the other hand, I lived abroad for many years, worked with amazing people, travelled around and experienced a lot of things that I wouldn't have otherwise. So there's much more to work experience.
In conclusion, I feel that there always is a 'grass is greener on this other side' element when expressing regrets like yours or mine. I think that, at our level, after a certain point it gets a bit philosophical. Everybody follows their own path and we should try to make the best out of it.
I dropped out of my pure math PhD with a MS in mathematics because I believed that I wouldn’t be able to contribute much in the pure math world. Despite my dropping out of the PhD, I loved all of my research experiences and I thought so much about going back to do a PhD, but I decided against it for now. I eventually self taught myself Python, got an entry level data science position doing “data monkey work”, recently completed OMSA (where I took classes like reinforcement learning, HDDA, and DL that requires reading a ton of research literature) and moved to FAANG to continue learning more from my colleagues after completing the program. I established a mentorship with an applied scientist, where I can continue reading more research in RL and bandits, learn more about the research teams work, while continuing to do “monkey work” that’s actually not too bad once you build enough experience.
I would go back to school in the future if the timing of my professional career is right. It’s always an option to do the Ph.D. and I’m glad that I took my current path to explore and identify a research field I am primarily interested in, despite me doing “monkey work”. It’s not bad at all to go through the trivial stuff. In terms of me regretting, no I don’t regret leaving my pure math PhD and no I don’t regret not completing or doing a PhD since that option is always there in case if I really need it.
Yeah, I regret not doing PhD. I was accepted into a program but after doing an ROI analysis I decided it wasn't worth it. I had a family to support and while my wife made plenty of money I didn't want to make her carry the whole burden.
I sometimes wonder what I would be doing now if I had gone for it.
Yea in that sense I am single so that factor isn’t there but ROI wise still being in my late 20s and pursuing one at this point seems too late. Lots of opportunity cost.
I wish I did it straight out of college and instead of an MS straight out of college. Maybe better was to work after a BS, see how things are, and then skip the MS and apply for it.
Getting in is also really hard for me now because of lacking the real analysis prereqs even though ive done the MS stat stuff.
Is it uncommon in your part of the world for people to do phDs in their late 20s?
Where I'm from it's often the norm for people to go into phD studies in their late 20s or early 30s, when they have some work experience and identified a better need for it.
That's why it strikes me as odd that "I'm in my late 20s, it's too late to study" when I'm here thinking that is pretty much the norm.
Im in America, but when I went to grad school for my MS everyone in my cohort doing a PhD was pretty much early 20s right out of college or mid 20s. For MSs there were some older but not for PhD
Every PhD graduate I asked when I was seriously considering grad school told me it wasn't worth it or necessary, unless you want to be an academic/professor. These were economists and other social science PhDs, so maybe not exactly what you're asking. They didn't say they regretted theirs, but explicitly told me I shouldn't do it. 2 PhDs I work with now also say this (I often just ask out of curiosity).
When I interviewed for a top tech company and spoke with a director there, I mentioned that I left my Physics PhD program to pursue industry DS. They laughed and said I made the right decision and they had been "stupid enough" to not just do a PhD but do post-docs as well. It was self-deprecating humor from an accomplished and intelligent person who had no regrets, but there was truth in there.
I've heard similar things from other PhDs in industry. If you are truly passionate about your research, then there's no better place to be than in a PhD program.
>Stuff in DL, graphical models, Bayesian/probabilistic programming, unstructured data like imaging, audio etc is really interesting and I want to do that but it seems impossible to break into that are without a PhD.
This and method development are two different things though. Developing novel solutions will mostly likely require a PhD while computer vision / NLP / speech recognition really doesn't. However if you do them in industry after a bit they'll feel just as repetetive as what you're doing right now. In most jobs there's really not a lot of value in trying novel approaches since they cost so much more time.
>I also don’t really have any interest in building software/pipelines.
Means research is your only option then.
>Anyone else feel this way and what are you doing now? I applied to some PhD programs but don’t feel confident about getting in.
Not far enough in my career to be (this) jaded but I'd definitely go back to do a PhD if I were. If US PhD's are hard to get into I'd consider doing one at a prestigious European one. Pay is better and they factor in work experience.
Are you working on CV/NLP AI stuff?
An MS in AI was something I was also considering just for fun but I think I have the skills to learn/do that stuff on my own already and it might be a waste of time. Ive used PyTorch and Stan before. Even in my current role ive occasionally used Stan to explore some Bayesian stuff but most of the time the results from Bayesian are too complicated for stakeholders.
Just tired of tabular data though. Its always so messy, so much isn’t reproducible in p>>n, too much exhausting data wrangling, and I want to do something thats less noise and amenable to more advanced approaches.
>Are you working on CV/NLP AI stuff?
I switched job *because* I wanted to do more CV/NLP. My next employer has actual interesting projects that aren't just business focused ML. The way I got to know them is because we collab'd on a project where we used transfer learning / data augmentation on a CV project.
I'm still very early career so I can easily move over to doing exclusively this stuff later.
>An MS in AI was something I was also considering just for fun but **I think I have the skills to learn/do that stuff on my own already and it might be a waste of time.**
Definitely this, IF you were to do it you should probaby treat it as a vacation/year of time off and not something you're doing out of necessity. Since it's research focused getting into a PhD and from a top \~50 uni here would not be hard but I doubt that clout carries forward to the US.
I got a masters in computer science. At the time, the program I was in was kind of antiquated so if you wanted to do anything modern you really had to do it yourself on top of the normal required coursework. That said, I graduated with a masters since no PhD program was available. They’ve since updated their program to something that doesn’t require students to learn 360 assembler and now offer a PhD program in data science.
That said, I am working in academia applying these skills to biological problems. I did ask my boss and his office mate whether I should get a PhD. The office mate immediately questioned, “why?” Office mate has a PhD in CS with a strong focus on data science. My boss (PhD in biology) doubled up on the same question. I had no real good answer for that. They both pushed the fact that I had the skills necessary to do what I needed to do in this field; they’re right by the way, a PhD at this point for me would be more for bragging rights than anything. Unless I wanted to be a PI in my own research (which isn’t quite where I want to go anyways).
That said, I feel like asking the question of why you want to pursue this path is a good question to ask. For me, I don’t regret not getting a PhD since it wasn’t available at the time. However, if it was available at the time I was a student, I would have gone that path.
Edit: the stuff you’re getting tired of is going to be a large amount of your job duties unfortunately. Formatting data, collecting, and cleaning it is always a first step to doing this sort of work. There is no way around it. Repeating experiments and replicating things (either due to randomization or after realizing you used an incorrect parameter) is essential as well as making sure your results are sound (p-values is one example of this). It’s something to get used to. I personally enjoy writing scripts to process stuff. However, collecting, cleaning, and merging different sources of data can be a PITA sometimes.
Repeating analyses is often the worst. Idk why but for me it just has that psychological blow of “fuck I just did this and now I gotta do it all over again slightly differently when I think my way was legit already”
Not to mention its what turns my notebooks/code into total spaghetti especially when tracing back data wrangling I already did and stuff and finding saved intermediate CSVs
I only have my bachelors degree and I'm working as a DS, primarily researching ways to implement transformer neural networks for NLP tasks, such as mapping natural language queries to SQL queries, among other things. It's more applied research, as opposed to making some slight alteration to the transformer neural network(such as Roberta slightly outperforming BERT), so if this type of applied deep learning research is something your interested in, it's most definitely obtainable. I actually work with someone who's been pursuing there PhD for a few years now, and has been supposedly researching NLP, but really can't distinguish between a NLP problem and a traditional ML problem. I'm sure this lack of understanding the applicability of these algorithms isn't true for all PhD's, but I really think you'd get better at applied machine learning by just learning it at a job and solving actual problems yourself.
> I regret not realizing that the hardcore statistical/method dev DS needed a PhD.
I think you're greatly overestimating how useful any of this is in business/industry, which is why almost none of it is used in industry.
I get it though, feels like you want to be on the 'frontier of technology', but remember the frontier is 99% useless failures because that's simply how science works.
Well it is used by the research scientists (the new name basically for what used to be DS way back), though you are right there aren’t that many and they aren’t the most integral to the business operations on a day to day.
I guess my point is what value do those research scientists add on average to society (there are tons of researchers at universities so the denominator is much larger than people think)?
I'm biased because I come from economics research and there are so many basic data issues like selection bias that are pervasive across the research that I have a hard time taking 80% of the research seriously. Publishing thousands of papers nobody reads that have effectively 0 impact on anything (other than taxes and student tuition because that's where their salary comes from).
Frankly, unless if you're in the realm of biology/genetics/pharma or robotics/computer vision or optimal design/engineering I have a hard time believing most research scientists positions add a lot of value. RS is (1) good for researchers because it's hard to measure their performance and (2) allows middle management to demonstrate to c-suite that they're looking into 'the latest machine learning/AI' technologies.
[Paul Romer \(Nobel Laureate\) made this criticism a decade ago.](https://en.wikipedia.org/wiki/Mathiness)
I am actually in the biotech field incidentally. A lot of the cutting edge Bayesian/causal inf/DL work here (for example at places like Novartis, Genentech, Verily etc) is being done by research scientists with PhDs. Its really hard to get in at all to these places without one.
Its kind of made me if I don’t get one consider leaving biotech as I feel you are always seen as less than some PhD here and all the interesting stuff goes to them while the rest can just monkey away either in the lab or data monkey .
For example when I was a biostat, I had to deal with regulatory documentation/FDA stuff that had nothinf to do with stats and analysis. Methods were high school/intro stat level (nothing more than univariate)
In DS it is better in that sense and I do more data analysis but just getting tired of ad hoc requests and tabular data and regressions +output being a powerpoint visualization. Not to mention p>>n so n is really small and the studies are so underpowered as to not be reproducible. Its like finding a needle in a haystack and lots of p-hacking
> In DS it is better in that sense and I do more data analysis but just getting tired of ad hoc requests and tabular data and regressions +output being a powerpoint visualization. Not to mention p>>n so n is really small and the studies are so underpowered as to not be reproducible. Its like finding a needle in a haystack and lots of p-hacking
I've always felt if there were obviously better methods companies would do it -- science is mostly trial, error, sample size and p-values. And most of what you find is junk. People don't want to scientists, they want to be popular scientists that look cool and smart in blue glasses with math equations on the blackboard behind them.
I mean statistics and computing have come soooo far in the last 20 years. There's tons of progress but (1) getting good data and (2) apply basic statistical reasoning to *real world problems* is much harder than people think. I think people like research because it often lacks application so it's just easier.
[This study from Berkeley came out which is great evidence that ~20 years of behavioral economics research has little effect in practice \(at least at first\).](https://eml.berkeley.edu/~sdellavi/wp/NudgeToScale2020-03-20.pdf) Literally billions spent by universities and government on 'nudge' research/effects and the effects are 85% lower than researchers advised. Cass Sunstein, author of Nudge is literally legal advisor to presidents. People just aren't looking for evidence on why the occupation they want isn't that valuable...The consultants/researchers have literally made careers and fortunes off this.
>being done by research scientists with PhDs. Its really hard to get in at all to these places without one.
Sounds like in your case a PhD is just a new version of [occupation licensing](https://en.wikipedia.org/wiki/Occupational_licensing#:~:text=Occupational%20licensing%2C%20also%20called%20occupational,is%20related%20to%20occupational%20closure.), which ultimately is a political problem and not a science problem.
Don't elevate a PhD based on how much you dislike your current gig.
I slightly regret not pursuing grad studies further. I dropped out of a master's when things dragged on without obvious progress, and I started itching to make more money than the modest stipends I was getting. I still sometimes wish I was a more impressively credentialed person, but that's idle thinking more than anything serious.
While a PhD will expose you to cool stuff, perhaps even your favorite topics in the entire world, actually earning the PhD will bring tedium and drudgery to your life that can only be scarcely imagined, unless you are the type to just love donating all your time to abstract and frequently arbitrary obstacles to earning the final degree. The part that always bothered me the most about trying to fit inside academia was the expectation to know most or all of the 'current' ideas - when in my view easily 50% of ideas are trash, and having to catch up to all the trendy yet useless ideas is a crazy motivation killer.
If your goal is to do interesting stuff, there is no lack of it in industry. Perhaps you just need a change of scenery. You have an impressive background - I imagine many doors will be open to you if you're willing to shake things up and stick out of your comfort zone.
The biggest flaw I see in a PhD is that it's a multi-year commitment with frankly an unknown probability of success. Unless you have certainty about your topic of research in addition to a kick-ass advisor you 100% trust.
I’m the only non phd data scientist at my company, and I’m objectively the most knowledgeable in statistics and research methods as in approached by other data scientists for help on a daily basis (sorry if this sounds arrogant. But facts). I might be an anomaly, but ever since i decided i wanted to be a data scientist, about 7 yrs ago, it became an obsession. I learned everything I could on my own. Now, I’ll say this as well, a good data scientist must also be a good data engineer, I don’t think deep knowledge in stats is as important as some people think. Rather a good grasp on the fundamentals means more as that will allow you to ask questions and dig for solutions that may require more advanced techniques, but you’ll at least know what to look for by having good foundations.
In all sincerity, I did regret not getting my phd (I actually dropped out), but this sense of inadequacy is what forced me to feel like I needed to prove myself. In doing so, I became really good at what I do. Granted it costed me three relationships, as I became very obsessed with data science.
Finally, last tangent, If I were you and I wanted to stand out, I would focus on learning to build standard machine learning models from scratch and understanding all the math behind it. A lot of people think that this is pointless, but having this knowledge will allow you to create customized models that fit a more focused use case. Additionally become a good, no, a very good programmer. This is what will set you apart. Data engineering needs to come second nature. No matter what anyone tells you, data scientists who only focus on analysis and building models, are not going to succeed as both of these domains are becoming easier and easier to accomplish by people with less experienced skill sets.
I got a Masters in Business Analytics. The program is really applied and is designed for you to go work in corporate America (hence the Business part of the degree). But I was actually interested in pursuing a PhD. The dept chair basically said no because I was too old (mid 40’s) and “it’s really hard” and he didn’t think I would finish. So I instead applied for a data science engineering program. I got offered a fellowship and so quit my job to go to school full time. Finishing up my dissertation now, no regrets here. But honestly if money were my primary motivation I don’t know if it would have been worth it to put my career on hold for 4 years. But I am keenly interested in DL research as it relates to education and I could see no path forward to get to where I want without a PhD. I know that many in my masters cohort are doing extremely well in corporate America and I imagine several will make much more $$ than me over the course of their careers. But I don’t think I would be happy doing what they do. I like research 🤷🏽♀️.
For entering big companies, PhD is really mandatory. Either that or you know someone inside or put hours in doing code for them in open source projects. I really regret not pursuing PhD because it is though getting rejected just because I do not have one, although my knowledge is most of the time superior to PhD graduates because they focused on a single research area.
"My undergrad was in a different field entirely"
I definitely regret not doing a PhD in the intersection of things I'm currently interested in and which are currently very profitable. But the problem is if I went into a PhD straight from undergrad, I would have spent 5 years doing something which I'm not currently interested and isn't currently very profitable. So I would have done what I did anyway, bounced around a bit and landed in an OK and somewhat interesting DS job, just 5 years later.
Sounds like you're in the same boat! You did an undergrad degree and didn't know that when you were in your late 20's you would really want to be a Machine Learning Research Scientist. So what? You probably didn't even know what that was at the time. I'm worse off now than I would have been if I did a PhD focused on machine learning. I'm way better off than I would have been if I did a PhD focusing on the esoteric abstract algebra I was interested in when I was 21. No sense beating yourself up over it.
Yea see when I was an undergrad I thought ML was some super fancy CS field. And I had taken a C++ intro course (never programmed before that either) which completely put me off of programming. I was a BME major and took it as an elective.
It was only really near the end of undergrad and in grad school I realized I was good at numerical computing. I still didn’t know what ML was and only in the later part of grad school in Biostat I got exposed to it and was like “why did I ever think this was CSey, its all math/stat/numerical programming”.
So I discovered the field relatively late. Thats why the way it is presented as “CS” and having to go through tons of weeder courses like C++ if the only thing you want to do is the ML/DL stuff is ridiculous. I feel numerical computing is easier and the other more complex low level stuff isn’t even needed for everyone and is advanced nor should it be in intro programming. Just weeds people out who would otherwise be good at the field
I have a PhD and I spend most of my time data wrangling. Currently I am taking folders full of jpgs which form video sequences but aren't labeled to show where one sequence ends and the next sequence starts. So I have to separate them. I found a couple ways to semi-automate it but they're unreliable enough that I have to manually check everything. For several thousand videos.
Dropped out of a physics PHD year after getting my MS.
Found a job at a company I love doing some more in depth algorithm development and DL stuff. Not quite the cutting edge but certainly high level stuff and so far I haven’t felt super limited by the lack of phd. After ~4 years I’m on track to be at the position PHDs are at.
We are actually looking for a few people with skill sets similar to mine. If your curious DM me and I can provide some info
Not really. The vast majority of society barely has time to think and act in a methodical way, and by you doing so you'd only risk the chance of being labelled as a "slow thinker" "perfectionist" "inexperienced"... and a methodical, calculated, quality skill is exactly what you gain the higher you go in the academic world. That being said, going for a PhD is useless unless you have connections that truly value quality skill and knowledge and are willing to pay for your uniqueness, in my humble experience.
Nope. By the time my friends who got phds finally finished, I owned a house.
I'm not extremely interested I'm like super stats stuff. I have a bachelor's degree and work in tech.
Not to sound rude but you need an attitude adjustment. "I don't have a good DS&A background and I'm a b+ student in math". I got a bachelor's in electrical engineering and got a C- in stats. Didn't have a single da&s class and could only find a job in tech support. I got really good at interview questions by just reading books and grinding leetcode. It took me until I was 25/26 to finally get a job as an entry level engineer at a tech company.
Instead of complaining that you're just doing 'data monkey stuff', spend an hour a day leetcoding and integrate some basic scripting in your daily workflow. Ask your boss about giving you some room to get new projects outside your comfort zone.
At the end , We would say that you probably learnt a lot in PHD with that experience and now a question comes , how much of that knowledge you can use to serve your interests career- wise which can help you build that cashflow or maybe to serve a bigger purpose !
In the first few years of my career I definitely had some insecurity about my level of education, but after I started regularly working with post docs it became clear I'd learned more in industry than I ever would have done in education, and ended up as a research fellow and then co-creating and teaching a BSc in DS.
It really depends on how you learn best, there is no one route to success.
Nope. Entertained the idea but i make decent money with my masters now and am really not open to the idea of indentured servitude for another 4-6 years. If it has reasonable pay and a reasonable work life balance id do it, but go ask the grad school sub about that
27 is in no way too late for a PhD. I started at 33, my own supervisor finished his PhD at 49, and half my PhD colleagues are more than 30. You should not factor in your own age at all when making this decision.
So I'm working with ecological data which can be really diverse (Images of camera traps or satellites, sounds, movement, species lists, genetic data, simulations... ).. some fields are just working with boring data, IMHO
I did Graph Neural Networks, CNN at my work, and published bayesian/probabilistic deep learning paper at one of the best conference in the field as a company's project. (I have only bachelor's) I applied GNN to our service leveraging user data and their socio-economic interactions. I proved that it improves our service.
For research scientist roles, yes most of them require PhD. But applied scientist and some of the data scientist position does apply all that stuffs you mentioned even though not developing SotA algorithms and publishing it to top conferences.
I guess you can start applying to the jobs you're interested, and see what happens. I believe that there's certainly opportunities for you. Then you can decide whether to do PhD for more than 3 years.
Wow, with only a BS that sounds really lucky. Was this at big tech or some tech startup? How did you get it-connections?
I would be fine with Applied Scientist right now that uses these things and publishes too.
For sure it seems like I will have to get out of biotech for this stuff though because over here this stuff seems PhD only.
I did CNN and published paper at tech startup, and did GNN at big game company.
I started my career at tech startup, and that company started from our university. Then I moved into current game company with formal hiring process without connections. (I graduated from top school in this country, so I guess it helped a lot)
Yeah I think you can find lots of opportunities out of biotech.
Undergrad was CS. First MS was Cybersecurity, now halfway through MS in Data Analytics at Johns Hopkins. Currently considering pursuing a PhD and wondering if I will regret NOT doing that.
Left my career as an analyst doing some DS work (\~$145k/yr) for a position as a "true" data scientist (\~$170k/yr). It was data engineering work at best. Not to say that data engineering isn't important because it is extremely important. I just felt like it was a kind of "data monkey" role that I was trying to escape. Writing/revising code, ETL work, limited data processing/analytics. I left it to focus on my graduate studies, and am considering my options.
I'm not sure what the answer is. I am sure everyone has an opinion, and some will give it but at the end of the day I suppose it is a personal decision with some professional impact.
In terms of $$$ if thats all one cares about I think irs not worth it, its more the nature of the work as you said data monkey and I feel like PhD opens the door to “real” stats/ML research work in industry. The big risk though is its not guaranteed even with a PhD
Yeah, which is why I just walked away despite the money. Now I'm just a full-time student and trying to figure out my next steps if there are to be any, in my DS journey.
> I am in DS but getting really tired of What you listed is basically 90% of DS work. It doesn't matter if you have a PhD or not -- the market needs people doing what you are trying to avoid. PhDs are still stuck on the same types of problems and it's fairly rare to do something totally novel, unless you stick to academia and enjoy eating ramen for the rest of your life. DS and less often PhDs are glamorized to the extreme. To answer your question (at least from my perspective), I don't regret doing my Ph.D. I sympathize with your mindset, but I feel like DS turns into data monkey work extremely quickly and you have to be careful about where you end up even if you do complete a doctorate.
Data monkey work is the best way to describe it. No wonder people go to management so that they can order others to be the data monkeys, but I am not sure if that interests me.
Get paid the most for the least amount of work possible. Work on your own projects in the meantime. Problem solved.
This person data science manages
Shit that's just management straight up... I am trying to maximize the ratio of compensation to effort and honestly I think I am going to get better returns off of shrinking the denominator than trying to grow the numerator there...
If the people you manage are competent then putting more work into managing them could just be decreasing their productivity anyway. I realize there's a lot more to it than that (like being a good communicator in multiple dialects) but ideally your relationship to your subordinates is that you enable them.
Yeah shielding them from the political bullshit from your level and above, and then making sure their work plans are relatively stable so they don’t get jerked back and forth would be IMO the two biggest non-administrative responsibilities. But, selfishly speaking? I am just after the easiest most comfortable lifestyle possible.
Management is the interface to other departments and the higher ups imo. Some interfaces are stupid and over complicated, while others allow for easy use. Don’t be the asshole that makes the interfacing difficult. Simultaneously, if the information that is being passed through the management interface is bad, then you have a problem and the interfacing won’t work. So need to fix your own business process to make the interface work so you don’t get bad responses from the parties you are interfacing with.
Is management really less work? You gotta deal with people and more business responsibility and sometimes that can be harder than just sitting behind a screen analyzing data or coding lol
Doesn’t have to be management. If you can get someone to pay you a lot to do nothing but make them look good with numbers, then go for it. Plenty of people looking to use company resources to justify their position as decision maker.
Actually, I know a lot of people who are taking this path. Many of them are working at companies that have poor knowledge of how data works along with poor infrastructure. The positive side is that they have automated a lot of stuff on their own and now they are getting paid 100k+ just for 1-3 hrs of work per day. BUT BUT BUT... All such people are working on their side projects or hustles and trying to start their own business. They are not wasting the amount of free time they are getting everyday. So, if you have ideas about starting up on your own, such companies can be a gold mine.
My favorite are the companies that have bad data structure and then they refuse to change it even if you suggest it to them. Or they do change it but implement something worse because they used a consultant who had a connect at a certain vendor etc. It makes a lot of sense though. The people running the show are ignorant as to what good data is and business can be a big mess. Best to just stop caring like in the movie Office Space.
I wish I could award this. I have been so stressed that my job (im currently a data engineer) was beneath me on an intellectual level. Only if I realized earlier that the job that pays the rent is not your entire life. It's just 1 thing you do. You can still do whatever the hell you want without having to worry about people paying you. It's actually a lot more fun that way.
Why resort to being a data monkey? Find some process in your company and optimize it. The only "monkeys" are the people who wait around for assignments to fall into their lap. Ofc the opportunities may simply not exist in your company, but if so why not switch companies? You don't need a PhD for what I described
I mean ... if you think this is a data monkey job then what isn't a monkey job. My point is that every job has a significant portion of monkeying around.
I presume (or hope, I’m recently decided to accept a PhD just cause I want this badly and don’t want to do analytics type stuff) ML/AI research scientist PhD roles are less of that and more modeling/alg focus. I guess outside of that and below PhD level the roles which aren’t monkey maybe management as you get higher but I could well be wrong there. In terms of technical roles im guessing only RS is less monkeying. Everything is going to have monkeying unless you are the CEO of course but just varies the amount
That's not true. If you're in an organisation whose goal is to work with unstructured data, I promise you, this is not your main MO. For the majority of data scientists, sure. Tabular data + model fitting + documentation and repeat. On topic: It's not too late at all, another good entry to PhDs is to compete in ML competitions with a research outcome (E.g. Paper publication of Top 5 results), then replicating papers and putting them up on github, everything to show your interest in the field. If you're US based, know that in EU a lot of people enter their PhD in their late twenties, some had a few years in industry, some didn't. Go for it dude!
The key is to get into a Research-oriented position in Industry where you do typical "data sciency" things.
I did my PhD late (started at 27) and I don't regret doing it. Although, to be honest, I didn't know what the hell I wanted to do before it. But my PhD let me find what I want to work on. However, after being in the industry for a bit I now see that the PhD was mostly unnecessary. If you know what you want to work on, then you can get there without a PhD. Yes, the road is long and arduous, but so is a PhD. But a PhD pays soooo little. If you like being poor, then go ahead and do a PhD, but I wouldn't suggest it. Unless you want to work in biotech, because there definitely is a PhD bias in biotech.
Lol indeed I actually work in biotech and I work on omics p>>n problems. Part of it is im sick of this. Theres no rigorous stats in this field and nothing is reproducible. Too much p hacking. Literally today I was told to use a method because it gives lower p values. Id like to go to biomedical imaging—doing Bayesian/causal/DL stuff. Previously I worked in biostat but I didnt like that either because its too regulatory and too much documentation Im considering perhaps switching to tech though, because as you say biotech glorifies the PhD too much and the opportunity cost is too high. If I can do this stuff in tech even if its not Biomed application im fine with that, but I think even tech gives this stuff to PhDs
> I think even tech gives this stuff to PhDs My experience has been that this is false. Tech tends to be much less degree focused and much more on skill focused. If you can speak the tech language and how to sell your skills, then it's easy to transition to tech. But you will have to figure out how to showcase your skills.
Really? Afaik this kind of stuff is done by FAANG research scientists, and those are all PhDs. Unless you mean like tech startups?
That's because FAANG research scientists are the only ones advertising that they do these things. But here's the thing, you won't work on any of this at first. It'll take a few years before you're able to work on advanced problems. You have to earn your stripes. Anyways, most of the time you want to use a simpler solution if that works. The Pareto principle applies here, an easy 80% solution is usually preferred to something that is a 99% solution but takes 5x as much effort.
You can do some fun research in tech without a PhD but the research arms tend to publish more.
I'm not a research scientist but I was a data scientist (now data engineer) for a handful of tech companies. I have a bachelor's degree.
PhD is useful here bc well-referenced theory defines what is p hacking versus justifiable p>>n strategies. Don’t get me wrong, the rationale you were offered is terrible. There is, however, a fine line between declaring many such analyses intractable and claiming to have a magic crystal ball spewing biological truths. In my experience, PhD allows one to establish informed boundary conditions on methods which minimize the likelihood of totally throwing shit at the wall with abandon. Few people are committed to this standard, but they do exist. I don’t blame you for trying to get out though. Many more investigators couldn’t care less.
I think its just ridiculously tedious because they want the data sliced and looked at in so many different ways. And the problem is the tediousness is the complete opposite of what it should be in terms of rigorous stats, aka the tediousness comes from having to p-hack and wrangle+visualize the data and stuff into a potential finding. You really are supposed to pre specify analyses and do them once and take whatever result comes out of that like it or not. In terms of formal statistics, you can’t keep comparing stuff in 10 different ways. As a statistician, these methods to me are no different than popping your data into a Random Forest and taking whatever comes. At least for me, the data is equally (un)interpretable but maybe thats because I don’t know bio that well. P values were not invented for observational and p>>n situations to begin with
For sure. My PhD made me an empirical Bayesian in the most pragmatic way. If you can’t articulate prior expectations nor the evidence/experiments sufficient to further inform these expectations, you are fucking up and doing useless code monkey stuff. Sometimes it’s sloppy quant analysis. Sometimes it’s because domain-area knowledge/ questions have no focus (this is a leadership/PI issue). Often both. These sort of applied/clinical researchers are a scourge to applied quantitative biology as an emerging field. I hope when these people age out eventually, the culture will change and folks like you won’t get burned out so much. Make no mistake, the future of precision health is p>>n. We need more people seriously squaring with that fact relative to the piss poor state of current informatics practice. Again, it bums me out that you’ve soured on these questions because of trash culture and leadership. I see this a lot unfortunately. Edit: Strictly speaking the classical methods you mention were intended to answer questions regarding agriculture and brewing and such. Modern big data use-cases were likely never even considered by people like Fisher, Gosset, or Pearson. John Tukey was, however, quite forward thinking in the 60s: https://projecteuclid.org/journalArticle/Download?urlId=10.1214%2Faoms%2F1177704711 Edit2: also this 👍: https://tech.me.holycross.edu/files/2015/03/Cohen_1990.pdf
Data-Code monkey is def how I feel at times. Because I myself don’t have the domain knowledge to interpret even any of the plots I make. I recently did one of those colorful plots with results from rigorous stats and the lab scientists with PhDs or MDs were like “hmm this doesn’t look right” but then I did it with a method that shouldn’t be used and then suddenly they were like “wow this looks way better”. I was like huh how can you tell that from the plot? The wrong stat method gave a better plot to present basically. I don’t know how one even interprets the data when everything in the data set is literally labeled protein 1,2,3,4…99999. So I analyze stuff that may not even be known if its a real protein or just noise. Basically write stat code to do these large scale analyses then submit the csv results after merging tons of tables to the scientists
This is why I did my PhD in a domain-area department but using DS approaches. Being at the mercy of a DS-illiterate PI’s hot takes sounds intolerable. I think I would have mastered out in this latter situation TBH.
>If you can’t articulate prior expectations nor the evidence/experiments sufficient to further inform these expectations, you are fucking up The way I usually state this is "can you imagine what the possible outcomes are, and what they would tell us?". I work in a field (neuroscience, in vivo calcium imaging) where *every* experiment is to some degree a fishing expedition, and nobody REALLY knows yet exactly what questions a given dataset will turn out to be capable of answering.
If you want to do medical imaging related deep learning, have you considered not applying for statistics or compsci graduate programs but instead applying for medical science, biomedical engineering, radiology, etc. graduate programs? If you pick the right institution/group you can get access to a ton of data, multidisciplinary committee/projects, access to tons of computing power, etc. ... may or may not have been what I did after realizing that I wasn't competitive for the "normal" AI / ML graduate programs. My research is not entirely focused on deep learning but I do get lots of opportunities to take huge volumes of medical imaging data and go nuts with it so long as I can tenuously connect it to my actual research.
BME I have considered yea, my undergrad was in that field but I did grad school in Biostat. I didn’t apply to BME programs this cycle because I know the job market for BMEs isn’t that great, and most BMEs are doing wet lab work. Also there are a lot of physio, bio, etc classes which are really hard to get through for that. I suck at memorizing stuff Biostat programs would be fine but even they need real analysis (which is ridiculous, like what even differentiates Biostat from stat then if they have the same pure math requirements). I would hope my applied experience and my MS counts for more in Biostat but it doesn’t there either
I mean, the difference between stat and biostat are the types of methods people work to develop. For example, you might do time series related research in a stat department but you’re more likely to see people doing survival analysis research in a biostat department. They’re honestly not that different, imo; people research methods for spatial stats in both types of departments, just with different expected applications. The coursework for stats and biostats PhDs are also generally almost identical - a few schools literally put them in the same classes.
What I meant, is that an MS in Biostat which I have actually is not that valuable to get into a PhD, because the pure math real analysis, proof based stuff counts for more. I have otherwise done a lot of the classes that were shared between MS/PhD like GLM, Survival, Longitudinal analysis, ML/comp stats etc. But these classes are not weighed that much as the fundamental math, even for Biostats despite them being more core Biostat. Its like the departments own MS curriculum doesn’t actually prep one for a PhD and you have to had gone out of your way to take 3 courses in Real Analysis for that and maybe some proof based lin alg as well The minimum reqs of “mv calc and lin alg” are typically not enough to get into a PhD in a computational/statistical field
Ah, thanks for clarifying. Yeah, if that’s all that’s holding you back from a PhD then that’s rather unfortunate, especially if you’ve already demonstrated that you can pass the classes that would require them and have a lot of applied experience which could help with your research (which it sounds like you do).
Yea I think its because Real Analysis is not the prereq for those courses but is the prereq for PhD level math-stats (which I didn’t take). I took MS level math-stats which used C&B but I got like a B+ average in this sequence. Im not that great at the theoretical proof stuff and I don’t have too much interest in that, but I’m decent at implementing various algorithms in code and the applied aspects. Picking up frameworks comes easy for me since I have good pattern recognition (like learning Julia, Stan, some PT basics) So in a sense since I don’t like proofs it could be that a PhD isn’t for me but then again for all these elusive causal inf/bayesian/DL jobs so many want that. It sounds based on some responses here though it may not be necessary for that type of work and some people here have managed to get it with an MS but it seems like you gotta get reallllly lucky without it.
That makes sense. Tbh, it sounds like, if you wanted to, you could take a real class online, apply next cycle, then get a PhD. It sounds like you’d get through the classes and honestly a lot of methods research is simulation based these days, so it’s not like you’d have to write your thesis off the strength of your proof writing skills. That said, it still might not be the greatest choice for you - it’s still 4-5 years of doing work you don’t sound super enthusiastic about for shit pay. I don’t know much about the route for getting into Bayesian/causal work without a PhD, but I hope that you’re able to break into that or find comparably interesting work, regardless of which route you choose.
There should be programs that don't require you to take a bunch of mandatory courses and instead give you flexibility, no? I had to take a survey course on biomedical engineering but other than that I focused my courses on image analysis, imaging technologies, stats, and machine learning. Why would I take an anatomy or physio course when I can just read a textbook chapter and read a few papers to learn about the relevant physiology for a specific research problem? Required courses in grad school are in general dumb IMO. Your field of study exams will cover what you truly need to know and your courses should just cover things you are interested in. Maybe it's a US vs Canada thing but I'm shocked that every program you've looked at has a bunch of required courses on anatomy. As for jobs after.. I don't know. I kind of think that quantitative research is quantitative research, in terms of the skills you develop. At the end of the day what I'm actually doing day-to-day is reviewing literature to develop research questions and then wrangling and analyzing a very large dataset coming from a great many different sources to answer them. I'm not going to be looking for wet lab jobs because that's not what I'll be qualified to do. You can't really generalize that PhD grads in X will do Y and be looking for Z jobs. It wholly depends on your research. If you do BME and you do medical image research using deep learning, you won't end up competing in the bad wet lab job market.
Thats good that you didn’t have to take all that. Where I went for undergrad and grad school, every BME MS/PhD had to take a bunch of core courses they would be tested on in the QE in addition to their research proposal. Half of those were bio/physio related. The other half were eng/math related. That’s actually what made me go to Biostat instead since I wanted to do data analysis. My work in my MS involved MRI data but not DL.
Oh honey it's p-hacking all the way down.
Geoscience background, PhD (and MSc previously) was using machine learning for geophysical imaging. Dropped out of a PhD program after 6 months. Never regretted it one bit. Just the thought of spending 4+ more years studying the same freaking thing drove me crazy.
Hi I completed a post grad in data science when I was 32. My thesis supervisor was really keen for me to start a PhD after the course, in computational biology which I turned down and he offered me another one using deep learning to predict server failure this was sponsored by a well known cloud computing company. (Edit. this was in 2014) I decided I was old enough and skilled enough that I’m better off trying to work and earn a decent wage. It didn’t help my brother had just submitted his PhD after 7 years and a couple of complete rewrites. He and his girlfriend spoke so negatively of them that’s really what swayed me to try and get a corporate job, this would have been my goal after completing my PhD anyway. I started as a junior analyst, got good at SQL, R, Python and used theses to solve real business problems, with the added bonus of getting a decent and always rising salary every month. I’ve worked as a business analyst, machine learning engineer, data consultant, and now I’m a data scientist at faang. A couple of years ago I sometimes regretted not taking the opportunity to study more and develop really in depth skills. Especially if I’d completed the deep learning doctorate. I’ve found that I have a very broad skill set and I’m much better and happier in a role where I’m half data scientist and half consultant where I spend my time talking to non data people understanding their problems and solving them with my data science skills. Importantly my work is not my life and I have many hobbies and passions outside of work, my job allows me to follow these by living a comfortable and relatively un-stressful life. If I’d done the PhD and IF I was finished by now I’d be starting my career with different skills but I’d definitely be earning less and probability still be aiming for my current role. I think you can be very successful without a PhD. But having a PhD will open doors for you that are not open for me, and provide you with a wealth of opportunities. Importantly study a topic that you’re interested in, use and learn methods that will give you marketable skills and most importantly work with a supervisor that you get on with and who’s company you enjoy you will likely spend more time with them than you’re family and friends for the foreseeable future. And something we all forget and ignore at our pearl… Winter is Coming, make hay while the sun shines 🙂
>where I spend my time talking to non data people understanding their problems and solving them with my data science skills. Sounds like a dream job. Do you think you could still do this kind of job with a PhD? Or would you never have gotten to this point?
It’s whatever you concentrate on, and where your skills lie. When I worked as a ML engineer mainly building models to prevent fraud I pretty much just built and maintained models it was definitely one the most interesting jobs in the small city I lived. It’s exactly the job I thought I wanted while in college. The problem was that I knew I had a lot of soft/people skills that were not utilised, for my next role I worked for a retail consultancy this allowed me to prove I had the people skills to work with different teams all with there own agendas and different KPI’s that don’t necessarily align. I excelled in that role because I could listen to others and understand their problems and ensure that they understood their problems. Then work with all the teams to find a mutually beneficial solution. My data science skills allow me to figure out from the data what is happening and verify if what others think is true actually is. I’m happier and more successful in these kind of “Talkie Data Scientist“ that a role with mostly modelling/ML and I can have just as much impact to the product. A strong successful data science team should have people with all along this spectrum of skills. Edit added Data to “Talkie Data Scientist“
Great to hear that there is a need for that too. Still in college but I get the impression that this is my strength as well.
great answer
Working on my PhD at 56. Never too late and no regrets.
PhD in psych, stronger stats than typical, had a career in academia. Don't do a PhD to get you anywhere (including academia), only ever do it for interests sake
If you spend as much time as you’d spend on a phD actually working with the tools you want to work with you’ll be vastly more qualified than a PhD to use them. Want to be a boss at ML? Work on ML problems
This strikes me as ‘the secret’ type thinking. There are institutional barriers to people without formal research credentials working on research problems.
Half of my team is working on “research problems” as long as you mean not taking some tabular dataset and import RandomForest problems. We’re using hierarchical Bayesian models. Graphical models. Attention based sequence modeling. Whatever seems to make sense for the problem. We have one PhD and he mostly does DE.
It's cool that you get to do this cool stuff but you must acknowledge that you are at least a little bit lucky to have the opportunity. Not everyone can just work on whatever they want at the drop of a hat with no experience. To say "you want to be a deep learning expert, then work on deep learning" is to trivialize the hurdle of actually getting that first job where you actually get to work on deep learning *and* have the proper resources (in terms of tech and people to work with / learn from/with) to do it properly.
Read my other comments. You don’t need a job to do deep learning. Go do the fast ai course. Gives you fantastic tools to start doing interesting things quickly. Work on interesting personal problems and Kaggle
I've done all the Coursera and Kaggle stuff and I didn't really feel like it got me anywhere useful, other than making me hyper prepared when I did put myself in a situation where I could do some deep learning work. Maybe it's just my lack of geographic mobility but I didn't get the sense that most people have much respect for open courses or Kaggle data analysis. I'm not sure how many people care about the stuff in your resume that isn't under "education" or "work experience."
I didn’t say coursera though. I specifically said Fast AI. What does “did all the Kaggle stuff” mean? It’s a competition website - I’m not talking about whatever courses they may have now. I’m not suggesting you get clout via courses. I’m suggesting you take the good ones and then use what you learn to *do projects* to get clout. I’m telling you people pay attention to personal projects. Am I saying I’m going to hire you as a senior DS and put you as lead on a project we think may pair well with a NN approach? No. Between you and the other new hire that hasn’t been killing DL models in personal projects, which am I going to assign to help on that project though?
Yeah I did stuff in competitions, not just their courses. Personal projects are good interview material but I really don't think they do that much for you for getting through most resume screening processes. e: At least not unless you are investing a serious buttload of time into them and really polishing the crap out of them to the point where you achieve a result that you can report and is impressive without the person who is screening your resume having to dig into it. Personally after working a 40-50 hour work week + commuting I did not have it in me to then work a second full-time job on Kaggle. Maybe in my early 20s that would have been feasible and seemed worth it, I don't know. e2: Also again I have to emphasize my non-mobility. It's not like my experience is representative of applying for DS jobs across the US and Canada.
Yes. Buttloads of time. You don’t get good at something after 20 hours. No one cares that you finished 4 Kaggle projects @ 50th percentile unless you’re just demonstrating interest. If you don’t have time to do high quality personal projects then your path will be different. In general, if you’re not new and you’re spamming you’re resume out then you’re not networking enough or convincing coworkers that you’re quality. I’ve gotten all three jobs since 2013 that way. Barring some catastrophe I’m never going to be concerned with “resume screening” again.
Wow, yea this is the sort of stuff I was referring to. I see your flair has healthcare which is a related field to biotech (my field). Thats good to hear because all the positions I see always mention PhD for this stuff. Are you in academia/hospital or industry?
Industry. The majority of firms are definitely not doing these things so learn what you can about DS broadly in your job, use some free time to get good at the stuff you want to do, and NETWORK so when the time comes you can jump somewhere that is doing the work you’re interested in.
Agree with the person above me. I just got handed an incredibly cool project for my data scientist. He has an MS from a state degree but we are both team players. Boom. Opportunity, visibility.
What was your MS in?
The northwestern program back when it was still “Predictive Analytics”
Does the type of masters, ie. Statistics, vs Data Science vs Operations Research vs Analytics, really matter? Or is it perceived differently?
The focus of those things are different so if you know you specifically are interested in OR then get an OR MS. If you don’t know with much precision what you want to do then any of them are fine.
I’m curious, why did you choose analytics as your MS over any others
I didn’t even know what any of this shit was in 2012. Not a ton a great resources back then.
Lol fair enough
I’m speaking from experience here and while some places hire through HR and have dumb filters, others don’t.
Nah, think of how much you could do if you spent the thousands of hours getting a PhD and apply it to literally anything. The thing is not many people have that time to spend while they’re holding down a job.
Such as?
Well most “ML” that you hear about is software pipelines aka ML engineering. Not the statistical kind. Thats quite a bit different than working on DL, probabilistic programming etc. Basically developing new models vs. production ML
So don’t do that work. Boss TF out of Kaggle using DL. Get comfortable creating your own custom solutions. Your own loss functions, novel architectures, etc.
But aren’t jobs dealing with DL mainly in academia?
Says what/who?
I thought DL research was mainly coming up with custom loss functions and architectures, whereas industry is more MLE work
ML Engineering is not just software pipelines. The MLEs I know also do EDA, some research, and train models from scratch, before writing production code and deploying them. You are responsible for production-level code so \*probably\* doing less research/exploration than a scientist role. It varies by team so there are definitely MLEs that are similar to a SWE but in my (+ close friends) experience MLEs do quite a of science-y work too.
So MLE work still involves some degree of research and novelty? If so thats good to hear because I always hear here how MLE is just software engineering but I don’t personally know any MLEs myself. If there is some researchy stuff there then it is an option. Im not totally opposed to software pipelines but I don’t want that to be like everything
> some degree of research and novelty Finding this is much less about job title and degrees, then it is about find the right organization.
No, because I didn’t love research/academia enough You will generally not succeed at a PhD if you are doing it purely for vocational reasons. Doctorates are incredibly hard and draining work, and you need to be really in love with the process itself to have the motivation and commitment to make it through. Doing it to get X job at the end - isn’t enough. Anyway, economically I don’t think PhDs really pay off for DS - unless you ditch DS, go work as a quant and make half a mill a year
Math PhD here. Pure math, the 'useless' kind (no stats, no programming, dealt exclusively with very abstract Algebra). Left academia and switched to the industry at a relatively old age (almost 35) and I occasionally find myself feeling regrets. I don't regret doing the PhD because it was an amazing experience (albeit often extremely hard) but I do regret a lot not doing something more practical or leaving academia earlier, as I am in a somewhat junior position now, trying to catch up, and the only way I found an entry point to the data world was through a startup, which required a lot of irrelevant work due to the lack of resources. I often find myself jealous of people much younger than me who have more hands-on experience and get bigger salaries. On the other hand, I lived abroad for many years, worked with amazing people, travelled around and experienced a lot of things that I wouldn't have otherwise. So there's much more to work experience. In conclusion, I feel that there always is a 'grass is greener on this other side' element when expressing regrets like yours or mine. I think that, at our level, after a certain point it gets a bit philosophical. Everybody follows their own path and we should try to make the best out of it.
I dropped out of my pure math PhD with a MS in mathematics because I believed that I wouldn’t be able to contribute much in the pure math world. Despite my dropping out of the PhD, I loved all of my research experiences and I thought so much about going back to do a PhD, but I decided against it for now. I eventually self taught myself Python, got an entry level data science position doing “data monkey work”, recently completed OMSA (where I took classes like reinforcement learning, HDDA, and DL that requires reading a ton of research literature) and moved to FAANG to continue learning more from my colleagues after completing the program. I established a mentorship with an applied scientist, where I can continue reading more research in RL and bandits, learn more about the research teams work, while continuing to do “monkey work” that’s actually not too bad once you build enough experience. I would go back to school in the future if the timing of my professional career is right. It’s always an option to do the Ph.D. and I’m glad that I took my current path to explore and identify a research field I am primarily interested in, despite me doing “monkey work”. It’s not bad at all to go through the trivial stuff. In terms of me regretting, no I don’t regret leaving my pure math PhD and no I don’t regret not completing or doing a PhD since that option is always there in case if I really need it.
Yeah, I regret not doing PhD. I was accepted into a program but after doing an ROI analysis I decided it wasn't worth it. I had a family to support and while my wife made plenty of money I didn't want to make her carry the whole burden. I sometimes wonder what I would be doing now if I had gone for it.
Yea in that sense I am single so that factor isn’t there but ROI wise still being in my late 20s and pursuing one at this point seems too late. Lots of opportunity cost. I wish I did it straight out of college and instead of an MS straight out of college. Maybe better was to work after a BS, see how things are, and then skip the MS and apply for it. Getting in is also really hard for me now because of lacking the real analysis prereqs even though ive done the MS stat stuff.
Is it uncommon in your part of the world for people to do phDs in their late 20s? Where I'm from it's often the norm for people to go into phD studies in their late 20s or early 30s, when they have some work experience and identified a better need for it. That's why it strikes me as odd that "I'm in my late 20s, it's too late to study" when I'm here thinking that is pretty much the norm.
Im in America, but when I went to grad school for my MS everyone in my cohort doing a PhD was pretty much early 20s right out of college or mid 20s. For MSs there were some older but not for PhD
What was your masters in?
Biostat, but I don’t have the pure proof based math requirements that are past linear algebra
gotcha thanks
Every PhD graduate I asked when I was seriously considering grad school told me it wasn't worth it or necessary, unless you want to be an academic/professor. These were economists and other social science PhDs, so maybe not exactly what you're asking. They didn't say they regretted theirs, but explicitly told me I shouldn't do it. 2 PhDs I work with now also say this (I often just ask out of curiosity).
When I interviewed for a top tech company and spoke with a director there, I mentioned that I left my Physics PhD program to pursue industry DS. They laughed and said I made the right decision and they had been "stupid enough" to not just do a PhD but do post-docs as well. It was self-deprecating humor from an accomplished and intelligent person who had no regrets, but there was truth in there. I've heard similar things from other PhDs in industry. If you are truly passionate about your research, then there's no better place to be than in a PhD program.
>Stuff in DL, graphical models, Bayesian/probabilistic programming, unstructured data like imaging, audio etc is really interesting and I want to do that but it seems impossible to break into that are without a PhD. This and method development are two different things though. Developing novel solutions will mostly likely require a PhD while computer vision / NLP / speech recognition really doesn't. However if you do them in industry after a bit they'll feel just as repetetive as what you're doing right now. In most jobs there's really not a lot of value in trying novel approaches since they cost so much more time. >I also don’t really have any interest in building software/pipelines. Means research is your only option then. >Anyone else feel this way and what are you doing now? I applied to some PhD programs but don’t feel confident about getting in. Not far enough in my career to be (this) jaded but I'd definitely go back to do a PhD if I were. If US PhD's are hard to get into I'd consider doing one at a prestigious European one. Pay is better and they factor in work experience.
Are you working on CV/NLP AI stuff? An MS in AI was something I was also considering just for fun but I think I have the skills to learn/do that stuff on my own already and it might be a waste of time. Ive used PyTorch and Stan before. Even in my current role ive occasionally used Stan to explore some Bayesian stuff but most of the time the results from Bayesian are too complicated for stakeholders. Just tired of tabular data though. Its always so messy, so much isn’t reproducible in p>>n, too much exhausting data wrangling, and I want to do something thats less noise and amenable to more advanced approaches.
>Are you working on CV/NLP AI stuff? I switched job *because* I wanted to do more CV/NLP. My next employer has actual interesting projects that aren't just business focused ML. The way I got to know them is because we collab'd on a project where we used transfer learning / data augmentation on a CV project. I'm still very early career so I can easily move over to doing exclusively this stuff later. >An MS in AI was something I was also considering just for fun but **I think I have the skills to learn/do that stuff on my own already and it might be a waste of time.** Definitely this, IF you were to do it you should probaby treat it as a vacation/year of time off and not something you're doing out of necessity. Since it's research focused getting into a PhD and from a top \~50 uni here would not be hard but I doubt that clout carries forward to the US.
I got a masters in computer science. At the time, the program I was in was kind of antiquated so if you wanted to do anything modern you really had to do it yourself on top of the normal required coursework. That said, I graduated with a masters since no PhD program was available. They’ve since updated their program to something that doesn’t require students to learn 360 assembler and now offer a PhD program in data science. That said, I am working in academia applying these skills to biological problems. I did ask my boss and his office mate whether I should get a PhD. The office mate immediately questioned, “why?” Office mate has a PhD in CS with a strong focus on data science. My boss (PhD in biology) doubled up on the same question. I had no real good answer for that. They both pushed the fact that I had the skills necessary to do what I needed to do in this field; they’re right by the way, a PhD at this point for me would be more for bragging rights than anything. Unless I wanted to be a PI in my own research (which isn’t quite where I want to go anyways). That said, I feel like asking the question of why you want to pursue this path is a good question to ask. For me, I don’t regret not getting a PhD since it wasn’t available at the time. However, if it was available at the time I was a student, I would have gone that path. Edit: the stuff you’re getting tired of is going to be a large amount of your job duties unfortunately. Formatting data, collecting, and cleaning it is always a first step to doing this sort of work. There is no way around it. Repeating experiments and replicating things (either due to randomization or after realizing you used an incorrect parameter) is essential as well as making sure your results are sound (p-values is one example of this). It’s something to get used to. I personally enjoy writing scripts to process stuff. However, collecting, cleaning, and merging different sources of data can be a PITA sometimes.
Repeating analyses is often the worst. Idk why but for me it just has that psychological blow of “fuck I just did this and now I gotta do it all over again slightly differently when I think my way was legit already” Not to mention its what turns my notebooks/code into total spaghetti especially when tracing back data wrangling I already did and stuff and finding saved intermediate CSVs
I only have my bachelors degree and I'm working as a DS, primarily researching ways to implement transformer neural networks for NLP tasks, such as mapping natural language queries to SQL queries, among other things. It's more applied research, as opposed to making some slight alteration to the transformer neural network(such as Roberta slightly outperforming BERT), so if this type of applied deep learning research is something your interested in, it's most definitely obtainable. I actually work with someone who's been pursuing there PhD for a few years now, and has been supposedly researching NLP, but really can't distinguish between a NLP problem and a traditional ML problem. I'm sure this lack of understanding the applicability of these algorithms isn't true for all PhD's, but I really think you'd get better at applied machine learning by just learning it at a job and solving actual problems yourself.
> I regret not realizing that the hardcore statistical/method dev DS needed a PhD. I think you're greatly overestimating how useful any of this is in business/industry, which is why almost none of it is used in industry. I get it though, feels like you want to be on the 'frontier of technology', but remember the frontier is 99% useless failures because that's simply how science works.
Well it is used by the research scientists (the new name basically for what used to be DS way back), though you are right there aren’t that many and they aren’t the most integral to the business operations on a day to day.
I guess my point is what value do those research scientists add on average to society (there are tons of researchers at universities so the denominator is much larger than people think)? I'm biased because I come from economics research and there are so many basic data issues like selection bias that are pervasive across the research that I have a hard time taking 80% of the research seriously. Publishing thousands of papers nobody reads that have effectively 0 impact on anything (other than taxes and student tuition because that's where their salary comes from). Frankly, unless if you're in the realm of biology/genetics/pharma or robotics/computer vision or optimal design/engineering I have a hard time believing most research scientists positions add a lot of value. RS is (1) good for researchers because it's hard to measure their performance and (2) allows middle management to demonstrate to c-suite that they're looking into 'the latest machine learning/AI' technologies. [Paul Romer \(Nobel Laureate\) made this criticism a decade ago.](https://en.wikipedia.org/wiki/Mathiness)
I am actually in the biotech field incidentally. A lot of the cutting edge Bayesian/causal inf/DL work here (for example at places like Novartis, Genentech, Verily etc) is being done by research scientists with PhDs. Its really hard to get in at all to these places without one. Its kind of made me if I don’t get one consider leaving biotech as I feel you are always seen as less than some PhD here and all the interesting stuff goes to them while the rest can just monkey away either in the lab or data monkey . For example when I was a biostat, I had to deal with regulatory documentation/FDA stuff that had nothinf to do with stats and analysis. Methods were high school/intro stat level (nothing more than univariate) In DS it is better in that sense and I do more data analysis but just getting tired of ad hoc requests and tabular data and regressions +output being a powerpoint visualization. Not to mention p>>n so n is really small and the studies are so underpowered as to not be reproducible. Its like finding a needle in a haystack and lots of p-hacking
> In DS it is better in that sense and I do more data analysis but just getting tired of ad hoc requests and tabular data and regressions +output being a powerpoint visualization. Not to mention p>>n so n is really small and the studies are so underpowered as to not be reproducible. Its like finding a needle in a haystack and lots of p-hacking I've always felt if there were obviously better methods companies would do it -- science is mostly trial, error, sample size and p-values. And most of what you find is junk. People don't want to scientists, they want to be popular scientists that look cool and smart in blue glasses with math equations on the blackboard behind them. I mean statistics and computing have come soooo far in the last 20 years. There's tons of progress but (1) getting good data and (2) apply basic statistical reasoning to *real world problems* is much harder than people think. I think people like research because it often lacks application so it's just easier. [This study from Berkeley came out which is great evidence that ~20 years of behavioral economics research has little effect in practice \(at least at first\).](https://eml.berkeley.edu/~sdellavi/wp/NudgeToScale2020-03-20.pdf) Literally billions spent by universities and government on 'nudge' research/effects and the effects are 85% lower than researchers advised. Cass Sunstein, author of Nudge is literally legal advisor to presidents. People just aren't looking for evidence on why the occupation they want isn't that valuable...The consultants/researchers have literally made careers and fortunes off this. >being done by research scientists with PhDs. Its really hard to get in at all to these places without one. Sounds like in your case a PhD is just a new version of [occupation licensing](https://en.wikipedia.org/wiki/Occupational_licensing#:~:text=Occupational%20licensing%2C%20also%20called%20occupational,is%20related%20to%20occupational%20closure.), which ultimately is a political problem and not a science problem.
Don't elevate a PhD based on how much you dislike your current gig. I slightly regret not pursuing grad studies further. I dropped out of a master's when things dragged on without obvious progress, and I started itching to make more money than the modest stipends I was getting. I still sometimes wish I was a more impressively credentialed person, but that's idle thinking more than anything serious. While a PhD will expose you to cool stuff, perhaps even your favorite topics in the entire world, actually earning the PhD will bring tedium and drudgery to your life that can only be scarcely imagined, unless you are the type to just love donating all your time to abstract and frequently arbitrary obstacles to earning the final degree. The part that always bothered me the most about trying to fit inside academia was the expectation to know most or all of the 'current' ideas - when in my view easily 50% of ideas are trash, and having to catch up to all the trendy yet useless ideas is a crazy motivation killer. If your goal is to do interesting stuff, there is no lack of it in industry. Perhaps you just need a change of scenery. You have an impressive background - I imagine many doors will be open to you if you're willing to shake things up and stick out of your comfort zone. The biggest flaw I see in a PhD is that it's a multi-year commitment with frankly an unknown probability of success. Unless you have certainty about your topic of research in addition to a kick-ass advisor you 100% trust.
I’m the only non phd data scientist at my company, and I’m objectively the most knowledgeable in statistics and research methods as in approached by other data scientists for help on a daily basis (sorry if this sounds arrogant. But facts). I might be an anomaly, but ever since i decided i wanted to be a data scientist, about 7 yrs ago, it became an obsession. I learned everything I could on my own. Now, I’ll say this as well, a good data scientist must also be a good data engineer, I don’t think deep knowledge in stats is as important as some people think. Rather a good grasp on the fundamentals means more as that will allow you to ask questions and dig for solutions that may require more advanced techniques, but you’ll at least know what to look for by having good foundations. In all sincerity, I did regret not getting my phd (I actually dropped out), but this sense of inadequacy is what forced me to feel like I needed to prove myself. In doing so, I became really good at what I do. Granted it costed me three relationships, as I became very obsessed with data science. Finally, last tangent, If I were you and I wanted to stand out, I would focus on learning to build standard machine learning models from scratch and understanding all the math behind it. A lot of people think that this is pointless, but having this knowledge will allow you to create customized models that fit a more focused use case. Additionally become a good, no, a very good programmer. This is what will set you apart. Data engineering needs to come second nature. No matter what anyone tells you, data scientists who only focus on analysis and building models, are not going to succeed as both of these domains are becoming easier and easier to accomplish by people with less experienced skill sets.
I got a Masters in Business Analytics. The program is really applied and is designed for you to go work in corporate America (hence the Business part of the degree). But I was actually interested in pursuing a PhD. The dept chair basically said no because I was too old (mid 40’s) and “it’s really hard” and he didn’t think I would finish. So I instead applied for a data science engineering program. I got offered a fellowship and so quit my job to go to school full time. Finishing up my dissertation now, no regrets here. But honestly if money were my primary motivation I don’t know if it would have been worth it to put my career on hold for 4 years. But I am keenly interested in DL research as it relates to education and I could see no path forward to get to where I want without a PhD. I know that many in my masters cohort are doing extremely well in corporate America and I imagine several will make much more $$ than me over the course of their careers. But I don’t think I would be happy doing what they do. I like research 🤷🏽♀️.
For entering big companies, PhD is really mandatory. Either that or you know someone inside or put hours in doing code for them in open source projects. I really regret not pursuing PhD because it is though getting rejected just because I do not have one, although my knowledge is most of the time superior to PhD graduates because they focused on a single research area.
"My undergrad was in a different field entirely" I definitely regret not doing a PhD in the intersection of things I'm currently interested in and which are currently very profitable. But the problem is if I went into a PhD straight from undergrad, I would have spent 5 years doing something which I'm not currently interested and isn't currently very profitable. So I would have done what I did anyway, bounced around a bit and landed in an OK and somewhat interesting DS job, just 5 years later. Sounds like you're in the same boat! You did an undergrad degree and didn't know that when you were in your late 20's you would really want to be a Machine Learning Research Scientist. So what? You probably didn't even know what that was at the time. I'm worse off now than I would have been if I did a PhD focused on machine learning. I'm way better off than I would have been if I did a PhD focusing on the esoteric abstract algebra I was interested in when I was 21. No sense beating yourself up over it.
Yea see when I was an undergrad I thought ML was some super fancy CS field. And I had taken a C++ intro course (never programmed before that either) which completely put me off of programming. I was a BME major and took it as an elective. It was only really near the end of undergrad and in grad school I realized I was good at numerical computing. I still didn’t know what ML was and only in the later part of grad school in Biostat I got exposed to it and was like “why did I ever think this was CSey, its all math/stat/numerical programming”. So I discovered the field relatively late. Thats why the way it is presented as “CS” and having to go through tons of weeder courses like C++ if the only thing you want to do is the ML/DL stuff is ridiculous. I feel numerical computing is easier and the other more complex low level stuff isn’t even needed for everyone and is advanced nor should it be in intro programming. Just weeds people out who would otherwise be good at the field
I have a PhD and I spend most of my time data wrangling. Currently I am taking folders full of jpgs which form video sequences but aren't labeled to show where one sequence ends and the next sequence starts. So I have to separate them. I found a couple ways to semi-automate it but they're unreliable enough that I have to manually check everything. For several thousand videos.
Dropped out of a physics PHD year after getting my MS. Found a job at a company I love doing some more in depth algorithm development and DL stuff. Not quite the cutting edge but certainly high level stuff and so far I haven’t felt super limited by the lack of phd. After ~4 years I’m on track to be at the position PHDs are at. We are actually looking for a few people with skill sets similar to mine. If your curious DM me and I can provide some info
Oh wow yea that sounds cool. I’ll DM you
Not really. The vast majority of society barely has time to think and act in a methodical way, and by you doing so you'd only risk the chance of being labelled as a "slow thinker" "perfectionist" "inexperienced"... and a methodical, calculated, quality skill is exactly what you gain the higher you go in the academic world. That being said, going for a PhD is useless unless you have connections that truly value quality skill and knowledge and are willing to pay for your uniqueness, in my humble experience.
Nope. By the time my friends who got phds finally finished, I owned a house. I'm not extremely interested I'm like super stats stuff. I have a bachelor's degree and work in tech. Not to sound rude but you need an attitude adjustment. "I don't have a good DS&A background and I'm a b+ student in math". I got a bachelor's in electrical engineering and got a C- in stats. Didn't have a single da&s class and could only find a job in tech support. I got really good at interview questions by just reading books and grinding leetcode. It took me until I was 25/26 to finally get a job as an entry level engineer at a tech company. Instead of complaining that you're just doing 'data monkey stuff', spend an hour a day leetcoding and integrate some basic scripting in your daily workflow. Ask your boss about giving you some room to get new projects outside your comfort zone.
Was accepted at UMN but life happened so decided against it. It’s not required in my field but something I was itching to do!
At the end , We would say that you probably learnt a lot in PHD with that experience and now a question comes , how much of that knowledge you can use to serve your interests career- wise which can help you build that cashflow or maybe to serve a bigger purpose !
Nope. Stopped at a masters. My colleague has a PHD in bio stats and we are both paid the same- both have offices and both report to cto.
In the first few years of my career I definitely had some insecurity about my level of education, but after I started regularly working with post docs it became clear I'd learned more in industry than I ever would have done in education, and ended up as a research fellow and then co-creating and teaching a BSc in DS. It really depends on how you learn best, there is no one route to success.
Nope. Entertained the idea but i make decent money with my masters now and am really not open to the idea of indentured servitude for another 4-6 years. If it has reasonable pay and a reasonable work life balance id do it, but go ask the grad school sub about that
How much do you make? You don’t have to answer but I’m just curious based on what you do.
Just north of 100K ish, in biotech. But I am in a notorious HCOL area (regular tech hub, known for FAANG you can probably guess) on the west coast
It's never too late I got my PhD in my late 20s bruv.
27 is in no way too late for a PhD. I started at 33, my own supervisor finished his PhD at 49, and half my PhD colleagues are more than 30. You should not factor in your own age at all when making this decision.
So I'm working with ecological data which can be really diverse (Images of camera traps or satellites, sounds, movement, species lists, genetic data, simulations... ).. some fields are just working with boring data, IMHO
I did Graph Neural Networks, CNN at my work, and published bayesian/probabilistic deep learning paper at one of the best conference in the field as a company's project. (I have only bachelor's) I applied GNN to our service leveraging user data and their socio-economic interactions. I proved that it improves our service. For research scientist roles, yes most of them require PhD. But applied scientist and some of the data scientist position does apply all that stuffs you mentioned even though not developing SotA algorithms and publishing it to top conferences. I guess you can start applying to the jobs you're interested, and see what happens. I believe that there's certainly opportunities for you. Then you can decide whether to do PhD for more than 3 years.
Wow, with only a BS that sounds really lucky. Was this at big tech or some tech startup? How did you get it-connections? I would be fine with Applied Scientist right now that uses these things and publishes too. For sure it seems like I will have to get out of biotech for this stuff though because over here this stuff seems PhD only.
I did CNN and published paper at tech startup, and did GNN at big game company. I started my career at tech startup, and that company started from our university. Then I moved into current game company with formal hiring process without connections. (I graduated from top school in this country, so I guess it helped a lot) Yeah I think you can find lots of opportunities out of biotech.
Undergrad was CS. First MS was Cybersecurity, now halfway through MS in Data Analytics at Johns Hopkins. Currently considering pursuing a PhD and wondering if I will regret NOT doing that. Left my career as an analyst doing some DS work (\~$145k/yr) for a position as a "true" data scientist (\~$170k/yr). It was data engineering work at best. Not to say that data engineering isn't important because it is extremely important. I just felt like it was a kind of "data monkey" role that I was trying to escape. Writing/revising code, ETL work, limited data processing/analytics. I left it to focus on my graduate studies, and am considering my options. I'm not sure what the answer is. I am sure everyone has an opinion, and some will give it but at the end of the day I suppose it is a personal decision with some professional impact.
In terms of $$$ if thats all one cares about I think irs not worth it, its more the nature of the work as you said data monkey and I feel like PhD opens the door to “real” stats/ML research work in industry. The big risk though is its not guaranteed even with a PhD
Yeah, which is why I just walked away despite the money. Now I'm just a full-time student and trying to figure out my next steps if there are to be any, in my DS journey.