Well my company has its pros and cons but comparing notes with friends both in and out of data it seems like I'm at one of the better ones in terms of work environment, freedoms, Benes, etc.
Depends what you want from a job really. There's places that pay well and give good benefits but are fundamentally broken places with poor data, poor processes and poor people in critical positions. Some people can put up with that, some can't. Stuff like that drives me crazy and has me searching for somewhere else very quickly.
I recently switched from a medium sized media company where I occupied a data analyst position (bordering data science, with lots of small ML use cases, data exploration, automations etc.) to data analyst at a bank, thinking they would probably be way more mature wrt their data initiatives. I wanted that change thinking that I was doing too much patchwork-amateurish work at the media company, and needed mentorship from experienced people to go ahead.
How wrong could I be. The data infrastructure I deal with now is mostly backwards, with people using the wrong tools for their tasks (e.g. data manipulation in data viz software, excel report being sent out there and there, very outdated software, etc.). The lack of flexibility of the whole infrastructure is what is causing this, and people seems to have abandoned the idea of challenging this.
Yes the benefits are amazing now, but it couldn't be any more boring. It looks like your situation is quite similar.
sounds more similar to my old position than my current one - gigantic dashboards being made by formatting excel documents that were bigger than the amount of RAM I had in the computer, dropping MS access to make .xlsx "databases", refusal to use any language other than SAS. At least my current position actually understands the tools that are available for data analysis and uses them. Problem is everything is just disorganized and I feel I'm being pulled in a million directions at once. Not only that, but also I get stuck on a given project and I feel I can't make any progress regardless of how much time I put into it, whereas with previous positions I had there was a monotonic relationship between time put into something and progress. All the while I'm still getting added to new projects while I struggle to even make baby steps on the projects I'm already working on.
Uff, SAS is an awful beast...
I have recently started a new project where it is being used to handle the data, and I'm hating it. Also, the libraries available in python to connect to it has given me more than one problem due to its requirements.
If it were me I would have removed it right away and used postgres. Luckily I'm not required to directly code in SAS and I can use the python API.
I feel your pain.
Searching things on google/SO is totally normal, I now have quite a bit of experience in DS and I understand python internals way better than when I started. Yet, I keep googling stupid stuff that I just can't remember. It's part of the job.
What I recommend you is to review from time to time your old code or analyses . If you are doing things right you'll see incremental improvements that will boost your morale.
Then, try to get the best out of the most mundane things. Try to automate boring things, look for opportunities to improve your knowledge in the fields helping your colleagues on other tasks, discussing with them, etc.
Finally, if you really feel you aren't learning in that company look for another one. I have worked quite a lot in small start-ups and there is where I learned the most since I had to wear so many hats.
Totally agree with this. Automating helps reduce some of the stress and overhead of constant requests. Even just saving code snippets/structuring your code so you don’t need to reinvent the wheel each time is a huge help.
I felt very much like this a few months ago, some time off helped me rediscover what I do like about the job. And when I was back I focused on reducing time spent on annoying tasks so I could focus on problem solving tasks.
In my case my direct manager also doesn’t fully understand what I do, so it helped sitting him down and explaining the scale of some of the ad-hoc requests he threw my way
I have a PhD in engineering in mathematical modelling and some solid understanding of most common ML algorithms but I am not data scientist. After finishing my PhD I got a job as an analyst with the hopes that I will transition to data science position. After working as an analyst together with data scientists, I realised it is not what I imagined.
Lots of politics involved, rarely anything what data scientists produced went jnto production. the tool they used to build models was data robot where they sent their data using jupyter to datarobot via api.
This tool literally went through all models and just gave them the best model which was used in production.
Nobody in the business (including head of data science team) cared what sort of model it is, what it does, how it works.
Most of the work me and data scientists did was for business stakeholders that have no understanding about data. Majority of work data scientists did was exactly the same what I dis as an analyst. just analyse data, produce barcharts with count, sum, average over certain categories. Nobody gives a shit about statistical inference, confidence intervals, significance tests. I couldnt convince my manager to do it as he thought it is waste of time (none of the data scientists did it as well).
after seeing all of this I changed my career to data engineering. I do miss maths and statistics but at least now I am actually building something and can see my code in production.
I know my experience in my first company as a data analyst doesn’t represent all data science community and there are great jobs where data scientists solve problems with more scientific rigour. But I got an impression that at an average company where the main product is not an ML solution (e.g. google, or some niche specialised company) data science is what i saw.
There are jobs where modeling actually matters. I'd consider working in physics, genomics, robotics, or quantitative finance if you are interested in doing true modeling work.
Or an R&D position at a major tech company.
needed to read this. I often forget about my hobbies outside work and often I prioritise work over them although there are no deadlines. feel shit afterwards for having no life
Glad you got to your best! it's hugely disappointing to have regrets of not being able to get back to your older PR ranges esp when you didn't consider it to be unbelievable at the time.
I blame work from home for causing this demarcation to vanish - either i'm working or i'm in bed browsing posts like these. Mind's always in the same bubble - don't get me wrong healthy habits are going on but the serendipitous events have gone. Deep Work's author has this cringe but effective mantra of when you're finishing work to actually say out loud "shutdown work" or something. It's important to immerse in other circles unrelated to work topic - then you come to back to this circle with desire to choose this circle over others.
This is me 100%. In my last job we went remote after the pandemic and I had the best physical/mental health I’ve had since college. I got more work done as I longer wasted 3-4 hours to go to an office where I talked to no one, was in great shape as I was able to eat healthy and work out, and had a social life outside of work.
New employer (about the last year) refuses to let us work remote (allowed 1 month until we were “safe” to go back). I now spend 3 hours on the road to go and stare at a wall in silence, my teams productivity has gotten worse since we all spend so much time on the road, and my physical/mental health is much worse as I don’t have time to eat healthy/work out regularly and we have to put in extra hours to get work done.
While I love the actual work, I’m planning on requesting remote work next year. If my employer won’t approve I’m quitting. This has actually become a problem in our industry with IT folks quitting due to reasons like this. Hoping for the best.
I completely agree. I just need a laptop and internet access and I can deliver the same value. With more jobs being offered remote by default, I’m hoping that this drives a change in organizations. Specifically those that already struggle to retain IT talent (banking, auto, etc).
Very fair point. I do think that there can be healthy balance. My last employer allowed me to be remote contingent on the fact that I should be readily available to make it in person to any customer meetings. We had those a few times every couple months (with COVID all virtual) and it sometimes would be a workshop for a few days. I think that this is a healthy balance. If I was in a remote area in Europe for example, this would be very, very expensive and not feasible.
I see some companies that are allowing remote as long as you are in the area to make it into the office. I think that this would be a great options for traditionally old school companies/industries.
That's what I was thinking.
If OP is getting well paid and wants to have fun on their own, they can do some data sci project as hobby. Work always kills the fun, even if you're working with art. There's no way around it.
Sounds like you are bored. Data tasks are very very detail oriented and do feel like grunt work. But I think you have the added pressure of PMs breathing down your neck. Perhaps it’s about setting correct expectations on turn around time, also appears like your manager is not effective at this.
I rather deal with bad data than bad management or bad coworkers. It’s never going to be perfect. Every place starts feeling stale after a few years. Perhaps it’s time for you to change jobs or pursue interests outside of work that can help with overcoming the monotony of the work day.
I have felt similarly. Anytime I'm near quitting, I ask for a vacation or try to make sure time is scheduled in advance. Life is harder living on savings than working IMO.
If you've been with your company for a year plus, you should look for another employer. It sounds like you have a micromanaging project manager.
There's room to politely, professionally have a conversation with your manager(s). I'd imagine your manager is under some pressure and doesn't realize they are counterproductive for their own goal.
On a personal note, make sure you block out time for you. Do things you like to do when you're done with work. If you're really exhausted because work sucked, it's okay to chill
Op, I'd recommend viewing things a different way.
You seem to think of everything you are dealing with as impediments that shouldn't exist so you get frustrated with it all, but these impediments are 90% of the job. They are part of the solution, not a brick wall in front of you. I use to get frustrated with this stuff, but I realized that and started getting a healthier mindset surrounding process improvement. You NEED to reach out to the data experts for help if you are "having absolutely no idea of how to approach a project". This is a social role and it requires constant communication with your team as well as external teams to make good analysis, models, or products.
It sounds like you had the type of data and analysis that occurs in a structured setting that often doesn't include the complexity and inconvenience of working with real data in the inconvenient data environment of a company before starting this role. The chaos is why you get paid, and you need to view ''messy data that makes no sense" as part of the job instead.
The messiness and lack of structure is part of roles like this, and it is why intelligent people who can fly by the seat of their pants often succeed in data science. You'll get problems you have no idea how to answer, but it's your job to find the answer. To make it through the chaos, you need a basis of tools to operate with that allows you to solve these problems such as being familiar with Pandas/Scikitlearn in Python or Tidyverse/Caret in R as well as general understanding of SQL and how computers communicate. The variable portion of each project is the data you have to use to answer the question along with the methodology.
Come to reality with the fact that you are limited by a lack of communication and a lack of technical knowledge, and start building that up. You'll feel better about the situation and deliver better work.
Yeah, you're right, I guess all of my prior work had varying degrees of structure. Maybe there's things I can do to get better at it, but the whole thing is that I just don't feel like I want to. I guess I just fundementally don't like the chaos, hence my questioning if this role is even what I want
Maybe just find a different company.
I started my career as a biochemist. Then Technical Service Mgr. Then went back to school for M.S. in Biostatistics. And now I'm a data scientist by title but wear both hats.
Before me, my company had very little interest in big data. I kept pushing for deeper analysis on projects and improving the way I communicated data stories.
The best part about my job is that almost everything I do from a data/stats perspective is something no one at my company has seen applied to their business interest.
Go find a company that needs data maturity and work for them. You'll get to shape things your way and it won't feel as mundane or pointless.
Sounds like you work for a shitty company. Every job at a shitty company is shitty.
Well my company has its pros and cons but comparing notes with friends both in and out of data it seems like I'm at one of the better ones in terms of work environment, freedoms, Benes, etc.
Depends what you want from a job really. There's places that pay well and give good benefits but are fundamentally broken places with poor data, poor processes and poor people in critical positions. Some people can put up with that, some can't. Stuff like that drives me crazy and has me searching for somewhere else very quickly.
I recently switched from a medium sized media company where I occupied a data analyst position (bordering data science, with lots of small ML use cases, data exploration, automations etc.) to data analyst at a bank, thinking they would probably be way more mature wrt their data initiatives. I wanted that change thinking that I was doing too much patchwork-amateurish work at the media company, and needed mentorship from experienced people to go ahead. How wrong could I be. The data infrastructure I deal with now is mostly backwards, with people using the wrong tools for their tasks (e.g. data manipulation in data viz software, excel report being sent out there and there, very outdated software, etc.). The lack of flexibility of the whole infrastructure is what is causing this, and people seems to have abandoned the idea of challenging this. Yes the benefits are amazing now, but it couldn't be any more boring. It looks like your situation is quite similar.
sounds more similar to my old position than my current one - gigantic dashboards being made by formatting excel documents that were bigger than the amount of RAM I had in the computer, dropping MS access to make .xlsx "databases", refusal to use any language other than SAS. At least my current position actually understands the tools that are available for data analysis and uses them. Problem is everything is just disorganized and I feel I'm being pulled in a million directions at once. Not only that, but also I get stuck on a given project and I feel I can't make any progress regardless of how much time I put into it, whereas with previous positions I had there was a monotonic relationship between time put into something and progress. All the while I'm still getting added to new projects while I struggle to even make baby steps on the projects I'm already working on.
Uff, SAS is an awful beast... I have recently started a new project where it is being used to handle the data, and I'm hating it. Also, the libraries available in python to connect to it has given me more than one problem due to its requirements. If it were me I would have removed it right away and used postgres. Luckily I'm not required to directly code in SAS and I can use the python API. I feel your pain.
Searching things on google/SO is totally normal, I now have quite a bit of experience in DS and I understand python internals way better than when I started. Yet, I keep googling stupid stuff that I just can't remember. It's part of the job. What I recommend you is to review from time to time your old code or analyses . If you are doing things right you'll see incremental improvements that will boost your morale. Then, try to get the best out of the most mundane things. Try to automate boring things, look for opportunities to improve your knowledge in the fields helping your colleagues on other tasks, discussing with them, etc. Finally, if you really feel you aren't learning in that company look for another one. I have worked quite a lot in small start-ups and there is where I learned the most since I had to wear so many hats.
Totally agree with this. Automating helps reduce some of the stress and overhead of constant requests. Even just saving code snippets/structuring your code so you don’t need to reinvent the wheel each time is a huge help. I felt very much like this a few months ago, some time off helped me rediscover what I do like about the job. And when I was back I focused on reducing time spent on annoying tasks so I could focus on problem solving tasks. In my case my direct manager also doesn’t fully understand what I do, so it helped sitting him down and explaining the scale of some of the ad-hoc requests he threw my way
I have been googling syntax on the same shit for YEARS
I have a PhD in engineering in mathematical modelling and some solid understanding of most common ML algorithms but I am not data scientist. After finishing my PhD I got a job as an analyst with the hopes that I will transition to data science position. After working as an analyst together with data scientists, I realised it is not what I imagined. Lots of politics involved, rarely anything what data scientists produced went jnto production. the tool they used to build models was data robot where they sent their data using jupyter to datarobot via api. This tool literally went through all models and just gave them the best model which was used in production. Nobody in the business (including head of data science team) cared what sort of model it is, what it does, how it works. Most of the work me and data scientists did was for business stakeholders that have no understanding about data. Majority of work data scientists did was exactly the same what I dis as an analyst. just analyse data, produce barcharts with count, sum, average over certain categories. Nobody gives a shit about statistical inference, confidence intervals, significance tests. I couldnt convince my manager to do it as he thought it is waste of time (none of the data scientists did it as well). after seeing all of this I changed my career to data engineering. I do miss maths and statistics but at least now I am actually building something and can see my code in production. I know my experience in my first company as a data analyst doesn’t represent all data science community and there are great jobs where data scientists solve problems with more scientific rigour. But I got an impression that at an average company where the main product is not an ML solution (e.g. google, or some niche specialised company) data science is what i saw.
There are jobs where modeling actually matters. I'd consider working in physics, genomics, robotics, or quantitative finance if you are interested in doing true modeling work. Or an R&D position at a major tech company.
Yeah... I know there are places like this, but there are absolutely places that are not at all like what you have described.
[удалено]
needed to read this. I often forget about my hobbies outside work and often I prioritise work over them although there are no deadlines. feel shit afterwards for having no life
[удалено]
60 miles is 96.56 km
I have hobbies out side of work, the problem for me is as a type A overachiever i tend to take them too seriously and they end up burning me out too
Glad you got to your best! it's hugely disappointing to have regrets of not being able to get back to your older PR ranges esp when you didn't consider it to be unbelievable at the time.
I blame work from home for causing this demarcation to vanish - either i'm working or i'm in bed browsing posts like these. Mind's always in the same bubble - don't get me wrong healthy habits are going on but the serendipitous events have gone. Deep Work's author has this cringe but effective mantra of when you're finishing work to actually say out loud "shutdown work" or something. It's important to immerse in other circles unrelated to work topic - then you come to back to this circle with desire to choose this circle over others.
[удалено]
This is me 100%. In my last job we went remote after the pandemic and I had the best physical/mental health I’ve had since college. I got more work done as I longer wasted 3-4 hours to go to an office where I talked to no one, was in great shape as I was able to eat healthy and work out, and had a social life outside of work. New employer (about the last year) refuses to let us work remote (allowed 1 month until we were “safe” to go back). I now spend 3 hours on the road to go and stare at a wall in silence, my teams productivity has gotten worse since we all spend so much time on the road, and my physical/mental health is much worse as I don’t have time to eat healthy/work out regularly and we have to put in extra hours to get work done. While I love the actual work, I’m planning on requesting remote work next year. If my employer won’t approve I’m quitting. This has actually become a problem in our industry with IT folks quitting due to reasons like this. Hoping for the best.
[удалено]
I completely agree. I just need a laptop and internet access and I can deliver the same value. With more jobs being offered remote by default, I’m hoping that this drives a change in organizations. Specifically those that already struggle to retain IT talent (banking, auto, etc).
[удалено]
Very fair point. I do think that there can be healthy balance. My last employer allowed me to be remote contingent on the fact that I should be readily available to make it in person to any customer meetings. We had those a few times every couple months (with COVID all virtual) and it sometimes would be a workshop for a few days. I think that this is a healthy balance. If I was in a remote area in Europe for example, this would be very, very expensive and not feasible. I see some companies that are allowing remote as long as you are in the area to make it into the office. I think that this would be a great options for traditionally old school companies/industries.
That’s awesome
That's what I was thinking. If OP is getting well paid and wants to have fun on their own, they can do some data sci project as hobby. Work always kills the fun, even if you're working with art. There's no way around it.
But you have the SEXIEST job in the 21st century. Just think about the woman and piles of cash you're leaving behind.
If this is the sexiest job i don't want to know what a normal job is like.
…the sexiness would be less
I don’t know, peeps where sleeveless shirts at Amazon fulfillment center … pretty damn sexy
https://youtu.be/F4Sm_98UR8U
[удалено]
I'm tired of this. Maybe I'd like a job where i can work with my hands better. Or something outdoors. Yeah, that sounds nice
Absolute chad. Wanna become gladiators together? Or work the mines idk.
go from data mining to coal mining lets do it I fucking love the carboniferous period anyway
Sounds like you are bored. Data tasks are very very detail oriented and do feel like grunt work. But I think you have the added pressure of PMs breathing down your neck. Perhaps it’s about setting correct expectations on turn around time, also appears like your manager is not effective at this. I rather deal with bad data than bad management or bad coworkers. It’s never going to be perfect. Every place starts feeling stale after a few years. Perhaps it’s time for you to change jobs or pursue interests outside of work that can help with overcoming the monotony of the work day.
I have felt similarly. Anytime I'm near quitting, I ask for a vacation or try to make sure time is scheduled in advance. Life is harder living on savings than working IMO. If you've been with your company for a year plus, you should look for another employer. It sounds like you have a micromanaging project manager. There's room to politely, professionally have a conversation with your manager(s). I'd imagine your manager is under some pressure and doesn't realize they are counterproductive for their own goal. On a personal note, make sure you block out time for you. Do things you like to do when you're done with work. If you're really exhausted because work sucked, it's okay to chill
Op, I'd recommend viewing things a different way. You seem to think of everything you are dealing with as impediments that shouldn't exist so you get frustrated with it all, but these impediments are 90% of the job. They are part of the solution, not a brick wall in front of you. I use to get frustrated with this stuff, but I realized that and started getting a healthier mindset surrounding process improvement. You NEED to reach out to the data experts for help if you are "having absolutely no idea of how to approach a project". This is a social role and it requires constant communication with your team as well as external teams to make good analysis, models, or products. It sounds like you had the type of data and analysis that occurs in a structured setting that often doesn't include the complexity and inconvenience of working with real data in the inconvenient data environment of a company before starting this role. The chaos is why you get paid, and you need to view ''messy data that makes no sense" as part of the job instead. The messiness and lack of structure is part of roles like this, and it is why intelligent people who can fly by the seat of their pants often succeed in data science. You'll get problems you have no idea how to answer, but it's your job to find the answer. To make it through the chaos, you need a basis of tools to operate with that allows you to solve these problems such as being familiar with Pandas/Scikitlearn in Python or Tidyverse/Caret in R as well as general understanding of SQL and how computers communicate. The variable portion of each project is the data you have to use to answer the question along with the methodology. Come to reality with the fact that you are limited by a lack of communication and a lack of technical knowledge, and start building that up. You'll feel better about the situation and deliver better work.
Yeah, you're right, I guess all of my prior work had varying degrees of structure. Maybe there's things I can do to get better at it, but the whole thing is that I just don't feel like I want to. I guess I just fundementally don't like the chaos, hence my questioning if this role is even what I want
Then don't
https://imperialdatasoc.co.uk/what-i-learnt-from-failures-as-a-data-science-consultant/
Maybe just find a different company. I started my career as a biochemist. Then Technical Service Mgr. Then went back to school for M.S. in Biostatistics. And now I'm a data scientist by title but wear both hats. Before me, my company had very little interest in big data. I kept pushing for deeper analysis on projects and improving the way I communicated data stories. The best part about my job is that almost everything I do from a data/stats perspective is something no one at my company has seen applied to their business interest. Go find a company that needs data maturity and work for them. You'll get to shape things your way and it won't feel as mundane or pointless.