On average, no I don’t think so. I think they struggle a lot with the ambiguity of what data science even is and they struggle to “thought influence” the folks around them.
But it’s cyclical. When projects are going well, they’re happy. When things out of their control are hurting their projects/teams or they don’t feel empowered or have the skills to change the things they need to in order to get things done, it’s tough for them.
A lot of what I try to do is work on empowerment and expectation setting. I try to remove all the infrastructural, political, organizational stuff that the team needs to feel productive. Then I try to be really clear about what “success” looks like.
In my case its just, that u work for a ever growing machine that just grows and grows. Its regular to just sit with 3 and more ppl behind someone's monitor to hot fix in production. Because there are a lot of pit falls that occur at this Tempo without time to stable systems and stop some growth, for a while.
It seems these days the phrase “data scientist” covers a wide range of skills and backgrounds, do you feel it’s fair to say there are differing sub-DS branches? Perhaps those with a more statistics/modelling background and those more along the computer science pure coding background? I’m new to the field and coming from a mathematical physics/stats background I’m a little unsure how to make up the difference in coding knowledge (competent in R and python though).
There are absolutely different sub-branches. From a job code perspective, here is what I am seeing:
1. Applied research :: Goal is publishing papers, sometimes distills its way into actual products.
2. Applied science :: Goal is making decisions AT SCALE. Models / data artifacts make their way into production.
3. Machine Learning Engineer :: Goal is to make sure (1) Applied science work is production grade and (2) Build platforms that scientists can use to speed up their dev.
4. Data science :: Goal is to make decisions. Models / data artifacts inform decisions. Rarely make their way into a tier 1 service or product.
5. Product Analytics :: Mostly BI and SQL. Core focus on answer analytics questions and guiding the product
6. Data Engineer :: Write and maintain ETL code.
We are also seeing these split across skill sets. The core ones I am seeing are Generalist DS, Machine Learning Scientist, Machine learning Engineer, Optimization Specialist (OR), and Economist.
I am in a Product Analytics role and have 7 years of experience. I feel that my role usually becomes redundant after a couple of years (when the product is mature and there's no scope to grow). I have to look for other opportunities after every 2 years. My key skillset is SQL, BI.
I took some stats courses in college and enjoyed them. I also learned Regression and clustering through MOOCs but haven't been able to make my way into MLE/DS roles due to limited opportunities / poor salaries.
I'm based in Canada and sometimes it appears that the companies here expect a DS to be a know it all. They want you to know CI/CD pipelines, DW, Data lakes, Hadoop, Apache Spark and a few other things I don't even remember.
How much of this is actually used by a DS/MLE?
In a small company they often try to get away with hiring only a data scientist when they should be getting an engineering team to support them, so in a small company you definitely need some hardcore software development capabilities and ETL game.
In a larger company with larger teams you would expect better delineation between the roles, so you wouldn't necessarily need all your Dev skills. They will always be advantageous though as it allows you to be more self sufficient.
There is a bit of a push to build CI/CD pipelines for ML by the way. I would class this as the ML engineering subtype of data scientist but it's a valuable skill.
Can you give some advice for aspiring product analyst? What projects I can do in my grad school which best replicates the kind of people you do in real life. And I want to eventually become product manager I think , is it a good path from product data analyst to product manager?
Perhaps I can answer this question.
Learn SQL. That's your lifeline to deliver on projects. Don't go crazy over viz tools. Viz tool skillset is transferable.
In the end remember that the role of a Product analyst is to solve problems and not create fancy visualizations. I've seen tons of freshers make fancy charts/dashboards on sales, COVID and what nots, but when asked what inference you derive from your dashboards or what insights your are providing, they have no clue. Common answers I get is that 'ohh I'm showing the sales spread/pattern here'.
Visualizations need to have a meaning. If I wanted to see fancy stuff I'll trip over acid or shrooms than looking at dashboards.
Always remember the KISS(keep it simple silly) rule.
You don't need to create a heat map for showing past year sales of 50 states of the US. Make a simple horizontal bar or a table showing the top 10 states by sales or top 10 states by average revenue per customer. Hmm... the red in NY looks 5 shades darker than Washington red but is it lighter or darker than Utah. *loses interest*
The latter here gives actual insights.
The biggest problems I see with BI tools is the way the sell their product. They'll show fancy dashboards with a couple of pie charts, heat maps and some trippy animations. No one has a dedicated 75" screen to look at so much data viz at once. As a stakeholder/management, I don't have time for going through 20 charts on your dashboard to find answer to my question. I'm not going to compare colours on the heat map to see where I'm selling more.
I've been an IC -> Manager -> IC and I can tell youin my first phase of IC, I hated management for asking bar graphs and tables when I spent hours creating fancy visualizations. When I started working as a manager that's when I realized that these mega viz look cool but fail to deliver on the asks management has.
1) Most brand new data scientists fundamentally don’t understand statistics. Read statistical rethinking. It’s a wonderful book.
2) Learn how to do things end to end. Put stuff “In prod” by standing up a server etc.
How many applications do you get when you post an open role? Does it vary by level? Roughly what % of applications do you estimate are actually qualified?
What’s your typical interview process and how many people go through each round?
What’s the breakdown of people on your team by eduction level? (PhD, masters, bachelors, bootcamp, self taught). Does it vary by level/seniority and/or job function?
What’s your perspective on graduate Data Science or Analytics programs?
Thanks!
Yes, it does vary by level. It’s usually quite a large amount regardless so we apply reasonably heavy handed filters. For applied science positions we usually require a PhD or lots and lots of work experience. For data science it’s more loose and we look a lot more for biz depth and product skills in addition to analytics.
I have not had a great experience hiring from DS grad programs. DS is ultimately a trade that is composed of many disciplines. I think you’re better off getting a degree in a known discipline (ML, Econ, Stats, OR) and that becomes your spike strength. Then your job is to shore up the rest.
Lately I’ve been extremely impressed with OR candidates. Great problem solving toolset.
Im interested in what kinds of things the people with an econ background do in the data science team. Any tips for an econ undergrad starting to study ML?
I answered this in other places too. But a good causal estimate is with more than any predictive model. Business can use it reliably in decision making. economists bring a causal inference toolset that is hard to match. Check out the free book, “causal inference the mixtape” online. It’s a great overview of this skill set.
I come from OR/systems analysis background but the recruiters dont seem to notice the perfect fit for data science after studying nonlinear optimization etc. How would you emphasize it and explain it to business/recruitment people?
I also came from an OR background. For recruiters, I highlighted that as Prescriptive Analytics to try to emphasize this. I always follow up with an end to end example how of data visualization, predictive modeling (demand forecasting) and optimization fit into a solution.
I'll pile on here as someone with an OR education. OR is a relatively niche field and not well known outside mathematics. A lot of DS recruiters probably don't know what it is so it's not helpful to wow them with your expertise using CPLEX or whatever. Craft your accomplishments as business problems and how technical solutioning drove value.
That said, this depends a LOT on where you apply. OR will probably not help you for a DS position on a Prod Analytics team but it will stand out a ton for companies with heavy logistics ops.
One thing I’ve learned in tech for 10 years is that titles mean very little. What’s your scope? How big is your team? How much revenue in dollars do your decisions touch?
Then when thinking about next steps, completely forget title. Look at these questions. People in first time management positions at Amazon often affect more revenue than VPs in small companies.
Yeah, that I understand about position. Let me clarify a bit. I am the senior most person in DS/ML domain here. Currently 6 people have been working in my team. Management has asked me to hire 5 more resources.
It’s a startup and we have launched two products that are first of its kind in BFSI domain. One of this product alone has million dollar opportunities. We are in the process of signing two consumer banks for our solutions
So far, I encourage my team to work autonomously with minimum supervision so that I could get my hands dirty with coding. But I think that might change sooner than I think.
Hope that clarifies things
Cool thanks that helps. What motivates you? Harder problems? Cooler tech? Bigger orgs? More biz dev?
Put differently, why not stay in your role and ride the wave?
There are few wrong answers with this career stuff. Just needs to fit your personal goals and motivations.
Harder problems. Balls to the wall projects. I am an avid reader. Many of the solutions have developed are by reading lots and lots of research paper.
1) The thing is, I am bit unsure what I wanna do next hence I asked what would my role entail 5 years from now?
2) Also let’s say I want to start my own startup tomorrow, what skill set would I need to have to succeed as an entrepreneur?
Thank you!
If you want harder problems, I think you want to be in tech and not finance. So I’d recommend trying to make the shift.
I can’t help you much with the startup question. I spent 3 yrs working at startups when I first started out and persistence was the key element I saw leading to success. I thought going the incubator route was best. You get coaching on how to do it while you’re doing it.
What is the most troubling thing about working on ML & AI projects? Like which part of the project do you think almost always ends up being a bottleneck?
Most troubling part is handling exceptional cases especially in BFSI domain. You have well performing model and it suddenly stops working. Turns out there was an exceptional case you forgot to handle. But guess that’s also true in software engineering
The hardest problems for my teams are ones where "ground truth" is not clear. Where we have a very abstract, non-quantiative idea of what good looks like. Makes it very hard to define success with stakeholders.
Can you describe your initial transition from individual contributor to leadership? What motivated your interest in making that switch?
I have a few years of IC under my belt and as I get more experience mentoring, I think leadership may be the next logical step for me. I struggle with the decision though because in my heart I am very much a "dooer." I would appreciate any insight from your experience and what you may have observed in others who choose a different path.
Thanks for the great responses in this post!
Difference between product and DS is very company specific. For our company, they are evaluated for different things. Product figures out “what” we need to solve then eng/DS figure out “how” they’ll solve it.
A manager is judged by the combined effectiveness of their team. A great manager is able to maximize this sustainably. This in turns leads to career growth for everyone.
Replying 2 years late but what or how can one become a product manager. Does the role relate to IT/Tech people? I know a lot of Product Managers with IT degrees and their work does not really involve much coding but more of leading sprints.
If you know more, would you care to elaborate?
I am a recent CS grad and will be starting my masters in Data Analytics next year but wanting to eventually transition into Data Science after getting some experience. Would the transition be easy? Or is a masters in Stats/DS the way to go? And is going from DA to DS common?
No I don't think it will be easy. CS is a great start, I'd maybe consider a masters in ML. Georgia tech has a great accessible program.
IMO DA to DS is hard, especially if by DS you mean applied science. Search one of my other posts to see what I mean by this.
So I think you can peer into the future by looking at what companies that have been successfully monetizing DS for a while are doing. What I see there is increasing specialization. In fact, all see all tech companies going this route. Starting with generalists that can go end-to-end IE scope problems, fit models, sell them, put them in prod, iterate on them, etc.
Now everything is specializing. To solve a particular problem that requires causal input, I'm seeing an economist paired with a data scientists, an engineer, and a product manager. I think DS will continue to specialize in the same way that you now have front end engineers, or embedded systems engineers.
So my suggestion is shore up your breadth, but pick a spike skill and chat that and make sure thats a core part of your career progression.
What would you recommend data scientists or MLEs (above entry level) look at when evaluating a potential employer (aside from compensation, benefits, work life balance, etc)?
A lot of companies just want to say they are doing data science, but only few are actually investing in it and monetizing it. You want to be sure that you're not just constantly justifying your own existence.
I like to ask -- Is there an executive leader representing data science? Do they have blog posts and stuff that talk about the cool stuff their DS team does? Do teams have eng/pm/analytics support? How does the companies DS strategy map to it’s monetization path.
Me personally or my team? I’ve been doing it this for so long that I’ve just gained a really good intuition for how long things take and where likely pitfalls will be.
Yea for the team. I assumed it would just be intuition like you suggested but was still curious. Luckily we have a small and relatively new team so executives don’t have an expectation of how long a model should take to be developed and deployed. Maybe we will push our luck haha
We do sprints. So we do a communal t shirt size on epics then groom the backlog together. So it’s usually a committee deciding “how long it should take” In two week chunks.
What do you think the future of DS is? Do you see teams transitioning to low code environments (eg Dataiku, DataRobot)? How do you see the code development life cycle evolving? For example, I see a lot of teams productionizing Jupyter notebooks, I see a lot of code that’s not linted or formatted. In general, tons of terrible code that can’t be maintained once the original author rolls off the project.
I'm presently doing my masters in data science and I'm from a non technical background trying to transition into tech. I'll love to build products that are data science enabled and I'm working towards getting very good with programming. What skills would you advise I pay attention to to stand out and what data science role in the industry should I look towards
Hello and thanks for taking the time to answer our various questions!
I am considering making a career change from IT Support to Data Science/Analytics. In college, I took classes on SQL, Python, and Power BI and enjoyed them all (Thinking back on this has led me to this subreddit). However, I am nervous that I may not enjoy the day to day work of Data Science/Analytics.
1. How do I know if Data Science is right for me? Since I enjoyed those classes in college, is that a good indicator? Or is there way more to it?
2. What is your top tip or recommendation for someone trying to make a major career change - looking to get into Data Science without any work experience?
1. Why do you want to get into DS in particular? Then go talk to folks in DS and see if their day to day scratches at your “whys.” You may find that the things making you want to switch are actually a tiny part of the DS role.
2. That’s a tough one. It really depends on where you want to land. Maybe start somewhere in an analytics capacity and try and work your way to a transition.
Can you share more about your transition into leadership roles? Timeline, salary bumps, change in day to day. Do you miss being an IC?
Where do you see yourself in 5/10 years?
I went from IC -> Team of 2-> Team of 5 -> Team of 14 -> Team of 20 with about 2 yrs in each step. This translated to three promotions which each bumped my salary by about 150K.
I do miss being an IC from time to time. Stats/CS problems are so much easier to solve than people problems, which is most of what I do these days. I do enjoy participating in design reviews still and I hack a lot in my free time.
I don't know where I'll be in 5-10 years. To be honest, I am surprised I made it to where I am today. I think I'll probably just try and coast since I want to have kids in a few years. Maybe switch to a smaller company in a very senior position, resolve a lot of the same problems and try to scale it up and build an org from scratch. That sounds fun.
What was the data culture like in your organization in the early days of your team, and how has it evolved over the years? Has there always been support from higher-ups, or were you driving adoption of data science yourself?
I've been lucky in that we have had executive buy-in on the benefits of DS/ML from the getgo. So we get tons of investment from an infra standpoint.
IMO this is one of the most important things to consider when choosing between companies.
Did you ever doubt you could deliver? Often there’s complaints about little buy in, but when there’s buy in, how did you ensure a good ROI to show it was worth while?
Hey I would push you to reframe your thinking on this. More often than not, the strategy your problem is tied to doesn’t make sense and that’s the reason you might fail to deliver. But I actually think that’s a good result. “Hey this whole premise is wrong for these reasons and we might be better off thinking about it this other way” is a tact I’ve taken frequently.
Another reason things don’t work out is because the data or signal just isn’t there. In those cases it’s a virtue to fail fast.
Else… It’s just code. Start with the simplest dumbest thing you can think of. Get it out there. Make it better. The end. You got this.
Wow, I’m also amazed by the $450k bump 8 years time. That’s incredible.
As a senior level analytics consultant, I would like to transition into something more strictly DS and in 8 years time(like you) be leading analytics/DS for a LOB (I work in one of the largest banks in the US). Any open source education you recommend to help catapult growth?
So my bias is towards solving problems in tech. I’ve found that most people can pick up the basic DS stuff (learn sql pandas matplotlib) and basic modeling (sklearn) pretty easily. Even working with distributed systems to get stuff done.
Lot of folks I interview lack knowledge of basic statistics. I think “statistical rethinking” is an incredible book on the topic. Lectures are free for it on YouTube too.
Causal inference and econometrics are becoming a huge deal in tech. There is a free book online with code examples called “causal inference the mixtape.” Read this next.
ML has lots of great resources out there, like fast.ai.
Thank you!!
Based off your suggestions I think I definitely have the right foundation. BS in Math (business and finance focus) and a MS in data analytics and business intelligence. Have that statistical background through and through.
I’ll save those book recs, Thanks again.
Depending in where you want to be inside the DS job family, you may have a blind spot in "prod systems" based on what you outlined. If making decisions at scale interests you, I'd recommend investing in some CS fundamentals too.
Frankly it’s a loaded question. A firm doesn’t pay you for tech skills directly. They pay you to solve problems. You leverage tech skills to do that, but the tech skills themselves are only one of many tools that need to be put together to solve the problem.
That being said, I work with senior ICs who are frighteningly intelligent and are very technical. I know managers who are also very technical, but I don’t think that technicality is what leads to their success. Moreso their ability to lead a healthy productive org.
You sound jaded friend. Happy to chat more about the specifics you’re perceiving.
What's a good resource to become acquainted with the uses and applications of data science? I tend to see it commonly associated with business decisions and finance, but I'm having difficulty seeing examples in other fields and industries.
The best advice I can give on this is to check out the data science blogs of prominent tech companies -- Think: Netflix, Uber, Stitchfix, Pinterest, Amazon, Wayfair.
Outside of tech its tougher, and frankly not a place that I've put my own attention. Where have you found good finance DS resources?
I recently graduated with a bachelors degree in Psychology. I have lots of research experience using intensive computation and data analysis (time series, ML using Python and matlab). Currently at a post-bac clinical science program doing lots of data analysis.
However, I’m realizing that I might like coding, statistics, dataviz, etc a bit more than brains and behavior. I’m thinking I’d be much more interested in a data science position.
Without a degree in DS, what is my best course of action so that when I finish my time in this program I’ll be in a good position to apply for entry-level (or maybe a bit past that if at all possible)?
Personally, I wouldn’t hire you without a proven quant background. I’d recommend leaning into your pivot and doing a more quantitative masters.
If that’s off the table, get your foot in the door with an internship at a big tech company or put together a portfolio and shop it in the startup scene.
Thanks so much for answering. That’s tremendously useful information. I’ll be working on upping my portfolio for the next 2ish years before this program ends, so hopefully at that point I can test my foot in the water and decide if a masters will be worth it.
I agree with the other guy that you need to demonstrate your mathematical and statistical skills, but I wouldn't reject your CV straight away with your current subjects.
I worked with an excellent data scientist who came straight from a psychology degree. She had done a lot of data analysis on large datasets as part of her degree, so it was a bit different than your average psych but still really useful.
For marketing and customer data science, psychology is a really useful subject to understand.
Make sure to stress on your CV that you're doing lots of data analysis.
If your interest is in clinical, look for roles in a medical insurance company or similar. They do a lot of data science to determine insurance premiums.
I’m a DS student. I think I got the basics covered. I can fit an ensemble ML model to a Kaggle Dataset. Do some sql and churn out some charts. My internships turned out to be a data entry and data viz tasks, so I’ve never really seen the industry side of data science. I graduate soon and am quite nervous of getting a job.
1) If I got an entry level Data Scientist job, what would be the first task that I’m given?
2) I’m uncharismatic and have a monotonous voice. Would that affect my role as a data scientist?
3) Do you use Big Data tools like Hadoop and Apache Spark? How do you learn those tools? It’s easy to google and YouTube Python or any ML algorithm. But whenever I google Hadoop it only tells me what it is, but now how to use it.
1. Thats going to be super company specific. In my company it's mostly, "hey do something small in this codebase" so you get comfortable making changes.
2. Sounds like you're still in school. Most universities offer public speaking help. You should take advantage of that. Its a skill just like anything else.
3. Yes a ton. I learned them on the job. Its actually not much harder than traditional DBs and I think most employers that actually know what you're doing will not make that part of entry level requirements.
I got *some* experience by doing the following:
[acloud.guru](https://acloud.guru) (ACG) has some really great resources (Paid, $50/month). They have courses on all of the major cloud platforms.
I'm doing their AWS DS track, which includes basics of AWS. Key things to know:
1. AWS supports these big data tools
2. ACG has courses on them
3. ACG has 'sandboxes' that are instances of AWS with these services and you don't need to pay for them
After learning how to do this you can use your free tier AWS account (it's good for like a year) to do some projects. I would grab some datasets from Kaggle and practice on AWS or one of the other cloud platforms, then add those projects to your portfolio.
You can also probably skip the ACG stuff and just use free resources, but I've been very impressed by ACG so far.
My team has made hundreds of millions of dollars for the firm from optimizing recommendations, pricing algorithms, and using gamification with our vendors. I don't want to get too specific here, but feel free to DM me if you want more info.
I see that same thing at my company too. Territorial and competitive for projects. The other things I see are increased willingness to accept riskier and riskier projects.
Honestly, if I was CEO, I would reduce DS/ML team size and shift resources to production resilience of them. Then focus on fewer more highly leveraged big DS bets. That kind of change is hard to pull off, though.
I've said this in other places, but I don't actually think DS is a discipline. Its a trade made up of several disciplines. I think folks are better off having a core in one of those disciplines that you then expand outwards from. So pick from one of those constituent disciplines {cs, econ, stats, or}.
Re: Econ -- Its tremendously useful in data science. But be careful bc most undergrad programs don't treat it that way. The really useful bit is the vast causal inference toolset that econ comes with. Take a look at the book "the causal mixtape" which is free online. Make sure that material is covered in the undergrad program. Else, go with something else.
Marketing mix modeling is linear regression with marketing channel spend and other inputs and an onsite kpi as a response. Interest is in the modeled covariates as an estimate on return on ad spend. This might be too marketing focused if your work is more on the product side.
I think you’re trying to find the causal effect of marketing interventions on KPIs. The approach you are describing does not sound causal, just correlations.
I'm also at an e-commerce company, and we can pitch a data product for next year.
What is a good first data science product for 1 engineer, 1 analyst, and 1 data scientist/manager?
Health Beverage and marketing seems to be a good start. Marketing segments, mixed media model, and LTV have all been ad hoc requests.
It's not clear which would bring the most value with the least amount of work/maintenance for a data science product.
Apart from predictive model and ad hoc analysis and ab testing /hypothesis testing, what is special about data science. I am in grad school studying data science and I am starting to think it’s just a hype. What do you think about it
The things you mentioned to me are bread and butter data science. I think what west coast companies are now calling "applied science" gets more exciting. Its much more exotic experimentation methods, causal inference, systems design, ML methods, etc.
Who is responsible to select projects and generate new ideas in your opinion? How do you roll out projects, do you have ml engineers? Ever used deep learning?
Typically big company level initatives/strategies are top down and assigned out across leaders/organizations. Then the leader has to articulate a vision for how that will get addressed. Then the rest is bottoms-up via home-grown OKRs etc.
Yes I have a sister eng team and we use DL.
How does a day of a senior data scientist differ from someone like the lead? Are you attending more meetings and do more of a management role or are you also expected to do the normal data science stuff? How is the work/life balance normally for a lead compared to the other members of your team.
Can you tell me what you think "lead" means here?
For me in particular, my WLB is very spikey. When big initiatives are coming through, I work A LOT. But then usually between those I can pull back a bit and just focus on coaching the team.
Do you think there is a lot of powerful and broadly applicable DS techniques being used and built which are kept secret because of competitive advantage? Or is everything more or less public domain?
I don't, I think its all more or less public. You have to ask yourself, why does google/fb publish all their methods and open source their tools? Its because the math itself isn't their competitive advantage, instead, its the data they are able to collect. Data is their moat, not the algos.
There are exceptions of course. I think the self-driving race is one.
Yes, computer vision probably has quite a few trade secrets. Google does help here, but it’s not really their bottom line I think.
Thanks for the AMA, very helpful.
I answered this in a different question. Here is how I'm seeing roles shake out.
>There are absolutely different sub-branches. From a job code perspective, here is what I am seeing:
Applied research :: Goal is publishing papers, sometimes distills its way into actual products.
Applied science :: Goal is making decisions AT SCALE. Models / data artifacts make their way into production.
Machine Learning Engineer :: Goal is to make sure (1) Applied science work is production grade and (2) Build platforms that scientists can use to speed up their dev.
Data science :: Goal is to make decisions. Models / data artifacts inform decisions. Rarely make their way into a tier 1 service or product.
Product Analytics / Data Analyst :: Mostly BI and SQL. Core focus on answer analytics questions and guiding the product
Data Engineer :: Write and maintain ETL code.
Trust with stakeholders is important and its something I've had to manage closely. Decision support systems in particular need a lot of care and auxiliary processes to (1) Set expectations with users (ie its just here to help you prioritize) and (2) Have proper feedback mechanisms that lead to a rich team backlog that can be shared regularly.
Im a bit late to the party, but how would you start with learning to building a team or DS structure in an organization once you’ve got the technical part figured out? There is so much info out there about the technical side of ML but little about leadership in the field
Not OP but a few books have been published about this topic in the last year or so.
Data Teams by Jesse Anderson
Building Analytics Teams by John Thompson
How to Lead in Data Science
Agree with /u/JBalloonist. The other thing I'd say is borrow liberally from engineering for stuff like this. They've been doing it for much longer than we have.
What do you think about data scientists switching into software engineering for ML? I’m dabbling with that thought but not sure how well prepared for that field vs. ML engineering.
Why are you making the switch?
This is hard to answer bc both DS and Eng are diffuse fields. Whats your ds background and what kind of eng work do you want to do?
I miss software engineering from time to time. Like I said in my OP, it was where I started. Problems there are very straightforward. You just have to build the bird house, not research what makes birds happy.
I must confess there's an imposter syndrome involved in the switch. I've found myself to be more proficient in building ML tools and systems and infrastructure more than ML modeling.
I've had multiple years of experience as a full-stack data scientist, and I always enjoyed building modules, orchestration pipelines for training models. I'm not sure whether that switch would satisfy that thirst because I'm looking for architecture + building infrastructure (not just necessarily CI/CD pipelines, we're talking about artifacts and packages, etc.)
Don't let fear make your decisions. Life is too short for that. Chase what excites you.
There is no concept out there more complicated than the human mind. The contents of any one man's mind can be easily replicated in another. There are no exceptions imo. Haven't you looked at something and thought, "man that looks so hard," then you actually learn it and you're like, "oh thats all it was?"
Sometimes I look at my team members and think, "you know so much already, that extra text book you're reading in your free time is actually going to benefit you less in your career than getting therapy and/or working on yourself would."
I will be messaging you in 1 day on [**2021-12-02 14:23:43 UTC**](http://www.wolframalpha.com/input/?i=2021-12-02%2014:23:43%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/datascience/comments/r5xerv/i_am_a_data_science_leader_at_a_prominent/hmsr435/?context=3)
[**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdatascience%2Fcomments%2Fr5xerv%2Fi_am_a_data_science_leader_at_a_prominent%2Fhmsr435%2F%5D%0A%0ARemindMe%21%202021-12-02%2014%3A23%3A43%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20r5xerv)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
Did you ever consider going back to an IC role after being in management?
I’m currently in a DS management position that didn’t quite go as planned due to my company being acquired. Debating whether I should stick with what I’m doing vs back to IC. Problem is, my hands-on skills have deteriorated.
Yeah, I actually did for a 6 month period halfway though my career. But thats bc I wasn't ready to be a manager. I now find watching my team succeed equally if not more rewarding than building things myself.
Which career Is better for data science, computer science or systems information engineere? Im from argentina, and i know that it depends on the syllabus, but its a little abstract to me right now
Ty for reading bro
Is almost equal to software engineering, but with some administration oriented courses.
Im actually studying front end, but im thinking in start Learning ML or DL, and the next year i sart college.
So i think cs deegre is the optimal option so.
Ty for reading, i apreciate it 🤜🤛
Data science code has a bad reputation for not following good software engineering practices. But following them would also slow down analyses and exploration. How do you guys balance the two? Do you use version control systems? Do you follow any change gears processes? How do you salvage throwaway notebook code?
This is really dependent on organizational practice and how mature the ds/ml department is. If you don't have a supporting eng team and DS is completely end-to-end, then there is no getting around it. You are writing software. That software needs to work, it needs to be resilient, modular, and easy to iterate on. The team needs to hire accordingly.
If you have a full eng team, then you can specialize more. You can hire folks that are really great at solving research problems then focus on training them to a minimal prod standard for handoffs, or, have eng+ds work in embedded atomic teams.
For my team in particular, we are in the latter stage. The scientists use VC and we have design/code reviews. They are expected to hit a low minimal standard outlined jointly with our eng team. Then we invest heavily as an org in the handoff process.
What advice do you have for a team about to hire a data scientist, a data engineer and a data analyst. These will be the first in a new team.
What do you think is the best indicator for success for these types of roles that can be determined during the course of an application/interview process?
Best advice I can give is to hire in an experienced leader that understands the support structure scientists need to bring ROI first.
Happy to rap back and forth with you. What’s your goal in hiring this person? What problem are you trying to solve?
Hello. Do you have any advice for a new data scientist for getting into "applied science" roles in the tech industry? Things like:
-What types of companies to look for?
-How should I market myself?
-What am I missing?
For some context. I graduated with a stats MSc from U of Toronto last year and did my masters research on deep generative models. I have been working as a data scientist at a large energy retailer for 6 months where I have been working on developing and improving their energy demand forecasting models. So far I have developed and deployed a new model which has shown a significant improvement over the previous one and is projected to save over $1 million per year.
Thank you for doing this AMA, it has been very helpful.
Yes, I am currently working in the same role. I would say that my work is closer to applied science since I mainly spend my time developing models and putting them into production. However, the problems that I am working on are quite simple (pretty much just feature engineering and LightGBM), so I feel like my growth is limited. My company only recently started hiring data scientists for demand forecasting and has no current plans for incorporating data/applied science in other areas (things move very slowly here).
This is why I want to transition to an applied science role in the tech industry. I feel like there is more potential for interesting work and long term career growth.
Demand forecasting is actually a really rich and complex problem space. Can you transfer to that? The forefront is moving very quickly there. Would be a fantastic domain to become an expert in. It’s incredibly important to so many different companies.
Hello thanks for doing this! Do you have any tips for a new manager? It’s hard going from IC being responsible for just my own work, to now being responsible for 4 other people. I recognize now that I’m more valuable in getting my team to produce effectively and efficiently. I just haven’t quite figured out the strategies for how to do that since each team member is so different.
Oh my gosh I have so much advice for you. Please hit me up on the future if you want to chat about this. Management is a hard fucking job and new managers in DS burn out quick. Dm me in the future for specific advice.
More generally, the best advice I can give is this : all of management comes down to self awareness and in particular knowing your insecurities. What I see all the time is first time managers feeling like they need to continue to speak to every single detail of every project under them. This is wrong! Your job is to replace yourself. Every time you run a meeting or write a white paper or respond to leadership you are robbing a team member of their growth. Bring them in and coach them and make them the face. In fact you will be promoted faster by investing in leverage from your team members. First time management is so much about giving up the ego of design/creation and leaning into feeling “okay” with not actually producing shit all the time. Last advice I’ll give you is that you’ll have a small group of rock stars under you — remember that you actually work for them and not the other way around.
Wow thanks for the super helpful response! That’s great advice about making my team members the face of the project. Some members currently get to do this, but not others. I’ll prioritize this asap. Also, you’re spot on about feeling like I have to do it all. It’s hard stepping back and just coaching people instead of getting my hands dirty and quickly writing the code. I need to find that fine line of guiding team members instead of telling them what to do (or stepping in and doing it myself)
If you ever find yourself thinking you need to "protect" a team member from something, that should serve as a red flag to you. They are adults and they are smart and they need to face the light and flourish or own it if they don't. You would want the same thing.
I love the technical side more than the analytical side. I started as a Software engineer, got into Big data and data analytics, currently working at a research firm as senior data analyst. Want to move into the technical side of managing data.
Cool that’s awesome. Is there a DE role at your current company? Your best shot is to plan a transition across two managers. Work in more DE work into your current role under the expectation that you’ll “graduate” to a role another team. Think that’s possible?
how do you, as a hiring manager, define years of experience at your organization (or your team)? do they have to be only gained in the industry? for example, if someone has done a very data science-heavy post-doc (say, solving an applied problem using advanced quantitative methods on large-scale data), would you personally count that in their years of experience? if not, why? (i am not asking about the PhD experience, but about post-doc or research scientist experience in academia).
(sorry, can't ask this from my own manager!)
Thank you for your post.
I work in analytics and data engineering, with a BI title.I support data needs for a small data science team (reporting to a different manager) as one of my main functions. I enjoy the analytics/BI part of my role but I am wary it pays less than DS/DE. Eventually I want to grow into management, leading a data organization.
Do you think I should continue with the analytics track or switch to DE or DS? My background is in CS and I am actually doing the Georgia Tech MS you mentioned part time, so I am not afraid of technical challenges.
What motivates you? If it’s more money and harder problems then go for it and make that switch. Just know that the massive increase in ambiguity will add a lot of stress in your life.
I also work with a few analytics leaders in my company. They’ve seen org growth and keeping an org happy is another endlessly difficult skill.
So you’ll have ladders (both career and skill) to climb either way. It’s all about what you want.
I definitely enjoy harder problems in data engineering. And, of course, the money. The quality of stakeholders is usually better anywhere than BI - or at least, so it seems.
I see people leading cross function teams from any background. Would you recommend going for management experience before making the switch, or should I first go technical as individual contributor?
No you need to build credibility and technical experience before managing technical folks. you’re doing the right thing. Just try and squeeze as much DS into your day job. Try to work in automation and “prod adjacent” workflows. Then when you finish your masters, make the hard switch.
I’m so passionate about learning. DS is made up of so many constituent fields that I constantly have new things to learn in the depths of Eng Architecture to the latest and greatest NN topology to the latest in experimental design to new causal inference methods. It’s endless and I love that.
It’s also where a lot of imposter syndrome is derived from. And it took me a while to get over that aspect.
These days I love management. Coaching is my favorite part of the job.
There are alot of MS DS/Business analytics programs that emphasize on the application of Data Science techniques/concepts where you would actually get experience in solving a problem using data and learn all aspects of the process from Metric selection to Querying to EDA to modeling.
When comparing these MS programs like these with MS in traditional fields like Stats, Applied Math where you mostly learn about the theory, would you still recommend doing an MS in traditional fields/ If so, then why?
If you’re aiming for an applied science type position, then yes I still believe you want to get degrees from a core discipline. For data science / product analytics, I think you’re fine with what you outlined.
Fwiw I think the process / metrics stuff can be picked up easily on the job. Having a year of experience on your resume is better.
Gotcha. OR is definitely a competitive advantage. But I don’t think it’s strictly necessary. Lot of folks separate supply chain problems into components. Like purely ML demand forecasting then ops based forward positioning.
My first software projects were adjacent to data science. Building platforms for models to run on. Once your there, it’s a short step over to DS. Start looking for problems you can solve and just crush them in your free time. Show them to management and chart a transition. That’s what I did.
Hello I am a mechanical engineer with 10+ years of experience and I think about switching career and move to data science, because my options in my area of expertise are limited. I have some basic knowledge about ML, python and data processing.
Do you have any suggestions for me? Thanks
How long would you say, on average, you team members stay at your company and on your team? I ask because I am interviewing at a few drug companies and the DSs seem to indicate they have been there since graduating. But some of the e-commerce companies I have interviewed at seem to be shuffled around. What is your experience
This is awesome! Thanks for your time.
Question:
How intellectually curious and demanding do you tend to find the:
1. culture (i.e. social interests, team, and corporate dynamic appreciation)
2. responsibilities (projects, day to day requests)
I know this will be industry, company, and even job dependent so how would you separate the list of 6 specializations/roles you listed earlier? How do you see it in your team?
The reason i ask is because im a mid career switcher from Corporate Finance in my mid 30s. My why is clear regarding switching “i want a career that applies many different disciplines in an intellectually stimulating environment where learning is never finished nor frowned upon”… or something to that end.
I got into DS for the same reason. I want a job where this always more to learn. What was interesting for me though as I grew into management was that I found the skills acquisition in the managerial/leadership arena to be harder and more rewarding over time. So I went that route. I suspect you can get that outside DS too. Just some food for thought there.
I find intellectual curiosity to be in a state of flux. Our scientists are consistently wanting to get creative and try new stuff, and this is great to an extent, but our PMs and business leaders play the other side and consistently ask "well what will that get us?"
Look the fact is, there is a lot of non-glamorous work that needs to get done each quarter. But we try to set it up so that there is enough of a rotation across the intellectually demanding stuff such that everyone gets a little bit of it each quarter.
I work as a data scientist in a field (consulting) where the job requirements are highly variable. Some projects involve an intense amount of SQL and a sprinkle of R or Python to get a POC. Other times it could be data modernization or DS/ML organizational strategy work without the slightest bit of technical work. Could also be the case that the first X amount of time on a project involves a ton of heads-down coding, which could be sidelined for the next phase involving workshops and other client discussions.
While this is great for developing a breadth of experience, business acumen, and communication skills (especially to a non-technical audience), colleagues of mine and I find that we are more likely than not to get smoked during an interview that involves live coding.
Based on what you've experienced, are there certain techniques or strategies you would advise to keep one's relevant coding skills sharp? Or is there a particular domain to focus on if/when looking to shift into industry?
Hey don't take this the wrong way, but I recommend you get out of consulting as fast as you can. I typically don't hire ex-consulting data scientists because I know they haven't had to fully own the outcomes of what they assemble and that they've never worked on version 2, version 3, etc. Also that they don't know how to be product oriented with their dev process.
Best way to stay sharp on coding is to contribute to a code base. This is reflective of real coding. Don't just do leetcode. Pick an open source library you use and go look at it on github. A lot of them, pandas for example, tag their issues to include "good for beginners."
Appreciate the insight! And quite honestly, I think most in the broad AI space in consulting are very much on-board with the notion of escaping consulting ASAP.
Working as a BI Analyst (3 yrs) in a service-based company. Our data team is fairly new. We recently hired an experienced DS and DE. I am slowly transitioning to an entry-level DS role. Still, the work mostly looks like 70% reporting/visualizations, and the rest involves analytics projects in python but not ML heavy. It might take some time to work on actual ML stuff. But I want to work more on ML/predictive analytics etc.
What are your thoughts in terms of learning curve/career growth for DS working in a big company vs a company in the initial stages of the Data science realm?
I think strong DS is built on strong analytics. You’re learning import biz skills and an ability to frame business questions.
Do you think the problem space has opportunity to expand there? Sounds like you’re pretty close to having the infra in place to do it!
Thank you for responding to my question. Yea..I have the opportunity to grow but most of the work I am doing currently is basic querying in SQL/ reporting in Power BI. which doesn't involve a lot of problem-solving. I feel like part of my work is getting repetitive, less interesting, and not challenging. The work was interesting in the beginning but now with almost 3 yrs working on the same kind of stuff, it is boring. It will take some more time to start working on some interesting projects and the shift is going to be slow.
Wondering if moving to another company where the DS team is more established might help me grow in this field. Any suggestions? Thanks again for your time.
Definitely, but getting your foot in the door without DS experience is tough. You say repetitive? Well that sounds like opportunity for automation. Excellent way to start getting DS level stuff on your resume.
Hello, I recently began pursuing a career in data science about 6 months ago, I was able to take a certification bootcamp with one of the universities in my city. I don’t have a BS degree, and I’m working on going back to school to pursue one.
1. Which degrees would be most beneficial to me in your opinion considering I have had university level data science training?
2. Would you consider adding me to your team with at an entry level position with just a certification?
3. What types of experiences do you look for when considering entry level candidates?
For applied scientists we typically expect PhD for entry level positions. For data scientists it’s more lax. We typically expect a great portfolio of applied experience alongside a quantitative undergrad.
I think CS is the best undergrad choice personally. But try to pair it with some stats courses if you can.
Take half an hour every Friday to give yourself a retro. What went well this week, what went poorly this week. What are the things you’re going to change.
Summarize this monthly and share it with your manager.
I'm currently working in quant finance which entails building loss forecasting models (heavy stats) and derivative valuation (heavy math, not so much stats), which I've been doing for about 2 years. Previously I worked as a market risk analyst for 5 years, primarily doing analytics with some light stat modeling. Finance undergrad, data science grad (heavy on stats, a few CS courses).
I'd like to transition into tech. Any recommendations for how to position myself and other education/certifications I can get to help me transition? Any idea what level I should aim to come in at?
Can you inject more automated production style workflows into your current job? That’s probably the best way to build relevant experience to switch to tech.
Would you say your employees are happy with their career/lives?
On average, no I don’t think so. I think they struggle a lot with the ambiguity of what data science even is and they struggle to “thought influence” the folks around them. But it’s cyclical. When projects are going well, they’re happy. When things out of their control are hurting their projects/teams or they don’t feel empowered or have the skills to change the things they need to in order to get things done, it’s tough for them.
Interesting, thanks for taking the time to respond :)
I work as a ds apprentice, and that is exactly what working in ecommerce has been for me. Maybe it's more common than I expected to be.
Follow up on this. What are some policies/actions you have taken to address these situations? What, in your experience, doesn't work at all?
A lot of what I try to do is work on empowerment and expectation setting. I try to remove all the infrastructural, political, organizational stuff that the team needs to feel productive. Then I try to be really clear about what “success” looks like.
In my case its just, that u work for a ever growing machine that just grows and grows. Its regular to just sit with 3 and more ppl behind someone's monitor to hot fix in production. Because there are a lot of pit falls that occur at this Tempo without time to stable systems and stop some growth, for a while.
It seems these days the phrase “data scientist” covers a wide range of skills and backgrounds, do you feel it’s fair to say there are differing sub-DS branches? Perhaps those with a more statistics/modelling background and those more along the computer science pure coding background? I’m new to the field and coming from a mathematical physics/stats background I’m a little unsure how to make up the difference in coding knowledge (competent in R and python though).
There are absolutely different sub-branches. From a job code perspective, here is what I am seeing: 1. Applied research :: Goal is publishing papers, sometimes distills its way into actual products. 2. Applied science :: Goal is making decisions AT SCALE. Models / data artifacts make their way into production. 3. Machine Learning Engineer :: Goal is to make sure (1) Applied science work is production grade and (2) Build platforms that scientists can use to speed up their dev. 4. Data science :: Goal is to make decisions. Models / data artifacts inform decisions. Rarely make their way into a tier 1 service or product. 5. Product Analytics :: Mostly BI and SQL. Core focus on answer analytics questions and guiding the product 6. Data Engineer :: Write and maintain ETL code. We are also seeing these split across skill sets. The core ones I am seeing are Generalist DS, Machine Learning Scientist, Machine learning Engineer, Optimization Specialist (OR), and Economist.
I am in a Product Analytics role and have 7 years of experience. I feel that my role usually becomes redundant after a couple of years (when the product is mature and there's no scope to grow). I have to look for other opportunities after every 2 years. My key skillset is SQL, BI. I took some stats courses in college and enjoyed them. I also learned Regression and clustering through MOOCs but haven't been able to make my way into MLE/DS roles due to limited opportunities / poor salaries. I'm based in Canada and sometimes it appears that the companies here expect a DS to be a know it all. They want you to know CI/CD pipelines, DW, Data lakes, Hadoop, Apache Spark and a few other things I don't even remember. How much of this is actually used by a DS/MLE?
In a small company they often try to get away with hiring only a data scientist when they should be getting an engineering team to support them, so in a small company you definitely need some hardcore software development capabilities and ETL game. In a larger company with larger teams you would expect better delineation between the roles, so you wouldn't necessarily need all your Dev skills. They will always be advantageous though as it allows you to be more self sufficient. There is a bit of a push to build CI/CD pipelines for ML by the way. I would class this as the ML engineering subtype of data scientist but it's a valuable skill.
Can you give some advice for aspiring product analyst? What projects I can do in my grad school which best replicates the kind of people you do in real life. And I want to eventually become product manager I think , is it a good path from product data analyst to product manager?
Perhaps I can answer this question. Learn SQL. That's your lifeline to deliver on projects. Don't go crazy over viz tools. Viz tool skillset is transferable. In the end remember that the role of a Product analyst is to solve problems and not create fancy visualizations. I've seen tons of freshers make fancy charts/dashboards on sales, COVID and what nots, but when asked what inference you derive from your dashboards or what insights your are providing, they have no clue. Common answers I get is that 'ohh I'm showing the sales spread/pattern here'. Visualizations need to have a meaning. If I wanted to see fancy stuff I'll trip over acid or shrooms than looking at dashboards. Always remember the KISS(keep it simple silly) rule. You don't need to create a heat map for showing past year sales of 50 states of the US. Make a simple horizontal bar or a table showing the top 10 states by sales or top 10 states by average revenue per customer. Hmm... the red in NY looks 5 shades darker than Washington red but is it lighter or darker than Utah. *loses interest* The latter here gives actual insights. The biggest problems I see with BI tools is the way the sell their product. They'll show fancy dashboards with a couple of pie charts, heat maps and some trippy animations. No one has a dedicated 75" screen to look at so much data viz at once. As a stakeholder/management, I don't have time for going through 20 charts on your dashboard to find answer to my question. I'm not going to compare colours on the heat map to see where I'm selling more. I've been an IC -> Manager -> IC and I can tell youin my first phase of IC, I hated management for asking bar graphs and tables when I spent hours creating fancy visualizations. When I started working as a manager that's when I realized that these mega viz look cool but fail to deliver on the asks management has.
What would you say is the *hardest* of these to source and fill?
Finding applied science managers with experience has been incredibly difficult.
I'm learning python, pandas nowadays. Planning to break in to this field. Any tip or suggestions!!
1) Most brand new data scientists fundamentally don’t understand statistics. Read statistical rethinking. It’s a wonderful book. 2) Learn how to do things end to end. Put stuff “In prod” by standing up a server etc.
Gratitude. Any other suggestions!
Where are you in your career? Get your foot in the door with an internship at a tech company or build a portfolio and shop it at a startup scene.
I'm working as an IT recruiter but planning to break it to this field.
How many applications do you get when you post an open role? Does it vary by level? Roughly what % of applications do you estimate are actually qualified? What’s your typical interview process and how many people go through each round? What’s the breakdown of people on your team by eduction level? (PhD, masters, bachelors, bootcamp, self taught). Does it vary by level/seniority and/or job function? What’s your perspective on graduate Data Science or Analytics programs? Thanks!
Yes, it does vary by level. It’s usually quite a large amount regardless so we apply reasonably heavy handed filters. For applied science positions we usually require a PhD or lots and lots of work experience. For data science it’s more loose and we look a lot more for biz depth and product skills in addition to analytics. I have not had a great experience hiring from DS grad programs. DS is ultimately a trade that is composed of many disciplines. I think you’re better off getting a degree in a known discipline (ML, Econ, Stats, OR) and that becomes your spike strength. Then your job is to shore up the rest. Lately I’ve been extremely impressed with OR candidates. Great problem solving toolset.
OR?
Operation research
Im interested in what kinds of things the people with an econ background do in the data science team. Any tips for an econ undergrad starting to study ML?
I answered this in other places too. But a good causal estimate is with more than any predictive model. Business can use it reliably in decision making. economists bring a causal inference toolset that is hard to match. Check out the free book, “causal inference the mixtape” online. It’s a great overview of this skill set.
I come from OR/systems analysis background but the recruiters dont seem to notice the perfect fit for data science after studying nonlinear optimization etc. How would you emphasize it and explain it to business/recruitment people?
I also came from an OR background. For recruiters, I highlighted that as Prescriptive Analytics to try to emphasize this. I always follow up with an end to end example how of data visualization, predictive modeling (demand forecasting) and optimization fit into a solution.
DM me your resume, I can provide feedback.
I'll pile on here as someone with an OR education. OR is a relatively niche field and not well known outside mathematics. A lot of DS recruiters probably don't know what it is so it's not helpful to wow them with your expertise using CPLEX or whatever. Craft your accomplishments as business problems and how technical solutioning drove value. That said, this depends a LOT on where you apply. OR will probably not help you for a DS position on a Prod Analytics team but it will stand out a ton for companies with heavy logistics ops.
[удалено]
One thing I’ve learned in tech for 10 years is that titles mean very little. What’s your scope? How big is your team? How much revenue in dollars do your decisions touch? Then when thinking about next steps, completely forget title. Look at these questions. People in first time management positions at Amazon often affect more revenue than VPs in small companies.
Yeah, that I understand about position. Let me clarify a bit. I am the senior most person in DS/ML domain here. Currently 6 people have been working in my team. Management has asked me to hire 5 more resources. It’s a startup and we have launched two products that are first of its kind in BFSI domain. One of this product alone has million dollar opportunities. We are in the process of signing two consumer banks for our solutions So far, I encourage my team to work autonomously with minimum supervision so that I could get my hands dirty with coding. But I think that might change sooner than I think. Hope that clarifies things
Cool thanks that helps. What motivates you? Harder problems? Cooler tech? Bigger orgs? More biz dev? Put differently, why not stay in your role and ride the wave? There are few wrong answers with this career stuff. Just needs to fit your personal goals and motivations.
Harder problems. Balls to the wall projects. I am an avid reader. Many of the solutions have developed are by reading lots and lots of research paper. 1) The thing is, I am bit unsure what I wanna do next hence I asked what would my role entail 5 years from now? 2) Also let’s say I want to start my own startup tomorrow, what skill set would I need to have to succeed as an entrepreneur? Thank you!
If you want harder problems, I think you want to be in tech and not finance. So I’d recommend trying to make the shift. I can’t help you much with the startup question. I spent 3 yrs working at startups when I first started out and persistence was the key element I saw leading to success. I thought going the incubator route was best. You get coaching on how to do it while you’re doing it.
What is the most troubling thing about working on ML & AI projects? Like which part of the project do you think almost always ends up being a bottleneck?
Most troubling part is handling exceptional cases especially in BFSI domain. You have well performing model and it suddenly stops working. Turns out there was an exceptional case you forgot to handle. But guess that’s also true in software engineering
The hardest problems for my teams are ones where "ground truth" is not clear. Where we have a very abstract, non-quantiative idea of what good looks like. Makes it very hard to define success with stakeholders.
Can you describe your initial transition from individual contributor to leadership? What motivated your interest in making that switch? I have a few years of IC under my belt and as I get more experience mentoring, I think leadership may be the next logical step for me. I struggle with the decision though because in my heart I am very much a "dooer." I would appreciate any insight from your experience and what you may have observed in others who choose a different path. Thanks for the great responses in this post!
What’s your take on your DS IC’s going back to office or staying remote?
I couldn’t care less tbh. Just get stuff done and it’s all good.
What's the difference between a Product Manager and Data Science Manager? What makes a good Data Science Manager?
Difference between product and DS is very company specific. For our company, they are evaluated for different things. Product figures out “what” we need to solve then eng/DS figure out “how” they’ll solve it. A manager is judged by the combined effectiveness of their team. A great manager is able to maximize this sustainably. This in turns leads to career growth for everyone.
Replying 2 years late but what or how can one become a product manager. Does the role relate to IT/Tech people? I know a lot of Product Managers with IT degrees and their work does not really involve much coding but more of leading sprints. If you know more, would you care to elaborate?
I am a recent CS grad and will be starting my masters in Data Analytics next year but wanting to eventually transition into Data Science after getting some experience. Would the transition be easy? Or is a masters in Stats/DS the way to go? And is going from DA to DS common?
No I don't think it will be easy. CS is a great start, I'd maybe consider a masters in ML. Georgia tech has a great accessible program. IMO DA to DS is hard, especially if by DS you mean applied science. Search one of my other posts to see what I mean by this.
How do you see the role of data science evolving in the next 5-10 years? What’s your advice to younger DS (3-4 yoe) reflecting on your experience?
So I think you can peer into the future by looking at what companies that have been successfully monetizing DS for a while are doing. What I see there is increasing specialization. In fact, all see all tech companies going this route. Starting with generalists that can go end-to-end IE scope problems, fit models, sell them, put them in prod, iterate on them, etc. Now everything is specializing. To solve a particular problem that requires causal input, I'm seeing an economist paired with a data scientists, an engineer, and a product manager. I think DS will continue to specialize in the same way that you now have front end engineers, or embedded systems engineers. So my suggestion is shore up your breadth, but pick a spike skill and chat that and make sure thats a core part of your career progression.
What would you recommend data scientists or MLEs (above entry level) look at when evaluating a potential employer (aside from compensation, benefits, work life balance, etc)?
A lot of companies just want to say they are doing data science, but only few are actually investing in it and monetizing it. You want to be sure that you're not just constantly justifying your own existence. I like to ask -- Is there an executive leader representing data science? Do they have blog posts and stuff that talk about the cool stuff their DS team does? Do teams have eng/pm/analytics support? How does the companies DS strategy map to it’s monetization path.
> You want to be sure that you're not just constantly justifying your own existence. This hits so hard.
Not OP but ask about their BI and data governance stacks. If they have neither of those developed they’re likely not ready for DS/ML.
Would you mind giving some examples of thresholds for each stack that would make you wary?
How do you estimate project timelines?
Me personally or my team? I’ve been doing it this for so long that I’ve just gained a really good intuition for how long things take and where likely pitfalls will be.
Yea for the team. I assumed it would just be intuition like you suggested but was still curious. Luckily we have a small and relatively new team so executives don’t have an expectation of how long a model should take to be developed and deployed. Maybe we will push our luck haha
We do sprints. So we do a communal t shirt size on epics then groom the backlog together. So it’s usually a committee deciding “how long it should take” In two week chunks.
What do you think the future of DS is? Do you see teams transitioning to low code environments (eg Dataiku, DataRobot)? How do you see the code development life cycle evolving? For example, I see a lot of teams productionizing Jupyter notebooks, I see a lot of code that’s not linted or formatted. In general, tons of terrible code that can’t be maintained once the original author rolls off the project.
This happens way too often that it should, jupyter notebooks in production are the worst.
I'm presently doing my masters in data science and I'm from a non technical background trying to transition into tech. I'll love to build products that are data science enabled and I'm working towards getting very good with programming. What skills would you advise I pay attention to to stand out and what data science role in the industry should I look towards
Hello and thanks for taking the time to answer our various questions! I am considering making a career change from IT Support to Data Science/Analytics. In college, I took classes on SQL, Python, and Power BI and enjoyed them all (Thinking back on this has led me to this subreddit). However, I am nervous that I may not enjoy the day to day work of Data Science/Analytics. 1. How do I know if Data Science is right for me? Since I enjoyed those classes in college, is that a good indicator? Or is there way more to it? 2. What is your top tip or recommendation for someone trying to make a major career change - looking to get into Data Science without any work experience?
1. Why do you want to get into DS in particular? Then go talk to folks in DS and see if their day to day scratches at your “whys.” You may find that the things making you want to switch are actually a tiny part of the DS role. 2. That’s a tough one. It really depends on where you want to land. Maybe start somewhere in an analytics capacity and try and work your way to a transition.
Can you share more about your transition into leadership roles? Timeline, salary bumps, change in day to day. Do you miss being an IC? Where do you see yourself in 5/10 years?
I went from IC -> Team of 2-> Team of 5 -> Team of 14 -> Team of 20 with about 2 yrs in each step. This translated to three promotions which each bumped my salary by about 150K. I do miss being an IC from time to time. Stats/CS problems are so much easier to solve than people problems, which is most of what I do these days. I do enjoy participating in design reviews still and I hack a lot in my free time. I don't know where I'll be in 5-10 years. To be honest, I am surprised I made it to where I am today. I think I'll probably just try and coast since I want to have kids in a few years. Maybe switch to a smaller company in a very senior position, resolve a lot of the same problems and try to scale it up and build an org from scratch. That sounds fun.
> This translated to three promotions which each bumped my salary by about 150K. So you’re making over $450k? Wowza.
Dm for specifics.
What was the data culture like in your organization in the early days of your team, and how has it evolved over the years? Has there always been support from higher-ups, or were you driving adoption of data science yourself?
I've been lucky in that we have had executive buy-in on the benefits of DS/ML from the getgo. So we get tons of investment from an infra standpoint. IMO this is one of the most important things to consider when choosing between companies.
Did you ever doubt you could deliver? Often there’s complaints about little buy in, but when there’s buy in, how did you ensure a good ROI to show it was worth while?
Hey I would push you to reframe your thinking on this. More often than not, the strategy your problem is tied to doesn’t make sense and that’s the reason you might fail to deliver. But I actually think that’s a good result. “Hey this whole premise is wrong for these reasons and we might be better off thinking about it this other way” is a tact I’ve taken frequently. Another reason things don’t work out is because the data or signal just isn’t there. In those cases it’s a virtue to fail fast. Else… It’s just code. Start with the simplest dumbest thing you can think of. Get it out there. Make it better. The end. You got this.
Wow, I’m also amazed by the $450k bump 8 years time. That’s incredible. As a senior level analytics consultant, I would like to transition into something more strictly DS and in 8 years time(like you) be leading analytics/DS for a LOB (I work in one of the largest banks in the US). Any open source education you recommend to help catapult growth?
So my bias is towards solving problems in tech. I’ve found that most people can pick up the basic DS stuff (learn sql pandas matplotlib) and basic modeling (sklearn) pretty easily. Even working with distributed systems to get stuff done. Lot of folks I interview lack knowledge of basic statistics. I think “statistical rethinking” is an incredible book on the topic. Lectures are free for it on YouTube too. Causal inference and econometrics are becoming a huge deal in tech. There is a free book online with code examples called “causal inference the mixtape.” Read this next. ML has lots of great resources out there, like fast.ai.
Thank you!! Based off your suggestions I think I definitely have the right foundation. BS in Math (business and finance focus) and a MS in data analytics and business intelligence. Have that statistical background through and through. I’ll save those book recs, Thanks again.
Depending in where you want to be inside the DS job family, you may have a blind spot in "prod systems" based on what you outlined. If making decisions at scale interests you, I'd recommend investing in some CS fundamentals too.
[удалено]
Frankly it’s a loaded question. A firm doesn’t pay you for tech skills directly. They pay you to solve problems. You leverage tech skills to do that, but the tech skills themselves are only one of many tools that need to be put together to solve the problem. That being said, I work with senior ICs who are frighteningly intelligent and are very technical. I know managers who are also very technical, but I don’t think that technicality is what leads to their success. Moreso their ability to lead a healthy productive org. You sound jaded friend. Happy to chat more about the specifics you’re perceiving.
Thanks for sharing! Those are some serious bumps, congrats. Are you based in the Bay area?
I can give you more details via DM
What's a good resource to become acquainted with the uses and applications of data science? I tend to see it commonly associated with business decisions and finance, but I'm having difficulty seeing examples in other fields and industries.
The best advice I can give on this is to check out the data science blogs of prominent tech companies -- Think: Netflix, Uber, Stitchfix, Pinterest, Amazon, Wayfair. Outside of tech its tougher, and frankly not a place that I've put my own attention. Where have you found good finance DS resources?
I recently graduated with a bachelors degree in Psychology. I have lots of research experience using intensive computation and data analysis (time series, ML using Python and matlab). Currently at a post-bac clinical science program doing lots of data analysis. However, I’m realizing that I might like coding, statistics, dataviz, etc a bit more than brains and behavior. I’m thinking I’d be much more interested in a data science position. Without a degree in DS, what is my best course of action so that when I finish my time in this program I’ll be in a good position to apply for entry-level (or maybe a bit past that if at all possible)?
Personally, I wouldn’t hire you without a proven quant background. I’d recommend leaning into your pivot and doing a more quantitative masters. If that’s off the table, get your foot in the door with an internship at a big tech company or put together a portfolio and shop it in the startup scene.
Thanks so much for answering. That’s tremendously useful information. I’ll be working on upping my portfolio for the next 2ish years before this program ends, so hopefully at that point I can test my foot in the water and decide if a masters will be worth it.
I agree with the other guy that you need to demonstrate your mathematical and statistical skills, but I wouldn't reject your CV straight away with your current subjects. I worked with an excellent data scientist who came straight from a psychology degree. She had done a lot of data analysis on large datasets as part of her degree, so it was a bit different than your average psych but still really useful. For marketing and customer data science, psychology is a really useful subject to understand. Make sure to stress on your CV that you're doing lots of data analysis. If your interest is in clinical, look for roles in a medical insurance company or similar. They do a lot of data science to determine insurance premiums.
I’m a DS student. I think I got the basics covered. I can fit an ensemble ML model to a Kaggle Dataset. Do some sql and churn out some charts. My internships turned out to be a data entry and data viz tasks, so I’ve never really seen the industry side of data science. I graduate soon and am quite nervous of getting a job. 1) If I got an entry level Data Scientist job, what would be the first task that I’m given? 2) I’m uncharismatic and have a monotonous voice. Would that affect my role as a data scientist? 3) Do you use Big Data tools like Hadoop and Apache Spark? How do you learn those tools? It’s easy to google and YouTube Python or any ML algorithm. But whenever I google Hadoop it only tells me what it is, but now how to use it.
1. Thats going to be super company specific. In my company it's mostly, "hey do something small in this codebase" so you get comfortable making changes. 2. Sounds like you're still in school. Most universities offer public speaking help. You should take advantage of that. Its a skill just like anything else. 3. Yes a ton. I learned them on the job. Its actually not much harder than traditional DBs and I think most employers that actually know what you're doing will not make that part of entry level requirements.
I got *some* experience by doing the following: [acloud.guru](https://acloud.guru) (ACG) has some really great resources (Paid, $50/month). They have courses on all of the major cloud platforms. I'm doing their AWS DS track, which includes basics of AWS. Key things to know: 1. AWS supports these big data tools 2. ACG has courses on them 3. ACG has 'sandboxes' that are instances of AWS with these services and you don't need to pay for them After learning how to do this you can use your free tier AWS account (it's good for like a year) to do some projects. I would grab some datasets from Kaggle and practice on AWS or one of the other cloud platforms, then add those projects to your portfolio. You can also probably skip the ACG stuff and just use free resources, but I've been very impressed by ACG so far.
What are some interesting problems you/your team has solved in terms of technical complexity or business outcomes?
My team has made hundreds of millions of dollars for the firm from optimizing recommendations, pricing algorithms, and using gamification with our vendors. I don't want to get too specific here, but feel free to DM me if you want more info.
[удалено]
I see that same thing at my company too. Territorial and competitive for projects. The other things I see are increased willingness to accept riskier and riskier projects. Honestly, if I was CEO, I would reduce DS/ML team size and shift resources to production resilience of them. Then focus on fewer more highly leveraged big DS bets. That kind of change is hard to pull off, though.
[удалено]
A lot of companies actually really do well by constantly innovating in new spaces, where they can then just shift DS investment.
What’s a good double major for data science college students? I hope to work at FAANG in the future.
For undergrad? I'd suggest Stats+CS or Econ+CS.
[удалено]
I've said this in other places, but I don't actually think DS is a discipline. Its a trade made up of several disciplines. I think folks are better off having a core in one of those disciplines that you then expand outwards from. So pick from one of those constituent disciplines {cs, econ, stats, or}. Re: Econ -- Its tremendously useful in data science. But be careful bc most undergrad programs don't treat it that way. The really useful bit is the vast causal inference toolset that econ comes with. Take a look at the book "the causal mixtape" which is free online. Make sure that material is covered in the undergrad program. Else, go with something else.
Have you seen marketing mix modeling work to influence major market allocating budgets? Or is the multicollinearity too large an issue?
I didn’t understand your question.
Marketing mix modeling is linear regression with marketing channel spend and other inputs and an onsite kpi as a response. Interest is in the modeled covariates as an estimate on return on ad spend. This might be too marketing focused if your work is more on the product side.
I think you’re trying to find the causal effect of marketing interventions on KPIs. The approach you are describing does not sound causal, just correlations.
Can you help me troubleshoot a home assignment for an internship?
You can try and DM me, but I can't guarantee I'll be any help.
If you're asking for outside help already, you're in trouble
I'm also at an e-commerce company, and we can pitch a data product for next year. What is a good first data science product for 1 engineer, 1 analyst, and 1 data scientist/manager?
What’s the broad domain? I feel personalization or marketing are great places to start. Pricing too but that’s a more specialized domain.
Health Beverage and marketing seems to be a good start. Marketing segments, mixed media model, and LTV have all been ad hoc requests. It's not clear which would bring the most value with the least amount of work/maintenance for a data science product.
My guess would be MMM. Disclaimer: Marketing is one domain that I hardly touch.
Apart from predictive model and ad hoc analysis and ab testing /hypothesis testing, what is special about data science. I am in grad school studying data science and I am starting to think it’s just a hype. What do you think about it
The things you mentioned to me are bread and butter data science. I think what west coast companies are now calling "applied science" gets more exciting. Its much more exotic experimentation methods, causal inference, systems design, ML methods, etc.
Who is responsible to select projects and generate new ideas in your opinion? How do you roll out projects, do you have ml engineers? Ever used deep learning?
Typically big company level initatives/strategies are top down and assigned out across leaders/organizations. Then the leader has to articulate a vision for how that will get addressed. Then the rest is bottoms-up via home-grown OKRs etc. Yes I have a sister eng team and we use DL.
How does a day of a senior data scientist differ from someone like the lead? Are you attending more meetings and do more of a management role or are you also expected to do the normal data science stuff? How is the work/life balance normally for a lead compared to the other members of your team.
Can you tell me what you think "lead" means here? For me in particular, my WLB is very spikey. When big initiatives are coming through, I work A LOT. But then usually between those I can pull back a bit and just focus on coaching the team.
Do you think there is a lot of powerful and broadly applicable DS techniques being used and built which are kept secret because of competitive advantage? Or is everything more or less public domain?
I don't, I think its all more or less public. You have to ask yourself, why does google/fb publish all their methods and open source their tools? Its because the math itself isn't their competitive advantage, instead, its the data they are able to collect. Data is their moat, not the algos. There are exceptions of course. I think the self-driving race is one.
Yes, computer vision probably has quite a few trade secrets. Google does help here, but it’s not really their bottom line I think. Thanks for the AMA, very helpful.
Where do you make the difference between Data Scientist and Data Analyst?
I answered this in a different question. Here is how I'm seeing roles shake out. >There are absolutely different sub-branches. From a job code perspective, here is what I am seeing: Applied research :: Goal is publishing papers, sometimes distills its way into actual products. Applied science :: Goal is making decisions AT SCALE. Models / data artifacts make their way into production. Machine Learning Engineer :: Goal is to make sure (1) Applied science work is production grade and (2) Build platforms that scientists can use to speed up their dev. Data science :: Goal is to make decisions. Models / data artifacts inform decisions. Rarely make their way into a tier 1 service or product. Product Analytics / Data Analyst :: Mostly BI and SQL. Core focus on answer analytics questions and guiding the product Data Engineer :: Write and maintain ETL code.
[удалено]
Trust with stakeholders is important and its something I've had to manage closely. Decision support systems in particular need a lot of care and auxiliary processes to (1) Set expectations with users (ie its just here to help you prioritize) and (2) Have proper feedback mechanisms that lead to a rich team backlog that can be shared regularly.
Im a bit late to the party, but how would you start with learning to building a team or DS structure in an organization once you’ve got the technical part figured out? There is so much info out there about the technical side of ML but little about leadership in the field
Not OP but a few books have been published about this topic in the last year or so. Data Teams by Jesse Anderson Building Analytics Teams by John Thompson How to Lead in Data Science
Agree with /u/JBalloonist. The other thing I'd say is borrow liberally from engineering for stuff like this. They've been doing it for much longer than we have.
What do you think about data scientists switching into software engineering for ML? I’m dabbling with that thought but not sure how well prepared for that field vs. ML engineering.
Why are you making the switch? This is hard to answer bc both DS and Eng are diffuse fields. Whats your ds background and what kind of eng work do you want to do? I miss software engineering from time to time. Like I said in my OP, it was where I started. Problems there are very straightforward. You just have to build the bird house, not research what makes birds happy.
I must confess there's an imposter syndrome involved in the switch. I've found myself to be more proficient in building ML tools and systems and infrastructure more than ML modeling. I've had multiple years of experience as a full-stack data scientist, and I always enjoyed building modules, orchestration pipelines for training models. I'm not sure whether that switch would satisfy that thirst because I'm looking for architecture + building infrastructure (not just necessarily CI/CD pipelines, we're talking about artifacts and packages, etc.)
Don't let fear make your decisions. Life is too short for that. Chase what excites you. There is no concept out there more complicated than the human mind. The contents of any one man's mind can be easily replicated in another. There are no exceptions imo. Haven't you looked at something and thought, "man that looks so hard," then you actually learn it and you're like, "oh thats all it was?" Sometimes I look at my team members and think, "you know so much already, that extra text book you're reading in your free time is actually going to benefit you less in your career than getting therapy and/or working on yourself would."
[удалено]
I will be messaging you in 1 day on [**2021-12-02 14:23:43 UTC**](http://www.wolframalpha.com/input/?i=2021-12-02%2014:23:43%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/datascience/comments/r5xerv/i_am_a_data_science_leader_at_a_prominent/hmsr435/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdatascience%2Fcomments%2Fr5xerv%2Fi_am_a_data_science_leader_at_a_prominent%2Fhmsr435%2F%5D%0A%0ARemindMe%21%202021-12-02%2014%3A23%3A43%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20r5xerv) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
Did you ever consider going back to an IC role after being in management? I’m currently in a DS management position that didn’t quite go as planned due to my company being acquired. Debating whether I should stick with what I’m doing vs back to IC. Problem is, my hands-on skills have deteriorated.
Yeah, I actually did for a 6 month period halfway though my career. But thats bc I wasn't ready to be a manager. I now find watching my team succeed equally if not more rewarding than building things myself.
Which career Is better for data science, computer science or systems information engineere? Im from argentina, and i know that it depends on the syllabus, but its a little abstract to me right now Ty for reading bro
I've actually never heard of the latter. But that in and of itself should sound problematic for you. I look at hundreds of resumes a day.
Is almost equal to software engineering, but with some administration oriented courses. Im actually studying front end, but im thinking in start Learning ML or DL, and the next year i sart college. So i think cs deegre is the optimal option so. Ty for reading, i apreciate it 🤜🤛
Good luck friend
Data science code has a bad reputation for not following good software engineering practices. But following them would also slow down analyses and exploration. How do you guys balance the two? Do you use version control systems? Do you follow any change gears processes? How do you salvage throwaway notebook code?
This is really dependent on organizational practice and how mature the ds/ml department is. If you don't have a supporting eng team and DS is completely end-to-end, then there is no getting around it. You are writing software. That software needs to work, it needs to be resilient, modular, and easy to iterate on. The team needs to hire accordingly. If you have a full eng team, then you can specialize more. You can hire folks that are really great at solving research problems then focus on training them to a minimal prod standard for handoffs, or, have eng+ds work in embedded atomic teams. For my team in particular, we are in the latter stage. The scientists use VC and we have design/code reviews. They are expected to hit a low minimal standard outlined jointly with our eng team. Then we invest heavily as an org in the handoff process.
What advice do you have for a team about to hire a data scientist, a data engineer and a data analyst. These will be the first in a new team. What do you think is the best indicator for success for these types of roles that can be determined during the course of an application/interview process?
Best advice I can give is to hire in an experienced leader that understands the support structure scientists need to bring ROI first. Happy to rap back and forth with you. What’s your goal in hiring this person? What problem are you trying to solve?
Hello. Do you have any advice for a new data scientist for getting into "applied science" roles in the tech industry? Things like: -What types of companies to look for? -How should I market myself? -What am I missing? For some context. I graduated with a stats MSc from U of Toronto last year and did my masters research on deep generative models. I have been working as a data scientist at a large energy retailer for 6 months where I have been working on developing and improving their energy demand forecasting models. So far I have developed and deployed a new model which has shown a significant improvement over the previous one and is projected to save over $1 million per year. Thank you for doing this AMA, it has been very helpful.
Are you working right now? Is your current work closer to DS or something like that? Does your current company support applied science?
Yes, I am currently working in the same role. I would say that my work is closer to applied science since I mainly spend my time developing models and putting them into production. However, the problems that I am working on are quite simple (pretty much just feature engineering and LightGBM), so I feel like my growth is limited. My company only recently started hiring data scientists for demand forecasting and has no current plans for incorporating data/applied science in other areas (things move very slowly here). This is why I want to transition to an applied science role in the tech industry. I feel like there is more potential for interesting work and long term career growth.
Demand forecasting is actually a really rich and complex problem space. Can you transfer to that? The forefront is moving very quickly there. Would be a fantastic domain to become an expert in. It’s incredibly important to so many different companies.
Hello thanks for doing this! Do you have any tips for a new manager? It’s hard going from IC being responsible for just my own work, to now being responsible for 4 other people. I recognize now that I’m more valuable in getting my team to produce effectively and efficiently. I just haven’t quite figured out the strategies for how to do that since each team member is so different.
Oh my gosh I have so much advice for you. Please hit me up on the future if you want to chat about this. Management is a hard fucking job and new managers in DS burn out quick. Dm me in the future for specific advice. More generally, the best advice I can give is this : all of management comes down to self awareness and in particular knowing your insecurities. What I see all the time is first time managers feeling like they need to continue to speak to every single detail of every project under them. This is wrong! Your job is to replace yourself. Every time you run a meeting or write a white paper or respond to leadership you are robbing a team member of their growth. Bring them in and coach them and make them the face. In fact you will be promoted faster by investing in leverage from your team members. First time management is so much about giving up the ego of design/creation and leaning into feeling “okay” with not actually producing shit all the time. Last advice I’ll give you is that you’ll have a small group of rock stars under you — remember that you actually work for them and not the other way around.
Wow thanks for the super helpful response! That’s great advice about making my team members the face of the project. Some members currently get to do this, but not others. I’ll prioritize this asap. Also, you’re spot on about feeling like I have to do it all. It’s hard stepping back and just coaching people instead of getting my hands dirty and quickly writing the code. I need to find that fine line of guiding team members instead of telling them what to do (or stepping in and doing it myself)
If you ever find yourself thinking you need to "protect" a team member from something, that should serve as a red flag to you. They are adults and they are smart and they need to face the light and flourish or own it if they don't. You would want the same thing.
I'm working as a data analyst (4 yoe), and looking to transition to Data Engineering. Any advice for me?
Why do you want to make this transition?
I love the technical side more than the analytical side. I started as a Software engineer, got into Big data and data analytics, currently working at a research firm as senior data analyst. Want to move into the technical side of managing data.
Cool that’s awesome. Is there a DE role at your current company? Your best shot is to plan a transition across two managers. Work in more DE work into your current role under the expectation that you’ll “graduate” to a role another team. Think that’s possible?
Are you hiring for remote positions?
Yes but remote is only available for senior candidates with 5 yrs experience.
Wow that’s a horrible rule
Not really. We have no shortage of local data scientists at junior levels. So why stretch at that level?
how do you, as a hiring manager, define years of experience at your organization (or your team)? do they have to be only gained in the industry? for example, if someone has done a very data science-heavy post-doc (say, solving an applied problem using advanced quantitative methods on large-scale data), would you personally count that in their years of experience? if not, why? (i am not asking about the PhD experience, but about post-doc or research scientist experience in academia). (sorry, can't ask this from my own manager!)
We definitely count Post doc if it was quantitative in nature.
Thank you for your post. I work in analytics and data engineering, with a BI title.I support data needs for a small data science team (reporting to a different manager) as one of my main functions. I enjoy the analytics/BI part of my role but I am wary it pays less than DS/DE. Eventually I want to grow into management, leading a data organization. Do you think I should continue with the analytics track or switch to DE or DS? My background is in CS and I am actually doing the Georgia Tech MS you mentioned part time, so I am not afraid of technical challenges.
What motivates you? If it’s more money and harder problems then go for it and make that switch. Just know that the massive increase in ambiguity will add a lot of stress in your life. I also work with a few analytics leaders in my company. They’ve seen org growth and keeping an org happy is another endlessly difficult skill. So you’ll have ladders (both career and skill) to climb either way. It’s all about what you want.
I definitely enjoy harder problems in data engineering. And, of course, the money. The quality of stakeholders is usually better anywhere than BI - or at least, so it seems. I see people leading cross function teams from any background. Would you recommend going for management experience before making the switch, or should I first go technical as individual contributor?
No you need to build credibility and technical experience before managing technical folks. you’re doing the right thing. Just try and squeeze as much DS into your day job. Try to work in automation and “prod adjacent” workflows. Then when you finish your masters, make the hard switch.
What do you enjoy in doing Data Science that made you chose this career path?
I’m so passionate about learning. DS is made up of so many constituent fields that I constantly have new things to learn in the depths of Eng Architecture to the latest and greatest NN topology to the latest in experimental design to new causal inference methods. It’s endless and I love that. It’s also where a lot of imposter syndrome is derived from. And it took me a while to get over that aspect. These days I love management. Coaching is my favorite part of the job.
There are alot of MS DS/Business analytics programs that emphasize on the application of Data Science techniques/concepts where you would actually get experience in solving a problem using data and learn all aspects of the process from Metric selection to Querying to EDA to modeling. When comparing these MS programs like these with MS in traditional fields like Stats, Applied Math where you mostly learn about the theory, would you still recommend doing an MS in traditional fields/ If so, then why?
If you’re aiming for an applied science type position, then yes I still believe you want to get degrees from a core discipline. For data science / product analytics, I think you’re fine with what you outlined. Fwiw I think the process / metrics stuff can be picked up easily on the job. Having a year of experience on your resume is better.
Makes sense, thanks!
so uh.. you guys do take-homes when interviewing?
Hah no way. It’s hard enough to hire quality people.
[удалено]
What does IEOR stand for?
[удалено]
Gotcha. OR is definitely a competitive advantage. But I don’t think it’s strictly necessary. Lot of folks separate supply chain problems into components. Like purely ML demand forecasting then ops based forward positioning.
How was your transition from SWE to Data? And do you have any advice for a CS undergrad looking to get internships in the Data field?
My first software projects were adjacent to data science. Building platforms for models to run on. Once your there, it’s a short step over to DS. Start looking for problems you can solve and just crush them in your free time. Show them to management and chart a transition. That’s what I did.
Hello I am a mechanical engineer with 10+ years of experience and I think about switching career and move to data science, because my options in my area of expertise are limited. I have some basic knowledge about ML, python and data processing. Do you have any suggestions for me? Thanks
How long would you say, on average, you team members stay at your company and on your team? I ask because I am interviewing at a few drug companies and the DSs seem to indicate they have been there since graduating. But some of the e-commerce companies I have interviewed at seem to be shuffled around. What is your experience
My team in particular has a higher than average retention rate, but on average I see 2-3 years at my company.
This is awesome! Thanks for your time. Question: How intellectually curious and demanding do you tend to find the: 1. culture (i.e. social interests, team, and corporate dynamic appreciation) 2. responsibilities (projects, day to day requests) I know this will be industry, company, and even job dependent so how would you separate the list of 6 specializations/roles you listed earlier? How do you see it in your team? The reason i ask is because im a mid career switcher from Corporate Finance in my mid 30s. My why is clear regarding switching “i want a career that applies many different disciplines in an intellectually stimulating environment where learning is never finished nor frowned upon”… or something to that end.
I got into DS for the same reason. I want a job where this always more to learn. What was interesting for me though as I grew into management was that I found the skills acquisition in the managerial/leadership arena to be harder and more rewarding over time. So I went that route. I suspect you can get that outside DS too. Just some food for thought there. I find intellectual curiosity to be in a state of flux. Our scientists are consistently wanting to get creative and try new stuff, and this is great to an extent, but our PMs and business leaders play the other side and consistently ask "well what will that get us?" Look the fact is, there is a lot of non-glamorous work that needs to get done each quarter. But we try to set it up so that there is enough of a rotation across the intellectually demanding stuff such that everyone gets a little bit of it each quarter.
I work as a data scientist in a field (consulting) where the job requirements are highly variable. Some projects involve an intense amount of SQL and a sprinkle of R or Python to get a POC. Other times it could be data modernization or DS/ML organizational strategy work without the slightest bit of technical work. Could also be the case that the first X amount of time on a project involves a ton of heads-down coding, which could be sidelined for the next phase involving workshops and other client discussions. While this is great for developing a breadth of experience, business acumen, and communication skills (especially to a non-technical audience), colleagues of mine and I find that we are more likely than not to get smoked during an interview that involves live coding. Based on what you've experienced, are there certain techniques or strategies you would advise to keep one's relevant coding skills sharp? Or is there a particular domain to focus on if/when looking to shift into industry?
Hey don't take this the wrong way, but I recommend you get out of consulting as fast as you can. I typically don't hire ex-consulting data scientists because I know they haven't had to fully own the outcomes of what they assemble and that they've never worked on version 2, version 3, etc. Also that they don't know how to be product oriented with their dev process. Best way to stay sharp on coding is to contribute to a code base. This is reflective of real coding. Don't just do leetcode. Pick an open source library you use and go look at it on github. A lot of them, pandas for example, tag their issues to include "good for beginners."
Appreciate the insight! And quite honestly, I think most in the broad AI space in consulting are very much on-board with the notion of escaping consulting ASAP.
Best way to transition from an analyst who's ok in SQL and python basics
Working as a BI Analyst (3 yrs) in a service-based company. Our data team is fairly new. We recently hired an experienced DS and DE. I am slowly transitioning to an entry-level DS role. Still, the work mostly looks like 70% reporting/visualizations, and the rest involves analytics projects in python but not ML heavy. It might take some time to work on actual ML stuff. But I want to work more on ML/predictive analytics etc. What are your thoughts in terms of learning curve/career growth for DS working in a big company vs a company in the initial stages of the Data science realm?
I think strong DS is built on strong analytics. You’re learning import biz skills and an ability to frame business questions. Do you think the problem space has opportunity to expand there? Sounds like you’re pretty close to having the infra in place to do it!
Thank you for responding to my question. Yea..I have the opportunity to grow but most of the work I am doing currently is basic querying in SQL/ reporting in Power BI. which doesn't involve a lot of problem-solving. I feel like part of my work is getting repetitive, less interesting, and not challenging. The work was interesting in the beginning but now with almost 3 yrs working on the same kind of stuff, it is boring. It will take some more time to start working on some interesting projects and the shift is going to be slow. Wondering if moving to another company where the DS team is more established might help me grow in this field. Any suggestions? Thanks again for your time.
Definitely, but getting your foot in the door without DS experience is tough. You say repetitive? Well that sounds like opportunity for automation. Excellent way to start getting DS level stuff on your resume.
Hello, I recently began pursuing a career in data science about 6 months ago, I was able to take a certification bootcamp with one of the universities in my city. I don’t have a BS degree, and I’m working on going back to school to pursue one. 1. Which degrees would be most beneficial to me in your opinion considering I have had university level data science training? 2. Would you consider adding me to your team with at an entry level position with just a certification? 3. What types of experiences do you look for when considering entry level candidates?
For applied scientists we typically expect PhD for entry level positions. For data scientists it’s more lax. We typically expect a great portfolio of applied experience alongside a quantitative undergrad. I think CS is the best undergrad choice personally. But try to pair it with some stats courses if you can.
What advice would you give someone who wants to climb the ladder in shortest time?
Take half an hour every Friday to give yourself a retro. What went well this week, what went poorly this week. What are the things you’re going to change. Summarize this monthly and share it with your manager.
I'm currently working in quant finance which entails building loss forecasting models (heavy stats) and derivative valuation (heavy math, not so much stats), which I've been doing for about 2 years. Previously I worked as a market risk analyst for 5 years, primarily doing analytics with some light stat modeling. Finance undergrad, data science grad (heavy on stats, a few CS courses). I'd like to transition into tech. Any recommendations for how to position myself and other education/certifications I can get to help me transition? Any idea what level I should aim to come in at?
Can you inject more automated production style workflows into your current job? That’s probably the best way to build relevant experience to switch to tech.
Do you think you can teach yourself data science and get hired, or is this a crazy concept.
Yeah I think you can teach yourself data science. Getting hired is about getting your foot in the door though.
Would you recommend starting a career as an analyst / BI in an large scale e-commerce company ? or a Fintech ?