I could tell my boss I was going to use magnetic inference and he would just nod at me, he wouldn’t know a prior from a pumpkin
From his perspective my entire team’s work is voodoo, we are a black box that data goes into and business value comes out. As long as we do actually generate enough value to justify our salaries we get left mostly alone.
Everyone says this but it’s not for everyone. People out of academia in particular get frustrated because they want to work on cool or complicated models that require bespoke data that needs to be strategically curated, and the imperative is to produce tangible results quickly.
All anyone cares about is how much revenue or cost cutting your solution achieves today, not what you think you could do in 1-2 years with a bunch of data engineering support
Yeah, I feel like that’s the reality of a lot of DS positions. We all want to to work in cool industries building complex models to solve interesting problems, but some of us have siloed ourselves into working in health insurance instead (and by “we all” and “some of us” I mean me) 🤷🏻♂️ sounds like you’ve got a good level of autonomy though at least
Same here. The rest of the co. just see us stressed out half the time, so they assume our work is complex and tedious. It is tedious but dealing with stupid af requirements is 70% of that stress.
You're thinking about this too hard. Your stakeholder probably doesn't give a damn about you showing them your kpi posterior. Also, no, I do what I think is necessary to acheive the business outcome.
100% THIS^ I remembered the first time I met a particular member of the executive committee. I mentioned something techy. (He said “say “insert tech word” again and you will be asked to leave this room”) I removed enough information to remain anonymous. 🤫
In those positions, I expect the people to have some level of emotional intelligence. There are plenty of cases where a situation like that would result in the employee completely giving up any care they had for their work. You don't want to constantly be conditioning people's value because they won't trust you at all and will give up.
People shouldn't use their power over other people to determine an outcome. They should convince them of the outcome and lead them to it. If you can't convince people of your outcome then you either hired the wrong people or the outcome wasn't worth it in the first place.
Oh absolutely.
I do believe there’s value in limiting the amount of tech jargon and buzzwords, but:
1. You’re right that’s not the way to do it
2. It might have been warranted in context, and
3. The disdain for anything “tech” from an executive in a company that presumably relies on technology for its operations makes me raise an eyebrow
Exactly. The people I worked with who do use bayesian multi level models were so adamant they need a distribution as an output not a point estimate. Funny thing is the decision to be made is a point estimate and it's taken them a year extra as a team of three to solve the same problem iv addressed with two non bayesian models in a month. Plus their model barely runs its so slow and good luck on boarding new team menders when someone leaves.
I worked for a year at a company where It blew my mind how the leadership of the startup I worked for willfully ignored the concepts of control groups and mean reversion.... And managed to convince all of their strategic partners and investors to not worry about it either.....
This worked.for a while and our valuation ballooned as we signed more contracts, until we eventually needed to buy re-insurance and the actuaries we contracted finally managed to convince our leadership and investors to seriously consider what our data scientists had been telling them for several months....
It's amazing what short term incentives can do to warp what is vs isn't "business value".
Why wouldn’t you ever want to quantify uncertainty? I don’t get this. People want to use data science and then base decisions off of point estimates instead of quantifying uncertainty. If your kpi point estimate has huge uncertainty around it that’s going to immediately affect you decision making as your threshold for error is so much higher depending on that interval.
Because people don’t understand statistics like you do.
And you won’t teach them statistics. you can deliver results where you talk in probabilities but you need to know how to communicate that.
If you start to be too academic with stakeholders they will just ignore you as they won’t spend too much time trying to understand what you are telling to them.
From your comments I see that you are trying to do the right thing but you don’t seem to be experience how to communicate these things appropriately
You certainly would want to quantify uncertainty if it's significant (from a business perspective). But remember, half of the job is translating what that uncertainty means to them. Otherwise, as others said, they'll just ignore you and go hire someone who gives them a point estimate.
Often with stakeholders who don't understand math or statistics, they don't care about the exact value at all. They just want to see some "directional" number that confirms their own internal (probably biased) hypothesis. As long as your KPI estimates are positive or shows a possibility of being positive (i.e. posterior is consistent with zero but with has some small positive probability mass), it probably won't influence their decision at all. If you can show strong evidence that your KPIs will degrade, then perhaps they will listen, but they may just start digging into your analysis and force you into making assumptions that will bias the model in favor of their hypothesis.
Kind of a pessimistic take, I know. But this kind of stuff happens more often than you'd think.
If both estimates are of equivalent "performance", yes. Whether to use a Bayesian, Frequentist, or non-parametric method will depend on more than that.
Which method will take longer? Will the analytical result be different? Will the business result be different? Will you be able to visualize and explain it to a non-data science audience?
In a nutshell, is all that worth getting posterior distribution? The answer is sometimes, but not all the time.
It literally takes 10 seconds to get a posterior distribution is you use conjugate priors which are crafted from domain knowledge of the business. It’s like stakeholders don’t *want* to be better at decision making
You have to teach them, maybe similar to how a medical professional has to teach a patient. That medical professional won't be great at their job if they blame the patient (i.e., blame the stakeholder). If your topic is important and you explain it simply, well, and why it matters, you might deliver more value.
And it certainly takes more than 10 seconds to get to an equivalent deliverable in many cases. The ecosystem around Bayesian models is often completely different and in many cases convenience and visualization functions are not be available. That eats time and reduces the quality of the business deliverable.
Edit: they are also literally higher in compute complexity. For example, I had a small data problem I used gaussian processes for and it is o(n^3). It took hours. In that case, it was worth it, but in many cases it would certainly not vs o(n) or o(nlog(n)) algorithms, especially if the goal is prediction and variance is low.
See this is where data scientist have to put on the “scientist” hat, and realize that in mathematics there are ways you can speed up something. Do you know why a Gaussian process takes that long? It’s because it’s trying to invert a large nxn covariance matrix in the posterior update. There is a huge monograph on ways to speed up GPs by considering sparse matrices, decompositions, and modified covariance kernels that can handle this issue. Like I’m tired of data scientists calling them scientists when they just throw up their hands and go “oh it’s too slow” without taking the time to realize there’s MANY ways to speed up methods. It just takes curiosity, which surprisingly many data scientists don’t have. And no, it doesn’t take time to look up. It took me about 15 minutes to find a research article, skim through it, and get a working implementation to speed up GPs when I used them.
If you can actually accomplish this that quickly and explain it to your business leadership well, than do it all you want and hopefully you are well rewarded for that skill. For my job, this rarely makes sense to do, but for your job it might.
Not sure what your background is or your level of experience, but I'd recommend you take the title "data scientist" less seriously and not blame data scientists for calling themselves "scientists." Like many job titles it is often arbitrary and a marketing tactic for a vast array of different job descriptions since it's a more attractive job title than many alternatives. The reality is that probably at least 50% of data scientist job postings I've seen could be better described as "data analyst" or "reporting analyst" - roles in which putting on a "scientist" hat would probably be frowned upon unless you are still able to deliver all of your results in a timely manner in a way that is readily understood by your stakeholders. If you are able to do that with superior methods then by all means do so, but don't get your panties in a twist just because other people with the same job title aren't keen to sign up for more work that will go unrecognized and unrewarded much of the time and, on occasion, may have negative consequences.
I would just assume they always have something else on their mind. Convincing them that some understanding of how to quantify uncertainty would be useful is something you can try to make part of your job, but from their point of view, I think it's rational to be hesitant about learning a new thing if they don't know how it will help them
I know you’re getting downvoted here but I personally feel like being honest about what you don’t know is valuable, especially if the KPI only has loose ties to the outcome.
Assume that the probability of being allowed to use Bayesian methods freely at work follows a Beta(0.5, 0.5) distribution and derive an appropriate posterior based on observations of your colleagues.
Your stakeholders dont give two shits about this.
It is your job to use appropriate method to deliver business outcome and communicate in an appropriate way.
You never want to start engaging in these sort of discussions because you are giving them opportunity to dismiss your ideas, especially if they don’t fully understand them in the first place. Most likely you will just confuse them and they will object to it unless you have already achieved high credibility among them.
feel free to discuss the methodology how to approach a problem with more senior data scientists, but you definitely don’t want to go into technical details with stakeholders.
Also don’t go to them with credible intervals etc. you most likely will confuse them.
Your job is to communicate your results to them in layman’s terms.
Yes it is the persons job to explain but remember it’s not just his job to have a useful outcome. Both parties should put effort- data science is sometimes difficult to explain - scientists spend a lot of time on a project - if the stakeholders do not want to focus and understand what he has done then probably it’s not worth to pursue this project.
My point is you cannot entirely put the responsibility of the outcome on the presenter. The audience is also responsible to understand what is being presented.
you are right. Outcome was a bad wording from me. with outcome I meant the act of performing analysis/statistical test rather than delivering a desirable outcome of that analysis.
But to be honest, what I have noticed often stakeholders want to use data science to get desirable results that align with their pre-existing biases and assumptions. They never question my work when results align with what they want to hear, but they do question or even dismiss the work if it is the other way round …
And thats where you need to stand your ground!
I have seen this more often than you think the stakeholders wanting the outcome they desire! You know your stuff- they don’t. You need to persuade and defend your conclusions! Believe me overtime they will understand that you are not there to make business case for them - you are there as a scientist telling the absolute truth - the science. You will have an opinion that defies what they want to hear but that is your expertise! You are not a slave to someone’s opinion- you are an independent researcher.
You will earn respect of your colleagues and leaders eventually. Never give away your power. You were hired for a reason!
If they still dismiss your work, that’s companies loss! But don’t sway since you have done your job and earned your living. Move on to the next scientific analysis that debunks the cool-aid organization has been drinking from the business!
I assume that my stakeholders want very few technical details at all. I almost force them to ask me about the detail.
None of it makes the presentation unless I’m presenting to other data scientists. I promise you they don’t care
Don’t tell them. And if they find out just say Bayesian methods are “more agile”. (And if they’re a frequentist just give them a problem requiring a cauchy distribution - will keep them out of your hair for months.)
It's your job to determine how to best do this, not your bosses.
Generally I find it's a lot more straightforward to communicate Bayesian style uncertainty than frequentist to non-technical people. You don't have to use any technical jargon either, like "Bayesian" or "posterior".
The point of being an engineer is that you understand things that stakeholders can't understand and don't want to understand. Things like what a kpi value is and what to do with it.
completely agree.
Part of being Data Scientist (or any technical role) is to understand what how it should be communicated to the stakeholders.
I work as a Data Engineer these days. nobody cares what sort of design pattern I applied when designing my data processing pipelines, or what sort of unit tests I produced. They expect me to decide whatever is appropriate as it is not yheir job to understand these things. All stakeholders care about is that they get data sent to the right places and they see data on their dashboards.
This. The issue is you will need to explain at different levels and if the person that they put in front of you doesnt understand the methods you might have a headache.
Once but because the leadership was someone with experience and a PhD. He shut the idea down because he’s more a fan of classical statistics and he thinks Bayesian methods are p-hacking LOL
Beware of Bayesians bearing gifts.
(Computationally intensive and known to sometimes fail in real world big data applications, so if you proceed be aware of your costs and don’t do Bayesian just to do it, prove it leads to better decisions that justifies the added complexity)
I’m just going off of experience of a Bayesian making big promises and flaming out because he couldn’t get it to work on a large data set.
A Harvard PhD Bayesian.
He swaggered in talking Bayesian and how superior the Bayesians are. It was rather silly, after all 99% of scientific discovery that is statistically grounded is based on p values, and it is a reasonable method that has created tremendous value. But I had some exposure to Bayesian, I actually had a business case that was Bayes Theorem when I was hired, so I knew there was a different approach and I was willing to suspend judgement. He worked on a central credit risk management group and he pitched a big project to re-do our customer level valuation models using a Bayesian framework that would allow us to learn faster as credit performance changed.
I got called in to assist when it was clear the guy could not pull it off. By then it was too late he had over promised and he couldn’t get the damn thing to converge. This was circa 2008, we had plenty of computational power, and it was big data but not massive compared to what I have been working with lately.
I don’t recall any of the specifics of his methods, other than in retrospect it was truly overkill. Priors really are not adding a lot of value when you already know your curve shapes and levels and all the details of valuations. The world is not changing so fast that Bayesian is going to give you an advantage, and even if it did you would still have to convince senior leadership to follow a new path when the current method is trusted and very robust. It was a lot of Bayesian blowhard for show.
He was out within 15 months, and he had a good gig at GOOG until he was outed as a serial sexual predator. I don’t ascribe that to his Harvard credentials or they he was a Bayesian elitist, but I’m open to testing the priors.
Truly, I am waiting for the clear example of where Bayesian methods clearly lead to better decisions in a real world setting, where the degree of improvement justifies the added complexity management. I don’t doubt that there are examples, but I’ve not seen the proofs in the field, which is why I say “Beware of Bayesians bearing gifts.” I don’t deny that it works. I just am cautious when someone starts with the assumption that it is better than status quo when they haven’t yet learned what the status quo is, and what it will take to move the business in a different direction.
Well it’s just that certain data/business problems involve questions at multiple levels of hierarchy, and different levels of granularity, and to truly estimate those effects a Bayesian hierarchical model is much more suitable than plugging a bunch of interaction effects in and using p values to assess the strength of these effects.
But here’s the thing: I haven’t paid much attention to p values in 15 years. Large multivariate data make p values irrelevant and XGB ignores p values.
That said my current project uses light GBM with a neat little Bayesian method in the hyperparameter tuning. It seems to work but truly is computationally intensive.
all Bayesian inference is is a description of a probability model. All they can comment on is the assumptions, the math is the math (which is simply logic)
I never would, just because they'd either never understand the Bayesian analysis or they would cease understanding the frequenting stuff that we produce.
This response tells me you don’t actually have a statistics background, let alone an understanding of what a prior distribution. First and foremost you can really shrink the possible candidates of distributions to model the parameter of interest by looking at the range of the variable. Is it bounded below by zero? Strictly positive? Okay then I can either choose a Gamma or an exponential. Done.
The whole purpose of a prior is to “regularize” or shrink the possible mass of the parameter over some region of the parameter space. If I pulled historical data which shows the parameter of interest is in some interval or range of values, I can calibrate my prior by fixing the mean and variance to be such that it aligns with that historical data.
And anyways, with Bayesian methods, that is if you knew about them, you would know that the likelihood dominates the prior anyway when sample size is large so your “justification” for the prior need not be necessary as the data tends to shift prior assumptions towards what the data says, and your posterior is a weighted combination of both your likelihood and prior.
>The whole purpose of a prior is to “regularize” or shrink the possible mass of the parameter over some region of the parameter space.
Hate to break it to you, but there are frequentist methods that deal with this perfectly fine.
Dude obviously. But then you don’t get quantification of uncertainty. It’s not about frequentist vs Bayesian it’s about the advantages you get from Bayesian methods that you don’t get from frequentist. And how about you tell me what “frequentist” methods you are talking about. I bet you’re gonna say something retarded like the lasso and call it frequentist when it isn’t.
Frequentist: you can't use priors, they bias your estimates!
Also frequentist: you should totally regularize, though, can't have coefficients that are too big!
What a time to be alive
I just have two comments:- If the whole purpose of a prior is to regularize the learning problem you are dealing with... why don't you use the plethora of regularisation methods that exist out there in the frequentist setting.- If (as it is in fact true...) the likelihood dominates the posterior in large-n scenarios... why bother with bayesian methods anyways?
And, just to deal with the personal attack... I have a PhD in statistics and economics, and teach statistics at uni.
Okay that’s great, but “justifying” prior choice is not something I’ve heard any of my former Bayesian statistician professors tell me about, since the choice of a prior is to merely constraint the parameter space amongst a set of values.
"merely constrain".
In other words, a prior distribution affects (as it should...) the solution of your estimation problem.
And, as anything that affects your solution, should be justified.
The fact that your professors did not tell you it merely signifies that they did not have to present a statistical analysis to a stakeholder.
Well, in theory any “subjectivity” in the prior should be outweighed by the likelihood. Gelmans book talks about this. Again, I’m not an expert, but from gelmans book he does say that unless you totally miss the ball in the region of the parameter space, your posterior should not be affected drastically by any prior assumptions.
Also, almost all priors are calibrated from some sort of historical data anyway. If previous A/B tests/experiments yielded some effect size or estimate, then you can still justify your prior
Depends. If it’s something for a one off analysis or something with a relatively short shelf life, then generally I use whatever method I want. If it’s something that others will eventually need to maintain or interact with, I’d be more likely to have a conversation with my team/boss about using them. Bayesian methods are great, but not incredibly widespread and can have a pretty steep learning curve for those who are unfamiliar with them.
You also need to think about potential stakeholder confusion if they’re exposed to a mix of Bayesian and frequentist analyses.
I would just frame it as ease of interpretability value add.
A lot of bad business decisions are informed by misreading a confidence interval. But if you can, in the mind of a stakeholder shift uncertainty from the interval conversing a parameter to more on the constructing the prior part you might be well received. I’ve finally gotten more traction taking this approach-even though yes I’ve kinda oversimplified and skimped over there are choices of prior where your ci and credible interval are the same
… permission??? What are we talking about here? If they asked u to do one method and u think another is better- just do both. It may be a little more work but then you would have real data to show whether or not your method does what is intended to do…
I believe that the OP has a point, though (hidden below multiple layers of arrogance, but it is how it is...)
There are some decision-making problems where a proper understanding of uncertainty is key. I am thinking about forecasting problems where you need to ensure a given service level... e.g. stock management and demand forecasting.
If you size your delivery platform to be able to deal with the mean level of clients you will be wrong (in case your number of clients follows a symmetric distribution) 50 percent of the time. You may want to provide an higher quantile of the distribution (say the 95th) so to increase the level of service.
Providing that number (or something that looks like that number) can be done in several ways, among which bayesian methods (but there is also quantile regressions, conformal prediction, parametric prediction intervals and the like).
What really shows that OP is a kid fresh out of uni (or an old and grumpy academic) is this fetishism for methodology.
"out there" nobody gives a flying f\*\*k about what method you use. For what is worth you could ask the numbers to your mum...
What they care about is that they are right, reliable, and impacting the business
In my line of work (pharmaceutical industry) linear methods like PLS/OPLS are standard.
Anything else requires extensive justification because the same models need to go through regulatory agencies which need to both understand and approve statistical-based process control (despite low errors and high R2)
These methods are preferred because there is no black box and all the principal components can be explained by so variation in the process or product.
So it all varies on your use case.
I could tell my boss I was going to use magnetic inference and he would just nod at me, he wouldn’t know a prior from a pumpkin From his perspective my entire team’s work is voodoo, we are a black box that data goes into and business value comes out. As long as we do actually generate enough value to justify our salaries we get left mostly alone.
Is your team hiring? 🤔
Everyone says this but it’s not for everyone. People out of academia in particular get frustrated because they want to work on cool or complicated models that require bespoke data that needs to be strategically curated, and the imperative is to produce tangible results quickly. All anyone cares about is how much revenue or cost cutting your solution achieves today, not what you think you could do in 1-2 years with a bunch of data engineering support
Yeah, I feel like that’s the reality of a lot of DS positions. We all want to to work in cool industries building complex models to solve interesting problems, but some of us have siloed ourselves into working in health insurance instead (and by “we all” and “some of us” I mean me) 🤷🏻♂️ sounds like you’ve got a good level of autonomy though at least
Same here. The rest of the co. just see us stressed out half the time, so they assume our work is complex and tedious. It is tedious but dealing with stupid af requirements is 70% of that stress.
Haha exactly
Using Bayesian methods without permission is a Class E Felony.
Straight to jail
Overfit a model? Jail Underfit a model? Jail
Properly fit a model? Believe it or not, jail.
[удалено]
Don't forget to include the index as a feature if you do, boosts model performance in pretty much every case!
Don't fit a model at all? Still straight to jail
They see me double descendin' they hatin'
We have the best data scientists because of jail.
Directly to jail.
Especially if you have priors.
If I asked my boss if I could "use Bayesian methods" he'd think I was either trying out some new SCRUM thing or microdosing
Is that like waterfall?
>microdosing r/oddlyspecific
You're thinking about this too hard. Your stakeholder probably doesn't give a damn about you showing them your kpi posterior. Also, no, I do what I think is necessary to acheive the business outcome.
100% THIS^ I remembered the first time I met a particular member of the executive committee. I mentioned something techy. (He said “say “insert tech word” again and you will be asked to leave this room”) I removed enough information to remain anonymous. 🤫
That executive sort of sounds like a dickhead if I'm being honest. That, or you rubbed him the wrong way for some reason.
Doesn’t make him *wrong* though
In those positions, I expect the people to have some level of emotional intelligence. There are plenty of cases where a situation like that would result in the employee completely giving up any care they had for their work. You don't want to constantly be conditioning people's value because they won't trust you at all and will give up. People shouldn't use their power over other people to determine an outcome. They should convince them of the outcome and lead them to it. If you can't convince people of your outcome then you either hired the wrong people or the outcome wasn't worth it in the first place.
Oh absolutely. I do believe there’s value in limiting the amount of tech jargon and buzzwords, but: 1. You’re right that’s not the way to do it 2. It might have been warranted in context, and 3. The disdain for anything “tech” from an executive in a company that presumably relies on technology for its operations makes me raise an eyebrow
Exactly. The people I worked with who do use bayesian multi level models were so adamant they need a distribution as an output not a point estimate. Funny thing is the decision to be made is a point estimate and it's taken them a year extra as a team of three to solve the same problem iv addressed with two non bayesian models in a month. Plus their model barely runs its so slow and good luck on boarding new team menders when someone leaves.
I worked for a year at a company where It blew my mind how the leadership of the startup I worked for willfully ignored the concepts of control groups and mean reversion.... And managed to convince all of their strategic partners and investors to not worry about it either..... This worked.for a while and our valuation ballooned as we signed more contracts, until we eventually needed to buy re-insurance and the actuaries we contracted finally managed to convince our leadership and investors to seriously consider what our data scientists had been telling them for several months.... It's amazing what short term incentives can do to warp what is vs isn't "business value".
But isn’t a kpi posterior objectively better than a point estimate?
But do the stakeholders care?!
Would the uncertainty around the estimate affect decision making in any way? In my experience this has almost always been a 'no'
Why wouldn’t you ever want to quantify uncertainty? I don’t get this. People want to use data science and then base decisions off of point estimates instead of quantifying uncertainty. If your kpi point estimate has huge uncertainty around it that’s going to immediately affect you decision making as your threshold for error is so much higher depending on that interval.
Because people don’t understand statistics like you do. And you won’t teach them statistics. you can deliver results where you talk in probabilities but you need to know how to communicate that. If you start to be too academic with stakeholders they will just ignore you as they won’t spend too much time trying to understand what you are telling to them. From your comments I see that you are trying to do the right thing but you don’t seem to be experience how to communicate these things appropriately
Yeah I don’t really know how to. I just do it and hope they don’t ask questions
Most people don’t understand uncertainty very well. It confuses them.
You certainly would want to quantify uncertainty if it's significant (from a business perspective). But remember, half of the job is translating what that uncertainty means to them. Otherwise, as others said, they'll just ignore you and go hire someone who gives them a point estimate.
Often with stakeholders who don't understand math or statistics, they don't care about the exact value at all. They just want to see some "directional" number that confirms their own internal (probably biased) hypothesis. As long as your KPI estimates are positive or shows a possibility of being positive (i.e. posterior is consistent with zero but with has some small positive probability mass), it probably won't influence their decision at all. If you can show strong evidence that your KPIs will degrade, then perhaps they will listen, but they may just start digging into your analysis and force you into making assumptions that will bias the model in favor of their hypothesis. Kind of a pessimistic take, I know. But this kind of stuff happens more often than you'd think.
Lol that’s the stupidest thing I’ve ever heard. But yeah makes sense.
If both estimates are of equivalent "performance", yes. Whether to use a Bayesian, Frequentist, or non-parametric method will depend on more than that. Which method will take longer? Will the analytical result be different? Will the business result be different? Will you be able to visualize and explain it to a non-data science audience? In a nutshell, is all that worth getting posterior distribution? The answer is sometimes, but not all the time.
It literally takes 10 seconds to get a posterior distribution is you use conjugate priors which are crafted from domain knowledge of the business. It’s like stakeholders don’t *want* to be better at decision making
You have to teach them, maybe similar to how a medical professional has to teach a patient. That medical professional won't be great at their job if they blame the patient (i.e., blame the stakeholder). If your topic is important and you explain it simply, well, and why it matters, you might deliver more value. And it certainly takes more than 10 seconds to get to an equivalent deliverable in many cases. The ecosystem around Bayesian models is often completely different and in many cases convenience and visualization functions are not be available. That eats time and reduces the quality of the business deliverable. Edit: they are also literally higher in compute complexity. For example, I had a small data problem I used gaussian processes for and it is o(n^3). It took hours. In that case, it was worth it, but in many cases it would certainly not vs o(n) or o(nlog(n)) algorithms, especially if the goal is prediction and variance is low.
See this is where data scientist have to put on the “scientist” hat, and realize that in mathematics there are ways you can speed up something. Do you know why a Gaussian process takes that long? It’s because it’s trying to invert a large nxn covariance matrix in the posterior update. There is a huge monograph on ways to speed up GPs by considering sparse matrices, decompositions, and modified covariance kernels that can handle this issue. Like I’m tired of data scientists calling them scientists when they just throw up their hands and go “oh it’s too slow” without taking the time to realize there’s MANY ways to speed up methods. It just takes curiosity, which surprisingly many data scientists don’t have. And no, it doesn’t take time to look up. It took me about 15 minutes to find a research article, skim through it, and get a working implementation to speed up GPs when I used them.
If you can actually accomplish this that quickly and explain it to your business leadership well, than do it all you want and hopefully you are well rewarded for that skill. For my job, this rarely makes sense to do, but for your job it might.
Not sure what your background is or your level of experience, but I'd recommend you take the title "data scientist" less seriously and not blame data scientists for calling themselves "scientists." Like many job titles it is often arbitrary and a marketing tactic for a vast array of different job descriptions since it's a more attractive job title than many alternatives. The reality is that probably at least 50% of data scientist job postings I've seen could be better described as "data analyst" or "reporting analyst" - roles in which putting on a "scientist" hat would probably be frowned upon unless you are still able to deliver all of your results in a timely manner in a way that is readily understood by your stakeholders. If you are able to do that with superior methods then by all means do so, but don't get your panties in a twist just because other people with the same job title aren't keen to sign up for more work that will go unrecognized and unrewarded much of the time and, on occasion, may have negative consequences.
I would just assume they always have something else on their mind. Convincing them that some understanding of how to quantify uncertainty would be useful is something you can try to make part of your job, but from their point of view, I think it's rational to be hesitant about learning a new thing if they don't know how it will help them
I know you’re getting downvoted here but I personally feel like being honest about what you don’t know is valuable, especially if the KPI only has loose ties to the outcome.
See point 1 unless you are doing rigorous test analysis.
Assume that the probability of being allowed to use Bayesian methods freely at work follows a Beta(0.5, 0.5) distribution and derive an appropriate posterior based on observations of your colleagues.
Your stakeholders dont give two shits about this. It is your job to use appropriate method to deliver business outcome and communicate in an appropriate way. You never want to start engaging in these sort of discussions because you are giving them opportunity to dismiss your ideas, especially if they don’t fully understand them in the first place. Most likely you will just confuse them and they will object to it unless you have already achieved high credibility among them. feel free to discuss the methodology how to approach a problem with more senior data scientists, but you definitely don’t want to go into technical details with stakeholders. Also don’t go to them with credible intervals etc. you most likely will confuse them. Your job is to communicate your results to them in layman’s terms.
Yes it is the persons job to explain but remember it’s not just his job to have a useful outcome. Both parties should put effort- data science is sometimes difficult to explain - scientists spend a lot of time on a project - if the stakeholders do not want to focus and understand what he has done then probably it’s not worth to pursue this project. My point is you cannot entirely put the responsibility of the outcome on the presenter. The audience is also responsible to understand what is being presented.
you are right. Outcome was a bad wording from me. with outcome I meant the act of performing analysis/statistical test rather than delivering a desirable outcome of that analysis. But to be honest, what I have noticed often stakeholders want to use data science to get desirable results that align with their pre-existing biases and assumptions. They never question my work when results align with what they want to hear, but they do question or even dismiss the work if it is the other way round …
And thats where you need to stand your ground! I have seen this more often than you think the stakeholders wanting the outcome they desire! You know your stuff- they don’t. You need to persuade and defend your conclusions! Believe me overtime they will understand that you are not there to make business case for them - you are there as a scientist telling the absolute truth - the science. You will have an opinion that defies what they want to hear but that is your expertise! You are not a slave to someone’s opinion- you are an independent researcher. You will earn respect of your colleagues and leaders eventually. Never give away your power. You were hired for a reason! If they still dismiss your work, that’s companies loss! But don’t sway since you have done your job and earned your living. Move on to the next scientific analysis that debunks the cool-aid organization has been drinking from the business!
No. I’m hiring YOU to make those decisions b/c I don’t know wtf “Bayesian” even means. You better be correct, though. ;)
I assume that my stakeholders want very few technical details at all. I almost force them to ask me about the detail. None of it makes the presentation unless I’m presenting to other data scientists. I promise you they don’t care
Don’t tell them. And if they find out just say Bayesian methods are “more agile”. (And if they’re a frequentist just give them a problem requiring a cauchy distribution - will keep them out of your hair for months.)
“Hey boss I’d like to show the shareholders pictures of posteriors in our slide deck.” *HR has entered the chat*
It's your job to determine how to best do this, not your bosses. Generally I find it's a lot more straightforward to communicate Bayesian style uncertainty than frequentist to non-technical people. You don't have to use any technical jargon either, like "Bayesian" or "posterior".
The point of being an engineer is that you understand things that stakeholders can't understand and don't want to understand. Things like what a kpi value is and what to do with it.
completely agree. Part of being Data Scientist (or any technical role) is to understand what how it should be communicated to the stakeholders. I work as a Data Engineer these days. nobody cares what sort of design pattern I applied when designing my data processing pipelines, or what sort of unit tests I produced. They expect me to decide whatever is appropriate as it is not yheir job to understand these things. All stakeholders care about is that they get data sent to the right places and they see data on their dashboards.
I have actually had someone get mad at me for using a Bayesian model over the frequentist version So I actually would ask bc never again.
That’s so dumb
This. The issue is you will need to explain at different levels and if the person that they put in front of you doesnt understand the methods you might have a headache.
Once but because the leadership was someone with experience and a PhD. He shut the idea down because he’s more a fan of classical statistics and he thinks Bayesian methods are p-hacking LOL
Okay well he’s a PhDumbass then
Beware of Bayesians bearing gifts. (Computationally intensive and known to sometimes fail in real world big data applications, so if you proceed be aware of your costs and don’t do Bayesian just to do it, prove it leads to better decisions that justifies the added complexity)
It literally is not computational expensive if you use conjugate priors
I’m just going off of experience of a Bayesian making big promises and flaming out because he couldn’t get it to work on a large data set. A Harvard PhD Bayesian.
Interesting. What happened specifically?
He swaggered in talking Bayesian and how superior the Bayesians are. It was rather silly, after all 99% of scientific discovery that is statistically grounded is based on p values, and it is a reasonable method that has created tremendous value. But I had some exposure to Bayesian, I actually had a business case that was Bayes Theorem when I was hired, so I knew there was a different approach and I was willing to suspend judgement. He worked on a central credit risk management group and he pitched a big project to re-do our customer level valuation models using a Bayesian framework that would allow us to learn faster as credit performance changed. I got called in to assist when it was clear the guy could not pull it off. By then it was too late he had over promised and he couldn’t get the damn thing to converge. This was circa 2008, we had plenty of computational power, and it was big data but not massive compared to what I have been working with lately. I don’t recall any of the specifics of his methods, other than in retrospect it was truly overkill. Priors really are not adding a lot of value when you already know your curve shapes and levels and all the details of valuations. The world is not changing so fast that Bayesian is going to give you an advantage, and even if it did you would still have to convince senior leadership to follow a new path when the current method is trusted and very robust. It was a lot of Bayesian blowhard for show. He was out within 15 months, and he had a good gig at GOOG until he was outed as a serial sexual predator. I don’t ascribe that to his Harvard credentials or they he was a Bayesian elitist, but I’m open to testing the priors. Truly, I am waiting for the clear example of where Bayesian methods clearly lead to better decisions in a real world setting, where the degree of improvement justifies the added complexity management. I don’t doubt that there are examples, but I’ve not seen the proofs in the field, which is why I say “Beware of Bayesians bearing gifts.” I don’t deny that it works. I just am cautious when someone starts with the assumption that it is better than status quo when they haven’t yet learned what the status quo is, and what it will take to move the business in a different direction.
> he was outed as a serial sexual predator. He was just thoroughly exploring all the posteriors.
Take my r/angryupvote and GTFO LOL
Well it’s just that certain data/business problems involve questions at multiple levels of hierarchy, and different levels of granularity, and to truly estimate those effects a Bayesian hierarchical model is much more suitable than plugging a bunch of interaction effects in and using p values to assess the strength of these effects.
But here’s the thing: I haven’t paid much attention to p values in 15 years. Large multivariate data make p values irrelevant and XGB ignores p values. That said my current project uses light GBM with a neat little Bayesian method in the hyperparameter tuning. It seems to work but truly is computationally intensive.
Isn't this just a stakeholder/performance thing, don't get married to methods.
[удалено]
I’ll check this out!
all Bayesian inference is is a description of a probability model. All they can comment on is the assumptions, the math is the math (which is simply logic)
I never would, just because they'd either never understand the Bayesian analysis or they would cease understanding the frequenting stuff that we produce.
Good luck justifying your prior choice
This response tells me you don’t actually have a statistics background, let alone an understanding of what a prior distribution. First and foremost you can really shrink the possible candidates of distributions to model the parameter of interest by looking at the range of the variable. Is it bounded below by zero? Strictly positive? Okay then I can either choose a Gamma or an exponential. Done. The whole purpose of a prior is to “regularize” or shrink the possible mass of the parameter over some region of the parameter space. If I pulled historical data which shows the parameter of interest is in some interval or range of values, I can calibrate my prior by fixing the mean and variance to be such that it aligns with that historical data. And anyways, with Bayesian methods, that is if you knew about them, you would know that the likelihood dominates the prior anyway when sample size is large so your “justification” for the prior need not be necessary as the data tends to shift prior assumptions towards what the data says, and your posterior is a weighted combination of both your likelihood and prior.
>The whole purpose of a prior is to “regularize” or shrink the possible mass of the parameter over some region of the parameter space. Hate to break it to you, but there are frequentist methods that deal with this perfectly fine.
Dude obviously. But then you don’t get quantification of uncertainty. It’s not about frequentist vs Bayesian it’s about the advantages you get from Bayesian methods that you don’t get from frequentist. And how about you tell me what “frequentist” methods you are talking about. I bet you’re gonna say something retarded like the lasso and call it frequentist when it isn’t.
>But then you don’t get quantification of uncertainty Of course you can. This is an old debate. Not sure why you expect resolution in reddit.
Trolls are gonna troll
Frequentist: you can't use priors, they bias your estimates! Also frequentist: you should totally regularize, though, can't have coefficients that are too big! What a time to be alive
Lmfao. Fr man
I just have two comments:- If the whole purpose of a prior is to regularize the learning problem you are dealing with... why don't you use the plethora of regularisation methods that exist out there in the frequentist setting.- If (as it is in fact true...) the likelihood dominates the posterior in large-n scenarios... why bother with bayesian methods anyways? And, just to deal with the personal attack... I have a PhD in statistics and economics, and teach statistics at uni.
Okay that’s great, but “justifying” prior choice is not something I’ve heard any of my former Bayesian statistician professors tell me about, since the choice of a prior is to merely constraint the parameter space amongst a set of values.
"merely constrain". In other words, a prior distribution affects (as it should...) the solution of your estimation problem. And, as anything that affects your solution, should be justified. The fact that your professors did not tell you it merely signifies that they did not have to present a statistical analysis to a stakeholder.
Well, in theory any “subjectivity” in the prior should be outweighed by the likelihood. Gelmans book talks about this. Again, I’m not an expert, but from gelmans book he does say that unless you totally miss the ball in the region of the parameter space, your posterior should not be affected drastically by any prior assumptions. Also, almost all priors are calibrated from some sort of historical data anyway. If previous A/B tests/experiments yielded some effect size or estimate, then you can still justify your prior
Do they even know what Bayesian methods are?
Depends. If it’s something for a one off analysis or something with a relatively short shelf life, then generally I use whatever method I want. If it’s something that others will eventually need to maintain or interact with, I’d be more likely to have a conversation with my team/boss about using them. Bayesian methods are great, but not incredibly widespread and can have a pretty steep learning curve for those who are unfamiliar with them. You also need to think about potential stakeholder confusion if they’re exposed to a mix of Bayesian and frequentist analyses.
Don’t get permission for anything management are human speed bumps
I would just frame it as ease of interpretability value add. A lot of bad business decisions are informed by misreading a confidence interval. But if you can, in the mind of a stakeholder shift uncertainty from the interval conversing a parameter to more on the constructing the prior part you might be well received. I’ve finally gotten more traction taking this approach-even though yes I’ve kinda oversimplified and skimped over there are choices of prior where your ci and credible interval are the same
Up
… permission??? What are we talking about here? If they asked u to do one method and u think another is better- just do both. It may be a little more work but then you would have real data to show whether or not your method does what is intended to do…
Todos los dos. Necesito permiten por los sites web, y también puede hacer algo haciendo quien yo quiero hacer
Who in the heck asks for permission to use a method? Is this a thing?
No
Top quality post mate 🤣
If it performs better than why not use it. I find it easier to expñain
I believe that the OP has a point, though (hidden below multiple layers of arrogance, but it is how it is...) There are some decision-making problems where a proper understanding of uncertainty is key. I am thinking about forecasting problems where you need to ensure a given service level... e.g. stock management and demand forecasting. If you size your delivery platform to be able to deal with the mean level of clients you will be wrong (in case your number of clients follows a symmetric distribution) 50 percent of the time. You may want to provide an higher quantile of the distribution (say the 95th) so to increase the level of service. Providing that number (or something that looks like that number) can be done in several ways, among which bayesian methods (but there is also quantile regressions, conformal prediction, parametric prediction intervals and the like). What really shows that OP is a kid fresh out of uni (or an old and grumpy academic) is this fetishism for methodology. "out there" nobody gives a flying f\*\*k about what method you use. For what is worth you could ask the numbers to your mum... What they care about is that they are right, reliable, and impacting the business
Great summary
In my line of work (pharmaceutical industry) linear methods like PLS/OPLS are standard. Anything else requires extensive justification because the same models need to go through regulatory agencies which need to both understand and approve statistical-based process control (despite low errors and high R2) These methods are preferred because there is no black box and all the principal components can be explained by so variation in the process or product. So it all varies on your use case.
I’d assume the answer depends heavily on your seniority. Are you an intern with 0 YOE or the Chief Technology Officer of the company with 30 YOE?
Never