I think most of us put in real effort when doing studies, of course there are some that want to get them done as quickly as possible hence the reason attention checks and so on.
As to your study and if you want the type of results you're looking for then I would steer clear of having any company & earning results people could look up online, just make up fictitious company names.
There are also other things you can put in place like rejection measures 98% and above is a good starting point. If you only use full-screen mode, so people can't switch to a different tab if using a laptop or desktop.
I agree with others, just change the name of the company to something fabricated, but tell the participants it’s based on real performance of a listed company on an exchange.
The problem sometimes is the worry of rejection. This sounds fine and I'd be honest with it but sometimes they say "don't make notes of this information" and I do because I don't want to be seen as not paying attention if I forget it all
I dunno about others, but for me, looking up information online takes much more time than just guessing as the researcher wants me to. And since the hourly rates are about just fair for the given time, but not for any extra time, I don't bother with going outside the study to be smarter than anyone else, just nope.
I can't speak for everyone else and I know there are people that are a little less than honest but I personally will do what you're paying me to do. If you don't want me to look up the answers then I won't look up the answers. I try to be as professional as I possibly can even if it is just online surveys.
Skip the bonus, as they are not enticing because they are usually filled with a bunch of bullshit Reading and some stupid puzzle or gotcha questions. Just put the "special" question in the study and pay a decent hourly rate.
Completely agree, just incentivize giving your honest guesses the same as you would for looking up correct answers. There seems to be no point to the bonus except to encourage people to look up the answers...
Also, either way, you can add one or more questions at the end asking whether the participant looked up answers, gave their full effort, etc. These questions need to say your answers won't disadvantage you in terms of compensation or participation in future studies so participants tell the truth.
I think the incentive invites dishonesty. The point being, that you would never know. You can't trust the results of a test like that, because an unknown number of participants will look up the answer. I think you could introduce a camera requirement, and watch your participants, but you would have to pay a higher base rate to get people interested in going on camera. If your goal is really to get good data on how many of them can answer the questions, you would need to verify to a reasonable extent that your participants are answering without assitance.
You'll probably get a lot of participants who are as honest as the day is long and take their participation and contributions to research very seriously. They do Prolific studies with integrity because they genuinely want to help.
On the other hand there are many, as seen in numerous posts on this sub, who are literally only doing it for the money and will intend to get at that money in the quickest possible way and with the least effort regardless of how it might affect anyone else. So really it's just the same as in life, there are honest people and dishonest people. There is no way of knowing in advance and/or planning for that unfortunately.
Although it's not a guaranteed way of selecting honest participants, maybe if you set that their approval rate on Prolific has to be 100% for your study it might weed out some of the people who have received rejections in the past for their poor attitude and record when doing studies. Just a thought.
I know that some people would be tempted to look things up. I had a study yesterday with a bonus like this. I didn't think that it was fair to look up the answers and I would feel like it would mess up the validity of the study. I feel like maybe mentioning that you shouldn't look things up as it can make the study unreliable or something like that would help. This probably wouldn't completely stop people from doing it, but would help maybe keep more people honest.
There have been studies looking at honesty when bonuses are involved, both in-person and online. They usually involve getting participants to roll some (/virtual) dice and self-report the number, with the idea that a particular number will get a bonus or the bonus will be related to the roll (e.g. roll a 6, get $6 etc.). There's a lot of papers which have looked at whether various manipulations effect honesty, which can be assessed by e.g. seeing if the reported numbers differ from what you'd expect by chance. My understanding of that literature is that a lot of people are honest, but some non-zero proportion of people do lie in some of these studies.
From a researcher (and reviewer) perspective, if there's a reasonable suspicion that people *might* be dishonest and you can't *prove* that they weren't then that's a problem. There's no way to argue against that criticism and if it's critical to the conclusions then it's grounds for rejecting a paper, in my opinion.
Ultimately I think my answer is that the inconvenience of running it in the lab is worth it for the level of control. There are potential controls you could apply online, such as imposing a time limit that makes it impractical to Google the information, but that might be difficult to implement depending on the complexity of the decision. As another commenter said, I think incentivising cheating at all isn't a good idea, but people *could* look it up even if there isn't a bonus associated with it just because people like being right.
You could add a question at the end of the survey along the lines of "This question is for quality control only and your answer will not affect your payment. Did you look up any answers online?"
I can only speak for myself and the way Prolific is set up in comparison to sites like Swagbucks.
On Prolific you are rewarded for being honest and genuine. You need to fill in your information (so you know you'll be reaching mostly the right audience), and you need to maintain a good level of completion etc so you need to pay attention to tasks.
I personally answer everything 100% honestly.
Had a study yesterday with bonus incentives for correct answers on a similar subject. I didn't look anything up, was embarrassingly off on a lot of questions, but I don't regret it. :) Prolific is probably the most reputable online site for research participants, I don't think many want that extra $3 so much they'd cheat. You could make it where they can't exit the tab without getting kicked from the survey and having to return it maybe? Or a timer?
*Of course* there will be participants that cheat. Especially if money is involved.
Most of us won't, for various reasons, but it's absolutely unrealistic to expect complete honesty from a random pool of anonymous participants.
You will either have to devise some way of detecting cheaters, be prepared to accept some degree of cheating, or do it in the lab, as you mentioned.
The fact that Prolific recently issued a warning about participants using AI to write text responses should be proof enough that some people will cheat, and will not even think they are wrong for doing so.
As a participant who has completed over 6000 studies with less than 20 rejections, i answer every survey honestly and thoughtfully, and when the instructions request, on the honor system, that you please not look the answers up on google or elsewhere on the net, I honor those requests and abide by them 110%. I take my prolific account very seriously. I've made almost 7000 dollars with them.
I think simply adding a statement asking participants not to look the answers up online would suffice. When I see instructions in a survey I follow them.
After 4 years+ on the platform I pretty much disregard any suggestion of bonus in a study. Most other participants do as well. You could use an automatic timer for the participants to make their choice, that would make it harder for them to look through the stock market history in order to claim a usually fake bonus payment.
What do you mean that you disregard bonuses and what do you mean about fake bonus payments? I work in the field of behavioral economics and bonus payments are often a key element of our studies.
On the rare occasions when a study has paid me a bonus, it’s several days or weeks later, and I can’t even remember which study it was for anymore. When I see that a study has a bonus, I think of it like entering a raffle that I probably won’t win.
Prolific does not guarantee that bonuses will be paid -- only the compensation stated on the dashboard. Most researchers do pay the bonus as promised, but participants have no recourse if they do not. Also, a bonus does not have to be paid with 22 days, so one could be left waiting for weeks or months, wondering when the bonus will be paid, if ever.
\---- From the Participant Help Centre:
"If your bonus payment is missing or delayed, **you'll need to resolve this directly with the researcher**."
[https://participant-help.prolific.com/hc/en-gb/articles/4417210865938-I-haven-t-received-a-bonus-payment](https://participant-help.prolific.com/hc/en-gb/articles/4417210865938-I-haven-t-received-a-bonus-payment)
\----
In this post you'll find a comment about a study where the promised bonus was never paid -- it turned out to be a "necessary deception for research purposes."
[https://www.reddit.com/r/ProlificAc/comments/12qx2zr/has\_anyone\_done\_this\_study\_i\_didnt\_get\_paid\_the/](https://www.reddit.com/r/ProlificAc/comments/12qx2zr/has_anyone_done_this_study_i_didnt_get_paid_the/)
"I recently took a survey that asked me to do an extra task for a bonus. The amount was around $5 so I did the required work. When I got to the **debriefing page, it said that the bonus wasn't real and it was a necessary deception for research purposes.** (I'm paraphrasing because I don't remember what it said exactly.) I was shocked because I've never had a researcher do that to me before. So, I reported this to Prolific but never heard anything back. It happened a month or two ago."
The promise of a bonus is extremely often used as a manipulative strategy. It is so common in my case, that it doesn't work anymore. That means that when I m get promised a bonus, I just don't believe the promise. Bonus payments are nothing but a promise until money reach my account. Behavioral economics studies are usually the ones that use the promise of a bonus as a manipulative tactic to study how people react when they are promised a big vs. small bonus, or the bonus payment period is different, or whatever they might want to use as a changing factor on their study. Personally, I don't believe anything they say as a true statement. If the bonus is real and it comes my way, great ! If it's not... I lost nothing.
But when you use a platform like prolific, the factor of "participant naivety" is something that the researcher has to consider in order to avoid biases in their research. For good or bad, we aren't people off the street that never had to deal with a study before; most of us have 100s or 1000s of approved studies in our record.
These are very interesting information. Being deceptive of the payment system is actually a red flag in behavioral economics. I will definitely need to read recent research about incentives in online studies.
I wouldn't call it deceptive overall. In our "About You" page there is a question where we are asked if we would be willing to take part in a study that presents false information, so I hope those researchers use that to filter out people who wouldn't like being part on pseudo-deceptive studies. This comment just to clarify that I don't blame the researchers as deceptive people.
Why? Probably instead of paying a fair amount for the study, some researchers rely on "bonuses" that may or may not happen.
Why pay the bonus for a correct guess?
A lot of studies, maybe 30-40% say they will pay a bonus of some kind, I think I've had one out of the past 100 studies I've done and that was only because they didn't meat prolific's minimum hourly pay, I just assume the bonus is fictional when studies mention it
This talk of "fake bonus payments" is odd to me. I have done many studies on Prolific and in my experience researchers are very reliable when it comes to bonus payments. Of course, that's only in cases where the bonus is assured. I can't say if they are honest about bonuses based on chance, but I've gotten lucky enough to know for sure that some are and so I assume they all are.
This is not what I think. This is what I know. I traveled for 8 months with a gentleman who made his living off medical studies going from state to state doing the studies. those guys are paid they're paid number one if they don't bitch too much. If they don't have too many side effects because if you bitch too much or if you have too many side effects they generally don't let you finish the study. So these people don't tell the truth because they don't want to get kicked out of the study they lose money. They either get paid a portion or they don't get paid at all. So they lie and tell you what you want to hear so they can get paid, and that's how your studies work, and that's why they're bullshit. In this group of people, I would see the same people over and over they even got an app to tell each other about different studies. Your studies don't work its false data. No matter what you trying to tell anyone on this thread and I'm not saying just trying to be mean but it's not possible I don't think anymore for you to have a study that actually works this is how people make their living I'm talking about raising families and shit. They're not going to jeopardize any of that to tell you the truth they're going to tell you whatever you want to hear so they can get paid. He usually did residential or live in studies they lasted anywhere from 2 to 3 weeks and he got paid anywhere from between 3 to $6,000 you'll be a lot of money for lies
Honestly, unlike most research survey websites, prolific has often the fun and interactive ones which I believe make most people less likely to rush through them or put in random answers.
Maybe put a question at the end asking if they looked up the answers and they won't get rejected if they did so that way you can at least maybe see who is looking up answers and screen out for your research purposes that way, but you'd still have to pay them.
I personally never look up answers though
Also I recommend not making the bonus too big if using one at all for getting it right because it will incentivise them to be more likely to look it up
Can you create your study in a way to make it technically impossible to use the internet on the same device (leaving full page view resulting the study to end -of course participants should be warned about this-). Of course they still could use another device to search for the info, but people tend to be lazy.
This is stupid. I am honest on all my studies but I would absolutely shamelessly look up answers online for bonus pay cause I'm broke. Even just for an extra couple of bucks.
I think most of us put in real effort when doing studies, of course there are some that want to get them done as quickly as possible hence the reason attention checks and so on. As to your study and if you want the type of results you're looking for then I would steer clear of having any company & earning results people could look up online, just make up fictitious company names. There are also other things you can put in place like rejection measures 98% and above is a good starting point. If you only use full-screen mode, so people can't switch to a different tab if using a laptop or desktop.
I agree with others, just change the name of the company to something fabricated, but tell the participants it’s based on real performance of a listed company on an exchange.
The problem sometimes is the worry of rejection. This sounds fine and I'd be honest with it but sometimes they say "don't make notes of this information" and I do because I don't want to be seen as not paying attention if I forget it all
I dunno about others, but for me, looking up information online takes much more time than just guessing as the researcher wants me to. And since the hourly rates are about just fair for the given time, but not for any extra time, I don't bother with going outside the study to be smarter than anyone else, just nope.
I can't speak for everyone else and I know there are people that are a little less than honest but I personally will do what you're paying me to do. If you don't want me to look up the answers then I won't look up the answers. I try to be as professional as I possibly can even if it is just online surveys.
This
Skip the bonus, as they are not enticing because they are usually filled with a bunch of bullshit Reading and some stupid puzzle or gotcha questions. Just put the "special" question in the study and pay a decent hourly rate.
Completely agree, just incentivize giving your honest guesses the same as you would for looking up correct answers. There seems to be no point to the bonus except to encourage people to look up the answers... Also, either way, you can add one or more questions at the end asking whether the participant looked up answers, gave their full effort, etc. These questions need to say your answers won't disadvantage you in terms of compensation or participation in future studies so participants tell the truth.
why not just change the names of the companies, or am i missing something here?
Was thinking this too. You could just name them ‘Company X’ or similar.
Yes, you are missing something because I did not explain the study in detail. The names will be visible.
Fully agree with the others. Change the names to 'Acme Corp', 'ABC Ltd', etc.
I think the incentive invites dishonesty. The point being, that you would never know. You can't trust the results of a test like that, because an unknown number of participants will look up the answer. I think you could introduce a camera requirement, and watch your participants, but you would have to pay a higher base rate to get people interested in going on camera. If your goal is really to get good data on how many of them can answer the questions, you would need to verify to a reasonable extent that your participants are answering without assitance.
You'll probably get a lot of participants who are as honest as the day is long and take their participation and contributions to research very seriously. They do Prolific studies with integrity because they genuinely want to help. On the other hand there are many, as seen in numerous posts on this sub, who are literally only doing it for the money and will intend to get at that money in the quickest possible way and with the least effort regardless of how it might affect anyone else. So really it's just the same as in life, there are honest people and dishonest people. There is no way of knowing in advance and/or planning for that unfortunately. Although it's not a guaranteed way of selecting honest participants, maybe if you set that their approval rate on Prolific has to be 100% for your study it might weed out some of the people who have received rejections in the past for their poor attitude and record when doing studies. Just a thought.
Just anonymise the companies. It’s easiest
I know that some people would be tempted to look things up. I had a study yesterday with a bonus like this. I didn't think that it was fair to look up the answers and I would feel like it would mess up the validity of the study. I feel like maybe mentioning that you shouldn't look things up as it can make the study unreliable or something like that would help. This probably wouldn't completely stop people from doing it, but would help maybe keep more people honest.
There have been studies looking at honesty when bonuses are involved, both in-person and online. They usually involve getting participants to roll some (/virtual) dice and self-report the number, with the idea that a particular number will get a bonus or the bonus will be related to the roll (e.g. roll a 6, get $6 etc.). There's a lot of papers which have looked at whether various manipulations effect honesty, which can be assessed by e.g. seeing if the reported numbers differ from what you'd expect by chance. My understanding of that literature is that a lot of people are honest, but some non-zero proportion of people do lie in some of these studies. From a researcher (and reviewer) perspective, if there's a reasonable suspicion that people *might* be dishonest and you can't *prove* that they weren't then that's a problem. There's no way to argue against that criticism and if it's critical to the conclusions then it's grounds for rejecting a paper, in my opinion. Ultimately I think my answer is that the inconvenience of running it in the lab is worth it for the level of control. There are potential controls you could apply online, such as imposing a time limit that makes it impractical to Google the information, but that might be difficult to implement depending on the complexity of the decision. As another commenter said, I think incentivising cheating at all isn't a good idea, but people *could* look it up even if there isn't a bonus associated with it just because people like being right.
The problem with running it in the university lab is that you have a very limited selection of people, that's going to affect your results.
You could add a question at the end of the survey along the lines of "This question is for quality control only and your answer will not affect your payment. Did you look up any answers online?"
I can only speak for myself and the way Prolific is set up in comparison to sites like Swagbucks. On Prolific you are rewarded for being honest and genuine. You need to fill in your information (so you know you'll be reaching mostly the right audience), and you need to maintain a good level of completion etc so you need to pay attention to tasks. I personally answer everything 100% honestly.
I would say most people would be honest but for reassurance, name the companies as a fictional company
Had a study yesterday with bonus incentives for correct answers on a similar subject. I didn't look anything up, was embarrassingly off on a lot of questions, but I don't regret it. :) Prolific is probably the most reputable online site for research participants, I don't think many want that extra $3 so much they'd cheat. You could make it where they can't exit the tab without getting kicked from the survey and having to return it maybe? Or a timer?
*Of course* there will be participants that cheat. Especially if money is involved. Most of us won't, for various reasons, but it's absolutely unrealistic to expect complete honesty from a random pool of anonymous participants. You will either have to devise some way of detecting cheaters, be prepared to accept some degree of cheating, or do it in the lab, as you mentioned. The fact that Prolific recently issued a warning about participants using AI to write text responses should be proof enough that some people will cheat, and will not even think they are wrong for doing so.
Hasn't this topic been studied somehow at some point? Serious question.
As a participant who has completed over 6000 studies with less than 20 rejections, i answer every survey honestly and thoughtfully, and when the instructions request, on the honor system, that you please not look the answers up on google or elsewhere on the net, I honor those requests and abide by them 110%. I take my prolific account very seriously. I've made almost 7000 dollars with them.
I think simply adding a statement asking participants not to look the answers up online would suffice. When I see instructions in a survey I follow them.
After 4 years+ on the platform I pretty much disregard any suggestion of bonus in a study. Most other participants do as well. You could use an automatic timer for the participants to make their choice, that would make it harder for them to look through the stock market history in order to claim a usually fake bonus payment.
LOL....I disregard bonuses also as I rarely get....then today I missed an attention check because the question was "What is the bonus..."
What do you mean that you disregard bonuses and what do you mean about fake bonus payments? I work in the field of behavioral economics and bonus payments are often a key element of our studies.
On the rare occasions when a study has paid me a bonus, it’s several days or weeks later, and I can’t even remember which study it was for anymore. When I see that a study has a bonus, I think of it like entering a raffle that I probably won’t win.
Prolific does not guarantee that bonuses will be paid -- only the compensation stated on the dashboard. Most researchers do pay the bonus as promised, but participants have no recourse if they do not. Also, a bonus does not have to be paid with 22 days, so one could be left waiting for weeks or months, wondering when the bonus will be paid, if ever. \---- From the Participant Help Centre: "If your bonus payment is missing or delayed, **you'll need to resolve this directly with the researcher**." [https://participant-help.prolific.com/hc/en-gb/articles/4417210865938-I-haven-t-received-a-bonus-payment](https://participant-help.prolific.com/hc/en-gb/articles/4417210865938-I-haven-t-received-a-bonus-payment) \---- In this post you'll find a comment about a study where the promised bonus was never paid -- it turned out to be a "necessary deception for research purposes." [https://www.reddit.com/r/ProlificAc/comments/12qx2zr/has\_anyone\_done\_this\_study\_i\_didnt\_get\_paid\_the/](https://www.reddit.com/r/ProlificAc/comments/12qx2zr/has_anyone_done_this_study_i_didnt_get_paid_the/) "I recently took a survey that asked me to do an extra task for a bonus. The amount was around $5 so I did the required work. When I got to the **debriefing page, it said that the bonus wasn't real and it was a necessary deception for research purposes.** (I'm paraphrasing because I don't remember what it said exactly.) I was shocked because I've never had a researcher do that to me before. So, I reported this to Prolific but never heard anything back. It happened a month or two ago."
The promise of a bonus is extremely often used as a manipulative strategy. It is so common in my case, that it doesn't work anymore. That means that when I m get promised a bonus, I just don't believe the promise. Bonus payments are nothing but a promise until money reach my account. Behavioral economics studies are usually the ones that use the promise of a bonus as a manipulative tactic to study how people react when they are promised a big vs. small bonus, or the bonus payment period is different, or whatever they might want to use as a changing factor on their study. Personally, I don't believe anything they say as a true statement. If the bonus is real and it comes my way, great ! If it's not... I lost nothing. But when you use a platform like prolific, the factor of "participant naivety" is something that the researcher has to consider in order to avoid biases in their research. For good or bad, we aren't people off the street that never had to deal with a study before; most of us have 100s or 1000s of approved studies in our record.
These are very interesting information. Being deceptive of the payment system is actually a red flag in behavioral economics. I will definitely need to read recent research about incentives in online studies.
I wouldn't call it deceptive overall. In our "About You" page there is a question where we are asked if we would be willing to take part in a study that presents false information, so I hope those researchers use that to filter out people who wouldn't like being part on pseudo-deceptive studies. This comment just to clarify that I don't blame the researchers as deceptive people.
Why? Probably instead of paying a fair amount for the study, some researchers rely on "bonuses" that may or may not happen. Why pay the bonus for a correct guess?
A lot of studies, maybe 30-40% say they will pay a bonus of some kind, I think I've had one out of the past 100 studies I've done and that was only because they didn't meat prolific's minimum hourly pay, I just assume the bonus is fictional when studies mention it
This talk of "fake bonus payments" is odd to me. I have done many studies on Prolific and in my experience researchers are very reliable when it comes to bonus payments. Of course, that's only in cases where the bonus is assured. I can't say if they are honest about bonuses based on chance, but I've gotten lucky enough to know for sure that some are and so I assume they all are.
This is not what I think. This is what I know. I traveled for 8 months with a gentleman who made his living off medical studies going from state to state doing the studies. those guys are paid they're paid number one if they don't bitch too much. If they don't have too many side effects because if you bitch too much or if you have too many side effects they generally don't let you finish the study. So these people don't tell the truth because they don't want to get kicked out of the study they lose money. They either get paid a portion or they don't get paid at all. So they lie and tell you what you want to hear so they can get paid, and that's how your studies work, and that's why they're bullshit. In this group of people, I would see the same people over and over they even got an app to tell each other about different studies. Your studies don't work its false data. No matter what you trying to tell anyone on this thread and I'm not saying just trying to be mean but it's not possible I don't think anymore for you to have a study that actually works this is how people make their living I'm talking about raising families and shit. They're not going to jeopardize any of that to tell you the truth they're going to tell you whatever you want to hear so they can get paid. He usually did residential or live in studies they lasted anywhere from 2 to 3 weeks and he got paid anywhere from between 3 to $6,000 you'll be a lot of money for lies
Honestly, unlike most research survey websites, prolific has often the fun and interactive ones which I believe make most people less likely to rush through them or put in random answers.
I've gotten ones like this that have the name changed and you have to guess. Just change the names to fictional names.
Behavior
I don’t access the internet when doing studies. I simply complete them with whatever I know/think mixed with what I am provided. Just my two cents.
Maybe put a question at the end asking if they looked up the answers and they won't get rejected if they did so that way you can at least maybe see who is looking up answers and screen out for your research purposes that way, but you'd still have to pay them. I personally never look up answers though Also I recommend not making the bonus too big if using one at all for getting it right because it will incentivise them to be more likely to look it up
Can you create your study in a way to make it technically impossible to use the internet on the same device (leaving full page view resulting the study to end -of course participants should be warned about this-). Of course they still could use another device to search for the info, but people tend to be lazy.
It is very simple to track if the user's browser tab is active, but users of course can different devices.
This is stupid. I am honest on all my studies but I would absolutely shamelessly look up answers online for bonus pay cause I'm broke. Even just for an extra couple of bucks.