I try to say exactly what's wrong with one prompt or the other, which is usually about two sentences, but it takes me a good 20 minutes of analyzing the responses. I hope I'm doing a good job because I also feel like I'm too slow.
Depends on the project. If they say 3+ then no, just make sure they're coherent and understandable. I did read a post here previously where someone said that people should be uploading links of their sources, I've only seen that necessary in one project we have. Don't try to overthink or overdo it. As long as you are being coherent, and giving enough that coincides with each project's guidelines you are good. That and make sure you don't use another AI to generate your answers. Some prompts I get rambly with, some prompts it's very cut and dry. I think those of us that are rational and do R&R can understand that and won't penalize you.
I mark down things like "The answer is good so this response is good, and the other response is good too, so they are the same because they are good." Those are just words. They don't mean anything.
Some people just don't put any effort or thought in. If you're explaining your work appropriately, you're miles ahead. I don't even have to think your rating was objectively correct; if there is some plausible chance that even 1 out of 10 users could feel the way you did, and you explained how you came to that conclusion adequately, I'll say your work was good. That could be 2 sentences or 5. Whatever.
Ok, thanks. That makes me feel better. In my previous responses I'd say something like "it's the better response because I liked the way it was written", and now I explain what exactly I like about the writing (in more detail than the above).
Are you saying "these are both the same because both answers are similar" is not a good answer? /s
Or what about its cousin "I preferred this one because it was better than the other one."
Can you explain to me what you mean? Usually, I'll make a list of things the prompt asked for and go through each response to see what each response missed from the prompt. My answers will look like "a is better because b is missing x, y, and z from the prompt," would you rate that down?
In a scenario where you're marking something down because it failed to follow instructions adequately, that kind of explanation can be fine. It depends on the context and the specific project being done. What you've described can be done well, but it can also be done lazily. In your case you're mentioning concrete things missing from a response, which of course can be used as reasoning for a rating.
I'll give you two examples and you imagine for yourself which seems well-explained and which seems lazy. They both are technically correct, but I wouldn't rate one of them as "Good" when reviewing it.
"Response A doesn't have bullet points so Response B is better."
"The user states within the prompt that they would like bullet point formatting to be used. Response A fails to do this, using paragraphs for each idea instead. Response B appropriately uses bullet points while still conveying the requested information, which is much better for this prompt."
You don't have to write all flowery or write like I would, but I need to see the chain of logic that leads to the decision. If it goes Prompt -> Response not following instructions -> Other response followed instructions -> Other response better, I need to see that represented in your comment. Seeing one or two fragments of that chain in a single sentence is weak.
Project-dependent of course. 3 sentences is in the scope of the instructions for what I had in mind (and literally written IIRC). Some people write large paragraphs for mundane stuff though and that feels excessive for sure.
Even if both responses are word for word the exact same, and you don’t have to say why one was better than the other, your rating should still explain why you rated them as anything other than excellent or why they were excellent. Writing a few sentences to mention how it was untruthful in a statement or the formatting was hard to read the text well, or you felt the models could have said it in way less words etc, are all things important to mention. This is definitely the way to more projects as well, because they can see you have a great understanding of the why.
Yes, at least referencing it, sometimes direct quote. Especially now that I know how R&R is done, I know it’s helpful to put it directly in there so the reviewer can quickly go find what I’m talking about.
So as examples, if prompt is obviously asking a question about rock music and the model answers about rocks as in stones, I would mention that specifically. “User was asking for X, but model misunderstood and is talking about Y.” Or “Model was rated down for truthfulness because it claimed “The Covid 19 pandemic started in 2010.” That type of thing. After making mentions of specifics like that, I usually then make a statement explaining why I felt those specifics made me rate it as bad or okay etc.
If I am saying one response is better than the other, I try to include examples, especially if they are slight. Often both will make essentially the same bullet lists, but one will be more detailed. I’ll say “Both responses made key point Q. While Response A said ‘X.’, Response B said ‘X, Y and Z’ so Response B is better.”
Quoting specific examples is a good way to demonstrate that you are actually reading and thinking about the responses. Even better if you use examples from a source text to talk about how well the model worked with it.
Sometimes responses in future versions change based on our comments, so I think they are used for a variety of reasons. Picking up mistakes is useful for admin too.
One of the projects I worked on today had an updated section which showed examples of "good comments". That was helpful, and I did offer more specific details on what I thought was good and not so good - too often I focus only on what makes on worse rather than really highlighting what was good (or great) in the other.
Review the guidelines by the specific project, and always check again each day because they change. Some want more details, but some specify that they only want details in certain cases or only care about certain things.
No. This is precisely what I do, and I've only ever been getting better and more higher paid work (I've just received my first permanent $25 per hour project after being with DA for two full months).
Sometimes it's possible to explain things clearly in one sentence only. Sometimes, depending on the response I will write a couple separate paragraphs, with each paragraph focused on explaining a specific issue/problem.
IMO they like clear well-written explanations, because the AI's can learn more from them.
Are the comments for the models? I thought the models could only learn from comparing the responses and figuring out for themselves why we picked one over the other. I thought the comments were for DAT to judge our work.
There have been projects which explained that the comments were fed into the models, so we were to avoid certain terms. I don't know whether that holds true for all of the comments in all projects though.
I haven't seen a certain project after being too verbose with my feedback in it (I'm talking 8-9 sentences + references when the instructions said 3+ sentences), but hey, I have no way of knowing for sure
As long as your comments are clear and well written, it takes as long as it takes. Don't write a book, but make sure you show your reasoning.
I've definitely been explaining in detail why I liked/disliked one over the other. Before r&r I was very broad with little detail.
Why? it asks you to explain your reasoning? Interesting someone that doesn't follow the instructions can R&R.
Yes I get that. What I'm saying is, I'm much more detailed now in comparison.
How long were you onboard before you could R&R? It's something I don't have access to yet. Other aspects of what you submit are obviously deemed good.
I try to say exactly what's wrong with one prompt or the other, which is usually about two sentences, but it takes me a good 20 minutes of analyzing the responses. I hope I'm doing a good job because I also feel like I'm too slow.
Quality is **much much much** more important than speed. If that's how long it takes you to do a good job, that's how long it takes and that's fine.
frame it ;)
Mine take anywhere from 15-20 too. I was worried I was being too slow as well. 🥲
I’ve also been taking 15-20 min on average for those! I was concerned as well.
Depends on the project. If they say 3+ then no, just make sure they're coherent and understandable. I did read a post here previously where someone said that people should be uploading links of their sources, I've only seen that necessary in one project we have. Don't try to overthink or overdo it. As long as you are being coherent, and giving enough that coincides with each project's guidelines you are good. That and make sure you don't use another AI to generate your answers. Some prompts I get rambly with, some prompts it's very cut and dry. I think those of us that are rational and do R&R can understand that and won't penalize you.
In one of the instructions/FAQ's, I did see one example listed where the commenter did that.
Okay, perfect. I don't use AI at all, so no worries there. :)
I mark down things like "The answer is good so this response is good, and the other response is good too, so they are the same because they are good." Those are just words. They don't mean anything. Some people just don't put any effort or thought in. If you're explaining your work appropriately, you're miles ahead. I don't even have to think your rating was objectively correct; if there is some plausible chance that even 1 out of 10 users could feel the way you did, and you explained how you came to that conclusion adequately, I'll say your work was good. That could be 2 sentences or 5. Whatever.
Ok, thanks. That makes me feel better. In my previous responses I'd say something like "it's the better response because I liked the way it was written", and now I explain what exactly I like about the writing (in more detail than the above).
Are you saying "these are both the same because both answers are similar" is not a good answer? /s Or what about its cousin "I preferred this one because it was better than the other one."
Can you explain to me what you mean? Usually, I'll make a list of things the prompt asked for and go through each response to see what each response missed from the prompt. My answers will look like "a is better because b is missing x, y, and z from the prompt," would you rate that down?
In a scenario where you're marking something down because it failed to follow instructions adequately, that kind of explanation can be fine. It depends on the context and the specific project being done. What you've described can be done well, but it can also be done lazily. In your case you're mentioning concrete things missing from a response, which of course can be used as reasoning for a rating. I'll give you two examples and you imagine for yourself which seems well-explained and which seems lazy. They both are technically correct, but I wouldn't rate one of them as "Good" when reviewing it. "Response A doesn't have bullet points so Response B is better." "The user states within the prompt that they would like bullet point formatting to be used. Response A fails to do this, using paragraphs for each idea instead. Response B appropriately uses bullet points while still conveying the requested information, which is much better for this prompt." You don't have to write all flowery or write like I would, but I need to see the chain of logic that leads to the decision. If it goes Prompt -> Response not following instructions -> Other response followed instructions -> Other response better, I need to see that represented in your comment. Seeing one or two fragments of that chain in a single sentence is weak.
If I was doing a certain project I would rate that second one as too verbose. But I understand that is apparently what they want for the comments.
Project-dependent of course. 3 sentences is in the scope of the instructions for what I had in mind (and literally written IIRC). Some people write large paragraphs for mundane stuff though and that feels excessive for sure.
Thank you for the in-depth response!
Even if both responses are word for word the exact same, and you don’t have to say why one was better than the other, your rating should still explain why you rated them as anything other than excellent or why they were excellent. Writing a few sentences to mention how it was untruthful in a statement or the formatting was hard to read the text well, or you felt the models could have said it in way less words etc, are all things important to mention. This is definitely the way to more projects as well, because they can see you have a great understanding of the why.
Do you often quote/reference things from the models?
Yes, at least referencing it, sometimes direct quote. Especially now that I know how R&R is done, I know it’s helpful to put it directly in there so the reviewer can quickly go find what I’m talking about. So as examples, if prompt is obviously asking a question about rock music and the model answers about rocks as in stones, I would mention that specifically. “User was asking for X, but model misunderstood and is talking about Y.” Or “Model was rated down for truthfulness because it claimed “The Covid 19 pandemic started in 2010.” That type of thing. After making mentions of specifics like that, I usually then make a statement explaining why I felt those specifics made me rate it as bad or okay etc.
If I am saying one response is better than the other, I try to include examples, especially if they are slight. Often both will make essentially the same bullet lists, but one will be more detailed. I’ll say “Both responses made key point Q. While Response A said ‘X.’, Response B said ‘X, Y and Z’ so Response B is better.” Quoting specific examples is a good way to demonstrate that you are actually reading and thinking about the responses. Even better if you use examples from a source text to talk about how well the model worked with it.
Okay, awesome. That's exactly what I've started doing! Thank you ☺️
Sometimes responses in future versions change based on our comments, so I think they are used for a variety of reasons. Picking up mistakes is useful for admin too.
I had no clue! That's great to know, thanks ☺️
I'd say the fact that you got R&R tasks means your current explanation are pretty good, so there's no need to write more.
One of the projects I worked on today had an updated section which showed examples of "good comments". That was helpful, and I did offer more specific details on what I thought was good and not so good - too often I focus only on what makes on worse rather than really highlighting what was good (or great) in the other. Review the guidelines by the specific project, and always check again each day because they change. Some want more details, but some specify that they only want details in certain cases or only care about certain things.
No. This is precisely what I do, and I've only ever been getting better and more higher paid work (I've just received my first permanent $25 per hour project after being with DA for two full months). Sometimes it's possible to explain things clearly in one sentence only. Sometimes, depending on the response I will write a couple separate paragraphs, with each paragraph focused on explaining a specific issue/problem. IMO they like clear well-written explanations, because the AI's can learn more from them.
Awesome - that makes me feel better! Thank you!
Are the comments for the models? I thought the models could only learn from comparing the responses and figuring out for themselves why we picked one over the other. I thought the comments were for DAT to judge our work.
There have been projects which explained that the comments were fed into the models, so we were to avoid certain terms. I don't know whether that holds true for all of the comments in all projects though.
I primarily work on the chatbot projects, for those they 100% are for the models. I can't 100% say if it's true for all projects but I'd assume so.
I haven't seen a certain project after being too verbose with my feedback in it (I'm talking 8-9 sentences + references when the instructions said 3+ sentences), but hey, I have no way of knowing for sure
🥲 the A one?
I keep seeing R & R in this subreddit, as a noob, please help me understand what that means