Differences

This shows you the differences between two versions of the page.

--- research:rater_instructions [2024/02/10 21:22] – admin
+++ research:rater_instructions [2024/02/11 20:08] (current) – [Different sizes] admin
@@ Line 5: / Line 5: @@
 \\
 \\
@@ Line 39: / Line 41: @@
 The AI model was asked to draw images with 1 through 16 of the different objects. However, the AI model often gets the number wrong, and we'd like to know exactly how it goes wrong. Therefore, our job is to count all of the objects in the image, even if there are more than 16.
+[{{ :research:survey_phone_screen_top.png?300|An image to rate and the "more than 20" question}}]
 However, sometimes the AI model's image is so obviously wrong that it just doesn't seem worth it to count all its mistakes. So, **if and only if we're sure** that there are more than 20 objects in an image, we can click an option for "Definitely more than 20" which will make the response slider-thingies disappear for that trial, so that we don't have to give a precise answer.
-If it's not immediately clear whether there are more than 20 objects, I recommend starting to count them, and then stopping if you get close to 20 and there are still lots of objects left.
+Recommendation: If it's not immediately clear whether there are more than 20 objects, start to count them, and then stop if you get close to 20 and there are still lots of objects left.
+===== DO count objects even if only a little bit is visible =====
+Even if only a tiny sliver of an object is visible (because most of it is outside of the frame, or most of it is hiding behind another object), **please do count it**.
 ====== Problematic counting situations that need judgment ======
@@ Line 72: / Line 80: @@
 The second response slider-thingy asks for the number of boundary clipped items. "Boundary" just means "the edge of the picture" in this case. If less than **50% of an object** is shown in the frame of the picture, it counts as boundary clipped.
+[{{ :research:boundary_clipping_example.png|Boundary clipping examples}}]
+Note: Objects that are partly occluded by (hidden by) //other objects// **don't count** as boundary clipped.
+In the example below, I counted 13 medicine balls, with 2 boundary clipped.
 ====== Consistency of objects ======
@@ Line 81: / Line 94: @@
 Imagine that there is a picture of two hot air balloons that are the same real world size (say, 60 feet tall). If one is far away, it might only take up 1/4" on the screen, while one that is near the front of the picture might take up 2". **These two balloons would be rated to be //the same// size.**
+[{{ :research:survey_phone_screen_sliders.png?300|The slider-thingies and some of the check boxes}}]
 Recommendation: Allow a little tolerance so that objects that are very nearly, but not exactly the same size can count as "the same size." When objects are noticeably different size, then rate them as "different sizes."
@@ Line 92: / Line 107: @@
 Most of the time, the objects will be different colors. When all of the objects of one of our key types (bocce ball, medicine ball, or hot air balloon) are the same color, then select this response.
-===== Different object type present =====
+===== Other object type present =====
 For example, if an image shows a person along with a bunch of bocce balls, then select this response. The same goes for an image of 10 medicine balls plus one hammer.
@@ Line 106: / Line 121: @@
 ====== When to leave a comment ======
+===== When an other object type is present =====
+If you check the box for "Other object type present," please add a comment reading "other: X" where X is the name of the other type of object.
+===== Noteworthy image features =====
 The final response field allows us to check a box and then enter a comment about the image.
@@ Line 116: / Line 137: @@
   * The image was ambiguous or confusing ("I can't tell what that thing in the corner is supposed to be!")
   * The image contained something truly awful (not just weird, but offensive or disgusting)
-  * The objects formed an unusual pattern (not just any pattern, but a noteworthy one)
+  * The objects formed an unusual pattern. (Not just any pattern, but a truly noteworthy one. Examples: if all the objects form a nice square grid, you **don't** need to comment on that, but if the objects form the numeral "9" please do comment on that.)