Each person in an audience has the opportunity see and provide feedback on one idea per test. We intentionally do not show one person more than one idea per test. We are looking for immediate gut reaction from that person on whether that idea is interesting to them and why it is or is not interesting to them.
Our threshold for audience volume is at the idea level and applied across the entire test. Each idea is shown an equal number of times to individual people. We intentionally do not optimize for better performing ideas to get more audience volume and worse performing ideas to get less audience volume.
Thirty people must respond to an idea for the idea to have reached a sufficient audience volume to be considered viable. See below for example test sizes by number of ideas:
6 ideas = 180 total respondents (6 ideas * 30 responses per idea)
10 ideas = 300 total respondents (10 ideas * 30 responses per idea)
30 ideas = 900 total respondents (30 ideas * 30 responses per idea)
Any test that gets sufficient audience volume will provide viable results across all ideas.
Each idea needs to receive at least 30 responses to be considered valid. Any idea with less than 30 responses will show as grey until it hits 30 responses.
Grey: Any idea that has less than 30 total responses will show as grey. Grey means you don’t have enough responses yet. Share your test more broadly to get more responses.
After an idea hits 30 total responses it will show one of the following colors. Please note that ideas can move from one color to another over time as results aggregate, with the exception of grey. Once an idea receives 30 votes that idea will never turn grey again.
Green: Any idea that receives 70% or more Yes votes will show as green. So for 30 responses this would be 21 or more Yes votes. Green means your idea is a winner and you should continue investing in deepening, designing, and developing it for this audience. Check out the Plans tool for more thoughts on prioritization.
Red: Any idea that receives less than 50% Yes votes will show as red. So for 30 responses this would be 14 or less Yes votes. Red means people don’t really like this idea and you should NOT continue investing in it for this audience.
Yellow: Any idea that receives 50% to less than 70% Yes votes will show as yellow. So for 30 responses this would be 15-20 Yes votes. Yellow means people are split. You may want to tweak this idea or the audience you tested with and test again to get more clear signal.
Testing ideas quickly connects prioritization and planning with real users!
If you are struggling to get a high enough audience volume across your test consider revisiting the number of ideas and perhaps consolidate or eliminate some ideas. You may also consider breaking out multiple tests perhaps with different audiences.
If you are having trouble finding your audience, please reach out to us.