1005126 - Randomize the order of the WebWeWant Questions

Status: VERIFIED FIXED

(Websites :: glow.mozilla.org, defect)

Description

[Chris More [:cmore]]

Given user research, website heatmaps, and the majority of visitors reading left to right, the order of the six questions on the glow website could influence the results of the survey. Even though the grid of the six questions is pretty tight and users should be able to quickly glance through the answers, there could be a statistical impact to the order. Even if there was no statistical impact with the order now, the perception that the results is skewed by the order is real.

We should randomize the order of the six questions on all layout widths, including mobile, to eliminate the reality or perception that the results are skewed toward specific locations.

Comment 1

[Chris More [:cmore]]

Should we try the randomization at the front-end template level or back-end? Pros/cons? 

If users refresh or bring up the question modal, will the questions be in the same order or a new random order?

Will this impact the ability to cache the website? Impact to performance?

Comment 2

[Paul [:pmac] McLanahan]

Attachment: pull-request

Comment 3

[Paul [:pmac] McLanahan]

I did it in the template and view (back end). This means that the HTML order is the one that continues to be used. I did it that way for a few reasons:

1. Not adding more work to the client. We're already taxing the browser quite a bit.
2. No affect on cache: We currently cache the page for 60 seconds per locale. This is so that the HTML will contain as recent as possible a timestamp for the JSON. So the order will be the same for 60 seconds at a time per locale.

If a user refreshes they will likely see the same order. If they do a hard refresh at the right time they'll see a new random order. Having the order be the same for 60 seconds at a time should be no issue though for the volume of traffic and votes we're seeing.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla/mrburns

https://github.com/mozilla/mrburns/commit/b511f6e9b9b7933b3a244d095b1b6f03a89f4bfb
Fix bug 1005126: Randomize issue choice order.

https://github.com/mozilla/mrburns/commit/7809fd22846dec771ae316e9aef22b63630cc99b
Merge pull request #183 from pmclanahan/randomiz-choices

Fix bug 1005126: Randomize issue choice order.

Comment 5

[Paul [:pmac] McLanahan]

This is now deployed to the dev instance for testing:

https://webwewant.allizom.org/en/

Comment 6

[Rebecca Billings [:rbillings]]

Looks good on Safari, Chrome, FF.