1490809 - Make a Nightly build for user testing of Milestone 3 (M3)

Status: RESOLVED FIXED

(Firefox :: WebPayments UI, enhancement, P1)

Description

[Matthew N. [:MattN]]

We need a self-extracting build of Firefox Nightly for users to use for user testing of Milestone 3.

See https://github.com/bgrins/firefox-user-testing-helpers for some helpers, instructions and ideas.

Comment 1

[Matthew N. [:MattN]]

Attachment: Bug 1490809 - Make a build for user testing of Payment Request. r?jaws,sfoster

Comment 2

[Matthew N. [:MattN]]

Here is a version 1 build (for Windows) ready for review:
https://drive.google.com/file/d/1gHNjg5V-lwo8JbHgFHWkXx3xXlNgaVPX/view

Comment 3

[Sharon Bautista (she/her) [:sbautista]]

I clicked through the build according to the research plan, and I noticed the following things. Can you please let me know if this is as intended?

When entering my billing address on the first screen for FTU, as I started typing, I saw a red error flash at the top of the modal. It might have been the "can't ship" error. I continued typing my address and the error did not re-appear. If this is a bug, I'm worried it might be off-putting to participants at the start of the interview.

Later in the flow, I also saw the invalid email error as I was typing my email address even though I wasn't done typing and had not moved on to another field.

I may have missed this in the design process, but is there a reason we're calling Preferences Firefox Options? None of the participants may notice, but once they're in Preferences, the address bar says Preferences, not Options.

Comment 4

[Matthew N. [:MattN]]

(In reply to sbautista from comment #3)
> I clicked through the build according to the research plan, and I noticed
> the following things. Can you please let me know if this is as intended?

Thanks for testing it out!

> When entering my billing address on the first screen for FTU, as I started
> typing, I saw a red error flash at the top of the modal. It might have been
> the "can't ship" error. I continued typing my address and the error did not
> re-appear. If this is a bug, I'm worried it might be off-putting to
> participants at the start of the interview.

If you mean shipping address, not billing address, then this is something I was also concerned about and was talking with UX about it last week but we thought it would be only visible if the user had an error so decided it was fine. Now I remember that it shows between the time the user clicks Next (which sets the selectedShippingAddress) and the time that the merchant responds with shipping options so it will show for everyone for a short amount of time. This is bug 1481243 but it depends on a DOM fix for the proper solution. I spent a while trying to figure out temporary solutions for the user test but didn't have much luck other than just hiding all generic errors on the credit card page.

> Later in the flow, I also saw the invalid email error as I was typing my
> email address even though I wasn't done typing and had not moved on to
> another field.

Hmm… that should only be the case if the field was already in the red invalid state before you focus the field. If not, can you file a bug blocking this one: https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=WebPayments%20UI&blocked=1490809

> I may have missed this in the design process, but is there a reason we're
> calling Preferences Firefox Options? None of the participants may notice,
> but once they're in Preferences, the address bar says Preferences, not
> Options.

That is the name that we always use on Windows/Linux. If you look in the main menu it should match there. Yes, the URL always says about:preferences but that's not specific to Web Payments so it's probably fine for the test.

Comment 5

[Jacqueline Savory [:jsavory] UX]

1. Sharon we talked about the error in our sync today, and will hide the error for the user study so people shouldn't see this. Unless they actually do have a shipping address that is not accepted and the merchant sends an error. 

2. Matt's understanding is correct, it should only appear if you task away from the field and then return to it. It shouldn't be appearing when you are just typing. 

3. I never knew this! I agree that we want what is in the UI, so Options makes sense for Windows and Preferences makes sense for Mac.

Comment 6

[Matthew N. [:MattN]]

(In reply to Jacqueline Savory [:jsavory] UX from comment #5)
> 1. Sharon we talked about the error in our sync today, and will hide the
> error for the user study so people shouldn't see this. Unless they actually
> do have a shipping address that is not accepted and the merchant sends an
> error. 

I'm making a new build with those changes and I will provide the new link when it's ready.

Try push: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=879ed6451e38e25b69638a9a5c2a14db6ff7fc2d


> ** Otherwise "More Info" => "Run Anyway" need

Sharon, was the Windows 10 security dialog ok to work with?

Comment 7

[Sharon Bautista (she/her) [:sbautista]]

Matt and Jacqueline: Thanks for the input. I will double check on the email error. And I learned something new about Preferences being called Options on Windows! 

The Windows 10 security dialog was fine. I noted the warning language for myself and will add a reminder in my discussion guide to reassure participants before that dialog comes up.

Comment 8

[Matthew N. [:MattN]]

Here is a version 2 build with the flash of the shipping error addressed: https://drive.google.com/file/d/1_iOcNpxPJ97NeXAyidyQSX0v5IbI4vVC/view

Comment 9

[Matthew N. [:MattN]]

Sharon noticed the lack of bug 1483412 in her test this time so I did a new try push with that cherry-picked on top of the same revision as version 2 so that I don't include bug 1429265 which needs more work.

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=91f6cdbbcdf12f3e2ad6d9896938de711557f061

Comment 10

[Matthew N. [:MattN]]

Here is the version 3 build: https://drive.google.com/file/d/1fBXMcIA0XWrufvKTGnQLv1VrP0JszCrL/view

I'm going to mark this as resolved now but if you find issues feel free to comment.

Comment 11

[Matthew N. [:MattN]]

Version 4 build: https://drive.google.com/file/d/1oRaOs_O1ACuq1TB87Ln7vOXKj5gQEQ69/view

The only changes are in the temp. profile:

+user_pref("dom.payments.response.timeout", 20000);
+user_pref("browser.crashReports.unsubmittedCheck.enabled", false);

These are based upon things noticed in the first two tests.

Comment 12

[Matthew N. [:MattN]]

Comment on attachment 9019252 [details]
Bug 1490809 - Make a build for user testing of Payment Request. r?jaws,sfoster

We have been using this fairly successfully for 3 participants already.

Comment 13

[Matthew N. [:MattN]]

Version 5 build: https://drive.google.com/file/d/17JTyx8eI72agowfAoYiZTiULmQKmLK05/view

Backs out https://hg.mozilla.org/mozilla-central/rev/95fe4f9c3203#l1.12 due to bug 1505141.

Comment 14

[Matthew N. [:MattN]]

Here are the revisions used for version 5:

o   443745:c45e8f2204c9 mozilla
|  Bug 1490809 - Make a build for user testing of Payment Request. r?jaws,sfoster
o   443744:75e9f4d8f4fd jwein
|   Bug 1483412 - Make the telephone number field required on the payer form if requestPayerPhone is true. r=MattN
o     443743:d71404ed02ef aiakab
|\   Merge inbound to mozilla-central a=merge

Comment 15

[Hani Yacoub, Desktop QA]

Removing the qe-verify+ flag since the feature is disabled.