701177 - Determine QA Process for Functional Snippets

Status: VERIFIED FIXED

(Snippets Graveyard :: General, defect)

Description

[Osmose [:osmose, :mkelly]]

Bug 682944 highlighted the fact that functional snippets need a better QA process to ensure snippets don't break about:home.

We need to investigate whether we want to define a manual testing process or look into running an automated test suite on snippets. We also need to check in with QA to get guidance on what to test for.

Comment 1

[Matt Brandt [:mbrandt]]

+1 from QA :-)

Comment 2

[Osmose [:osmose, :mkelly]]

What about setting up some selenium tests to check these things? Can we access localStorage in a selenium test? Also, can we pass parameters to an automated selenium test runner to indicate which snippet to test?

Comment 3

[Osmose [:osmose, :mkelly]]

So, a bit more background:

We want to test particular snippets on the about:home page. We need it to be within a browser session because some parts of about:home depend on code included within the browser.

about:home uses a URL stored in localStorage to download and inject snippets into the page. We can create snippets on a staging server and give them a specific URL so that only that one snippet is served to help in testing. We have an addon that can change the parameters in this URL to point to the specific snippet if we can't alter localStorage directly.

Right now the main things to test are the Restore Previous Session button and the search box. I'd like to be able to trigger a test runner and specify a URL to use to download snippets.

Comment 4

[David Burns :automatedtester]

This should be fairly trivial to do with Selenium WebDriver.

On IRC there was a mention of a extension so will show an example below. It will require some tweaking to run on the WebQA grid.

from selenium import webdriver
from selenium.webdriver.common.by import By

#Profile setup if needed
profile = webdriver.Profile()
profile.add_extension('path/to/extension')
profile.set_preference("foo", "bar")
profile.update_preferences()

#Getting a browser
browser = webdriver.Firefox(firefox_profile=profile)
browser.get("about:home")

#Finding and using elements on the page
search = browser.find_element(By.NAME, "q")
search.send_keys("Webdriver")

#Running JavaScript, like calling localStorage
driver.execute_script("return localStorage.getItem('foo');")


Once you have your tests running I would have a look at https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation#Selenium to see how WebQA do things.

If you have any questions just yell!

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

I don't know how we'll cover the first part of the "Right now the main things to test are the Restore Previous Session button and the search box." case; David, won't we have to test that with Mozmill, since it's in chrome?

Comment 6

[David Burns :automatedtester]

@stephend

about:home loads a webpage. Since we just need to tell the browser to load the page we can then manipulate it with WebDriver. @mkelly hasn't suggested that he needs to access the browser chrome other than running a few JS calls for localStorage. If he needs to access that then we may need to look at Mozmill but if its just for the page then WebDriver is the way forward imho

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to David Burns :automatedtester from comment #6)
> @stephend
> 
> about:home loads a webpage. Since we just need to tell the browser to load
> the page we can then manipulate it with WebDriver. @mkelly hasn't suggested
> that he needs to access the browser chrome other than running a few JS calls
> for localStorage. If he needs to access that then we may need to look at
> Mozmill but if its just for the page then WebDriver is the way forward imho

Yeah, cool; while I knew about:home loads a webpage (I and Raymond tested that in the service rewrite, in fact), I didn't realize the Restore Previous Session page was also HTML/XHTML (thought it might be chrome): http://mxr.mozilla.org/mozilla-central/source/browser/base/content/aboutHome.xhtml

So, good, good! :-)

Comment 8

[Laura Forrest]

@stephend Can you or whoever is on point from QA, document the QA process we establish here somewhere in the wiki? Perhaps somewhere around here? https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#Overview or wherever you see fit. Would be good to have! 

At that point I can go ahead and file a bug to start QA of existing functional snippets. Who should I assign those to?

Comment 9

[Laura Forrest]

StephenD Followed up via email and it looks like Raymond will be able to help us out to QA functional snippets until a new hire can work on in starting January. 

@Stephend - Is there anything else that should be included here in order to close this bug? I suggest documentation in Comment 8 but perhaps that can be done on the QA end of things if needed. For now, I'm comfortable assigning all functional bugs that need QA to Raymond. 

Yes?

Thanks!

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Laura Forrest from comment #9)
> StephenD Followed up via email and it looks like Raymond will be able to
> help us out to QA functional snippets until a new hire can work on in
> starting January. 
> 
> @Stephend - Is there anything else that should be included here in order to
> close this bug? I suggest documentation in Comment 8 but perhaps that can be
> done on the QA end of things if needed. For now, I'm comfortable assigning
> all functional bugs that need QA to Raymond. 

Laura/Mike, I've taken a stab at documenting a process over at https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#QA_Process; can you take a look?  Particularly, are the steps Les listed to test, in https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#Using_a_non-production_service_with_about:home) still valid? 

> Yes?

Yes :-)
 
> Thanks!

You're welcome; sorry for the delay -- wanted to make sure we're approaching this correctly and sustainably.

Comment 11

[Laura Forrest]

Thanks Stephen - looks like this can be closed!

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Laura Forrest from comment #11)
> Thanks Stephen - looks like this can be closed!

Yep; we're taking care of this manually for now, and we'll put a Q1 goal for automation (to cover the essentials) on the map for 2012.

Verified FIXED.