Last Comment Bug 701177 - Determine QA Process for Functional Snippets
: Determine QA Process for Functional Snippets
Status: VERIFIED FIXED
:
Product: Snippets
Classification: Other
Component: General (show other bugs)
: unspecified
: All All
: -- normal
: ---
Assigned To: Stephen Donner [:stephend]
:
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-09 13:30 PST by Michael Kelly [:mkelly,:Osmose]
Modified: 2012-01-08 18:30 PST (History)
4 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Michael Kelly [:mkelly,:Osmose] 2011-11-09 13:30:51 PST
Bug 682944 highlighted the fact that functional snippets need a better QA process to ensure snippets don't break about:home.

We need to investigate whether we want to define a manual testing process or look into running an automated test suite on snippets. We also need to check in with QA to get guidance on what to test for.
Comment 1 Matt Brandt [:mbrandt] 2011-11-09 13:34:36 PST
+1 from QA :-)
Comment 2 Michael Kelly [:mkelly,:Osmose] 2011-11-09 14:15:51 PST
What about setting up some selenium tests to check these things? Can we access localStorage in a selenium test? Also, can we pass parameters to an automated selenium test runner to indicate which snippet to test?
Comment 3 Michael Kelly [:mkelly,:Osmose] 2011-11-09 14:41:35 PST
So, a bit more background:

We want to test particular snippets on the about:home page. We need it to be within a browser session because some parts of about:home depend on code included within the browser.

about:home uses a URL stored in localStorage to download and inject snippets into the page. We can create snippets on a staging server and give them a specific URL so that only that one snippet is served to help in testing. We have an addon that can change the parameters in this URL to point to the specific snippet if we can't alter localStorage directly.

Right now the main things to test are the Restore Previous Session button and the search box. I'd like to be able to trigger a test runner and specify a URL to use to download snippets.
Comment 4 David Burns :automatedtester 2011-11-09 16:23:07 PST
This should be fairly trivial to do with Selenium WebDriver.

On IRC there was a mention of a extension so will show an example below. It will require some tweaking to run on the WebQA grid.

from selenium import webdriver
from selenium.webdriver.common.by import By

#Profile setup if needed
profile = webdriver.Profile()
profile.add_extension('path/to/extension')
profile.set_preference("foo", "bar")
profile.update_preferences()

#Getting a browser
browser = webdriver.Firefox(firefox_profile=profile)
browser.get("about:home")

#Finding and using elements on the page
search = browser.find_element(By.NAME, "q")
search.send_keys("Webdriver")

#Running JavaScript, like calling localStorage
driver.execute_script("return localStorage.getItem('foo');")


Once you have your tests running I would have a look at https://wiki.mozilla.org/QA/Execution/Web_Testing/Automation#Selenium to see how WebQA do things.

If you have any questions just yell!
Comment 5 Stephen Donner [:stephend] 2011-11-11 02:51:38 PST
I don't know how we'll cover the first part of the "Right now the main things to test are the Restore Previous Session button and the search box." case; David, won't we have to test that with Mozmill, since it's in chrome?
Comment 6 David Burns :automatedtester 2011-11-11 02:56:28 PST
@stephend

about:home loads a webpage. Since we just need to tell the browser to load the page we can then manipulate it with WebDriver. @mkelly hasn't suggested that he needs to access the browser chrome other than running a few JS calls for localStorage. If he needs to access that then we may need to look at Mozmill but if its just for the page then WebDriver is the way forward imho
Comment 7 Stephen Donner [:stephend] 2011-11-11 03:13:23 PST
(In reply to David Burns :automatedtester from comment #6)
> @stephend
> 
> about:home loads a webpage. Since we just need to tell the browser to load
> the page we can then manipulate it with WebDriver. @mkelly hasn't suggested
> that he needs to access the browser chrome other than running a few JS calls
> for localStorage. If he needs to access that then we may need to look at
> Mozmill but if its just for the page then WebDriver is the way forward imho

Yeah, cool; while I knew about:home loads a webpage (I and Raymond tested that in the service rewrite, in fact), I didn't realize the Restore Previous Session page was also HTML/XHTML (thought it might be chrome): http://mxr.mozilla.org/mozilla-central/source/browser/base/content/aboutHome.xhtml

So, good, good! :-)
Comment 8 Laura Forrest 2011-11-11 09:55:59 PST
@stephend Can you or whoever is on point from QA, document the QA process we establish here somewhere in the wiki? Perhaps somewhere around here? https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#Overview or wherever you see fit. Would be good to have! 

At that point I can go ahead and file a bug to start QA of existing functional snippets. Who should I assign those to?
Comment 9 Laura Forrest 2011-11-14 15:31:51 PST
StephenD Followed up via email and it looks like Raymond will be able to help us out to QA functional snippets until a new hire can work on in starting January. 

@Stephend - Is there anything else that should be included here in order to close this bug? I suggest documentation in Comment 8 but perhaps that can be done on the QA end of things if needed. For now, I'm comfortable assigning all functional bugs that need QA to Raymond. 

Yes?

Thanks!
Comment 10 Stephen Donner [:stephend] 2011-11-15 10:49:05 PST
(In reply to Laura Forrest from comment #9)
> StephenD Followed up via email and it looks like Raymond will be able to
> help us out to QA functional snippets until a new hire can work on in
> starting January. 
> 
> @Stephend - Is there anything else that should be included here in order to
> close this bug? I suggest documentation in Comment 8 but perhaps that can be
> done on the QA end of things if needed. For now, I'm comfortable assigning
> all functional bugs that need QA to Raymond. 

Laura/Mike, I've taken a stab at documenting a process over at https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#QA_Process; can you take a look?  Particularly, are the steps Les listed to test, in https://wiki.mozilla.org/Firefox/Projects/Firefox_Start/Snippet_Service#Using_a_non-production_service_with_about:home) still valid? 

> Yes?

Yes :-)
 
> Thanks!

You're welcome; sorry for the delay -- wanted to make sure we're approaching this correctly and sustainably.
Comment 11 Laura Forrest 2011-12-29 09:59:21 PST
Thanks Stephen - looks like this can be closed!
Comment 12 Stephen Donner [:stephend] 2012-01-08 18:30:44 PST
(In reply to Laura Forrest from comment #11)
> Thanks Stephen - looks like this can be closed!

Yep; we're taking care of this manually for now, and we'll put a Q1 goal for automation (to cover the essentials) on the map for 2012.

Verified FIXED.

Note You need to log in before you can comment on or make changes to this bug.