Closed Bug 919923 Opened 11 years ago Closed 11 years ago

Create a section in the HTML report to output the build id, gecko, gaia version

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::UI Tests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
2.0 S1 (9may)

People

(Reporter: davehunt, Assigned: davehunt)

References

Details

Attachments

(2 files)

Create a section in the HTML report to output build details. v1.0
8.26 KB, patch
jgriffin
: review+
Details | Diff | Splinter Review
Example results HTML for device.
99.57 KB, text/html
Bebe
: feedback+
stephend
: feedback+
Details
Originally filed as: https://github.com/mozilla/gaia-ui-tests/issues/1340 We should output the build details in the HTML report, including build ID and gecko/gaia revisions and versions.
Would you please assign this bug to me (and provide some suggestions)? Thanks!
Hey Jonathan! I've assigned the bug to you. Basically we need to take the gecko, gaia, and build revisions out of the sources.xml file that accompanies our device builds. We already do this for a tool named b2gperf, and you can see that code here: https://github.com/mozilla/b2gperf/blob/0a238fa1021d43c74d1218a7c408e7a4c4dc9e75/b2gperf/b2gperf.py#L54 We first attempt to pull the application.zip file and read the resources/gaia_commit.txt file for the gaia commit, but this will only work if Gaia has been flashed on top of our nightly builds. I would expect a --sources command line argument to be added to our test runner here: https://github.com/mozilla-b2g/gaia/blob/597b217834474b64b2c2bf786ee974823b3d01f4/tests/python/gaia-ui-tests/gaiatest/runtests.py#L79 and then create a summary section in the HTML report that outputs these values here: https://github.com/mozilla-b2g/gaia/blob/597b217834474b64b2c2bf786ee974823b3d01f4/tests/python/gaia-ui-tests/gaiatest/runtests.py#L189 Take a look through the above links. I'm sure you'll have some more questions so please feel free to ask here, or find me (davehunt) in #automation on irc.mozilla.org
Assignee: nobody → lj.ditrapani
Status: NEW → ASSIGNED
Dave, Thanks for the pointers. I've been slowing digesting all this information. I just want to do a sanity check. Here's what I understand along with a few basic questions. I've been reading the gaia-ui-tests README at https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/README.md Which says I need Marionett enable Firefox; which I can do with: # Enable marionette firefox -marionette Will this marionette-enabled firefox be acting like a server to the B2G device? I'll need B2G with Gaia running on it. Do I need to set up both Firefox OS on an emulator and B2G Desktop to fix this bug, or is B2G Desktop enough? (I'm assuming running Gaia directly on Firefox is insufficient to run these tests, correct?) By the way, I updated the link on https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner It used to link to: https://github.com/mozilla/gaia-ui-tests It now links to: https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests Is it ok for me to do that? - Jonathan
(In reply to L. Jonathan Di Trapani [:ditrapani] from comment #3) > Do I need to set up both Firefox OS on an emulator and B2G Desktop to fix > this bug, or is B2G Desktop enough? (I'm assuming running Gaia directly on > Firefox is insufficient to run these tests, correct?) Unfortunately you're going to need to do some of this work blind. You can run the tests using the Firefox OS Simulator (b2g desktop) but you won't be able to obtain the gecko and gaia revisions using that. These values are pulled from files on the device or provided alongside the device builds. I can provide you with a cut down version of one of these files, or you can infer the structure of them from the code in b2gperf that already extracts the revisions from them. > By the way, I updated the link on > https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner > It used to link to: > https://github.com/mozilla/gaia-ui-tests > It now links to: > https://github.com/mozilla-b2g/gaia/tree/master/tests/python/gaia-ui-tests > Is it ok for me to do that? Yes, thanks for doing that! :)
Dave Hunt (:davehunt), Is the information we want (build id, gecko, gaia version) in the sources.xml file or in the resources/gaia_commit.txt within the application.zip file? Is the reason for the --sources command line argument to provide the path to the sources.xml file? Inferring the structure of the resources/gaia_commit.txt within the application.zip: It's a bunch of "key: value" pairs separated by newlines. So I can just scan through each line until I find the keys we are interested in. In b2gperf, the "Gaia revision:" value is the only value you extract. Does that contain all the needed information or are there other, separate keys that contain the build id and gecko version? How do I find out what those keys are? "Build id:"? "Gecko version:"? I would expect the structure of the sources.xml file to be...xml.
(In reply to L. Jonathan Di Trapani [:ditrapani] from comment #5) > Dave Hunt (:davehunt), > > Is the information we want (build id, gecko, gaia version) in the > sources.xml file or in the resources/gaia_commit.txt within the > application.zip file? That just contains the Gaia commit, and is only present if we've updated the version of Gaia on the device. That means we should try to obtain that, but fall back to the sources.xml. The gaia_commit.txt file is in the format: 7f7743c283a0d7491290e7d1cfb36930363384f0 1380662990 The first line is the commit, the second is the build ID. We should probably grab both for the HTML report. > Is the reason for the --sources command line argument to provide the path to > the sources.xml file? Yep, this should be the path to sources.xml > Inferring the structure of the resources/gaia_commit.txt within the > application.zip: > It's a bunch of "key: value" pairs separated by newlines. So I can just > scan through each line until I find the keys we are interested in. In > b2gperf, the "Gaia revision:" value is the only value you extract. Does > that contain all the needed information or are there other, separate keys > that contain the build id and gecko version? How do I find out what those > keys are? "Build id:"? "Gecko version:"? It's simpler than that, see above. > I would expect the structure of the sources.xml file to be...xml. Yeah, no surprises there. Here's a snippet: <?xml version="1.0" ?><manifest> ... <project name="gecko.git" path="gecko" remote="mozillaorg" revision="4d7401a1778508ff6942e6cf4dded618cefc63db"/> ... <project name="platform_build" path="build" remote="b2g" revision="59605a7c026ff06cc1613af3938579b1dddc6cfe"> ... <project name="gaia.git" path="gaia" remote="mozillaorg" revision="e7c011371aa6af696033d4b867fb9152a6985efa"/> ... </manifest>
Depends on: 928452
I've filed bug 928452 which should make this task much easier by moving the revision seeking to a separate package.
Hello Dave (:davehunt), Updated documentation: ======================= I updated some more documentation to point to the correct gaia project instead of the older gaia-ui-tests project on github. - gaia-ui-tests README.md https://github.com/lj-ditrapani/gaia/blob/master/tests/python/gaia-ui-tests/README.md - Gaia UI Tests Guidelines https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/ Automated_testing/gaia-ui-tests - Gaia UI Tests Environment Setup https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/ Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Installation - Gaia UI Tests Run Tests https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Platform/ Automated_testing/gaia-ui-tests/Gaia_UI_Tests_Run_Tests What I have done so far ======================= Following the above documentation, I tried to get gaiatest to run. I installed firefox standard build dependecies follawing https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/Linux_Prerequisites (I probably didn't need to do that) I setup a virtual environment. Cloned gaia and ran setup.py in the gaia-ui-tests directory which installed a bunch of python eggs. Downloaded and unpacked the latest nightly build of B2G Desktop (18.0 i686 linux from 09 Oct). I ran it with ./b2g and it seems to work fine. I updated the gaia/profile/user.js by adding at the beginning of the file: user_pref('marionette.force-local', true); and started b2g again (it says marionette-server.js is loaded and it is listening on port 2828, which seems to be what we want it to do) Trying to use gcli ------------------ When I try to use gcli to send commands to B2G Desktop, I see the B2G Desktop respond to the command on the command line I launched the B2G process on, but the device does not respond and the gcli command crashes with a python "KeyError: 'id'" runtime exception. Trying to run gaiatest ------------------ When I try to run gaiagtest with: gaiatest --address=localhost:2828 --type=b2g-antenna-bluetooth-carrier-camera-sdcard-wifi-xfail gaiatest/tests/manifest.ini The gaiatest just prints the standard "ACKNOWLEDGE RISKS" banner and exits. The b2g process does not respond in anyway and I don't see any indication of any tests being executed. Questions =========== So I still can't even get gaiatest running. What should I try next? Am I doing something wrong? Should I try a full emulator instead or am I just wasting my time? I really want to make progress on this but I've already spent quite a lot of time trying to understand all these pieces and how they fit together and I don't want to keep spinning my wheels. I saw your previous update. Do you want me to create a separate project on github for this? Thanks for your help! I appreciate your time in helping me become of productive member of the community! - Jonathan
Hey Johnathan, I've provided a few comments below, but let's take the troubleshooting to IRC or the mailing list. You can find me in #automation on irc.mozilla.org or subscribe to the dev-automation mailing list at: https://lists.mozilla.org/listinfo/dev-automation > I updated some more documentation to point to the correct gaia project > instead of the older gaia-ui-tests project on github. Great, many thanks! > - gaia-ui-tests README.md > https://github.com/lj-ditrapani/gaia/blob/master/tests/python/gaia-ui-tests/ > README.md Could you submit any documentation updates as a pull request? Please also file a bug and request review for your changes (I can help you with this). > I installed firefox standard build dependecies follawing > https://developer.mozilla.org/en-US/docs/Developer_Guide/Build_Instructions/ > Linux_Prerequisites > > (I probably didn't need to do that) Nope, but it's good practice for when you're building your own version of the software! ;) > I updated the gaia/profile/user.js by adding at the beginning of the file: > > user_pref('marionette.force-local', true); That should be no longer needed, Marionette should work straight-off. If this was documented somewhere, can you also please update this? > When I try to use gcli to send commands to B2G Desktop, I see the B2G > Desktop respond to the command on the command line I launched the B2G > process on, but the device does not respond and the gcli command crashes > with a python "KeyError: 'id'" runtime exception. This could be related to a recent change in Marionette. Please try again but with Marionette client 0.6.0 and the very latest desktop build. You may have been unfortunate in that you tried this between these being updated. > The gaiatest just prints the standard "ACKNOWLEDGE RISKS" banner and exits. > The b2g process does not respond in anyway and I don't see any indication of > any tests being executed. Have you completed the steps to acknowledge the risks? See https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks for details. > What should I try next? See my previous comments. Your perseverance is appreciated, especially your documentation updates, which will help the next contributors. > I saw your previous update. Do you want me to create a separate project on > github for this? I'd like to first get some feedback from my team on bug 928452. If you're interested on working on this, we'd be happy to mentor you. The idea would be to put this code into a separate package that could be used on the command line or as an API so we don't keep writing the same code over and over. :)
I had 2 problems. 1. The link in the gaia-ui-tests readme and some other documentation pointed to an old version of B2G Desktop (18) instead of the most recent. Now I'm using the most recent version. I updated the link in the gaia-ui-tests README. 2. I didn't acknowledge the risks in the testvars.json file. This one was my fault as I didn't read all the way through the risks page. I updated the gaia-ui-tests file to be more clear on this. Now I can get the tests running and I've started to poke around in the runtests.py file. Progress! :) The updated version of the README.md is here. https://github.com/lj-ditrapani/gaia/blob/master/tests/python/gaia-ui-tests/README.md The fork is synch with the mozilla project as of this weekend. I would appreciate some direction in filing a bug and submitting a pull request to update the documentation. Do you have a link with instructions? Thanks for your help last week; it was great to finally get things running. Hopefully I'll have more time this week to finish this up. Yeah, bug 928452 sounds like a good next step for me once I finish this one. Thanks again! - Jonathan
(In reply to L. Jonathan Di Trapani [:ditrapani] from comment #10) > I had 2 problems. > > 1. The link in the gaia-ui-tests readme and some other documentation pointed > to an old version of B2G Desktop (18) instead of the most recent. Now I'm > using the most recent version. I updated the link in the gaia-ui-tests > README. Sorry about that! Thanks for fixing the link. It also turns out that the warning in the README about the desktop build on Windows is no longer valid, so we can remove that too. > 2. I didn't acknowledge the risks in the testvars.json file. This one was > my fault as I didn't read all the way through the risks page. I updated the > gaia-ui-tests file to be more clear on this. This is somewhat intentional. We could provide an example that can be copy/pasted but it's possible that the user wouldn't really understand the risks, so we make it necessary for them to read through the entire page. > Now I can get the tests running and I've started to poke around in the > runtests.py file. Progress! :) Great! :) > The updated version of the README.md is here. > https://github.com/lj-ditrapani/gaia/blob/master/tests/python/gaia-ui-tests/ > README.md > The fork is synch with the mozilla project as of this weekend. > > I would appreciate some direction in filing a bug and submitting a pull > request to update the documentation. Do you have a link with instructions? You can create a bug by clicking this link: https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20OS&component=Gaia%3A%3AUI%20Tests make sure you have a suitable summary and description, and indicate that you'll be working on it. In this case the summary might be 'Improve documentation for users getting starting with desktop builds' or similar. Once you've done that, create a branch for the bug in your gaia repository and include your changes. Push this to your fork and submit a pull request. You can then add a link to this pull request as an attachment in the bug you filed, and you can request review from me. To request a review, click the details link for the attachment in bugzilla. On the left side are dropdowns with various labels. Find the "review" entry and change the flag to "?", and enter the email address of the person who will be reviewing the patch. Remember to submit changes! > Thanks for your help last week; it was great to finally get things running. > Hopefully I'll have more time this week to finish this up. Yeah, bug 928452 > sounds like a good next step for me once I finish this one. Thanks again! You're very welcome. Let me know if you have any other questions.
(In reply to Dave Hunt (:davehunt) from comment #11) Dave, I created bug 935319 https://bugzilla.mozilla.org/show_bug.cgi?id=935319 to update the documentation along with the pull request. I couldn't figure out how to assign the bug to myself. Do I have permissions to do that? Please let me know what else you need to get the documentation updated. - Jonathan
(In reply to L. Jonathan Di Trapani [:ditrapani] from comment #12) > (In reply to Dave Hunt (:davehunt) from comment #11) > I created bug 935319 https://bugzilla.mozilla.org/show_bug.cgi?id=935319 to > update the documentation along with the pull request. I couldn't figure out > how to assign the bug to myself. Do I have permissions to do that? Please > let me know what else you need to get the documentation updated. Thanks, I've assigned it to you.
Jonathan any updates on this? Can we help you?
(In reply to Florin Strugariu [:Bebe] from comment #14) > Jonathan any updates on this? > Can we help you? This depends on bug 928452, which I've requested feedback from several people on. If we need to get this into the HTML report sooner then we can do so, and later refactor it to use the new package. I'm still happy to mentor either way.
(In reply to Florin Strugariu [:Bebe] from comment #14) > Jonathan any updates on this? > Can we help you? Sorry, for the late reply. I was really busy during the holiday season with work and family and haven't had time for this. Back in November, I had finally gotten the gaia-ui-tests running, but was not sure how to reasonably develop the code. If I make changes to the code, how can I test that the changes work properly? The gaia-ui-tests take quite some time to execute. So running through all the tests every time I change a line of code is not reasonable. If I kill the test process after it starts (instead of waiting for all tests to complete), the localhost+port socket seems to still be bound, so I can't immediately re-run the test. I have to wait a certain amount of time until the port is available again. I'm not sure why that is. I imagine the solution would be to configure the gaia-ui-tests to run only one or two quick tests, ignoring the rest, so it executes faster. But I don't know how to set that up. I just don't know how to get a reasonable development flow going with this project.
Hi Jonathan, I would hold off on this bug for now as it's dependent on bug 928452, which is nearing completion. I'd be happy to mentor you through this once it's resolved, but there's not much point working on it before then. The gaiatest harness has a number of unit tests, so you can run just those to check if you've introduced a regression, however a full run is always advisable before submitting any patches. Run just the unit tests using --type=b2g /path/to/tests/unit/manifest.ini. You can also point to just an individual test file in order to just run a single test.
Dave any updates on this? I know we will love to see this in the reports
(In reply to Florin Strugariu [:Bebe] from comment #18) > Dave any updates on this? > I know we will love to see this in the reports Bebe: I'm not sure what I can update other than what's already here. The dependent bug is resolved and mozversion is available. This bug is currently assigned to Jonathan so we should ask if he still has time to work on it, or has made any progress. Jonathan: Let us know if you'd still like to work on this. If not, we'll release it so someone else can take it. Thanks.
Flags: needinfo?(lj.ditrapani)
(In reply to Dave Hunt (:davehunt) from comment #19) > Jonathan: Let us know if you'd still like to work on this. If not, we'll > release it so someone else can take it. Thanks. Yes, I'd still like to work on this. I'll start tomorrow. I only have time in the evening (Eastern time zone) and weekends. It will probably take me a couple of days to get spun back up on this. I'm sure I'll have questions/need mentoring. Thanks.
Flags: needinfo?(lj.ditrapani)
Great, thanks Jonathan. Just let us know if you have any questions. :)
Hey Jonathan, any progress on this patch? Is there anything we can help with? If you're having trouble finding the time then it might be worth freeing this one up for someone else to work on, and we can perhaps find something else for you to take. I know several people who are keen to see this particular bug resolved.
Flags: needinfo?(lj.ditrapani)
Assignee: lj.ditrapani → nobody
Flags: needinfo?(lj.ditrapani)
Sorry, I thought I would have time to get back into this, but my work situation has changed drastically and I'm a lot busier than expected. I went ahead and assigned it to nobody so someone else can take it.
Thanks Jonathan, let us know if things change again and you have free time to contribute. :)
I worked on this today, will upload a patch shortly.
Assignee: nobody → dave.hunt
This adds mozversion as a dependency and outputs the version, build id, and revision for gecko, the date and revision for gaia, and the device id and firmware details (if available). It also adds a --sources command line argument for optionally providing a patch to the sources.xml file. I'll attach a sample HTML report as an example of the output.
Attachment #8399582 - Flags: review?(jgriffin)
Attachment #8399582 - Flags: review?(jgriffin) → review+
Comment on attachment 8399583 [details] Example results HTML for device. I don't recall who raised the original bug, so asking for feedback from a few people I know will use these changes. Feel free to comment on this example HTML report, or the patch, or both! :)
Attachment #8399583 - Flags: feedback?(zcampbell)
Attachment #8399583 - Flags: feedback?(stephen.donner)
Attachment #8399583 - Flags: feedback?(florin.strugariu)
Attachment #8399583 - Flags: feedback?(florin.strugariu) → feedback+
Comment on attachment 8399583 [details] Example results HTML for device. Looks great; just what we're used to :-)
Attachment #8399583 - Flags: feedback?(stephen.donner) → feedback+
Submitted to try, results look good: https://tbpl.mozilla.org/?tree=Try&rev=4bae5a6c40c9
Keywords: checkin-needed
Whiteboard: [mentor=davehunt][lang=py]
Attachment #8399583 - Flags: feedback?(zcampbell)
https://hg.mozilla.org/mozilla-central/rev/a4fbca208278
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → 1.5 S1 (9may)
Not seeing this in http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Master/job/b2g.hamachi.mozilla-central.ui/671/HTML_Report/? and friends -- what's left (on any side[s]) to make this happen? Thanks! (I did notice we don't pull in mozversion (as a package) on http://selenium.qa.mtv2.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui/671/consoleFull -- should we be doing so?)
(In reply to Stephen Donner [:stephend] from comment #33) > Not seeing this in > http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Master/job/b2g.hamachi. > mozilla-central.ui/671/HTML_Report/? and friends -- what's left (on any > side[s]) to make this happen? Thanks! > > (I did notice we don't pull in mozversion (as a package) on > http://selenium.qa.mtv2.mozilla.com:8080/job/b2g.hamachi.mozilla-central.ui/ > 671/consoleFull -- should we be doing so?) This is a Marionette client fix, so it will be dependent on a new release of the PyPI package, and bumping gaiatest to depend on the new version. I didn't see this as high enough priority to prompt a bump/release, but please feel free to request a release via bugzilla.
(In reply to Dave Hunt (:davehunt) from comment #34) > This is a Marionette client fix, so it will be dependent on a new release of > the PyPI package, and bumping gaiatest to depend on the new version. I > didn't see this as high enough priority to prompt a bump/release, but please > feel free to request a release via bugzilla. Thx, Dave; not sure if there are risky changes yet-to-be-pushed, but I've filed bug 998365.
(In reply to Stephen Donner [:stephend] from comment #35) > (In reply to Dave Hunt (:davehunt) from comment #34) > > > This is a Marionette client fix, so it will be dependent on a new release of > > the PyPI package, and bumping gaiatest to depend on the new version. I > > didn't see this as high enough priority to prompt a bump/release, but please > > feel free to request a release via bugzilla. > > Thx, Dave; not sure if there are risky changes yet-to-be-pushed, but I've > filed bug 998365. I don't think a version of the Marionette client has been released since this change landed, so we first need to bump/release Marionette. Then we just need to bump the marionette_client dependency in gaiatest. We shouldn't need to bump/release gaiatest at all as our tests use gaiatest from master.
I've raised bug 1001461 to bump and release Marionette client. We will also need to bump the dependency for gaiatest once this is available on PyPI.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: