Open Bug 1132057 Opened 5 years ago Updated 9 months ago

[mozlog] HTML log should show the passed/failed tests on the top of the page as default

Categories

(Testing :: Mozbase, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: Bebe, Unassigned, Mentored)

Details

(Whiteboard: [lang=py][good first bug])

Currently the MozLog  HTML report showing the tests results in the order they where processed by the test runner.
Having a test suite that has a lot of skipped tests these will processed at the beginning of test suite so the test results.


Can we update the mozlog to show the "important data" first and the other information at the bottom of the HTML file.
I think the best solution here might be to hide less important results by default. In most cases the important results would be unexpected errors and failures. We could introduce a filter section at the top of the report that by default only shows these, and if there's no results to display show an appropriate message - ideally with a link to show all results.

Here's an example of how Mozmill dashboards filter the results by status: http://mozmill-daily.blargon7.com/#/functional/report/beeca5bc62c04337e8365d933bb29dae

We talked about sorting by reverse execution time, and test duration, but I think these aren't very intuitive for the user.
Mentor: dave.hunt
OS: Linux → All
Hardware: x86_64 → All
Summary: [mozlog] MozLog should show the passed/ failed tests on the top of the page as default → [mozlog] HTML log should show the passed/failed tests on the top of the page as default
Essentially the same code is used for my pytest-html plugin, which has been enhanced to sort result by default. See https://github.com/davehunt/pytest-html/pull/6 for the changes. These should be relatively easy to port to mozlog.
Whiteboard: [lang=py][good first bug]
I'd like to give this one a go.
Thanks Jacob, I've assigned it to you - let me know if you have any questions.
Assignee: nobody → jacob.harrowmortelliti
Assignee: jacob.harrowmortelliti → nobody
hello! can i take this bug?
Sure, I've assigned it to you.
Assignee: nobody → bullionareboy
(In reply to Dave Hunt (:davehunt) from comment #6)
> Sure, I've assigned it to you.

Thank You! Please could you point me in the right direction where i need to start off with this, what all i need to know and the basic prerequisites. 
Also are you present on IRC if i need to contact again?
I've not looked at this code for a while myself, but let's see...

The source code for mozlog is here[0] you can clone this locally using Mercurial[1] by running:

$ hg clone https://hg.mozilla.org/mozilla-central

The code for the HTML formatter for mozlog can be found here[2]. Very similar code to mozlog is used for the pytest-html plugin, and a default sort order for result was implemented here[3]. I think something similar could be fairly easily applied to mozlog so failures are elevated to the top of the report.

I'm davehunt on IRC, and can be found in #ateam. I'm based in the UK so you're most likely to see me during the day in my timezone (GMT).

[0] http://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/mozlog/
[1] https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/index.html
[2] http://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/mozlog/mozlog/formatters/html
[3] https://github.com/davehunt/pytest-html/pull/6/files
Hey are there any updates on this? I'm looking for a good starting project. I see since the last time this bug has been discussed the pytest-html link (for an example of a similar implementation) is dead. This seems to be place to find it now: https://github.com/pytest-dev/pytest-html/pull/6/files
Hey mattschmo! The pytest-html project moved within the pytest-dev organisation in GitHub as you pointed out. There's also been further changes to the plugin that write the results to the HTML in the default sort order so that it's not necessary for the initial page load to perform a sort. Other enhancements that might be worth bringing across include a result outcome filter and the option to hide details from results. Feel free to work on the default sort order, and maybe you'd also be interested in bringing across some of these other enhancements?
Assignee: bullionareboy → nobody
Cool sounds good, Dave. I'll take a deeper look at this when I get home from work tonight, and post any questions that may come up here.
Hi Dave! I want to contribute to mozilla and I have downloaded the source code. I want a bug to start with and I was thinking about this one. Also I am only a beginner in python though I have done some competitive coding in c++. How do I begin?
Hey p23keshav! Thanks for your interest in this bug, however mattschmo has already expressed interest in it a couple of weeks ago. Let's see if they're okay for someone else to take it before you spend any time on it.
Flags: needinfo?(mattschmo)
Okay! Thank You
Hey guys! Yeah so I cloned the repo but haven't made much progress as I've encountered performance issue on my personal laptop. I'm using zsh on a mac, and anytime I try to access the repo via the command line, my CPU spikes to 100 processing the mercurial history.

I haven't found a fix for that yet, and unfortunately it makes it very hard to work on the code base when simply changing a directory takes down my computer for 45-60 seconds.

p23keshav feel free to take a stab at it!
Hi Dave!! I would like to work on it. I have the code but as I am a beginner in python I do not understand the html.py very well.
Assignee: nobody → p23keshav
Hi Dave! Thanks
Do I have to work with only html.py or with main.js also?
First you'll want to install virtualenv and create an environment. You can do this using pip install virtualenv or I prefer to use virtualenvwrapper:

$ pip install virtualenvwrapper
$ mkvirtualenv bug1132057

Then from this virtual environment run setup_development.py found in testing/mozbase to install the various mozbase packages (including mozlog). Note that mozbase currently only support Python 2.x, and not 3.x.

$ cd testing/mozbase
$ python setup_development.py

You can then run the existing tests for mozlog using:

$ python test.py mozlog/tests/manifest.ini

It might then make sense to write a failing test for the new functionality. This would probably make sense in test_structured.py under TestHTMLFormatter. Here's an example that would log each of the potential test outcomes:

> def test_result_order(self):
>     self.logger.suite_start([])
>     for outcome in ['PASS', 'FAIL', 'OK', 'ERROR', 'TIMEOUT', 'CRASH', 'ASSERT', 'SKIP']:
>         test = '{}_test'.format(outcome.lower())
>         self.logger.test_start(test)
>         self.logger.test_end(test, outcome)
>     self.logger.suite_end()

You'd need to write your own assert that parses the HTML in self.loglines and checks that the results are in the correct order. Once you have this test failing as expected, I would suggest taking a look at how pytest-html orders the results by outcome and seeing if you can apply a similar approach to mozlog. This change was made to pytest-html in the following commit: https://github.com/pytest-dev/pytest-html/commit/a1956abb4db1b2603203261761eb22a048a1687a

See how you go, and feel free to ask questions here. It's been a while since I worked on mozbase myself, but hopefully I'll be able to help.
Flags: needinfo?(mattschmo)
Hi Dave!
I installed all requirements. unfortunately i don't found test.py in testing/mozbase directory. 

but i found in docs examples code to get HTML output of logs

http://mozbase.readthedocs.io/en/latest/_static/structured_example.py

python ../../config/pythonpath.py -I ../../config/ structured_example.py --log-html=-

output file contents one pass and one fail test. If I underst this file must be post-processed in correct order. I want to start from this point.

Possible i made something wrong, plz correct me.
Flags: needinfo?(dave.hunt)
(In reply to kpstsp from comment #19)
> I installed all requirements. unfortunately i don't found test.py in
> testing/mozbase directory. 

It's been a while since I ran the mozbase tests myself.. ahal: could you help kpstsp with the correct way to run the existing tests for mozlog?

> output file contents one pass and one fail test. If I underst this file must
> be post-processed in correct order. I want to start from this point.

Rather than post-processing the file, I would introduce a sort order when adding results to the `result_rows` list in `make_result_html`. A similar approach was done in https://github.com/pytest-dev/pytest-html/commit/a1956abb4db1b2603203261761eb22a048a1687a, where a TestResult object was created with a __lt__ method that would allow us to place it into a list of results in the desired order.
Flags: needinfo?(dave.hunt)
Flags: needinfo?(ahalberstadt)
Try ./mach python-test testing/mozbase/mozlog

I'm on PTO until late May, so won't be checking any follow up mail here.
Flags: needinfo?(ahalberstadt)
Thanks Andrew!
I runned test and being discouraged with this messages:

TEST-PASS | /media/user/data/project/mozilla-central/testing/mozbase/mozlog/tests/test_structured.py | TestBuffer.test_buffer_size      
 0:00.99 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:00.99                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:00.99 test_structured.py: error: unrecognized arguments: --log-unittest=-                                                                   
 0:01.00 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.00                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.00 test_structured.py: error: unrecognized arguments: --log-unittest-level=error                                                         
 0:01.00 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.00                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.00 test_structured.py: error: unrecognized arguments: --log-html=-                                                                       
 0:01.00 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.00                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.00 test_structured.py: error: unrecognized arguments: --log-html-level=error                                                             
 0:01.01 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.01                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.01 test_structured.py: error: unrecognized arguments: --log-tbpl=-                                                                       
 0:01.01 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.01                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.01 test_structured.py: error: unrecognized arguments: --log-tbpl-level=error                                                             
 0:01.02 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.02                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.02 test_structured.py: error: unrecognized arguments: --log-xunit=-                                                                      
 0:01.02 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.02                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.02 test_structured.py: error: unrecognized arguments: --log-xunit-level=error                                                            
 0:01.02 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.03                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.03 test_structured.py: error: unrecognized arguments: --log-errorsummary=-                                                               
 0:01.03 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.03                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.03 test_structured.py: error: unrecognized arguments: --log-errorsummary-level=error                                                     
 0:01.03 usage: test_structured.py [-h] [--log-raw LOG_RAW]                                                                                    
 0:01.03                           [--log-raw-level LOG_RAW_LEVEL]                                                                             
 0:01.04 test_structured.py: error: unrecognized arguments: --log-mach=-                                                                       
:                                                                                                                                              


What's the command  i need run for get logs in HTML format? 
Like what


python ../../config/pythonpath.py -I ../../config/ structured_example.py --log-html=-
This output is expected as it's testing that the formats can be restricted and any otherwise valid format will raise an exception. See https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozlog/tests/test_structured.py#843-857 for the test.
I'm a student from coventry university studying an opensource module, I would like to be assigned this as my first bug! Could you please point me in the right direction of how to start off?
hello! can i take this bug? I need it as I am studying open source module in Coventry uni
As there hasn't been any activity on this bug for 8 months I'm happy to re-assign it. Syed: As you commented first, I am inclined to assign this to you, however I see you have commented on 10 bugs recently asking to work on them. I'm concerned that you won't be able to work on so many bugs at the same time. Let me know if you're still interested in working on this, or if you'd like to let Prince take this one.
Assignee: p23keshav → nobody
Flags: needinfo?(alis105)
I am still interested
Flags: needinfo?(alis105)
Assignee: nobody → alis105
Hi Dave! Interested in working on this bug. Where do I start and how can I get a sense of what the present output for the tests looks like? I have read the comments but would like any other comments you have regarding this bug.
Flags: needinfo?(dave.hunt)
Are you still using the HTML reports anywhere? We've had a few contributors interested in working on this, but it's been a while since it was first opened. Do you think it's still a worthwhile enhancement?
Flags: needinfo?(dave.hunt) → needinfo?(hskupin)
I'm not sure which type of HTML report this is. For Marionette we have something like https://taskcluster-artifacts.net/TvMhCOaeSaKocnU7JxC2Bg/0/public/test_info/report.html. So the results are displayed at the top.
Flags: needinfo?(hskupin)

Is anyone working on this? Else, I can take this up.

Assignee: alis105 → nobody
You need to log in before you can comment on or make changes to this bug.