817269 - Update OrangeFactor/Logparser with new OS/platform/testsuite/repo names

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

jgriffin: Also, I noticed that at the moment, the TBPL submission to OrangeFactor uses TBPL's UI-specific platform names (which have changed recently, eg macosx64 became osx-10-6, android names changed etc), rather than the buildbot (or some other) name, which are why pages like this:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=761987&startday=2012-11-24&endday=2012-12-01&tree=trunk
...don't display the platform properly.

To fix, we just update:
https://hg.mozilla.org/automation/orangefactor/file/40507604ec6e/html/scripts/woo.utils.js#l7

...however, it doesn't seem right that the OF submission depends on what the TBPL UI is calling things surely? Or was the idea that if buildbot buildernames changed, only TBPL had to be updated (as long as the TBPL UI names remained the same) and no OF update required?

I guess I don't mind either way too much (after all, whatever we choose, we'll have a mix of platform names in OF, since there is no easy way to correct all the old rows, given how many changes have been made to the TBPL UI name over the months/years) - I just wanted to make sure this was intentional.

Comment 1

[Ed Morley [:emorley]]

Also need to add some testsuite names, eg robocop (a la bug 748430).

Comment 3

[Jonathan Griffin (:jgriffin)]

The problem with OF is that it has to correlate data coming from two different sources: TBPL, and buildbot logs via pulse, and these use different ways of identifying platform/os/testname.  At least when OF was initially written, TBPL did not have access to the same builder name strings as exist in the raw buildbot logs; I'm not sure whether or not that's still the case.  Thus, the need for OF to recognize these sets of strings.

For example, the raw buildbot log for mochitest-4 on linux64 has a builder string of mozilla-central_fedora64_test-mochitest-4.  TBPL uses a different string, Rev3 Fedora 12x64 mozilla-central opt test mochitest-4.  This latter string does not appear in the raw buildbot logs that OF parses, only the prettified logs generated by TBPL, and AFAIK, TBPL doesn't know about the former.

In turn, TBPL regex's those long builder strings into platform names that OF has been taught to recognize.

This kind of thing has long been an aggravation, but solving it correctly would likely require changes to buildbot and a lot of downstream code.

If you see something I'm missing, please let me know!

Comment 4

[Ed Morley [:emorley]]

Grepping the last few days of buildbot json, gives the following test names:
{
chrome
chromez
chrome_mac
crashtest
crashtest-2
crashtest-3
crashtest-ipc
dirty
dirtypaint
dromaeo
dromaeojs
gaia-ui-test
jetpack
jsreftest
jsreftest-1
jsreftest-2
jsreftest-3
marionette
marionette-webapi
mochitest-1
mochitest-2
mochitest-3
mochitest-4
mochitest-5
mochitest-6
mochitest-7
mochitest-8
mochitest-browser-chrome
mochitest-other
nochrome
other
plain-reftest-1
plain-reftest-2
plain-reftest-3
plain-reftest-4
reftest
reftest-1
reftest-10
reftest-2
reftest-3
reftest-4
reftest-5
reftest-6
reftest-7
reftest-8
reftest-9
reftest-ipc
reftest-no-accel
remote-tp4m_nochrome
remote-trobocheck
remote-trobocheck2
remote-trobopan
remote-troboprovider
remote-ts
remote-tsvg
robocop
svg
svgr
tp
tpn
xpcshell
xperf
}

Many of these are missing from logparser's config.py

Comment 5

[Ed Morley [:emorley]]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> If you see something I'm missing, please let me know!

Ah I see, thank you for the explanation.

Comment 6

[Ed Morley [:emorley]]

Attachment: Logparser part 1: Tidy test_list

Will make the next diff clearer.

Comment 7

[Ed Morley [:emorley]]

Attachment: Logparser part 2: Bring test_list up to date

Using the list in comment 4.

Comment 8

[Ed Morley [:emorley]]

(More patches for logparser and OF to come)

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 693495 [details] [diff] [review]
Logparser part 2: Bring test_list up to date

Review of attachment 693495 [details] [diff] [review]:
-----------------------------------------------------------------

Note that the logparser won't handle Marionette logs correctly.  It won't die, but it won't identify any errors either.  I'll file a separate bug for this.

Comment 10

[Ed Morley [:emorley]]

Many of the entries in logparser's config.py are completely out of date, however it's hard to update them, since unlike the OF configs, they need to match buildbot's platform/test names and not TBPL's. I'm going to have a stab here if only to close this long open bug out (and primarily am interested in updating OF's config itself), but it's not really affecting OF's functionality - and this will all be EOL anyway, once we switch OF over to using treeherder as a datasource.

Comment 11

[Ed Morley [:emorley]]

Attachment: Logparser 1: Sort platform and test name lists alphabetically

This patch had review previously.

Comment 12

[Ed Morley [:emorley]]

Attachment: Logparser 2: Sync trees_to_watch with those in orangefactor.conf

Syncing with the list of trees from:
https://hg.mozilla.org/automation/orangefactor/file/default/server/orangefactor.conf#l6

Comment 13

[Ed Morley [:emorley]]

Attachment: Logparser 3: Sync platform_list with those from buildbot

Syncing with the list of platforms from recent jobs found by a grep of the buildjson output:
https://secure.pub.build.mozilla.org/builddata/buildjson/

Comment 14

[Ed Morley [:emorley]]

Attachment: Logparser 4: Sync test_list with those from buildbot

Same as above, but for tests.

Comment 15

[Ed Morley [:emorley]]

Attachment: OF 1: Improve readability of platform_map

This is a no-op. With this, the diffs will be less horrendous for later changes.

Comment 16

[Ed Morley [:emorley]]

Attachment: OF 2: Update platform list

Updated with the latest platform names from TBPL.

Comment 17

[Ed Morley [:emorley]]

Attachment: OF 3: Sort test suites alphabetically

This will make it easier to find the desired test suite when scrolling through the UI select, as well as make this list easier to keep up to date.

Comment 18

[Ed Morley [:emorley]]

Attachment: OF 4: Bring test suites list up to date

Helpfully unlike the platform list, the test names used aren't those in TBPL, but from buildbot - so this list is based on the same updates in the logparser part 4 patch (and so derived from a grep of recent buildjson).

Comment 19

[Ed Morley [:emorley]]

Attachment: OF 5: Allow filtering of asan jobs

TBPL views ASAN jobs as an additional build type. OF already displays asan jobs as such when viewing the single-bug intermittent failure stats page - this just adds the ability to filter by asan on the main OrangeFactor overview.

Comment 20

[Ed Morley [:emorley]]

Attachment: OF 6: Remove obsolete platform exclusion entries

Win64 jobs no longer have that platform name and are also not being run on
trunk - so won't skew the stats.

Comment 21

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8460204 [details] [diff] [review]
Logparser 2: Sync trees_to_watch with those in orangefactor.conf

Review of attachment 8460204 [details] [diff] [review]:
-----------------------------------------------------------------

::: logparser/config.py
@@ +73,5 @@
>                    'mozilla-aurora',
>                    'mozilla-beta',
> +                  'mozilla-esr31',
> +                  'mozilla-b2g30_v1_4',
> +                  'mozilla-b2g28_v1_3']

I assume we should include the new mozilla-b2g32_v2_0 tree as well.

Comment 22

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8460205 [details] [diff] [review]
Logparser 3: Sync platform_list with those from buildbot

Review of attachment 8460205 [details] [diff] [review]:
-----------------------------------------------------------------

::: logparser/config.py
@@ +44,5 @@
> +                 'macosx64_gecko',
> +                 'macosx64_gecko-debug',
> +                 'macosx64_gecko_localizer',
> +                 'mavericks',
> +                 'mock-hw',

Is this a real thing we should parse logs for?

Comment 23

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8460217 [details] [diff] [review]
OF 5: Allow filtering of asan jobs

Review of attachment 8460217 [details] [diff] [review]:
-----------------------------------------------------------------

...but I don't think we're actually populating OF with entries that use a buildtype of 'asan'.  See http://hg.mozilla.org/automation/logparser/file/4b0fd863cdb7/logparser/tinderboxparser.py#l164.

Comment 24

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8460221 [details] [diff] [review]
OF 6: Remove obsolete platform exclusion entries

Review of attachment 8460221 [details] [diff] [review]:
-----------------------------------------------------------------

This should be fine, now that we automatically exclude hidden builders.

Comment 25

[Ed Morley [:emorley]]

(In reply to Jonathan Griffin (:jgriffin) from comment #21)
> I assume we should include the new mozilla-b2g32_v2_0 tree as well.

Yeah will need to add to OF as well; will attach another patch.

(In reply to Jonathan Griffin (:jgriffin) from comment #22)
> Comment on attachment 8460205 [details] [diff] [review]
> Logparser 3: Sync platform_list with those from buildbot
> 
> ::: logparser/config.py
> @@ +44,5 @@
> > +                 'macosx64_gecko',
> > +                 'macosx64_gecko-debug',
> > +                 'macosx64_gecko_localizer',
> > +                 'mavericks',
> > +                 'mock-hw',
> 
> Is this a real thing we should parse logs for?

These were extracted from the platform field in the buildjson output.

macosx64_gecko* comes from jobs who have buildername eg "b2g_mozilla-b2g32_v2_0_macosx64_gecko build" etc
mavericks from eg "Rev5 MacOSX Mavericks 10.9 cedar talos svgr"
mock-hw from eg: "fuzzer-mock-hw"

Guessing we should leave the fuzzer one out then.
Updated locally.

(In reply to Jonathan Griffin (:jgriffin) from comment #23)
> Comment on attachment 8460217 [details] [diff] [review]
> OF 5: Allow filtering of asan jobs
>
> ...but I don't think we're actually populating OF with entries that use a
> buildtype of 'asan'.  See
> http://hg.mozilla.org/automation/logparser/file/4b0fd863cdb7/logparser/
> tinderboxparser.py#l164.

Yeah that's what I thought - however it seems we use the data submitted by TBPL instead when it comes to filtering build type (yey for consistency), and so asan already appears in the production instance failure details:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1039862
(see the "Build Type" column)

...and with this patch filtering by asan worked when I tested locally.

Comment 26

[Ed Morley [:emorley]]

Attachment: OF 7: Add support for mozilla-b2g32_v2_0

Comment 27

[Ed Morley [:emorley]]

Attachment: Logparser 5: Add support for mozilla-b2g32_v2_0

Comment 28

[Ed Morley [:emorley]]

https://hg.mozilla.org/automation/logparser/rev/425cda2c2644
https://hg.mozilla.org/automation/logparser/rev/7125c12108fd
https://hg.mozilla.org/automation/logparser/rev/c5cb4428fb6c
https://hg.mozilla.org/automation/logparser/rev/07c09f48b3ef
https://hg.mozilla.org/automation/logparser/rev/7fa7bc6cf70d

https://hg.mozilla.org/automation/orangefactor/rev/00aeb256d46d
https://hg.mozilla.org/automation/orangefactor/rev/f49a60b29169
https://hg.mozilla.org/automation/orangefactor/rev/1c44b736cae1
https://hg.mozilla.org/automation/orangefactor/rev/ef3b6697da91
https://hg.mozilla.org/automation/orangefactor/rev/3c5f5672d577
https://hg.mozilla.org/automation/orangefactor/rev/be699209020c
https://hg.mozilla.org/automation/orangefactor/rev/b4f49b1aa780

OF update/deploy:

[[e-mail]3 ~]$ cd ~/apps/orangefactor/src/orangefactor/ && hg pull -u -v
pulling from http://hg.mozilla.org/automation/orangefactor/
searching for changes
adding changesets
adding manifests
adding file changes
added 10 changesets with 12 changes to 5 files
resolving manifests
getting README.txt
getting html/scripts/woo.utils.js
getting requirements/prod.txt
getting server/handlers.py
getting server/orangefactor.conf
5 files updated, 0 files merged, 0 files removed, 0 files unresolved

[[e-mail]3 orangefactor]$ cat server/orangefactor.conf | egrep '^trees'
trees = mozilla-central, mozilla-inbound, b2g-inbound, fx-team, mozilla-aurora, mozilla-beta, mozilla-esr31, mozilla-b2g32_v2_0, mozilla-b2g30_v1_4, mozilla-b2g28_v1_3

[[e-mail]3 ~]# /etc/init.d/orangefactor stop; /etc/init.d/orangefactor start
stopping orangefactor                                      [  OK  ]
starting orangefactorspawn-fcgi: child spawned successfully: PID: 26855
                                                           [  OK  ]

[[e-mail]3 ~]# cd /home/webtools/apps/orangefactor/src/orangefactor/ && ./deploy.sh

[[e-mail]3 orangefactor]# egrep 'asan|b2g32' /usr/share/nginx/html/orangefactor/scripts/woo.utils.js
      'mozilla-b2g32_v2_0': '',
      'asan': '',

The logparser update/deploy is blocked on bug 1043388.

Comment 29

[Ed Morley [:emorley]]

Logparser update/deploy:

[[e-mail]1 ~]$ cd ~/apps/logparser/src/logparser
[[e-mail]1 logparser]$ hg pull -u -v
pulling from http://hg.mozilla.org/automation/logparser
searching for changes
all local heads known remotely
adding changesets
adding manifests
adding file changes
added 6 changesets with 6 changes to 2 files
resolving manifests
getting .hgignore
getting logparser/config.py
2 files updated, 0 files merged, 0 files removed, 0 files unresolved

[[e-mail]1 ~]# service logparser stop; service logparser start
stopping logparser                                         [  OK  ]
starting logparser                                         [  OK  ]