1040418 - Update log parser regex & bug suggestion blacklist to match TBPL

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Cameron Dawson [:camd]]

We have several areas where we have regular expressions for data and log parsing.  The Sheriff's should maintain these files going forward to ensure that we keep up to date with changes that are needed.


Here are the files that will need maintenance.  In most cases they somewhat mirror the files used in TBPL.  However, the structure *is* a bit different here and there.  Much of this was copied from TBPL directly, and converted to python.  So you'll even see similar comments and ordering.  I've done my best to keep the order the same in the various lists.  Hopefully this will make sense when you look into it.

Job data that comes from buildbot:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py

Log parsing for errors:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/log_parser/parsers.pyx#L250

Hopefully this will be self-explanatory enough.  But if not, please don't hesitate to contact me and we can work through it together.

Comment 2

[Cameron Dawson [:camd]]

In addition to the files mentioned above when updating regexes, etc: when we update the names of the platforms, it should be done directly in this file: https://github.com/mozilla/treeherder-ui/blob/master/webapp/app/js/values.js

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There were some changes in the TBPL symbols in bug 1013691 that are yet to be reflected in Treeherder.

Comment 6

[Ed Morley [:emorley]]

UI-side...

As of:
https://github.com/mozilla/treeherder-ui/blob/13088bae0aee302bf0bbb8cdb4635f351581c292/webapp/app/js/values.js

the file is up to date compared to:
https://hg.mozilla.org/webtools/tbpl/file/06af9dad7e20/js/Config.js#l345

Looking at service next.

Comment 7

[Ed Morley [:emorley]]

Attachment: Part 1: Correct Android & B2G architecture

We have three architectures for our Android builds: armv7, armv6 and x86.
The armv6 jobs were mostly already correct, but many of the armv7 ones were previously listed as x86.

Note that in addition to this, we currently don't differentiate between the arch on which a build was compiled, and the arch it is intended to be run on - however I've filed bug 1056928 for that, and this patch at least makes things more consistent until then.

Comment 8

[Ed Morley [:emorley]]

The tests passed when run locally.

Comment 9

[Ed Morley [:emorley]]

Another file with regex:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/log_parser/utils.py#L7

Comment 10

[Cameron Dawson [:camd]]

Comment on attachment 8476801 [details] [diff] [review]
Part 1: Correct Android & B2G architecture

Review of attachment 8476801 [details] [diff] [review]:
-----------------------------------------------------------------

Ed, this looks great.  Please feel free to merge this to master.

Comment 11

[Ed Morley [:emorley]]

Comment on attachment 8476801 [details] [diff] [review]
Part 1: Correct Android & B2G architecture

https://github.com/mozilla/treeherder-service/commit/0ae443a35bf677df829d7a83c79abd5693cc8551

Comment 12

[Ed Morley [:emorley]]

Attachment: Part 2: Fix flake8 warnings & comment typos in parsers.pyx

Fixes the following warnings:
treeherder/log_parser/parsers.pyx:217:31: E201 whitespace after '{'
treeherder/log_parser/parsers.pyx:221:1: E302 expected 2 blank lines, found 1
treeherder/log_parser/parsers.pyx:228:5: E303 too many blank lines (2)
treeherder/log_parser/parsers.pyx:249:24: E222 multiple spaces after operator
treeherder/log_parser/parsers.pyx:254:25: E125 continuation line with same indent as next logical line
treeherder/log_parser/parsers.pyx:264:51: E231 missing whitespace after ','
treeherder/log_parser/parsers.pyx:275:9: E126 continuation line over-indented for hanging indent
treeherder/log_parser/parsers.pyx:277:5: E121 continuation line under-indented for hanging indent
treeherder/log_parser/parsers.pyx:328:1: E302 expected 2 blank lines, found 1
treeherder/log_parser/parsers.pyx:350:9: E265 block comment should start with '# '
treeherder/log_parser/parsers.pyx:362:1: E302 expected 2 blank lines, found 1
treeherder/log_parser/parsers.pyx:375:32: F841 local variable 'e' is assigned to but never used

Comment 13

[Ed Morley [:emorley]]

Attachment: Part 3: Move log error terms containing regex out of the simple substring list

Terms in IN_SEARCH_TERMS are only used in a 'in' membership check, which only does a simple substring match. This moves terms that contain regex to RE_ERR_SEARCH which actually will use them in a re.search() (some were listed in both).

Comment 14

[Cameron Dawson [:camd]]

Comment on attachment 8480731 [details] [diff] [review]
Part 3: Move log error terms containing regex out of the simple substring list

Ed-- Is line 286 also regex?

Comment 15

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #14)
> Ed-- Is line 286 also regex?

No, but the escaping there is both unnecessary and causing that term to not work as expected (one of my local changes fixes this) :-)

Comment 16

[Ed Morley [:emorley]]

Attachment: Part 4: Remove unnecessary (and broken) escaping from substring search terms

Comment 17

[Ed Morley [:emorley]]

Attachment: Part 5: Sync log parser regex with recent TBPL additions

The treeherder equivalent of:
https://hg.mozilla.org/webtools/tbpl/rev/03d9ff82b54a
https://hg.mozilla.org/webtools/tbpl/rev/9941c86059cb
https://hg.mozilla.org/webtools/tbpl/rev/c6d684e3be13
https://hg.mozilla.org/webtools/tbpl/rev/611685ea448d
https://hg.mozilla.org/webtools/tbpl/rev/ae716547090e

TBPL now uses "/fatal error/i", but the two cases we are interested in are either all uppercase, or all lowercase - so I went with two entries in IN_SEARCH_TERMS rather than using (?i) in RE_ERR_SEARCH. Let me know if you'd prefer the latter instead.

Comment 19

[Ed Morley [:emorley]]

Attachment: Part 6: Clean up log parser regex expressions

This patch should be a no-op - it just cleans up the expressions used.

* Switch uses of "[0-9]" to "\d"
* Switch uses of "[ ]+" to " +"
* Remove unnecessary escaping

Comment 20

[Ed Morley [:emorley]]

All attached patches here are now landed (plus a typo followup for part 5, that isn't caught by the tests - I've filed bug 1060765 for improving test coverage):
https://github.com/mozilla/treeherder-service/commit/78b73b81ed05c7777040e5ad3cfad87ac2d3154c
https://github.com/mozilla/treeherder-service/commit/6164e51a2320bb395f5e7944d918d97d597a6993
https://github.com/mozilla/treeherder-service/commit/0a00acfd8bab1cc8cc3b110cdc936c6a6e9fc2b6
https://github.com/mozilla/treeherder-service/commit/b019c9a1b3faec711fdeeccc96f23db4241acb7a
https://github.com/mozilla/treeherder-service/commit/7d0549abeb017de2ca60cb0d89c44c78678ddc88
https://github.com/mozilla/treeherder-service/commit/34c7d78a7270335d7a2aab640e9ccbb02415e961

There's more to do with syncing job types, but I'll break that out to bug 1060763.

Comment 21

[Ed Morley [:emorley]]

(In reply to Cameron Dawson [:camd] from comment #0)
> Here are the files that will need maintenance. 
> 
> Job data that comes from buildbot:
> https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/
> buildbot.py
> 
> Log parsing for errors:
> https://github.com/mozilla/treeherder-service/blob/master/treeherder/
> log_parser/parsers.pyx#L250

(In reply to Cameron Dawson [:camd] from comment #2)
> In addition to the files mentioned above when updating regexes, etc: when we
> update the names of the platforms, it should be done directly in this file:
> https://github.com/mozilla/treeherder-ui/blob/master/webapp/app/js/values.js

(In reply to Ed Morley [:edmorley] from comment #9)
> Another file with regex:
> https://github.com/mozilla/treeherder-service/blob/master/treeherder/
> log_parser/utils.py#L7

Also new platforms need adding to PLATFORM_ORDER:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/webapp/api/resultset.py#L14

Comment 22

[Ed Morley [:emorley]]

When we deploy changes on dev/staging/production, are the parsers recompiled automatically, or are manual steps required? If the latter, will just remember to remind people to re-compile when pushing to prod for future requests etc :-)

Comment 23

[Jonathan Eads ( :jeads )]

The parser changes are recompiled automatically on production on every code push:
https://github.com/mozilla/treeherder-service/blob/master/deployment/update/update.py#L74

There are two bash scripts on dev/stage that carry out the same compile step automatically that we run to push new code changes. However these scripts do not automatically push code to the workers which run on separate AWS instances so that does need to be done manually. We will do this explicitly on dev/stage for these changes.

Comment 24

[Ed Morley [:emorley]]

Ah thank you :-)