630559 - Nightly reports either do not exist or they're empty

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[(no longer active)]

See the page in the URL field.  Most of the 4 beta nightlies do not have a report, and the 3.6 and 3.5 versions that do have report links, they're mostly empty.

Comment 1

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Some changes have been made to the builds script for 1.7.7, which may or may not help here. They're in the 1.7.7 branch and once I can get those changes merged then we can test on stage and see if there are remaining issues.

https://bugzilla.mozilla.org/show_bug.cgi?id=628067#c3

Comment 2

[Robert Helmer [:rhelmer]]

Looking into this. Here's something relatively new in the logs. I only see this for 4.0b11pre and then later 4.0b12pre:

"""
2011-02-14 07:05:02,935 WARNING - IndexError: list index out of range
2011-02-14 07:05:02,938 INFO - Did not find build entries in builds table for firefox 4.0b12pre linux-i686 20110214030347
http://hg.mozilla.org/mozilla-central/rev/26421a3b68f3
2011-02-14 07:05:02,942 CRITICAL - Caught Error: <class 'psycopg2.DataError'>
2011-02-14 07:05:02,942 CRITICAL - invalid input syntax for integer: "20110214030347
http://hg.mozilla.org/mozilla-central/rev/26421a3b68f3"
LINE 5:       (E'firefox', E'4.0b12pre', E'linux-i686', E'2011021403...
"""

We also seem to get this regularly:
"""
    if build_file[2]:
2011-02-14 07:05:01,734 WARNING - IndexError: list index out of range
2011-02-14 07:05:01,735 WARNING - Caught Error: <type 'exceptions.IndexError'>
2011-02-14 07:05:01,735 WARNING - list index out of range
2011-02-14 07:05:01,735 WARNING - trace back follows:
2011-02-14 07:05:01,736 WARNING - Traceback (most recent call last):
2011-02-14 07:05:01,736 WARNING - File "/data/socorro/application/socorro/cron/builds.py", line 140, in fetchAndRecordNightlyBuilds
    if build_file[3]:
2011-02-14 07:05:01,737 WARNING - IndexError: list index out of range
2011-02-14 07:05:01,740 INFO - Did not find build entries in builds table for firefox 3.5.18pre linux-i686 20110214030349
2011-02-14 07:05:01,747 INFO - Inserted the following build: firefox 3.5.18pre linux-i686 20110214030349 061b6c646c10   firefox-3.5.18pre.en-US.linux-i686.txt
2011-02-14 07:05:01,798 WARNING - Caught Error: <type 'exceptions.IndexError'>
2011-02-14 07:05:01,798 WARNING - list index out of range
2011-02-14 07:05:01,799 WARNING - trace back follows:
2011-02-14 07:05:01,799 WARNING - Traceback (most recent call last):
2011-02-14 07:05:01,800 WARNING - File "/data/socorro/application/socorro/cron/builds.py", line 135, in fetchAndRecordNightlyBuilds
"""

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #1)
> Some changes have been made to the builds script for 1.7.7, which may or may
> not help here. They're in the 1.7.7 branch and once I can get those changes
> merged then we can test on stage and see if there are remaining issues.
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=628067#c3

Stage has the latter stack from comment 2, but the former now throw this instead:

"""
2011-02-14 07:05:06,477 WARNING - IndexError: list index out of range
2011-02-14 07:05:06,493 INFO - Did not find build entries in builds table for firefox 4.0b12pre linux-i686 20110214030347
http://hg.mozilla.org/mozilla-central/rev/26421a3b68f3
2011-02-14 07:05:06,496 CRITICAL - Caught Error: psycopg2.ProgrammingError
2011-02-14 07:05:06,496 CRITICAL - column "platform_changeset" of relation "builds" does not exist
LINE 3:       (product, version, platform, buildid, platform_changes...
"""

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to comment #2)
> We also seem to get this regularly:
> """
>     if build_file[2]:
> 2011-02-14 07:05:01,734 WARNING - IndexError: list index out of range
> 2011-02-14 07:05:01,735 WARNING - Caught Error: <type 'exceptions.IndexError'>
> 2011-02-14 07:05:01,735 WARNING - list index out of range
> 2011-02-14 07:05:01,735 WARNING - trace back follows:
> 2011-02-14 07:05:01,736 WARNING - Traceback (most recent call last):
> 2011-02-14 07:05:01,736 WARNING - File
> "/data/socorro/application/socorro/cron/builds.py", line 140, in
> fetchAndRecordNightlyBuilds
>     if build_file[3]:
> 2011-02-14 07:05:01,737 WARNING - IndexError: list index out of range
> 2011-02-14 07:05:01,740 INFO - Did not find build entries in builds table for
> firefox 3.5.18pre linux-i686 20110214030349
> 2011-02-14 07:05:01,747 INFO - Inserted the following build: firefox 3.5.18pre
> linux-i686 20110214030349 061b6c646c10   firefox-3.5.18pre.en-US.linux-i686.txt
> 2011-02-14 07:05:01,798 WARNING - Caught Error: <type 'exceptions.IndexError'>
> 2011-02-14 07:05:01,798 WARNING - list index out of range
> 2011-02-14 07:05:01,799 WARNING - trace back follows:
> 2011-02-14 07:05:01,799 WARNING - Traceback (most recent call last):
> 2011-02-14 07:05:01,800 WARNING - File
> "/data/socorro/application/socorro/cron/builds.py", line 135, in
> fetchAndRecordNightlyBuilds
> """

Ah, this is addressed by https://bugzilla.mozilla.org/show_bug.cgi?id=628067#c12

Just focusing on the other one for now (comment 3).

Comment 5

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Does the builds table on stage match the 1.7.6 schema?  It should contain the fields platform_changeset, app_changeset_1 and app_changeset_2, per:
http://code.google.com/p/socorro/wiki/SocorroUpgrade#builds_table

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #3)
> (In reply to comment #1)
> > Some changes have been made to the builds script for 1.7.7, which may or may
> > not help here. They're in the 1.7.7 branch and once I can get those changes
> > merged then we can test on stage and see if there are remaining issues.
> > 
> > https://bugzilla.mozilla.org/show_bug.cgi?id=628067#c3
> 
> Stage has the latter stack from comment 2, but the former now throw this
> instead:
> 
> """
> 2011-02-14 07:05:06,477 WARNING - IndexError: list index out of range
> 2011-02-14 07:05:06,493 INFO - Did not find build entries in builds table for
> firefox 4.0b12pre linux-i686 20110214030347
> http://hg.mozilla.org/mozilla-central/rev/26421a3b68f3
> 2011-02-14 07:05:06,496 CRITICAL - Caught Error: psycopg2.ProgrammingError
> 2011-02-14 07:05:06,496 CRITICAL - column "platform_changeset" of relation
> "builds" does not exist
> LINE 3:       (product, version, platform, buildid, platform_changes...
> """

Looks like a schema change (from bug 562114) needs to be applied, it's in schema.py

Looks like it's in the 1.7.6 instructions:
http://code.google.com/p/socorro/wiki/SocorroUpgrade#Socorro_1.7.6

Ryan, how do you generally get these sorts of changes done on stage?

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #6)
> Looks like a schema change (from bug 562114) needs to be applied, it's in
> schema.py
> 
> Looks like it's in the 1.7.6 instructions:
> http://code.google.com/p/socorro/wiki/SocorroUpgrade#Socorro_1.7.6
> 
> Ryan, how do you generally get these sorts of changes done on stage?

Answering my own question, "by filing IT bug 634001" (now resolved). 

Testing on stage now.

Comment 8

[Robert Helmer [:rhelmer]]

Hmm seems better, fewer errors but seeing a few of these:

"""
2011-02-14 15:07:57,870 CRITICAL - Caught Error: psycopg2.DataError
2011-02-14 15:07:57,871 CRITICAL - invalid input syntax for integer: "2011021403
0007
http://hg.mozilla.org/mozilla-central/rev/393fbb42cfe3
http://hg.mozilla.org/comm-central/rev/d6059bdc7053"
"""

Looks like this is Thunderbird 3.3a3pre, and it's including both mozilla-central and comm-central repos, maybe we're not handling this case?:

See http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/thunderbird-3.3a3pre.en-US.mac.txt for an example.

Comment 9

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

In theory the c-c changeset should be "app_changeset_1", which should be supported by Ryan's changes for bug 562114 (though I can't pretend to follow the code).

Why is the buildID wrapped to two lines in the error output?  Is that just an 80-char limit in the logging?

Comment 10

[Robert Helmer [:rhelmer]]

Attachment: split on newline not space

This seems to work in my dev env if I split on newline rather than space; otherwise no split happens and we end up with something like '20110215030013\nhttp://hg.mozilla.org/mozilla-central/rev/2c646d10b9c7\nhttp://hg.mozilla.org/comm-central/rev/21879c945223' for the buildid.

Comment 11

[Robert Helmer [:rhelmer]]

Comment on attachment 512526 [details] [diff] [review]
split on newline not space

Thanks ryan!
Committed revision 2927.

Comment 12

[Robert Helmer [:rhelmer]]

I think this'll probably work on staging now and go out with the next Socorro release (1.7.7), I'll verify once staging updates.

Comment 13

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

I'm not sure what happened to some of the items (like parseChangeset()) from https://bug562114.bugzilla.mozilla.org/attachment.cgi?id=460667 . Rob - if you think this could cause an issue moving forward, then open up another ticket so we can get that resolved.

Comment 14

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #11)
> Comment on attachment 512526 [details] [diff] [review]
> split on newline not space
> 
> Thanks ryan!
> Committed revision 2927.

(Again, don't know the code, but…) Isn't that going to break things everywhere that bug 549958 hasn't yet landed (i.e., break all the branches)?  I hope to get approval for landing that in the next branch cycle, but for now the branches are still space-delimited and the trunk is newline-delimited.

Comment 15

[Robert Helmer [:rhelmer]]

(In reply to comment #14)
> (In reply to comment #11)
> > Comment on attachment 512526 [details] [diff] [review]
> > split on newline not space
> > 
> > Thanks ryan!
> > Committed revision 2927.
> 
> (Again, don't know the code, but…) Isn't that going to break things everywhere
> that bug 549958 hasn't yet landed (i.e., break all the branches)?  I hope to
> get approval for landing that in the next branch cycle, but for now the
> branches are still space-delimited and the trunk is newline-delimited.

Hmm I didn't see any problems testing this locally, I thought we caught this and printed a warning before getting to this point. Ryan do you know offhand? I can dig into it more if not.

Comment 16

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Rob - The script should be able to handle any errors that it encounters and error out safely if the file is not in the desired format.  

But now that I'm re-reading the ticket that was originally filed in bug 562114, we will need to account for both spaces/tabs in files that are in the old format, as well as line breaks in files that are in the new format.  Will split() without any parameters handle spaces/tabs as well as line breaks?

Comment 17

[Robert Helmer [:rhelmer]]

(In reply to comment #16)
> Rob - The script should be able to handle any errors that it encounters and
> error out safely if the file is not in the desired format.  
> 
> But now that I'm re-reading the ticket that was originally filed in bug 562114,
> we will need to account for both spaces/tabs in files that are in the old
> format, as well as line breaks in files that are in the new format.  Will
> split() without any parameters handle spaces/tabs as well as line breaks?

Looks like split handle that, as well as mixed tab/spaces:

>>> "test1 test2".split()
['test1', 'test2']
>>> "test1\ntest2".split()
['test1', 'test2']
>>> "test1\ntest2 test3".split()
['test1', 'test2', 'test3']

Do we have the format(s) for these files documented anywhere ? :) That might help to make sure we're covering everything...

Comment 18

[(not currently active) Ted Mielczarek]

Not really, except in the discussion on bug 474610 and bug 549958. bug 562114 comment 0 offers a pretty succinct explanation, though.

Comment 20

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Am I following the status correctly: this fix won't be deployed to production until 1.7.7?

Comment 21

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

The fix does appear to be working correctly on stage, though :) https://crash-stats.stage.mozilla.com/products/Firefox/builds

Comment 22

[Matt Brandt [:mbrandt]]

Verified using a link checker. Thx stephend for pointing me to this tool :)