874113 - deploy latest version of crash-data-tools

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Description

[Robert Helmer [:rhelmer]]

There is code from http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools deployed to /data/crash-data-tools/ on socorro stage and prod (I am not exactly sure how it gets deployed there!)

I've checked in a workaround for bug 867342 to http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/rev/0d9be01ab7ce which needs to be deployed.

This can happen any time, we don't need a downtime window or to sync with a Socorro release (this code is used by exactly one cron job, which runs daily)

Comment 1

[Brandon Burton [:solarce]]

I did a little digging and this is deployed from Puppet SVN

puppet/trunk/modules/socorro/files/prod/crash-data-tools

I chatted with :rhelmer and think we should switch this to a local checkout of Hg so I'll work on making that happen

Comment 2

[Brandon Burton [:solarce]]

I've made the changes to stage

bburton@ironbars [08:09:01] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn status
D       modules/webapp/files/socorro-stage/crash-data-tools
D       modules/webapp/files/socorro-stage/crash-data-tools/crashfinder.py
D       modules/webapp/files/socorro-stage/crash-data-tools/macdebugids.json
D       modules/webapp/files/socorro-stage/crash-data-tools/addonids-local.json
D       modules/webapp/files/socorro-stage/crash-data-tools/jsondb.py
D       modules/webapp/files/socorro-stage/crash-data-tools/addonids-amo.json
D       modules/webapp/files/socorro-stage/crash-data-tools/amo-ids.txt
D       modules/webapp/files/socorro-stage/crash-data-tools/per-crash-core-count.py
D       modules/webapp/files/socorro-stage/crash-data-tools/per-crash-interesting-modules.py
D       modules/webapp/files/socorro-stage/crash-data-tools/addonids.py
D       modules/webapp/files/socorro-stage/crash-data-tools/macdebugids.py
bburton@ironbars [08:15:41] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "cleaning up crash-data-tools on stage, moving to hg checkout, bug 874113"
Deleting       trunk/modules/webapp/files/socorro-stage/crash-data-tools
Sending        trunk/modules/webapp/manifests/admin/socorro.pp
Transmitting file data .
Committed revision 66104.

[[e-mail]1 crash-data-tools]# hg clone http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools  .
requesting all changes
adding changesets
adding manifests
adding file changes
added 102 changesets with 118 changes to 15 files
updating to branch default
11 files updated, 0 files merged, 0 files removed, 0 files unresolved
[[e-mail]1 crash-data-tools]# ls
addonids-amo.json    addonids.py  crashfinder.py  macdebugids.json  per-crash-core-count.py
addonids-local.json  amo-ids.txt  jsondb.py       macdebugids.py    per-crash-interesting-modules.py
[[e-mail]1 crash-data-tools]# puppetctl run --noop
(did nothing as expected)

[[e-mail]1 crash-data-tools]# hg summary
parent: 101:0d9be01ab7ce tip
 bug 867342 - ignore crash reason, take 2
branch: default
commit: (clean)
update: (current)

I assume you'll be able to see if things are happy with the next cron run? If so, let's wait for that and then do prod tomorrow

Comment 3

[Brandon Burton [:solarce]]

Did the cron job on stage complete as expected with the updated files?

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Brandon Burton [:solarce] from comment #3)
> Did the cron job on stage complete as expected with the updated files?

Hmm no they are all 0-byte still :( 

Let me dig a little bit into it today and get back to you.

Comment 5

[Robert Helmer [:rhelmer]]

Here's the traceback, digging into it now:

Traceback (most recent call last):
  File "/data/crash-data-tools/per-crash-interesting-modules.py", line 45, in <module>
    import macdebugids
  File "/data/crash-data-tools/macdebugids.py", line 59, in <module>
    local_db = LibraryInfoMap("macdebugids")
  File "/data/crash-data-tools/jsondb.py", line 56, in __init__
    self._map = json.load(io)
  File "/usr/lib64/python2.6/json/__init__.py", line 267, in load
    parse_constant=parse_constant, **kw)
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
    obj, end = self._scanner.iterscan(s, **kw).next()
  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib64/python2.6/json/decoder.py", line 183, in JSONObject
    value, end = iterscan(s, idx=end, context=context).next()
  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib64/python2.6/json/decoder.py", line 198, in JSONObject
    raise ValueError(errmsg("Expecting property name", s, end - 1))
ValueError: Expecting property name: line 101 column 3 (char 6755)

Comment 6

[Robert Helmer [:rhelmer]]

This is just the python JSON parser being picky about trailing spaces. I'll push an update to the hg repo.

Comment 7

[Robert Helmer [:rhelmer]]

Pushed to http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/rev/0a9b16dd44bb

Please update, hopefully that will fix!

Can you also rerun the cron so we don't have to wait until tomorrow to verify? It takes a while I suggest using screen. Run as socorro user on socorroadm stage:

# Correlation report for crash-analysis
05 00 * * * socorro /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #6)
> This is just the python JSON parser being picky about trailing spaces. I'll
> push an update to the hg repo.

Er, meant to say "trailing comma"

Comment 9

[Brandon Burton [:solarce]]

[[e-mail]1 crash-data-tools]# hg pull
pulling from http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
(run 'hg update' to get a working copy)
[[e-mail]1 crash-data-tools]# hg update
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
[[e-mail]1 crash-data-tools]# hg status
[[e-mail]1 crash-data-tools]# hg summary
parent: 102:0a9b16dd44bb tip
 fix syntax error
branch: default
commit: (clean)
update: (current)

Started in screen

[[e-mail]1 crash-data-tools]# sudo -u socorro /data/socorro/application/scripts/crons/cron_libraries.sh
[2013-05-22 21:28:59] lock cron_libraries
[2013-05-22 21:28:59] Processing for DATE: 20130522 and WEEK: 20130520
[2013-05-22 21:28:59] Phase 1: start
[2013-05-22 21:28:59] Phase 1: Product: Firefox
[2013-05-22 21:28:59] Running psql query for version list.
[2013-05-22 21:28:59] Phase 1: Version: 22.0 start
[2013-05-22 21:28:59] Running psql query for uuids and passing to hbaseClient.py

Comment 10

[Brandon Burton [:solarce]]

Redone with the logging redirection

[[e-mail]1 crash-data-tools]# su - socorro
[[e-mail]1 ~]$ /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1

Comment 11

[Brandon Burton [:solarce]]

The cron job finished

[[e-mail]1 ~]$ /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1
[[e-mail]1 ~]$

Comment 12

[Brandon Burton [:solarce]]

How's it looking?

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to Brandon Burton [:solarce] from comment #12)
> How's it looking?

Oh, it looks like a totally different (and unchanged) part of this just doesn't work on staging hbase :( 

I think we are fine to update prod, especially if we back up the old one. As I mentioned before this code is used for exactly one cronjob, which currently isn't working, so we can't really make it much worse :)

Comment 14

[Robert Helmer [:rhelmer]]

This is ready to go on production.

Comment 15

[Brandon Burton [:solarce]]

Prod has been updated, running the cron now

[[e-mail]1 crash-data-tools]# su - socorro
[[e-mail]1 ~]$ /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1

Comment 16

[Brandon Burton [:solarce]]

Cron job finished on prod

[[e-mail]1 ~]$ /data/socorro/application/scripts/crons/cron_libraries.sh > /var/log/socorro/cron_libraries.log 2>&1
[[e-mail]1 ~]$ date
Tue May 28 14:25:29 PDT 2013
[[e-mail]1 ~]$

Comment 17

[Robert Helmer [:rhelmer]]

lgtm, thanks!