562233 - Push Socorro 1.6.2 to production on 4/28/10 (off regular schedule)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Thomson :laura]

We really need to get this out this week to clear staging for 1.7.  Aravind agreed to push Wednesday pm.

Instructions for upgrade:
http://code.google.com/p/socorro/wiki/SocorroUpgrade#Socorro_1.6.2

I'll post a tag once we finish testing this last bug.

Comment 1

[matthew zeier [:mrz]]

Start time?  Duration?  What goes offline?

Comment 2

[Shyam Mani [:fox2mike]]

Over to Aravind.

Comment 3

[Laura Thomson :laura]

mrz: discussed with aravind yesterday.  Probably around 5pm, duration < 10 minutes.  We have new db columns + index on new column (should be fast), config changes/update/restart for the processor, and config changes and updates for the UI.  Collectors will continue to run and no data should be missed.

Comment 4

[matthew zeier [:mrz]]

There's another bug tracking rebooting all the hadoop servers - is this a good time to do that too?

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

Dunno, depends on what Aravind says about staging.

Comment 6

[Laura Thomson :laura]

Tag is:
http://socorro.googlecode.com/svn/tags/releases/1.6.2_r2010_20100428/

aravind: you'll need to svn switch to this tag.

Comment 7

[Aravind Gottipati [:aravind]]

In the past I haven't switched the processors/monitor to tags, I would just update to the latest in trunk.  Will that work or do I need to switch processors/monitor to this tag as well?

Comment 8

[Laura Thomson :laura]

That's the intention of tagging releases, so yep.

Also, yes, it's ok to reboot the Hadoop boxes at this time.

Comment 9

[K Lars Lohn [:lars] [:klohn]]

this release includes a change to the processor.  That change is reflected in the tag for this release.  http://code.google.com/p/socorro/wiki/SocorroUpgrade#Socorro_1.6.2 in the processor section includes instructions to stop the monitors and processors, pull from this tag, and then restart the processors and monitor.  There should be no complications with pulling from this tag as opposed to pulling from trunk.  

Monitor does not need to be updated, but updating it would harm nothing.

Comment 10

[Aravind Gottipati [:aravind]]

I will update the bug as I go along, so you guys should be able to see if I missed anything.

Shut down all the processors and monitor.

db updates done.  Here is the output.

breakpad=> alter table reports_20090202 no inherit reports;
ALTER TABLE
breakpad=> alter table reports_20090202
breakpad->   add column hangid TEXT
breakpad-> ;
ALTER TABLE
breakpad=> alter table reports
breakpad->   add column hangid TEXT
breakpad-> ;
ALTER TABLE
breakpad=> alter table reports_20090202 inherit reports;
ALTER TABLE
breakpad=> CREATE INDEX reports_20100426_hangid_idx ON reports_20100426(hangid);
CREATE INDEX
breakpad=>

Comment 11

[Aravind Gottipati [:aravind]]

Here is the output from the svn switch.

[root@cm-breakpad02 scripts]# svn switch http://socorro.googlecode.com/svn/tags/releases/1.6.2_r2010_20100428/scripts
U    config/processorconfig.py.dist
Updated to revision 2011.
[root@cm-breakpad02 scripts]# cd ../socorro/
[root@cm-breakpad02 socorro]# svn switch http://socorro.googlecode.com/svn/tags/releases/1.6.2_r2010_20100428/socorro
U    unittest/processor/testProcessorWithExternalBreakpad.py
U    unittest/processor/testProcessor.py
U    unittest/cron/testBugzilla.py
U    unittest/cron/testDailyUrl.py
 U   unittest/cron
U    unittest/database/testSchemaPartitionedTable.py
 U   unittest/testlib/createJsonDumpStore.py
 U   unittest/testlib/testDB.py
 U   unittest/testlib/loggerForTest.py
 U   unittest/monitor
U    processor/processor.py
G    processor/externalProcessor.py
U    cron/dailyUrl.py
U    database/schema.py
U    collector/submitter.py
U    collector/crashstorage.py
Updated to revision 2011.
[root@cm-breakpad02 socorro]#

Comment 12

[Aravind Gottipati [:aravind]]

Okay, all processors and monitor switched to the tag and restarted.

Config changed, so this line now reads

signatureSentinels.default = ['_purecall', ('mozilla::ipc::RPCChannel::Call(IPC::Message*, IPC::Message*)', lambda x: 'CrashReporter::CreatePairedMinidumps(void*, unsigned long, nsAString_internal*, nsILocalFile**, nsILocalFile**)' in x)]

Comment 13

[Aravind Gottipati [:aravind]]

UI updates picked up the following changes.

[root@mradm02 webapp-php]# svn switch http://socorro.googlecode.com/svn/tags/releases/1.6.2_r2010_20100428/webapp-php
U    application/models/report.php
U    application/models/topcrashers.php
U    application/models/common.php
U    application/config/codebases.php
U    application/controllers/query.php
U    application/controllers/report.php
U    application/libraries/MY_SearchReportHelper.php
U    application/views/layout.php
 U   application/views/mtbf
D    application/views/common/list_topcrashers_old.php
A    application/views/common/hang_details.php
U    application/views/common/query_form.php
U    application/views/common/list_topcrashers.php
U    application/views/report/do_list.php
U    application/views/report/index.php
U    css/screen.css
A    img/ajax-loader16x16.gif
A    img/3rdparty
A    img/3rdparty/silk
A    img/3rdparty/silk/chart_curve.png
A    img/3rdparty/silk/readme.txt
A    img/3rdparty/fatcow
A    img/3rdparty/fatcow/brick16x16.png
A    img/3rdparty/fatcow/application16x16.png
A    img/3rdparty/fatcow/stop16x16.png
U    js/socorro/query.js
U    js/socorro/report_list.js
U    js/socorro/topcrash.js
 U   js/socorro
Updated to revision 2011.

Config updated to point so it now says this.
$config['vcsMappings'] = array(
    'cvs' => array(
        'cvs.mozilla.org/cvsroot' =>
          'http://bonsai.mozilla.org/cvsblame.cgi?file=%(file)s&rev=%(revision)s&mark=%(line)s#%(line)s'
    ),
    'hg' => array(
        'hg.mozilla.org/mozilla-central' =>
            'http://hg.mozilla.org/mozilla-central/annotate/%(revision)s/%(file)s#l%(line)s',
    'hg.mozilla.org/comm-central' =>
        'http://hg.mozilla.org/comm-central/index.cgi/annotate/%(revision)s/%(file)s#l%(line)s'
    )
);

Will flush the NS caches in like 5 minutes, other than that, I should be all done here.

Comment 14

[Aravind Gottipati [:aravind]]

Flushed the NS cache.