571228 - Push Socorro 1.7 to production

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Thomson :laura]

The tag for this release is http://socorro.googlecode.com/svn/tags/releases/1.7_r2148_20100610/

Upgrade instructions are at 
http://code.google.com/p/socorro/wiki/SocorroUpgrade#Socorro_1.7

We have a pre-release meeting in #362 at 10 PT, and will push immediately after, probably around 11PT.

Comment 1

[Aravind Gottipati [:aravind]]

Working on collector configs, waiting for Daniel to turn off crash submissions into HBase.

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

Downloaded and extracted new hbase version

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

Symlinked to 0.20.5
copied CDH 2 hadoop jars to hadoop/lib

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

LZO jar and .so copied to /usr/lib/hbase-0.20.5/lib and .so symlinked.

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

Official production configs for hbase 0.20.5 copied from ~deinspanjer/hbase_conf to /usr/lib/hbase-0.20.5/conf

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

hbase-0.20.5-20100602 pushed to all production cluster
chowned to hadoop.hadoop
symlinked to hbase-0.20.5

Comment 7

[Aravind Gottipati [:aravind]]

Turned off crash submissions to HBase.  Still collecting into nfs.

Still working on other config files.

Comment 8

[Aravind Gottipati [:aravind]]

Copied the maintenance page over the index page, that should propagate out in like 10 minutes.

Comment 9

[Daniel Einspanjer [:dre] [:deinspanjer]]

Thrift stopped.  Starting hbase master shutdown

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

Master stopped
updated /usr/lib/hbase to point at hbase-0.20.5
Starting master

Comment 11

[Daniel Einspanjer [:dre] [:deinspanjer]]

Master started.  Waiting for regions to be assigned.

Comment 12

[Daniel Einspanjer [:dre] [:deinspanjer]]

Ran set_meta_block_cache.
Upgrading schema according to https://svn.mozilla.org/metrics/hadoop/socorro/hbase/schema_migration.txt

Comment 13

[Daniel Einspanjer [:dre] [:deinspanjer]]

Almost done with expensive migrations.  Create table statements will be fast.

Comment 14

[Daniel Einspanjer [:dre] [:deinspanjer]]

Wrong with almost done.  The second alter is taking much longer than the first.  Unfortunately, I don't have *any* way to tell progress at the moment..

Need one more migration to up the max region file size for the existing crash_reports table. Not in the migration file currently.

alter 'crash_reports', {METHOD => 'table_att', MAX_FILESIZE => '1073741824'}

Comment 15

[Daniel Einspanjer [:dre] [:deinspanjer]]

Second one took 35 minutes.
Doing the table_att MAX_FILESIZE one now.
I think the third alter might be quicker like the first.  The second was probably very slow because the flags: column family already existed.

Comment 16

[Daniel Einspanjer [:dre] [:deinspanjer]]

MAX_FILESIZE alter done. 792 seconds.
Doing last alter now.

Comment 17

[Daniel Einspanjer [:dre] [:deinspanjer]]

Last alter took 162 seconds.
Enabling crash reports table now.

Comment 18

[Daniel Einspanjer [:dre] [:deinspanjer]]

Done enabling big table.
Creating new tables now.

Comment 19

[Daniel Einspanjer [:dre] [:deinspanjer]]

Done with all migrations.
Started thrift servers.
HBase upgrade complete

Comment 20

[Aravind Gottipati [:aravind]]

Done with the collectors, monitor, processors and the middleware layer.

Working on the php front end next.

Comment 21

[Aravind Gottipati [:aravind]]

I am now done with webapp and cron jobs as well.  All done..

Comment 22

[Austin King [:ozten]]

Please restart web service layer. webapp-php can't talk to it.

Comment 23

[Austin King [:ozten]]

Did a crash me now for d62d8c64-206c-4523-852d-ba5b12100610

it's in hadoop

http://crash-stats.mozilla.com/dumps/d62d8c64-206c-4523-852d-ba5b12100610.jsonz
is a 500 system error
"Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.3 (Red Hat) Server at dm-bp-mware01.mozilla.org Port 80"

Comment 24

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

The advanced search page at http://crash-stats.mozilla.com/query is encountering connection reset errors with every single query and has not completed a successful query yet.

Comment 25

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Upon the first page load at http://crash-stats.mozilla.com/query, I am receiving this error:

"The maximum query date range you can perform is days. Admins may log in to increase query date range limits. Query results have been narrowed to the default range of ."

This leads me to believe that a config file is missing certain values.  Please ensure this array is found at the bottom of application/config/application.php:

/**
 * The query range limit for users who have the role of user and admin.
 *
 * @see My_SearchReportHelper->normalizeDateUnitAndValue()
 */
$config['query_range_defaults'] = array(
    'admin' => array(
        'range_default_value' => 14,
        'range_default_unit' => 'days',
        'range_limit_value_in_days' => 120
    ),
    'user' => array(
        'range_default_value' => 14,
        'range_default_unit' => 'days',
        'range_limit_value_in_days' => 30
    )
);

Comment 26

[Austin King [:ozten]]

(In reply to comment #22)
(In reply to comment #23)
These are fixed.

Comment 27

[Aravind Gottipati [:aravind]]

Running the daily crash job now.  Its set to run in cron at 00:15

Comment 28

[Austin King [:ozten]]

Verification status:

Many features are working except:

1) ADU Report as noted in Comment #27
2) Most Search Queries are timing out

Performance has been improving and load on Postgres has dropped form 8 to 6 to 1 in the last hour. We're going to build out #1 and regroup at 6:40pm to see how #2 is looking.

Comment 29

[Austin King [:ozten]]

(In reply to comment #28)
WTF... #2 advanced search is fixed.

Comment 30

[Austin King [:ozten]]

http://crash-stats.mozilla.com/daily is working now.

Verification complete.

Comment 31

[Stephen Donner [:stephend] Not actively reading bugmail]

I can't verify all the bugs pushed in 1.7, but I've run through a series of post-push tests, and it's looking good to me (plus comment 29 and comment 30; thanks, Austin!)

Verified.