602201 - Test Socorro 1.7 and 1.8 on HBase 0.89 on Metrics research cluster

Status: RESOLVED WORKSFORME

(Socorro :: General, task, P1)

Description

[Laura Thomson :laura]

As above.  Let's find out what we need to do to make the upgrade work before we get to PHX.

Comment 1

[Robert Helmer [:rhelmer]]

For reference here is the stacktrace we see on Socorro 1.7 when trying to insert a crash from the collector:

2010-09-28 19:06:10,274 ERROR - trace back follows:
  File "/data/www/crash-reports.mozilla.com/application/socorro/collector/crashstorage.py", line 349, in save_raw
    self.hbaseConnection.put_json_dump(uuid, jsonData, dump.read(), number_of_retries=2)
  File "/data/www/crash-reports.mozilla.com/application/socorro/hbase/hbaseClient.py", line 143, in f
    result = fn(self, *args, **kwargs)
  File "/data/www/crash-reports.mozilla.com/application/socorro/hbase/hbaseClient.py", line 675, in put_json_dump
    self.client.mutateRow('crash_reports', row_id, mutationList) # unit test marker 233
  File "/data/www/crash-reports.mozilla.com/thirdparty/hbase/hbase.py", line 1251, in mutateRow
    self.recv_mutateRow()
  File "/data/www/crash-reports.mozilla.com/thirdparty/hbase/hbase.py", line 1274, in recv_mutateRow
    raise result.io

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: hbase test client using Thrift, from xstevens

Comment 3

[Robert Helmer [:rhelmer]]

Xavier and I were able to track this down to a missing column ("id") on the crash_reports table. 

On my dev Socorro 1.7 instance, the collector is able to insert crashes into HBase 0.89 on cm-hadoop25, and the processor is able to get crashes out and insert into my dev PostgreSQL.

I'll try Socorro 1.8 next.

Comment 4

[Robert Helmer [:rhelmer]]

Created bug 602545 to make this kind of problem easier for Socorro to detect in the future.

Comment 5

[Robert Helmer [:rhelmer]]

With Socorro 1.8, I was getting an exception around http://code.google.com/p/socorro/source/browse/trunk/socorro/storage/hbaseClient.py#672 which I think is due to the clock being off on my dev VM (NTP sync is now set up). However the exception is not logged, execution continues, and when it gets to http://code.google.com/p/socorro/source/browse/trunk/socorro/storage/hbaseClient.py#726 the exception that appears in the logs is "local variable 'now' referenced before assignment", which threw me off at first.

I'll file a followup bug about this.

Comment 6

[Robert Helmer [:rhelmer]]

This seems to work ok on my local dev instance. I think this is ready for staging.

I tested this with both Socorro 1.7 and 1.8, against cm-hadoop-dev02 (same version of HBase as production) and cm-hadoop25 (HBase 0.89):

1) insert crashes to local dev collector (via submitter.py)
2) local dev postgresql reports table is populated

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> With Socorro 1.8, I was getting an exception around
> http://code.google.com/p/socorro/source/browse/trunk/socorro/storage/hbaseClient.py#672
> which I think is due to the clock being off on my dev VM (NTP sync is now set
> up).

I am leaning towards this being a Socorro bug and not caused by the clock sync problem. Filed bug 604229 with more info and to follow-up.