1079144 - File 20141005-modulelist.txt didn't get generated

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[Ludovic Hirlimann [:Usul]]

I've followed the runbook - but it failed. When I tried again I got :

[[e-mail]1 ~]$ export JAVA_HOME=/usr/lib/jvm/java-1.6.0; export HADOOP_GW="cherry-gw.cherry.metrics.phx1.mozilla.com"; /data/socorro/application/scripts/crons/cron_modulelist.sh
Fatal exit code: 6
pig run failed

Comment 1

[Robert Helmer [:rhelmer]]

The run triggered from cron was almost done but was killed:

2014-10-07 08:40:36,941 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 96% complete
Killed by signal 1.

Manual run failed because the output file already exists:

2014-10-07 08:57:55,025 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: 
<file ./socorro/analysis/modulelist.pig, line 37, column 0> Output Location Validation Failed for: 'hdfs://nameservice1/user/socorro/modulelist-141006-141006 More info to follow:
Output directory hdfs://nameservice1/user/socorro/modulelist-141006-141006 already exists
Details at logfile: socorro-modulelist.log

I'll clean up the output file and try another manual run.

Comment 2

[Robert Helmer [:rhelmer]]

Removed:

[[e-mail]1 ~]$ export JAVA_HOME=/usr/lib/jvm/java-1.6.0; export HADOOP_GW="cherry-gw.cherry.metrics.phx1.mozilla.com";
[[e-mail]1 ~]$ DATE=`date -d 'yesterday' +%y%m%d`
[[e-mail]1 ~]$ ssh $HADOOP_GW "hadoop fs -rmr modulelist-${DATE}-${DATE}"
rmr: DEPRECATED: Please use 'rm -r' instead.
14/10/07 09:09:29 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.
Moved: 'hdfs://nameservice1/user/socorro/modulelist-141006-141006' to trash at: hdfs://nameservice1/user/socorro/.Trash/Current


Running manually now (same as comment 0)

Comment 3

[Ludovic Hirlimann [:Usul]]

of course today's file didn't get generated : 20141005-modulelist.tx :(

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #2)
> Removed:
> 
> [[e-mail]1 ~]$ export JAVA_HOME=/usr/lib/jvm/java-1.6.0;
> export HADOOP_GW="cherry-gw.cherry.metrics.phx1.mozilla.com";
> [[e-mail]1 ~]$ DATE=`date -d 'yesterday' +%y%m%d`

Oops this should have been:
DATE=141005

> [[e-mail]1 ~]$ ssh $HADOOP_GW "hadoop fs -rmr
> modulelist-${DATE}-${DATE}"

Did this again, rerunning comment 0 now.