Closed Bug 938339 Opened 11 years ago Closed 11 years ago

blobber failed to upload minidump but didn't give any useful debugging information

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bhearsum, Unassigned)

References

Details

Attachments

(2 files)

Bump blobuploader version to 1.0.1b
938 bytes, patch
rail
: review+
rail
: checked-in+
Details | Diff | Splinter Review
Bump blobuploader version to 1.0.2b
940 bytes, patch
rail
: review+
rail
: checked-in+
Details | Diff | Splinter Review 
From https://tbpl.mozilla.org/php/getParsedLog.php?id=30521103&tree=Try:
01:12:02     INFO - Running command: ['C:\\slave\\test\\build\\venv\\Scripts\\python', 'C:\\slave\\test\\build\\venv\\Scripts\\blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', 'C:\\slave\\test\\oauth.txt', '-b', 'try', '-d', 'C:\\slave\\test\\build\\blobber_upload_dir']
01:12:02     INFO - Copy/paste: C:\slave\test\build\venv\Scripts\python C:\slave\test\build\venv\Scripts\blobberc.py -u https://blobupload.elasticbeanstalk.com -a C:\slave\test\oauth.txt -b try -d C:\slave\test\build\blobber_upload_dir
01:12:08     INFO -  (blobuploader) - INFO - Open directory for files ...
01:12:08     INFO -  (blobuploader) - INFO - Uploading C:\slave\test\build\blobber_upload_dir\92eca2ba-4ee3-4718-bf39-9eda87271b3e.dmp ...
01:12:08     INFO -  (blobuploader) - INFO - Using https://blobupload.elasticbeanstalk.com
01:12:08     INFO -  (blobuploader) - INFO - Uploading, attempt #1.
01:12:09     INFO -  (blobuploader) - CRITICAL - Something went wrong on blobber!
01:12:09     INFO -  (blobuploader) - INFO - Upload failed. Trying again ...
01:12:09     INFO -  (blobuploader) - INFO - Done attempting.
01:12:09     INFO -  (blobuploader) - INFO - Iteration through files over.


If we're catching an exception around that CRITICAL message, maybe we should be setting print_exc=True in that log message? Or otherwise logging the traceback before eating it?
I'm testing out uploading logcats on ash and seeing something similar, not sure if it's the same error.

From https://tbpl.mozilla.org/php/getParsedLog.php?id=31178248&tree=Ash&full=1:
11:58:16     INFO -  (blobuploader) - INFO - Open directory for files ...
11:58:16     INFO -  (blobuploader) - INFO - Uploading /home/cltbld/talos-slave/test/build/blobber_upload_dir/emulator-5554.log ...
11:58:16     INFO -  (blobuploader) - INFO - Using https://blobupload.elasticbeanstalk.com
11:58:16     INFO -  (blobuploader) - INFO - Uploading, attempt #1.
11:58:16     INFO -  (blobuploader) - CRITICAL - Unexpected error in client: <type 'exceptions.ValueError'>
11:58:16     INFO -  (blobuploader) - INFO - Done attempting.
11:58:16     INFO -  (blobuploader) - INFO - Iteration through files over.

Though oddly the xpcshell job which uses the same mozharness script had no errors:
https://tbpl.mozilla.org/php/getParsedLog.php?id=31178248&tree=Ash&full=1
Where does this need to be fixed? I couldn't locate blobberc.py upon quick inspection.
Flags: needinfo?(bhearsum)
(In reply to Andrew Halberstadt [:ahal] from comment #2)
> Where does this need to be fixed? I couldn't locate blobberc.py upon quick
> inspection.

I'm not sure either...Rail?
Flags: needinfo?(bhearsum) → needinfo?(rail)
it comes from https://github.com/MihaiTabara/blobuploader
Flags: needinfo?(rail)
Yeah :-( Unfortunately it seems to be not very accurate as it is. Should bring some improvements there to narrow down easier the errors. Could be an error when reading the file :-/

Can we re-trigger the same job in debug mode ?
I don't think there is. Though in general, I believe that tracebacks should never be hidden behind a "debug" or "verbose" level, there isn't any reason to.

I submitted a pull request to add the traceback, though I don't know how to get it into after merge:
https://github.com/MihaiTabara/blobuploader/pull/5
*get it into production after merge
I merged it. Let me push another version and ask someone to push it inbound to our puppetagain data. Afterwards, we're good to go for a re-trigger.

Thanks a lot for the patch!
@Rail: can you please push this (1.0.1b) in our puppetagain servers?

https://pypi.python.org/pypi/blobuploader

I'll push the patch for mozharness in a few seconds.
Flags: needinfo?(rail)
* Bumped blobuploader version to 1.0.1b ( https://github.com/MihaiTabara/blobuploader )

! Please do *not* land this patch until the python package does not lie in our puppetagain servers. Currently it lies only in https://pypi.python.org/pypi/blobuploader
Attachment #8340470 - Flags: review?(rail)
Attachment #8340470 - Flags: review?(aki)
(In reply to Mihai Tabara [:mtabara] from comment #9)
> @Rail: can you please push this (1.0.1b) in our puppetagain servers?

This is done now, but it may take ~1hr to sync to all servers.
Flags: needinfo?(rail)
Attachment #8340470 - Flags: review?(rail)
Attachment #8340470 - Flags: review?(aki)
Attachment #8340470 - Flags: review+
Comment on attachment 8340470 [details] [diff] [review]
Bump blobuploader version to 1.0.1b

pushed to default https://hg.mozilla.org/build/mozharness/rev/5e7b7164d07c
cedar should pick this up
Attachment #8340470 - Flags: checked-in+
(In reply to Mihai Tabara [:mtabara] from comment #8)
> Thanks a lot for the patch!

Np, thanks for landing everything!
Depends on: 945012
Any updates with that error on uploading logcats with the new blobuploader version?
Flags: needinfo?(ahalberstadt)
I tried re-triggering, but that didn't pick up my changes. I guess I need to push?

If so, results will be in the b2g reftests:
https://tbpl.mozilla.org/?tree=Ash&rev=cb16a4ba3e99
Flags: needinfo?(ahalberstadt)
I'm still seeing the old <value error> in the log instead of the traceback:
https://tbpl.mozilla.org/php/getParsedLog.php?id=31325796&tree=Ash&full=1

Note this is on Ash.. are the new changes rolled out there yet?
Flags: needinfo?(tabara.mihai)
There's a "[ ... ] Installing blobuploader==1.0b into virtualenv [...]" in the log which means it's not pulling out the correct version of blobuploader. Weird I'd say ... it seems like the change made it to production branch of mozharness: http://hg.mozilla.org/build/mozharness/file/1a2c01691638/mozharness/mozilla/blob_upload.py

AFAIK, Ash runs on default branch of mozharness but hence the changes are already in production branch, they should also be in default too, right?

I'm out of ideas. Let me ping rail for this.
Rail, any idea why the job's not pulling the new version of blobuploader?
Flags: needinfo?(tabara.mihai) → needinfo?(rail)
ash uses aki's repo, this is why the version is still old
Flags: needinfo?(rail)
Right, duh. I'll merge it. Thanks.
Doh, how could I forget that?!? 

When I spammed Aki's mozharness with patches for days/weeks I knew this ... People have the tendency to forget things very fast, don't they? :-)

@rail: thanks for the tip
@ahal: please let me know the results after the re-triggering
(In reply to Mihai Tabara [:mtabara] from comment #20)
> Doh, how could I forget that?!? 
> 
> When I spammed Aki's mozharness with patches for days/weeks I knew this ...
> People have the tendency to forget things very fast, don't they? :-)

Yeah.. I use ash mozharness more than anyone I think, and still it didn't click.

Anyway, the error is:
  (blobuploader) - CRITICAL - Unexpected error in client: Traceback (most recent call last):
    File "/home/cltbld/talos-slave/test/build/venv/bin/blobberc.py", line 87, in upload_file
      ret = post_file(host, auth, filename, branch, hashalgo, blobhash)
    File "/home/cltbld/talos-slave/test/build/venv/bin/blobberc.py", line 144, in post_file
      url = urlparse.urljoin(host, '/blobs/{}/{}'.format(hashalgo, blobhash))
  ValueError: zero length field name in format
Flags: needinfo?(tabara.mihai)
Looks like this is happening with slaves running python 2.6 (which explains why it works for some jobs and not others).

See the accepted answer of http://stackoverflow.com/questions/10054122/valueerror-zero-length-field-name-in-format-python for more details.
Flags: needinfo?(tabara.mihai)
@ahal: thanks! I should've thought of that when I've implemented it. I had too some other issues with dict comprehensions for python 2.6 vs python 2.7 while working on blobber.

I'll patch it up today. Will ping here when it's done.
Bugfix here: https://github.com/MihaiTabara/blobuploader/commit/4813b5fe197a5b9ea97754c0073eed5c36a8072f
Bumped blobuploader version to 1.0.2b https://pypi.python.org/pypi/blobuploader

@rail: when you catch a minute, please have a short look and sync it up to our PuppetAgain.
I'll patch the mozharness version for when the sync is over.

Thank you.
Flags: needinfo?(rail)
(In reply to Mihai Tabara [:mtabara] from comment #24)
> @rail: when you catch a minute, please have a short look and sync it up to
> our PuppetAgain.

Thanks for the fix. Sync in progress. I'll push the patch once the file is synced.
Flags: needinfo?(rail)
Comment on attachment 8342450 [details] [diff] [review]
Bump blobuploader version to 1.0.2b

https://hg.mozilla.org/build/mozharness/rev/744423b7ebbb
Attachment #8342450 - Flags: review?(rail)
Attachment #8342450 - Flags: review+
Attachment #8342450 - Flags: checked-in+
Comment on attachment 8342450 [details] [diff] [review]
Bump blobuploader version to 1.0.2b

merged to production.
:ahal should the things work OK now, can we close this bug?
Flags: needinfo?(ahalberstadt)
Yep, seems to be working on Ash. Thanks!
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: needinfo?(ahalberstadt)
Resolution: --- → FIXED
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: