Closed
Bug 938339
Opened 11 years ago
Closed 11 years ago
blobber failed to upload minidump but didn't give any useful debugging information
Categories
(Release Engineering :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bhearsum, Unassigned)
References
Details
Attachments
(2 files)
938 bytes,
patch
|
rail
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
940 bytes,
patch
|
rail
:
review+
rail
:
checked-in+
|
Details | Diff | Splinter Review |
From https://tbpl.mozilla.org/php/getParsedLog.php?id=30521103&tree=Try:
01:12:02 INFO - Running command: ['C:\\slave\\test\\build\\venv\\Scripts\\python', 'C:\\slave\\test\\build\\venv\\Scripts\\blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', 'C:\\slave\\test\\oauth.txt', '-b', 'try', '-d', 'C:\\slave\\test\\build\\blobber_upload_dir']
01:12:02 INFO - Copy/paste: C:\slave\test\build\venv\Scripts\python C:\slave\test\build\venv\Scripts\blobberc.py -u https://blobupload.elasticbeanstalk.com -a C:\slave\test\oauth.txt -b try -d C:\slave\test\build\blobber_upload_dir
01:12:08 INFO - (blobuploader) - INFO - Open directory for files ...
01:12:08 INFO - (blobuploader) - INFO - Uploading C:\slave\test\build\blobber_upload_dir\92eca2ba-4ee3-4718-bf39-9eda87271b3e.dmp ...
01:12:08 INFO - (blobuploader) - INFO - Using https://blobupload.elasticbeanstalk.com
01:12:08 INFO - (blobuploader) - INFO - Uploading, attempt #1.
01:12:09 INFO - (blobuploader) - CRITICAL - Something went wrong on blobber!
01:12:09 INFO - (blobuploader) - INFO - Upload failed. Trying again ...
01:12:09 INFO - (blobuploader) - INFO - Done attempting.
01:12:09 INFO - (blobuploader) - INFO - Iteration through files over.
If we're catching an exception around that CRITICAL message, maybe we should be setting print_exc=True in that log message? Or otherwise logging the traceback before eating it?
Comment 1•11 years ago
|
||
I'm testing out uploading logcats on ash and seeing something similar, not sure if it's the same error.
From https://tbpl.mozilla.org/php/getParsedLog.php?id=31178248&tree=Ash&full=1:
11:58:16 INFO - (blobuploader) - INFO - Open directory for files ...
11:58:16 INFO - (blobuploader) - INFO - Uploading /home/cltbld/talos-slave/test/build/blobber_upload_dir/emulator-5554.log ...
11:58:16 INFO - (blobuploader) - INFO - Using https://blobupload.elasticbeanstalk.com
11:58:16 INFO - (blobuploader) - INFO - Uploading, attempt #1.
11:58:16 INFO - (blobuploader) - CRITICAL - Unexpected error in client: <type 'exceptions.ValueError'>
11:58:16 INFO - (blobuploader) - INFO - Done attempting.
11:58:16 INFO - (blobuploader) - INFO - Iteration through files over.
Though oddly the xpcshell job which uses the same mozharness script had no errors:
https://tbpl.mozilla.org/php/getParsedLog.php?id=31178248&tree=Ash&full=1
Comment 2•11 years ago
|
||
Where does this need to be fixed? I couldn't locate blobberc.py upon quick inspection.
Flags: needinfo?(bhearsum)
Reporter | ||
Comment 3•11 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #2)
> Where does this need to be fixed? I couldn't locate blobberc.py upon quick
> inspection.
I'm not sure either...Rail?
Flags: needinfo?(bhearsum) → needinfo?(rail)
Comment 4•11 years ago
|
||
it comes from https://github.com/MihaiTabara/blobuploader
Flags: needinfo?(rail)
Comment 5•11 years ago
|
||
Yeah :-( Unfortunately it seems to be not very accurate as it is. Should bring some improvements there to narrow down easier the errors. Could be an error when reading the file :-/
Can we re-trigger the same job in debug mode ?
Comment 6•11 years ago
|
||
I don't think there is. Though in general, I believe that tracebacks should never be hidden behind a "debug" or "verbose" level, there isn't any reason to.
I submitted a pull request to add the traceback, though I don't know how to get it into after merge:
https://github.com/MihaiTabara/blobuploader/pull/5
Comment 7•11 years ago
|
||
*get it into production after merge
Comment 8•11 years ago
|
||
I merged it. Let me push another version and ask someone to push it inbound to our puppetagain data. Afterwards, we're good to go for a re-trigger.
Thanks a lot for the patch!
Comment 9•11 years ago
|
||
@Rail: can you please push this (1.0.1b) in our puppetagain servers?
https://pypi.python.org/pypi/blobuploader
I'll push the patch for mozharness in a few seconds.
Flags: needinfo?(rail)
Comment 10•11 years ago
|
||
* Bumped blobuploader version to 1.0.1b ( https://github.com/MihaiTabara/blobuploader )
! Please do *not* land this patch until the python package does not lie in our puppetagain servers. Currently it lies only in https://pypi.python.org/pypi/blobuploader
Attachment #8340470 -
Flags: review?(rail)
Attachment #8340470 -
Flags: review?(aki)
Comment 11•11 years ago
|
||
(In reply to Mihai Tabara [:mtabara] from comment #9)
> @Rail: can you please push this (1.0.1b) in our puppetagain servers?
This is done now, but it may take ~1hr to sync to all servers.
Flags: needinfo?(rail)
Updated•11 years ago
|
Attachment #8340470 -
Flags: review?(rail)
Attachment #8340470 -
Flags: review?(aki)
Attachment #8340470 -
Flags: review+
Comment 12•11 years ago
|
||
Comment on attachment 8340470 [details] [diff] [review]
Bump blobuploader version to 1.0.1b
pushed to default https://hg.mozilla.org/build/mozharness/rev/5e7b7164d07c
cedar should pick this up
Attachment #8340470 -
Flags: checked-in+
Comment 13•11 years ago
|
||
(In reply to Mihai Tabara [:mtabara] from comment #8)
> Thanks a lot for the patch!
Np, thanks for landing everything!
Comment 14•11 years ago
|
||
Any updates with that error on uploading logcats with the new blobuploader version?
Flags: needinfo?(ahalberstadt)
Comment 15•11 years ago
|
||
I tried re-triggering, but that didn't pick up my changes. I guess I need to push?
If so, results will be in the b2g reftests:
https://tbpl.mozilla.org/?tree=Ash&rev=cb16a4ba3e99
Flags: needinfo?(ahalberstadt)
Comment 16•11 years ago
|
||
I'm still seeing the old <value error> in the log instead of the traceback:
https://tbpl.mozilla.org/php/getParsedLog.php?id=31325796&tree=Ash&full=1
Note this is on Ash.. are the new changes rolled out there yet?
Flags: needinfo?(tabara.mihai)
Comment 17•11 years ago
|
||
There's a "[ ... ] Installing blobuploader==1.0b into virtualenv [...]" in the log which means it's not pulling out the correct version of blobuploader. Weird I'd say ... it seems like the change made it to production branch of mozharness: http://hg.mozilla.org/build/mozharness/file/1a2c01691638/mozharness/mozilla/blob_upload.py
AFAIK, Ash runs on default branch of mozharness but hence the changes are already in production branch, they should also be in default too, right?
I'm out of ideas. Let me ping rail for this.
Rail, any idea why the job's not pulling the new version of blobuploader?
Flags: needinfo?(tabara.mihai) → needinfo?(rail)
Comment 18•11 years ago
|
||
ash uses aki's repo, this is why the version is still old
Flags: needinfo?(rail)
Comment 19•11 years ago
|
||
Right, duh. I'll merge it. Thanks.
Comment 20•11 years ago
|
||
Doh, how could I forget that?!?
When I spammed Aki's mozharness with patches for days/weeks I knew this ... People have the tendency to forget things very fast, don't they? :-)
@rail: thanks for the tip
@ahal: please let me know the results after the re-triggering
Comment 21•11 years ago
|
||
(In reply to Mihai Tabara [:mtabara] from comment #20)
> Doh, how could I forget that?!?
>
> When I spammed Aki's mozharness with patches for days/weeks I knew this ...
> People have the tendency to forget things very fast, don't they? :-)
Yeah.. I use ash mozharness more than anyone I think, and still it didn't click.
Anyway, the error is:
(blobuploader) - CRITICAL - Unexpected error in client: Traceback (most recent call last):
File "/home/cltbld/talos-slave/test/build/venv/bin/blobberc.py", line 87, in upload_file
ret = post_file(host, auth, filename, branch, hashalgo, blobhash)
File "/home/cltbld/talos-slave/test/build/venv/bin/blobberc.py", line 144, in post_file
url = urlparse.urljoin(host, '/blobs/{}/{}'.format(hashalgo, blobhash))
ValueError: zero length field name in format
Flags: needinfo?(tabara.mihai)
Comment 22•11 years ago
|
||
Looks like this is happening with slaves running python 2.6 (which explains why it works for some jobs and not others).
See the accepted answer of http://stackoverflow.com/questions/10054122/valueerror-zero-length-field-name-in-format-python for more details.
Flags: needinfo?(tabara.mihai)
Comment 23•11 years ago
|
||
@ahal: thanks! I should've thought of that when I've implemented it. I had too some other issues with dict comprehensions for python 2.6 vs python 2.7 while working on blobber.
I'll patch it up today. Will ping here when it's done.
Comment 24•11 years ago
|
||
Bugfix here: https://github.com/MihaiTabara/blobuploader/commit/4813b5fe197a5b9ea97754c0073eed5c36a8072f
Bumped blobuploader version to 1.0.2b https://pypi.python.org/pypi/blobuploader
@rail: when you catch a minute, please have a short look and sync it up to our PuppetAgain.
I'll patch the mozharness version for when the sync is over.
Thank you.
Flags: needinfo?(rail)
Comment 25•11 years ago
|
||
Attachment #8342450 -
Flags: review?(rail)
Comment 26•11 years ago
|
||
(In reply to Mihai Tabara [:mtabara] from comment #24)
> @rail: when you catch a minute, please have a short look and sync it up to
> our PuppetAgain.
Thanks for the fix. Sync in progress. I'll push the patch once the file is synced.
Flags: needinfo?(rail)
Comment 27•11 years ago
|
||
Comment on attachment 8342450 [details] [diff] [review]
Bump blobuploader version to 1.0.2b
https://hg.mozilla.org/build/mozharness/rev/744423b7ebbb
Attachment #8342450 -
Flags: review?(rail)
Attachment #8342450 -
Flags: review+
Attachment #8342450 -
Flags: checked-in+
Comment 28•11 years ago
|
||
I've uploaded 1.0.2b to http://pypi.pub.build.mozilla.org using https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Other_Duties#Python_packages, to resolve the failures to install it on pine.
Comment 29•11 years ago
|
||
Comment on attachment 8342450 [details] [diff] [review]
Bump blobuploader version to 1.0.2b
merged to production.
Comment 30•11 years ago
|
||
:ahal should the things work OK now, can we close this bug?
Updated•11 years ago
|
Flags: needinfo?(ahalberstadt)
Comment 31•11 years ago
|
||
Yep, seems to be working on Ash. Thanks!
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: needinfo?(ahalberstadt)
Resolution: --- → FIXED
Assignee | ||
Updated•8 years ago
|
Component: Tools → General
You need to log in
before you can comment on or make changes to this bug.
Description
•