1426346 - Intermittent Error: HTTPSConnectionPool(host='crash-stats.mozilla.com', port=443): Read timed out. and MemoryError during later upload attempt

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=152494244&repo=mozilla-beta

Uploading symbol file "target.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"
Attempt 1 of 5...
Error: HTTPSConnectionPool(host='crash-stats.mozilla.com', port=443): Read timed out.
Retrying...
Attempt 2 of 5...
Traceback (most recent call last):
  File "upload_symbols.py", line 101, in <module>
    sys.exit(main())
  File "upload_symbols.py", line 80, in main
    timeout=120)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 290, in prepare
    self.prepare_body(data, files)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 427, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 145, in _encode_files
    body, content_type = encode_multipart_formdata(new_fields)
  File "/usr/lib/python2.7/dist-packages/urllib3/filepost.py", line 101, in encode_multipart_formdata
    return body.getvalue(), content_type
MemoryError
[taskcluster 2017-12-20 00:37:01.771Z] === Task Finished ===
[taskcluster 2017-12-20 00:37:01.771Z] Unsuccessful task run with exit code: 1 completed in 256.46 seconds

Comment 1

[Chris AtLee [:catlee]]

ted, are we trying to load the entire file into memory first?

Were you working on a way to avoid downloading/uploading the symbols in the symbol submission tasks?

Comment 2

[(not currently active) Ted Mielczarek]

(In reply to Chris AtLee [:catlee] from comment #1)
> ted, are we trying to load the entire file into memory first?

I didn't *think* requests would be that dumb, but I guess maybe it does? The docs say:

"In the event you are posting a very large file as a multipart/form-data request, you may want to stream the request. By default, requests does not support this, but there is a separate package which does - requests-toolbelt."

http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file

(In reply to Chris AtLee [:catlee] from comment #1)
> Were you working on a way to avoid downloading/uploading the symbols in the
> symbol submission tasks?

Yes! In fact I just landed bug 1423881 yesterday. It has a few prerequisites we'd need if we wanted to uplift it, though.

Comment 3

[Intermittent Failures Robot]

1 failures in 590 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1426346&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 462 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 1

Platform breakdown:
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1426346&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There are similar upload failures:

https://treeherder.mozilla.org/logviewer.html#?job_id=154775720&repo=mozilla-beta
https://treeherder.mozilla.org/logviewer.html#?job_id=154775939&repo=mozilla-beta

Uploading symbol file "target.crashreporter-symbols-full.zip" to "https://crash-stats.mozilla.com/symbols/upload"
Attempt 1 of 5...
Traceback (most recent call last):
  File "upload_symbols.py", line 101, in <module>
    sys.exit(main())
  File "upload_symbols.py", line 80, in main
    timeout=120)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 290, in prepare
    self.prepare_body(data, files)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 427, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 145, in _encode_files
    body, content_type = encode_multipart_formdata(new_fields)
  File "/usr/lib/python2.7/dist-packages/urllib3/filepost.py", line 101, in encode_multipart_formdata
    return body.getvalue(), content_type
MemoryError

Comment 6

[Intermittent Failures Robot]

4 failures in 788 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 4

Platform breakdown:
* linux32-devedition: 2
* linux64: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1426346&startday=2018-01-08&endday=2018-01-14&tree=all

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Found in triaging. I think this bug outlived its usefulness. Closing this but please re-open if I'm wrong.