MemoryError exception ingesting data on vagrant

RESOLVED FIXED

Status

Tree Management
Treeherder: Data Ingestion
P1
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: camd, Assigned: wlach)

Tracking

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
I'm hitting memory errors when ingesting data on a freshly created vagrant instance (after destroy) on master.

In one tab I run:
./bin/run_gunicorn

in the second, I run:
celery -A treeherder worker -B --concurrency 5

As I wait, I get an error:
[2015-09-23 13:46:33,038] WARNING [treeherder.etl.pushlog:120] Unable to get last push from cache for 'cedar', getting all pushes
[2015-09-23 13:46:42,080] INFO [treeherder.etl.th_publisher:32] collection loading request for project cedar: resultset
[2015-09-23 13:46:42,968] WARNING [treeherder.etl.pushlog:120] Unable to get last push from cache for 'cypress', getting all pushes
[2015-09-23 13:46:50,112: ERROR/MainProcess] Task fetch-hg-push-logs[61ae163b-642b-427f-9e5c-e741f9ca74a6] raised unexpected: MemoryError()
Traceback (most recent call last):
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/vagrant/treeherder/treeherder/etl/tasks/buildapi_tasks.py", line 57, in fetch_hg_push_log
    process.run(repo_url + '/json-pushes/?full=1&version=2', repo_name)
  File "/home/vagrant/treeherder/treeherder/etl/pushlog.py", line 121, in run
    extracted_content = self.extract(source_url)
  File "/home/vagrant/treeherder/treeherder/etl/pushlog.py", line 78, in extract
    return response.json()
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/requests/models.py", line 812, in json
    return json.loads(self.content.decode(encoding), **kwargs)
  File "/home/vagrant/venv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
MemoryError


Also:

[2015-09-23 13:47:13,668] ERROR [treeherder.etl.mixins:33] Error fetching https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js
Traceback (most recent call last):
  File "/home/vagrant/treeherder/treeherder/etl/mixins.py", line 31, in extract
    return json.loads(handler.read())
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/simplejson/__init__.py", line 505, in loads
    return _default_decoder.decode(s)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 1107088 (char 1107087)
[2015-09-23 13:47:13,679: ERROR/MainProcess] Task fetch-buildapi-running[f346b121-189f-46fe-a3e8-6b90bdbe0d3a] raised unexpected: TypeError("'NoneType' object has no attribute '__getitem__'",)
Traceback (most recent call last):
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/vagrant/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/vagrant/treeherder/treeherder/etl/tasks/buildapi_tasks.py", line 25, in fetch_buildapi_running
    RunningJobsProcess().run()
  File "/home/vagrant/treeherder/treeherder/etl/buildapi.py", line 498, in run
    filter_to_job_group=filter_to_job_group)
  File "/home/vagrant/treeherder/treeherder/etl/buildapi.py", line 296, in transform
    for project, revisions in data[source].iteritems():
TypeError: 'NoneType' object has no attribute '__getitem__'

Comment 1

2 years ago
Tweaking summary so people on IRC who've been hitting the same issue can find it.
Priority: -- → P1
Summary: Memory errors ingesting data on vagrant → MemoryError exception ingesting data on vagrant

Comment 2

2 years ago
Is maybe mysql 5.6 taking up more of the VM's memory?
I think so, I saw this when doing local conversions of the perfherder data when testing bug 1192976.

Our vagrant instance is currently only configured with 1G of memory, which is ridiculously small. Let's at least double that, then we can tweak mysql settings if we still notice a problem.
Created attachment 8665450 [details]
PR
Assignee: nobody → wlachance
Attachment #8665450 - Flags: review?(emorley)
(Reporter)

Comment 5

2 years ago
Wlach: for some reason your PR attachment just brings up the number of the bug for me.  I think it needs the PR link.  Thanks for doing this!
Comment on attachment 8665450 [details]
PR

https://github.com/mozilla/treeherder/pull/1003
Comment on attachment 8665450 [details]
PR

https://github.com/mozilla/treeherder/pull/1003
Created attachment 8665454 [details] [review]
actual PR
Attachment #8665450 - Attachment is obsolete: true
Attachment #8665450 - Flags: review?(emorley)
Attachment #8665454 - Flags: review?(emorley)
(In reply to Cameron Dawson [:camd] from comment #5)
> Wlach: for some reason your PR attachment just brings up the number of the
> bug for me.  I think it needs the PR link.  Thanks for doing this!

PR attachment fail. Fixed now, I hope. :)

Updated

2 years ago
Attachment #8665454 - Flags: review?(emorley) → review+

Comment 10

2 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b151ab45c70f9390e5b6d876887a5b3d05d2f843
Bug 1207816 - Bump default vagrant memory to 2G from 1G

It seems like 1G is too little for mysql as it is currently configured,
bumping the default memory on the Vagrant instance is an easy way of
addressing this.

https://github.com/mozilla/treeherder/commit/8444cc13cc57ee98a57bc9bc9bedf42151ee9982
Merge pull request #1003 from wlach/1207816

 	Bug 1207816 - Bump default vagrant memory to 2G from 1G
I'm pretty sure this should fix the issues we're seeing, feel free to reopen if that isn't the case.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.