1617430 - Create a beefier worker type for codeql database generation job

Status: RESOLVED INVALID

(Release Engineering :: Firefox-CI Administration, task)

Description

[Tom Ritter [:tjr]]

We made a new static analysis bounty, one of which is using CodeQL. To do this, the researcher must generate a database of Firefox which takes many hours (5-16 or something I think?) and requires a beefy machine.

I want to pre-generate the database for every release of Firefox. (We are also investigating using codeql ourselves as static-analysis, which might involve generating a database on every mozilla-central push, depending on how efficient we can make database generation.)

I would like to experiment with different instance types to figure out which is the most cost effective for time to generate. (i.e. should i take more time on a cheaper instance or less time on a more expensive instance.)

Right now though, I'm just trying to develop it into a job we can run. In https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=289990584&revision=4cbe215093455e5866010b4a51f62282ea8127a8 I ran out of memory, after adding 'use more memory plz Java' a parameter in https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=290030785&revision=da0611d5043704cb7825cfd501e4e181043e17db I ran out of disk space.

Could I have a 4xlarge instance (or just something with 16 cores and 32 GB of memory; or more) with 40 more GB of space than what is currently allocated to the t-linux-xlarge to try to get my job working? I just need one of these at a time. Maybe call it t-linux-4xlarge? (Or after whatever instance type you use...)

Also, what's the best way to go about testing the cost effectiveness of different worker types? I imagine just do a test generate on several of them and see how long it takes. I think there's a developer-playground AWS account we have for that sort of thing....

Comment 1

[Tom Ritter [:tjr]]

Good news, seems we don't need this.