Closed Bug 1036675 Opened 11 years ago Closed 11 years ago

fxOS: Add 319MB config for 2.0 Flame Performance Automation Tests

Categories

(Firefox OS Graveyard :: Infrastructure, defect, P1)

Product:
Firefox OS Graveyard
Component:
Infrastructure
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: dwong)

References

Details

(Keywords: perf, Whiteboard: [c=automation p= s= u=2.0])

We should use a 273MB config for 2.0 (mozilla-aurora) perf runs on Flame devices; we're currently running with the full-memory configuration.
Bug 1036670 was just filed about issues booting with that memory config.
Depends on: 1036670
Ouch, guess we'll need to wait until that's fixed.
From mlee: we want 2.0 runs in both configurations, 273MB and full memory.
Keywords: perf
Priority: -- → P1
Whiteboard: [c=automation p= s= u=]
Summary: Move 2.0 Flame perf tests to 273MB config → Add 273MB config for 2.0 Flame perf tests
We upgraded b2g-14 to v22 with fonts and used the font flashing script from https://bugzilla.mozilla.org/show_bug.cgi?id=1032874 to add the font. The plan is to update all the perf phones to v22 with fonts tomorrow. If https://bugzilla.mozilla.org/show_bug.cgi?id=1036670 is solved by tomorrow. We'll add that as well and update this bug after we are done.
(In reply to raymond [:retornam] (needinfo? me) from comment #4) > We upgraded b2g-14 to v22 with fonts and used the font flashing script from > https://bugzilla.mozilla.org/show_bug.cgi?id=1032874 to add the font. The > plan is to update all the perf phones to v22 with fonts tomorrow. If > https://bugzilla.mozilla.org/show_bug.cgi?id=1036670 is solved by tomorrow. > We'll add that as well and update this bug after we are done. We have upgraded all the nodes to v122 with fonts.We are waiting on https://bugzilla.mozilla.org/show_bug.cgi?id=1032874 to be resolved so we can switch the phones to using 273MB.
I think you'll be blocked by bug 1036670 for automation.
(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #6) > I think you'll be blocked by bug 1036670 for automation. Hence the dependency :-)
Can we move this forward now that bug 1036670 has been fixed? Please remember that we want to add 273MB Flame configs not replace the existing 1GB Flame ones. Thanks!
Flags: needinfo?(mozbugs.retornam)
I'm on PTO today. Dylan can you handle this? If not this will have to wait till I get back on Tuesday
Flags: needinfo?(mozbugs.retornam) → needinfo?(dwong)
Yes, so with the new minis, and current flames add perf nodes that have throttled memory? We currently have 3 sims available.
Flags: needinfo?(dwong)
I don't think we'd need to add more nodes (although that might be a good idea), we would need new Jenkins jobs that build with the suitable memory limitation. We also need to ensure that we report to datazilla appropriately. I would suggest making the device name 'flame-273MB'.
So I will copy current perf jobs, with the device entered into datazilla as 'flame-273MB' with the scripts adjusted to include the throttling. I'm going to test locally a bit first to see in what cases the throttled memory goes back to normal and whether it'll stay throttled after testing.
I have created new jobs on the new jenkins wherein it will throttle memory before running tests and reset them back to auto at test conclusion.
It's actually having some trouble running so I've disabled the projects. Sorry for the spam.
(In reply to Dylan Chun Wong [:dwong] from comment #13) > I have created new jobs on the new jenkins wherein it will throttle memory > before running tests and reset them back to auto at test conclusion. If the Jenkins build fails then it's like the reset will not run. We should therefore set the memory allocation at the start of every job. (In reply to Dylan Chun Wong [:dwong] from comment #14) > It's actually having some trouble running so I've disabled the projects. > Sorry for the spam. What issues are you seeing? I've implemented this today for the Eideticker tests.
I'll add the resets to the other perf jobs and move them to the start of the job so that I don't need a wait as suggested. The issue was mostly not waiting long enough for the device to be recognized on adb again, but this should fix it.
The new jobs did not trigger when a new build was available. I have fixed this now.
Mike: Should we also make these 319MB now, based on bug 1041984? Also, it's been suggested that we should only test on the restricted memory configurations. Once we have some successful builds on the limited memory configurations, can we disable/delete the unrestricted ones? This will also free up devices and allow us to keep up with b2g-inbound builds and Gaia commits.
Flags: needinfo?(mlee)
Yes we should make them 319MB per the 273MB build stability issues reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1041984#c2 Re: disabling or deleting the unrestricted configs, we should keep a 512MB config as a control. 1GB is really not very useful for assessing performance as it puts no real strain on the system.
Flags: needinfo?(mlee)
Whiteboard: [c=automation p= s= u=] → [c=automation p= s= u=2.0]
For the control 512MB config, let's run them less frequently than per-commit; we don't have the bandwidth to run both configs on all checkins.
I can update the name and memory configurations of the jobs to 319 from 273 as soon as work week events are over.
Will updating the names be automatically picked up by Datazilla or is that another thing to be done?
I'm not too sure. For the 273 jobs when initially added I simply changed the device name that is sent to datazilla from Flame to Flame-273MB. Could someone else confirm if anything else needs to be done?
All 273MB jobs have been updated to 319MB and the new device name submitted is now 'flame-319MB'. Two of the job names still have 273 due to the fact that you cannot rename a job while it is being built.
Thanks Dylan. (In reply to Dylan Chun Wong [:dwong] from comment #23) > I'm not too sure. For the 273 jobs when initially added I simply changed the > device name that is sent to datazilla from Flame to Flame-273MB. Could > someone else confirm if anything else needs to be done? @jeads, can you confirm if changes need to be made to Datazilla to reflect the new names and if so can you make those changes? Thanks, Mike
Assignee: nobody → dwong
Status: NEW → ASSIGNED
Flags: needinfo?(jeads)
Summary: Add 273MB config for 2.0 Flame perf tests → fxOS: Add 319MB config for 2.0 Flame Performance Automation Tests
No changes are needed to datazilla to display results from a new device name, however I do see an issue where history is shown for flame-319MB from before we were submitting results for it. This could be due to the mac address matching results for a different device name. I'll raise a bug for this. The other thing you might want to do is to set the default device to be flame-319MB. I'll also raise a bug for this and needinfo to determine if this is something we want to do. Finally, I will raise another bug to take care of switching the auto memory results to 512MB, but on a less frequent schedule.
Flags: needinfo?(jeads)
Mike: I notice the summary change to explicitly mention 2.0, does this not apply to 2.1 (current master)?
Flags: needinfo?(mlee)
(In reply to Dave Hunt (:davehunt) from comment #26) > No changes are needed to datazilla to display results from a new device > name, however I do see an issue where history is shown for flame-319MB from > before we were submitting results for it. This could be due to the mac > address matching results for a different device name. I'll raise a bug for > this. Bug 1046056 > The other thing you might want to do is to set the default device to be > flame-319MB. I'll also raise a bug for this and needinfo to determine if > this is something we want to do. Bug 1046057 > Finally, I will raise another bug to take care of switching the auto memory > results to 512MB, but on a less frequent schedule. Bug 1046060
Hi Dave, We want 319 MB Flame configurations for 2.0 and all subsequent releases for which Flame will be the reference device.
Flags: needinfo?(mlee)
Component: WebQA → Infrastructure
Product: Testing → Firefox OS
This was done. I'll raise a separate issue for potentially removing the 512MB configurations.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.