If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

[Tracking Bug] Downtime for March 11th



Release Engineering
8 years ago
4 years ago


(Reporter: catlee, Assigned: catlee)


Firefox Tracking Flags

(Not tracked)




8 years ago
Planned events:

* Land bug 550876
* Restart pm02
* Add-ons branch (bug 542910)
* reconfig of Talos master to enable addonsmgr branch

Comment 2

8 years ago
* reset Try
Assignee: armenzg → catlee
Depends on: 550256

Comment 3

8 years ago
This downtime has commenced.

Comment 4

8 years ago
Memory usage for buildbot before restart is: 59%

Started with:
 nohup ./start_buildbot.sh &

Pruned events with:
  python ~/tools/buildfarm/maintenance/purge_events.py

Memory usage after restart is: 60% while pm01 is 22%.

In my opinion, it shouldn't be such a big difference.

We have discovered that changes.pck is huge on pm02, 200MB, while on pm01 is only 1.4MB.

After removing it and start it again pm02's memory usage stayed at around 45%.

The differences of these masters are the following:
* 254 builders vs 351 builders
* 3 branches vs 5 branches + 6 mobile branches
* 2 HgAllLocalesPoller vs 3 HgAllLocalesPoller (currently they are even - mobile repacks-on-change are disabled)
* pm02 has three extra pollers (mobile, mc and m192) beside the poller differences added on misc.py per branch

I have noticed that staging-master02 also had a really high memory usage (>50%) and after restarting and pruning it is at 44% (almost like production-master02). It seems like the difference of slaves is irrelevant.

Comment 5

8 years ago
The only thing left in here is restarting the try repo. We are waiting on IT.

Comment 6

8 years ago
All done here.  Try server will be done at a later time.
Last Resolved: 8 years ago
No longer depends on: 550256
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.