Very frequent hangs during leak tests on new linux build system

RESOLVED FIXED

Status

Release Engineering
General Automation
--
major
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: philor, Assigned: jhopkins)

Tracking

(Depends on: 1 bug, {intermittent-failure})

other
All
Linux
intermittent-failure
Dependency tree / graph

Firefox Tracking Flags

(firefox17 fixed)

Details

(Whiteboard: [qa-])

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15386072&tree=Mozilla-Inbound
Linux x86-64 mozilla-inbound leak test build on 2012-09-20 10:50:41 PDT for push 5543e5e5bcd3
slave: bld-linux64-ec2-041

localhost.localdomain - - [20/Sep/2012 14:46:12] "GET /grid.html HTTP/1.1" 200 -
++DOMWINDOW == 14 (0x3b9ff90) [serial = 19] [outer = 0x46a88d0]

command timed out: 3600 seconds without output, attempting to kill
program finished with exit code 247

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387455&tree=Mozilla-Inbound
bld-linux64-ec2-010

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388348&tree=Mozilla-Inbound
bld-linux64-ec2-033

https://tbpl.mozilla.org/php/getParsedLog.php?id=15387848&tree=Mozilla-Inbound
bld-linux64-ec2-019

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389762&tree=Mozilla-Inbound
bld-linux64-ec2-021

https://tbpl.mozilla.org/php/getParsedLog.php?id=15389924&tree=Mozilla-Inbound
bld-linux64-ec2-045

https://tbpl.mozilla.org/php/getParsedLog.php?id=15388873&tree=Firefox
bld-linux64-ec2-056
(Reporter)

Comment 1

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15391409&tree=Firefox
bld-linux64-ec2-035
https://tbpl.mozilla.org/php/getParsedLog.php?id=15394022&tree=Firefox
s: bld-linux64-ec2-035
https://tbpl.mozilla.org/php/getParsedLog.php?id=15394069&tree=Firefox
s: bld-linux64-ec2-026
(Reporter)

Comment 4

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395290&tree=Fx-Team
bld-linux64-ec2-058
(Reporter)

Comment 5

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395794&tree=Mozilla-Inbound
bld-linux64-ec2-033
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395788&tree=Mozilla-Inbound
bld-linux64-ec2-031
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395760&tree=Mozilla-Inbound
bld-linux64-ec2-005
(Reporter)

Comment 6

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15395678&tree=Mozilla-Inbound
bld-linux64-ec2-032
Depends on: 793079
(Reporter)

Comment 7

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15398365&tree=Cedar
bld-linux64-ec2-045
(Reporter)

Comment 8

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15390311&tree=Cedar
bld-centos6-hp-017
cltbld    1221  1208  0 16:35 ?        00:00:00 Xvfb :2 -screen 0 1280x1024x24
cltbld    1700     1  0 16:36 ?        00:00:08 /tools/buildbot-0.8.4-pre-moz2/bin/python2.7 /tools/buildbot/bin/twistd --no_save --logfile /builds/slave/twistd.log --python /builds/slave/buildbot.tac
cltbld    7402     1  0 20:36 ?        00:00:00 /usr/libexec/gam_server
cltbld    7531  7505  0 20:37 ?        00:00:00 python leaktest.py -- --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7535  7531  0 20:37 ?        00:00:06 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log
cltbld    7536  7531  0 20:37 ?        00:00:00 /usr/bin/perl /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/fix-linux-stack.pl
cltbld    7606  7535  0 20:37 ?        00:00:00 /builds/slave/try-lnx-dbg/build/obj-firefox/dist/bin/firefox-bin -no-remote -profile /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log

minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp
No stack trace given in the log...

PROCESS-CRASH | Main app process exited normally | application crashed (minidump found)
Crash dump filename: /builds/slave/try-lnx-dbg/build/obj-firefox/_leaktest/leakprofile/minidumps/010ac078-817a-80ff-51dee7bf-6e49c685.dmp
No symbols path given, can't process dump.
(Reporter)

Comment 11

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15400937&tree=Mozilla-Inbound
bld-linux64-ec2-009
https://tbpl.mozilla.org/php/getParsedLog.php?id=15402549&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15402237&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15402830&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15403762&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15405558&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15406071&tree=Mozilla-Inbound

Updated

5 years ago
Summary: Very frequent hangs during leak tests on AWS → Very frequent hangs during leak tests on new linux build system
This is also happening on the HP machines.

Updated

5 years ago
Severity: normal → major
(Reporter)

Comment 14

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15412541&tree=Firefox
bld-linux64-ec2-058

Updated

5 years ago
Assignee: nobody → jhopkins
(Reporter)

Comment 15

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15409208&tree=Mozilla-Inbound
bld-centos6-hp-019
https://tbpl.mozilla.org/php/getParsedLog.php?id=15408761&tree=Mozilla-Inbound
bld-linux64-ec2-011
https://tbpl.mozilla.org/php/getParsedLog.php?id=15414483&tree=Mozilla-Inbound
bld-linux64-ec2-035
https://tbpl.mozilla.org/php/getParsedLog.php?id=15414214&tree=Mozilla-Inbound
bld-linux64-ec2-016
(Reporter)

Comment 16

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15414893&tree=Mozilla-Inbound
bld-linux64-ec2-024
(Reporter)

Comment 17

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15416637&tree=Mozilla-Inbound
bld-linux64-ec2-069
https://tbpl.mozilla.org/php/getParsedLog.php?id=15417461&tree=Mozilla-Inbound
bld-linux64-ec2-020
https://tbpl.mozilla.org/php/getParsedLog.php?id=15418406&tree=Mozilla-Inbound
bld-linux64-ec2-033
https://tbpl.mozilla.org/php/getParsedLog.php?id=15418167&tree=Mozilla-Inbound
bld-linux64-ec2-027
https://tbpl.mozilla.org/php/getParsedLog.php?id=15418594&tree=Mozilla-Inbound
bld-linux64-ec2-025
(Reporter)

Comment 18

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15418849&tree=Mozilla-Inbound
bld-linux64-ec2-068
https://tbpl.mozilla.org/php/getParsedLog.php?id=15419337&tree=Mozilla-Inbound
bld-linux64-ec2-070
https://tbpl.mozilla.org/php/getParsedLog.php?id=15420092&tree=Mozilla-Inbound
bld-linux64-ec2-035
https://tbpl.mozilla.org/php/getParsedLog.php?id=15420843&tree=Mozilla-Inbound
bld-linux64-ec2-051
https://tbpl.mozilla.org/php/getParsedLog.php?id=15420689&tree=Mozilla-Inbound
bld-linux64-ec2-026
https://tbpl.mozilla.org/php/getParsedLog.php?id=15429506&tree=Mozilla-Inbound
s: bld-linux64-ec2-014

https://tbpl.mozilla.org/php/getParsedLog.php?id=15428396&tree=Mozilla-Inbound
s: bld-linux64-ec2-063

https://tbpl.mozilla.org/php/getParsedLog.php?id=15426607&tree=Mozilla-Inbound
s: bld-linux64-ec2-031
(Reporter)

Comment 20

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15433875&tree=Mozilla-Inbound
bld-linux64-ec2-007
(In reply to Chris AtLee [:catlee] from comment #9)
> minidump is at http://people.mozilla.org/~catlee/debug-stack.dmp

The dump is not very useful without the symbol files for the corresponding build.
https://tbpl.mozilla.org/php/getParsedLog.php?id=15435250&tree=Mozilla-Inbound
s: bld-linux64-ec2-059

https://tbpl.mozilla.org/php/getParsedLog.php?id=15434763&tree=Mozilla-Inbound
s: bld-linux64-ec2-046
https://tbpl.mozilla.org/php/getParsedLog.php?id=15441080&tree=Firefox
s: bld-linux64-ec2-049
https://tbpl.mozilla.org/php/getParsedLog.php?id=15441766&tree=Firefox
s: bld-linux64-ec2-034
(Reporter)

Comment 25

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15443534&tree=Fx-Team
bld-linux64-ec2-036
(Reporter)

Comment 26

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15445629&tree=Fx-Team
bld-linux64-ec2-052
(Reporter)

Comment 27

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15446193&tree=Mozilla-Inbound
bld-linux64-ec2-004
https://tbpl.mozilla.org/php/getParsedLog.php?id=15446385&tree=Mozilla-Inbound
bld-linux64-ec2-061
(Reporter)

Comment 28

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15446553&tree=Mozilla-Inbound
bld-linux64-ec2-008
(Reporter)

Comment 29

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15449123&tree=Fx-Team
bld-linux64-ec2-057
https://tbpl.mozilla.org/php/getParsedLog.php?id=15449019&tree=Fx-Team
bld-linux64-ec2-013
(Reporter)

Comment 30

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15448893&tree=Mozilla-Inbound
bld-linux64-ec2-061
https://tbpl.mozilla.org/php/getParsedLog.php?id=15448818&tree=Mozilla-Inbound
bld-linux64-ec2-058
(Reporter)

Comment 31

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15450534&tree=Firefox
bld-centos6-hp-006
(Reporter)

Comment 32

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15451521&tree=Mozilla-Inbound
bld-centos6-hp-009
(Reporter)

Comment 33

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15452214&tree=Mozilla-Inbound
bld-linux64-ec2-048
possibly related?

--------------------------------------------------------------------------------
Exception in /builds/buildbot/try1/master/twistd.log:
2012-09-22 16:45:28-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
            self._runCallbacks()
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/builds/buildbot/try1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1072, in <lambda>
            d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 274, in createSummary
            if m:
          File "/builds/buildbot/try1/lib/python2.6/site-packages/buildbotcustom/steps/test.py", line 75, in formatBytes
            bytes=float(str(bytes) + '.0')
        exceptions.ValueError: invalid literal for float(): __codesizeDiff.0
https://tbpl.mozilla.org/php/getParsedLog.php?id=15453652&tree=Mozilla-Inbound
s: bld-linux64-ec2-049

https://tbpl.mozilla.org/php/getParsedLog.php?id=15456429&tree=Firefox
s: bld-linux64-ec2-034
(Reporter)

Comment 36

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15456564&tree=Firefox
bld-linux64-ec2-070
(Reporter)

Comment 37

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15458017&tree=Mozilla-Inbound
bld-linux64-ec2-051
(Reporter)

Comment 38

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15458944&tree=Firefox
bld-centos6-hp-006
(Reporter)

Comment 39

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459037&tree=Firefox
 bld-linux64-ec2-048
(Reporter)

Comment 40

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459523&tree=Mozilla-Inbound
bld-linux64-ec2-059
(Reporter)

Comment 41

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15459594&tree=Fx-Team
bld-linux64-ec2-058
(Reporter)

Comment 42

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15463026&tree=Mozilla-Inbound
bld-linux64-ec2-008
(Reporter)

Comment 43

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15463155&tree=Mozilla-Inbound
bld-linux64-ec2-059
(Reporter)

Comment 44

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15464433&tree=Firefox
bld-centos6-hp-006
(Reporter)

Comment 45

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15466015&tree=Mozilla-Inbound
bld-centos6-hp-019
(Reporter)

Comment 46

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15467119&tree=Mozilla-Inbound
bld-linux64-ec2-031
(Reporter)

Comment 47

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15468789&tree=Firefox
bld-linux64-ec2-006
(Reporter)

Comment 48

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15469079&tree=Firefox
bld-centos6-hp-009
(Reporter)

Comment 49

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15469866&tree=Mozilla-Inbound
bld-linux64-ec2-045
(Reporter)

Comment 50

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471395&tree=Mozilla-Inbound
bld-linux64-ec2-020
(Reporter)

Comment 51

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471459&tree=Mozilla-Inbound
bld-linux64-ec2-048
(Reporter)

Comment 52

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15471591&tree=Mozilla-Inbound
bld-linux64-ec2-022

Updated

5 years ago
Blocks: 793675
(Reporter)

Comment 53

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15478425&tree=Mozilla-Inbound
 bld-linux64-ec2-046
(Reporter)

Comment 54

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15473262&tree=Mozilla-Inbound
bld-linux64-ec2-020
(Reporter)

Comment 55

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15479030&tree=Firefox
bld-linux64-ec2-069
(Reporter)

Comment 56

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15483879&tree=Mozilla-Inbound
bld-linux64-ec2-063

Updated

5 years ago
Depends on: 793782
(Reporter)

Comment 57

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15487266&tree=Mozilla-Inbound
bld-linux64-ec2-046
(Reporter)

Comment 58

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15488914&tree=Mozilla-Inbound
bld-linux64-ec2-064
(Reporter)

Comment 59

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15489605&tree=Mozilla-Inbound
bld-linux64-ec2-038
(Reporter)

Comment 60

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15490382&tree=Mozilla-Inbound
bld-linux64-ec2-019
(Reporter)

Comment 61

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15490966&tree=Mozilla-Inbound
bld-linux64-ec2-020
https://tbpl.mozilla.org/php/getParsedLog.php?id=15491199&tree=Mozilla-Inbound
bld-linux64-ec2-017
(Reporter)

Comment 62

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15491799&tree=Mozilla-Inbound
bld-linux64-ec2-031
(Reporter)

Comment 63

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15493559&tree=Mozilla-Inbound
bld-linux64-ec2-062
(Reporter)

Comment 64

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15494473&tree=Mozilla-Inbound
bld-linux64-ec2-021
(Reporter)

Comment 65

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15495269&tree=Mozilla-Inbound
bld-linux64-ec2-019
(Reporter)

Comment 66

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15500612&tree=Mozilla-Inbound
bld-linux64-ec2-019
(Reporter)

Comment 67

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15502565&tree=Mozilla-Inbound
bld-linux64-ec2-026
(Reporter)

Comment 68

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15504933&tree=Mozilla-Inbound
bld-linux64-ec2-031
https://tbpl.mozilla.org/php/getParsedLog.php?id=15504831&tree=Mozilla-Inbound
bld-linux64-ec2-028
https://tbpl.mozilla.org/php/getParsedLog.php?id=15505345&tree=Mozilla-Inbound
bld-linux64-ec2-002
https://tbpl.mozilla.org/php/getParsedLog.php?id=15504593&tree=Mozilla-Inbound
bld-linux64-ec2-035
(Reporter)

Comment 69

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15505285&tree=Fx-Team
bld-centos6-hp-009
https://tbpl.mozilla.org/php/getParsedLog.php?id=15512600&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15511527&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15511671&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15511558&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15511558&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15508500&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15508501&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15514205&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15516137&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=15517124&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15517566&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15517269&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15519510&tree=Mozilla-Inbound

Updated

5 years ago
Depends on: 794174
(Assignee)

Comment 75

5 years ago
Created attachment 664586 [details] [diff] [review]
proposed mozilla-central workaround suggested by :ted
Attachment #664586 - Flags: review?(ted.mielczarek)
Attachment #664586 - Flags: review?(catlee)

Updated

5 years ago
Attachment #664586 - Flags: review?(catlee) → review+
(Reporter)

Comment 76

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15520501&tree=Mozilla-Inbound
Attachment #664586 - Flags: review?(ted.mielczarek) → review+
(Assignee)

Comment 77

5 years ago
Comment on attachment 664586 [details] [diff] [review]
proposed mozilla-central workaround suggested by :ted

Landed in http://hg.mozilla.org/mozilla-central/rev/aa3266ba9cab
Attachment #664586 - Flags: checked-in+
Looks like this is fixed now!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 79

5 years ago
Hi, I'm back.

https://tbpl.mozilla.org/php/getParsedLog.php?id=15844614&tree=Mozilla-Aurora
status-firefox17: --- → affected
(Reporter)

Comment 80

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15852637&tree=Mozilla-Aurora
(Reporter)

Comment 81

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=15863875&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=15862929&tree=Mozilla-Aurora

and fixed-17 in https://hg.mozilla.org/releases/mozilla-aurora/rev/dda7c25ecc38
status-firefox17: affected → fixed
Whiteboard: [orange] → [orange][qa-]

Updated

5 years ago
Depends on: 813039
Keywords: intermittent-failure
Whiteboard: [orange][qa-] → [qa-]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.