1423055 - disable some Buildbot testing so we can enable the full suite of tests on project-branches

Reporter

Description

•

5 years ago

from callek:

context: Would be awesome if we could disable a lot of BB testing so we can enable the full suite of tests on project-branches (e.g. graphics) so we can actually cause the decision task to fail if a BBB task is attempted when no BB job is configured.

Motive: We hit “too many builders” when I tried to enable the full set of stuff on graphics/stuff, so we had to disable that error.

Alin Selagea [:aselagea]

Assignee

Comment 1

•

5 years ago

Attached file graphics_tests.txt — Details

We can help with this, but would need some additional info. 

These would be the current graphics tests present on the the bb-masters.

As for the builds, we have only two of them:
WINNT 5.2 graphics leak test build SigningScriptFactory
WINNT 6.1 x86-64 graphics leak test build SigningScriptFactory

I'd have two questions:
  1. which of the above builds/tests would you like to remove?
  2. does 1 apply for the other project branches as well?

Flags: needinfo?(bugspam.Callek)

Justin Wood (:Callek)

Comment 2

•

5 years ago

In particular this issue is that there are too many test builders configured in buildbot in general.

For example the suites defined here: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#559

we disable some here, and do similar stuff around there: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#3107

you can get the list of jobs that roughly should be supported on a branch with a gecko checkout on that branch using `./mach taskgraph full --json > /tmp/jobs.json` and then using gron or some other inspection tool to find the buildbot-bridge jobs.

From my current local copy of m-c, this is what I see that still needs to be defined, but many of the tests in the configs can go.

We likely need MOST of those tests for at least mozilla-esr52, but I bet 57 and newer can drop a large portion.

If you are confused or need more info, lets meet via vidyo tomorrow morning.

callek@Centaurus:~/mozilla/hg/mozilla-central$ for job in `grep workerType /tmp/jobs_test1.json | grep buildbot-bridge | sed 's/^json\[\"\(.*\)\"\].*/\1/' | sort -u`; do grep $job /tmp/jobs_test1.json ; done | grep task.payload.buildername | sed 's/^.*\= \"\(.*\)\".*/\1/' | sort -u
Linux x86-64 mozilla-central periodic file update
release-mozilla-central-fennec_bncr_sub
release-mozilla-central-fennec_bouncer_aliases
release-mozilla-central-fennec_mark_as_shipped
release-mozilla-central-fennec_uptake_monitoring
release-mozilla-central-fennec_version_bump
Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g2-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g2-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g3-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g3-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g4-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g4-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g5-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos g5-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos h1-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos h2-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos other-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos other-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-singletons-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-singletons-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos perf-reftest-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos speedometer-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos speedometer-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central pgo talos tp6-stylo-threads-e10s
Ubuntu HW 12.04 x64 mozilla-central talos chromez-e10s
Ubuntu HW 12.04 x64 mozilla-central talos chromez-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs-e10s
Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g1-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g1-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g2-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g2-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g3-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g3-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g4-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g4-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g5-e10s
Ubuntu HW 12.04 x64 mozilla-central talos g5-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos h1-e10s
Ubuntu HW 12.04 x64 mozilla-central talos h2-e10s
Ubuntu HW 12.04 x64 mozilla-central talos other-e10s
Ubuntu HW 12.04 x64 mozilla-central talos other-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-e10s
Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-singletons-e10s
Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-singletons-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos perf-reftest-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos speedometer-e10s
Ubuntu HW 12.04 x64 mozilla-central talos speedometer-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos svgr-e10s
Ubuntu HW 12.04 x64 mozilla-central talos svgr-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos tp5o-e10s
Ubuntu HW 12.04 x64 mozilla-central talos tp5o-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos tp6-e10s
Ubuntu HW 12.04 x64 mozilla-central talos tp6-stylo-disabled-e10s
Ubuntu HW 12.04 x64 mozilla-central talos tp6-stylo-threads-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos chromez-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos dromaeojs-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos g1-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos g3-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos g4-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos g5-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos h1-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos h2-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos other-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos perf-reftest-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos perf-reftest-singletons-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos speedometer-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos svgr-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos tp5o-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos tp6-e10s
Ubuntu HW 12.04 x64 qr mozilla-central talos tp6-stylo-threads-e10s
Windows 10 64-bit Code Coverage mozilla-central debug test reftest-e10s-1
Windows 10 64-bit Code Coverage mozilla-central debug test reftest-e10s-2
Windows 10 64-bit Code Coverage mozilla-central debug test reftest-no-accel-e10s-1
Windows 10 64-bit Code Coverage mozilla-central debug test reftest-no-accel-e10s-2
Windows 10 64-bit DevEdition mozilla-central opt test reftest-e10s
Windows 10 64-bit DevEdition mozilla-central opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-central debug test reftest-1
Windows 10 64-bit mozilla-central debug test reftest-2
Windows 10 64-bit mozilla-central debug test reftest-e10s-1
Windows 10 64-bit mozilla-central debug test reftest-e10s-2
Windows 10 64-bit mozilla-central debug test reftest-no-accel-1
Windows 10 64-bit mozilla-central debug test reftest-no-accel-2
Windows 10 64-bit mozilla-central debug test reftest-no-accel-e10s-1
Windows 10 64-bit mozilla-central debug test reftest-no-accel-e10s-2
Windows 10 64-bit mozilla-central opt test reftest-e10s
Windows 10 64-bit mozilla-central opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-central pgo talos chromez-e10s
Windows 10 64-bit mozilla-central pgo talos dromaeojs-e10s
Windows 10 64-bit mozilla-central pgo talos g1-e10s
Windows 10 64-bit mozilla-central pgo talos g2-e10s
Windows 10 64-bit mozilla-central pgo talos g4-e10s
Windows 10 64-bit mozilla-central pgo talos g5-e10s
Windows 10 64-bit mozilla-central pgo talos h1-e10s
Windows 10 64-bit mozilla-central pgo talos h2-e10s
Windows 10 64-bit mozilla-central pgo talos other-e10s
Windows 10 64-bit mozilla-central pgo talos perf-reftest-e10s
Windows 10 64-bit mozilla-central pgo talos perf-reftest-singletons-e10s
Windows 10 64-bit mozilla-central pgo talos speedometer-e10s
Windows 10 64-bit mozilla-central pgo talos svgr-e10s
Windows 10 64-bit mozilla-central pgo talos tp5o-e10s
Windows 10 64-bit mozilla-central pgo talos tp6-e10s
Windows 10 64-bit mozilla-central pgo talos tp6-stylo-threads-e10s
Windows 10 64-bit mozilla-central pgo talos xperf-e10s
Windows 10 64-bit mozilla-central pgo test reftest-e10s
Windows 10 64-bit mozilla-central pgo test reftest-no-accel-e10s
Windows 10 64-bit mozilla-central talos chromez-e10s
Windows 10 64-bit mozilla-central talos dromaeojs-e10s
Windows 10 64-bit mozilla-central talos g1-e10s
Windows 10 64-bit mozilla-central talos g2-e10s
Windows 10 64-bit mozilla-central talos g4-e10s
Windows 10 64-bit mozilla-central talos g5-e10s
Windows 10 64-bit mozilla-central talos h1-e10s
Windows 10 64-bit mozilla-central talos h2-e10s
Windows 10 64-bit mozilla-central talos other-e10s
Windows 10 64-bit mozilla-central talos perf-reftest-e10s
Windows 10 64-bit mozilla-central talos perf-reftest-singletons-e10s
Windows 10 64-bit mozilla-central talos speedometer-e10s
Windows 10 64-bit mozilla-central talos svgr-e10s
Windows 10 64-bit mozilla-central talos tp5o-e10s
Windows 10 64-bit mozilla-central talos tp6-e10s
Windows 10 64-bit mozilla-central talos tp6-stylo-threads-e10s
Windows 10 64-bit mozilla-central talos xperf-e10s
Windows 7 32-bit mozilla-central pgo talos chromez-e10s
Windows 7 32-bit mozilla-central pgo talos dromaeojs-e10s
Windows 7 32-bit mozilla-central pgo talos g1-e10s
Windows 7 32-bit mozilla-central pgo talos g2-e10s
Windows 7 32-bit mozilla-central pgo talos g4-e10s
Windows 7 32-bit mozilla-central pgo talos g5-e10s
Windows 7 32-bit mozilla-central pgo talos h1-e10s
Windows 7 32-bit mozilla-central pgo talos h2-e10s
Windows 7 32-bit mozilla-central pgo talos other-e10s
Windows 7 32-bit mozilla-central pgo talos perf-reftest-e10s
Windows 7 32-bit mozilla-central pgo talos perf-reftest-singletons-e10s
Windows 7 32-bit mozilla-central pgo talos speedometer-e10s
Windows 7 32-bit mozilla-central pgo talos svgr-e10s
Windows 7 32-bit mozilla-central pgo talos tp5o-e10s
Windows 7 32-bit mozilla-central pgo talos tp6-e10s
Windows 7 32-bit mozilla-central pgo talos tp6-stylo-threads-e10s
Windows 7 32-bit mozilla-central pgo talos xperf-e10s
Windows 7 32-bit mozilla-central talos chromez-e10s
Windows 7 32-bit mozilla-central talos dromaeojs-e10s
Windows 7 32-bit mozilla-central talos g1-e10s
Windows 7 32-bit mozilla-central talos g2-e10s
Windows 7 32-bit mozilla-central talos g4-e10s
Windows 7 32-bit mozilla-central talos g5-e10s
Windows 7 32-bit mozilla-central talos h1-e10s
Windows 7 32-bit mozilla-central talos h2-e10s
Windows 7 32-bit mozilla-central talos other-e10s
Windows 7 32-bit mozilla-central talos perf-reftest-e10s
Windows 7 32-bit mozilla-central talos perf-reftest-singletons-e10s
Windows 7 32-bit mozilla-central talos speedometer-e10s
Windows 7 32-bit mozilla-central talos svgr-e10s
Windows 7 32-bit mozilla-central talos tp5o-e10s
Windows 7 32-bit mozilla-central talos tp6-e10s
Windows 7 32-bit mozilla-central talos tp6-stylo-threads-e10s
Windows 7 32-bit mozilla-central talos xperf-e10s

Flags: needinfo?(bugspam.Callek)

Alin Selagea [:aselagea]

Assignee

Comment 3

•

5 years ago

Thanks Callek for the hint!

So I did a similar thing for a local copy of m-i:
e.g. "./mach taskgraph full --json > m-i.json"

Then I used a little Python snippet to search for the jobs that run via BBB:
"
import json

buildernames = []
data = json.load(open('m-i.json'))
for key in data.keys():
    if data[key]['task']['workerType']== 'buildbot-bridge':
        buildernames.append(data[key]['task']['payload']['buildername'])

for build in sorted(buildernames):
    print build
"

I then copied the resulted entries to a local file. As far as I know, all the talos jobs run via BBB so we shouldn't touch them. Besides talos, we still have the following jobs running via BBB that we'll need to keep:
$ cat m-i.json | grep -v "talos"
Linux x86-64 mozilla-inbound periodic file update
release-mozilla-inbound-fennec_bncr_sub
release-mozilla-inbound-fennec_bouncer_aliases
release-mozilla-inbound-fennec_mark_as_shipped
release-mozilla-inbound-fennec_uptake_monitoring
release-mozilla-inbound-fennec_version_bump
Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-e10s-1
Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-e10s-2
Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-no-accel-e10s-1
Windows 10 64-bit Code Coverage mozilla-inbound debug test reftest-no-accel-e10s-2
Windows 10 64-bit DevEdition mozilla-inbound opt test reftest-e10s
Windows 10 64-bit DevEdition mozilla-inbound opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-inbound debug test reftest-1
Windows 10 64-bit mozilla-inbound debug test reftest-2
Windows 10 64-bit mozilla-inbound debug test reftest-e10s-1
Windows 10 64-bit mozilla-inbound debug test reftest-e10s-2
Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-1
Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-2
Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-e10s-1
Windows 10 64-bit mozilla-inbound debug test reftest-no-accel-e10s-2
Windows 10 64-bit mozilla-inbound opt test reftest-e10s
Windows 10 64-bit mozilla-inbound opt test reftest-e10s
Windows 10 64-bit mozilla-inbound opt test reftest-e10s
Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-inbound opt test reftest-no-accel-e10s
Windows 10 64-bit mozilla-inbound pgo test reftest-e10s
Windows 10 64-bit mozilla-inbound pgo test reftest-no-accel-e10s

Alin Selagea [:aselagea]

Assignee

Comment 4

•

5 years ago

Attached file bb_jobs.txt — Details

I then checked Buildbot DB for the jobs that ran during the past month:
e.g 'select buildername from buildrequests where submitted_at>1510012800 and complete=1;'

Dumped the output into a file, then removed the following:
  - duplicate entries
  - talos jobs - already mentioned above why we still need them
  - ESR52 jobs - most of them run in Buildbot, so I think we shouldn't touch them for now
  - Try jobs - :kmoir worked in bug 1384706 to stop invoking buildbot jobs by default, yet we should still have a way of testing ESR/Release patches (available via "--buildot" in the TryChooser synthax), so I'd say we shouldn't touch Try as well
  - Thunderbird jobs
e.g. 'cat bb_jobs.txt | sort -u | grep -Ev "talos"\|"esr52"\|"try"\|"comm"'

The remaining jobs can be seen in this attachment.

Based on that, I *think* we can do the following:
1. keep the talos jobs as they are now
2. don't touch ESR52 and Try
3. for Fx >= 57, drop *all* the tests except:
    - reftest
    - reftest-e10s
    - reftest-no-accel
    - reftest-no-accel-e10s
    - mochitest-clipboard
    - mochitest-clipboard-e10s
    - mochitest-media-e10s
4. leave the builders as they are
    - noticed that ReleaseDuty still uses some project branches (e.g. jamun) for repacks, update verify builds, etc.

Looping in :nthomas and :jmaher here so that we can have a clear idea of what we still want to keep before proceeding with removing these BB jobs.

Attachment #8935439 - Flags: feedback?(nthomas)

Attachment #8935439 - Flags: feedback?(jmaher)

Attachment #8935439 - Flags: feedback?(bugspam.Callek)

Joel Maher ( :jmaher ) (UTC -8)

Comment 5

•

5 years ago

thanks for the heads up, this looks reasonable for now, hopefully in due time we can get that list reduced.

Justin Wood (:Callek)

Comment 6

•

5 years ago

(In reply to Alin Selagea [:aselagea][buildduty] from comment #4)
> Based on that, I *think* we can do the following:
> 1. keep the talos jobs as they are now
> 2. don't touch ESR52 and Try
> 3. for Fx >= 57, drop *all* the tests except:
>     - reftest
>     - reftest-e10s
>     - reftest-no-accel
>     - reftest-no-accel-e10s
>     - mochitest-clipboard
>     - mochitest-clipboard-e10s
>     - mochitest-media-e10s
> 4. leave the builders as they are
>     - noticed that ReleaseDuty still uses some project branches (e.g. jamun)
> for repacks, update verify builds, etc.

That plan sounds good to me as well.

Nick Thomas [:nthomas] (UTC+12)

Comment 7

•

5 years ago

Your inspection methods seem fine to me. I defer to jmaher and Callek for details on the tests.

Nick Thomas [:nthomas] (UTC+12)

Updated

•

5 years ago

Attachment #8935439 - Flags: feedback?(nthomas) → feedback+

Joel Maher ( :jmaher ) (UTC -8)

Updated

•

5 years ago

Attachment #8935439 - Flags: feedback?(jmaher) → feedback+

Alin Selagea [:aselagea]

Assignee

Comment 8

•

5 years ago

Attached patch bug1423055.patch — Details — Splinter Review

Attachment #8935772 - Flags: review?(bugspam.Callek)

Alin Selagea [:aselagea]

Assignee

Comment 9

•

5 years ago

Attached file diff.txt — Details

Test diff.

(test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat old_sorted | wc -l
13650
(test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat new_sorted | wc -l
5072

That means 8578 tests were dropped.

Alin Selagea [:aselagea]

Assignee

Comment 10

•

5 years ago

Attached file remaining_tests.txt — Details

And here are the remaining tests (excluding talos, esr52 and try).

(test_Q4)[aselagea@dev-master2.bb.releng.use1.mozilla.com test_Q4]$ cat new_sorted | grep -Ev talos\|esr52\|try | wc -l
721

I also ran 'test-masters.sh' script to confirm the patch is working fine on other masters, all tests passed.

Alin Selagea [:aselagea]

Assignee

Updated

•

5 years ago

Assignee: nobody → aselagea

Justin Wood (:Callek)

Comment 11

•

5 years ago

Comment on attachment 8935772 [details] [diff] [review]
bug1423055.patch

Review of attachment 8935772 [details] [diff] [review]:
-----------------------------------------------------------------

Lets do it, I think this is good based on joel's feedback earlier.

Attachment #8935772 - Flags: review?(bugspam.Callek) → review+

Alin Selagea [:aselagea]

Assignee

Comment 12

•

5 years ago

Comment on attachment 8935772 [details] [diff] [review]
bug1423055.patch

https://hg.mozilla.org/build/buildbot-configs/rev/1eca1f3f8393
https://hg.mozilla.org/build/buildbot-configs/rev/fd2fb97fc031

Attachment #8935772 - Flags: checked-in+

Alin Selagea [:aselagea]

Assignee

Updated

•

5 years ago

Status: NEW → RESOLVED

Closed: 5 years ago

Resolution: --- → FIXED

Justin Wood (:Callek)

Updated

•

5 years ago

Attachment #8935439 - Flags: feedback?(bugspam.Callek)

Nobody; OK to take it and work on it

Updated

•

5 years ago

Component: General Automation → General

graphics_tests.txt 5 years ago Alin Selagea [:aselagea] 55.22 KB, text/plain		Details
bb_jobs.txt 5 years ago Alin Selagea [:aselagea] 15.92 KB, text/plain	nthomas : feedback+ jmaher : feedback+	Details
bug1423055.patch 5 years ago Alin Selagea [:aselagea] 2.11 KB, patch	Callek : review+ aselagea : checked-in+	Details \| Diff \| Splinter Review
diff.txt 5 years ago Alin Selagea [:aselagea] 660.69 KB, text/plain		Details
remaining_tests.txt 5 years ago Alin Selagea [:aselagea] 51.10 KB, text/plain		Details

Bugzilla

disable some Buildbot testing so we can enable the full suite of tests on project-branches

Categories

(Release Engineering :: General, enhancement)

Tracking

(Not tracked)

People

(Reporter: jlund, Assigned: aselagea, Mentored)

References

Details

Crash Data

Security

(public)

User Story

Attachments

(5 files)

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Updated

Updated

Comment 8

Comment 9

Comment 10

Updated

Comment 11

Comment 12

Updated

Updated

Updated