Closed Bug 1213325 Opened 6 years ago Closed 5 years ago

refactor desktop tests to run more like build-linux.sh

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

Attachments

(1 file)

For Linux builds, we have a shell script to wrap mozharness and convert environment variables to mozharness options, set up the environment, etc.

This has the advantage of cleanliness -- that script is run from in-tree, so it's very easy to edit it as needed.  It gets us away from putting custom mozharness configs in docker images, or requiring long command-lines in our task descriptions.

For parity, if for nothing else, we should do something similar for tests.
Blocks: 1212881
I'm almost done -- just having pulseaudio problems while running locally.  Potentially related:
  http://stackoverflow.com/questions/27637465/record-local-audio-in-a-docker-container
Bug 1213325: refactor desktop-test to use an in-tree script; r?ahal

This generally makes the approach look more like that for desktop-build.
Attachment #8673239 - Flags: review?(ahalberstadt)
So this patch includes everything except the pulseaudio bits, since those are still causing crashes.

Here's a task run with this setup:
  https://tools.taskcluster.net/task-inspector/#cHP7S0_PT5Wqv__HAtA--g/0

I'm not sure if that got far enough??
The existing tester script does this:

----
Xvfb :0 -nolisten tcp -screen 0 1600x1200x24 2>/dev/null &
export DISPLAY=:0

pulseaudio --start
----

which means that pulseaudio starts up before the X server is up and running, and thus never really talks to it.  That is probably why it doesn't crash.
Starting pulseaudio before Xvfb doesn't seem to help:

https://tools.taskcluster.net/task-inspector/#WXqXcu5GRKOEWkAE6YEbKw/1
----
# start up the pulseaudio daemon.  Note that it's important this occur
# before the Xvfb startup.
if $NEED_PULSEAUDIO; then
    mkdir -p /home/worker/artifacts/logs
    pulseaudio --start --daemonize \
        --disallow-exit -vvv \
        --log-target=file:/home/worker/artifacts/logs/pulseaudio.log
fi
+ true
+ mkdir -p /home/worker/artifacts/logs
+ pulseaudio --start --daemonize --disallow-exit -vvv --log-target=file:/home/worker/artifacts/logs/pulseaudio.log
[taskcluster] === Task Finished ===
[taskcluster] Successful task run with exit code: 0 completed in 119.899 seconds
----

To be clear there's more to the script after that `pulseaudio` invocation.  But rather than do that stuff, the task just finishes successfully at that point, or in my experience maybe a few ms later.  pulseaudio.log doesn't have much of interest, finishing with

----
I  [pulseaudio] main.c: Daemon startup successful.
----
Comment on attachment 8673239 [details]
MozReview Request: Bug 1213325: refactor desktop-test to use an in-tree script; r?ahal

https://reviewboard.mozilla.org/r/21867/#review19651

::: testing/taskcluster/scripts/tester/test-linux.sh:8
(Diff revision 1)
> +# Taskcluster friendly wrapper for performing fx desktop builds via mozharness.

nit: desktop tests

::: testing/taskcluster/scripts/tester/test-linux.sh:80
(Diff revision 1)
> +# Add some mozharness config tweaks.  TODO: once tests are no longer running
> +# side-by-site with Buildbot, get rid of this
> +cat <<'EOF' > remove_executables.py
> +config = {
> +    # We bake this directly into the tester image now...
> +    "download_minidump_stackwalk": False,
> +    "minidump_stackwalk_path": "/usr/local/bin/linux64-minidump_stackwalk",
> +    "exes": {}
> +}
> +EOF
> +config_cmds="${config_cmds} --config-file $PWD/remove_executables.py"

Couldn't we check in a taskcluster specific config into mozharness rather than baking this into the image?

::: testing/taskcluster/scripts/tester/test-linux.sh:92
(Diff revision 1)
> +# Mozharness would ordinarily do the checkouts itself, but they are disabled
> +# here (--no-checkout-sources, --no-clone-tools) as the checkout is performed above.
> +
> +python2.7 $WORKSPACE/${MOZHARNESS_SCRIPT} ${config_cmds} \
> +  --no-read-buildbot-config \
> +  --installer-url $INSTALLER_URL \
> +  --test-packages-url $TEST_PACKAGES_URL \
> +  --download-symbols ondemand \
> +  "${@}"

This isn't going to play well with my patch and landing it will almost certainly break mulet.

I sort of prefer if the mozharness invocation was left in the yaml files. E.g https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/mulet_mochitests.yml#12

(in my patch I have this invocation factored out into a base.yml file)
Attachment #8673239 - Flags: review?(ahalberstadt)
(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> I'm not sure if that got far enough??

You should see tests actually running (and in most cases, passing), see logs for:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6210c226e2e0

Doesn't look like the test harness ever got invoked in that log.
Running the task interactively gives the same result as running with NEED_PULSEAUDIO=false.

In that case, the early parts of pulseaudio.log look familiar, but:
----
I  [pulseaudio] main.c: Daemon startup successful.
D  [pulseaudio] core-subscribe.c: Dropped redundant event due to change event.
I  [pulseaudio] module-suspend-on-idle.c: Sink auto_null idle for too long, suspending ...
D  [pulseaudio] sink.c: Suspend cause of sink auto_null is 0x0004, suspending
D  [pulseaudio] core.c: Hmm, no streams around, trying to vacuum.
I  [pulseaudio] module-device-restore.c: Synced.
I  [pulseaudio] main.c: Got signal SIGINT.
I  [pulseaudio] main.c: Exiting.
I  [pulseaudio] main.c: Daemon shutdown initiated.
I  [pulseaudio] module.c: Unloading "module-device-restore" (index: #0).
I  [pulseaudio] module.c: Unloaded "module-device-restore" (index: #0).
I  [pulseaudio] module.c: Unloading "module-stream-restore" (index: #1).
D  [pulseaudio] protocol-dbus.c: Interface org.PulseAudio.Ext.StreamRestore1 removed from object /org/pulseaudio/stream_restore1
I  [pulseaudio] module.c: Unloaded "module-stream-restore" (index: #1).
I  [pulseaudio] module.c: Unloading "module-card-restore" (index: #2).
I  [pulseaudio] module.c: Unloaded "module-card-restore" (index: #2).
I  [pulseaudio] module.c: Unloading "module-augment-properties" (index: #3).
I  [pulseaudio] module.c: Unloaded "module-augment-properties" (index: #3).
I  [pulseaudio] module.c: Unloading "module-udev-detect" (index: #4).
I  [pulseaudio] module.c: Unloaded "module-udev-detect" (index: #4).
I  [pulseaudio] module.c: Unloading "module-native-protocol-unix" (index: #5).
I  [pulseaudio] module.c: Unloaded "module-native-protocol-unix" (index: #5).
I  [pulseaudio] module.c: Unloading "module-gconf" (index: #6).
I  [pulseaudio] module.c: Unloaded "module-gconf" (index: #6).
I  [pulseaudio] module.c: Unloading "module-default-device-restore" (index: #7).
I  [pulseaudio] module.c: Unloaded "module-default-device-restore" (index: #7).
I  [pulseaudio] module.c: Unloading "module-rescue-streams" (index: #8).
I  [pulseaudio] module.c: Unloaded "module-rescue-streams" (index: #8).
I  [pulseaudio] module.c: Unloading "module-null-sink" (index: #9).
D  [pulseaudio] module-always-sink.c: Autoloaded null-sink removed
D  [pulseaudio] core-subscribe.c: Dropped redundant event due to change event.
D  [null-sink] module-null-sink.c: Thread shutting down
I  [pulseaudio] sink.c: Freeing sink 0 "auto_null"
I  [pulseaudio] source.c: Freeing source 0 "auto_null.monitor"
I  [pulseaudio] module.c: Unloaded "module-null-sink" (index: #9).
I  [pulseaudio] module.c: Unloading "module-always-sink" (index: #10).
I  [pulseaudio] module.c: Unloaded "module-always-sink" (index: #10).
I  [pulseaudio] module.c: Unloading "module-intended-roles" (index: #11).
I  [pulseaudio] module.c: Unloaded "module-intended-roles" (index: #11).
I  [pulseaudio] module.c: Unloading "module-suspend-on-idle" (index: #12).
I  [pulseaudio] module.c: Unloaded "module-suspend-on-idle" (index: #12).
I  [pulseaudio] module.c: Unloading "module-position-event-sounds" (index: #13).
I  [pulseaudio] module.c: Unloaded "module-position-event-sounds" (index: #13).
I  [pulseaudio] module.c: Unloading "module-filter-heuristics" (index: #14).
I  [pulseaudio] module.c: Unloaded "module-filter-heuristics" (index: #14).
I  [pulseaudio] module.c: Unloading "module-filter-apply" (index: #15).
I  [pulseaudio] module.c: Unloaded "module-filter-apply" (index: #15).
I  [pulseaudio] module.c: Unloading "module-switch-on-port-available" (index: #16).
I  [pulseaudio] module.c: Unloaded "module-switch-on-port-available" (index: #16).
I  [pulseaudio] main.c: Daemon terminated.
----
Interesting -- so even without pulseaudio, it's barfing.

My run (without NEED_PULSEAUDIO):

----
18:28:14     INFO - Running pre-action listener: _resource_record_pre_action
18:28:14     INFO - Running pre-action listener: _set_gcov_prefix
18:28:14     INFO - Running main action method: run_tests
18:28:14     INFO - Running pre test command disable_screen_saver with 'xset s off s reset'
18:28:14     INFO - Running command: ('xset', 's', 'off', 's', 'reset') in /home/worker/workspace/build
18:28:14     INFO - Copy/paste: xset s off s reset
5 XSELINUXs still allocated at reset
SCREEN: 0 objects of 176 bytes = 0 total bytes 0 private allocs
DEVICE: 4 objects of 96 bytes = 384 total bytes 0 private allocs
CLIENT: 0 objects of 160 bytes = 0 total bytes 0 private allocs
WINDOW: 0 objects of 48 bytes = 0 total bytes 0 private allocs
PIXMAP: 1 objects of 16 bytes = 16 total bytes 0 private allocs
GC: 0 objects of 56 bytes = 0 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
CURSOR_BITS: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 5 objects, 400 bytes, 0 allocs
4 DEVICEs still allocated at reset
DEVICE: 4 objects of 96 bytes = 384 total bytes 0 private allocs
CLIENT: 0 objects of 160 bytes = 0 total bytes 0 private allocs
WINDOW: 0 objects of 48 bytes = 0 total bytes 0 private allocs
PIXMAP: 1 objects of 16 bytes = 16 total bytes 0 private allocs
GC: 0 objects of 56 bytes = 0 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
CURSOR_BITS: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 5 objects, 400 bytes, 0 allocs
1 PIXMAPs still allocated at reset
PIXMAP: 1 objects of 16 bytes = 16 total bytes 0 private allocs
GC: 0 objects of 56 bytes = 0 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
CURSOR_BITS: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 1 objects, 16 bytes, 0 allocs
[dix] Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/100dpi/:unscaled, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/75dpi/:unscaled, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/Type1, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/100dpi, removing from list!
[dix] Could not init font path element /usr/share/fonts/X11/75dpi, removing from list!
[dix] Could not init font path element /var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType, removing from list!
18:28:14     INFO - Return code: 0
18:28:14     INFO - Running post-action listener: _package_coverage_data
18:28:14     INFO - Running post-action listener: _resource_record_post_action
18:28:14     INFO - Running post-run listener: _resource_record_post_run
18:28:15     INFO - Total resource usage - Wall time: 13s; CPU: 59.0%; Read bytes: 13287424; Write bytes: 418799616; Read time: 328; Write time: 383372
18:28:15     INFO - install - Wall time: 14s; CPU: 59.0%; Read bytes: 13287424; Write bytes: 418799616; Read time: 328; Write time: 383372
18:28:15     INFO - run-tests - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
18:28:15     INFO - Running post-run listener: _upload_blobber_files
18:28:15  WARNING - Blob upload gear skipped. Missing cmdline options.
18:28:15     INFO - Copying logs to upload dir...
18:28:15     INFO - mkdir: /home/worker/workspace/build/upload/logs

cleanup
+ cleanup
+ '[' -n 21 ']'
+ kill 21
+ false
[taskcluster] === Task Finished ===
[taskcluster] Successful task run with exit code: 0 completed in 175.762 seconds
----

----
18:32:09     INFO - Running pre test command disable_screen_saver with 'xset s off s reset'
18:32:09     INFO - Running command: ('xset', 's', 'off', 's', 'reset') in /home/worker/build
18:32:09     INFO - Copy/paste: xset s off s reset
18:32:09     INFO - Return code: 0
18:32:09     INFO - #### Running mochitest suites
18:32:09     INFO - mkdir: /home/worker/build/blobber_upload_dir
18:32:09     INFO - ENV: MOZ_UPLOAD_DIR is now /home/worker/build/blobber_upload_dir
18:32:09     INFO - ENV: MINIDUMP_SAVE_PATH is now /home/worker/build/blobber_upload_dir
....
----

So basically, it dies right after running `xset`.
OK, this is definitely a docker bug; running via `docker run` on the provisioned EC2 host itself:

----
worker@taskcluster-worker:~$ bash /home/worker/bin/test.sh

: GECKO_HEAD_REPOSITORY         ${GECKO_HEAD_REPOSITORY:=https://hg.mozilla.org/mozilla-central}
+ : GECKO_HEAD_REPOSITORY https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central
: GECKO_HEAD_REV                ${GECKO_HEAD_REV:=default}
+ : GECKO_HEAD_REV bug1213325@default
: WORKSPACE                     ${WORKSPACE:=/home/worker/workspace}
+ : WORKSPACE /home/worker/workspace

  .......

# support multiple, space delimited, config files
config_cmds=""
cleanup
root@ip-172-31-11-45:~# echo $?
0
----

So, my interactive shell died along with all the stuff in the scripts it was running, and dropped me back to the shell on the EC2 host.  I don't see anything interesting in papertrail (including from the kernel) around that time.  Admittedly my `docker run` invocation didn't involve `--devices`, but that is hardly an excuse to crash the container.
Correction to the previous: that had omitted the --device arguments to `docker run`.  Adding those arguments lets it run up through the mkdir and then exit.  Looking at an strace of the mozharness process shows it doing what it says: copying logs to the upload dir, then exiting gracefully.  It's always hard to follow the flow control in Mozharness, but it's quite possible that this is due to some difference in how I'm invoking mozharness.  So, not figuring into the pulseaudio crashes.
Indeed, the "${@}" substitution isn't working because I forgot to pass the args through when dropping root privileges.  So, that's easy to fix.

(..and apologies for those of you new to my bug-as-blog style..)

So we know that:

 - the 'tester' image runs pulseaudio and Xvfb as root; desktop-test does not
 - running pulseaudio can cause a docker container to crash
   - definitely when no --device options are used (not a good excuse, but it happens)
 - not running pulseaudio allows the task to succeed (modulo not actually running tests)
 - docker-worker selects an arbitrary device ID that is not used by another docker instance and mounts the corresponding devices [1]

I wonder if, depending on the device ID selected by docker, pulseaudio sometimes can't find the device and thus crashes docker.  Maybe docker-worker could map those devices to deviceId 0 within the 

[1] in one case:
+ ls -al /dev/snd
total 0
drwxr-xr-x 2 root root      140 Oct 13 22:14 .
drwxr-xr-x 6 root root      400 Oct 13 22:14 ..
crw-rw---- 1 root audio 116, 53 Oct 13 22:14 controlC9
crw-rw---- 1 root audio 116, 52 Oct 13 22:14 pcmC9D0c
crw-rw---- 1 root audio 116, 51 Oct 13 22:14 pcmC9D0p
crw-rw---- 1 root audio 116, 50 Oct 13 22:14 pcmC9D1c
crw-rw---- 1 root audio 116, 49 Oct 13 22:14 pcmC9D1p
Huh. Running pulseaudio as root works.  Blergh.

  https://tools.taskcluster.net/task-inspector/#AtcJbDsaTfChaOPiwC5ZOw/0

Onto the next problem: video doesn't work
Hm, maybe system mode isn't the right plan:

+ sudo -H pulseaudio --system --daemonize                                                                               
W: [pulseaudio] main.c: Running in system mode, but --disallow-exit not set!                                            
W: [pulseaudio] main.c: Running in system mode, but --disallow-module-loading not set!                                  
N: [pulseaudio] main.c: Running in system mode, forcibly disabling SHM mode!                                            
N: [pulseaudio] main.c: Running in system mode, forcibly disabling exit idle time!
I managed to replicate this on my laptop.  Sure enough, everything gets SIGKILL -- but why?

[pid 26802] socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4
[pid 26802] connect(4, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = -1 ENOENT (No such file or directory)
[pid 26802] close(4)                    = 0
[pid 26802] munmap(0x7f6e61999000, 2109656) = 0
[pid 26802] munmap(0x7f6e6177f000, 2200480 <unfinished ...>
[pid 26785] +++ killed by SIGKILL +++
[pid 26806] +++ killed by SIGKILL +++
[pid 26802] +++ killed by SIGKILL +++
+++ killed by SIGKILL +++

26802 is the sudo process -- this was with sudo'ing to root to run pulseaudio.  Without (just running as worker):

[pid 27253] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 27253] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=18, si_uid=1000, si_status=0, si_utime=0, si_stime=2} ---
[pid 27253] wait4(-1, 0x7fffc24e4898, WNOHANG, NULL) = -1 ECHILD (No child processes)
[pid 27253] rt_sigreturn({mask=[]})     = 0
[pid 27253] rt_sigaction(SIGINT, {0x456010, [], SA_RESTORER, 0x7f26d7145150}, {0x43f370, [], SA_RESTORER, 0x7f26d7145150}, 8) = 0
[pid 27253] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
[pid 27253] read(255, "\n# run XVfb in the background, i"..., 3722) = 1752
[pid 27253] write(2, "\n", 1)           = 1
[pid 27253] rt_sigprocmask(SIG_BLOCK, NULL,  <unfinished ...>
[pid 27274] +++ killed by SIGKILL +++
[pid 27253] +++ killed by SIGKILL +++
[pid 27272] +++ killed by SIGKILL +++
+++ killed by SIGKILL +++

27253 is the shell process running test-linux.sh.

So, running as root appears not to help.
Stracing the docker process and its children shows the same, with no `kill(..)` aside from some kill(.., SIG_0) which are generally a way to probe whether a process is alive.
I'm not totally losing my mind -- pulseaudio doesn't work in the b2g `tester` image, either:

root@1cc97c41f959:~#  ~worker/bin/entrypoint sh -c 'pulseaudio --check || echo no'
+ export DISPLAY=:0
+ DISPLAY=:0
+ pulseaudio --start
+ Xvfb :0 -nolisten tcp -screen 0 1600x1200x24
W: [pulseaudio] main.c: This program is not intended to be run as root (unless --system is specified).
+ '[' '!' -z '' ']'
+ buildbot_step 'Running tests' sh -c pulseaudio --check '||' echo no
========= Started Running tests (results: 0, elapsed: 0 secs) (at 2015-10-14 18:51:19.310636) =========
W: [pulseaudio] main.c: This program is not intended to be run as root (unless --system is specified).
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] core-util.c: Failed to connect to system bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
E: [pulseaudio] pid.c: Daemon already running.
E: [pulseaudio] main.c: pa_pid_file_create() failed.
no
========= Finished Running tests (results: 0, elapsed: 0 secs) (at 2015-10-14 18:51:19.323061) =========

So, I'm punting the PA stuff out to bug 1214809.
Comment on attachment 8673239 [details]
MozReview Request: Bug 1213325: refactor desktop-test to use an in-tree script; r?ahal

Bug 1213325: refactor desktop-test to use an in-tree script; r?ahal

This generally makes the approach look more like that for desktop-build.
Attachment #8673239 - Flags: review?(ahalberstadt)
Comment on attachment 8673239 [details]
MozReview Request: Bug 1213325: refactor desktop-test to use an in-tree script; r?ahal

https://reviewboard.mozilla.org/r/21867/#review19909

Thanks, looks great! So I guess this means in the taskcluster configs we replace 'entrypoint + python desktop_unittest.py' with 'test.sh'? And then have some sort of 'mozharness_arguments' variable that contains the command line we pass into it?

::: testing/taskcluster/scripts/tester/test-linux.sh:50
(Diff revisions 1 - 2)
> +    echo "Pulseaudio is not yet supported in desktop-test"

nit: add the docker pulseaudio bug number here
Attachment #8673239 - Flags: review?(ahalberstadt) → review+
docker images push'd
Duplicate of this bug: 1212881
https://hg.mozilla.org/mozilla-central/rev/4b1a3c29d838
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Dustin, you added remove_executables.py to mozharness root. I assume this was by accident? This file is already present in configs and should reside only there, right?

http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/remove_executables.py
http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/remove_executables.py
Flags: needinfo?(dustin)
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.