Closed Bug 414049 Opened 12 years ago Closed 9 years ago

unit test boxes should upload a screen shot after testing, on Linux

Categories

(Testing :: General, enhancement, P5)

All
Linux
enhancement

Tracking

(Not tracked)

RESOLVED FIXED
mozilla2.0b5

People

(Reporter: ted, Assigned: karlt)

References

Details

Attachments

(5 files, 3 obsolete files)

The unit test boxes sometimes have mysterious failures due to focus, which can happen if someone logs in and leaves a console window open or probably other silly things.  They should upload a screenshot somewhere (linked in the log file) so we could quickly spot check for that sort of problem.
Severity: normal → enhancement
Mass move of Core:Testing bugs to mozilla.org:ReleaseEngineering. Filter on RelEngMassMove to ignore.
Component: Testing → Release Engineering
Product: Core → mozilla.org
QA Contact: testing → release
Version: Trunk → other
Component: Release Engineering → Release Engineering: Future
For reference, I do something similar to this on an Xvfb instance using:
import -window root -display :50 /path/to/screenshot.png

but "import" is from ImageMagick, so there's probably a simpler solution.

on OS X:
/usr/sbin/screencapture -x /path/to/screenshot.png
Attached file win32 screenshot utility (obsolete) —
Here's a command-line Win32 program to save a screenshot to a PNG file that I wrote in < 100 lines of C++.
Oops, I missed a spot in the license header.
Attachment #360708 - Attachment is obsolete: true
I'd love to see this behind an xpcom interface so that we could use it while running the tests, too. In particular, that'd be nifty to have for doing l10n screenshots.
That wasn't my intention. I was thinking that the buildbot would just call an external process and upload the png somewhere. Mixing in XPCOM adds a lot of complexity, and then each test harness would have to learn how to call it etc.
The Linux refplatform is sorely lacking in tools to make this happen. The best I appear able to do is using "xwd", but there's no xwdtopnm, so I can't turn it into a useful image format. It has gnome-screenshot, but AFAICT there's no way to run that non-interactively.
If we got netpbm-progs installed, then the following would work just fine:
xwd -root -silent | xwdtopnm | pnmtopng > screenshot.png
Attached file GDK screenshot utility
Turns out it's like 5 lines of code if you use GDK. This WFM on the Linux refplatform without any extra software.
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
Just for information, on Windows the ImageGrab module from the Python Imaging Library is another handy way to take screenshots directly from Python (http://www.pythonware.com/library/pil/handbook/imagegrab.htm) (but it requires installing PIL, so it's not really self-contained).
Comment on attachment 360916 [details]
GDK screenshot utility


>  return TRUE == gdk_pixbuf_save (screenshot, filename, "png",
>                  &error, NULL);

should be the opposite, so that it returns with an exit status of 0 if gdk_pixbuf_save succeeds.
Ted: this is assigned to you, but I'm guessing it's not on your radar any more. It came up during our future queue triage today. 

Do you still think this is useful, and do you plan to pick it up again at some point?
I think it's useful, yeah. I know we've still hit some "random dialog on screen" errors in recent history. I think it's just a matter of getting these screenshot tools built (or sticking binaries in the build/tools repo) and then adding steps to upload them somewhere. (The somewhere might be a sticking point?)

As to whether I'm the one to pick it up, well, that's a good question. I'm not likely to have the time or experience to hack buildbot configs to make this happen.
firefox/tinderbox-builds/<epoch_time> alongside the build/symbols/tests perhaps, via UPLOAD_EXTRA_FILES. Unless you want more than a 24 hour lifetime.
Still a good idea, but I'm not likely to finish it.
Assignee: ted.mielczarek → nobody
I stuck the source of both files here, just to have a better place to track them:
http://hg.mozilla.org/users/tmielczarek_mozilla.com/screenshot-tools/

(I also fixed the return values as syp pointed out in comment 11.)
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P5
Status: ASSIGNED → NEW
Whiteboard: [unittest][automation]
This would have helped us catch bug 578448 right away.
I believe this will be helpful for the WarOnOranges.
I will consider taking this on specially for XP oranges that will come in few weeks.
I wonder if we could just dump the PNG to a data URI in the log.  The only downside I see to that is that the log size would grow substantially on failure, but it would make this quite a bit simpler (we could trigger the screenshots from the mochitest harness rather than the build automation)
Assignee: nobody → jhford
I set up some screenshots on timeouts (motivated by bug 557456).

I didn't know you were working on this, sorry John.  I wasn't meaning to steal this bug from you, but please use this if it is helpful.  (Ignore the "xset"; that was extra info I wanted.)

http://hg.mozilla.org/try/rev/91e501a090e4
http://hg.mozilla.org/try/rev/602ab418a40a

I don't know about IPC on MS systems, so haven't tried anything there.

It's working on X11, but Mac's screencapture doesn't seem to be doing what I
wanted when the file is /dev/stdout:

4411 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_CrossSiteXHR_origin.html | Test timed out.
SCREENSHOT: data:image/png;base64,
4412 INFO SimpleTest finished /tests/content/base/test/test_CrossSiteXHR_origin.html in 315835ms

Passing "-" as the file parameter to screencapture gives:
  screencapture: no file specified
  ... exited with code 255

The gdk screen capture program originated from ted's tool.  I was probably
going to move it from tools/screenshot to testing/tools/screenshot.

The screenshots on the Linux test machines were about 400kB when base64 encoded.
If that's a bit large, then there are things that can be done to help:
* Surround by a few lines so that they don't show up in summaries.
* Change the desktop background to something simpler.
* Install ImageMagick and do color reduction.
Attached patch patch for GDK screenshot utility (obsolete) — Splinter Review
Contents of http://hg.mozilla.org/try/raw-rev/91e501a090e4 , since they go away when try gets cleaned out.
It looks like this should be part of the test harness given the direction of this bug.
Assignee: jhford → nobody
Component: Release Engineering → General
Product: mozilla.org → Testing
QA Contact: release → general
Version: other → unspecified
Whiteboard: [unittest][automation]
Added some screensaver info to help with identifying bug 585286.

Asking dbaron for review since he wrote a similar utility in bug 570878.
Attachment #463755 - Attachment is obsolete: true
Attachment #464728 - Flags: review?
Attachment #464728 - Flags: review? → review?(dbaron)
I think it's worth adding this now even though it's only implemented for Linux builds.  It would help to debug the many focus issues on Linux as well as with bug 585286.  The screen is only dumped on the first timeout so as not to make the logs unnecessarily large.
Attachment #463756 - Attachment is obsolete: true
Attachment #464729 - Flags: review?(ted.mielczarek)
Attachment #464730 - Flags: review? → review?(gavin.sharp)
Attachment #464730 - Flags: review?(gavin.sharp) → review+
Comment on attachment 464728 [details] [diff] [review]
Add a GDK/X11 screenshot to png utility

r=dbaron

(At some point we might want a privileged DOMWindowUtils screenshot API in addition to this...)
Attachment #464728 - Flags: review?(dbaron) → review+
Comment on attachment 464729 [details] [diff] [review]
dump screen on first mochitest time out (GTK)

># HG changeset patch
># Parent 0bbc6d5340aeb1ed3b44bc3ce0870d8839340167
># User Karl Tomlinson <karlt+@karlt.net>
>b=414049 dump screen on first mochitest time out (GTK) r?ted
>
>diff --git a/build/automation.py.in b/build/automation.py.in
>--- a/build/automation.py.in
>+++ b/build/automation.py.in
>+  if UNIXISH:
>+    def dumpScreen(self, utilityPath):
>+      self.haveDumpedScreen = True;
>+
>+      screentopng = os.path.join(utilityPath, "screentopng")
>+      if os.path.exists(screentopng):
>+        command = [screentopng]
>+      else: # try ImageMagick
>+        command = ["import", "-window", "root",
>+                   "+dither", "-colors", "16", "png:-"]

Don't bother with this fallback, you're building screentopng as part of the build, you can reasonably expect it to be there.

>+
>+      try:
>+        dumper = self.Process(command, bufsize=-1,
>+                              stdout=subprocess.PIPE, close_fds=True)
>+      except OSError, err:
>+        self.log.info("Failed to start %s for screenshot: %s",
>+                      command[0], err.strerror)
>+        return
>+
>+      image = dumper.stdout.read()
>+      status = dumper.wait()
>+      if status != 0:
>+        self.log.info("%s exited with code %d", command[0], status)
>+        return
>+
>+      import base64
>+      encoded = base64.b64encode(image)
>+      self.log.info("SCREENSHOT: data:image/png;base64,%s", encoded)

This is kind of unfortunate, but maybe we can hack TBPL to linkify these.

>diff --git a/testing/mochitest/Makefile.in b/testing/mochitest/Makefile.in
>--- a/testing/mochitest/Makefile.in
>+++ b/testing/mochitest/Makefile.in
>@@ -146,16 +146,22 @@ endif
> ifeq ($(OS_ARCH),Darwin)
> TEST_HARNESS_BINS += fix_macosx_stack.py
> endif
> 
> ifeq ($(OS_ARCH),Linux)
> TEST_HARNESS_BINS += fix-linux-stack.pl
> endif
> 
>+ifeq ($(MOZ_WIDGET_TOOLKIT),gtk2)
>+ifdef MOZ_X11

You could write this as:

ifeq (gtk2_1,$(MOZ_WIDGET_TOOLKIT)_$(MOZ_X11))

r=me with those changes.
Attachment #464729 - Flags: review?(ted.mielczarek) → review+
Blocks: 589668
http://hg.mozilla.org/mozilla-central/rev/d926be226157
http://hg.mozilla.org/mozilla-central/rev/094a1b4d19b1
http://hg.mozilla.org/mozilla-central/rev/6c9e53b7751f

Filed bug 589668 to track adding implementations for other platforms.
Assignee: nobody → karlt
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla2.0b5
OS: All → Linux
Depends on: 606048
Summary: unit test boxes should upload a screen shot after testing → unit test boxes should upload a screen shot after testing, on Linux
Version: unspecified → Trunk
You need to log in before you can comment on or make changes to this bug.