Investigate adding Facebook screenshot-tests regression tool to automation

REOPENED
Unassigned

Status

()

P5
normal
REOPENED
3 years ago
3 months ago

People

(Reporter: mcomella, Unassigned, Mentored)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Facebook has a tool called screenshot-tests [1] that will take a screenshot of a page, compare it to a previous screenshot, and check if anything has changed. This is a good way to catch errors when changing code that will affect many different configurations.

We should check it out, see how easy it'd be to run on a variety of API levels, and, if dope, add these tests to automation.

[1]: https://facebook.github.io/screenshot-tests-for-android/
Oh, this is hot.  It's quite tied to Gradle -- I don't think it's feasible to do this without running Gradle (in some form) in automation.
This would be a great mentored bug – I'd say first steps would be to get this running in the tree (and we can do automation in a follow-up).

Nick, do you have some more guidance?
Mentor: michael.l.comella, nalexander
Flags: needinfo?(nalexander)
(In reply to Michael Comella (:mcomella) from comment #2)
> This would be a great mentored bug – I'd say first steps would be to get
> this running in the tree (and we can do automation in a follow-up).
> 
> Nick, do you have some more guidance?

My guess is that we just need to add the gradle configuration, like:

https://facebook.github.io/screenshot-tests-for-android/#gradle-setup

and then try to take a screenshot in a test.  The tricky part will be setting the right test runner (https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#14), or making our existing test runner use the screenshot stuff (https://dxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/FennecInstrumentationTestRunner.java).

As for running in automation, this looks just like Bug 1064004, with slightly different parameters in the post_flight_build_mach_commands.  I guess we need to make sure the emulator we run isn't headless.
Flags: needinfo?(nalexander)
Whiteboard: [platform-rel-Facebook]

Updated

2 years ago
platform-rel: --- → ?
platform-rel: ? → ---
Whiteboard: [platform-rel-Facebook]
Comment hidden (spam)
Restrict Comments: true

Comment 5

10 months ago
Mass closing Firefox for Android :: Testing bugs with no progress in 2017.

If this bug is important to you, please re-open.
Status: NEW → RESOLVED
Last Resolved: 10 months ago
Resolution: --- → WONTFIX
It'd be ideal if we could do this.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
Priority: -- → P5
You need to log in before you can comment on or make changes to this bug.