1232863 - Investigate adding Facebook screenshot-tests regression tool to automation

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: Testing, defect, P5)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Facebook has a tool called screenshot-tests [1] that will take a screenshot of a page, compare it to a previous screenshot, and check if anything has changed. This is a good way to catch errors when changing code that will affect many different configurations.

We should check it out, see how easy it'd be to run on a variety of API levels, and, if dope, add these tests to automation.

[1]: https://facebook.github.io/screenshot-tests-for-android/

Comment 1

[Nick Alexander :nalexander [he/him]]

Oh, this is hot.  It's quite tied to Gradle -- I don't think it's feasible to do this without running Gradle (in some form) in automation.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

This would be a great mentored bug – I'd say first steps would be to get this running in the tree (and we can do automation in a follow-up).

Nick, do you have some more guidance?

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Michael Comella (:mcomella) from comment #2)
> This would be a great mentored bug – I'd say first steps would be to get
> this running in the tree (and we can do automation in a follow-up).
> 
> Nick, do you have some more guidance?

My guess is that we just need to add the gradle configuration, like:

https://facebook.github.io/screenshot-tests-for-android/#gradle-setup

and then try to take a screenshot in a test.  The tricky part will be setting the right test runner (https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#14), or making our existing test runner use the screenshot stuff (https://dxr.mozilla.org/mozilla-central/source/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/FennecInstrumentationTestRunner.java).

As for running in automation, this looks just like Bug 1064004, with slightly different parameters in the post_flight_build_mach_commands.  I guess we need to make sure the emulator we run isn't headless.

Comment 5

[Geoff Brown [:gbrown]]

Mass closing Firefox for Android :: Testing bugs with no progress in 2017.

If this bug is important to you, please re-open.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

It'd be ideal if we could do this.

Comment 7

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 8

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.