1950158 - mach test and mach android-test cannot be used to run one specific test file

Status: RESOLVED FIXED

(Fenix :: Tooling, defect)

Description

[Rob Wu [:robwu]]

bug 1897186 added support for using ./mach test (and ./mach android-test) to run tests in android-components. While trying that out, I noticed the inability to run one specific test - it always runs all tests.

I tried the following:

• ./mach test mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt
• ./mach android-test --subproject=ac --test=mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt

The following gradlew commands works:

JAVA_HOME=$HOME/.mozbuild/jdk/jdk-17.0.14+7/Contents/Home ANDROID_HOME=$HOME/.mozbuild/android-sdk-macosx ANDROID_SDK_ROOT=$HOME/.mozbuild/android-sdk-macosx ./gradlew :feature-addons:testDebugUnitTest --tests PermissionsDialogFragmentTest

By "working" I meant that it at least tries to run the test. The test result is cached and re-used unless I modify the test file, which was reported before as bug 1936263. Also, I don't see any test output. Due to this all, I can imagine how the bug was missed during development - there is no obvious output in the console that could tell whether the test ran or not...

Comment 1

[Rob Wu [:robwu]]

Attachment: Bug 1950158 - Run specific test instead of all AC tests with mach

Comment 2

[Aaditya Dhingra [:adhingra]]

Hey, I tried running the command above and get this:

When a test fails:

./mach test mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

10 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':feature-addons:testDebugUnitTest'.
> There were failing tests. See the report at: file:///Users/adhingra/mozilla-unified-2/mobile/android/android-components/components/feature/addons/build/reports/tests/testDebugUnitTest/index.html

* Try:
> Run with --scan to get full insights.

BUILD FAILED in 10s
 0:10.35 
Overall Summary

When a test passes:

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
 0:14.17 
Overall Summary
===============

I agree that the output is not clear when it passes, planning to work on this soon!

Comment 3

[Rob Wu [:robwu]]

(In reply to Aaditya Dhingra [:adhingra] from comment #2)

I agree that the output is not clear when it passes, planning to work on this soon!

Is there an Bugzilla ticket tracking this that I can follow?

Comment 4

[Aaditya Dhingra [:adhingra]]

Attachment: Bug 1950158 - Show better output for mach test for android.

Comment 5

[Aaditya Dhingra [:adhingra]]

With https://bugzilla.mozilla.org/show_bug.cgi?id=1950158#c4:

Output when test is successful:

./mach test mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
Configuration on demand is an incubating feature.
mozconfig.gradle> Loading mach environment into a gradle extension property
settings.gradle> Read local.properties: /Users/adhingra/mozilla-unified-2/local.properties
settings.gradle> Disabled auto-publication of application-services. Enable it by settings 'autoPublish.application-services.dir' in local.properties
settings.gradle> Disabled auto-publication of Glean. Enable it by settings 'autoPublish.glean.dir' in local.properties

> Task :feature-addons:testDebugUnitTest
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

[Incubating] Problems report is available at: file:///Users/adhingra/mozilla-unified-2/objdir-frontend/gradle/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 9s
402 actionable tasks: 23 executed, 379 up-to-date
 0:09.68 
Overall Summary
===============

Output when test fails:

./mach test mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
Configuration on demand is an incubating feature.
mozconfig.gradle> Loading mach environment into a gradle extension property
settings.gradle> Read local.properties: /Users/adhingra/mozilla-unified-2/local.properties
settings.gradle> Disabled auto-publication of application-services. Enable it by settings 'autoPublish.application-services.dir' in local.properties
settings.gradle> Disabled auto-publication of Glean. Enable it by settings 'autoPublish.glean.dir' in local.properties

> Task :feature-addons:testDebugUnitTest
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

> Task :feature-addons:testDebugUnitTest

PermissionsDialogFragmentTest > build dialog FAILED
    java.lang.AssertionError at PermissionsDialogFragmentTest.kt:106

10 tests completed, 1 failed

> Task :feature-addons:testDebugUnitTest FAILED

[Incubating] Problems report is available at: file:///Users/adhingra/mozilla-unified-2/objdir-frontend/gradle/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':feature-addons:testDebugUnitTest'.
> There were failing tests. See the report at: file:///Users/adhingra/mozilla-unified-2/mobile/android/android-components/components/feature/addons/build/reports/tests/testDebugUnitTest/index.html

* Try:
> Run with --scan to get full insights.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.12.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 9s
402 actionable tasks: 23 executed, 379 up-to-date
 0:09.66 
Overall Summary
===============

Comment 6

[Rob Wu [:robwu]]

Thanks for jumping on this so quickly. It is certainly an issue that I'd like to see fixed, but it would probably be better to file a new bug and continue there, so that people can follow the bug and track its fix even after the patch to this bug report has landed.

I gave your patch a try, and shared feedback at https://phabricator.services.mozilla.com/D239368#8266508

Comment 7

[Aaditya Dhingra [:adhingra]]

Moved to Bug 1950183: https://phabricator.services.mozilla.com/D239390

Example output:

./mach test mobile/android/android-components/components/feature/addons/src/test/java/mozilla/components/feature/addons/ui/PermissionsDialogFragmentTest.kt
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8

SUITE: mozilla.components.feature.addons.ui.PermissionsDialogFragmentTest
  TEST: hide private browsing checkbox when the add-on does not allow running in private windows
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
  SUCCESS
  TEST: dialog with origin permissions shows the first five domains at the top of the list
  SUCCESS
  TEST: build dialog
  TEST-UNEXPECTED-FAIL | mozilla.components.feature.addons.ui.PermissionsDialogFragmentTest.build dialog | java.lang.AssertionError: expected:<2> but was:<3>
  TEST: dialog must have all the styles of the feature promptsStyling object
  SUCCESS
  TEST: dialog with origin permissions allows for toggling all domains shown
  SUCCESS
  TEST: handles add-ons without permissions
  SUCCESS
  TEST: clicking on dialog buttons notifies lambdas
  SUCCESS
  TEST: build dialog for optional permissions
  SUCCESS
  TEST: dismiss the permissions dialog when an origin permission does not match the normalization requirements
  SUCCESS
  TEST: dismissing the dialog notifies deny lambda
  SUCCESS

10 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':feature-addons:testDebugUnitTest'.
> There were failing tests. See the report at: file:///Users/adhingra/mozilla-unified-2/mobile/android/android-components/components/feature/addons/build/reports/tests/testDebugUnitTest/index.html

* Try:
> Run with --scan to get full insights.

BUILD FAILED in 18s
 0:18.55 
Overall Summary
===============

Thanks for helping out!!

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/77bc3597e850
Run specific test instead of all AC tests with mach r=ohall,adhingra

Comment 9

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/77bc3597e850