1395643 - Autophone - disable all Unit Tests until reliability is restored

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, enhancement)

Description

[Bob Clary [:bc] (inactive)]

Autophone's Unit Tests have become extremely unreliable due to a number of failures. I am going to remove all Unit Tests from the schedule except on try until this is sorted out. I will try to diagnose then this began so we have some indication of the root cause. Before we enable these tests again we will have to prove that they are stable on try.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: bug-1395643-v1.patch

Comment 2

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/2aa36cbf2201973ca12dadd188b5e80539bf7b57
cancelled pending jobs
rebooted
deployed 2017-08-31 13:00:00

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: application-timed-out-with-no-output.txt

I collected the errors for August and extracted the application timed out with no output errors. This shows the problems with application time outs began around

      9  20170829231727  mozilla-inbound     
      5  20170829231804  autoland            
      5  20170830084509  mozilla-central     

when the error became more consistent and the counts jumped.

The transition from android-api-15 to android-api-16 caught me unawares. We missed testing builds for the following until I made the change to support android-api-16:

autoland        2017-08-29 11:48:09 - 2017-08-29 22:50:41
mozilla-central 2017-08-29 18:28:36 - 2017-08-29 22:38:06
mozilla-inbound 2017-08-29 18:39:46 - 2017-08-29 23:03:35

The jump in errors corresponds to the change. RyanVM suggested that this might be due to the api change and I am beginning to believe him.

Comment 4

[Bob Clary [:bc] (inactive)]

Ryan: Fyi, I'm beginning to be convinced though I don't know why. The android_version checks shouldn't have been affected by this change as far as I know.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: SecurityError: Permission denied to access property "wrappedJSObject" on cross-origin object

Remember that Autophone Unit tests do not run on every push. They only run when the specified list of directories is changed. Therefore when an error occurs on a push it is either due to the push or earlier pushes which did not change the required directories. It makes diagnosis trickier.

SecurityError: Permission denied to access property "wrappedJSObject" on cross-origin object spiked just as the failures spiked. It had occurred a couple of times previously but first appeared in vloume in a try push on 2017-08-23

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2b99165a25a6901fd270d03dc8226003762696a4
[e-mail] Bug 1388938 - Disable channel annotation and flashblock in tests harnesses.r?hchang

NOTE: Bug 1388938

These prefs should be added to all of the test harnesses that disable Safe Browsing:

  privacy.trackingprotection.annotate_channels = false
  plugins.flashBlock.enabled = false

Pushed by [e-mail] on 2017-08-24 10:57 PDT

https://hg.mozilla.org/integration/autoland/rev/3cbad3a6fc6e
Disable channel annotation and flashblock in tests harnesses.r=hchang

The error then did not reoccur until 2017-08-29 in the following pushes

merge
mozilla-inbound 20170829231727
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=a06b235e4a0bdd9ae6e52a174981ba0c9b3fa232

autoland 20170830012828
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=72ffe7066fc077f45c8b11813189c4621282addc
 Wed Aug 30, 1:28:28 - [e-mail]
 Bug 1385934: Use RTL layout attr in activity_stream_topsites_page. r=liuche

mozilla-inbound 20170829231727
merge mozilla-inbound 20170829231727
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=a06b235e4a0bdd9ae6e52a174981ba0c9b3fa232

The try push for Bug 1388938 is clearly implicated though I don't know why it did not become a regular error after it landed on autoland. Perhaps the merges were faulty.

I am doing a run on autophone-4 reporting to staging for the period I missed due to the api change. It is currently running and is available at https://treeherder.allizom.org/#/jobs?repo=autoland&filter-searchStr=autophone&exclusion_profile=false&fromchange=e3c7045f22a712f062e45ab45642c55fca0e48f7&group_state=expanded&tochange=04b1b915604a918556f7b6d9ce36ca591752715c

fmarier: Do you know if your patch would be responsible for these SecurityErrors? Is the problem that I haven't set the proper preferences as you mentioned in the bug?

Comment 6

[François Marier [:francois]]

(In reply to Bob Clary [:bc:] from comment #5)
> fmarier: Do you know if your patch would be responsible for these
> SecurityErrors? Is the problem that I haven't set the proper preferences as
> you mentioned in the bug?

I'll be honest, I don't really understand what I'm looking at here.

I would not expect this patch, which disables flashblocking and various Quantum optimizations based on the tracking protection list, to have any effect on permissions.

Does the problem actually go away if you set one of these prefs to true?

Comment 7

[Bob Clary [:bc] (inactive)]

I haven't tested it yet. My test run for the missing builds due to the api change is still running and hasn't reproduced the problem yet. If I can reproduce it on my test system, I'll try the pref changes to see if it makes a difference.

It does seem weird. Your patch had been in the tree for almost a week before I started seeing problems. The only thing that led me to you was the fact that your try run had a number of these SecurityErrors.

Comment 8

[Bob Clary [:bc] (inactive)]

Went again and disabled the devices that no longer have tests:
https://github.com/mozilla/autophone/commit/6212ae793d2e872882b2c2b3ea958a8154624b53

Comment 9

[Intermittent Failures Robot]

356 failures in 939 pushes (0.379 failures/push) were associated with this bug in the last 7 days. 

This is the #3 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 247
* mozilla-central: 58
* mozilla-inbound: 48
* mozilla-beta: 3

Platform breakdown:
* android-6-0-armv8-api16: 161
* android-7-1-armv8-api16: 113
* android-4-4-armv7-api16: 64
* android-5-0-armv8-api16: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395643&startday=2017-08-28&endday=2017-09-03&tree=all