If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

[tier-3] Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Available update has been found

RESOLVED INCOMPLETE

Status

Testing
Firefox UI Tests
RESOLVED INCOMPLETE
11 months ago
7 days ago

People

(Reporter: Treeherder Bug Filer, Unassigned)

Tracking

(Depends on: 1 bug, {bulk-close-intermittents, intermittent-failure})

Version 3
bulk-close-intermittents, intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox51 affected)

Details

(Whiteboard: [stockwell unknown])

(Reporter)

Description

11 months ago
treeherder
Filed by: hskupin [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=4079767&repo=mozilla-aurora

https://firefox-ui-tests.s3.amazonaws.com/4997a0b0-8df8-4cad-9c1a-64fe089993a4/log_info.log

Updated

11 months ago
status-firefox51: --- → affected
Summary: Intermittent test_direct_update.py TestDirectUpdate.test_update | AssertionError: Available update has been found → Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Available update has been found
This failure happened a lot on Sunday Nov 6th for update number 3 and 4:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=8027a7b723c9db7254bdb0bea3bc8937220daa7e&filter-searchStr=update&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=4082851

Maybe somehow related to bug 1285340
See Also: → bug 1285340

Comment 2

10 months ago
30 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 30

Platform breakdown:
* osx-10-9: 5
* osx-10-11: 5
* windows8-64: 4
* osx-10-10: 4
* windows7-64: 3
* windowsxp: 2
* windows8-32: 2
* windows7-32: 2
* linux64: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2016-11-07&endday=2016-11-13&tree=all

Comment 3

7 months ago
8 failures in 836 pushes (0.01 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-aurora: 5
* mozilla-beta: 3

Platform breakdown:
* windows8-64: 4
* windows8-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-02-06&endday=2017-02-12&tree=all

Comment 4

7 months ago
7 failures in 833 pushes (0.008 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-beta: 3
* mozilla-central: 2
* mozilla-aurora: 2

Platform breakdown:
* windows8-64: 5
* windows8-32: 1
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-02-13&endday=2017-02-19&tree=all

Comment 5

7 months ago
23 failures in 182 pushes (0.126 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* mozilla-beta: 23

Platform breakdown:
* windows7-64: 5
* linux64: 4
* windows8-64: 3
* windows8-32: 3
* osx-10-9: 2
* osx-10-11: 2
* osx-10-10: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-02-24&endday=2017-02-24&tree=all

Comment 6

7 months ago
23 failures in 812 pushes (0.028 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-beta: 23

Platform breakdown:
* windows7-64: 5
* linux64: 4
* windows8-64: 3
* windows8-32: 3
* osx-10-9: 2
* osx-10-11: 2
* osx-10-10: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-02-20&endday=2017-02-26&tree=all

Comment 7

7 months ago
7 failures in 783 pushes (0.009 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-central: 4
* mozilla-release: 2
* mozilla-aurora: 1

Platform breakdown:
* windows8-64: 4
* windows8-32: 1
* windows7-64: 1
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-02-27&endday=2017-03-05&tree=all

Comment 8

6 months ago
31 failures in 790 pushes (0.039 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-aurora: 31

Platform breakdown:
* windows8-64: 6
* windows7-64: 5
* linux32: 5
* windows8-32: 4
* windows7-32: 4
* osx-10-9: 2
* osx-10-11: 2
* osx-10-10: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-03-06&endday=2017-03-12&tree=all
Did we turn off updates for Nightly builds over the weekend? I can see lots of failures on Windows and Linux.
Flags: needinfo?(rail)
Flags: needinfo?(bugspam.Callek)
AFAIK, there was an issue in one of the nightly builds on Friday and recall that someone disabled the updates. I also have vague memory, that someone was going to send an announcement to one of the lists.
Flags: needinfo?(rail)
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #10)
> AFAIK, there was an issue in one of the nightly builds on Friday and recall
> that someone disabled the updates. I also have vague memory, that someone
> was going to send an announcement to one of the lists.

Yes, RyanVM did the announce, but also said it was enabled, so I *think* its all set.

https://groups.google.com/d/topic/mozilla.dev.platform/2_9L32K32h8/discussion
Flags: needinfo?(bugspam.Callek)
can I assume that all these failures were related to a misconfigured nightly job this weekend?  the failures seem to have stopped!
Whiteboard: [stockwell fixed]
Thanks. That would explain it. And yes, we all all back to normal for nightly builds off mozilla-central.

Comment 14

6 months ago
13 failures in 777 pushes (0.017 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-aurora: 10
* mozilla-beta: 2
* mozilla-esr52: 1

Platform breakdown:
* windows8-64: 5
* windows8-32: 4
* windows7-64: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-03-13&endday=2017-03-19&tree=all
Duplicate of this bug: 1303705

Comment 16

5 months ago
11 failures in 867 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 6
* mozilla-aurora: 3
* mozilla-central: 2

Platform breakdown:
* windows8-64: 8
* windows8-32: 2
* windows7-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 17

5 months ago
6 failures in 894 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-aurora: 3
* mozilla-release: 2
* mozilla-esr52: 1

Platform breakdown:
* windows8-64: 3
* windows7-32: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-04-10&endday=2017-04-16&tree=all
With yesterdays tests we have a spike of this failure for mozilla-aurora. When I checked mozmill-ci I noticed that it was missed to change the channel from aurora to auroratest. I made this change for now.

Comment 19

5 months ago
19 failures in 817 pushes (0.023 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-aurora: 12
* mozilla-esr52: 4
* mozilla-release: 3

Platform breakdown:
* windows8-64: 7
* windows7-64: 5
* windowsxp: 3
* windows8-32: 2
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-04-17&endday=2017-04-23&tree=all

Comment 20

5 months ago
9 failures in 883 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 8
* mozilla-central: 1

Platform breakdown:
* windows8-64: 5
* windows8-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-04-24&endday=2017-04-30&tree=all

Comment 21

5 months ago
10 failures in 770 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 5
* mozilla-release: 4
* mozilla-central: 1

Platform breakdown:
* windows8-64: 6
* windows8-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-05-01&endday=2017-05-07&tree=all

Comment 22

4 months ago
10 failures in 777 pushes (0.013 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-esr52: 5
* mozilla-release: 3
* mozilla-beta: 2

Platform breakdown:
* windows8-64: 5
* windowsxp: 4
* windows8-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-05-15&endday=2017-05-21&tree=all
I caught an interesting case for direct updates on mozilla-beta:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=a6edfa473c7d&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta(&selectedJob=101240456

Here we clearly get a valid snippet but the AUS service tells us that there is no update available:

https://treeherder.mozilla.org/logviewer.html#?job_id=101240456&repo=mozilla-beta&lineNumber=2697
https://treeherder.mozilla.org/logviewer.html#?job_id=101240456&repo=mozilla-beta&lineNumber=7096

08:16:00     INFO -  'build_pre': {'buildid': u'20170504103226',
08:16:00     INFO -                'channel': u'beta',
08:16:00     INFO -                'disabled_addons': None,
08:16:00     INFO -                'locale': u'pl',
08:16:00     INFO -                'mar_channels': set(['firefox-mozilla-beta',
08:16:00     INFO -                                     'firefox-mozilla-release']),
08:16:00     INFO -                'update_snippet': '<?xml version="1.0"?>\n<updates>\n    <update type="minor" displayVersion="54.0 Beta 10" appVersion="54.0" platformVersion="54.0" buildID="20170522172523" detailsURL="https://www.mozilla.org/pl/firefox/54.0/releasenotes/">\n        <patch type="complete" URL="http://download.mozilla.org/?product=firefox-54.0b10-complete&amp;os=win&amp;lang=pl&amp;force=1" hashFunction="sha512" hashValue="10b826b52bb3c8a698cee43c743189a4ea5ae16b626d68c3706c520fa2ca5706d3e34e63ddb7380fa9afb8779e2bb72e1137e0a5d62d2aaa1ebd9605ccb1a3b5" size="54414805"/>\n    </update>\n</updates>',
08:16:00     INFO -                'update_url': u'https://aus5.mozilla.org/update/6/Firefox/54.0/20170504103226/WINNT_x86-msvc-x86/pl/beta/Windows_NT%206.3.0.0%20(x86)(noBug1296630v1)(nowebsense)/SSE3/default/default/update.xml?force=1',
08:16:00     INFO -                'user_agent': u'Mozilla/5.0 (Windows NT 6.3; rv:54.0) Gecko/20100101 Firefox/54.0',
08:16:00     INFO -                'version': u'54.0'},
08:16:00     INFO -  'fallback': False,

The update available status gets retrieved by checking the currently selected deck of the about window:

https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/harness/firefox_ui_harness/testcases.py#133

Checking the logs in more detail I can see the following failure of the application updater:

08:15:59     INFO -  *** AUS:SVC Checker:onError - request.status: 2152398890
08:15:59     INFO -  *** AUS:SVC getStatusTextFromCode - transfer error: Serwer proxy nie został odnaleziony (należy sprawdzić połączenie z Internetem), code: 2152398890

It looks like there was a problem with the network, and the proxy server has not been found.

So we should take failures into account for the above update available check. And if there is a failure we have to bail out.

I will file a new bug for this.
Depends on: 1367319

Comment 24

3 months ago
3 failures in 814 pushes (0.004 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 2
* mozilla-esr52: 1

Platform breakdown:
* windows8-32: 2
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 25

3 months ago
22 failures in 166 pushes (0.133 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-beta: 22

Platform breakdown:
* windows8-64: 4
* windows7-64: 4
* windows8-32: 2
* windows7-32: 2
* osx-10-9: 2
* osx-10-11: 2
* osx-10-10: 2
* linux64: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-06-23&endday=2017-06-23&tree=all
on mozilla-beta this test had 22 failures yesterday alone, this seems to be a new issue given the fact that we only had 3 failures in the previous week.

:whimboo- can you take a look at this?
Flags: needinfo?(hskupin)
Whiteboard: [stockwell fixed] → [stockwell needswork]

Comment 27

3 months ago
23 failures in 892 pushes (0.026 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 23

Platform breakdown:
* windows8-64: 5
* windows7-64: 4
* windows8-32: 2
* windows7-32: 2
* osx-10-9: 2
* osx-10-11: 2
* osx-10-10: 2
* linux64: 2
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-06-19&endday=2017-06-25&tree=all
(In reply to Joel Maher ( :jmaher) from comment #26)
> on mozilla-beta this test had 22 failures yesterday alone, this seems to be
> a new issue given the fact that we only had 3 failures in the previous week.
> 
> :whimboo- can you take a look at this?

These failures were due to incorrect setup of the beta channel (55b3 was available instead of 55b4). They should be ignored (I've also informed people to mark such failure as "expected fail").
Note that I've also removed the associations to this bug and have reclassified all corresponding failing jobs as "expected fail" due to the incorrect setup of the channel.
Flags: needinfo?(hskupin)
Summary: Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Available update has been found → [tier-3] Intermittent test_direct_update.py TestDirectUpdate.test_update, test_fallback_update.py TestFallbackUpdate.test_update | AssertionError: Available update has been found

Comment 30

3 months ago
8 failures in 718 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-release: 7
* mozilla-esr52: 1

Platform breakdown:
* windows8-64: 3
* windows8-32: 3
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-06-26&endday=2017-07-02&tree=all
this appears fixed, do we know why this isn't failing anymore?
Whiteboard: [stockwell needswork] → [stockwell unknown]
This is a kinda meta bug for every issue which could cause updates not to be found. If it's unclear people star it against this bug, without proper investigation. :/
Please also see bug 1367319 in the dependency list, which should give a better error. But given the unclear future of the update tests I don't change anything right now.

Comment 34

2 months ago
2 failures in 720 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* windows8-64: 1
* windows8-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 35

2 months ago
5 failures in 888 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-release: 3
* mozilla-esr52: 2

Platform breakdown:
* windows8-64: 4
* windows8-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 36

a month ago
11 failures in 901 pushes (0.012 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-release: 11

Platform breakdown:
* linux64: 3
* windows8-64: 2
* windows8-32: 2
* windows7-64: 2
* windows7-32: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1316567&startday=2017-08-07&endday=2017-08-13&tree=all
Status: NEW → RESOLVED
Last Resolved: 7 days ago
Keywords: bulk-close-intermittents
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.