1210466 - test_call_log_groups.py : AssertionError: '12/01/2014' != u'12/01/14'

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Max Ivanov[:MaxIvanov]]

Description: Test case fails with "AssertionError"

Steps:
1. Update a flame device to 20151001030229
2. Run this test case http://lxr.mozilla.org/gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer/test_call_log_groups.py

Actual result:
"AssertionError"

Expected result:
Test completes successfully.

Repro Rate: Reproduced via Jenkins (0/5 passing)
http://jenkins1.qa.scl3.mozilla.com/job/flame-kk.ui.adhoc.bitbar/265/HTML_Report/

Traceback (most recent call last):
File "/var/lib/jenkins/jobs/flame-kk.ui.adhoc.bitbar/workspace/.env/lib/python2.7/site-packages/marionette_client-0.17-py2.7.egg/marionette/marionette_test.py", line 296, in run
testMethod()
File "/var/lib/jenkins/jobs/flame-kk.ui.adhoc.bitbar/workspace/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer/test_call_log_groups.py", line 52, in test_call_log_groups
self.assertEqual('{:%m/%d/%Y}'.format(self.past_date_1), group_calls[2].header_text)
AssertionError: '12/01/2014' != u'12/01/14'

Comment 1

[Max Ivanov[:MaxIvanov]]

Attachment: index.png

Comment 2

[Martijn Wargers (dead)]

This started failing because of bug 1206306.

Comment 3

[GitHub Autolander Bot]

Attachment: [gaia] mwargers:1210466 > mozilla-b2g:master

Comment 4

[Martijn Wargers (dead)]

Comment on attachment 8668932 [details] [review]
[gaia] mwargers:1210466 > mozilla-b2g:master

This fixes the failure for me. I also had to change the timezone for this test, otherwise I would get this failure:
TEST-UNEXPECTED-FAIL | test_call_log_groups.py TestCallLogGroups.test_call_log_groups | AssertionError: u'12/01/14' != u'11/30/14'
- 12/01/14
?  ^  -
+ 11/30/14
?  ^ +


Traceback (most recent call last):
  File "/Users/mwargers/.virtualenvs/zapion_test2/lib/python2.7/site-packages/marionette_client-0.17-py2.7.egg/marionette/marionette_test.py", line 296, in run
    testMethod()
  File "/Users/mwargers/B2G/gaia_clean/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer/test_call_log_groups.py", line 53, in test_call_log_groups
    self.assertEqual(u'{:%m/%d/%y}'.format(self.past_date_1), group_calls[2].header_text)
TEST-INFO took 93998ms


Not exactly sure why that is needed, but I guess it has something to do with the usage of UTC, which is then converted to a certain time on the phone.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

Why this error did show up for me when I was working on the original patch? I would have investigate on my own. Sorry for that, 
Martijn!

Comment 6

[Martijn Wargers (dead)]

Zibi, no problem. Gaia UI tests are not running on Treeherder, so there is no way you could have known. This has been going on for a while and is not an ideal situation. In the next quarter we want to get it back to report to Treeherder.

Comment 7

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8668932 [details] [review]
[gaia] mwargers:1210466 > mozilla-b2g:master

git grep '{:%m/%d/%Y}' showed only the 2 lines you changed, which is good.

I agree with you regarding the timezone change. It seems related to bug 1206306#c8> In order to make the test less error-prone I think we can force the timezone to UTC.

Comment 8

[Martijn Wargers (dead)]

I removed this part:
-        self.data_layer.set_setting('time.timezone', 'Europe/Istanbul')

And started an adhoc job to see if this works fine in the Mountain View lab:
http://jenkins1.qa.scl3.mozilla.com/view/Mozilla%20Lab/job/flame-kk.ui.adhoc/894/

I'm still surprised that I need this setting locally to get this test passing.
Setting the timezone to Europe/Paris also makes the test pass for me, but setting the timezone Europe/Amsterdam, I get the same failure.

Comment 9

[Martijn Wargers (dead)]

Right adhoc job this time: http://jenkins1.qa.scl3.mozilla.com/view/Mozilla%20Lab/job/flame-kk.ui.adhoc/895/

Comment 10

[Martijn Wargers (dead)]

Ok, the adhoc job is passing.

Comment 11

[No-Jun Park [:njpark]]

Comment on attachment 8668932 [details] [review]
[gaia] mwargers:1210466 > mozilla-b2g:master

LGTM

Comment 12

[Martijn Wargers (dead)]

Merged: https://github.com/mozilla-b2g/gaia/commit/7ff515a65174cebf484cf0e17af82144103a73f0

Zibi, do you have an idea why I had to set the time.timezone setting to 'Europe/Istanbul' or 'Europe/Paris' to get not the failure as in comment 4?
Setting the time.timezone setting to 'Europe/Amsterdam' also gives me that failure.

Comment 13

[Martijn Wargers (dead)]

Zibi, see question in comment 12.

Comment 14

[Zibi Braniecki [:zbraniecki][:gandalf]]

My guess is that your test is timezone dependent so when it's run in one timezone it may show one day, and in another, another.

You should force all assertions to either be timezone dependent on both sides, or timezone independent on both sides (forced to UTC).

Comment 15

[Martijn Wargers (dead)]

I don't see how this test is timezone dependant and even if it was how it can be that it works and not works in the cases mentioned in comment 14. There should have been no difference between 'Europe/Paris' and 'Europe/Amsterdam' timezone.