807273 - OrangeFactor should use a consistent timezone throughout, otherwise failure counts per day appear incorrect

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

Don't know if related to bug 807272?

(In reply to Phil Ringnalda (:philor) from bug 795206 comment #32)
> I don't think Orange Factor actually, you know, *works*. I'd guess without
> having kept track that this is up to more like 20-30 failures per weekday,
> certainly not OF's 16 trunk, 17 total for the week (there was a push on m-a
> or m-b yesterday where I starred this twice on one *push*, so 1 off-trunk in
> the last week is certainly wrong).

http://brasstacks.mozilla.com/orangefactor/?display=Bug&tree=trunk&startday=2012-10-24&endday=2012-10-31&bugid=795206


(In reply to Phil Ringnalda (:philor) from bug 792873 comment #10)
> And, sorry, but we're back to pasting logs - I thought Orange Factor would
> take the place of them, but it apparently ignores a huge percentage of them,
> since rather than "6 failures on trunk in the last week" it should be more
> like "60 failures on trunk in the last 12 hours."
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=16607399&tree=Mozilla-
> Inbound

http://brasstacks.mozilla.com/orangefactor/?display=Bug&tree=trunk&startday=2012-10-24&endday=2012-10-31&bugid=792873

Comment 1

[Phil Ringnalda (:philor)]

Bugzilla says bug 610183 was starred 34 times yesterday; https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=610183&startday=2012-10-28&endday=2012-11-04&tree=all says 6.

Comment 2

[Ed Morley [:emorley]]

jgriffin, do you have any ideas about this? I use OrangeFactor as the basis for deciding which bugs/people to hassle, so if the toporange order is inaccurate I've been giving some people grief unnecessarily :-)

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Ed Morley [:edmorley UTC+0] from comment #2)
> jgriffin, do you have any ideas about this? I use OrangeFactor as the basis
> for deciding which bugs/people to hassle, so if the toporange order is
> inaccurate I've been giving some people grief unnecessarily :-)

I think this may just be a timing issue.  From bug 610183:

 TinderboxPushlog Robot 2012-11-03 18:03:58 PDT

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=16730883&tree=Mozilla-Inbound
Rev3 WINNT 6.1 mozilla-inbound debug test jsreftest on 2012-11-03 17:22:08
slave: talos-r3-w7-040

This comment (and the failure) were on 2012-11-03, but it appears in OF under Nov 4.

Possibly this is just a PST/UTC issue, or something similar.

Comment 4

[Phil Ringnalda (:philor)]

Ah, interesting, you not only have to know that today doesn't exist until tomorrow (an enormous knock on the suitability of OF to be a replacement for bug comments), you also have to know that tomorrow starts at a different time than it does for any other Mozilla property :)

Comment 5

[Jonathan Griffin (:jgriffin)]

(In reply to Phil Ringnalda (:philor) from comment #4)
> Ah, interesting, you not only have to know that today doesn't exist until
> tomorrow (an enormous knock on the suitability of OF to be a replacement for
> bug comments), you also have to know that tomorrow starts at a different
> time than it does for any other Mozilla property :)

Yes, we should definitely figure out the root cause of this and fix it.  :)

Comment 6

[Jonathan Griffin (:jgriffin)]

I'm thinking the right fix is to change this line not to convert to UTC:

http://hg.mozilla.org/users/jgriffin_mozilla.com/mozautolog/file/ccbbea99ccdc/mozautolog/mozautolog.py#l310

But I'll have to test that, and I don't have time in the next couple of days.

Comment 7

[Ed Morley [:emorley]]

Morphing this to be about the issue in comment 6.

Bug 797811 is now the meta for making sure all failures appear correctly in OF.

Comment 8

[Mark Côté [:mcote]]

Btw the new version of flot that's due out in the next few months has timezone support, so we could conceivably add an option to display in the user's timezone.

Comment 9

[Ed Morley [:emorley]]

Not actively working on this right now, may come back to later, but for now sticking back in the pool.

Comment 10

[Ed Morley [:emorley]]

Wontfix in favour of OrangeFactor v2 which will consume Treeherder's API, and likely be written from scratch.

Comment 11

[Ed Morley [:emorley]]

So navigating to a page like:
http://localhost:8080/?display=OrangeFactor&tree=trunk&startday=2015-09-01&endday=2015-09-03

Results in json-pushes calls of form:
http://hg.mozilla.org/mozilla-central/json-pushes?startdate=2015-09-01&enddate=2015-09-04
http://hg.mozilla.org/integration/mozilla-inbound/json-pushes?startdate=2015-09-01&enddate=2015-09-04
http://hg.mozilla.org/integration/b2g-inbound/json-pushes?startdate=2015-09-01&enddate=2015-09-04
http://hg.mozilla.org/integration/fx-team/json-pushes?startdate=2015-09-01&enddate=2015-09-04

According to pushloghtml for the same params, this gets converted to a range of the form:
  after 2015-09-01 00:00:00, before 2015-09-04 00:00:00

Where by inspection of the commits/push info the pushlog info appears to be UTC.

On the ElasticSearch classification data side:

"date" (which gets returned from the OF API as "date") is currently the date the job was scheduled, according to treeherder-rabbitmq1's localtime, which is UTC. See:
https://github.com/mozilla/treeherder/blob/b8872e574249fb9226628b22ce8fd83f81899248/treeherder/etl/classification_mirroring.py#L46-L49

"starttime" (which gets returned from the OF API as "starttime") is the Unix timestamp of when the job started running. See:
https://github.com/mozilla/treeherder/blob/b8872e574249fb9226628b22ce8fd83f81899248/treeherder/etl/classification_mirroring.py#L56

"timestamp" (which gets returned from the OF API as "classification_time") is the Unix timestamp of when the failure was classified. See:
https://github.com/mozilla/treeherder/blob/b8872e574249fb9226628b22ce8fd83f81899248/treeherder/etl/classification_mirroring.py#L61

Now given that all of our ES data is bucketed by a UTC 'date' (which is just a YYYY-MM-DD string), then doing timezone conversions will require a complete rewrite of OrangeFactor to instead use "starttime" (with likely ElasticSearch perf implications, since there won't be discrete buckets).

As such, let's just stick to UTC throughout. We should:
1) Make the timezone clearer in the UI
2) Double check the pushlog handling timezone conversion isn't broken (it's doing conversion at the moment, but probably shouldn't be)

Comment 12

[Ed Morley [:emorley]]

Attachment: Part 1: Add comments to make date formats/timezones more obvious

Also re-arrange the date processing in getPushesByDate() to be slightly clearer.

Comment 13

[Ed Morley [:emorley]]

Attachment: Part 2: Use gmtime() rather than localtime() in getPushesByDate()

Since the YYYY-MM-DD strings returned from getPushesByDate() will be used against those from ElasticSearch, so need to be UTC. However using localtime() converts to the timezone on the server, which in the case of brasstacks is Pacific Time.

Comment 14

[Ed Morley [:emorley]]

Attachment: Part 3: Label the dates in the UI as being in UTC

Comment 15

[Jonathan Griffin (:jgriffin)]

Thanks for fixing this up.

Comment 16

[Ed Morley [:emorley]]

remote:   https://hg.mozilla.org/automation/orangefactor/rev/71edac61e339
remote:   https://hg.mozilla.org/automation/orangefactor/rev/2a70bcd7e5c7
remote:   https://hg.mozilla.org/automation/orangefactor/rev/94b531abea2e

Deployed this and bug 1202385:

[[e-mail]3 ~]$ cd ~/apps/orangefactor/src/orangefactor/ && hg pull -uv
pulling from http://hg.mozilla.org/automation/orangefactor/
searching for changes
all local heads known remotely
adding changesets
adding manifests
adding file changes
added 4 changesets with 5 changes to 3 files
resolving manifests
getting html/scripts/woo.bugs.js
getting html/scripts/woo.main.js
getting server/handlers.py
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
[[e-mail]3 orangefactor]$ logout
[[e-mail]3 ~]# cd /home/webtools/apps/orangefactor/src/orangefactor/ && ./deploy.sh
[[e-mail]3 orangefactor]# /etc/init.d/orangefactor stop; /etc/init.d/orangefactor start
stopping orangefactor                                      [  OK  ]
starting orangefactorspawn-fcgi: child spawned successfully: PID: 29601
                                                           [  OK  ]