Closed Bug 1208102 Opened 9 years ago Closed 9 years ago

Treeherder is submitting the classification dates to ElasticSearch with the wrong timezone


(Tree Management :: Treeherder, defect, P2)



(Not tracked)



(Reporter: emorley, Assigned: emorley)




(1 file)


There are both entries from 23rd and 24th Sept.

Must be more timezone related fun, sigh.
This will likely be fixed by bug 1209339.
Depends on: 1209339
(In reply to Ed Morley [:emorley] from comment #1)
> This will likely be fixed by bug 1209339.

Hmm or not.

So for this page:

Classifications are shown from midday 23rd sept through to almost 8am 24th sept, so this must be a pacific vs UTC timezone issue.

The API call for that page is:

Picking the last occurrence:
* Job start time as shown in the OF UI: 24 Sep 2015, 07:38 UTC
* Job start time shown in the raw log: 1443080294.8 -> 09/24/2015 @ 7:38am (UTC)

From the OF API response:
      "buildtype": "debug",
      "branch": "try",
      "treeherder_job_id": 11835104,
      "date": "2015-09-23",
      "bug": "1207774",
      "machinename": "t-w864-ix-040",
      "classification_time": "1443391353",
      "platform": "windows8-64",
      "starttime": "1443080294",
      "test": "mochitest-browser-chrome-2",
      "revision": "a96306a84fc8"

classification_time -> 09/27/2015 @ 10:02pm (UTC)
starttime -> matches what the raw log says: 09/24/2015 @ 7:38am (UTC)

So basically "date" is wrong.

This is confirmed by looking at the raw data for that classification in ElasticSearch:
    "buildtype": "debug",
    "treeherder_job_id": 11835104,
    "os": "windows8-64",
    "tree": "try",
    "who": "SNIP",
    "buildname": "Windows 8 64-bit try debug test mochitest-browser-chrome-2",
    "date": "2015-09-23",
    "bug": "1207774",
    "machinename": "t-w864-ix-040",
    "rev": "a96306a84fc8",
    "starttime": "1443080294",
    "timestamp": "1443391353",
    "type": "Mochitest Browser Chrome"


So Treeherder is in fact submitting the 'date' field in Pacific, not UTC after all, even though the rabbitmq timezone is UTC:

[emorley@treeherder-rabbitmq1.private.scl3 ~]$ date
Tue Sep 29 11:17:56 UTC 2015
Component: OrangeFactor → Treeherder
Priority: -- → P2
Summary: Results from the next day appearing in single page view → Treeherder is submitting the classification dates to ElasticSearch with the wrong timezone

[emorley@treeherder-rabbitmq1.private.scl3 treeherder-service]$ date
Tue Sep 29 11:40:50 UTC 2015

[emorley@treeherder-rabbitmq1.private.scl3 ~]$ cat /etc/sysconfig/clock

[emorley@treeherder-rabbitmq1.private.scl3 ~]$ ls -l /etc/localtime
lrwxrwxrwx 1 root root 23 May 30  2014 /etc/localtime -> /usr/share/zoneinfo/UTC

[emorley@treeherder-rabbitmq1.private.scl3 treeherder-service]$ ../venv/bin/python ./ shell
>>> import datetime
datetime.datetime(2015, 9, 29, 4, 37, 11, 902909)

[emorley@treeherder-rabbitmq1.private.scl3 treeherder-service]$ ../venv/bin/python
>>> import datetime
datetime.datetime(2015, 9, 29, 11, 39, 46, 530121)

So it looks like the Django python environment has overridden the timezone datetime uses, even though the Django docs imply that the settings/ line of:
TIME_ZONE = "America/Los_Angeles"
...should only affect internal Django functionality.

I've found this that shows that Django does mess with the environment variable TZ, which is presumably why datetime behaviour changes:

Either way, Treeherder should always have been using utcfromtimestamp() not fromtimestamp() (the latter is a big footgun, why it's called that in the std python library I don't know; localfromtimestamp() would be much more useful) - so let's fix that. In another bug we can look at changing TIME_ZONE in settings/ to UTC, but there might be wider implications so not going to do that here.
Attachment #8667231 - Flags: review?(mdoglio) → review+
Commit pushed to master at
Bug 1208102 - Submit the job start time date as UTC to ElasticSearch

Since `.fromtimestamp()` is being overridden by Django's TIME_ZONE and
so uses Pacific Time, even though rabbitmq's server time is UTC.
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.