1135991 - We generate today's buildjson file by using PT timezone instead of UTC

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

This is not blocking any of my work, however, it simply shows that there are some issues on what time we generate buildjson files.

For few hours every day, we generate the files, however, they don't contain any jobs completed for the last 6 hours before midnight.

Only jobs completed (endtime) within a UTC day are included into a days worth of status.

This is why build4hrs has probably been proven the best choice for jobs running on the same day.

We should probably change the cronjob to run according to UTC's midnight rather than PT's midnight.

Downloading a buildjson file after 4pm PT will always yield the same size (You have to wait 15 minutes for the next run).

#############################################################

armenzg@armenzg-thinkpad:~/repos/ci_tools$ python mozci/sources/.buildjson.py
1424649600 Mon, 23 Feb 2015 00:00:00  () Sun, 22 Feb 2015 16:00:00 -0800 (PST)
1424736000 Tue, 24 Feb 2015 00:00:00  () Mon, 23 Feb 2015 16:00:00 -0800 (PST)

armenzg@armenzg-thinkpad:~/repos/ci_tools$ ls -l ~/moz/patches/builds-2015-02-23_09_*
-rw-rw-r-- 1 armenzg armenzg 18641472 Feb 23 21:26 /home/armenzg/moz/patches/builds-2015-02-23_09_26_et.js.gz
-rw-rw-r-- 1 armenzg armenzg 18641472 Feb 23 21:39 /home/armenzg/moz/patches/builds-2015-02-23_09_39_et.js.gz

armenzg@armenzg-thinkpad:~/repos/ci_tools$ cat mozci/sources/.buildjson.py
from mozci.sources.buildjson import _fetch_buildjson_day_file
from mozci.utils.tzone import pacific_time as pt
from mozci.utils.tzone import utc_time as ut

builds = _fetch_buildjson_day_file("2015-02-23")

list = []
for job in builds:
    list.append(job["endtime"])

list.sort()
print "%s %s %s" % (list[0], ut(list[0]), pt(list[0]))
print "%s %s %s" % (list[-1], ut(list[-1]), pt(list[-1]))

Comment 1

[Armen [:armenzg]]

GH issue: https://github.com/armenzg/mozilla_ci_tools/issues/46

Comment 2

[Armen [:armenzg]]

dustin, do you have information on where we time the generation of a daily buildjson file?

It seems there is info about builds-4hr in here:
https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#report_crontask

It is probably in the same place where we need to adjust the time.
Could you please help me find the code for this cronjob?

Comment 3

[Chris Cooper [:coop] (he/him)]

[[e-mail]3 home]# crontab -u buildapi -l
# HEADER: This file was autogenerated at Thu Nov 27 15:43:14 -0800 2014 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: buildapi-waittimes
MAILTO="[e-mail]"
1 6 * * * /var/lib/buildapi/waittimes.sh
# Puppet Name: buildapi-today
MAILTO="[e-mail]"
*/15 * * * * /var/lib/buildapi/report-today.sh
# Puppet Name: buildapi-running
MAILTO="[e-mail]"
* * * * * /var/lib/buildapi/running.sh
# Puppet Name: buildapi-daily
MAILTO="[e-mail]"
0 0 * * * /var/lib/buildapi/report-daily.sh
# Puppet Name: buildapi-4hour
MAILTO="[e-mail]"
* * * * * /var/lib/buildapi/report-4hr.sh

Comment 4

[Chris Cooper [:coop] (he/him)]

report-daily.sh seems to be the one you want. That script does the following:

#!/bin/sh
buildapi_lib="/var/lib/buildapi"
buildapi_lock="/var/lock/buildapi"
buildapi_log="/var/log/buildapi"
buildapi_www="/data/releng/www/buildapi"
reporter_cfg="/data/releng/www/buildapi/reporter.cfg"
builddata_dir="/mnt/netapp/relengweb/builddata"
lockfile=${buildapi_lock}/builds-daily

(
    /usr/bin/flock -w 1 200 || exit
    python2.7 "${buildapi_www}/run-reporter" \
        --config ${reporter_cfg} -z \
        -o ${builddata_dir}/buildjson/builds-$(date -d yesterday +\%Y-\%m-\%d).js.gz \
        --startdate $(date -d yesterday +\%Y-\%m-\%d) \
        >> ${buildapi_log}/reporter-daily.log 2>&1
    rm -f ${lockfile}
) 200>${lockfile}

Comment 5

[Armen [:armenzg]]

Coop, I actually want report-today.sh. Could you please paste it for me?

If it also calls run-reporter, could you please help me find the code for that? I can't find it in mxr.
I think that run-reporter calls buildapi/scripts/reporter.py.

Thanks!

Comment 6

[Armen [:armenzg]]

coop, could you please help me grab such file?
I want to work on this tomorrow Friday.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #6)
> coop, could you please help me grab such file?
> I want to work on this tomorrow Friday.

Here you go:

http://people.mozilla.org/~coop/bug1135991/

Comment 8

[Armen [:armenzg]]

Attachment: report-today.diff

Notice that I tested this change by pointing to my user folder [1] and not by what this patch shows.

This generated a builds-2015-03-13.js.gz which contained jobs with endtimes within the expected utc day [2].
If run it against today's generated file I get similar data [3]

If you want me to, I can write a script that would compare the file I produce with the one produced automatically.
I can create a similar cronjob and I have a 15 minutes window.

By looking at this, I have noticed that we don't need the buildapi-daily crontask [4]
The reason is that we would not need it anymore as far as I can tell.

If we compare report-daily and report-today, this is what I can extract from it [5]
What we do in report-daily is create the builds-*.js.gz of yesterday (instead of today).
The --startdate also gives the date of yesterday and we're lucky that past midnight's PT day is always the same as UTC day.

Even today, without my changes, I actually don't think we need this crontask at all.
From 4pm PT (this time has probably changed -+1hour after the DST changes) all the way until midnight PT, we generate every 15 minutes yesterday's UTC builds-*.js.gz file.
It is only after midnight that we generate the new UTC day file and report-daily will re-generate one last time the builds day file of yesterday (which report-today already had been generating for the last 6 hours).

I have also noticed that we have stopped generating the l10n reports and the symlink is broken:
http://builddata.pub.build.mozilla.org/buildjson/l10n_reports/2014-03-25.txt

[1]
[[e-mail]3 bapi]# diff -pU 1 report-today.new.sh report-today.final.sh
--- report-today.new.sh	2015-03-13 07:03:21.288497766 -0700
+++ report-today.final.sh	2015-03-13 07:10:24.622107923 -0700
@@ -1,9 +1,8 @@
 #!/bin/sh
-azg="/home/armenzg/bapi"
 buildapi_lib="/var/lib/buildapi"
-buildapi_lock=${azg}
-buildapi_log=${azg}
+buildapi_lock="/var/lock/buildapi"
+buildapi_log="/var/log/buildapi"
 buildapi_www="/data/releng/www/buildapi"
 reporter_cfg="/data/releng/www/buildapi/reporter.cfg"
-builddata_dir=${azg}
+builddata_dir="/mnt/netapp/relengweb/builddata"
 lockfile=${buildapi_lock}/builds-today

[2]
MacAir ci_tools git:[master*] $ ls -l builds-2015-03-13.js
-rw-r--r--  1 armenzg  staff  137917769 13 Mar 10:07 builds-2015-03-13.js
MacAir ci_tools git:[master*] $ python scripts/misc/dump_buildjson_endtimes.py
1426204801 Fri, 13 Mar 2015 00:00:01 +0000 (UTC) Thu, 12 Mar 2015 16:00:01 -0800 (PST)
1426255394 Fri, 13 Mar 2015 14:03:14 +0000 (UTC) Fri, 13 Mar 2015 06:03:14 -0800 (PST)

[3]
MacAir ci_tools git:[master*] $ python scripts/misc/dump_buildjson_endtimes.py
1426204801 Fri, 13 Mar 2015 00:00:01 +0000 (UTC) Thu, 12 Mar 2015 16:00:01 -0800 (PST)
1426255195 Fri, 13 Mar 2015 13:59:55 +0000 (UTC) Fri, 13 Mar 2015 05:59:55 -0800 (PST)
MacAir ci_tools git:[master*] $ ls -l builds-2015-03-13.js
-rw-r--r--  1 armenzg  staff  137271624 13 Mar 10:14 builds-2015-03-13.js

[4]
[[e-mail]3 bapi]# diff -pU 1 crontab.orig.txt crontab.new.txt
--- crontab.orig.txt	2015-03-13 06:57:30.120017612 -0700
+++ crontab.new.txt	2015-03-13 07:18:23.689592905 -0700
@@ -13,5 +13,2 @@ MAILTO="[e-mail]"
 * * * * * /var/lib/buildapi/running.sh
-# Puppet Name: buildapi-daily
-MAILTO="[e-mail]"
-0 0 * * * /var/lib/buildapi/report-daily.sh
 # Puppet Name: buildapi-4hour

[5]
[[e-mail]3 bapi]# diff -U2 report-daily.orig.sh report-today.orig.sh
--- report-daily.orig.sh	2015-03-13 06:58:21.807824032 -0700
+++ report-today.orig.sh	2015-03-13 07:03:22.180511688 -0700
@@ -6,13 +6,25 @@
 reporter_cfg="/data/releng/www/buildapi/reporter.cfg"
 builddata_dir="/mnt/netapp/relengweb/builddata"
-lockfile=${buildapi_lock}/builds-daily
+lockfile=${buildapi_lock}/builds-today
+builds_today=${builddata_dir}/buildjson/builds-$(date +\%Y-\%m-\%d).js.gz
+builds_yesterday=${builddata_dir}/buildjson/builds-$(date -d yesterday +\%Y-\%m-\%d).js.gz

 (
     /usr/bin/flock -w 1 200 || exit
+
     python2.7 "${buildapi_www}/run-reporter" \
         --config ${reporter_cfg} -z \
-        -o ${builddata_dir}/buildjson/builds-$(date -d yesterday +\%Y-\%m-\%d).js.gz \
-        --startdate $(date -d yesterday +\%Y-\%m-\%d) \
-        >> ${buildapi_log}/reporter-daily.log 2>&1
+        -o ${builds_today} \
+        --startdate $(date +\%Y-\%m-\%d) \
+        >> ${buildapi_log}/reporter-today.log 2>&1
+
+    # also generate the l10n report, based on that report and yesterday's
+    python2.7 ${buildapi_www}/buildapi/buildapi/scripts/l10n_report.py \
+        -o ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt \
+        ${builds_yesterday} ${builds_today} \
+        >> ${buildapi_log}/reporter-today.log 2>&1
+
+    # and fix up some links
+    ln -sf ${builddata_dir}/buildjson/l10n_reports/$(date +\%Y-\%m-\%d).txt ${builddata_dir}/buildjson/l10n_reports/latest.txt
     rm -f ${lockfile}
 ) 200>${lockfile}

Comment 9

[Armen [:armenzg]]

To rephase the issue.
All throughout a PT day (from midnight to midnight) we generate today's buildjson file (e.g. builds-2015-03-13.js.gz) even thought the contents of the file only contain jobs that completed by the end of the UTC day.
This means that after UTC midnight (it used to be 4pm PT), we generate again and again the same file every 15 minutes with the same contents (all jobs that completed within UTC's day).

This means that after 4pm/5pm PT until midnight PT:
* PT day is today (e.g. 2015-03-13)
* UTC day is tomorrow (e.g. 2015-03-14)

My patch changes the return of a day date to be UTC instead of PT.
This means that after 4pm/5pm we will call the scripts with a UTC day instead of a PT day.

From
--startdate $(date +\%Y-\%m-\%d)
to
--startdate $(date --utc +\%Y-\%m-\%d)

Comment 10

[Armen [:armenzg]]

Or even more clearly.

After 4pm/5pm PT we should be generating:
http://builddata.pub.build.mozilla.org/builddata/buildjson/builds-2015-03-14.js.gz (New UTC day)
instead of:
http://builddata.pub.build.mozilla.org/builddata/buildjson/builds-2015-03-13.js.gz

Comment 11

[Armen [:armenzg]]

I'm also trying to write code to verify that we download these files properly.
In some way, mozci downloaded and decompressed for jmaher one of those files but it was incomplete. [1][2][3]

The current download code is in here [4]. requests decompresses it automatically which means that the file we write is uncompressed and I can't compare it to the header's content-length.[5]

How would you verify that we transfered the file properly?

I'm looking at downloading the file raw and maybe create an md5 for it or some sort of metadata so I don't keep both the compress and uncompressed files.

Maybe I can change report-daily to generate the hash for all of those buildjson files.

[1]
MacAir ci_tools git:[master*] $ ls -l builds-2015-03-04.*
-rw-r--r--  1 armenzg  staff  241080700  5 Mar 03:36 builds-2015-03-04.good.js
-rw-r--r--  1 armenzg  staff  134067492 13 Mar 14:20 builds-2015-03-04.js

[2] http://people.mozilla.org/~jmaher/builds-2015-03-04.js
[3] http://builddata.pub.build.mozilla.org/builddata/buildjson/builds-2015-03-04.js.gz
[4] https://github.com/armenzg/mozilla_ci_tools/blob/master/mozci/sources/buildjson.py#L21
[5] (Pdb) req.headers
{'content-length': '23338306', 'content-encoding': 'gzip', 'x-backend-server': 'web2.releng.webapp.scl3.mozilla.com', 'accept-ranges': 'bytes', 'expires': 'Fri, 13 Mar 2015 21:16:03 GMT', 'keep-alive': 'timeout=5, max=718', 'server': 'Apache', 'last-modified': 'Thu, 05 Mar 2015 08:36:52 GMT', 'connection': 'Keep-Alive', 'etag': '"1641d42"', 'cache-control': 'max-age=60', 'date': 'Fri, 13 Mar 2015 21:15:03 GMT', 'access-control-allow-origin': '*', 'content-type': 'text/plain; charset=UTF-8'}

Comment 12

[Armen [:armenzg]]

I'm currently the only consumer of these *day* files.
There's no risk AFAIK and it is a very simple change.
I'm only adding --utc to every date call.

Comment 13

[Armen [:armenzg]]

Thanks for the review and deployment!
I see the change on buildapi.

Since we still run report-daily.sh, the last generation time still shows after midnight [1].
I will check after 5pm what the timestamp is for builds-2015-03-24.js.gz and make sure that builds-2015-03-25.js.gz starts being generated.

[1]
[ ]	builds-2015-03-24.js.gz	24-Mar-2015 05:31 	12M	 
[ ]	builds-2015-03-23.js.gz	24-Mar-2015 00:03 	20M

Comment 14

[Armen [:armenzg]]

Woohoo! We did it! We have the data of the jobs from the UTC future!

[ ]	builds-2015-03-24.js.gz	24-Mar-2015 16:47 	25M	 
[ ]	builds-2015-03-25.js.gz	24-Mar-2015 17:00 	123 	 

We now can have at most 15 minutes of a gap! :P
At midnight we fix this gap.
I should probably change the other report and change the cronjob to run at 7pm every day since it is always in tomorrow's UTC time.

Comment 15

[Armen [:armenzg]]

Attachment: report-daily.diff

Same idea as the previous patch.

[[e-mail]3 ~]# date -d yesterday +\%Y-\%m-\%d
2015-03-23
[[e-mail]3 ~]# date --utc -d yesterday +\%Y-\%m-\%d
2015-03-24
[[e-mail]3 ~]# date
Tue Mar 24 17:53:30 PDT 2015

Comment 16

[Armen [:armenzg]]

Attachment: crontab.diff

Run at 19:00 instead of midnight.
I think we could 17:01pm and still be fine all year.

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 8582799 [details] [diff] [review]
crontab.diff

Review of attachment 8582799 [details] [diff] [review]:
-----------------------------------------------------------------

Can you add a comment about why it's at 1900?

Comment 18

[Armen [:armenzg]]

Attachment: crontab.diff

Carrying forward the r+

Comment 19

[Armen [:armenzg]]

dustin, could you please land these two patches for me?
Thanks!

Comment 20

[Armen [:armenzg]]

And done!
Thank you for your help.

[ ]	builds-2015-03-23.js.gz	24-Mar-2015 00:03 	20M	 
[ ]	builds-2015-03-24.js.gz	25-Mar-2015 00:06 	25M	 
[ ]	builds-2015-03-25.js.gz	25-Mar-2015 19:03 	26M	 <-- notice last generation time
[ ]	builds-2015-03-26.js.gz	26-Mar-2015 05:46 	12M