926868 - gofaster is not refreshing since end of september

Status: RESOLVED FIXED

(Testing Graveyard :: GoFaster, defect)

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

The date suggests that gofaster is using http://build.mozilla.org for its data, as that was taken down on Sept 30 (bug 604688).

Comment 2

[Ed Morley [:emorley]]

I've found:
https://github.com/mozilla/gofaster/blob/master/bin/fetch-and-process-builddata.sh#L21

Comment 3

[Ed Morley [:emorley]]

Dustin, what should the new URL be? I can't seem to find it at my inital secure.oub.m.o guesses...? :-)

That csv is output here:
http://mxr.mozilla.org/build-central/source/braindump/reports/buildfaster_report.sh#9

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

That's on builddata:
  http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz

Comment 5

[Ed Morley [:emorley]]

Attachment: Update csv source to builddata.pub.build.m.o

build.m.o is no more, the csv now resides at:
http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz

Once this is checked in, happy for me to push to prod on brasstacks? Presuming I just pull the repo there?

Comment 6

[William Lachance (:wlach)]

Comment on attachment 817811 [details] [diff] [review]
Update csv source to builddata.pub.build.m.o

Review of attachment 817811 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. One minor suggestion.

::: bin/fetch-and-process-builddata.sh
@@ +17,5 @@
>      esac
>  done
>  
>  if [ $LOCAL == 0 ]; then
> +    wget http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz -O - | gunzip > $CSV_OUTPUT

So, in retrospect it might have been better to make the location a define at the top of the file like:

BUILDDATA_LOCATION=http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz

then just: 

wget $BUILDDATA_LOCATION -O - | gunzip > $CSV_OUTPUT

Comment 7

[William Lachance (:wlach)]

(In reply to Ed Morley (Mostly away until 21st October) [:edmorley UTC+1] from comment #5)
> Created attachment 817811 [details] [diff] [review]
> Update csv source to builddata.pub.build.m.o
> 
> build.m.o is no more, the csv now resides at:
> http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz
> 
> Once this is checked in, happy for me to push to prod on brasstacks?
> Presuming I just pull the repo there?

Oh, and I have no problem with you pushing to prod on brasstacks, though at this point I'm not sure if I'm the right one to ask. I'm still happy to review patches, but IMO someone more closely involved with build efficiency should be responsible for keeping things running (interested?).

Comment 8

[Ed Morley [:emorley]]

Thank you :-)

Landed with the switch to a define:
https://github.com/mozilla/gofaster/commit/1933afb85fa3a640cf3264d105b14d5c66234585

Comment 9

[Ed Morley [:emorley]]

[[e-mail]3 ~]# su - webtools
[[e-mail]3 ~]$ cd ~/apps/gofaster_dashboard/
[[e-mail]3 gofaster_dashboard]$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       bin/unit2-2.6
#       deploy.sh
#       lib64
#       src/dashboard/server/nohup.out
nothing added to commit but untracked files present (use "git add" to track)

[[e-mail]3 gofaster_dashboard]$ git pull
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (1/1), done.
remote: Total 4 (delta 3), reused 4 (delta 3)
Unpacking objects: 100% (4/4), done.
From git://github.com/mozilla/gofaster
   8be50ce..1933afb  master     -> origin/master
Updating 8be50ce..1933afb
Fast-forward
 bin/fetch-and-process-builddata.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

...

[[e-mail]3 ~]$ ./apps/gofaster_dashboard/bin/fetch-and-process-builddata.sh
--2013-10-16 16:59:55--  http://builddata.pub.build.mozilla.org/reports/buildfaster.csv.gz

Resolving builddata.pub.build.mozilla.org... 63.245.215.57
Connecting to builddata.pub.build.mozilla.org|63.245.215.57|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2162713 (2.1M) [text/plain]
Saving to: â?oSTDOUTâ??

100%[================================================>] 2,162,713   10.0M/s   in 0.2s

2013-10-16 16:59:55 (10.0 MB/s) - written to stdout [2162713/2162713]

-> http://brasstacks.mozilla.com/gofaster/#/executiontime/build now looks up to date :-)

(Filed bug 927664 for the files not tracked in the repo).

Comment 10

[Ed Morley [:emorley]]

Oh and the data is imported once a day, at midnight PDT:

[[e-mail]3 gofaster_dashboard]# crontab -l -u webtools
0 0 * * *   /home/webtools/apps/gofaster_dashboard/bin/fetch-and-process-builddata.sh
...

Comment 11

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #7)
> Oh, and I have no problem with you pushing to prod on brasstacks, though at
> this point I'm not sure if I'm the right one to ask. I'm still happy to
> review patches, but IMO someone more closely involved with build efficiency
> should be responsible for keeping things running (interested?).

Possibly... ;-)