979498 - Improve the way we publish results from jenkins for b2g

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

We currently use the "publish over ssh plugin" to sychronize the eideticker dashboard with eideticker.mozilla.org:

https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin

This is starting to get really slow. I think this is because it's copying everything over, every time. It would be nice if we could use rsync here to do this, perhaps just using a modified version of the existing script that's inside eideticker?

https://github.com/mozilla/eideticker/blob/master/bin/sync-dashboard.sh

Comment 1

[William Lachance (:wlach)]

See also bug 978806: sync should be a seperate job.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Allow overriding of dashboard local path and remote username

Local path could now be different if --output-path is used, and allowing the username to be specified allows us to use alternate destinations (useful during testing)

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8404739 [details] [diff] [review]
Allow overriding of dashboard local path and remote username

Review of attachment 8404739 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, this shouldn't break anything we have in production currently.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed as:
https://github.com/mozilla/eideticker/commit/eafba4587bc33f89fa0de6d8b06f43cfd7579a93

Comment 5

[William Lachance (:wlach)]

As discussed on irc, the way to fix this is the same as the way to fix bug 950588. It looks something like:

(1) add a mode to copy-dashboard.py to just copy the raw numbers (not the metadata/videos/profiles) and device metadata o a local directory
(2) update jenkins to pull existing results from server to a custom directory inside the workspace before running the tests
(3) when running the tests, update results inside the custom directory
(4) finally, sync the custom directory to the server (or soon, ec2 bucket) after the tests have run

Comment 6

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #5)
> As discussed on irc, the way to fix this is the same as the way to fix bug
> 950588. It looks something like:

That should be 960588.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Updated eideticker-ci:
https://github.com/mozilla/eideticker-ci/commit/756894cbea5b5c858fea1186f645f7e53b892ce2

Comment 9

[William Lachance (:wlach)]

Attachment: Allow to skip downloading metadata when copying dashboard

As discussed on irc. This patch will make it faster to download a copy of existing dashboard data for updating purposes (where we don't need the metadata).

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The rsync is now working. I tested by pushing to a new dashboard at b2g-test, and the sync took ~8 minutes. The next build (currently running) will push to the original dashboard and should take considerably less time as only the last few results will need to be pushed.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8404823 [details] [diff] [review]
Allow to skip downloading metadata when copying  dashboard

Review of attachment 8404823 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with or without the argument changed to --skip-metadata. Once this lands I'll prepare a patch for the CI to wipe out the dashboard from the workspace, and copy-dashboard.py before running the tests. This will mean the workspace will only contain historical testdata and only the most recent test metadata. This will save considerable disk space on the system. Also, by archiving this dashboard delta and using rsync we should also prevent accidentally wiping out data in the production dashboard, or at least make it relatively easy to restore missing data.

::: bin/copy-dashboard.py
@@ +67,4 @@
>  parser.add_option("--full-mirror", action="store_true",
>                    default=False, dest="full_mirror",
>                    help="Download videos, profiles to disk")
> +parser.add_option("--no-metadata", action="store_false",

I think a better argument would be --skip-metadata

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

As noted on bug 929222 I'll also make the sync occur after each test rather than at the end, so the dashboard is updated more frequently and a test failure will not prevent any successful results form being published.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Hunt (:davehunt) from comment #10)
> The rsync is now working. I tested by pushing to a new dashboard at
> b2g-test, and the sync took ~8 minutes. The next build (currently running)
> will push to the original dashboard and should take considerably less time
> as only the last few results will need to be pushed.

01:49:40.765 + ./bin/sync-dashboard.sh
01:49:43.812 sending incremental file list
...
01:49:49.229 sent 6526862 bytes  received 5003 bytes  687564.74 bytes/sec
01:49:49.229 total size is 803105381  speedup is 122.95

Comment 14

[William Lachance (:wlach)]

(In reply to Dave Hunt (:davehunt) from comment #11)

> ::: bin/copy-dashboard.py
> @@ +67,4 @@
> >  parser.add_option("--full-mirror", action="store_true",
> >                    default=False, dest="full_mirror",
> >                    help="Download videos, profiles to disk")
> > +parser.add_option("--no-metadata", action="store_false",
> 
> I think a better argument would be --skip-metadata

Agreed. Pushed with that changed:

https://github.com/mozilla/eideticker/commit/7f4fb42c61b5230a5ff84ff23fb2de61011d5f48

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is all finished. We now copy down the dashboard (without metrics) before the test is run, and sync the dashboard between tests. This uses negligible disk space, and means the uploading of the dashboard is more frequent and much faster.