1230134 - [tc-gip] Make the logs private

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Johan Lorenzo [:jlorenzo]]

Once bug 1132214 lands, we'll be able to make the logs accessible to specific people only.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: 1230134.patch

I changed the configuration, factorized the device.json, and put the jobs back in the job flags.

By the way, when you see "log" in this context:
> artifacts:
>   'private/logs':
>     type: directory
>     path: '/home/worker/upload/logs/'
>     expires: '{{#from_now}}1 year{{/from_now}}'
>
> log: 'private/logs/live.log'

you might be confused. Maybe we should rebrand "log" into "live_log_location". What do you think, Greg?

Comment 2

[Greg Arndt [:garndt]]

Comment on attachment 8695273 [details] [diff] [review]
1230134.patch

Review of attachment 8695273 [details] [diff] [review]:
-----------------------------------------------------------------

Left a comment about the artifact namespace.  I think we should probably start following this new process now to limit the people that can see this information.  I can help out with whatever necessary.

::: testing/taskcluster/tasks/phone_test.yml
@@ +29,5 @@
>        GAIA_REV: '{{{gaia_rev}}}'
>  
>      # All builds share a common artifact directory for ease of uploading.
>      artifacts:
> +      'private/device.json':

So, now that we're starting to carve out sections of namespaces for private artifacts, we probably should start following the process here. Check out the docs [1]  that dustin put together (it's awesome and very informative).  Specifically the artifacts section.  right now we broadly give a lot of employees access to "private/*" so we would to evaluate if this should be more specific to those that need access to this information.  Perhaps we setup a namespace for the b2g/gaia team such that anyone that gets a scope to get an artifact for that prefix can download these files, but no one outside of that namespace could do so.  Hopefully that makes sense and doesn't complicate things much.  Just figured we should start following this new process going forward.


[1] http://docs.taskcluster.net/devel/namespaces/

@@ +40,5 @@
>          path: '/home/worker/upload/logs/'
>          expires: '{{#from_now}}1 year{{/from_now}}'
>  
> +
> +    log: 'private/logs/live.log'

See comment about artifact namespaces.

::: testing/taskcluster/tasks/tests/flame_kk_gaia_ui_test_functional.yml
@@ -36,5 @@
>          --type=b2g-dsds
>          --log-mach=-
>          --log-raw=/home/worker/upload/logs/raw.log
>          gaia/source/tests/python/gaia-ui-tests/gaiatest/tests/functional/manifest.ini
> -    artifacts:

Great clean up, thanks for moving that into the best test definition.

Comment 3

[Greg Arndt [:garndt]]

Comment on attachment 8695273 [details] [diff] [review]
1230134.patch

Removing flag until we work out the namespaces for these artifacts and get that merged into the docs.

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: 1230134-1.diff

I propose to use the namespace "b2g". It's consistent with the b2g-inbound and the b2g-repo.

Comment 5

[Greg Arndt [:garndt]]

This patch looks good to me but I don't want to flag r+ just yet until the worker lands with the other changes.  Just to prevent this from accidentally landing

Comment 6

[Greg Arndt [:garndt]]

I'm re-enabling the workers now with your changes to the worker code.  Things should be ok, but if you want to push these changes and monitor closely first that would be good :) just to make sure logs are private as you intend them.

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: MozReview Request: Bug 1230134 - [tc-gip] Make the logs private r=garndt

Review commit: https://reviewboard.mozilla.org/r/28521/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/28521/

Comment 8

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8699996 [details]
MozReview Request: Bug 1230134 - [tc-gip] Make the logs private r=garndt

I had some merge issues. The changes are the same than the previous patch. Carrying r+.

Comment 9

[Johan Lorenzo [:jlorenzo]]

https://hg.mozilla.org/integration/b2g-inbound/rev/6b0936bf07f1f4110864b60779e7ae049edc0007
Bug 1230134 - [tc-gip] Make the logs private r=garndt

Comment 10

[Johan Lorenzo [:jlorenzo]]

https://hg.mozilla.org/integration/b2g-inbound/rev/85ee5c3af6321d5b9c2f8c3e4640eceac8884467
Backed out Bug 1230134 - [tc-gip] Make the logs private

Comment 11

[Johan Lorenzo [:jlorenzo]]

I backed out the changes. All the tc-gip jobs failed, with an error like this one:

> [taskcluster] taskId: WJOESChxRQaMksS8A-Hgqg, workerId: i-55be5796
> [taskcluster] using cache "tc-vcs" -> /home/worker/.tc-vcs
> 
> [taskcluster:error] `task.payload` format is invalid json schema errors:
>  {
>   "validation": {
>     "artifacts": {
>       "schema": {
>         "log": {
>           "type": "object"
>         }
>       }
>     }
>   }
> }
> [taskcluster] Unsuccessful task run with exit code: -1 completed in 17.913 seconds

Comment 12

[Greg Arndt [:garndt]]

Ok, I know how to fix this, and will submit a PR for you to review once I'm done.

Comment 13

[Greg Arndt [:garndt]]

I also will add a test for this in docker-worker.  WE have other schema tests, but it doesn't test for this particular type of payload.

Comment 14

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8699996 [details]
MozReview Request: Bug 1230134 - [tc-gip] Make the logs private r=garndt

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28521/diff/1-2/

Comment 15

[Johan Lorenzo [:jlorenzo]]

https://hg.mozilla.org/integration/b2g-inbound/rev/ba6c0f7881c68dcaa76c4325a2b4f09945f395f1
Bug 1230134 - [tc-gip] Make the logs private r=garndt

Comment 16

[Johan Lorenzo [:jlorenzo]]

On the stage instance of treeherder, the jobs are back executing: https://treeherder.allizom.org/#/jobs?repo=b2g-inbound&revision=ba6c0f7881c6

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ba6c0f7881c6