1433157 - Please add ship-it related scopes to moz-tree:level:\d

Reporter

Description

•

6 years ago

In bug 1431764, we're adding a new worker type. Then, we would need:

> queue:create-task:????:scriptworker-prov-v1/shipit-dev
=> highest on moz-tree:level:3 and lowest on moz-tree:level:1

> queue:create-task:highest:scriptworker-prov-v1/shipit-v1
on moz-tree:level:3

> project:releng:ship-it:staging
on moz-tree:level:1

> project:releng:ship-it:production
on moz-tree:level:3


I hope previous experiences (like bug 1420459) make me get all the scopes right, this time :)

Dustin J. Mitchell [:dustin] (he/him)

Comment 1

•

6 years ago

I gave the `releng` LDAP group permisison to modify the moz-tree:* roles, so you should be able to do this now.

It sounds like these are gecko-only, so they should probably be added to moz-tree:level:L:gecko.

Johan Lorenzo [:jlorenzo]

Reporter

Comment 2

•

6 years ago

After looking at the current state of roles, I added:
> project:releng:ship-it:staging
> queue:create-task:very-low:scriptworker-prov-v1/shipit-dev
to moz-tree:level:1:*[1]. Having * will allow Thunderbird devs to try using this worker.

Then, I added:
> project:releng:ship-it:production
> queue:create-task:highest:scriptworker-prov-v1/shipit-v1
to project:releng:release:mozilla-{beta,release}[2][3].

How does this look to you, Dustin? 

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A1%3A*
[2] https://tools.taskcluster.net/auth/roles/project%3Areleng%3Arelease%3Amozilla-beta
[3] https://tools.taskcluster.net/auth/roles/project%3Areleng%3Arelease%3Amozilla-release

Flags: needinfo?(dustin)

Dustin J. Mitchell [:dustin] (he/him)

Comment 3

•

6 years ago

That makes perfect sense! :)

It might make sense to think about `project:releng:dev-release:..` roles too, to handle the very-low and shipit-staging scopes?

Flags: needinfo?(dustin)

Johan Lorenzo [:jlorenzo]

Reporter

Comment 4

•

6 years ago

Oh noes, I broke ship_action tasks today[1]. So it appears `project:releng:release:mozilla-beta` isn't inherited by the only scope the action task relies on `repo:hg.mozilla.org/releases/mozilla-beta:*`. Because it held the release, I worked around the issue by adding to missing scope [2]. I don't think this solution should stand.

In order to understand where I should put the scope, I backtracked `queue:create-task:highest:scriptworker-prov-v1/pushapk-v1*` (it should be used in the same context as the shipit scope). I didn't manage to find where its explicitly defined. [3] with the filter "direct ownership" shows `project:releng:nightly:level-3:*` (and some cli clients). However, [4] doesn't show anything used in the release promotion process.

Then, I don't know what happens. Aki and Callek checked a few things and went out of idea. Dustin, would you have a way to show how a scope is inherited, when inheritance has more than 1 parent-child relationship? If we solve this issue, then we would be able to set up `project:releng:dev-release:..` roles.

[1] {Firefox,Devedition} 59.0b6 https://tools.taskcluster.net/groups/C5eTcZdrSbix4IP2nYMJuw/tasks/ZvJbYO0BQFy7qXBO3YC-6w/runs/0/logs/public%2Flogs%2Flive_backing.log#L2026
[2] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko
[3] https://tools.taskcluster.net/auth/scopes/queue%3Acreate-task%3Ahighest%3Ascriptworker-prov-v1%2Fpushapk-v1*
[4] https://tools.taskcluster.net/auth/scopes/assume%3Arole%3Aproject%3Areleng%3Anightly%3Alevel-3%3A*

Flags: needinfo?(dustin)

Dustin J. Mitchell [:dustin] (he/him)

Comment 5

•

6 years ago

There's no such thing as role inheritance, so that's easy.. rather, roles are assumed, one to the next -- more like macro inclusion, really.

I don't see anything in the expanded scopes for that createTask call that includes "pushapk-v1*", but I do see queue:create-task:highest:scriptworker-prov-v1/dep-pushapk -- maybe that's what you meant?

Anyway, the roles involved in that createTask call are right in [1]:

[task 2018-02-02T15:21:38.825193Z]   "assume:moz-tree:level:1:gecko",
[task 2018-02-02T15:21:38.825205Z]   "assume:moz-tree:level:2:gecko",
[task 2018-02-02T15:21:38.825217Z]   "assume:moz-tree:level:3:gecko",
[task 2018-02-02T15:21:38.825234Z]   "assume:project:releng:branch:gecko:level-3:mozilla-beta",
[task 2018-02-02T15:21:38.825253Z]   "assume:project:releng:feature:buildbot:gecko:level-3:mozilla-beta",
[task 2018-02-02T15:21:38.825310Z]   "assume:project:releng:feature:taskcluster-docker-routes-v1:gecko:level-3:mozilla-beta",
[task 2018-02-02T15:21:38.825362Z]   "assume:project:releng:feature:taskcluster-docker-routes-v2:gecko:level-3:mozilla-beta",
[task 2018-02-02T15:21:38.825395Z]   "assume:project:releng:nightly:level-3:mozilla-beta",
[task 2018-02-02T15:21:38.825416Z]   "assume:project:taskcluster:gecko:level-1-sccache-buckets",
[task 2018-02-02T15:21:38.825435Z]   "assume:project:taskcluster:gecko:level-2-sccache-buckets",
[task 2018-02-02T15:21:38.825465Z]   "assume:project:taskcluster:gecko:level-3-sccache-buckets",
[task 2018-02-02T15:21:38.825515Z]   "assume:project:taskcluster:level-1-sccache-buckets",
[task 2018-02-02T15:21:38.825535Z]   "assume:project:taskcluster:level-2-sccache-buckets",
[task 2018-02-02T15:21:38.825552Z]   "assume:project:taskcluster:level-3-sccache-buckets",
[task 2018-02-02T15:21:38.825568Z]   "assume:repo:hg.mozilla.org/releases/mozilla-beta:*",

The action is running with the last one, and it is directly and indirectly including the others.

https://tools.taskcluster.net/auth/roles/repo:hg.mozilla.org%2freleases%2fmozilla-beta:*
    assume:project:releng:branch:gecko:level-3:mozilla-beta
    assume:project:releng:feature:taskcluster-docker-routes-v1:gecko:level-3:mozilla-beta
    assume:project:releng:feature:taskcluster-docker-routes-v2:gecko:level-3:mozilla-beta
    assume:project:releng:feature:buildbot:gecko:level-3:mozilla-beta

So, chasing those down in turn (and ignoring the features since they are clearly not relevant just based on the names)

https://tools.taskcluster.net/auth/roles/project%3Areleng%3Abranch%3Agecko%3Alevel-3%3A*
    assume:moz-tree:level:3:gecko
    index:insert-task:gecko.cache.level-3.*
    index:insert-task:gecko.v2.<..>.*
    queue:route:coalesce.v1.<..>.*
    queue:route:coalesce.v1.builds.<..>.*
    queue:route:index.gecko.cache.level-3.*
    queue:route:index.gecko.v2.<..>.*
    queue:route:index.releases.v1.<..>.*
    queue:route:tc-treeherder-stage.<..>.*
    queue:route:tc-treeherder-stage.v2.<..>.*
    queue:route:tc-treeherder.<..>.*
    queue:route:tc-treeherder.v2.<..>.*
    secrets:get:project/releng/gecko/build/level-3/*

That leaves only moz-tree:level:3:gecko!  That's matched by both

https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko
    assume:moz-tree:level:2:gecko
    assume:project:taskcluster:gecko:level-3-sccache-buckets
    assume:project:taskcluster:level-3-sccache-buckets
    queue:create-task:highest:scriptworker-prov-v1/shipit-v1

https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3A*
    auth:aws-s3:read-write:public-qemu-images/repository/hg.mozilla.org/mozilla-central/*
    docker-worker:cache:level-3-*
    docker-worker:feature:balrogStageVPNProxy
    docker-worker:feature:balrogVPNProxy
    docker-worker:image:taskclusterprivate/taskcluster-vpn-proxy:*
    generic-worker:cache:level-3-*
    generic-worker:os-group:*
    index:insert-task:docker.images.v2.level-3.*
    index:insert-task:gecko.cache.level-3.*
    project:releng:balrog:server:beta
    project:releng:balrog:server:release
    project:releng:beetmover:bucket:release
    project:releng:ship-it:production
    queue:cancel-task:gecko-level-3/*
    queue:create-task:aws-provisioner-v1/ami-test*
    queue:create-task:aws-provisioner-v1/android-api-*
    queue:create-task:aws-provisioner-v1/b2gbuild*
    queue:create-task:aws-provisioner-v1/b2gtest*
    queue:create-task:aws-provisioner-v1/balrog
    queue:create-task:aws-provisioner-v1/dbg-*
    queue:create-task:aws-provisioner-v1/desktop-test*
    queue:create-task:aws-provisioner-v1/flame-kk*
    queue:create-task:aws-provisioner-v1/gecko-1-*
    queue:create-task:aws-provisioner-v1/gecko-2-*
    queue:create-task:aws-provisioner-v1/gecko-3-*
    queue:create-task:aws-provisioner-v1/gecko-decision
    queue:create-task:aws-provisioner-v1/gecko-images
    queue:create-task:aws-provisioner-v1/gecko-misc
    queue:create-task:aws-provisioner-v1/gecko-symbol-upload
    queue:create-task:aws-provisioner-v1/gecko-t-*
    queue:create-task:aws-provisioner-v1/loan-1-*
    queue:create-task:aws-provisioner-v1/loan-t-*
    queue:create-task:aws-provisioner-v1/mulet-debug
    queue:create-task:aws-provisioner-v1/mulet-opt
    queue:create-task:aws-provisioner-v1/opt-*
    queue:create-task:aws-provisioner-v1/rustbuild
    queue:create-task:aws-provisioner-v1/spidermonkey
    queue:create-task:aws-provisioner-v1/symbol-upload
    queue:create-task:aws-provisioner-v1/taskcluster-images
    queue:create-task:aws-provisioner-v1/testdroid-device
    queue:create-task:buildbot-bridge/buildbot-bridge
    queue:create-task:dummy-test-provisioner/dummy-test-type
    queue:create-task:gecko-t-tc-worker/*
    queue:create-task:highest:aws-provisioner-v1/ami-test*
    queue:create-task:highest:aws-provisioner-v1/android-api-*
    queue:create-task:highest:aws-provisioner-v1/b2gbuild*
    queue:create-task:highest:aws-provisioner-v1/b2gtest*
    queue:create-task:highest:aws-provisioner-v1/balrog
    queue:create-task:highest:aws-provisioner-v1/dbg-*
    queue:create-task:highest:aws-provisioner-v1/desktop-test*
    queue:create-task:highest:aws-provisioner-v1/flame-kk*
    queue:create-task:highest:aws-provisioner-v1/gecko-1-*
    queue:create-task:highest:aws-provisioner-v1/gecko-2-*
    queue:create-task:highest:aws-provisioner-v1/gecko-3-*
    queue:create-task:highest:aws-provisioner-v1/gecko-decision
    queue:create-task:highest:aws-provisioner-v1/gecko-images
    queue:create-task:highest:aws-provisioner-v1/gecko-misc
    queue:create-task:highest:aws-provisioner-v1/gecko-symbol-upload
    queue:create-task:highest:aws-provisioner-v1/gecko-t-*
    queue:create-task:highest:aws-provisioner-v1/loan-1-*
    queue:create-task:highest:aws-provisioner-v1/loan-t-*
    queue:create-task:highest:aws-provisioner-v1/mulet-debug
    queue:create-task:highest:aws-provisioner-v1/mulet-opt
    queue:create-task:highest:aws-provisioner-v1/opt-*
    queue:create-task:highest:aws-provisioner-v1/rustbuild
    queue:create-task:highest:aws-provisioner-v1/spidermonkey
    queue:create-task:highest:aws-provisioner-v1/symbol-upload
    queue:create-task:highest:aws-provisioner-v1/taskcluster-images
    queue:create-task:highest:aws-provisioner-v1/testdroid-device
    queue:create-task:highest:buildbot-bridge/buildbot-bridge
    queue:create-task:highest:dummy-test-provisioner/dummy-test-type
    queue:create-task:highest:gecko-t-tc-worker/*
    queue:create-task:highest:localprovisioner/*
    queue:create-task:highest:null-provisioner/buildbot
    queue:create-task:highest:null-provisioner/buildbot-try
    queue:create-task:highest:packetnet/*
    queue:create-task:highest:releng-hardware/gecko-t-*
    queue:create-task:highest:scl3-puppet/os-x-10-10-gw
    queue:create-task:highest:scl3-puppet/os-x-build-gw
    queue:create-task:highest:scriptworker-prov-v1/dep-pushapk
    queue:create-task:highest:scriptworker-prov-v1/depsigning
    queue:create-task:highest:scriptworker-prov-v1/dummy-worker-transpar
    queue:create-task:highest:scriptworker-prov-v1/signing-linux-dev
    queue:create-task:highest:tc-worker-provisioner/*
    queue:create-task:highest:test-dummy-provisioner/*
    queue:create-task:localprovisioner/*
    queue:create-task:null-provisioner/buildbot
    queue:create-task:null-provisioner/buildbot-try
    queue:create-task:packetnet/*
    queue:create-task:releng-hardware/gecko-t-*
    queue:create-task:scl3-puppet/os-x-10-10-gw
    queue:create-task:scl3-puppet/os-x-build-gw
    queue:create-task:tc-worker-provisioner/*
    queue:create-task:test-dummy-provisioner/*
    queue:get-artifact:private/docker-worker/*
    queue:route:index.docker.images.v2.level-3.*
    queue:route:index.gecko.cache.level-3.*
    queue:scheduler-id:gecko-level-3
    secrets:get:project/releng/gecko/build/level-3/*
    secrets:get:project/releng/snapcraft/firefox/edge
    secrets:get:project/taskcluster/gecko/build/level-3/*

Anyway, you could add that extra scope either in moz-tree:level:3:gecko if every level-3 gecko (non-comm) branch should have that scope.  Otherwise, if it should only exist for beta for example, we don't have a great place to put it right now, but I have some ideas (in production-branches.json, via bug 1407681)

Flags: needinfo?(dustin)

Comment hidden (Intermittent Failures Robot)

2 failures in 735 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* gecko-decision: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1433157&startday=2018-01-29&endday=2018-02-04&tree=all

Johan Lorenzo [:jlorenzo]

Reporter

Comment 7

•

6 years ago

Dustin and I chatted on IRC. I was actually looking for pushapk-v1 (not dep-pushapk)

He found `project:releng:nightly:level-3:*`[1] does define pushapk-v1. Then, the chain is the following:
* A ship_fennec task[2] uses the scope `repo:hg.mozilla.org/releases/mozilla-beta:*`[3]
* This scope matches `repo:hg.mozilla.org/releases/mozilla-beta:cron:nightly-*`[4]
* which itself assumes `assume:project:releng:nightly:level-3:mozilla-beta`[5] which contains the right scope.

The current description of the later scope reads:
> (XXXCallek: Delete scopes after Bug 1348834 lands/deploys queue:route:index.project.releng.funsize.level-3.* )

Bug 1348834 is fixed. Therefore, we should stop using it. Moreover, on-push builds are all nightlies, but not all of them are meant to be shipped. Thus, I'd suggest to migrate scopes from[5] to `project:releng:release:mozilla-beta`[6] (some of them already are). Then we should add [6] to the scopes of the "ship_*" action tasks. 

Simon, Callek, Aki: Does this sound reasonable/feasible to you? 

[1] https://tools.taskcluster.net/auth/roles/project%3Areleng%3Anightly%3Alevel-3%3A*
[2] https://tools.taskcluster.net/groups/G_4VI3JkSMGAmrULs-61Kg/tasks/bkTIJE8wQTifKGZEPGel-w/details
[3] https://tools.taskcluster.net/auth/roles/repo%3Ahg.mozilla.org%2Freleases%2Fmozilla-beta%3A*
[4] https://tools.taskcluster.net/auth/roles/repo%3Ahg.mozilla.org%2Freleases%2Fmozilla-beta%3Acron%3Anightly-*
[5] https://tools.taskcluster.net/auth/roles/project%3Areleng%3Anightly%3Alevel-3%3Amozilla-beta
[6] https://tools.taskcluster.net/auth/roles/project%3Areleng%3Arelease%3Amozilla-beta

Depends on: 1348834

Flags: needinfo?(sfraser)

Flags: needinfo?(bugspam.Callek)

Flags: needinfo?(aki)

Simon Fraser [:sfraser] ⌚️GMT

Comment 8

•

6 years ago

Sounds good to me

Flags: needinfo?(sfraser)

Aki Sasaki (not active)

Comment 9

•

6 years ago

(In reply to Johan Lorenzo [:jlorenzo] from comment #7)
> Dustin and I chatted on IRC. I was actually looking for pushapk-v1 (not
> dep-pushapk)
> 
> He found `project:releng:nightly:level-3:*`[1] does define pushapk-v1. Then,
> the chain is the following:
> * A ship_fennec task[2] uses the scope
> `repo:hg.mozilla.org/releases/mozilla-beta:*`[3]
> * This scope matches
> `repo:hg.mozilla.org/releases/mozilla-beta:cron:nightly-*`[4]
> * which itself assumes
> `assume:project:releng:nightly:level-3:mozilla-beta`[5] which contains the
> right scope.
> 
> The current description of the later scope reads:
> > (XXXCallek: Delete scopes after Bug 1348834 lands/deploys queue:route:index.project.releng.funsize.level-3.* )
> 
> Bug 1348834 is fixed. Therefore, we should stop using it.

Are we deleting these scopes? I think cleanup is good, but we should make sure a nightly + staging-release graph succeeds afterwards.

> Moreover, on-push
> builds are all nightlies, but not all of them are meant to be shipped. Thus,
> I'd suggest to migrate scopes from[5] to
> `project:releng:release:mozilla-beta`[6] (some of them already are). Then we
> should add [6] to the scopes of the "ship_*" action tasks. 

- I think mozilla-beta is wrong for mozilla-release, maple, etc tasks.
- The ship_* action tasks take the scopes from the tree, so that would mean adding project:releng:release:mozilla-beta to the maple, birch, and mozilla-release trees, which also sounds wrong unless we rename.
- mozilla-central uses pushapk-v1 for nightly graphs, no?
- Not including this in level 3 or other shared roles means no one but Releng can trigger nightly graphs, or retrigger failed pushapk tasks. I'm not sure this is the right thing. Once everything is using hooks to elevate scopes, we can possibly do something like this. Until then this may have usability concerns.
  - Additionally, we appear to be rating pushapk security above, say, balrog or release signing. Given the relative sizes of the user populations, this is probably backwards.

I would lean towards having pushapk-v1 in level3 roles. We have scriptworker scope checks to narrow this down to per-tree.

Flags: needinfo?(aki)

Johan Lorenzo [:jlorenzo]

Reporter

Comment 10

•

6 years ago

Thank you very much for raising these concerns, Aki! I agree with you, level3 seems to be the best solution.

> Are we deleting these scopes? I think cleanup is good, but we should make sure a nightly + staging-release graph succeeds afterwards.

I think we should get rid of it, because of the description. I agree we should save it somewhere, just in case a nightly or a release breaks. I can handle this. 


> - I think mozilla-beta is wrong for mozilla-release, maple, etc tasks.
That's a fairly valid point. We could address this particular point by setting `project:releng:release:*`, but that wouldn't solve the next points. 

> - The ship_* action tasks take the scopes from the tree, so that would mean adding project:releng:release:mozilla-beta to the maple, birch, and mozilla-release trees, which also sounds wrong unless we rename.
`project:releng:release:*` sounds bad for staging releases branches. We don't want them to use the production scriptworker instances. 

> - mozilla-central uses pushapk-v1 for nightly graphs, no?
That's true. We can leave the nightly scopes on central. In my opinion, the release scope should have a (slightly) different architecture than the nightly ones (because of this case, notably)

> - Not including this in level 3 or other shared roles means no one but Releng can trigger nightly graphs, or retrigger failed pushapk tasks.
That's bad, indeed. I don't see a better way than putting the scopes in the level3 role, like you suggested.

> - Additionally, we appear to be rating pushapk security above, say, balrog or release signing.
That's fair. Maybe in a glorious future, the balrog/beetmover workers that publish a release should have a more restrictive set of scopes. As of today, that's backwards, I agree.

bhearsum@mozilla.com (:bhearsum)

Comment 11

•

6 years ago

Did some changes happen to the scopes in the past few hours? The latest push to maple is failing with:
[task 2018-02-12T19:51:10.268953Z] You do not have sufficient scopes. You are missing the following scopes:
[task 2018-02-12T19:51:10.268986Z] 
[task 2018-02-12T19:51:10.269001Z] {
[task 2018-02-12T19:51:10.269028Z]   "AllOf": [
[task 2018-02-12T19:51:10.269070Z]     "project:releng:signing:cert:release-signing",
[task 2018-02-12T19:51:10.269099Z]     {
[task 2018-02-12T19:51:10.269131Z]       "AnyOf": [
[task 2018-02-12T19:51:10.269214Z]         "queue:create-task:highest:scriptworker-prov-v1/signing-linux-v1",
[task 2018-02-12T19:51:10.269262Z]         "queue:create-task:very-high:scriptworker-prov-v1/signing-linux-v1",
[task 2018-02-12T19:51:10.269307Z]         "queue:create-task:high:scriptworker-prov-v1/signing-linux-v1",
[task 2018-02-12T19:51:10.269358Z]         "queue:create-task:medium:scriptworker-prov-v1/signing-linux-v1",
[task 2018-02-12T19:51:10.269407Z]         "queue:create-task:low:scriptworker-prov-v1/signing-linux-v1",
[task 2018-02-12T19:51:10.269451Z]         "queue:create-task:very-low:scriptworker-prov-v1/signing-linux-v1"
[task 2018-02-12T19:51:10.269476Z]       ]
[task 2018-02-12T19:51:10.269497Z]     }
[task 2018-02-12T19:51:10.269514Z]   ]
[task 2018-02-12T19:51:10.269529Z] }

Aki Sasaki (not active)

Comment 12

•

6 years ago

I just added those two to https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko , which makes me nervous, but we fall back to cot scopes verification if someone tries to abuse it. In theory we should be unblocked for maple staging releases.

Aki Sasaki (not active)

Comment 13

•

6 years ago

project:releng:signing:cert:nightly-signing too.
At some point we'll downgrade maple and birch to dep-signing, possibly after the no-bbb project.

Johan Lorenzo [:jlorenzo]

Reporter

Comment 14

•

6 years ago

(In reply to Ben Hearsum (:bhearsum) from comment #11)
I'm fairly certain I didn't touch any scope/role yesterday. I just looked at them.

I'm sure [1] didn't have these scopes yesterday and 11 days ago. I'm worried something else changed. Do we have a way to track recent changes in scopes, Dustin?

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko

Justin Wood (:Callek)

Comment 15

•

6 years ago

Nothing to add.

Flags: needinfo?(bugspam.Callek)

Dustin J. Mitchell [:dustin] (he/him)

Comment 16

•

6 years ago

Did this get sorted out?

Flags: needinfo?(jlorenzo)

Johan Lorenzo [:jlorenzo]

Reporter

Comment 17

•

6 years ago

Yeah, we ended up using https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko for all release-related scopes. There's nothing more to add, as of today. Sorry, I should have closed this bug earlier.

Status: NEW → RESOLVED

Closed: 6 years ago

Flags: needinfo?(jlorenzo)

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Assignee

Updated

•

5 years ago

Component: Service Request → Operations and Service Requests

Bugzilla

Quick Search

Please add ship-it related scopes to moz-tree:level:\d

Categories

(Taskcluster :: Operations and Service Requests, task)

Tracking

(Not tracked)

People

(Reporter: jlorenzo, Unassigned)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Comment 14

Comment 15

Comment 16

Comment 17

Updated