1288434 - Add task to mark releases as "shipped"

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

Now that ship-it notify is dead and not messing the status we can automate marking releases as shipped.

* The task should be dependent on "Publish release human decision task"
* Still has to use BB/BBB to talk to SCL3 (ship-it)

The call is very simple, similar to:
https://github.com/mozilla-releng/ship-it/blob/bc2c01de271399df099b12e7c289b97bbcbd2de5/kickoff/static/kickoff.js#L274-L288

Probably we can implement it similar to mark_as_completed() http://hg.mozilla.org/build/tools/file/tip/lib/python/kickoff/__init__.py#l89 and call         

  self.release_api.update(release['name'], status='shipped', shippedAt='2016-07-07 23:05:06')

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Mark as shipped mozharness patch

@rail:

I tripped in using Swagger at some point and decided to go along and finish the implementation without it for the moment. I was thinking of filing a bug afterwards as an improvement. For the moment, just wanted to throw this up for a quick feedback. Few questions if I may:

1. does "shipitapi" sounds good as a pypi name package?
2. can we upload http://hg.mozilla.org/build/tools/file/tip/misc/certs/ca-bundle.crt along with the pypi package?
3. will I have the credentials/api_root to access Ship It API endpoint in oauth.txt? If yes, where can I lookup for the exact variables names?
4. Does it sound reasonable to "rebuild" the release name by building up the string based on product, version and buildnumber?

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8782478 [details] [diff] [review]
Mark as shipped mozharness patch

Review of attachment 8782478 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Mihai Tabara [:mtabara] from comment #1)
> 1. does "shipitapi" sounds good as a pypi name package?

yup, sgtm.

> 2. can we upload
> http://hg.mozilla.org/build/tools/file/tip/misc/certs/ca-bundle.crt along
> with the pypi package?

I think this should be in the certify package (used by requests).

> 3. will I have the credentials/api_root to access Ship It API endpoint in
> oauth.txt? If yes, where can I lookup for the exact variables names?

I think we should add those and use BBB to deal with netflows :/

> 4. Does it sound reasonable to "rebuild" the release name by building up the
> string based on product, version and buildnumber?

If by "rebuild" you mean constructing the name of release without any external dependency, then yes, it's OK. We do the same in releasetasks.

::: testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
@@ +104,5 @@
> +        release_api = Release(auth, api_root=api_root,
> +                              timeout=self.config['timeout'])
> +        _now = datetime.now()
> +        shipped_at = "{}-{}-{} {}:{}:{}".format(_now.year, _now.month, _now.day,
> +                                                _now.hour, _now.minute, _now.second)

you can probably use time.strftime() instead of format.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Mark as shipped mozharness patch

(In reply to Rail Aliiev [:rail] from comment #2)
> Comment on attachment 8782478 [details] [diff] [review]
> Mark as shipped mozharness patch
> 
> Review of attachment 8782478 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> (In reply to Mihai Tabara [:mtabara] from comment #1)
> > 1. does "shipitapi" sounds good as a pypi name package?
> 
> yup, sgtm.

will do so, thanks.

> 
> > 2. can we upload
> > http://hg.mozilla.org/build/tools/file/tip/misc/certs/ca-bundle.crt along
> > with the pypi package?
> 
> I think this should be in the certify package (used by requests).

I double-checked the CA-bundle used in tools repo (by both Balrog and Ship-it lib/python/.../api.py and they both use https://github.com/MihaiTabara/build-tools/blob/master/misc/certs/ca-bundle.crt which at its turn is a sync from Mozilla official certdata file - https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt. Having that said, I will add <certifi> package as a dependency and use <certifi.where()> to reference the CA bundle when the calls API calls are made via requests. 

 
> > 3. will I have the credentials/api_root to access Ship It API endpoint in
> > oauth.txt? If yes, where can I lookup for the exact variables names?
> 
> I think we should add those and use BBB to deal with netflows :/

how do I do this? who should I poke for some help here?
> 
> > 4. Does it sound reasonable to "rebuild" the release name by building up the
> > string based on product, version and buildnumber?
> 
> If by "rebuild" you mean constructing the name of release without any
> external dependency, then yes, it's OK. We do the same in releasetasks.
> 

Great, thanks.

> ::: testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
> @@ +104,5 @@
> > +        release_api = Release(auth, api_root=api_root,
> > +                              timeout=self.config['timeout'])
> > +        _now = datetime.now()
> > +        shipped_at = "{}-{}-{} {}:{}:{}".format(_now.year, _now.month, _now.day,
> > +                                                _now.hour, _now.minute, _now.second)
> 
> you can probably use time.strftime() instead of format.

done, thanks for the feedback.

Comment 4

[Rail Aliiev [:rail]]

(In reply to Mihai Tabara [:mtabara] from comment #3)
> > I think we should add those and use BBB to deal with netflows :/
> 
> how do I do this? who should I poke for some help here?

If you use BBB, you don't need to do anything extra.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] from comment #4)
> (In reply to Mihai Tabara [:mtabara] from comment #3)
> > > I think we should add those and use BBB to deal with netflows :/
> > 
> > how do I do this? who should I poke for some help here?
> 
> If you use BBB, you don't need to do anything extra.

And where can I inspect a sample of that oauth.txt file? Not sure how those username/passwd variables are named in the file.

Comment 6

[Rail Aliiev [:rail]]

This is the template: https://dxr.mozilla.org/build-central/source/puppet/modules/buildmaster/templates/BuildSlaves-build.py.erb. Puppet generates BuildSlaves.py, which is copied to slaves in one of the steps, http://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l4428

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

23:13:09 <mtabara|away> rail: to add ship-it credentials I'll need to patch the above template with some shipit_username/shipit_password, iiuc. but what do we do for api_root? doesn't seem the right place
23:13:51 <rail> it should go somewhere under mozharness/configs I think
23:16:29 <mtabara|away> postrelease_firefox_<channel> maybe? 
23:18:53 <rail> maybe
23:19:05 <rail> maybe a separate file

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8782986 [details] [diff] [review]
Mark as shipped mozharness patch

Review of attachment 8782986 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
@@ +14,5 @@
> +import sys
> +from datetime import datetime
> +from shipitapi import API
> +
> +sys.path.insert(1, os.path.dirname(os.path.dirname(sys.path[0])))

I think you need to move the shipitapi import under this line.

@@ +88,5 @@
> +                self.info("Overriding %s with %s" % (prop, props[prop]))
> +                self.config[prop] = props.get(prop)
> +            else:
> +                self.warning("%s could not be found within buildprops" % prop)
> +                return

Should it be "continue" instead of "return"?

@@ +102,5 @@
> +                                        self.config["credentials_file"])
> +        credentials = {}
> +        execfile(credentials_file, credentials)
> +        # TODO: identify the proper credentials and api_root from config file
> +        auth = (credentials['username'], credentials['password'])

I'd use the same approach we use for balrog (see http://hg.mozilla.org/build/puppet/file/tip/modules/buildmaster/templates/BuildSlaves-build.py.erb#l28), where you specify the username in the config and the password in the credentials file. This way you can have different credentials for prod and staging and per product (thunderbird would have a different username and permissions, when we implement them)

@@ +103,5 @@
> +        credentials = {}
> +        execfile(credentials_file, credentials)
> +        # TODO: identify the proper credentials and api_root from config file
> +        auth = (credentials['username'], credentials['password'])
> +        api_root = credentials['api_root']

I think it'd be better to not store the API root in the credentials file. Just put it into the config.

::: testing/mozharness/scripts/release/shipitapi.py
@@ +1,5 @@
> +from datetime import datetime
> +import logging
> +import certifi
> +import requests
> +from redo import retry

I'm not sure if you can import certify, requests and redo here, because they are installed as a part of the main script, what happens after all top level imports. As a work around you may want to call the imports just before you use the modeules .

@@ +26,5 @@
> +    """
> +
> +    auth = None
> +    url_template = None
> +    verify = certifi.where()

I wonder what happens if you run this against prod and staging. One of them may have a self-signed (or signed by mozilla root).

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] from comment #8)
> Comment on attachment 8782986 [details] [diff] [review]
> Mark as shipped mozharness patch
> 
> Review of attachment 8782986 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
> @@ +14,5 @@
> > +import sys
> > +from datetime import datetime
> > +from shipitapi import API
> > +
> > +sys.path.insert(1, os.path.dirname(os.path.dirname(sys.path[0])))
> 
> I think you need to move the shipitapi import under this line.

Not sure why. The shipitapi is a package in the virtual env, it's unrelated with the mozharness relative paths insertion, right?

> @@ +88,5 @@
> > +                self.info("Overriding %s with %s" % (prop, props[prop]))
> > +                self.config[prop] = props.get(prop)
> > +            else:
> > +                self.warning("%s could not be found within buildprops" % prop)
> > +                return
> 
> Should it be "continue" instead of "return"?

Hm ... I need all those three args to build up the name. Should one of those goes missing in the buildprops.json, I might "reconstruct" a wrong release name and end up touching misleading stuff. If I have "continue" over "return" there and if by accident one of the (product, version, buildnumber) is missing, then I would be building a bogus release name. Or am I?

> ::: testing/mozharness/scripts/release/shipitapi.py
> @@ +1,5 @@
> > +from datetime import datetime
> > +import logging
> > +import certifi
> > +import requests
> > +from redo import retry
> 
> I'm not sure if you can import certify, requests and redo here, because they
> are installed as a part of the main script, what happens after all top level
> imports. As a work around you may want to call the imports just before you
> use the modeules .

Hm, not sure I understand here. Isn't <shipitapi> a wanna-be standalone pypi package? We don't even need the redo, requests and certifi in the main script venv deps as all the logic is in the pypi package. We can add them as dependencies in the setup.py for the pypi so they get installed along with the shipitapi package. Wouldn't that work?
 
> @@ +26,5 @@
> > +    """
> > +
> > +    auth = None
> > +    url_template = None
> > +    verify = certifi.where()
> 
> I wonder what happens if you run this against prod and staging. One of them
> may have a self-signed (or signed by mozilla root).

Wouldn't chain of trust solve this by default?

Comment 10

[Rail Aliiev [:rail]]

(In reply to Mihai Tabara [:mtabara] from comment #9)
> Not sure why. The shipitapi is a package in the virtual env, it's unrelated
> with the mozharness relative paths insertion, right?

The patch tells me that shipitapi is a part of mozharness. ;)
 
> > @@ +88,5 @@
> Hm ... I need all those three args to build up the name. Should one of those
> goes missing in the buildprops.json, I might "reconstruct" a wrong release
> name and end up touching misleading stuff. If I have "continue" over
> "return" there and if by accident one of the (product, version, buildnumber)
> is missing, then I would be building a bogus release name. Or am I?

I'd still use continue in this case and add an explicit check after that loop and barf is something is missing.

> Hm, not sure I understand here. Isn't <shipitapi> a wanna-be standalone pypi
> package? We don't even need the redo, requests and certifi in the main
> script venv deps as all the logic is in the pypi package. We can add them as
> dependencies in the setup.py for the pypi so they get installed along with
> the shipitapi package. Wouldn't that work?

If you package shipitapi as a separate package then yes. In that case you would need to add it to "virtualenv_modules" and import before you use it (it won't be available until you run "activate-virtualenv".

> Wouldn't chain of trust solve this by default?

I don't think it solves it, because we may use a self-signed cert on a staging instance of ship-it.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

for posterity:

23:36:21 <rail> in that case you want to include shipitapi to the list of the virtualenv packages
23:36:35 <rail> and import it later runtime
23:37:55 <mtabara|afk> yes, I think I can only import shipitapi package in the virtual env list and it will automatically add requests, redo and certifi, would it?
23:38:01 <rail> or something like https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/release/beet_mover.py#185
23:39:18 <rail> and probably you wouldn't need to specify requests and certify then
23:39:28 <rail> in the script 
23:39:31 <mtabara|afk> yes
23:39:43 <mtabara|afk> it'll drag them automatically as shipitapi dependencies
23:39:51 <rail> yup
23:40:05 <mtabara|afk> and I'll import them before using them in the shipitapi.py-whatever script
23:40:41 <rail> yeah, shipitapi can use normal imports in that case

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] from comment #10)
> (In reply to Mihai Tabara [:mtabara] from comment #9)
> > Not sure why. The shipitapi is a package in the virtual env, it's unrelated
> > with the mozharness relative paths insertion, right?
> 
> The patch tells me that shipitapi is a part of mozharness. ;)

Sorry for the misunderstanding - shipitapi is supposed to be a package, yes.

>  
> > > @@ +88,5 @@
> > Hm ... I need all those three args to build up the name. Should one of those
> > goes missing in the buildprops.json, I might "reconstruct" a wrong release
> > name and end up touching misleading stuff. If I have "continue" over
> > "return" there and if by accident one of the (product, version, buildnumber)
> > is missing, then I would be building a bogus release name. Or am I?
> 
> I'd still use continue in this case and add an explicit check after that
> loop and barf is something is missing.

Okay, will do so.

> 
> > Hm, not sure I understand here. Isn't <shipitapi> a wanna-be standalone pypi
> > package? We don't even need the redo, requests and certifi in the main
> > script venv deps as all the logic is in the pypi package. We can add them as
> > dependencies in the setup.py for the pypi so they get installed along with
> > the shipitapi package. Wouldn't that work?
> 
> If you package shipitapi as a separate package then yes. In that case you
> would need to add it to "virtualenv_modules" and import before you use it
> (it won't be available until you run "activate-virtualenv".
> 
> > Wouldn't chain of trust solve this by default?
> 
> I don't think it solves it, because we may use a self-signed cert on a
> staging instance of ship-it.

Ah, right, makes sense. So it can default to the certifi.where() CA_bundles but in the same time I need to find an alternative way to overwrite it in case of need?

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: shipitapi as Pypi package

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

Pypi for shipitapi is https://pypi.python.org/pypi/shipitapi
@rail: I've added you as Owner 

Note to self: once we've got everything set, TODO: add jlund and catlee as owners too

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Mark as shipped mozharness patch, refactored version.

* addressed the comments

TODISCUSS:
1. If postrelease_* files from mh/configs are to be used or should we use separate ones just for this task?
2. If login credentials are properly split between puppet and mozharness configs?
3. I didn't address it in the current patch, but how about:

<pseudocode>
if self.config["ca_certs"]
> release_api = Release(auth, api_root=api_root, ca_certs=self.config["ca_certs], timeout ...

else
> release_api = Release(auth, api_root=api_root, timeout ...
</pseudocode>

This way we can add different CA bundles for staging environment and specify them via configs (+ probably tweak puppet too)

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8784272 [details] [diff] [review]
Mark as shipped mozharness patch, refactored version.

Review of attachment 8784272 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
@@ +36,5 @@
> +        super(MarkReleaseAsShipped, self).__init__(
> +            config_options=self.config_options,
> +            require_config_file=require_config_file,
> +            config={
> +                # TODO: import shipitapi instead here!

Not sure if I am following the comment... Otherwise LGTM! Is the only remaining part is a puppet patch now?

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara] from comment #14)
> Pypi for shipitapi is https://pypi.python.org/pypi/shipitapi
> @rail: I've added you as Owner 
> 
> Note to self: once we've got everything set, TODO: add jlund and catlee as
> owners too

This is done now

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1288434 - fix buildbot props in mark release as shipped.

Review commit: https://reviewboard.mozilla.org/r/74500/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74500/

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1288434 - Add bb-configs for mark release as shipped task.

Review commit: https://reviewboard.mozilla.org/r/74534/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74534/

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1288434 - Add bb-custom builder for mark release as shipped task.

Review commit: https://reviewboard.mozilla.org/r/74538/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74538/

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8785258 [details]
Bug 1288434 - Add bb-configs for mark release as shipped task.

https://reviewboard.mozilla.org/r/74534/#review72376

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 8785259 [details]
Bug 1288434 - Add bb-custom builder for mark release as shipped task.

https://reviewboard.mozilla.org/r/74538/#review72374

::: process/release.py:2033
(Diff revision 1)
>          }
>      }
>      builders.append(bouncer_aliases_builder)
>  
> +    # mark release as shipped
> +    mark_as_shipped_mh_cfg= {

A nit: add a space before =

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8785223 [details]
Bug 1288434 - fix buildbot props in mark release as shipped.

https://reviewboard.mozilla.org/r/74500/#review72378

Comment 24

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Scheduling builder from releasetasks

PR in releasetasks to define the builder in the TC release graph.

Comment 25

[Mihai Tabara [:mtabara]⌚️GMT]

Leftovers:
* to merge all above patches
* puppet patch
* integration with release-runner (set the postrelease_mark_as_shipped_enabled to True) to activate the TC scheduling
* removing the step from releasewarrior once this is shipped

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785259 [details]
Bug 1288434 - Add bb-custom builder for mark release as shipped task.

https://reviewboard.mozilla.org/r/74538/#review72374

> A nit: add a space before =

Thanks for catching that, will address it before pushing upstream.

Comment 27

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1288434 - Add Ship-It credentials in BuildSlaves.py.

Review commit: https://reviewboard.mozilla.org/r/74594/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74594/

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8785386 [details]
Bug 1288434 - Add Ship-It credentials in BuildSlaves.py.

https://reviewboard.mozilla.org/r/74594/#review72444

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 8785386 [details]
Bug 1288434 - Add Ship-It credentials in BuildSlaves.py.

remote:   https://hg.mozilla.org/build/puppet/rev/2d58c6ec75d0
remote:   https://hg.mozilla.org/build/puppet/rev/5028ebdb4607

Comment 30

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1288434 - tweak release runner for mark as shipped builder.

Review commit: https://reviewboard.mozilla.org/r/74602/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74602/

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2199b0d2d4f3
Add task to mark release as shipped. r=rail a=release DONTBUILD

Comment 32

[Rail Aliiev [:rail]]

Comment on attachment 8785223 [details]
Bug 1288434 - fix buildbot props in mark release as shipped.

https://hg.mozilla.org/integration/mozilla-inbound/rev/2199b0d2d4f3
https://hg.mozilla.org/releases/mozilla-aurora/rev/e02ff3651298
https://hg.mozilla.org/releases/mozilla-beta/rev/805edff00ab6
https://hg.mozilla.org/releases/mozilla-release/rev/24d254ae0e82
https://hg.mozilla.org/releases/mozilla-esr45/rev/f0545cc75580

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 8785403 [details]
Bug 1288434 - tweak release runner for mark as shipped builder.

https://reviewboard.mozilla.org/r/74602/#review72452

Comment 34

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/3dc06ea07b8d

Comment 35

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/91327dcd59ef

Comment 36

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785259 [details]
Bug 1288434 - Add bb-custom builder for mark release as shipped task.

http://hg.mozilla.org/build/buildbotcustom/rev/91327dcd59ef
http://hg.mozilla.org/build/buildbotcustom/rev/ca1e83fb0a25

Comment 37

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785258 [details]
Bug 1288434 - Add bb-configs for mark release as shipped task.

http://hg.mozilla.org/build/buildbot-configs/rev/3dc06ea07b8d
http://hg.mozilla.org/build/buildbot-configs/rev/9377d9b0a3e9

Comment 38

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785287 [details] [review]
Scheduling builder from releasetasks

https://github.com/mozilla/releasetasks/commit/7ee0d7c7dc6ea10eade3431f82d2fae89b3251cb

Comment 39

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785403 [details]
Bug 1288434 - tweak release runner for mark as shipped builder.

http://hg.mozilla.org/build/tools/rev/6e9c74164f98

Comment 40

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8784265 [details]
shipitapi as Pypi package

* Transfered ownership under mozilla-releng Github repo.
https://github.com/mozilla-releng/shipitapi

* Pypi package is now owned by catlee, jlund, rail and mtabara
https://pypi.python.org/pypi/shipitapi

Comment 41

[Mihai Tabara [:mtabara]⌚️GMT]

Leftovers before we can close this:
1. add the https://pypi.python.org/pypi/shipitapi Pypi package to mozilla-internal Pypi 
2. once we confirm everything is running okay in the next beta - wipe off this section https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#4-post-release-step 

@rail: can you help me out please with some information regarding 1? I forgot how the process works within puppet. It's been 3 years since I last used it :)

Comment 42

[Rail Aliiev [:rail]]

Me too! :) BTW, this is not the puppet pypi, this is the internal pypi mirror, something like https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_RelengAPI?

Comment 43

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2199b0d2d4f3

Comment 44

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] from comment #42)
> Me too! :) BTW, this is not the puppet pypi, this is the internal pypi
> mirror, something like
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_RelengAPI?

Ah, right, makes sense ;) 

I think it's worth waiting for a confirmation with 49.0b8-build1 to make sure it works before pushing anything to our internal Pypi mirror to avoid dealing with multiple consecutive version bumps. Personally, I'd wait for bug 1298343 completion too but am not sure how long we're supposed to leave the infra rely on http://pypi.python.org/ only before mirror it internally.

Will follow-up on this after we notice its behavior in 49.0b8 release today.

Comment 45

[Mihai Tabara [:mtabara]⌚️GMT]

We can't install anything from pypi.python.org. Rail took care of copying the file under our internal mirror: http://pypi.pub.build.mozilla.org/pub/shipitapi-0.0.3.tar.gz

Comment 46

[Rail Aliiev [:rail]]

this is what I did:

ssh relengwebadm.private.scl3.mozilla.com
https://pypi.python.org/packages/1b/80/77cd9ea4f02b39c2b560940cc88ea8a43ff962ec52292a199bd8dc09d131/shipitapi-0.0.3.tar.gz#md5=efeca33cdd6186715a49fcb272341ea6
chmod 664 shipitapi-0.0.3.tar.gz 
sudo mv -vi  shipitapi-0.0.3.tar.gz /mnt/netapp/relengweb/pypi/pub

Comment 47

[Rail Aliiev [:rail]]

Missing "wget" somewhere ^ ;)

Comment 48

[Mihai Tabara [:mtabara]⌚️GMT]

Sadly the build misbehaved with its first public representation. 

Few things here:
1. When I cloned the version-bump script in releasetasks, I forgot to rename the notification id
2. There is an error with respect to the builnumber config

08:26:41     INFO - Using buildbot properties:
08:26:41     INFO - {
08:26:41     INFO -     "project": "", 
08:26:41     INFO -     "product": "firefox", 
08:26:41     INFO -     "build_number": 1, 
08:26:41     INFO -     "taskId": "OH4kQzlASAGa_Dq2-8L5gA", 
08:26:41     INFO -     "repository": "", 
08:26:41     INFO -     "buildername": "release-mozilla-beta-firefox_mark_as_shipped", 
08:26:41     INFO -     "basedir": "/builds/slave/rel-m-beta-fx_mark_as_shipped-", 
08:26:41     INFO -     "buildnumber": 0, 
08:26:41     INFO -     "platform": null, 
08:26:41     INFO -     "version": "49.0b8", 
08:26:41     INFO -     "release_promotion": true, 
08:26:41     INFO -     "script_repo_revision": "68d24e6f784c7e375cf6c84c5c92496464d4f7e0", 
08:26:41     INFO -     "branch": "mozilla-beta", 
08:26:41     INFO -     "master": "http://buildbot-master71.bb.releng.use1.mozilla.com:8001/", 
08:26:41     INFO -     "revision": "68d24e6f784c7e375cf6c84c5c92496464d4f7e0", 
08:26:41     INFO -     "slavename": "bld-linux64-spot-074", 
08:26:41     INFO -     "next_version": "49.0b9", 
08:26:41     INFO -     "repo_path": "releases/mozilla-beta"
08:26:41     INFO - }
08:26:41     INFO - Overriding product with firefox
08:26:41     INFO - Overriding version with 49.0b8
08:26:41  WARNING - buildnumber could not be found within buildprops
Traceback (most recent call last):
  File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 110, in <module>
    MarkReleaseAsShipped().run_and_exit()
  File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 56, in __init__
    "mark-as-shipped",
  File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 1563, in __init__
    self._pre_config_lock(rw_config)
  File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 80, in _pre_config_lock
    raise Exception("%s not found in configs" % missing_props)
Exception: ['buildnumber'] not found in configs

3. as per Rail's observation: "if props.get(prop) should be probably `prop in props` so you handle when a prop is False or 0 or someting"

Comment 49

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8785223 [details]
Bug 1288434 - fix buildbot props in mark release as shipped.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/74500/diff/1-2/

Comment 50

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara] from comment #48)
> Sadly the build misbehaved with its first public representation. 
> 
> Few things here:
> 1. When I cloned the version-bump script in releasetasks, I forgot to rename
> the notification id

PR: https://github.com/mozilla/releasetasks/pull/195

> 2. There is an error with respect to the builnumber config
> 3. as per Rail's observation: "if props.get(prop) should be probably `prop
> in props` so you handle when a prop is False or 0 or someting"

https://reviewboard.mozilla.org/r/74498

Comment 51

[Rail Aliiev [:rail]]

Comment on attachment 8785223 [details]
Bug 1288434 - fix buildbot props in mark release as shipped.

https://reviewboard.mozilla.org/r/74500/#review73220

Comment 52

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8ca77a9ae536
fix buildbot props in mark release as shipped. r=rail a=release DONTBUILD

Comment 53

[Rail Aliiev [:rail]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8ca77a9ae536
https://hg.mozilla.org/releases/mozilla-aurora/rev/16bf54c7ba56
https://hg.mozilla.org/releases/mozilla-beta/rev/9982e9c96463
https://hg.mozilla.org/releases/mozilla-release/rev/db79f58ee7e1
https://hg.mozilla.org/releases/mozilla-esr45/rev/9d4b5d253fde

Comment 54

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8ca77a9ae536

Comment 55

[Mihai Tabara [:mtabara]⌚️GMT]

Leftovers in this bug once we confirm it's working with 49.0b9:

* adjust https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#4-post-release-step to mention it's automatic for release promotion
* adjust the release templates in release warrior (beta, esr, release) to contain only one step under "published release tasks part 1 part 2"

Comment 56

[Mihai Tabara [:mtabara]⌚️GMT]

Looks like we can't talk to ship-it from slaves:

07:31:18     INFO - ##### Finished create-virtualenv step (success)
07:31:18     INFO - #####
07:31:18     INFO - #####
07:31:18     INFO - ##### Running activate-virtualenv step.
07:31:18     INFO - #####
07:31:18     INFO - Running main action method: activate_virtualenv
07:31:18     INFO - #####
07:31:18     INFO - ##### Finished activate-virtualenv step (success)
07:31:18     INFO - #####
07:31:18     INFO - #####
07:31:18     INFO - ##### Running mark-as-shipped step.
07:31:18     INFO - #####
07:31:18     INFO - Running main action method: mark_as_shipped
07:31:19     INFO - Mark the release as shipped with 2016-09-02 14:31:19 timestamp
07:31:19     INFO - Starting new HTTPS connection (1): ship-it.mozilla.org
07:32:19     INFO - #####
07:32:19     INFO - ##### Finished mark-as-shipped step (failed)
07:32:19     INFO - #####
07:32:19    FATAL - Uncaught exception: Traceback (most recent call last):
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 1768, in run
07:32:19    FATAL -     self.run_action(action)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 1707, in run_action
07:32:19    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 1647, in _possibly_run_method
07:32:19    FATAL -     return getattr(self, method_name)()
07:32:19    FATAL -   File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 106, in mark_as_shipped
07:32:19    FATAL -     status='shipped', shippedAt=shipped_at)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 110, in update
07:32:19    FATAL -     url_template_vars=url_template_vars).content
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 59, in request
07:32:19    FATAL -     verify=self.verify, timeout=self.timeout, auth=self.auth)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
07:32:19    FATAL -     resp = self.send(prep, **send_kwargs)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
07:32:19    FATAL -     r = adapter.send(request, **kwargs)
07:32:19    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/requests/adapters.py", line 459, in send
07:32:19    FATAL -     raise ConnectTimeout(e, request=request)
07:32:19    FATAL - ConnectTimeout: HTTPSConnectionPool(host='ship-it.mozilla.org', port=443): Max retries exceeded with url: /csrf_token (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10f04d0>, 'Connection to ship-it.mozilla.org timed out. (connect timeout=60)'))
07:32:19    FATAL - Running post_fatal callback...
07:32:19    FATAL - Exiting -1
07:32:19     INFO - Running post-run listener: copy_logs_to_upload_dir
07:32:19     INFO - Copying logs to upload dir...
program finished with exit code 255
elapsedTime=60.585027

More from https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_mark_as_shipped-bm77-build1-build2.txt.gz

Not sure what's to be done yet but I'll leave this for future reference.

Comment 57

[Mihai Tabara [:mtabara]⌚️GMT]

Slaves can now talk to ship-it. We've tested the builder successfully whilst shipping 49.0 and 45.4.0esr. Everything worked as expected \o/

(In reply to Mihai Tabara [:mtabara] from comment #55)
> Leftovers in this bug once we confirm it's working with 49.0b9:
> 
> * adjust
> https://github.com/mozilla/releasewarrior/blob/master/how-tos/relpro.md#4-
> post-release-step to mention it's automatic for release promotion
> * adjust the release templates in release warrior (beta, esr, release) to
> contain only one step under "published release tasks part 1 part 2"

Done per https://github.com/mozilla/releasewarrior/pull/27

Comment 58

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.