1117849 - Can we have a way to determine the canonical URL for the /latest directory

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The URL such as https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-kk-eng/latest/ is updated whenever there's a new build, however there's no easy way to determine the URL that this is a link to.

Currently we are polling the contents of /latest to determine when a new build is available. At this point we download it, and store it for flashing to our devices.

We are now implementing a new device lab run remotely by a partner, who will be responsible for downloading and flashing the phones. It's therefore necessary for us to pass them a URL of the build to use. If we can't specify a unique URL for each build then we won't be able to rerun against previous builds, and our partner may even be picking up the wrong build. It also makes it harder for our partner to implement caching of the builds to prevent wasting bandwidth on identical builds.

Once solution that might work for us is to have a build.properties file in the directory that contains something like BUILD_URL=https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/b2g-inbound-flame-kk-eng/20150105071441/ however I'm open to other suggestions.

It may also make sense to have another way to trigger these tests with the URL provided as a parameter. I'd be happy to explore this also.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

catlee: Any idea who would be a good person to help with this?

Comment 2

[Chris AtLee [:catlee]]

The 'latest' symlink is created by this code:
http://hg.mozilla.org/build/mozharness/file/70550db540e8/scripts/b2g_build.py#l813

One idea I had was dropping a 'latest.txt' file or something into that directory alongside the symlink. The text file would contain the name of the directory that latest points to. Would something like that work? Is that what you had in mind as well with your build.properties file?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

That would work well for us. If this file is in the format suggested above we should be able to use a Jenkins plugin to set an environment variable for use by the remaining build steps. I guess if I write the patch, I'll be able to suggest the format. Assigning this to me. :)

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Create .properties file for symlinks. v1.0

As far as I could see an upload would need to use rsync, which isn't possible for a file in the same destination as the symlink. This patch uses echo to create the file via SSH. Let me know if there's a more suitable alternative. Also, is there a good way to test this patch?

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8548972 [details] [diff] [review]
Create .properties file for symlinks. v1.0

Review of attachment 8548972 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +808,5 @@
> +                   ]
> +            retval = self.run_command(cmd)
> +            if retval != 0:
> +                self.error("failed to delete properties file")
> +                self.return_code = 2

what's the purpose of removing it first? doesn't the subsequent step overwrite it?

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Chris AtLee [:catlee] from comment #5)
> Comment on attachment 8548972 [details] [diff] [review]
> Create .properties file for symlinks. v1.0
> 
> Review of attachment 8548972 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/b2g_build.py
> @@ +808,5 @@
> > +                   ]
> > +            retval = self.run_command(cmd)
> > +            if retval != 0:
> > +                self.error("failed to delete properties file")
> > +                self.return_code = 2
> 
> what's the purpose of removing it first? doesn't the subsequent step
> overwrite it?

We do the same with the symlink. It may be so that if there is an issue writing the file that we don't have a 'latest' that is out of date. An out of date file/symlink could go largely unnoticed, whereas if the file is not present any dependent processes would likely fail, therefore highlighting the problem.

Comment 7

[Chris AtLee [:catlee]]

The code for symlinks was added in bug 815784, and I think it doesn't apply here, since the property file won't be treated as a directory.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Create .properties file for symlinks. v1.1

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

What's a good way to test this? Should we land it on an alternate branch first?

Comment 10

[Chris AtLee [:catlee]]

We could land on Ash first.

Comment 11

[Jonathan Griffin (:jgriffin)]

You can also test these kinds of changes on try now, by modifying testing/mozharness/mozharness.json to point to a clone of the mozharness repo that you've made at your own hg user repo (like I've done at https://hg.mozilla.org/users/jgriffin_mozilla.com/mozharness/).

Ash is probably going away soon, at least as a mozharness sandbox.

Another option is to land this change on mozharness, retrigger jobs on cedar and make sure they don't fail and back your commit out promptly if they do.  This has the risk of getting merged to production by releng while you're doing your experimentation, though.

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

If I tested using try and a clone of mozharness I'm suspecting that the builds would not be uploaded anywhere and the symlinks therefore also wouldn't be created. Assuming that landing this and retriggering on cedar will create the appropriate symlink in https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/cedar-flame-kk-eng/ I shall go ahead with your second suggestion.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed in:
https://hg.mozilla.org/build/mozharness/rev/9892549f8789

Builds retriggered on cedar:
https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=2dea8b3c6c91

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like this is working, but the URL can't be resolved:

Here's the results from the current contents of https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/cedar-hamachi/latest.properties

Expected: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/cedar-hamachi-eng/20150122160649/

Actual: http://pvtbuilds.pvt.build.mozilla.org//pvt/mozilla.org/b2gotoro/tinderbox-builds/cedar-hamachi-eng/20150122160649

I see that the remote host is mapped to the correct host in https://hg.mozilla.org/build/mozharness/file/af1461521190/mozharness/mozilla/proxxy.py#l33 but not sure if that's particularly accessible from b2g_build.py. If it's not, perhaps it would be enough to use the relative URL in the properties file?

Comment 15

[Jordan Lund (:jlund)]

a mozharness patch has from this bug is in production

Comment 16

[Chris AtLee [:catlee]]

Hmmm...yeah, wouldn't use proxxy for that.

A relative path makes sense, the client requesting it already knows which host to connect to. We have bug 1016980 for cleaning up these names.

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Use relative paths in latest.properties file. v1.0

Comment 18

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed in:
https://hg.mozilla.org/build/mozharness/rev/6506f3e066b8

Builds retriggered on cedar:
https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=2dea8b3c6c91

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This appears to be working on cedar:
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/tinderbox-builds/cedar-flame-kk-eng/latest.properties

Comment 20

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Rename key to B2G_BUILD_PATH to reduce ambiguity. v1.0

Sorry for yet another patch. I realised today that using 'path' might get us into trouble when injecting this into environment variables, which is what we're planning to do with Jenkins.

Comment 21

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Landed in:
https://hg.mozilla.org/build/mozharness/rev/069e6f8e814b

Comment 22

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/6506f3e066b8

Comment 23

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is working beautifully! :)