Closed Bug 914111 Opened 9 years ago Closed 9 years ago

Make gecko and gaia archives from device builds publicly available

Categories

(Release Engineering :: General, defect)

All
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mwu, Assigned: aki)

References

Details

(Whiteboard: [b2g] [participation])

Attachments

(2 files, 6 obsolete files)

We should make gaia and gecko tarballs/zips available. In other words, b2g-26.0a1.en-US.android-arm.tar.gz and gaia.zip . They're generated from publicly available code and contain no proprietary bits.
For all device builds?
Whiteboard: [b2g]
Hmm, good question. Maybe it would be better if we stuck to the emulator and publicly available devices. So, Hamachi, Unari, and Inagi. (and Nexus 4 when that starts)
Just to be explicit then, which devices shouldn't we publish gaia/gecko for?
To err on the side of caution, let's not publish for Leo and Helix at the moment.
This is very exciting news. 

I believe this will be a significant improvement over the current situation for many in our community who have devices but are unable to track the tip of development where bug finding and fixing is likely to be easiest and more impactful.

How is this prioritized and is there anything that I can do to help move this forward.
Whiteboard: [b2g] → [b2g] [participation]
Here's my situation: I vet community members to Gaia UI Tests (in Python), have them start with Desktop, and then in warranted cases, give them an Inari + send them builds, nearly daily -- I'd like to change it (here) so they (and anyone else who bootloader-flashes a ZTE Open, etc.) can find and test daily builds.

:mwu, per my understanding, we also shouldn't publish for Hamachi/Buri, no?  (Even though those devices, I've confirmed, are otherwise identical to their shipping counterpart.)  Or is there a completely Mozilla RIL-only build for that, too?  (As you know, we flash with comm-RIL repacks, but flash with -n).
When dealing with third parties (acquiring top tier content) for Firefox OS, we frequently need to have developers using cutting edge b2g features.  It's extremely frustrating because we can't point them to download images and instructions; instead we must ask them to ship us the phone so we can update them, then mail them back.  This makes the content acquisition process take significantly longer than it needs to.

I've been working on an add on to help developers quickly flash back and forth between 1.0, 1.1, 1.2, 1.3+.
https://github.com/nickdesaulniers/fxos-version-control

It's not ready yet, I had a working prototype that I've started rewriting to be a cross platform version of the initial prototype (where everything was hard coded).  It would be great to ship an add on for Firefox that would flash your phone for you.  It could even recover phones that were soft bricked if we provide instructions for how to manually reboot each phone into its bootloader.

The biggest blocker is that device images are private, except for geeksphones.  The other is that bootloaders are locked on devices shipping to end users.  Both could probably be fixed via branding agreements.
Hey guys, now that we're transitioning to Buri devices, this issue got crucial.
Any updates?
Flags: needinfo?(nick)
I think the same concerns from bug 917621 would apply here - that is, we may not be able to redistribute these files because they contain compiled copies of audio/video codecs.
(In reply to Chris AtLee [:catlee] from comment #9)
> I think the same concerns from bug 917621 would apply here - that is, we may
> not be able to redistribute these files because they contain compiled copies
> of audio/video codecs.

Not a concern. These files only contain free codecs - VP8/Theora/Vorbis/etc.
Is that a green light for publishing our builds (for released devices)?
Flags: needinfo?(nick) → needinfo?(mwu)
I wouldn't have opened this bug if I thought there were issues. I don't know what people are waiting on.
Flags: needinfo?(mwu)
Let's do this!
Do we need this for the per-checkin builds, or just for nightly builds?

Do we need to have the engineering builds publicly available as well?
This has some implications for disk space usage on the ftp server. Probably not going to be a problem, but please check with me before we go live with it.
Nightly is likely enough if we need to watch the disk space usage. Engineering builds can be pretty useful, so I would like them as well if possible.
Assignee: nobody → aki
Attached patch public_upload_wip (obsolete) — Splinter Review
https://github.com/escapewindow/mozharness/compare/gecko-gaia-upload
Ready for testing.

Pretty sure the b2g/nightly and b2g/tinderbox-builds dirs on stage.m.o will Just Work; I'm a lot less sure about the b2gtry stuff, which may need a user account configured on stage.
Attached patch in-tree configsSplinter Review
Looks like there's no named package-gecko target to package gecko for the emulator, so there will be no gecko package for emulator{,jb}.
Attachment #8359554 - Flags: review?(catlee)
Comment on attachment 8359553 [details] [diff] [review]
in-tree configs

It'll be easier to test once this lands, because mapper.
Attachment #8359553 - Flags: review?(catlee)
Able to test dep builds without mapper...

There's a b2gtry on stage.m.o, yay!
I'll have to tweak the upload paths to match the desktop stage.m.o paths, boo.
Comment on attachment 8359554 [details] [diff] [review]
nuke the unused emulator-ics and unagi config dirs

Review of attachment 8359554 [details] [diff] [review]:
-----------------------------------------------------------------

I'd rather finish up bug 916134 and nuke 'emulator'. Nuking unagi is fine.
Attachment #8359554 - Flags: review?(catlee) → review-
Attachment #8359553 - Flags: review?(catlee) → review+
https://hg.mozilla.org/mozilla-central/rev/82a4e7daeb13
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Is there a public URL now for these builds?
Oops, forgot to [leave open]
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I think this might work, but I still need to rework the script to use the new config format: https://github.com/escapewindow/mozharness/commit/78e98477cb24775fa2d36aa1dc05e00206307825
Nick:
On a high level, do you see anything objectionable in this patch?
I am porting some upload.py logic into b2g_build.py... I thought that might be easiest.

I'm thinking this may take quite a bit of testing, and I'd rather find out I'm on the wrong path earlier.
Attachment #8358659 - Attachment is obsolete: true
Attachment #8360779 - Flags: feedback?(nthomas)
Comment on attachment 8360779 [details] [diff] [review]
(needs testing) public_upload_wip

This cleanup is awesome to see!

>diff --git a/configs/b2g/releng-beta.py b/configs/b2g/releng-beta.py
>+    "upload": {
>+        "default": {
>+            "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"),
>+            "ssh_user": "b2gbld",
>+            "upload_remote_host": "pvtbuilds2.dmz.scl3.mozilla.com",
>+            "upload_remote_path": "/pub/mozilla.org/b2g/tinderbox-builds/%(branch)s-%(target)s/%(buildid)s",

I think Dustin would like to do away with pvtbuilds2, so we could swap to using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a path prefix of '/mnt/pvt_builds' for anything that should stay private though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub).

>diff --git a/configs/b2g/releng-emulator.py b/configs/b2g/releng-emulator.py
>+            "post_upload_cmd": "post_upload.py --tinderbox-builds-dir ${branch}s-%(target)s -p b2g -i %(buildid)s --revision %(revision)s --release-to-tinderbox-dated-builds",            "post_upload_nightly_cmd": "post_upload.py --tinderbox-builds-dir %(branch)s-%(target)s -b %(branch)s -p b2g -i %(buildid)s --revision %(revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated",
>+        },
>+    },

Missing newline prior to post_upload_nightly_cmd.

>diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py
>+    def _do_postupload_upload(self, upload_dir, ssh_key, ssh_user, remote_host,
...
>+            for dirpath, dirname, filenames in os.walk(upload_dir):
>+                for f in filenames:
>+                    path = '%s/%s' % (dirpath, f)

os.path.join() for portability ?

>+            postupload_key = 'post_upload_ngithly_cmd'

Typo in nightly.

>+        if os.path.exists(dirs['abs_public_upload_dir']) and self.config['upload'].get('public'):

Is it possible we might get here with an empty dir, or will gaia always get uploaded ?
Attachment #8360779 - Flags: feedback?(nthomas) → feedback+
(In reply to Nick Thomas [:nthomas] from comment #31)
> Comment on attachment 8360779 [details] [diff] [review]
> (needs testing) public_upload_wip
> 
> This cleanup is awesome to see!

Thanks!

> >diff --git a/configs/b2g/releng-beta.py b/configs/b2g/releng-beta.py
> >+    "upload": {
> >+        "default": {
> >+            "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"),
> >+            "ssh_user": "b2gbld",
> >+            "upload_remote_host": "pvtbuilds2.dmz.scl3.mozilla.com",
> >+            "upload_remote_path": "/pub/mozilla.org/b2g/tinderbox-builds/%(branch)s-%(target)s/%(buildid)s",
> 
> I think Dustin would like to do away with pvtbuilds2, so we could swap to
> using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a
> path prefix of '/mnt/pvt_builds' for anything that should stay private
> though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub).

I think we assume the remote path is equivalent to the path in the URL, so I may need to create a /pvt softlink to /mnt/pvt_builds.  Puppet?

> >diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py
> >+    def _do_postupload_upload(self, upload_dir, ssh_key, ssh_user, remote_host,
> ...
> >+            for dirpath, dirname, filenames in os.walk(upload_dir):
> >+                for f in filenames:
> >+                    path = '%s/%s' % (dirpath, f)
> 
> os.path.join() for portability ?

I'm explicitly using '/' because this is the directory / filepath that will be passed over ssh to the post_upload.py call, and I assume we will never have a windows upload server.  I may be wrong, but I hope not.

Maybe I'll add a comment.

> >+        if os.path.exists(dirs['abs_public_upload_dir']) and self.config['upload'].get('public'):
> 
> Is it possible we might get here with an empty dir, or will gaia always get
> uploaded ?

Gaia's always added.  I nuke the public upload dir during clobber, and only recreate it by copy_to_upload_dir() calls, so I think we shouldn't have an empty dir.
(In reply to Aki Sasaki [:aki] from comment #32)
> > I think Dustin would like to do away with pvtbuilds2, so we could swap to
> > using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a
> > path prefix of '/mnt/pvt_builds' for anything that should stay private
> > though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub).
> 
> I think we assume the remote path is equivalent to the path in the URL, so I
> may need to create a /pvt softlink to /mnt/pvt_builds.  Puppet?

There actually is one already, and in your patch we're using pvtbuilds.pvt whenever the path starts with /pvt. So that leaves the private builds in /pub case. I'm actually don't understand what the distinction is between pub and pvt any more, and the access controls are all wacky (yay, legal issues). Perhaps it's time to rationalise that too.
Testing's going well for private tinderbox-builds (with softlink!)
Getting this for post_upload.py public nightly though:

13:04:24     INFO - Copy/paste: ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa dev-stage01.srv.releng.scl3.mozilla.com "post_upload.py --tinderbox-builds-dir mozilla-central-hamachi-eng -b mozilla-central -p b2g -i 20140117152452 --revision bbe08810e87b --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated /tmp/tmp.uSXhlgN814/ \"/tmp/tmp.uSXhlgN814//sources.xml\" \"/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip\" \"/tmp/tmp.uSXhlgN814//gaia.zip\" \"/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.tar.gz\""
13:04:24     INFO -  sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-hamachi-eng', '-b', 'mozilla-central', '-p', 'b2g', '-i', '20140117152452', '--revision', 'bbe08810e87b', '--release-to-tinderbox-dated-builds', '--release-to-latest', '--release-to-dated', '/tmp/tmp.uSXhlgN814/', '/tmp/tmp.uSXhlgN814//sources.xml', '/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip', '/tmp/tmp.uSXhlgN814//gaia.zip', '/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.tar.gz']
13:04:24     INFO -  Traceback (most recent call last):
13:04:24     INFO -    File "/usr/local/bin/post_upload.py", line 551, in <module>
13:04:24     INFO -      func(options, upload_dir, files)
13:04:24     INFO -    File "/usr/local/bin/post_upload.py", line 201, in ReleaseToLatest
13:04:24     INFO -      CopyFileToDir(f, upload_dir, latestPath)
13:04:24     INFO -    File "/usr/local/bin/post_upload.py", line 103, in CopyFileToDir
13:04:24     INFO -      tmp_fd, tmp_path = tempfile.mkstemp(dir=dest_dir)
13:04:24     INFO -    File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp
13:04:24     INFO -      return _mkstemp_inner(dir, prefix, suffix, flags)
13:04:24     INFO -    File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner
13:04:24     INFO -      fd = _os.open(file, flags, 0600)
13:04:24     INFO -  OSError: [Errno 13] Permission denied: '/home/ftp/pub/b2g/nightly/latest-mozilla-central/tmpnykvkH'
13:04:24    ERROR - Return code: 1
13:04:24    ERROR - failed to run post_upload.py --tinderbox-builds-dir mozilla-central-hamachi-eng -b mozilla-central -p b2g -i 20140117152452 --revision bbe08810e87b --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated!

I think I just need to track down an issue with my cmdline... maybe the trailing slash?
Oh, /home/ftp/pub. Hm.
(In reply to Aki Sasaki [:aki] from comment #35)
> Oh, /home/ftp/pub. Hm.

I think this is a ffxbld/b2gbld issue, since stage uses ffxbld and pvtbuilds uses b2gbld, but dev-stage01 can't have both users own the same directory tree.  Did a recursive chmod g+w on b2g/nightly and am trying again.
Attached patch gecko-gaia-upload (obsolete) — Splinter Review
Phew, that took a lot longer than I expected.

Successful staging dep run:
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_emulator_dep/builds/6/steps/run_script/logs/stdio

Successful staging nightly run:
http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_hamachi_eng_nightly/builds/6/steps/run_script/logs/stdio
Attachment #8359554 - Attachment is obsolete: true
Attachment #8360779 - Attachment is obsolete: true
Attachment #8362066 - Flags: review?(nthomas)
Comment on attachment 8362066 [details] [diff] [review]
gecko-gaia-upload

Review of attachment 8362066 [details] [diff] [review]:
-----------------------------------------------------------------

I didn't look at this in great detail because I ended up doing that for the earlier feedback request, and the interdiff was small. There is one big issue though ...

::: configs/b2g/releng-emulator.py
@@ +17,5 @@
> +    "upload": {
> +        "default": {
> +            "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"),
> +            "ssh_user": "b2gbld",
> +            "upload_remote_host": "pvtbuilds.pvt.build.mozilla.org",

The effect of taking this as-is would be to put private bits on the public ftp server. If you log in to both pvtbuilds2 and pvtbuilds.pvt and compare /pub/mozilla.org/b2g/nightly/ you'll see what I mean (the former has private stuff, the later is public b2g desktop builds). 

My suggestion to swap to this was actually a bad idea, at least in the short-term, so lets just stick with pvtbuilds2 for now, in all these config files where it's already used.
Attachment #8362066 - Flags: review?(nthomas) → review-
Comment on attachment 8363065 [details] [diff] [review]
back to pvtbuilds2

r+. Please watch carefully after this lands in case we missed something.
Attachment #8363065 - Flags: review?(nthomas) → review+
Comment on attachment 8363065 [details] [diff] [review]
back to pvtbuilds2

Landed on default, kicked off emulator builds on cypress.
http://hg.mozilla.org/build/mozharness/rev/9e6b874c256b
Attachment #8363065 - Flags: checked-in+
Attached patch silence_public_upload (obsolete) — Splinter Review
Ok. The uploads look good on Cypress, and we have softlinks on pvtbuilds tinderbox-builds.

We are getting this message I'd like to patch (just skip it for now; we can parse post_upload.py output later if needed).  Not urgent:

18:12:50     INFO - Copy/paste: ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org "post_upload.py --tinderbox-builds-dir cypress-generic-debug -p b2g -i 20140120083431 --revision 45a22c4901bb --release-to-tinderbox-dated-builds /tmp/tmp.7XJbZeIsPm/ /tmp/tmp.7XJbZeIsPm//gaia.zip /tmp/tmp.7XJbZeIsPm//sources.xml /tmp/tmp.7XJbZeIsPm//gaia-tests.zip /tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip /tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.tests.zip"
18:12:51     INFO -  sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'cypress-generic-debug', '-p', 'b2g', '-i', '20140120083431', '--revision', '45a22c4901bb', '--release-to-tinderbox-dated-builds', '/tmp/tmp.7XJbZeIsPm/', '/tmp/tmp.7XJbZeIsPm//gaia.zip', '/tmp/tmp.7XJbZeIsPm//sources.xml', '/tmp/tmp.7XJbZeIsPm//gaia-tests.zip', '/tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip', '/tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.tests.zip']
18:12:51     INFO -  http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/gaia.zip
18:12:51     INFO -  http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/sources.xml
18:12:51     INFO -  http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/gaia-tests.zip
18:12:51     INFO -  http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip
18:12:52     INFO -  http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/b2g-29.0a1.en-US.android-arm.tests.zip
18:12:53     INFO - Return code: 0
18:12:53     INFO - Upload successful: http://stage.mozilla.org//tmp/tmp.7XJbZeIsPm/
Attachment #8363399 - Flags: review?(nthomas)
(I'll merge to production tomorrow morning.)
Attachment #8363399 - Flags: review?(nthomas) → review+
(In reply to Aki Sasaki [:aki] from comment #43)
> (I'll merge to production tomorrow morning.)

I just did that: https://hg.mozilla.org/build/mozharness/rev/8e4b5b9fbe71
I think we're done here.
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Is there a public url for these builds now (if so, where)?
Flags: needinfo?(aki)
/nightly/ doesn't seem to have all device builds, only emulator builds and hamachi builds. Is this intentional?  Should we be pointing people to /tinderbox-builds/ then?
The logs have the upload locations.
* not all device builds upload depend builds to tinderbox-builds
* not all device builds have nightly builds
* since this was just enabled, we haven't done a full set of nightlies across all branches yet

What device/branch are you specifically concerned about?
I'd like to see combinations of:

Devices: [unagi, inari, hamachi, mako (n4)]
FxOS versions: [1.0, 1.1, 1.2, 1.3, master (1.4)]

We send many devices to third party app developers and have been in a pinch when they need to update devices.  It would be great to point them to a specific dir for each device so they could quickly find a build for what version of FxOS they're looking for.

I'd also like to write an add on that can do this automatically when a phone is connected, so having a known dir to look for will help.
(In reply to Nick Desaulniers [:\n] from comment #52)
> I'd like to see combinations of:
> 
> Devices: [unagi, inari, hamachi, mako (n4)]
> FxOS versions: [1.0, 1.1, 1.2, 1.3, master (1.4)]
> 
> We send many devices to third party app developers and have been in a pinch
> when they need to update devices.  It would be great to point them to a
> specific dir for each device so they could quickly find a build for what
> version of FxOS they're looking for.
> 
> I'd also like to write an add on that can do this automatically when a phone
> is connected, so having a known dir to look for will help.

I think we're going to have to manually keep that updated, especially since 1.4 is going to move to mozilla-aurora then to b2g30_v1_4 later.

Maybe we should have a b2g equivalent of http://nightly.mozilla.org/ ?

Also, unagi builds are no longer supported.
Depends on: 963148
Ok, this is eating a ton of disk space on ftp.m.o... overall ~4TB aiui, and it's putting all infrastructure at risk.

I'm thinking about:
* turning off all public try uploads
* turning off all public depend uploads except for emulator (nightly only for devices)

Any objections?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Sorry, I mispoke on IRC. There is 3TB of space used by /pub/mozilla.org/b2g/tindebox-builds/*gecko (eg mozilla-inbound-linux32_gecko some 270GB), but I don't see any gecko/gaia archives there. 

I'm sure they're contributing elsewhere, because there's a clear downward trend on the free space which starts on the 21st or 22nd, but don't have numbers yet.
Attachment #8363399 - Flags: checked-in+ → checked-in-
Attachment #8363065 - Flags: checked-in+ → checked-in-
See Also: → 963768
Whiteboard: [b2g] [participation] → [b2g] [participation] [tree-closing]
Whiteboard: [b2g] [participation] [tree-closing] → [b2g] [participation]
(In reply to Aki Sasaki [:aki] from comment #54)
> Ok, this is eating a ton of disk space on ftp.m.o... overall ~4TB aiui, and
> it's putting all infrastructure at risk.
> 
> I'm thinking about:
> * turning off all public try uploads
> * turning off all public depend uploads except for emulator (nightly only
> for devices)
> 
> Any objections?

Nick: I'm willing to do the above, which should limit the amount of additional load on ftp.m.o, but only if we think it won't kill ftp.m.o again anyway.  Do you have any more data?

If the above won't help, maybe this bug should be morphed into an S3 upload or something.

... also, I'm not finding the ftp filling up bug to link here: do you have it handy?
Flags: needinfo?(nthomas)
Depends on: 963768
Pretty much the same, but:

* combined the two previous patches
* updated the new fota configs
* removed the public upload config from the try config file
* only pushes public bits if self.query_is_nightly()

The other change we might want is to add nightly builds for more devices + emulator.
Attachment #8363065 - Attachment is obsolete: true
Attachment #8363399 - Attachment is obsolete: true
Attachment #8367530 - Flags: review?(nthomas)
[12:35]	<hwine>	aki: re https://bugzilla.mozilla.org/show_bug.cgi?id=914111 - check in with catlee before you deploy, since that will impact the ipsec traffic issue
Flags: needinfo?(catlee)
should be fine if it's only pushing bits nightly
Flags: needinfo?(catlee)
Comment on attachment 8367530 [details] [diff] [review]
public_upload_nightly_only

Sorry, I've expended all my tokens and will have to look at this first thing next week.
Comment on attachment 8367530 [details] [diff] [review]
public_upload_nightly_only

I agree with catlee that this should be OK on ftp.m.o. Sorry for the delay.
Attachment #8367530 - Flags: review?(nthomas) → review+
Flags: needinfo?(nthomas)
Ok, we're uploading to ftp.m.o again, but only for nightlies (not depend builds, not try)
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Aaaaaaaaaaaand relanded try bustage fix http://hg.mozilla.org/build/mozharness/rev/b66a53a7144d
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.