530597 - Create an ssh poller replacement for MobileFtpPoller

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Description

[Aki Sasaki (not active)]

These have been moved to tinderbox-builds/1234567890/en-US.
We can't key off the multi-l10n tarball because that's only built nightly.

The MobileFtpPoller is pretty much a copy of the Talos one, and is hardcoded to look in a 1234567890 subdir of the directory you're looking at. I'd like to

a) have that subdir regex hardcode be optional and configurable and
b) allow for an additional subdir (e.g. 1234567890/en-US) to be appended.

Comment 1

[Aki Sasaki (not active)]

I spent a few hours trying to get this working.  I might be misunderstanding something, but afaict I can't get this to work without a lot of overhead, so I'm working on uploading the en-US tarball in the standard directory as the quick fix.

Comment 2

[Aki Sasaki (not active)]

Attachment: en-US tarball fix

Basically updating the multi-l10n maemo factory to only move the en-US deb into en-US/, but also upload the en-US tar.bz2 (at the root level).

Should fix things for now. I do think we'll end up revisiting this whole upload thing at some point... if we do I think we should think about switching to uploading to mobile/ instead of firefox/ and creating different dirs per platform like we do with firefox.  That's a whole different set of bugs though.

Comment 3

[Aki Sasaki (not active)]

Attachment: oops, remove that leftover rename

Same thing, with less testing cruft.

Comment 4

[Armen [:armenzg]]

Comment on attachment 414191 [details] [diff] [review]
oops, remove that leftover rename

We will revisit it for sure and I agree with your changes specially with posting to "mobile" instead of "firefox"

Comment 5

[Armen [:armenzg]]

Allow me to give you a patch that will help us both. It will be based on yours.

Comment 6

[Armen [:armenzg]]

Attachment: [buildbotcustom] upload tar ball to the root level as well as the "en-US" subdirectory

This copies the tar ball to the root level before uploading.
By doing this, we will have the tar ball under en-US as well as in the root level.
I want the Maemo l10n nightly builds to pull the tar ball and the deb from the same place (the "en-US' directory).

Let me test the patch on staging before I ask you for the review.

Comment 7

[Armen [:armenzg]]

Comment on attachment 414272 [details] [diff] [review]
[buildbotcustom] upload tar ball to the root level as well as the "en-US" subdirectory

Tested. It uploads the tar ball to root and "en-US" at the same time.

Comment 8

[Aki Sasaki (not active)]

Hah, I had the same fix a couple hours before my patch. I thought we wouldn't want two copies of the en-US tarball, I guess I was wrong.

However, it looks like the tinderbox-builds tarballs are getting picked up by production-mobile-master, so I'm lowering the priority on this.

I think the real fix here is to not use an ftp poller at all... scraping a single directory's web page doesn't tell us enough about subdirectories.  Instead, I'm thinking an ssh poller that ssh's in to stage and does a

  find root1 root2 -name WILDCARD -print -exec ls -lT {} \;

As long as we limit the roots to not be something ridiculous, this should be fast. Maybe time this out if it's taking too long?

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 414272 [details] [diff] [review]
[buildbotcustom] upload tar ball to the root level as well as the "en-US" subdirectory

This looks good, and thanks for the patch. However, I'm removing the review on this due to bug shift. If the ssh poller doesn't work, I'll r this and land it.

Comment 10

[Aki Sasaki (not active)]

I'd like to see this, but I'm hoping that with

a) schedulerdb + pods
b) removing xulrunner
c) make upload
d) triggering tests in schedulerdb instead of polling

this will become even less of an issue.

Comment 11

[Aki Sasaki (not active)]

Not planning on working on this, and it seems like I was the one that wanted this most.

> I'd like to see this, but I'm hoping that with
> 
> a) schedulerdb + pods

Coming soon to Maemo

> b) removing xulrunner

Done

> c) make upload

Coming soon to Maemo (hopefully q3)

> d) triggering tests in schedulerdb instead of polling

Ditto

As well as splitting the multilocale build (hopefully q3) will make this completely optional.