408674 - Need a command-line script to automatically add new entries

Status: RESOLVED FIXED

(Webtools :: Bouncer, enhancement, P1)

Description

[Michael Morgan [:morgamic]]

Bouncer needs to be able to add new entries from the command line in order to fit into the build bootstrap.

When a build completes, and bits are staged, build has to manually create entries.  We need to create a simple utility that can interface with Bouncer and create product entries as they are completed.

John or Rob can you list what params are available from the bootstrap?  Bouncer needs:
1) product name
2) platform
3) locale

Will these be used one-at-a-time or will the script need to accept arrays of values?

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

For each release, we ship 9 products; complete-installer, partialupdate, completeupdate... each on 3 o.s. For each of these 9 products, we have to enter three values: productname, o.s., fullpathname. 

(Interesting, we do not specify locales, just only en-US).

I think passing variable length array of triplet-values makes most sense, if thats doable. Note: some releases do not have updates, so would be passing a shorter array.



Here's one example set of data:

Firefox-3.0b2 	win 	/firefox/releases/3.0b2/win32/en-US/Firefox%20Setup%203.0%20Beta%202.exe
Firefox-3.0b2 	linux 	/firefox/releases/3.0b2/linux-i686/en-US/firefox-3.0b2.tar.bz2
Firefox-3.0b2 	osx 	/firefox/releases/3.0b2/mac/en-US/Firefox%203.0%20Beta%202.dmg
Firefox-3.0b2-Complete 	osx 	/firefox/releases/3.0b2/update/mac/en-US/firefox-3.0b2.complete.mar
Firefox-3.0b2-Complete 	linux 	/firefox/releases/3.0b2/update/linux-i686/en-US/firefox-3.0b2.complete.mar
Firefox-3.0b2-Complete 	win 	/firefox/releases/3.0b2/update/win32/en-US/firefox-3.0b2.complete.mar
Firefox-3.0b2-Partial-3.0b1 	osx 	/firefox/releases/3.0b2/update/mac/en-US/firefox-3.0b1-3.0b2.partial.mar
Firefox-3.0b2-Partial-3.0b1 	win 	/firefox/releases/3.0b2/update/win32/en-US/firefox-3.0b1-3.0b2.partial.mar
Firefox-3.0b2-Partial-3.0b1 	linux 	/firefox/releases/3.0b2/update/linux-i686/en-US/firefox-3.0b1-3.0b2.partial.mar

Comment 2

[Robert Helmer [:rhelmer]]

I think comment 1 makes sense, once bouncer supports multiple locales we should just switch back to making it use the SHA1SUMS file instead of this kind of explicit manual configuration, IMHO, so configuration just becomes a mapping of "Firefox-3.0b2" -> "path/to/sha1sum".

Passing an array to a command-line tool seems kind of weird to me; doing it one-at-a-time seems a little cleaner like:

$ ./add_bouncer_entry -h
Usage: add_bouncer_entry <product> <platform> <path>
$ ./add_bouncer_entry Firefox-3.0b2 win '/firefox/...'

What would it look like on the command line to pass this data in as an array of tuples? Like:

$ ./add_bouncer_entry [(Firefox-3.0b2, win, /firefox/...), (Firefox-3.0b2, linux, /firefox/...)]

That seems pretty strange for a command line tool... if we want to have more of the logic inside this tool we should just make it a little smarter:

$ ./add_bouncer_entry -h
Usage: add_bouncer_entry [--product, --version, --no-partials, --no-complete, 
                          --partial-override]
$ ./add_bouncer_entry --release=Firefox --version=3.0b2

The script would infer the path name, and assume that we want partials and completes and that they point to the previous release unless overridden.

This is kind of tricky because we have weird rules about filenames if it's a beta or alpha release, so it's need to parse the version string and do the right thing, but we could teach all of this to the script instead of having it elsewhere. If we do it this way, it'd be nice to have a class for each deliverable which is instantiated with some basic info and has methods to give us this kind of data; we could use this for tons of other stuff e.g.:

>>> build = Build('Firefox', '3.0b2', 'win', 'en-US')
>>> print build.getFtpPath()
/firefox/releases/3.0b2/win32/en-US/Firefox%20Setup%203.0%20Beta%202.exe

So, I'd rather have either:
a) a dumb script that takes entries one-at-a-time, which means there needs to be an app feeding it this info that knows wtf it is doing, or
b) a smart script that does the right thing without my intervention

I don't think b would be that hard to do, and if we split the build-specific stuff out to classes it'd be easy to deal with and beneficial for other scripts like this.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found this during triage... any update?

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

morgamic: gentle ping. Any update on having automation be able to add entries to bouncer like this?

Comment 5

[[e-mail] (:bhearsum)]

I'm going to be doing the Bootstrap/Buildbot side of this - I'm happy to tackle this part too, if I can get some guidance first.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

We now add 3 products and 24 locations manually for 3.6-based releases. It'd be really great if we could add those in some automated fashion.

Comment 7

[Fred Wenzel [:wenzel]]

This can probably become a management command for the django app in bug 535808.

Comment 8

[Fred Wenzel [:wenzel]]

I added a management command to the tuxedo branch (commit: http://github.com/fwenzel/tuxedo/commit/aaa445b04c544e35ee7d04c87419ff717a3024e6):

$ python manage.py mirror_add_location --help
Usage: manage.py mirror_add_location [options] <product_name> <os_name> <location_path>

Add a new location to the database.
All arguments are mandatory. Add the --locale option to define a locale
for this location.

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=all output
  --settings=SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath=PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Print traceback on exception
  -l LOCALE, --locale=LOCALE
                        Locale code for this location. Default: none.
                        Currently known locales: af, ak, ar, as, ast-ES, be,
                        bg, bn-BD, bn-IN, br-FR, ca, ca-valencia, cs, cy, da,
                        de, de-AT, de-CH, de-DE, dsb, el, en-AU, en-CA, en-GB,
                        en-NZ, en-US, en-ZA, eo, es-AR, es-CL, es-ES, es-MX,
                        et, eu, fa, fi, fj-FJ, fr, fur-IT, fy-NL, ga-IE, gl,
                        gu-IN, he, hi, hi-IN, hr, hsb, hu, hy-AM, id, is, it,
                        ja, ka, kk, kn, ko, ku, la, lt, lv, mg, mi, mk, ml,
                        mn, mr, nb-NO, ne-NP, nl, nn-NO, nr, nso, oc, or, pa-
                        IN, pl, pt-BR, pt-PT, rm, ro, ru, rw, si, sk, sl, sq,
                        sr, sr-Latn, ss, st, sv-SE, ta, ta-IN, ta-LK, te, th,
                        tn, tr, ts, tt-RU, uk, ur, ve, vi, wo, xh, zh-CN, zh-
                        TW, zu.
  --version             show program's version number and exit
  -h, --help            show this help message and exit

$ python manage.py mirror_add_location Firefox-3.0b2 win "/firefox/releases/3.0b2/win32/en-US/Firefox%20Setup%203.0%20Beta%202.exe" --locale="en-US"
Successfully added location ('Firefox-3.0b2', 'win', '/firefox/releases/3.0b2/win32/en-US/Firefox%20Setup%203.0%20Beta%202.exe') for locale en-US with id 35598.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

And
 python manage.py mirror_add_product 'Firefox-3.0b2'
by the looks. Woo!

Is the locale handling changing or have we just not exposed it before ?

Comment 10

[Fred Wenzel [:wenzel]]

(In reply to comment #9)
> And
>  python manage.py mirror_add_product 'Firefox-3.0b2'
> by the looks. Woo!

Yes, indeed, I thought we need that as well, otherwise you'd need to log in manually before you can actually add a location when its product doesn't exist yet.

Anything else that needs to be added from the command line?

> Is the locale handling changing or have we just not exposed it before ?

It is changing. The patch for that has been lingering on trunk for-ev-ar and now I am tying the strings together, so we can properly deploy it. The basic idea is that we store in the database which locale a specific file is for, and removed the cheesy "replace en-US with whatever locale the user asked for" hack.

Note that there will be one location entry per locale in the DB, it's not a many-to-many relationship of any sort. (In other words, the bot just needs to call this command once for each locale, creating a new entry each time.)

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

That may have an interesting effect on the sentry process if it has to poll 75 locales per platform instead of just one. justdave, is sentry already parallelised ? Let me know if there is a better place to discuss this.

Comment 12

[Fred Wenzel [:wenzel]]

(In reply to comment #11)
> That may have an interesting effect on the sentry process if it has to poll 75
> locales per platform instead of just one.

Traffic-wise it should be fine, as we don't pull the file down, we just look if it's there.

> justdave, is sentry already
> parallelised ? Let me know if there is a better place to discuss this.

Not quite yet: That's bug 400876.

Comment 13

[[e-mail] (:bhearsum)]

Any idea when this is headed into production? We're still trying to automate adding bouncer entries with each release...

Comment 14

[Fred Wenzel [:wenzel]]

I am planning on finishing the current bouncer work next week, and I'll send out an email about it. We'll have to work together (Webdev, IT, and Build) to get this staged and tested properly, and when that works we can push it to production.