481304 - Add functionality to import/upload l10n priority and metric data into tiki db for l10n dashboard

Status: RESOLVED FIXED

(support.mozilla.org :: Localization, task, P1)

Description

[E. Cooper [:ecooper]]

The core of the new l10n dashboard is getting data that's currently hand/script tabulated into the new tiki db table. We'll need a script or admin page to import this data.

Currently, Laura is talking with IT for suggestions (ease of use and sustainability of solution).

In the meantime, if we could attach a sample of what we would be pulling it, it'd provide a good start.

Comment 1

[E. Cooper [:ecooper]]

Laura any update on this front?

Comment 2

[Laura Thomson :laura]

I think the right way to do this is to run a cron job on prod that pulls data from a specified location in sumotools, and if it finds it, then updates the database.  

Not sure of any connectivity issues between prod and the outside world, which is why we need IT's input.  I pinged Dave in IRC earlier but didn't get an answer.  Ccing server-ops/mrz on this bug for feedback.

Comment 3

[David Tenser [:djst]]

Laura's proposed solution in comment 2 has been approved by IT (Dave) so I think we're good to go. We still need to settle on a sensible format to import the data from.

Comment 4

[E. Cooper [:ecooper]]

(In reply to comment #3)
> Laura's proposed solution in comment 2 has been approved by IT (Dave) so I
> think we're good to go. We still need to settle on a sensible format to import
> the data from.

CSV would be easiest. We can use the first line to define the list (list name, list title, list description) and then have the remaining list be the articles.

If one file would have multiple lists, we can just define a list delimitator.

Comment 5

[E. Cooper [:ecooper]]

I feel like I've spelled something incorrectly.

I imagine, we'll also eventually need some sort of tiki/web-based solution for editing, adding to, and deleting lists.

Obviously, the cron comes first.

Comment 6

[E. Cooper [:ecooper]]

Cww do you have an preference on list format at all? Or should I just do it?

Comment 7

[[:Cww]]

CSV works for me but really I'm using perl and there's a perl module for everything so I can adapt if needbe.  That said, a simple mysql command to shove data from one select function into a table is by far the easiest since retrieval would just be sql queries.

To be fair, I'm not sure what I would be doing with these lists... I assume that there'll be some gui-dashboard pretty htmlbased thingy that the localizers will be seeing.  As for summarizing the results each week, what will end up happening is I'll be scripting something to shove into a new table in sumo_processed on sumotools.  So whatever you output, I'll just be reading into a table.

Comment 8

[David Tenser [:djst]]

(In reply to comment #6)
> Cww do you have an preference on list format at all? Or should I just do it?

From my POV, I'm totally fine with you deciding what format you want to import, since Cheng can adapt the dumped files to whatever format you choose.

Comment 9

[E. Cooper [:ecooper]]

Do we have a folder within webroot where we keep maintenance/cli scripts (sort of like Mediawiki does)? If we don't, can we start one?

Keeping scripts out of webroot would be ideal, but if we have to write wrappers within webroot like we currently do for indexing forums, we might as well just have a folder some place within webroot.

Cww, if you're using perl, we can just use the PHP serialization module on CPAN. That'd make things super easy for me. ;)

Comment 10

[E. Cooper [:ecooper]]

Attachment: Cron script to fetch metrics/page lists from sumotools

I'm still sort of in the dark about the network/location of sumotools, but this should work. If Cww is okay with dealing with PHP::Serialization in his Perl scripts. If not, it's easily changed. 

The only caveat is that tiki doesn't have a centralized location for maintenance scripts like Mediawiki, so I'm not sure where to put it.

Comment 11

[Laura Thomson :laura]

Comment on attachment 367539 [details]
Cron script to fetch metrics/page lists from sumotools

Should be fine *assuming* we have allow_url_fopen on in prod.  Let's test it on stage.

Comment 12

[E. Cooper [:ecooper]]

My question is where to put it. If we put it in /scripts, it'll likely need to be run from somewhere in /webroot via a wrapper anyway since it uses tiki libraries.

Comment 13

[Laura Thomson :laura]

I'd put it in /scripts for safety.  Check with IT what the include_path is on prod, and you can always set_include_path so the library inclusions will work correctly.

Comment 14

[E. Cooper [:ecooper]]

r23552 in trunk.

I created tiki_script_base.php. It allows scripts using tikilib to run from /scripts. So we should be set. It just took a little tinkering to figure out what include paths needed to be set (assuming webroot can be accessed from /scripts via ../webroot/). Any further maintenance scripts that need access to tiki can be run from /scripts now.

fetch_sumo_lists.php also uses curl, which it should have been using from the beginning. I'm not sure why I left that in.

Now we just have to get a formatted file on sumotools.

Comment 15

[David Tenser [:djst]]

(In reply to comment #14)
> Now we just have to get a formatted file on sumotools.

I'll try to get to this today: https://bugzilla.mozilla.org/show_bug.cgi?id=484243

Comment 16

[[:Cww]]

Sigh... sorry for dropping the ball on this I had to declare bugmail bankruptcy yesterday since I'd gotten overloaded on bugmail (700 in 3 days!) so I missed the call for action.  I'm on it now.

Comment 17

[E. Cooper [:ecooper]]

Bug to test this on staging can be found here: https://bugzilla.mozilla.org/show_bug.cgi?id=484809

Comment 18

[E. Cooper [:ecooper]]

bug 484809 was a success. Marking as fixed.

As mentioned in comment 14, any further (and current) tiki scripts requiring access to TikiLib::query and other libraries, can include /scripts/tiki_scripts_base.php.

Comment 19

[David Tenser [:djst]]

LP: Could you verify that this was indeed upstreamed as part of the l10n dashboard feature?

Comment 20

[Louis-Philippe Huberdeau]

This is clearly sumo only. The libraries it uses in tiki are in place and everything should work out of the box.