1167761 - Implement asset re-use in ingestion format for splice

Status: RESOLVED FIXED

(Content Services Graveyard :: Tiles: Administration Front-End, defect)

Description

[Olivier Yiptong [:oyiptong]]

With State/DMA, there will be potentially many more tiles which will increase the time it takes to do ingestion.

Already, the ingestion process is cumbersome due to our need to do table locks in the database.

One way to make this less painful is to make the file upload faster, and this is by re-using assets instead of expecting duplicates in the ingestion format

Comment 1

[Olivier Yiptong [:oyiptong]]

The input for ingestion currently consists of a sequence of Tiles defined by a JSON object.

e.g.

> {
>   "STAR/en-US": [
>     {
>       "bgColor": "#0096DD", 
>       "directoryId": 402, 
>       "imageURI": "data:image/png;base64,...", 
>       "title": "Firefox Marketplace", 
>       "type": "affiliate", 
>       "url": "https://marketplace.firefox.com/"
>     }, 
>     {
>       "bgColor": "#62BE06", 
>       "directoryId": 403, 
>       "imageURI": "data:image/png;base64,...", 
>       "title": "Customize Firefox", 
>       "type": "affiliate", 
>       "url": "https://addons.mozilla.org/en-US/android/"
>     }, 
>     {
>       "bgColor": "#F37C00", 
>       "directoryId": 404, 
>       "imageURI": "data:image/png;base64,...", 
>       "title": "Firefox Help and Support", 
>       "type": "affiliate", 
>       "url": "https://support.mozilla.org/en-US/products/mobile"
>     }, 
>     {
>       "bgColor": "#CE4E41", 
>       "directoryId": 405, 
>       "imageURI": "data:image/png;base64,...", 
>       "title": "The Mozilla Project", 
>       "type": "affiliate", 
>       "url": "https://www.mozilla.org/en-US/"
>     }
>   ]
> }

The data URI in the imageURI and enhancedImageURI properties is often duplicated.
We want to only declare the base64 encoded image once and re-use across many geo/locale objects.

One way to do so would be to make the new ingestion format as follows:

> {
>   "assets": {
>     "someId": "... b64 encoded image here ...",
>     "someId2": "... b64 encoded image here ...",
>     "someId3": "... b64 encoded image here ..."
>   },
>   "distributions": {
>     "STAR/en-US": [
>       {
>         "bgColor": "#0096DD", 
>         "directoryId": 402, 
>         "imageURI": "#someId", 
>         "title": "Firefox Marketplace", 
>         "type": "affiliate", 
>         "url": "https://marketplace.firefox.com/"
>       }, 
>       {
>         "bgColor": "#62BE06", 
>         "directoryId": 403, 
>         "imageURI": "#someId", 
>         "title": "Customize Firefox", 
>         "type": "affiliate", 
>         "url": "https://addons.mozilla.org/en-US/android/"
>       }, 
>       {
>         "bgColor": "#F37C00", 
>         "directoryId": 404, 
>         "imageURI": "#someId2", 
>         "title": "Firefox Help and Support", 
>         "type": "affiliate", 
>         "url": "https://support.mozilla.org/en-US/products/mobile"
>       }, 
>       {
>         "bgColor": "#CE4E41", 
>         "directoryId": 405, 
>         "imageURI": "#someId3", 
>         "title": "The Mozilla Project", 
>         "type": "affiliate", 
>         "url": "https://www.mozilla.org/en-US/"
>       }
>     ]
>   }
> }

Comment 2

[Olivier Yiptong [:oyiptong]]

We would also need a converter, to convert the old-style distribution to the new style.

The original format needs to work and be automatically translated.

Comment 3

[Nan Jiang [:nanj]]

That's definitely the way to go. We could push it even further.
Apart from the URIs, we can abstracting all the fields, which have potential duplicates

Comment 4

[Nan Jiang [:nanj]]

Continue my last comment...

For instance, we can put "type" mapping into the assets as well,

> "type": { 1: "affiliate", 2: "organic", 3: "sponsored" }

Then we only store the type #, instead of the actual value in the payload.

Comment 5

[[github robot]]

Commit pushed to master at https://github.com/mozilla/splice

https://github.com/mozilla/splice/commit/53fbb4657f8f087cf6ec18d9ef1ef5a3fbddad2f
Fix Bug 1167761 - Implement asset re-use in ingestion format for splice

Squashed commit of the following:

commit 8eeb3a15110d80d226a087cbdd338cd5827ec228
Author: Nan Jiang <[e-mail]>
Date:   Mon Jun 29 14:21:16 2015 -0400

    Revert the change in CHANGELOG.md

commit 14130e280844ba4ba43e9e2396e463e6fd13f9ef
Author: Nan Jiang <[e-mail]>
Date:   Mon Jun 29 14:08:05 2015 -0400

    Bump up version to 1.1.24

commit 246f5d6caad2d44ddb9506023bcf05e043507aa3
Author: Nan Jiang <[e-mail]>
Date:   Thu Jun 25 16:03:04 2015 -0700

    Remove unnecessary object copies

commit 9f38464adfd7b5d93f289d2cf24ded913404ac8f
Author: Nan Jiang <[e-mail]>
Date:   Thu Jun 25 12:06:06 2015 -0700

    Add more error info for invalid payload

commit 7b8739031b55dea686c395f40f17683d2e1f06c7
Author: Nan Jiang <[e-mail]>
Date:   Fri Jun 19 15:27:55 2015 -0400

    Fixes after review

commit 11ca2a4deb6912c0f031bdad22fe0f7e452b4d2e
Author: Nan Jiang <[e-mail]>
Date:   Thu Jun 18 17:45:06 2015 -0400

    Change the payload to compact format on S3 and front-end

commit bc69ea5af4c526f595c4de9834c1750cd75d85c7
Author: Nan Jiang <[e-mail]>
Date:   Wed Jun 10 18:10:14 2015 -0400

    Increase test coverage for #73

commit 168b888905dc0355b1e28c5ee40736cef93a56af
Author: Nan Jiang <[e-mail]>
Date:   Wed Jun 10 16:10:09 2015 -0400

    Pep8

commit 316502d7955b358f6c858082aecc042e517a6771
Merge: 890166f 422957f
Author: Nan Jiang <[e-mail]>
Date:   Wed Jun 10 15:30:25 2015 -0400

    Merge branch 'master' into bug-1167761-reuse-assets-in-ingestion

commit 890166f290083dc9e06f900416722b1a1b52ed3c
Author: Nan Jiang <[e-mail]>
Date:   Tue Jun 2 10:46:50 2015 -0400

    Add assets re-use in ingestion format for backend

closes #73