1117195 - After updating the Plugincheck Database the Plugincheck Website should use the new data within Minutes NOT Days

Status: RESOLVED INCOMPLETE

(Plugin Check Graveyard :: Database, defect)

Description

[DJ-Leith]

After updating the Plugincheck Database the Plugincheck Website
should use the new data within Minutes NOT Days.

The delay between 'updating the Plugincheck Database' and the relevant
information being used by the 'Plugincheck Website' is a concern.


Bug 1084537 "Flash sometimes displayed as up to date whilst vulnerable, on Windows 7"
Illustrates an example of this, going back to 2014-10-17.

Neither Carsten Book [:Tomcat] (who has added many plugins to the Plugincheck Database
in the last two years) nor Schalk Neethling [:espressive] (who has worked on the
Plugincheck Website in 2013 and 2014) have been able to fix this.  Both have write
access to the Plugincheck Database and have commented in bug 1084537.

I think the underlying cause may be unwanted caching in the Mozilla infrastructure
and I have CCed some people to get 'other eyes' on this.
Feel free to redirect to whoever you think is most able to review this.
  I have a specific question for Justin Dolske in comment # 2.

Please read about the Background to the 'Plugincheck Service' in
bug 956905 comment # 148 onwards, before you think about this bug.

It is important to remember that, since May 2014, there are
TWO 'communication channels' between the Plugincheck Database and
the Plugincheck Website.

Release (Fx 34 and Fx 34.0.5) and older use the 'dynamic URLs',
Beta + (Fx 35, Fx 36 and Fx 37) use the 'JSON List'.



In this bug, consider the following chronology:

***
2014-12-09
***
  Adobe release Flash "15.0.0.246" (with fix for CVE-2014-9163)

  Adobe publish:

  Security updates available for Adobe Flash Player
  Release date: December 9, 2014
  Vulnerability identifier: APSB14-27
  https://helpx.adobe.com/security/products/flash-player/apsb14-27.html
    where Windows users are strongly recommended Flash "16.0.0.235".

  Security Updates available for Adobe Reader and Acrobat
  Release date: December 9, 2014
  Vulnerability identifier: APSB14-28
  http://helpx.adobe.com/security/products/reader/apsb14-28.html

***
2014-12-10
***

2014-12-10 at 01:13:58 PST 
  Schalk Neethling [:espressive],
  in bug 1109488 comment # 1 "Update Adobe Flash Versions"
  has updated the Plugincheck Database.
      While there is not much detail in bug 1109488 the update
      of the Database does include Flash "16.0.0.235" for Windows.
        Evidence for this is in bug 1084537 comment # 20.
      This means that visitors to the Plugincheck Website,
      using Firefox on Windows, with Flash less than "16.0.0.235"
      should have their Flash plugin reported as "vulnerable".

      Knowing this, I tested with "15.0.0.246" and I was told
      "Up to Date", IN ERROR,
      for 2 days (using the 'JSON List' - Aurora)
      and 6 days (using the 'dynamic URL method' - Release).

2014-12-10 at 11:05 PST
  Bug 1109795 "Blocklist Flash versions vulnerable to CVE-2014-9163
  (15.0.0.242 and below, 11.2.202.424 on linux)" 
  Reported by Daniel Veditz [:dveditz] 

2014-12-10 13:24 PST
As I am still getting Adobe Reader "11.0.9.29" reported as "Up to Date"
and I can not find a bug to update the Plugincheck Database for this I filed
Bug 1109858 "Adobe Reader (and Acrobat) for Windows and Macintosh - plugins 
vulnerable 2014-12-09 - APSB14-28" 
  The Plugincheck Database was updated by Schalk Neethling [:espressive] on
  2014-12-12 at 00:39:33 PST 
      Unfortunately, the data in the Database is wrong for Reader 11.0.9
      (see bug 1117189).

***
2014-12-11
***

2014-12-11 05:54:08 PST (bug 1109795 comment # 6)
> The block for Windows and Mac is now live: https://addons.mozilla.org/blocked/p794
> 
> The block for Linux using version numbers is now 
> staged: https://addons-dev.allizom.org/blocked/p588. Please test.

So the Plugincheck Database was updated before the Blocklist went live - VERY GOOD.
However, when users used the Plugincheck Website they were told that their
Flash plugin was "Up to Date" in ERROR.

This caused confusion, threads at SUMO and some evidence in bugzilla.



2014-12-11 at 09:14:55 PST
Konstantin Kolinko updates bug 1084537 comment # 12:

> I am seeing this now. I have Windows 7, Firefox 34.0.5 (locale: Russian) 
> and Shockware Flash 15.0.0.239, but 'plugin check' page says that it is up-to-date
> https://www.mozilla.org/ru/plugincheck/

He adds the very helpful 'dynamic URL' that was used to
'collect the data from the Plugincheck Database' and use it in the
'assessment of the Flash plugin' when he visited the 'Plugincheck Website'.

https://plugins.mozilla.org/pfs/v2?appID=%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D&appRelease=34&appVersion=20141126041045&clientOS=Windows&chromeLocale=ru-RU&detection=version_available&mimetype=application%2Fx-shockwave-flash+application%2Ffuturesplash&callback=C 

The contents are Attached to bug 1084537 comment # 14
https://bug1084537.bugzilla.mozilla.org/attachment.cgi?id=8535066

We see that the data was "fetched"
BEFORE the Database was Updated (on 2014-12-10 at 01:13:58 PST)
>        'fetched': '2014-11-26T15:34:14-08:00'
and so the data is stale, and it results in the WRONG report.

      In fact, the "fetched" was about 5 min after
      the PREVIOUS Plugincheck Database update for Flash (bug 1105307 comment # 1)
          'modified': '2014-11-26T23:27:18+00:00',
          'created': '2014-11-26T23:27:18+00:00',
          'fetched': '2014-11-26T15:34:14-08:00'
      Note Time Zone, "fetched" is -8 hours (it is PST)
      Updated by Carsten Book [:Tomcat] 2014-11-26 at 07:36:12 PST (time in bugzilla).

Plugincheck should have reported Flash "15.0.0.239", in Konstantin's case,
on 2014-12-11, as "vulnerable" IF the communication from the Plugincheck
Database to the Plugincheck Website was working well.


***
2014-12-12
***
https://bug1109981.bugzilla.mozilla.org/attachment.cgi?id=8535433
"plugin-check-german-2014-12-12-7.43-GMT.PNG"
Screenshot, by Harald Fassler (from bug 1109981 comment # 2)
Shockwave Flash 15.0.0.239 still reported as "Up to Date" / "Aktuell".

By 2014-12-12 I see the correct result for Flash, "vulnerable"
when using the 'JSON List'.

However, I continue to see the WRONG result for Flash ("Up to Date")
when using Release and the 'dynamic URLs'.
  See Bug 1084537 comment # 20 for detail.


***
2014-12-13
***

Konstantin Kolinko on 2014-12-13 at 11:29:12 PST, in bug 1084537 comment # 21, said:

> In bug 1109981 comment #1 it was said that
> 
> > The 'Plugincheck Database' was updated,
> > by Schalk Neethling [:espressive] on 2014-12-10 at 01:13:58 PST.
> > (see bug 1109488).
> 
> FYI, the plugin check page still lists my Shockwave Flash 15.0.0.239 as "Up-to-date".

***
2015-12-14
***

Still have the WRONG result at Plugincheck and we see bugs like this:
Bug 1111356 "Firefox offers conflicting information regarding plugin vulnerability" 

(from bug 1111356 comment # 5 on 2014-12-17)
> Don't see the relevance of that link. Version 11.2.202.424 is vulnerable. 
> I already knew that. As the title of the bug says "Firefox offers conflicting 
> information regarding plugin vulnerability". Providing a link to Adobe's website 
> doesn't resolve the fact that Firefox offers conflicting information to end users,
> and in this case is actually *encouraging* end users to use a version of Flash 
> that has a critical vulnerability by informing users that version 11.2.202.424
> is actually up to date.


***
2014-12-16
***
(see bug 1110578 comment # 14)
> This bug is still a problem, so is clearly not fixed. 
> As of 2014-12-16T20:06:21Z Flash version 11.2.202.424 is shown as vulnerable
> on web pages, but is apparently "up to date" on the plugin check webpage.


I was STILL, 6 days after the Database was updated, getting the WRONG result on
Release, using the 'dynamic URLs'. See bug 1084537 comment # 28.

Now in this example, the 'Plugincheck Database' was updated,
by Schalk Neethling [:espressive] on 2014-12-10 at 01:13:58 PST.
(see bug 1109488).

I FIRST saw the correct report - "vulnerable" on 2014-12-12 (two days after the
Database was updated) using the 'JSON List' - Aurora.

I was still seeing the WRONG report - "Up to Date" on 2014-12-16 (six days after
the Database was updated) using the 'dynamic URL' method - Release.


DJ-Leith

continued

Comment 1

[DJ-Leith]

A few general points,
in ADDITION to those made in bug 956905 comment # 148 onwards.

First, see bug 991664 "Test Automation for PluginCheck"
It is short, and there are many good points including:

> The plugin check site is ranked as the #2 site (#1 is mozilla.org) that
> the public lands on. I would suggest we identify scenarios that should
> never fail - start with happy path smoke tests - and automate those first.


***
Testing
***
While 'rapid updating of the Plugincheck Database' is undoubtedly both the
most important and hardest part of running the 'Plugincheck Service'
having 'old plugins' to do testing is also challenging.

Not only do you have to install the plugins you want to test
you also have to 'take the risk of using vulnerable plugins'
when you 'delay updating' (one has to switch off the 'automatic updates'
of these plugins) in order to prove that the Plugincheck Website
is now able to 'correctly detect and report' the "vulnerable" plugins.

It is hard to find 'old plugins' for testing (if you want to do a 'fresh
install of a plugin').

You also, as there are TWO 'communication channels', the 'dynamic URLs' and
the 'JSON List' - which provide DIFFERENT data, have to test 'both methods'
if you wish to verify that 'all is OK'.
See bug 956905 comment # 148 section on "4. Test and QA." for more.


***
The 'dynamic URLs'.
***
Remember these are used (and have always been used) by
Firefox Release and older.
The 'JSON List' has only EVER been used by Firefox Beta +.

D1.
How are the 'dynamic URLs' created?
I don't KNOW.  I have searched.
I've asked, in several bugs, and nobody has provided an answer for
me to cite here.

The answer may well be deduced if you read the actual code that creates the
'dynamic URLs'.

D2.
When are the 'dynamic URLs' used?
When the browser visit the 'Plugincheck Website' the plugins are enumerated.
For EACH 'plugin found by enumeration' a 'dynamic URL' is used to
'collect some data that contains information about *that* plugin'.
If you have seven plugins then seven 'dynamic URLs' are used
and seven 'tests are done as part of the assessment of each plugin'.

D3.
What data is in the 'dynamic URLs'?
If you browse to a 'dynamic URL' you will see that the response
is similar to a 'report from a database'.
The data contains information about the "latest" version of the
'plugin the dynamic URL is checking' and "other" versions of the plugin.

If YOUR plugin is the same version as the "latest" (or newer) it will
be reported as "Up to Date".  If it matches any of the "other" plugins
it will (I think) be reported as "vulnerable".
I am not sure how / if it could be reported as
'out of date but not "vulnerable" AKA "outdated"'.
The original Perfidies code (bug 956905 comment # 68) includes:

> Old 4.1 Each can be classified / categorised as
> 
> DISABLE:          "should_disable", (perfidies.js - line 165 ff for comments)
> VULNERABLE:       "vulnerable",
> MAYBE_VULNERABLE: "maybe_vulnerable",
> OUTDATED:         "outdated",
> MAYBE_OUTDATED:   "maybe_outdated",
> CURRENT:          "latest",
> UNKNOWN:          "unknown",
> NEWER:            "newer". 

So you might think that
'the dynamic URL, for a particular plugin (e.g. Java), might contain
ALL the known data about that plugin'.

It is slightly more subtle than this, it is not ALL the 'known data'.

As seen in bug 985968 comment # 38 (to introduce the topic).

Java 7 U51 was being reported as "vulnerable" after Java 8 was
added to the Database.  The 'dynamic URL' was selecting ONLY ONE
"latest" 'Java Plugin' (in this case the Java 8) and so the
'Java 7 U51 data' was NOT in the 'data sent to the Website
by the dynamic URL method'.

BOTH Java 7 U51 and Java 8 were 'safe to use',
both were "latest" and both were in the Plugincheck Database.

bug 985968 comment # 40 has the URL (next)
https://bug985968.bugzilla.mozilla.org/attachment.cgi?id=8407195

and then
bug 985968 comment # 41 to discuss the actual data.

There is lots of detail, I concluded:

> Some conclusions:
> 
> I think the 'Java 7 U51' data is still in the database
> BUT
> because 'all the Java from Sun / Oracle' data are selected
> as 'one bucket of data', that match the aliases or literal criteria,
> (lines 10 to 27) and only ONE (which has the 'highest version number')
> is then deemed "latest" and 'sent to the plugincheck website'.
> We have 'Java 7 U51' detected - a 'false positive' - *this bug*.
> 
> It is possible to have 'more than one type of plugin to do deal
> with Java'.  You could have the IcedTea plugin.


D4.
An unanswered doubt, asked in bug 956905 comment # 149, is
does the 'creation of the dynamic URLs',
or the use of these 'dynamic URLs' depend on
ANY infrastructure or code that is 'being retired along with the PFS'?

I will ask again, in comment # 2.


***
The 'JSON List'
***

J1.
What is in the 'JSON List'?

In bug 956905 comment # 75 Schalk Neethling listed 37 plugins
that were going to be included in the 'JSON List'.

The 'JSON List' went live in May 2014. 

From bug 1020133 comment # 33
https://bug1020133.bugzilla.mozilla.org/attachment.cgi?id=8487601
In this example, from 2014-09-10, there are 5,428 lines in the 'JSON List'
(including 8 lines of Scratchpad comment).

By 2014-11-14 (bug 1084537 comment # 7) there were
5,750 lines of data (including 8 lines of Scratchpad comment).

You can see the 'raw data' by browsing directly to:
https://plugins.mozilla.org/en-us/plugins_list.json

Bug 1101613 has reduced the number of plugins in the Database
and so the 'JSON List' is now
3,616 lines of data (including 8 lines of Scratchpad comment).

J2.
When is the 'JSON List' generated?

I don't know.

Bug 1105483 "Add a 'Generated' Date and Time stamp to the top of the
'Plugincheck JSON List'"
was opened to add this data.

If a browser visits the Plugincheck Website, to do a Plugincheck,
should 'the browser that visits' generate a 'fresh JSON List
by making a query to the Plugincheck Database'?
I don't think so.

There is plenty of scope for the 'JSON List' to be generated
regularly and then be made available to the Plugincheck Website.
  So the 'end point' https://plugins.mozilla.org/en-us/plugins_list.json
  could be used by the 'Plugincheck Website'.
  This 'end point' could 'get the data from the Database'.
  The Plugincheck Website need not have 'a direct connection to the Database'
I'll discuss this below (in the 'push' section).

J3.
If you use the 'JSON List' are the results accurate?

Unfortunately, they are NOT as good as the existing method using the
'dynamic URLs'.

A good example of this is bug 1020133 "Improve Adobe Acrobat plugin reporting".

Part of the problem has been the lack of 'testing resource' for the 'JSON List'
  method.  Many of the bugs in the 'existing Plugincheck' have been ironed out
  over the years.
  People on Nightly, Aurora and Beta are LESS likely to have 'old plugins'
  that can 'trigger reports' of "vulnerable".  So, there few bug reports
  from those who are NOT 'actively looking' for accurate plugin tests
  using the 'JSON List'.

Another part of the problem is that the 'order of assessment' is different.
  In the 'JSON List' method the 'list drives the assessment'.
  The 'detection code', used at the Plugincheck Website to assess
  'is the plugin under test (e.g. Java) "vulnerable" (or not)?',
  was developed to use the 'fuller and MORE SPECIFIC data' that is
  delivered by the 'dynamic URLs'.

Another example is bug 1038685 "Some plugins don't appear in plugincheck
  website even though they are installed and are seen in plugins_list.json"


***
How frequently does the data in the Plugincheck Database change?
***

How many new versions of plugins have been added to the Plugincheck
Database since 2014-09-30?
  Useful to have an idea of how frequent changes to the database are.

Assumption:
  If all the updates to the Plugincheck Database are documented in public
  bugzilla bugs
  (I know of one case where there was a Flash update that happened
  'outwith public documentation', it was also outwith these last 3 months).
  In bug 1101613 I documented 4 changes. Flash on 2014-11-26 (bug 1105307).
  There was also bug 1082813 "Java 7u71 and 8u25 released"
  In addition, there are the two in comment # 0 of this bug.
  Also, I know of 4 changes that have been 'requested' (i.e. have bugs)
  but not yet done. So only about 12 (if we include those).
  If we assume that this is a 'typical quarter of a year' this gives us
  about 40-50 per year.


***
Push on updating the Plugincheck Database
***

Is there a 'push' when the Database is updated?

Given that
there are only about 40-50 times per year and
it is important that changes made to the database SHOULD invalidate any 
'old cached data that is now NOT the latest information about the plugins'
a 'push' would seem to be a good idea.

I envisage:

A. Whenever the Plugincheck Database is up dated,
  'push' to invalidate the 'JSON List'
  and regenerate a 'fresh JSON List'.

B. Every 6 min / 20 min / hour / 4 hours / 8 hours / day,
  whether there has been a 'push' or not,
  regenerate a 'fresh JSON List'.

  So if, for whatever reason, the 'push failed' within a few minutes
  a 'fresh JSON List' would be available.
    How often the 'regenerate the JSON List by schedule' would be
    done would be an operational decision.
    
    I would hope that the 'regular regeneration' or 'refresh' of the
    'JSON List' should be closer to every hour as opposed to a
    long time (many hours).  We want the 'Plugincheck Website' to
    use the 'best data from the Plugincheck Database'.

    If the 'JSON List' is cached it should expire so that
    the assessment at the 'Plugincheck Website' uses
    a recent 'JSON List'.

As I don't know how the 'dynamic URLs' are made or if they
'expire' (I have seen data that was "fetched" 3 weeks before)
I can't comment on this.

It is a concern that the BOTH the 'dynamic URL' method and the
'JSON List' method failed to get the 'new data about Flash'
into use by the 'Plugincheck Website'. We had Flash being
reported as "Up to Date" for days after the Database was updated.


Other information on 'push'.

In bug 1084537 comment # 16 on 2014-12-11 at 16:07:51 PST I asked:

> FAO Carsten Book,
> please can you review all of this bug.
> 
> On two recent occasions we have had a
> 'Flash version added to the Plugincheck Database';
> and then we have seen some 'odd results'.
> 
> We have had the SAME plugin reported as
> "Up to Date" and "vulnerable" on the same day! 
> 
Lots more in that comment.

Carsten replied in bug 1084537 comment # 17
> Hi,
> 
> moving away from working on plugincheck (i guess espressive will be
> the number one contact in the future). 
> 
> For your points:
> 
> "When somebody adds an Adobe Flash plugin to the 'Plugincheck Database'
> do they have to 'do something' to create the 'dynamic URLs'?"
> 
> no its just pasting data in some fields on a website like version number 
> etc - so all backround stuff like dynamic url etc is done automatically,
> like when we update a plugin the one who update the data does no
> direct sql or something insert manually its all done via a gui -> database


Daniel Veditz [:dveditz]
on 2014-12-12 at 15:42:30 PST in bug 1110578 comment # 3 asked: 
> Schalk: is there a manual "push" step to get the database info live
> that didn't happen? (but has obviously happened now.)

Schalk Neethling [:espressive]
on 2014-12-15 at 01:40:41 PST in bug 1110578 comment # 11 said:
> Updating the information for the plugincheck page involves updating the
> database at plugins.m.o [this is independent of any changes to plugin
> statuses anywhere else, such as blocklists etc.] but, after the update
> happened there, there is also a cache period that needs to expire on
> mozilla.org but, the length of time mentioned here is much longer than
> the cache TTL.

Now in this example, from comment # 0, the 'Plugincheck Database' was updated,
by Schalk Neethling [:espressive] on 2014-12-10 at 01:13:58 PST.
(see bug 1109488).

I FIRST saw the correct report - "vulnerable" on 2014-12-12 (two days after the
Database was updated) using the 'JSON List' - Aurora.

I was still seeing the WRONG report - "Up to Date" on 2014-12-16 (six days after
the Database was updated) using the 'dynamic URL' method - Release.

DJ-Leith

Comment 2

[DJ-Leith]

FAO Justin Dolske [:Dolske]

I believe that the 'plugincheck using enumeration' was developed using
'ideas, infrastructure and code' from the "Plugin Finder Service" (and later PFS2).
As noted in bug 956905 comment # 149, you closed more than 60 PFS bugs.

My question is, with respect to the whole 'Plugincheck Service':
"Are there ANY dependencies on PFS or PFS2?"

In particular,
does the 'creation of the dynamic URLs',
or the use of these 'dynamic URLs' depend on
ANY infrastructure or code that is 'being retired along with the PFS'?

DJ-Leith

Comment 3

[Justin Dolske [:Dolske]]

(In reply to DJ-Leith from comment #0)

>   I have a specific question for Justin Dolske in comment # 2.

I happened to catch my name mentioned here, but I don't know what you're asking.

Please make an effort to be CONCISE and FOCUSED in your bug reports. You comments have been rather long and rambling around multiple topics, so they're very difficult to read and understand.

PFS and the Plugin Check service have absolutely nothing to do with each other.

Comment 4

[u597032]

Is this bug still relevant or can it be closed by now?

Comment 5

[Schalk Neethling [:espressive]]

(In reply to opi123 from comment #4)

Is this bug still relevant or can it be closed by now?

Same here. I doubt that this service still exists so I would vote to close it.