931907 - Email users with crashes in bitguard

Status: RESOLVED FIXED

(Socorro :: Data request, task, P1)

Description

[Benjamin Smedberg]

We're seeing crashes with newer bitguard DLLs, and there is currently no way to add this to the DLL blocklist because it is an APPINIT DLL which gets loaded before Firefox starts.

Without action from bitguard, which has not been responsive this will be a big issue with our release population, and so I want to go ahead and deploy an automated email response to these users which recommends that they uninstall bitguard.

* I couldn't find an existing SUMO article specifically about bitguard, but I think we should have such an article which recommends that users uninstall this software. 
* We can easily deploy a custom classifier to note these crashes, or make this the first use of the support classifier.
* Firefox 25 does not have the user locale in the crash report, but Firefox 26 does (bug 765288). We should start out sending English emails and try to also send out localized versions when the locale and a translation are available.
* I believe that this should replace rather than supplement the normal support email that we send out from bug 814647 unless that is very complex.

Adrian, are you the right person to own this?

Comment 1

[Benjamin Smedberg]

This was prompted by bug 925459, by the way.

Comment 2

[Matt Grimes [:Matt_G]]

Looping in Verdi for documentation. I know Cheng is already working on something as well, so please connect.

Comment 3

[[:Cww]]

Firefox user,

We're sorry that your Firefox recently crashed. After analysis, the crash was determined to be caused by the BitGuard program. BitGuard is often installed unintentionally while installing or modifying other (often free) software downloaded from the internet. It can change Firefox search and homepage settings without your consent and is known to repeatedly crash many recent releases of Firefox.

We recommend that you remove BitGuard to help improve the stability of Firefox. You can do so by uninstalling it from the Add/Remove Programs area of the Control Panel. You may also need to run additional malware detection and cleanup services to prevent it from reinstalling itself. We recommend the free version of Malwarebytes which you can download from here: http://www.malwarebytes.org/products/malwarebytes_free/

For more information on BitGuard as well as detailed instructions for its removal, please see http://malwaretips.com/blogs/remove-bitguard-exe-virus. For more information on Firefox crashes as well as help on restoring your original Firefox home page or search engines, please use our support site at http://support.mozilla.org

Firefox Crash Team

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

I am indeed the one who should own this. 

Here is what needs to be done if we want to achieve this: 

 * create a new email in ExactTarget
 * add a way to detect BitGuard crashes into Socorro
 * store that data in our PostgreSQL database, preferably in the reports table
 * extend the current automatic_email cron job to be able to send different emails depending on the classification of the crash

I don't know anything about the second and third points of that list. Do we already have that kind of logic? Can we easily store that data in PostgreSQL?

Comment 5

[K Lars Lohn [:lars] [:klohn]]

adrian:

issue 2: the classifier system would be a fine way to tag these crashes - however, the  classifier doesn't exist yet in the processor (aside from the skunk classifier).  It is targeted for processor2014.  However, it could be back-ported to the current HybridProcessor or LegacyProcessor fairly quickly.  Two weeks out is an optimistic minimum.   Once the processor infrastructure is in place, the signatures in Bug 25459 could be used as data for the classifier.

issue 3: the classifier system was conceived to save classifications into the processed_crash json.  That means that it would be in HBase and Elasticsearch.  We're working on storing the processed crash into Postgres, but I cannot imagine that happening before the end of the year.

Comment 6

[K Lars Lohn [:lars] [:klohn]]

bad paste - that should have been bug 925459

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

So an option would be for me to change the automatic_email cron to get data from elasticsearch (probably as well as PostgreSQL), in order to be able to get the classifier data from the processed JSON. And then haste the use of the classifier to put that data in our database, so we can use it in our emailing job. Sounds like a good plan on the long term, but will take some time to be ready. 

An other option would be to make some sort of temporary classifying system, storing a list of signatures associated with a "type" and then using those signatures to choose what email to send. That means having a new table in postgres, but doing something that will be made obsolete by the classifier. Certainly faster, no need to switch to elasticsearch, but implies crappy coding. :)

Comment 8

[Verdi [:verdi] Best to slack me]

Here is the support article: https://support.mozilla.org/kb/uninstall-bitguard-fix-crashes-firefox-25

When including a link to the article, be sure to use this version (without a locale string or any extra parameters at the end).

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

Laura, what should we do here?

Comment 10

[Laura Thomson :laura]

I'm ok with either of the solutions in comment #7. Since I think this is wanted soon, let's go with the fast and crappy option.

Comment 11

[Benjamin Smedberg]

Lars, what do we need to do to set up the support classifier? It will be dirt-simple: search the dump for bitguard.dll and classify as "bitguard".

Comment 12

[[DEACTIVATED] Adrian Gaudebert]

I am actively working on this. I rewrote the automatic emails cron job to use elasticsearch instead of postgres. I am currently fighting a very weird elasticsearch bug that happens during unit tests. After that, I will use the classification available in the processed crash to chose which email get sent. 

Cheng, could you please create the new email in ExactTarget, and give it a name like 'socorro_bitguard_en'?

Comment 13

[[DEACTIVATED] Adrian Gaudebert]

Pull request: https://github.com/mozilla/socorro/pull/1727

Comment 14

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/6e3aa7a5d3f176d4d1cef333c41ac45da9eca3ea
Fixes bug 931907 - Rewrote automatic-emails cron job to use elasticsearch, added a classification system to send different emails. r=twobraids,ErikRose

https://github.com/mozilla/socorro/commit/0e1be1d5b962b4cedd0dde67b05209d886d697c7
Merge pull request #1727 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch

Fixes bug 931907 - Rewrote automatic-emails cron job to use elasticsearch and support classifications.

Comment 15

[Laura Thomson :laura]

I think this shouldn't be closed - we still need the classifier, and the email text. I also want to talk in this week's meeting about how we can be more agile on these types of requests as we'd like to do more of this sort of stuff.

Lars, can you comment on the classifier?

Cheng, can you comment on the email text?

Comment 16

[[DEACTIVATED] Adrian Gaudebert]

@Laura the classifier is already in there (and if I'm correct it is running in production), and the email exists, I forgot to mention it here but it was created in ExactTarget by Tyler Downer. 

Confirmed by lars in IRC:
> 19:52 <lars> adrian: the only support classifier running in prod right now is the BitGuardClassifier

Comment 17

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/ebb45fd4ac4c9ff16f0efbd6fc388ef4e601a087
Fixes bug 931907 - Removed limitation of 10 results from elasticsearch, added unit test. r=ErikRose

https://github.com/mozilla/socorro/commit/3d222b54bcf4aabd6eeda75b09b3c088107836e8
Merge pull request #1796 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch-fix

Fixes bug 931907 - Removed limitation of 10 results from elasticsearch, ...

Comment 18

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/f33baf491b2e4647a0d8929cf0b5307980449223
Fixes bug 931907 - Fixed migration script from PSQL to ES.

https://github.com/mozilla/socorro/commit/52f64095ace45953b83fd713f387108441e58f77
Merge pull request #1797 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch-fix

Fixes bug 931907 - Fixed migration script from PSQL to ES.

Comment 19

[[DEACTIVATED] Adrian Gaudebert]

Before the release, please do the following from the admin box:

1. Create a temporary config file (see attachment in next comment).

2. Run the following script from /data/socorro/:

./application/script/move_emails.py --admin.conf=/path/to/the/config/file.ini

Let it run. That's it!

Comment 20

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Simple config file for migration script

This might not work well, depending on when the configuration update is done. In any case, all that is needed is to load the configuration of PostgreSQL and elasticsearch, that's all.