Closed Bug 931907 Opened 11 years ago Closed 11 years ago

Email users with crashes in bitguard


(Socorro :: Data request, task, P1)


(Not tracked)



(Reporter: benjamin, Assigned: adrian)




(1 file)

We're seeing crashes with newer bitguard DLLs, and there is currently no way to add this to the DLL blocklist because it is an APPINIT DLL which gets loaded before Firefox starts.

Without action from bitguard, which has not been responsive this will be a big issue with our release population, and so I want to go ahead and deploy an automated email response to these users which recommends that they uninstall bitguard.

* I couldn't find an existing SUMO article specifically about bitguard, but I think we should have such an article which recommends that users uninstall this software. 
* We can easily deploy a custom classifier to note these crashes, or make this the first use of the support classifier.
* Firefox 25 does not have the user locale in the crash report, but Firefox 26 does (bug 765288). We should start out sending English emails and try to also send out localized versions when the locale and a translation are available.
* I believe that this should replace rather than supplement the normal support email that we send out from bug 814647 unless that is very complex.

Adrian, are you the right person to own this?
Blocks: 925459
This was prompted by bug 925459, by the way.
Looping in Verdi for documentation. I know Cheng is already working on something as well, so please connect.
Firefox user,

We're sorry that your Firefox recently crashed. After analysis, the crash was determined to be caused by the BitGuard program. BitGuard is often installed unintentionally while installing or modifying other (often free) software downloaded from the internet. It can change Firefox search and homepage settings without your consent and is known to repeatedly crash many recent releases of Firefox.

We recommend that you remove BitGuard to help improve the stability of Firefox. You can do so by uninstalling it from the Add/Remove Programs area of the Control Panel. You may also need to run additional malware detection and cleanup services to prevent it from reinstalling itself. We recommend the free version of Malwarebytes which you can download from here:

For more information on BitGuard as well as detailed instructions for its removal, please see For more information on Firefox crashes as well as help on restoring your original Firefox home page or search engines, please use our support site at

Firefox Crash Team
OS: Windows 7 → All
Hardware: x86_64 → All
I am indeed the one who should own this. 

Here is what needs to be done if we want to achieve this: 

 * create a new email in ExactTarget
 * add a way to detect BitGuard crashes into Socorro
 * store that data in our PostgreSQL database, preferably in the reports table
 * extend the current automatic_email cron job to be able to send different emails depending on the classification of the crash

I don't know anything about the second and third points of that list. Do we already have that kind of logic? Can we easily store that data in PostgreSQL?

issue 2: the classifier system would be a fine way to tag these crashes - however, the  classifier doesn't exist yet in the processor (aside from the skunk classifier).  It is targeted for processor2014.  However, it could be back-ported to the current HybridProcessor or LegacyProcessor fairly quickly.  Two weeks out is an optimistic minimum.   Once the processor infrastructure is in place, the signatures in Bug 25459 could be used as data for the classifier.

issue 3: the classifier system was conceived to save classifications into the processed_crash json.  That means that it would be in HBase and Elasticsearch.  We're working on storing the processed crash into Postgres, but I cannot imagine that happening before the end of the year.
bad paste - that should have been bug 925459
So an option would be for me to change the automatic_email cron to get data from elasticsearch (probably as well as PostgreSQL), in order to be able to get the classifier data from the processed JSON. And then haste the use of the classifier to put that data in our database, so we can use it in our emailing job. Sounds like a good plan on the long term, but will take some time to be ready. 

An other option would be to make some sort of temporary classifying system, storing a list of signatures associated with a "type" and then using those signatures to choose what email to send. That means having a new table in postgres, but doing something that will be made obsolete by the classifier. Certainly faster, no need to switch to elasticsearch, but implies crappy coding. :)
Here is the support article:

When including a link to the article, be sure to use this version (without a locale string or any extra parameters at the end).
Laura, what should we do here?
Flags: needinfo?(laura)
I'm ok with either of the solutions in comment #7. Since I think this is wanted soon, let's go with the fast and crappy option.
Flags: needinfo?(laura)
Lars, what do we need to do to set up the support classifier? It will be dirt-simple: search the dump for bitguard.dll and classify as "bitguard".
Assignee: nobody → adrian
I am actively working on this. I rewrote the automatic emails cron job to use elasticsearch instead of postgres. I am currently fighting a very weird elasticsearch bug that happens during unit tests. After that, I will use the classification available in the processed crash to chose which email get sent. 

Cheng, could you please create the new email in ExactTarget, and give it a name like 'socorro_bitguard_en'?
Priority: -- → P1
Commits pushed to master at
Fixes bug 931907 - Rewrote automatic-emails cron job to use elasticsearch, added a classification system to send different emails. r=twobraids,ErikRose
Merge pull request #1727 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch

Fixes bug 931907 - Rewrote automatic-emails cron job to use elasticsearch and support classifications.
Closed: 11 years ago
Resolution: --- → FIXED
Blocks: 958595
I think this shouldn't be closed - we still need the classifier, and the email text. I also want to talk in this week's meeting about how we can be more agile on these types of requests as we'd like to do more of this sort of stuff.

Lars, can you comment on the classifier?

Cheng, can you comment on the email text?
Flags: needinfo?(lars)
Resolution: FIXED → ---
Flags: needinfo?(cwwmozilla)
@Laura the classifier is already in there (and if I'm correct it is running in production), and the email exists, I forgot to mention it here but it was created in ExactTarget by Tyler Downer. 

Confirmed by lars in IRC:
> 19:52 <lars> adrian: the only support classifier running in prod right now is the BitGuardClassifier
Closed: 11 years ago11 years ago
Flags: needinfo?(lars)
Flags: needinfo?(cwwmozilla)
Resolution: --- → FIXED
Target Milestone: --- → 72
Commits pushed to master at
Fixes bug 931907 - Removed limitation of 10 results from elasticsearch, added unit test. r=ErikRose
Merge pull request #1796 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch-fix

Fixes bug 931907 - Removed limitation of 10 results from elasticsearch, ...
Commits pushed to master at
Fixes bug 931907 - Fixed migration script from PSQL to ES.
Merge pull request #1797 from AdrianGaudebert/931907-automatic-emails-with-elasticsearch-fix

Fixes bug 931907 - Fixed migration script from PSQL to ES.
Before the release, please do the following from the admin box:

1. Create a temporary config file (see attachment in next comment).

2. Run the following script from /data/socorro/:

./application/script/ --admin.conf=/path/to/the/config/file.ini

Let it run. That's it!
This might not work well, depending on when the configuration update is done. In any case, all that is needed is to load the configuration of PostgreSQL and elasticsearch, that's all.
You need to log in before you can comment on or make changes to this bug.