Open Bug 1197866 Opened 4 years ago Updated 10 months ago

Use a more efficient scheme for tracking visited URLs in GlobalHistory

Categories

(Firefox for Android :: General, defect)

All
Android
defect
Not set

Tracking

()

ASSIGNED

People

(Reporter: rnewman, Assigned: andrew, Mentored, NeedInfo)

References

Details

(Whiteboard: [priority:low])

Attachments

(3 files, 22 obsolete files)

9.63 KB, patch
nalexander
: review+
Details | Diff | Splinter Review
13.63 KB, patch
nalexander
: review+
Details | Diff | Splinter Review
4.08 KB, patch
nalexander
: review+
Details | Diff | Splinter Review
https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/GlobalHistory.java?offset=1900#46

We hold a Set of every visited URI. When this becomes too much, we throw it away -- worst-case, we do that on every page load.

We can do much better. We never need to retrieve URLs from this set, so why store the strings?!

Two options:

1. Don't store the strings, store the hashes.
2. Don't store anything in a Set; use a Bloom filter, like [1]. This is way better.


If we use a Bloom filter, *don't throw this fixed-size data structure away on memory pressure*. This will dramatically reduce the amount of work done around page load time on low-memory devices.

We can also consider persisting the Bloom filter in the CP, which takes us to purely incremental work. Yay!


[1] https://github.com/MagnusS/Java-BloomFilter/blob/master/src/com/skjegstad/utils/BloomFilter.java
Is this only open to the Mozilla Project veterans? If not, I'd like to take a stab at implementing this solution. I have a few questions, however.

1. The location of the GlobalHistory class seems to have changed since rnewman's comment was posted. I can now see the class at https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java. Did some refactoring take place?
2. I looked at the code in the GlobalHistory class but I couldn't figure out where we throw away the set of every visited URI. Seems like it pulls it from a persistent store and saves it in a cache structure/set.
3. I read through the description of a Bloom filter and also the link at [1] in rnewman's comment. One downside of using a Bloom filter is it's susceptibility to "false positives". But, if I understood this correctly, it doesn't apply to this particular situation because we know for a fact that all the URIs in the set have been visited (else they wouldn't be in the set in the first place). Is this understanding correct?
Flags: needinfo?(rnewman)
(In reply to swaroop.rao from comment #1)
> Is this only open to the Mozilla Project veterans? If not, I'd like to take
> a stab at implementing this solution.

Nope, it's open to anyone.


> 1. The location of the GlobalHistory class seems to have changed since
> rnewman's comment was posted. I can now see the class at
> https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/
> mozilla/gecko/GlobalHistory.java. Did some refactoring take place?

Yes, we moved files to match common Android project conventions.


> 2. I looked at the code in the GlobalHistory class but I couldn't figure out
> where we throw away the set of every visited URI. Seems like it pulls it
> from a persistent store and saves it in a cache structure/set.

http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html

We use a SoftReference in GlobalHistory to store a reference to a Set. At any time, memory pressure can cause a GC, and the garbage collector can decide that the reference can be nulled and the Set discarded.


> 3. I read through the description of a Bloom filter and also the link at [1]
> in rnewman's comment. One downside of using a Bloom filter is it's
> susceptibility to "false positives". But, if I understood this correctly, it
> doesn't apply to this particular situation because we know for a fact that
> all the URIs in the set have been visited (else they wouldn't be in the set
> in the first place). Is this understanding correct?

We would be susceptible to false positives; the point here is that we'd be getting rid of the set altogether and relying on the filter.

The false positive rate can be specified as part of the filter's creation parameters. We're guessing that we can reach an acceptable false positive rate -- 0.01%? 0.005%? -- that gives us good space efficiency and in every other respect behaves better than a SoftReference<HashSet<String>>.

The false positive rate here will simply cause a tiny, tiny proportion of unvisited URIs to be shown as visited when rendering a page. I'm betting that this is a worthwhile price to pay to get faster lookups, faster inserts, and constant space.

You can measure the size and lookup/insert times for the HashSet approach for common history sizes ranging from a few hundred to, perhaps, 10,000 URLs. Then measure the size and lookup/insert times for a few Bloom filters with various false positive rates. Then we can make an informed decision.
Flags: needinfo?(rnewman)
Hi, is anyone working on this bug right now? If not, I'd gladly work on it.
I measured GeckoGlobalHistory storing 14,845 URIs with 1,587,499 characters,
roughly 2.5 MB using Android 8.0 with compact Strings and almost twice as much
with earlier versions.  Switching from a HashMap<String> to HashMap<Long>
reduces this to 850 KB.  We can further compact this with an implementation like
GNU Trove TLongHashSet.
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as a HashSet<Long> instead of
HashSet<String> reduces this to less than 1 MB by removing the characters and
halving the object overhead by using Long instead of String and its array.
Using only 64 bits of the SHA-256 hash should not have collisions given the
limited size of the map.
Attachment #8943731 - Flags: review?(rnewman)
Assignee: nobody → andrew
Status: NEW → ASSIGNED
Comment on attachment 8943731 [details] [diff] [review]
Represent visited URLs with SHA-256 hash

Review of attachment 8943731 [details] [diff] [review]:
-----------------------------------------------------------------

My initial questions about this:

- Why SHA-256? I don't believe we need a cryptographic hash here, but am I missing something?
- If this map gets big enough to be a concern space-wise, then presumably it contains a lot of strings. What's the CPU impact of taking the SHA-256 hash of 10,000 long strings in a loop? Consider that in some edge cases this map is rebuilt frequently -- it's discarded in order to free memory, then rebuilt when next needed.
- What is the collision likelihood of 64 bits of SHA-256 on 10,000 URLs? 100,000?
- What is the false positive rate of a Bloom filter of equivalent memory size? If it's acceptable, then I believe the Bloom filter will offer better performance.
- I chose SHA-256 because both Fennec and hardware acceleration support it.  I can change this to Utils.sha1 although it performs only slightly faster.  Java String.hashCode lacks enough bits and has obvious collisions: https://stackoverflow.com/questions/9406775/why-does-string-hashcode-in-java-have-many-conflicts
- I lack ARM benchmarks but Intel hashes 1 million 89 byte strings in 1963 ms: https://automationrhapsody.com/md5-sha-1-sha-256-sha-512-speed-performance/ .  We should expect that this set is rebuilt less often given the smaller memory pressure.
- The strong hash function is effectively random so if we have 2**32 URLs we should expect 1 collision via birthday paradox.  Repeating this with a 32-bit int we should expect 1 collision after only 2**16 URLs, a plausible number for some users.
- We could also use a Bloom filter but given that the size of a Trove hash table will be about 200 KB for my use case it does not seem like a worthwhile optimization.  It is also harder to reason about BF sizing, how collisions affect the table, and what kind of false positive rate we should choose.  The current patch combined with GNU Trove will give most of the benefit without worrying about these issues.
Flags: needinfo?(rnewman)
(In reply to Andrew Gaul from comment #7)
> - I chose SHA-256 because both Fennec and hardware acceleration support it. 

I'm a little concerned about this causing an immediate init of the Java crypto system, which we know from experience (Bug 959652) to be expensive on devices.


> - I lack ARM benchmarks but Intel hashes 1 million 89 byte strings in 1963
> ms:

I'd be very interested in seeing real-world timings (say, 1K, 10K, and 100K URLs) for the current approach versus your implementation. There is a space-time tradeoff here, and choice of hash function is one of the knobs we can tweak.

This kind of change must be tested on a device.


> - We could also use a Bloom filter but given that the size of a Trove hash
> table

… how big is Trove as a dependency? We care a lot about APK size, and one does not take on an external dependency lightly.


> worthwhile optimization.  It is also harder to reason about BF sizing, how
> collisions affect the table, and what kind of false positive rate we should
> choose.  The current patch combined with GNU Trove will give most of the
> benefit without worrying about these issues.

In general I would rather not take on an external general-purpose library dependency like Trove in order to address this case.

I would rather leap from built-in (HashSet) to specialized (custom fast non-colliding hash function, Bloom or Cuckoo filter). In both cases all of the relevant code would live in-tree.
Flags: needinfo?(rnewman)
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as a TLongHashSet instead of
HashSet<String> reduces this to a few hundred kilobytes by removing the
characters and halving the object overhead by using long instead of String and
its char array.  Using only 64 bits of the SHA-1 hash should not have collisions
given the limited size of the map.
Attachment #8948620 - Flags: review?(rnewman)
I benchmarked my patch on a Nexus 7 2013, before and after:

02-05 21:25:31.977 18263 18297 W GeckoGlobalHistory: Rebuilt visisted link set containing 15049 URLs in 651 ms
02-05 22:40:01.733 22705 22741 W GeckoGlobalHistory: Rebuilt visisted link set containing 15049 URLs in 1273 ms

Hashing takes an additional 300 ms while String.getBytes takes 300 ms.

I added another patch to add a slimmed down Trove dependency, weighing in at 3,000 lines of code.  I made no modifications; I merely copied the minimum set of source files to get TLongHashSet to compile.  Note that this reduces the previously cited size from 2.5 MB to 200 KB.

I switched to using SHA-1 based on your recommendation which was much faster, previously 2484 ms!  There exist lower quality hash functions although this will require analysis that my limited URL history is not suited to.

I disagree about using a Bloom filter for the visited set.  BF are most often used in combination with backend lookups, avoiding the lookup in the negative case.  Given that many URLs will positively match this will require expensive SQLite lookups.  Relying on only the Bloom filter would necessitate many bits to get a low fpp and thus many random memory lookups.  The analysis is complicated and it does not seem worth it to further reduce the 200 KB memory use of TLongHashSet.
(In reply to Andrew Gaul from comment #11)
> I benchmarked my patch on a Nexus 7 2013, before and after:

Could you clarify what these two times represent? I'm not sure what you mean by "before" and "after". Do you mean that without your changes is ~700ms, and with your changes is ~1300ms (but less memory)?


> I disagree about using a Bloom filter for the visited set.  BF are most
> often used in combination with backend lookups, avoiding the lookup in the
> negative case.  Given that many URLs will positively match this will require
> expensive SQLite lookups.

If the error is low-consequence — as it is here — no secondary lookup is necessary. Bloom filters give a probabilistic yes/definite no, so if we're OK with some small possibility of false positive (as discussed in Comment 2), then we don't need to fall through to another structure.
Flags: needinfo?(andrew)
Before measures the existing code, using more memory, while after includes the patch I submitted, using less memory.

Showing an unvisited URL as visited is low consequence in that the browser does not explode, however, incorrectly showing pornhub.com as visited might raise some eyebrows!  I disagree several parts of comment #2; tuning Bloom filters requires a fair amount of thought.  Particularly, lookup performance will be worse probing 20 or more bits for a sufficiently low fpp and this will tend to spill more valuable data from L1 cache for many lookups.  Also note that users will look up many more items than exist in the filter, particularly when visited link aggregators like Reddit, and the fpp should be based on this larger lookup set than the smaller visited set.
Flags: needinfo?(andrew) → needinfo?(rnewman)
Whiteboard: [good next bug][lang=java] → [good next bug][lang=java][MemShrink]
Attached patch Add MurmurHash3 (obsolete) — Splinter Review
This code is in the public domain and originated at:

https://github.com/yonik/java_util
Attachment #8969971 - Flags: review?(michael.l.comella)
Attachment #8969971 - Flags: review?(bugzilla)
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as a TLongHashSet instead of
HashSet<String> reduces this to a few hundred kilobytes by removing the
characters and halving the object overhead by using long instead of String and
its char array.  Using only 64 bits of the Murmush hash should have few
collisions given the limited size of the map.
Attachment #8969972 - Flags: review?(michael.l.comella)
Attachment #8969972 - Flags: review?(bugzilla)
Attachment #8948620 - Attachment is obsolete: true
Attachment #8948620 - Flags: review?(bugzilla)
Attachment #8943731 - Attachment is obsolete: true
Attachment #8943731 - Flags: review?(bugzilla)
Attachment #8948622 - Flags: review?(michael.l.comella)
Attachment #8948622 - Flags: review?(bugzilla)
I have changed the SHA-1 hash to a Murmur3 hash to reduce the run-time and potentially remove the overhead of referencing java crypto.  The updated benchmarks on my Nexus 7 2013, before and after:

04-21 22:48:16.844 26520 26563 W GeckoGlobalHistory: Rebuilt visited link set containing 35900 URLs in 2192 ms
04-21 23:07:37.929 27365 27413 W GeckoGlobalHistory: Rebuilt visited link set containing 35914 URLs in 2455 ms

With the larger URL set size the before overhead costs roughly 6 MB while the after overhead roughly 500 KB.  The before numbers are roughly double on older Android versions.  This patch reduces the visited set by over 10x so I have tagged MemShrink since I believe this represents a large improvement.
Flags: needinfo?(michael.l.comella)
Unfortunately, I'm only reviewing critical FF-Android bugs at the moment.

Susheel, would you be able to find a reviewer for this?
Flags: needinfo?(michael.l.comella) → needinfo?(sdaswani)
Andrew, are all these changes covered by unit tests? I think we can review it but I want to understand what's the risk of regressions.
Flags: needinfo?(sdaswani) → needinfo?(andrew)
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as a TLongHashSet instead of
HashSet<String> reduces this to a few hundred kilobytes by removing the
characters and halving the object overhead by using long instead of String and
its char array.  Using only 64 bits of the Murmush hash should have few
collisions given the limited size of the map.
Attachment #8972226 - Flags: review?(sdaswani)
Attachment #8972227 - Flags: review?(sdaswani)
Attachment #8969972 - Attachment is obsolete: true
Flags: needinfo?(andrew)
Attachment #8969972 - Flags: review?(bugzilla)
Vlad, can you, Petru or Andrei review this patch and the associated tests?

Thanks!
Flags: needinfo?(vlad.baicu)
Previously GlobalHistory lacked test coverage so I added a simple test.  However, this change is more about changing one of the underlying data structures for memory efficiency, e.g., switching out LinkedList for ArrayList, and not logic changes.
Andrew do we need to perform a migration or is the data structure only used in memory?
Flags: needinfo?(andrew)
No migration needed; this data structure is rebuilt in-memory on every start or after the garbage collector invalidates the SoftReference.
Flags: needinfo?(andrew)
Whiteboard: [good next bug][lang=java][MemShrink] → [good next bug][lang=java][MemShrink][Leanplum 61]
Whiteboard: [good next bug][lang=java][MemShrink][Leanplum 61] → [good next bug][lang=java][MemShrink][Leanplum][61]
Whiteboard: [good next bug][lang=java][MemShrink][Leanplum][61] → [Leanplum] [61]
Attachment #8972226 - Flags: review?(sdaswani) → review?(vlad.baicu)
Attachment #8972227 - Flags: review?(sdaswani) → review?(vlad.baicu)
While I am not very experienced with Murmurhash hash, after some research, I have discovered that Murmurhash has pretty much the same chance of collision as the other hashes over a wide range of data. (https://www.strchr.com/hash_functions) 

Patch lgtm r+
Flags: needinfo?(vlad.baicu)
Attachment #8972226 - Flags: review?(vlad.baicu) → review+
Comment on attachment 8972227 [details] [diff] [review]
Add simple test for GlobalHistory

Review of attachment 8972227 [details] [diff] [review]:
-----------------------------------------------------------------

Based on Andrew's review, I'm setting the flags to r+
Attachment #8972227 - Flags: review?(vlad.baicu) → review+
Attachment #8948622 - Flags: review?(vlad.baicu)
Attachment #8969971 - Flags: review?(vlad.baicu)
Thanks for the review Andrei! I'm setting an NI for snorp - want to make sure he blesses this.
Flags: needinfo?(snorp)
Attachment #8969971 - Flags: review?(vlad.baicu) → review+
Attachment #8948622 - Flags: review?(vlad.baicu) → review+
Neat. lgtm!
Flags: needinfo?(snorp)
Nick, can you push this through review - neither Vlad or I can, and both Vlad and Snorp are happy with the change.
Flags: needinfo?(nalexander)
Attached patch Add MurmurHash3 (obsolete) — Splinter Review
Rebased and addressed fallthrough warnings.
Attachment #8969971 - Attachment is obsolete: true
Attachment #8969971 - Flags: review?(bugzilla)
Attachment #8976781 - Flags: review?(vlad.baicu)
Comment on attachment 8976781 [details] [diff] [review]
Add MurmurHash3

Review of attachment 8976781 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me
Attachment #8976781 - Flags: review?(vlad.baicu) → review+
I've been asked to shepherd this through to landing, but it's not easy to tell what the current state is.  Are all the attached patches needed?  If so, please label them like

```
Bug 1197866 - Part 1: blah blah blah. r=person

Detailed comments here.
```

I'm concerned that we're incorporating code with a license requirement (i.e., a message must appear) but that there are no changes to the licensing file (https://searchfox.org/mozilla-central/source/toolkit/content/license.html), which suggests there's been no legal review of this CERN license.  Maybe that's fine, maybe it's not, and I'm not the person to say.  In any case, I see _zero_ discussion of license in this ticket, which raises flags.  If we need to talk about licensing, Mike Hoye (:mhoye) is the person to NI.

I see that rnewman flagged concerns about APK size, and I have the same concerns.  Try builds before and after would go a long way towards assuaging the correctness and size penalty of these changes; and it would clarify what parts are actually required, which are obsolete, etc.

Sorry to layer on process, but that's what's needed here...
Flags: needinfo?(nalexander)
Whiteboard: [Leanplum] [61] → [priority:low]
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attached patch Part 1. Add MurmurHash3 (obsolete) — Splinter Review
This code is in the public domain and originated at:

https://github.com/yonik/java_util
Attachment #8948622 - Attachment is obsolete: true
Attachment #8948622 - Flags: review?(bugzilla)
Attachment #8972226 - Attachment is obsolete: true
Attachment #8972227 - Attachment is obsolete: true
Attachment #8976781 - Attachment is obsolete: true
Attachment #9013913 - Flags: review?(nalexander)
Attachment #9013912 - Flags: review?(nalexander)
Attachment #9013911 - Flags: review?(nalexander)
Sorry for my months-delayed response but I believe I have addressed your suggestions.  I replaced the LGPL Trove dependency with a hand-written long primitive hash map and verified that the murmur hash has a public domain license.  I am not sure about the level of rigor you want for apk size; locally I measured 38296960 vs. 38293686, an increase of 3274 bytes.  Finally I modified the commit messages as you suggested.  I am struggling a bit with the hg/bugzilla workflow but I am keen to finish this now that I have some spare cycles.
Flags: needinfo?(nalexander)
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9013912 - Attachment is obsolete: true
Attachment #9013912 - Flags: review?(nalexander)
Attachment #9014132 - Flags: review?(nalexander)
Assignee: andrew → nobody
Status: ASSIGNED → NEW
Assignee: nobody → andrew
QA Contact: sdaswani
Assignee: andrew → nobody
QA Contact: sdaswani
(In reply to Andrew Gaul from comment #36)
> Sorry for my months-delayed response but I believe I have addressed your
> suggestions.  I replaced the LGPL Trove dependency with a hand-written long
> primitive hash map and verified that the murmur hash has a public domain
> license.  I am not sure about the level of rigor you want for apk size;
> locally I measured 38296960 vs. 38293686, an increase of 3274 bytes. 
> Finally I modified the commit messages as you suggested.  I am struggling a
> bit with the hg/bugzilla workflow but I am keen to finish this now that I
> have some spare cycles.

Sorry for the slow review.  It's in my queue and I'll reply by the end of the week (Pacific).  Thanks, Andrew!
Flags: needinfo?(nalexander)
Comment on attachment 9014132 [details] [diff] [review]
Part 2. Represent visited URLs with Murmur hash

Review of attachment 9014132 [details] [diff] [review]:
-----------------------------------------------------------------

Overall, this looks very good.  Major comments:

1) I would very much prefer to make the fill percentage calculations integer operations rather than floating point operations.  (I can't easily reason about floating point operations.)  That means keeping some integer state and updating it when you resize.  That will let you assert that the number of non-SENTINEL elements is > 0 and < elems.length - 1.  And it'll make checking for resize needed a little easier.

2) Javadoc comments through-out, please.  E.g., you return a `bool` on `add(long)` -- I know what you intend, but spell it out.

3) Should this be in GeckoView's `util/` directory?  I happen to know there are relevant things happening in GeckoView land over in https://bugzilla.mozilla.org/show_bug.cgi?id=1494713 and consumers might want this more generally.  snorp, lina -- I'd like this to be on your radar.  Maybe GV should ship a default history set, and it could be this?

r- is f+: I want to look at this again before landing.

::: mobile/android/app/src/test/java/org/mozilla/gecko/SimpleLongOpenHashSetTest.java
@@ +1,1 @@
> +/* -*- Mode: Java; c-basic-offset: 4; tab-width: 20; indent-tabs-mode: nil; -*-

Tests are usually Public Domain.  Can you follow whatever the license block is in this directory?

@@ +12,5 @@
> +
> +public final class SimpleLongOpenHashSetTest {
> +    @Test
> +    public void testEmpty() {
> +        SimpleLongOpenHashSet set = new SimpleLongOpenHashSet();

What happens in degenerate cases: size 0, size 1, size 2?

::: mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java
@@ +217,5 @@
>      }
> +
> +    /** Calculate hash of input. */
> +    private static long hashUrl(byte[] input, MurmurHash3.LongPair pair) {
> +        MurmurHash3.murmurhash3_x64_128(input, 0, input.length, 0, pair);

There's a note about avoiding the `.getBytes()` in your Murmur hash implementation.  Can we use that?  Also, why are we hashing to 128 bits and then immediately taking 64?  Why not just hash to 64 
(which is supported, I thought)?

Also, please say that we _do not need or want_ a cryptographic hash here: speed is required, good hash properties are not really required.

::: mobile/android/base/java/org/mozilla/gecko/SimpleLongOpenHashSet.java
@@ +7,5 @@
> +
> +import java.util.Arrays;
> +
> +/**
> + * Simple open hash table which specializes for the primitive long type which

nit: A simple open hash set, specialized for primitive long values.

There is no support for removing individual values but clearing the entire set is supported.

The implementation uses linear probing.

@@ +34,5 @@
> +        if (elem == SENTINEL) {
> +            ++elem;
> +        }
> +        int index = hash(elem);
> +        while (true) {

You're subtly using that the underlying array can never be full, correct?  Otherwise you might loop over the entire array again and again.

@@ +91,5 @@
> +        }
> +    }
> +
> +    private int hash(long elem) {
> +        return Math.abs(Long.valueOf(elem).hashCode()) % elems.length;

Worth a comment to say that Java's modulus maintains the sign bit of the input value.  Also, `Long.valueOf().hashCode()` allocates -- Prefer https://docs.oracle.com/javase/8/docs/api/java/lang/Long.html#hashCode-long- please.

Also, why are we hashing our input long, which is itself a hash?  Can we avoid that?
Attachment #9014132 - Flags: review?(nalexander)
Attachment #9014132 - Flags: review-
Attachment #9014132 - Flags: feedback?(snorp)
Attachment #9014132 - Flags: feedback?(lina)
Comment on attachment 9013913 [details] [diff] [review]
Part 1. Add MurmurHash3

Review of attachment 9013913 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/base/java/org/mozilla/gecko/MurmurHash3.java
@@ +1,4 @@
> +package org.mozilla.gecko;
> +
> +/**
> + *  The MurmurHash3 algorithm was created by Austin Appleby and placed in the public domain.

Emphasize that Murmur hash is _not_ a cryptographic hash and should never be used as such.
Attachment #9013913 - Flags: review?(nalexander) → review+
Comment on attachment 9013911 [details] [diff] [review]
Part 3. Add simple test for GlobalHistory

Review of attachment 9013911 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/app/src/test/java/org/mozilla/gecko/GlobalPageMetadataTest.java
@@ +112,5 @@
>  
> +    @Test
> +    public void testGlobalHistory() throws Exception {
> +        GlobalHistory history = GlobalHistory.getInstance();
> +        history.mVisitedCache = new SoftReference<>(new SimpleLongOpenHashSet());

Does this race against destroying the SoftReference?  Maybe keep the underlying hash set reference yourself and comment.

This is a theoretical concern, but who knows...

@@ +117,5 @@
> +        String exampleDotCom = "http://example.com/";
> +        assertFalse(history.containsUri(exampleDotCom));
> +        history.addUri(exampleDotCom);
> +        assertTrue(history.containsUri(exampleDotCom));
> +        assertFalse(history.containsUri(exampleDotCom + 2));

This is weird: `+ "2.html"` or something, not string plus number :)
Attachment #9013911 - Flags: review?(nalexander) → review+
Comment on attachment 9014132 [details] [diff] [review]
Part 2. Represent visited URLs with Murmur hash

Review of attachment 9014132 [details] [diff] [review]:
-----------------------------------------------------------------

Seems ok to me, but I didn't look all that closely. We're going to need similar caching for the visited link set in GeckoView, though, so this is certainly interesting.
Attachment #9014132 - Flags: feedback?(snorp) → feedback+
Attached patch Part 1: Add MurmurHash3 (obsolete) — Splinter Review
This code is in the public domain and originated at:

https://github.com/yonik/java_util
Attachment #9017670 - Flags: review?(nalexander)
Assignee: nobody → andrew
Status: NEW → ASSIGNED
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9017671 - Flags: review?(nalexander)
Attachment #9017672 - Flags: review?(nalexander)
Attachment #9017671 - Flags: feedback?(lina)
Attachment #9013911 - Attachment is obsolete: true
Attachment #9013913 - Attachment is obsolete: true
Attachment #9014132 - Attachment is obsolete: true
Attachment #9014132 - Flags: feedback?(lina)
I addressed most suggestiosn in the updated patches and respond to
others below:

> 1) I would very much prefer to make the fill percentage calculations
> integer operations rather than floating point operations.

I changed this to integer arithmetic but am not sure I addressed your
suggestion.  Could you look at the changes?

> 3) Should this be in GeckoView's `util/` directory?

Happy to implement suggestions to make this reusable by other
components.

> There's a note about avoiding the `.getBytes()` in your Murmur hash
> implementation.  Can we use that?  Also, why are we hashing to 128
> bits and then immediately taking 64?  Why not just hash to 64 (which
> is supported, I thought)?

The source provides two implementations:

    int murmurhash3_x86_32(CharSequence)
    void murmurhash3_x64_128(byte[], LongPair)

The former specializes for CharSequence but only returns 32 bits of
hash.  The latter returns two longs which GlobalHistory xors into a
single 64-bit quantity.  Adapting the CharSequence optimization from the
former to the latter looks tricky.  Do we need to implement this since
it only applies to the single element insert path?  The bulk rebuilding
path avoids calling String.getBytes.
(In reply to Andrew Gaul from comment #46)
> I addressed most suggestiosn in the updated patches and respond to
> others below:
> 
> > 1) I would very much prefer to make the fill percentage calculations
> > integer operations rather than floating point operations.
> 
> I changed this to integer arithmetic but am not sure I addressed your
> suggestion.  Could you look at the changes?

Will do.

> > 3) Should this be in GeckoView's `util/` directory?
> 
> Happy to implement suggestions to make this reusable by other
> components.

Let's push to follow-up.

> > There's a note about avoiding the `.getBytes()` in your Murmur hash
> > implementation.  Can we use that?  Also, why are we hashing to 128
> > bits and then immediately taking 64?  Why not just hash to 64 (which
> > is supported, I thought)?
> 
> The source provides two implementations:
> 
>     int murmurhash3_x86_32(CharSequence)
>     void murmurhash3_x64_128(byte[], LongPair)
> 
> The former specializes for CharSequence but only returns 32 bits of
> hash.  The latter returns two longs which GlobalHistory xors into a
> single 64-bit quantity.  Adapting the CharSequence optimization from the
> former to the latter looks tricky.  Do we need to implement this since
> it only applies to the single element insert path?  The bulk rebuilding
> path avoids calling String.getBytes.

Sure, this is fine by me.
Flags: needinfo?(bugzilla)
Comment on attachment 9017671 [details] [diff] [review]
Part 2: Represent visited URLs with Murmur hash

Review of attachment 9017671 [details] [diff] [review]:
-----------------------------------------------------------------

OK, I think this is fine.  Good work!
Attachment #9017671 - Flags: review?(nalexander) → review+
Comment on attachment 9017672 [details] [diff] [review]
Part 3: Add simple test for GlobalHistory

Review of attachment 9017672 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/app/src/test/java/org/mozilla/gecko/GlobalPageMetadataTest.java
@@ +112,5 @@
>  
> +    @Test
> +    public void testGlobalHistory() throws Exception {
> +        GlobalHistory history = GlobalHistory.getInstance();
> +        // hold referece to prevent GC race

nit: "reference", full sentence:

// Hold reference to prevent racing with GC.
Attachment #9017672 - Flags: review?(nalexander) → review+
Thanks for your patience, Andrew.  Bombs away!
Comment on attachment 9017671 [details] [diff] [review]
Part 2: Represent visited URLs with Murmur hash

Review of attachment 9017671 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM, too. :-) We'll definitely want some kind of caching like this in GeckoView. Checking the visited status of a link involves storing the full `nsIURI` in memory, an IPC message to the parent, and an event dispatch to Java, and that's before we get to the JNA calls into our Rust Places/Sync library, and querying SQLite. Storing a set of visited URLs in the content process would let us bypass that entirely; storing the hashes on the Java side would still avoid calling into Places.
Attachment #9017671 - Flags: feedback?(lina) → feedback+
(In reply to Lina Cambridge (she/her) [:lina] from comment #51)
> Storing a set of visited URLs in the content process...

I meant a set of visited URL hashes. 😅
Attachment #9019202 - Flags: review?(nalexander)
Attachment #9017672 - Attachment is obsolete: true
Attachment #9019203 - Flags: review?(nalexander)
Attachment #9019202 - Attachment is obsolete: true
Attachment #9019202 - Flags: review?(nalexander)
> nit: "reference", full sentence:

Fixed nit.
I've see checkstyle failures at https://treeherder.mozilla.org/#/jobs?repo=try&author=nalexander%40mozilla.com&selectedJob=207282509

Andrew, can you address?  You can run this locally with |mach android checkstyle|.
Flags: needinfo?(andrew)
Attached patch Part 1: Add MurmurHash3 (obsolete) — Splinter Review
This code is in the public domain and originated at:

https://github.com/yonik/java_util
Attachment #9019474 - Flags: review?(nalexander)
Attachment #9017670 - Attachment is obsolete: true
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9019475 - Flags: review?(nalexander)
Attachment #9017671 - Attachment is obsolete: true
To address the Checkstyle errors I moved MurmurHash3 to the thirdparty directory, where it should have been to begin with.
Flags: needinfo?(andrew)
Pushed by mozilla@noorenberghe.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b9befdce34d
Part 1: Add MurmurHash3. r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/a80113dba9c1
Part 2: Represent visited URLs with Murmur hash. r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/32db311632da
Part 3: Add simple test for GlobalHistory. r=nalexander
Keywords: checkin-needed
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9020143 - Flags: review?(nalexander)
Attachment #9019475 - Attachment is obsolete: true
Incorrect truncation from long to int caused the failures, introduced when addressing a review comment and relying on the unit tests instead of live testing.  I have added coverage to SimpleLongOpenHashSetTest.testEdgeElements and fixed SimpleLongOpenHashSet.index to address these.
Flags: needinfo?(andrew)
See Also: → 1503481
(In reply to Andrew Gaul from comment #64)
> Incorrect truncation from long to int caused the failures, introduced when
> addressing a review comment and relying on the unit tests instead of live
> testing.  I have added coverage to
> SimpleLongOpenHashSetTest.testEdgeElements and fixed
> SimpleLongOpenHashSet.index to address these.

I think I understand the change.  

-        // remove sign bit since modulus preserves it
-        return (int) Math.abs(elem) % elems.length;
+        // Remove sign bit since modulus preserves it.
+        return (((int) (elem ^ (elem >>> 32))) & Integer.MAX_VALUE) % elems.length;

You're ORing the top and bottom 32 bits of the 64 bit input, then dropping the top 32 bits, casting to int (which will always succeed due to the AND dropping the top 32 bits), then taking modulus?

That makes sense.  Bump the comment to say what you're doing, please.

It looks like there's an extraneous change:

diff --git a/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java b/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java
--- a/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java
+++ b/mobile/android/base/java/org/mozilla/gecko/GlobalHistory.java
@@ -10,6 +10,8 @@ import java.security.NoSuchAlgorithmExce
 import java.util.LinkedList;
 import java.util.Queue;

+import com.github.yonik.MurmurHash3;
+

Is that needed?

Also, in `testEdgeElements`, can you test `Long.MAX_VALUE` and `Long.MAX_VALUE - 1`?

Thanks!  I'll run this through try now as well.  (It would be good for you to have try access, Andrew.)
Comment on attachment 9020143 [details] [diff] [review]
Part 2: Represent visited URLs with Murmur hash

Review of attachment 9020143 [details] [diff] [review]:
-----------------------------------------------------------------

r- just for nits.
Attachment #9020143 - Flags: review?(nalexander) → review-
(In reply to Nick Alexander :nalexander [he/him] from comment #67)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=8d8fb2fecc22f52732409d6e9ae83e150e1d3487

This looks okay except for the checkstyle issues in https://taskcluster-artifacts.net/LOE0hU73R3W5Kug0_FEaYA/0/public/logs/live_backing.log.
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9020143 - Attachment is obsolete: true
Attachment #9022287 - Flags: review?(nalexander)
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9022289 - Flags: review?(nalexander)
Attachment #9022287 - Attachment is obsolete: true
Attachment #9022287 - Flags: review?(nalexander)
(In reply to Nick Alexander :nalexander [he/him] from comment #65)
> You're ORing the top and bottom 32 bits of the 64 bit input, then dropping
> the top 32 bits, casting to int (which will always succeed due to the AND
> dropping the top 32 bits), then taking modulus?
> 
> That makes sense.  Bump the comment to say what you're doing, please.

XORing actually, taken from Long.hashCode, which presumably gives a higher-quality result than just truncation.  I added a comment explaining the operation.

> It looks like there's an extraneous change:
> 
> +import com.github.yonik.MurmurHash3;
> 
> Is that needed?

I put MurmurHash3 in its own package since it originates elsewhere and does not follow Mozilla formatting.  ./mach android checkstyle succeeds for me locally; I do not understand how to reproduce your try error?

> Also, in `testEdgeElements`, can you test `Long.MAX_VALUE` and
> `Long.MAX_VALUE - 1`?

Done.

> Thanks!  I'll run this through try now as well.  (It would be good for you
> to have try access, Andrew.)

You sponsored me about a year but I cannot log in.  I am trying to determine the cause of this in bug 1504332.
This code is in the public domain and originated at:

https://github.com/yonik/java_util
This set can contain tens of thousands of entries each with 100 characters,
consuming several megabytes.  Representing this as specialized primitive long
set instead of HashSet<String> reduces this to a few hundred kilobytes by
removing the characters and halving the object overhead by using long instead of
String and its char array.  Using only 64 bits of the Murmush hash should have
few collisions given the limited size of the map.
Attachment #9022289 - Attachment is obsolete: true
Attachment #9019474 - Attachment is obsolete: true
Attachment #9019203 - Attachment is obsolete: true
Flags: needinfo?(nalexander)
Flags: needinfo?(nalexander)
Attachment #9032329 - Flags: review+
Patches look good, try push looks good -- I think this is ready to go!
Keywords: checkin-needed
Pushed by rmaries@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/59020fd50495
Part 2: Represent visited URLs with Murmur hash. r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/84da51f01bd9
Part 3: Add simple test for GlobalHistory. r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/df8ca6189da3
Part 1: Add MurmurHash3. r=nalexander
Keywords: checkin-needed
Backed out for failing android at layout/style/test/test_animations_omta.html.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=df8ca6189da3b8aa9cf830566f42a48ca8009a68

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=218029827&repo=mozilla-inbound&lineNumber=1994

[task 2018-12-20T01:48:37.214Z] 01:48:37     INFO -  245 INFO TEST-START | layout/style/test/test_animations_omta.html
[task 2018-12-20T01:56:44.850Z] 01:56:44     INFO -  wait for org.mozilla.fennec_aurora complete; top activity=org.mozilla.fennec_aurora
[task 2018-12-20T01:56:45.456Z] 01:56:45     INFO -  org.mozilla.fennec_aurora unexpectedly found running. Killing...
[task 2018-12-20T01:56:45.456Z] 01:56:45     INFO -  TEST-INFO | started process screentopng
[task 2018-12-20T01:56:45.794Z] 01:56:45     INFO -  TEST-INFO | screentopng: exit 0
[task 2018-12-20T01:57:00.867Z] 01:57:00  WARNING -  TEST-UNEXPECTED-FAIL | layout/style/test/test_animations_omta.html | application timed out after 370 seconds with no output
[task 2018-12-20T01:57:00.868Z] 01:57:00     INFO -  remoteautomation.py | Application ran for: 0:11:02.448264
[task 2018-12-20T01:57:01.179Z] 01:57:01     INFO -  Contents of /data/anr/traces.txt:
[task 2018-12-20T01:57:01.180Z] 01:57:01     INFO -  ----- pid 1975 at 2018-12-19 17:56:45 -----
[task 2018-12-20T01:57:01.180Z] 01:57:01     INFO -  Cmd line: org.mozilla.fennec_aurora
[task 2018-12-20T01:57:01.181Z] 01:57:01     INFO -  JNI: CheckJNI is on; workarounds are off; pins=0; globals=273 (plus 5 weak)
[task 2018-12-20T01:57:01.181Z] 01:57:01     INFO -  DALVIK THREADS:
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -  (mutexes: tll=0 tsl=0 tscl=0 ghl=0)
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -  "main" prio=5 tid=1 NATIVE
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x414c9578 self=0x2a00d090
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    | sysTid=1975 nice=0 sched=0/0 cgrp=apps handle=1073811452
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    | state=S schedstat=( 45968446153 15968033603 45065 ) utm=3084 stm=1512 core=0
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    #00  pc 0001c5a4  /system/lib/libc.so (__futex_syscall3+8)
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    #01  pc 0000e688  /system/lib/libc.so (__pthread_cond_timedwait_relative+48)
[task 2018-12-20T01:57:01.183Z] 01:57:01     INFO -    #02  pc 0000e6e8  /system/lib/libc.so (__pthread_cond_timedwait+64)
[task 2018-12-20T01:57:01.184Z] 01:57:01     INFO -    #03  pc 00052e97  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.185Z] 01:57:01     INFO -    #04  pc 00053461  /system/lib/libdvm.so (dvmChangeStatus(Thread*, ThreadStatus)+30)
[task 2018-12-20T01:57:01.186Z] 01:57:01     INFO -    #05  pc 00048d29  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.186Z] 01:57:01     INFO -    #06  pc 00039afd  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.187Z] 01:57:01     INFO -    #07  pc 0004b7a9  /system/lib/libandroid_runtime.so (android::AndroidRuntime::getJNIEnv()+16)
[task 2018-12-20T01:57:01.187Z] 01:57:01     INFO -    #08  pc 0005eed5  /system/lib/libandroid_runtime.so (android::NativeDisplayEventReceiver::dispatchVsync(long long, int, unsigned int)+20)
[task 2018-12-20T01:57:01.188Z] 01:57:01     INFO -    #09  pc 0005f0b9  /system/lib/libandroid_runtime.so (android::NativeDisplayEventReceiver::handleEvent(int, int, void*)+80)
[task 2018-12-20T01:57:01.188Z] 01:57:01     INFO -    #10  pc 00015129  /system/lib/libutils.so (android::Looper::pollInner(int)+468)
[task 2018-12-20T01:57:01.188Z] 01:57:01     INFO -    #11  pc 000151d5  /system/lib/libutils.so (android::Looper::pollOnce(int, int*, int*, void**)+92)
[task 2018-12-20T01:57:01.189Z] 01:57:01     INFO -    #12  pc 00067b69  /system/lib/libandroid_runtime.so (android::NativeMessageQueue::pollOnce(_JNIEnv*, int)+22)
[task 2018-12-20T01:57:01.189Z] 01:57:01     INFO -    #13  pc 0001dc4c  /system/lib/libdvm.so (dvmPlatformInvoke+112)
[task 2018-12-20T01:57:01.190Z] 01:57:01     INFO -    #14  pc 0004dcab  /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+394)
[task 2018-12-20T01:57:01.190Z] 01:57:01     INFO -    #15  pc 000385e1  /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+8)
[task 2018-12-20T01:57:01.191Z] 01:57:01     INFO -    #16  pc 00000214  /dev/ashmem/dalvik-jit-code-cache (deleted)
[task 2018-12-20T01:57:01.191Z] 01:57:01     INFO -    at android.os.MessageQueue.nativePollOnce(Native Method)
[task 2018-12-20T01:57:01.192Z] 01:57:01     INFO -    at android.os.MessageQueue.next(MessageQueue.java:132)
[task 2018-12-20T01:57:01.192Z] 01:57:01     INFO -    at android.os.Looper.loop(Looper.java:124)
[task 2018-12-20T01:57:01.192Z] 01:57:01     INFO -    at android.app.ActivityThread.main(ActivityThread.java:5103)
[task 2018-12-20T01:57:01.192Z] 01:57:01     INFO -    at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-12-20T01:57:01.192Z] 01:57:01     INFO -    at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-12-20T01:57:01.193Z] 01:57:01     INFO -    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:737)
[task 2018-12-20T01:57:01.194Z] 01:57:01     INFO -    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
[task 2018-12-20T01:57:01.194Z] 01:57:01     INFO -    at dalvik.system.NativeStart.main(Native Method)
[task 2018-12-20T01:57:01.194Z] 01:57:01     INFO -  "GeckoIconTask" prio=5 tid=18 WAIT
[task 2018-12-20T01:57:01.195Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41bb57b0 self=0x2a266770
[task 2018-12-20T01:57:01.195Z] 01:57:01     INFO -    | sysTid=2060 nice=0 sched=0/0 cgrp=apps handle=707875376
[task 2018-12-20T01:57:01.196Z] 01:57:01     INFO -    | state=S schedstat=( 164758417 476640542 114 ) utm=14 stm=2 core=0
[task 2018-12-20T01:57:01.196Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.197Z] 01:57:01     INFO -    - waiting on <0x41bb58d8> (a java.lang.VMThread) held by tid=18 (GeckoIconTask)
[task 2018-12-20T01:57:01.197Z] 01:57:01     INFO -    at java.lang.Thread.parkFor(Thread.java:1205)
[task 2018-12-20T01:57:01.198Z] 01:57:01     INFO -    at sun.misc.Unsafe.park(Unsafe.java:325)
[task 2018-12-20T01:57:01.199Z] 01:57:01     INFO -    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:159)
[task 2018-12-20T01:57:01.199Z] 01:57:01     INFO -    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2019)
[task 2018-12-20T01:57:01.200Z] 01:57:01     INFO -    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:413)
[task 2018-12-20T01:57:01.201Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1013)
[task 2018-12-20T01:57:01.201Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1073)
[task 2018-12-20T01:57:01.202Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
[task 2018-12-20T01:57:01.202Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.204Z] 01:57:01     INFO -  "AsyncTask #1" prio=5 tid=17 WAIT
[task 2018-12-20T01:57:01.204Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41ae1af0 self=0x2a2ec590
[task 2018-12-20T01:57:01.204Z] 01:57:01     INFO -    | sysTid=2044 nice=10 sched=0/0 cgrp=apps/bg_non_interactive handle=707851112
[task 2018-12-20T01:57:01.205Z] 01:57:01     INFO -    | state=S schedstat=( 7460278 17726731 7 ) utm=0 stm=0 core=0
[task 2018-12-20T01:57:01.205Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.205Z] 01:57:01     INFO -    - waiting on <0x41ae1c88> (a java.lang.VMThread) held by tid=17 (AsyncTask #1)
[task 2018-12-20T01:57:01.206Z] 01:57:01     INFO -    at java.lang.Thread.parkFor(Thread.java:1205)
[task 2018-12-20T01:57:01.206Z] 01:57:01     INFO -    at sun.misc.Unsafe.park(Unsafe.java:325)
[task 2018-12-20T01:57:01.206Z] 01:57:01     INFO -    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:159)
[task 2018-12-20T01:57:01.207Z] 01:57:01     INFO -    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2019)
[task 2018-12-20T01:57:01.207Z] 01:57:01     INFO -    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:413)
[task 2018-12-20T01:57:01.208Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1013)
[task 2018-12-20T01:57:01.208Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1073)
[task 2018-12-20T01:57:01.208Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
[task 2018-12-20T01:57:01.209Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.209Z] 01:57:01     INFO -  "WifiManager" prio=5 tid=16 NATIVE
[task 2018-12-20T01:57:01.210Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41a91548 self=0x2a2eba10
[task 2018-12-20T01:57:01.210Z] 01:57:01     INFO -    | sysTid=2043 nice=0 sched=0/0 cgrp=apps handle=707827376
[task 2018-12-20T01:57:01.211Z] 01:57:01     INFO -    | state=S schedstat=( 3283242 22271360 10 ) utm=0 stm=0 core=0
[task 2018-12-20T01:57:01.211Z] 01:57:01     INFO -    #00  pc 0001c3dc  /system/lib/libc.so (epoll_wait+12)
[task 2018-12-20T01:57:01.212Z] 01:57:01     INFO -    #01  pc 00014fb1  /system/lib/libutils.so (android::Looper::pollInner(int)+92)
[task 2018-12-20T01:57:01.212Z] 01:57:01     INFO -    #02  pc 000151d5  /system/lib/libutils.so (android::Looper::pollOnce(int, int*, int*, void**)+92)
[task 2018-12-20T01:57:01.212Z] 01:57:01     INFO -    #03  pc 00067b69  /system/lib/libandroid_runtime.so (android::NativeMessageQueue::pollOnce(_JNIEnv*, int)+22)
[task 2018-12-20T01:57:01.213Z] 01:57:01     INFO -    #04  pc 0001dc4c  /system/lib/libdvm.so (dvmPlatformInvoke+112)
[task 2018-12-20T01:57:01.213Z] 01:57:01     INFO -    #05  pc 0004dcab  /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+394)
[task 2018-12-20T01:57:01.213Z] 01:57:01     INFO -    #06  pc 000385e1  /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+8)
[task 2018-12-20T01:57:01.214Z] 01:57:01     INFO -    #07  pc 00027060  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.214Z] 01:57:01     INFO -    #08  pc 0002b580  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
[task 2018-12-20T01:57:01.214Z] 01:57:01     INFO -    #09  pc 0005fcbd  /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+292)
[task 2018-12-20T01:57:01.215Z] 01:57:01     INFO -    #10  pc 0005fce7  /system/lib/libdvm.so (dvmCallMethod(Thread*, Method const*, Object*, JValue*, ...)+20)
[task 2018-12-20T01:57:01.215Z] 01:57:01     INFO -    #11  pc 00054a6f  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.215Z] 01:57:01     INFO -    #12  pc 0000ca58  /system/lib/libc.so (__thread_entry+72)
[task 2018-12-20T01:57:01.216Z] 01:57:01     INFO -    #13  pc 0000cbd4  /system/lib/libc.so (pthread_create+208)
[task 2018-12-20T01:57:01.216Z] 01:57:01     INFO -    at android.os.MessageQueue.nativePollOnce(Native Method)
[task 2018-12-20T01:57:01.216Z] 01:57:01     INFO -    at android.os.MessageQueue.next(MessageQueue.java:132)
[task 2018-12-20T01:57:01.217Z] 01:57:01     INFO -    at android.os.Looper.loop(Looper.java:124)
[task 2018-12-20T01:57:01.217Z] 01:57:01     INFO -    at android.os.HandlerThread.run(HandlerThread.java:61)
[task 2018-12-20T01:57:01.217Z] 01:57:01     INFO -  "Thread-83" prio=5 tid=15 NATIVE
[task 2018-12-20T01:57:01.217Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x4178c9e0 self=0x2a309018
[task 2018-12-20T01:57:01.218Z] 01:57:01     INFO -    | sysTid=2030 nice=0 sched=0/0 cgrp=apps handle=707373960
[task 2018-12-20T01:57:01.218Z] 01:57:01     INFO -    | state=S schedstat=( 19794967568 21869398834 38585 ) utm=1661 stm=318 core=0
[task 2018-12-20T01:57:01.218Z] 01:57:01     INFO -    #00  pc 0001c5a4  /system/lib/libc.so (__futex_syscall3+8)
[task 2018-12-20T01:57:01.219Z] 01:57:01     INFO -    #01  pc 0000e688  /system/lib/libc.so (__pthread_cond_timedwait_relative+48)
[task 2018-12-20T01:57:01.219Z] 01:57:01     INFO -    #02  pc 0000e6e8  /system/lib/libc.so (__pthread_cond_timedwait+64)
[task 2018-12-20T01:57:01.219Z] 01:57:01     INFO -    #03  pc 0034de9d  /data/app-lib/org.mozilla.fennec_aurora-1/libxul.so
[task 2018-12-20T01:57:01.220Z] 01:57:01     INFO -    at dalvik.system.NativeStart.run(Native Method)
[task 2018-12-20T01:57:01.220Z] 01:57:01     INFO -  "pool-1-thread-1" prio=5 tid=14 WAIT
[task 2018-12-20T01:57:01.220Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41a8f868 self=0x2a1e5e50
[task 2018-12-20T01:57:01.220Z] 01:57:01     INFO -    | sysTid=2001 nice=0 sched=0/0 cgrp=apps handle=706634400
[task 2018-12-20T01:57:01.221Z] 01:57:01     INFO -    | state=S schedstat=( 24573388 36126978 38 ) utm=1 stm=1 core=0
[task 2018-12-20T01:57:01.221Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.221Z] 01:57:01     INFO -    - waiting on <0x41a8f9b0> (a java.lang.VMThread) held by tid=14 (pool-1-thread-1)
[task 2018-12-20T01:57:01.222Z] 01:57:01     INFO -    at java.lang.Thread.parkFor(Thread.java:1205)
[task 2018-12-20T01:57:01.222Z] 01:57:01     INFO -    at sun.misc.Unsafe.park(Unsafe.java:325)
[task 2018-12-20T01:57:01.222Z] 01:57:01     INFO -    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:159)
[task 2018-12-20T01:57:01.223Z] 01:57:01     INFO -    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2019)
[task 2018-12-20T01:57:01.223Z] 01:57:01     INFO -    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:413)
[task 2018-12-20T01:57:01.223Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1013)
[task 2018-12-20T01:57:01.224Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1073)
[task 2018-12-20T01:57:01.224Z] 01:57:01     INFO -    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
[task 2018-12-20T01:57:01.224Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.225Z] 01:57:01     INFO -  "actionMode" prio=5 tid=13 WAIT
[task 2018-12-20T01:57:01.225Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41a83390 self=0x2a1e80b0
[task 2018-12-20T01:57:01.225Z] 01:57:01     INFO -    | sysTid=2000 nice=0 sched=0/0 cgrp=apps handle=706643200
[task 2018-12-20T01:57:01.225Z] 01:57:01     INFO -    | state=S schedstat=( 403003 40644439 2 ) utm=0 stm=0 core=0
[task 2018-12-20T01:57:01.226Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.226Z] 01:57:01     INFO -    - waiting on <0x41a83390> (a java.util.Timer$TimerImpl)
[task 2018-12-20T01:57:01.226Z] 01:57:01     INFO -    at java.lang.Object.wait(Object.java:364)
[task 2018-12-20T01:57:01.227Z] 01:57:01     INFO -    at java.util.Timer$TimerImpl.run(Timer.java:214)
[task 2018-12-20T01:57:01.227Z] 01:57:01     INFO -  "Gecko" prio=5 tid=12 SUSPENDED
[task 2018-12-20T01:57:01.227Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41750c20 self=0x2a1c8b48
[task 2018-12-20T01:57:01.227Z] 01:57:01     INFO -    | sysTid=1992 nice=0 sched=0/0 cgrp=apps handle=706515464
[task 2018-12-20T01:57:01.228Z] 01:57:01     INFO -    | state=S schedstat=( 271337169370 155759616854 351810 ) utm=23270 stm=3863 core=0
[task 2018-12-20T01:57:01.228Z] 01:57:01     INFO -    #00  pc 0001c5a4  /system/lib/libc.so (__futex_syscall3+8)
[task 2018-12-20T01:57:01.229Z] 01:57:01     INFO -    #01  pc 0000e688  /system/lib/libc.so (__pthread_cond_timedwait_relative+48)
[task 2018-12-20T01:57:01.229Z] 01:57:01     INFO -    #02  pc 0000e6e8  /system/lib/libc.so (__pthread_cond_timedwait+64)
[task 2018-12-20T01:57:01.229Z] 01:57:01     INFO -    #03  pc 00052e97  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.229Z] 01:57:01     INFO -    #04  pc 00053461  /system/lib/libdvm.so (dvmChangeStatus(Thread*, ThreadStatus)+30)
[task 2018-12-20T01:57:01.230Z] 01:57:01     INFO -    #05  pc 0003892f  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.230Z] 01:57:01     INFO -    #06  pc 0003981f  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.230Z] 01:57:01     INFO -    #07  pc 0003e5bf  /system/lib/libdvm.so
[task 2018-12-20T01:57:01.231Z] 01:57:01     INFO -    #08  pc 012ae33f  /data/app-lib/org.mozilla.fennec_aurora-1/libxul.so
[task 2018-12-20T01:57:01.231Z] 01:57:01     INFO -    at org.mozilla.gecko.mozglue.GeckoLoader.nativeRun(Native Method)
[task 2018-12-20T01:57:01.231Z] 01:57:01     INFO -    at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:466)
[task 2018-12-20T01:57:01.232Z] 01:57:01     INFO -  "GeckoBackgroundThread" daemon prio=5 tid=11 NATIVE
[task 2018-12-20T01:57:01.232Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x4173de90 self=0x2a04fe60
[task 2018-12-20T01:57:01.232Z] 01:57:01     INFO -    | sysTid=1989 nice=0 sched=0/0 cgrp=apps handle=704971440
[task 2018-12-20T01:57:01.232Z] 01:57:01     INFO -    | state=S schedstat=( 1560229726 3618184608 1063 ) utm=134 stm=22 core=0
[task 2018-12-20T01:57:01.233Z] 01:57:01     INFO -    #00  pc 0001c3dc  /system/lib/libc.so (epoll_wait+12)
[task 2018-12-20T01:57:01.233Z] 01:57:01     INFO -    #01  pc 00014fb1  /system/lib/libutils.so (android::Looper::pollInner(int)+92)
[task 2018-12-20T01:57:01.233Z] 01:57:01     INFO -    #02  pc 000151d5  /system/lib/libutils.so (android::Looper::pollOnce(int, int*, int*, void**)+92)
[task 2018-12-20T01:57:01.234Z] 01:57:01     INFO -    #03  pc 00067b69  /system/lib/libandroid_runtime.so (android::NativeMessageQueue::pollOnce(_JNIEnv*, int)+22)
[task 2018-12-20T01:57:01.234Z] 01:57:01     INFO -    #04  pc 0001dc4c  /system/lib/libdvm.so (dvmPlatformInvoke+112)
[task 2018-12-20T01:57:01.234Z] 01:57:01     INFO -    #05  pc 0004dcab  /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+394)
[task 2018-12-20T01:57:01.235Z] 01:57:01     INFO -    #06  pc 000385e1  /system/lib/libdvm.so (dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+8)
[task 2018-12-20T01:57:01.235Z] 01:57:01     INFO -    #07  pc 00000214  /dev/ashmem/dalvik-jit-code-cache (deleted)
[task 2018-12-20T01:57:01.235Z] 01:57:01     INFO -    at android.os.MessageQueue.nativePollOnce(Native Method)
[task 2018-12-20T01:57:01.236Z] 01:57:01     INFO -    at android.os.MessageQueue.next(MessageQueue.java:132)
[task 2018-12-20T01:57:01.236Z] 01:57:01     INFO -    at android.os.Looper.loop(Looper.java:124)
[task 2018-12-20T01:57:01.236Z] 01:57:01     INFO -    at org.mozilla.gecko.util.GeckoBackgroundThread.run(GeckoBackgroundThread.java:43)
[task 2018-12-20T01:57:01.237Z] 01:57:01     INFO -  "Binder_2" prio=5 tid=10 NATIVE
[task 2018-12-20T01:57:01.237Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41745bb0 self=0x2a04eb98
[task 2018-12-20T01:57:01.237Z] 01:57:01     INFO -    | sysTid=1987 nice=0 sched=0/0 cgrp=apps handle=704964944
[task 2018-12-20T01:57:01.237Z] 01:57:01     INFO -    | state=S schedstat=( 10398292 6485252 18 ) utm=0 stm=1 core=0
[task 2018-12-20T01:57:01.238Z] 01:57:01     INFO -    #00  pc 0001b2d4  /system/lib/libc.so (__ioctl+8)
[task 2018-12-20T01:57:01.238Z] 01:57:01     INFO -    #01  pc 0002b94b  /system/lib/libc.so (ioctl+14)
[task 2018-12-20T01:57:01.238Z] 01:57:01     INFO -    #02  pc 0001b539  /system/lib/libbinder.so (android::IPCThreadState::talkWithDriver(bool)+140)
[task 2018-12-20T01:57:01.239Z] 01:57:01     INFO -    #03  pc 0001bcd7  /system/lib/libbinder.so (android::IPCThreadState::joinThreadPool(bool)+154)
[task 2018-12-20T01:57:01.239Z] 01:57:01     INFO -    #04  pc 0001faa5  /system/lib/libbinder.so
[task 2018-12-20T01:57:01.239Z] 01:57:01     INFO -    #05  pc 00011a75  /system/lib/libutils.so (android::Thread::_threadLoop(void*)+216)
[task 2018-12-20T01:57:01.240Z] 01:57:01     INFO -    #06  pc 0004b629  /system/lib/libandroid_runtime.so (android::AndroidRuntime::javaThreadShell(void*)+68)
[task 2018-12-20T01:57:01.240Z] 01:57:01     INFO -    #07  pc 00011569  /system/lib/libutils.so
[task 2018-12-20T01:57:01.240Z] 01:57:01     INFO -    #08  pc 0000ca58  /system/lib/libc.so (__thread_entry+72)
[task 2018-12-20T01:57:01.241Z] 01:57:01     INFO -    #09  pc 0000cbd4  /system/lib/libc.so (pthread_create+208)
[task 2018-12-20T01:57:01.241Z] 01:57:01     INFO -    at dalvik.system.NativeStart.run(Native Method)
[task 2018-12-20T01:57:01.241Z] 01:57:01     INFO -  "Binder_1" prio=5 tid=9 NATIVE
[task 2018-12-20T01:57:01.241Z] 01:57:01     INFO -    | group="main" sCount=1 dsCount=0 obj=0x41745ac0 self=0x2a04e188
[task 2018-12-20T01:57:01.242Z] 01:57:01     INFO -    | sysTid=1986 nice=0 sched=0/0 cgrp=apps handle=704800896
[task 2018-12-20T01:57:01.242Z] 01:57:01     INFO -    | state=S schedstat=( 15015247 12477313 15 ) utm=1 stm=0 core=0
[task 2018-12-20T01:57:01.242Z] 01:57:01     INFO -    #00  pc 0001b2d4  /system/lib/libc.so (__ioctl+8)
[task 2018-12-20T01:57:01.243Z] 01:57:01     INFO -    #01  pc 0002b94b  /system/lib/libc.so (ioctl+14)
[task 2018-12-20T01:57:01.243Z] 01:57:01     INFO -    #02  pc 0001b539  /system/lib/libbinder.so (android::IPCThreadState::talkWithDriver(bool)+140)
[task 2018-12-20T01:57:01.243Z] 01:57:01     INFO -    #03  pc 0001bcd7  /system/lib/libbinder.so (android::IPCThreadState::joinThreadPool(bool)+154)
[task 2018-12-20T01:57:01.244Z] 01:57:01     INFO -    #04  pc 0001faa5  /system/lib/libbinder.so
[task 2018-12-20T01:57:01.244Z] 01:57:01     INFO -    #05  pc 00011a75  /system/lib/libutils.so (android::Thread::_threadLoop(void*)+216)
[task 2018-12-20T01:57:01.244Z] 01:57:01     INFO -    #06  pc 0004b629  /system/lib/libandroid_runtime.so (android::AndroidRuntime::javaThreadShell(void*)+68)
[task 2018-12-20T01:57:01.245Z] 01:57:01     INFO -    #07  pc 00011569  /system/lib/libutils.so
[task 2018-12-20T01:57:01.245Z] 01:57:01     INFO -    #08  pc 0000ca58  /system/lib/libc.so (__thread_entry+72)
[task 2018-12-20T01:57:01.245Z] 01:57:01     INFO -    #09  pc 0000cbd4  /system/lib/libc.so (pthread_create+208)
[task 2018-12-20T01:57:01.246Z] 01:57:01     INFO -    at dalvik.system.NativeStart.run(Native Method)
[task 2018-12-20T01:57:01.246Z] 01:57:01     INFO -  "FinalizerWatchdogDaemon" daemon prio=5 tid=8 WAIT
[task 2018-12-20T01:57:01.246Z] 01:57:01     INFO -    | group="system" sCount=1 dsCount=0 obj=0x41742958 self=0x2a025ed8
[task 2018-12-20T01:57:01.247Z] 01:57:01     INFO -    | sysTid=1985 nice=0 sched=0/0 cgrp=apps handle=704799528
[task 2018-12-20T01:57:01.247Z] 01:57:01     INFO -    | state=S schedstat=( 1555273 66688213 11 ) utm=0 stm=0 core=0
[task 2018-12-20T01:57:01.247Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.248Z] 01:57:01     INFO -    - waiting on <0x414d16d8> (a java.lang.Daemons$FinalizerWatchdogDaemon)
[task 2018-12-20T01:57:01.248Z] 01:57:01     INFO -    at java.lang.Object.wait(Object.java:364)
[task 2018-12-20T01:57:01.248Z] 01:57:01     INFO -    at java.lang.Daemons$FinalizerWatchdogDaemon.waitForObject(Daemons.java:230)
[task 2018-12-20T01:57:01.248Z] 01:57:01     INFO -    at java.lang.Daemons$FinalizerWatchdogDaemon.run(Daemons.java:207)
[task 2018-12-20T01:57:01.249Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.249Z] 01:57:01     INFO -  "FinalizerDaemon" daemon prio=5 tid=7 WAIT
[task 2018-12-20T01:57:01.249Z] 01:57:01     INFO -    | group="system" sCount=1 dsCount=0 obj=0x417427a8 self=0x2a025638
[task 2018-12-20T01:57:01.250Z] 01:57:01     INFO -    | sysTid=1984 nice=0 sched=0/0 cgrp=apps handle=704797320
[task 2018-12-20T01:57:01.250Z] 01:57:01     INFO -    | state=S schedstat=( 89199640 183705304 63 ) utm=5 stm=3 core=0
[task 2018-12-20T01:57:01.250Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.251Z] 01:57:01     INFO -    - waiting on <0x414ba568> (a java.lang.ref.ReferenceQueue)
[task 2018-12-20T01:57:01.251Z] 01:57:01     INFO -    at java.lang.Object.wait(Object.java:401)
[task 2018-12-20T01:57:01.251Z] 01:57:01     INFO -    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:102)
[task 2018-12-20T01:57:01.252Z] 01:57:01     INFO -    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:73)
[task 2018-12-20T01:57:01.252Z] 01:57:01     INFO -    at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:170)
[task 2018-12-20T01:57:01.253Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.253Z] 01:57:01     INFO -  "ReferenceQueueDaemon" daemon prio=5 tid=6 WAIT
[task 2018-12-20T01:57:01.253Z] 01:57:01     INFO -    | group="system" sCount=1 dsCount=0 obj=0x41742640 self=0x2a045a70
[task 2018-12-20T01:57:01.254Z] 01:57:01     INFO -    | sysTid=1983 nice=0 sched=0/0 cgrp=apps handle=704929472
[task 2018-12-20T01:57:01.254Z] 01:57:01     INFO -    | state=S schedstat=( 10980371 18219718 42 ) utm=0 stm=1 core=0
[task 2018-12-20T01:57:01.254Z] 01:57:01     INFO -    at java.lang.Object.wait(Native Method)
[task 2018-12-20T01:57:01.255Z] 01:57:01     INFO -    - waiting on <0x414ba490>
[task 2018-12-20T01:57:01.255Z] 01:57:01     INFO -    at java.lang.Object.wait(Object.java:364)
[task 2018-12-20T01:57:01.255Z] 01:57:01     INFO -    at java.lang.Daemons$ReferenceQueueDaemon.run(Daemons.java:130)
[task 2018-12-20T01:57:01.255Z] 01:57:01     INFO -    at java.lang.Thread.run(Thread.java:841)
[task 2018-12-20T01:57:01.256Z] 01:57:01     INFO -  "Compiler" daemon prio=5 tid=5 VMWAIT

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/ccc4c538f5135129eafaec791ebd41f8f8701152
Flags: needinfo?(andrew)
Are we really sure this is the fault of this patch?  I saw this on try (https://treeherder.mozilla.org/#/jobs?repo=try&revision=54bb935b6bdae47fc757fa87df3eec4b2ae66bc8&selectedJob=217827799) and felt it looked like pre-existing flakiness.  There's nothing here that's likely to impact OMTA, although it is of course possible, since this does impact link highlighting.
Flags: needinfo?(rmaries)
(In reply to Andreea Pavel [:apavel] from comment #82)
> Nick: yes. job below this push are green, on the push are perma-failing. 
> 
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> inbound&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexce
> ption&group_state=expanded&searchStr=android%2C4.
> 3%2Capi16%2B%2Copt%2Cmochitests%2Ctest-android-em-4.3-arm7-api-16%2Fopt-
> mochitest-22%2Cm%2822%29&selectedJob=218029827

Many thanks!  We'll investigate.
You need to log in before you can comment on or make changes to this bug.