1170981 - ./mach mochitest stuck at http://mochi.test:8888/redirect.html

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Kan-Ru Chen [:kanru] (UTC+9)]

STR:

  revision 14be4237f855368ba4132f3af4152c0e6ce0737d
  ./mach build
  ./mach mochitest

The test window stuck at "redirecting..." and then timed out

I tested this on two Debian stable box and both are reproducible.

Comment 1

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: screenshot

Comment 2

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: log.txt

Comment 3

[(not currently active) Ted Mielczarek]

I'm guessing this is stuck running mochitest-chrome for some reason. I would guess it's fallout from one of ahal's recent changes.

Comment 4

[Andrew Halberstadt [:ahal]]

Strange, it looks like the mach formatter is being used even though the tbpl formatter should be the default.

Can you paste the full output of |mach mochitest --log-tbpl -|? Also what happens if you run the specific suite with |mach mochitest -f browser|?

Comment 5

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: log2.txt

(In reply to Andrew Halberstadt [:ahal] from comment #4)
> Strange, it looks like the mach formatter is being used even though the tbpl
> formatter should be the default.

I thought I should use --log-mach...

> Can you paste the full output of |mach mochitest --log-tbpl -|? Also what
> happens if you run the specific suite with |mach mochitest -f browser|?

Log attached. |mach mochitest -f browser| stuck too and the log looks identical.

Comment 6

[Kan-Ru Chen [:kanru] (UTC+9)]

Hummm.. wrong log. Somehow running this from remote starts crashing the browser.

Comment 7

[Andrew Halberstadt [:ahal]]

(In reply to Kan-Ru Chen [:kanru] from comment #5)
> I thought I should use --log-mach...

Yeah, the mach logger is good for running locally, it just doesn't paste into text attachments very well :).

I don't think this is a regression from the patch I recently landed because there is no "Now running mochitest-<flavor>" message in the log. What branch are you trying to run these on? If central/inbound, does pulling in the latest changes help?

Comment 8

[Kan-Ru Chen [:kanru] (UTC+9)]

I bisected it. The first bad commit is http://hg.mozilla.org/mozilla-central/rev/622af20a746a5030dffa04bb741856cff5464e56

I don't know how but reverting it does fix this issue for me.

Comment 9

[Benjamin Smedberg]

I have no theory why this would be happening.

When you say it's stuck, is the window responding just not doing the network load, or is it actually deadlocked?

Comment 10

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #9)
> I have no theory why this would be happening.
> 
> When you say it's stuck, is the window responding just not doing the network
> load, or is it actually deadlocked?

The window is responding. Just the redirect.html doesn't proceed to actually run the tests.

Comment 11

[Kan-Ru Chen [:kanru] (UTC+9)]

I think the crucial part is |let storage = Services.storage;| now will not always run before NSS initialization so bug 717904 is back. If I put |let storage = Services.storage;| to the original position then everything is normal again. We need a batter way to ensure the initialization sequence is correct.

Comment 12

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: MozReview Request: Bug 1170981 - Load the storage service before NSS

Bug 1170981 - Load the storage service before NSS

Comment 13

[Kan-Ru Chen [:kanru] (UTC+9)]

Comment on attachment 8617146 [details]
MozReview Request: Bug 1170981 - Load the storage service before NSS

Bug 1170981 - Load the storage service before NSS

Comment 14

[Kan-Ru Chen [:kanru] (UTC+9)]

Marco, could you also take a look at bug 717904. I think we need to make sure the sqlite database is initialized in correct order. Also, do you know how to test this?

Comment 15

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Comment on attachment 8617146 [details]
MozReview Request: Bug 1170981 - Load the storage service before NSS

https://reviewboard.mozilla.org/r/10557/#review9589

Ship It!

Comment 16

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Kan-Ru Chen [:kanru] from comment #14)
> Marco, could you also take a look at bug 717904. I think we need to make
> sure the sqlite database is initialized in correct order. Also, do you know
> how to test this?

For an add-on manager test you'd better ask Mossop, I don't know much about that testing code.
I am aware of bug 717904, I participated to IRC discussions about it and we filed bug 730495 to find a way to ensure proper initialization. That bug actually has the solution explained in the comments, but nobody implemented it yet.
We should basically compile internal versions of SQLite with an additional define, while system Sqlite should be unaffected cause it doesn't use jemalloc.

Comment 17

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Marco Bonardo [::mak] from comment #16)
> (In reply to Kan-Ru Chen [:kanru] from comment #14)
> > Marco, could you also take a look at bug 717904. I think we need to make
> > sure the sqlite database is initialized in correct order. Also, do you know
> > how to test this?
> 
> For an add-on manager test you'd better ask Mossop, I don't know much about
> that testing code.
> I am aware of bug 717904, I participated to IRC discussions about it and we
> filed bug 730495 to find a way to ensure proper initialization. That bug
> actually has the solution explained in the comments, but nobody implemented
> it yet.
> We should basically compile internal versions of SQLite with an additional
> define, while system Sqlite should be unaffected cause it doesn't use
> jemalloc.

Thanks, Marco. I'll try to fix the root cause.

Comment 18

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/61e83526454d

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/61e83526454d