1195736 - intermittent internal error: "file error - nav_link: not found" (also manifests as fields_lhs: not found)

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[Ed Morley [:emorley]]

1) Visit https://bugzilla.mozilla.org/buglist.cgi?quicksearch=P1%20%3Atreeherder&list_id=12480992
2) Click on any of the bug rows returned, to visit the bug

-> Internal Error: "file error - nav_link: not found"

Doesn't occur if visiting the bug directly; presume due to the previous/next bug navigation thing that is shown when a search has been used. I'm using the modal bug UI.

Comment 1

[David Lawrence [:dkl]]

Hmm this doesn't happen for me and I am using the modal view as well.

Comment 2

[Ed Morley [:emorley]]

Is list_id specific to me?
Can you repro if you create your own search?

Comment 3

[David Lawrence [:dkl]]

Nope. Tried it with my own list_id and worked:

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=P1%20%3Atreeherder&list_id=12481251

Also created a saved search and re-ran it from the footer without issue.

dkl

Comment 4

[:glob ✱]

i've spent some time trying to track this one down.

according to template-toolkit it means that the nav_link block isn't found in the compiled template, so it's looking for a file.  weird thing is if i look at the actual compiled template the block is there.

nothing environmental or code changed that would impact this when it started happening, and it's an infrequent intermittent.

Comment 6

[Josh Matthews [:jdm]]

I see this permanently when trying to view bug 1209095. It works fine in a private window or in Chrome; presumably it's something related to cookies in my active profile?

"file error - fields_lhs: not found"

Comment 7

[:Gijs (he/him)]

(In reply to Josh Matthews [:jdm] from comment #6)
> I see this permanently when trying to view bug 1209095. It works fine in a
> private window or in Chrome; presumably it's something related to cookies in
> my active profile?
> 
> "file error - fields_lhs: not found"

Based on when I've seen this and when others have reported it, I think it happens if the bug is part of a bug list, and it's been in an unrestored tab for a while (read: I haven't loaded it in some time). Maybe it's possible if there's cookie information that indicates the bug is part of a list, but the list has been removed? (I'm assuming the cached lists of bugs get nuked after some period of time) Glob, is that plausible?

Comment 8

[:glob ✱]

(In reply to :Gijs Kruitbosch from comment #7)
> [snip] Glob, is that plausible?

unlikely.  it's the template engine itself that's throwing this error; it's saying that after compiling the template it's unable to find the sub for the fields_lhs block.  however if you examine the compiled template it's present.

Comment 9

[Josh Matthews [:jdm]]

Attachment: curl command to reproduce the internal error

I used the 'copy as cURL' feature of the network devtools, and the result is an internal error page.

Comment 10

[Josh Matthews [:jdm]]

FTR, after logging out and logging in again I can't reproduce the previous error in the browser. Haven't tried cURL.

Comment 11

[Dylan Hardison [:dylan] (he/him)]

Thanks glob. The problem is related to this code:

https://github.com/mozilla/webtools-bmo-bugzilla/blob/master/Bugzilla/User.pm#L769-L782

Should be able to repro by setting the referer to buglist.cgi with random non-existing list_id values.

Comment 12

[Dylan Hardison [:dylan] (he/him)]

I haven't run any code yet, but looking around, check_quietly() is throwing an error at a distance --

https://github.com/mozilla/webtools-bmo-bugzilla/blob/master/Bugzilla/Search/Recent.pm#L103-L110

that should be fine, except we have a die handler (from Bugzilla::Sentry) sees an exception come through, it eventually calls ThrowTemplateError() which calls exit(), which is just another exception... it's unclear what happens from there, but I'll probably be able to fix it soon, possibly along with Bug 1216300.

Also, what the actual help. We can't just parse the referer header! WHAT WHAT WHAT.

That is all.

Comment 13

[Dylan Hardison [:dylan] (he/him)]

As a note, why we see this coming from the templates is that the stack-walking code that sentry uses gets confused.

Comment 14

[Dylan Hardison [:dylan] (he/him)]

Attachment: 1195736_1.patch

Alright, this patch makes it so we don't install a __DIE__ handler under mod_perl. instead, we capture the error in the log_error() handler method, sent it off to sentry, and store it in a dynamically scoped variable. We then read out the exception in the main handler code. If we've already sent stuff, we do nothing (because that's the safest thing to do), otherwise we throw a generic error (ThrowTemplateError).

Comment 15

[Dylan Hardison [:dylan] (he/him)]

It turns out this doesn't entirely address the issue -- and the root cause is a little more weird :-)

Template Toolkit is not re-entrant, we can't call into $template->process while doing a $template->process.

This bug was actually introduced by https://github.com/mozilla/webtools-bmo-bugzilla/commit/01076cee38eb2a5b806e62e62c945cd2c71bfae1. However the fix for it is applicable to upstream bugzilla as well:
during $template->process(), error_mode should always be ERROR_MODE_DIE.

Comment 16

[Dylan Hardison [:dylan] (he/him)]

Attachment: 1195736_2.patch

Alright, this fixes recursive calls to template->process, at least from error handling routines.

Comment 17

[Dylan Hardison [:dylan] (he/him)]

I moved the other changes to bug 1262676.

Comment 18

[Dylan Hardison [:dylan] (he/him)]

For testing this, you can use RefControl (https://addons.mozilla.org/en-US/firefox/addon/refcontrol/) to set a Refer containing an invalid list_id. Something like http://bugzilla.vm/bmo/buglist.cgi?quicksearch=%40dylan&list_id=1650000 should do

Comment 19

[David Lawrence [:dkl]]

Comment on attachment 8738850 [details] [diff] [review]
1195736_2.patch

Review of attachment 8738850 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl

Comment 20

[Dylan Hardison [:dylan] (he/him)]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   cc0c32b..33f6155  master -> master

Comment 21

[Dylan Hardison [:dylan] (he/him)]

Reverting due to test failures

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   197afbe..49293a6  master -> master

Comment 22

[Dylan Hardison [:dylan] (he/him)]

exceptions raised inside templates are always caught, and they get stored in $template->error
to fix our recursive/re-entrancy problem with handling errors we can add a new error mode: ERROR_MODE_TEMPLATE
and before calling $template->SUPER::process() in Bugzilla::Template->process(), we set that error mode
we then add code to ThrowUserError and ThrowCodeError so those construct Template::Exception objects
it turns out (and this what made me happy) that we always pass $template->error to ThrowTemplateError
so I just have to add code to ThrowTemplateError() to match on TT2 errors from bugzilla and pass them back to ThrowUserError/ThrowCodeError for formatting

Comment 23

[Dylan Hardison [:dylan] (he/him)]

This means you can catch bugzilla errors in templates as well,
[% TRY %]
[% some_janky_code %]
[% CATCH bugzilla.user.account_locked %]
...
[% END %]

Comment 24

[Dylan Hardison [:dylan] (he/him)]

Attachment: 1195736_5.patch

Comment 25

[David Lawrence [:dkl]]

Comment on attachment 8743032 [details] [diff] [review]
1195736_5.patch

Review of attachment 8743032 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl. Fixes the issue for me.

Comment 26

[Dylan Hardison [:dylan] (he/him)]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   dab15f5..dacce0f  master -> master