intermittent internal error: "file error - nav_link: not found" (also manifests as fields_lhs: not found)

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
General
--
major
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: emorley, Assigned: dylan)

Tracking

Production

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

3 years ago
1) Visit https://bugzilla.mozilla.org/buglist.cgi?quicksearch=P1%20%3Atreeherder&list_id=12480992
2) Click on any of the bug rows returned, to visit the bug

-> Internal Error: "file error - nav_link: not found"

Doesn't occur if visiting the bug directly; presume due to the previous/next bug navigation thing that is shown when a search has been used. I'm using the modal bug UI.
Hmm this doesn't happen for me and I am using the modal view as well.
(Reporter)

Comment 2

3 years ago
Is list_id specific to me?
Can you repro if you create your own search?
Nope. Tried it with my own list_id and worked:

https://bugzilla.mozilla.org/buglist.cgi?quicksearch=P1%20%3Atreeherder&list_id=12481251

Also created a saved search and re-ran it from the footer without issue.

dkl
i've spent some time trying to track this one down.

according to template-toolkit it means that the nav_link block isn't found in the compiled template, so it's looking for a file.  weird thing is if i look at the actual compiled template the block is there.

nothing environmental or code changed that would impact this when it started happening, and it's an infrequent intermittent.
Assignee: nobody → glob
Summary: Internal Error: "file error - nav_link: not found" when visiting bugs from search results → intermittent internal error: "file error - nav_link: not found" (also manifests as fields_lhs: not found)
Duplicate of this bug: 1232489

Comment 6

2 years ago
I see this permanently when trying to view bug 1209095. It works fine in a private window or in Chrome; presumably it's something related to cookies in my active profile?

"file error - fields_lhs: not found"
(In reply to Josh Matthews [:jdm] from comment #6)
> I see this permanently when trying to view bug 1209095. It works fine in a
> private window or in Chrome; presumably it's something related to cookies in
> my active profile?
> 
> "file error - fields_lhs: not found"

Based on when I've seen this and when others have reported it, I think it happens if the bug is part of a bug list, and it's been in an unrestored tab for a while (read: I haven't loaded it in some time). Maybe it's possible if there's cookie information that indicates the bug is part of a list, but the list has been removed? (I'm assuming the cached lists of bugs get nuked after some period of time) Glob, is that plausible?
Flags: needinfo?(glob)
(In reply to :Gijs Kruitbosch from comment #7)
> [snip] Glob, is that plausible?

unlikely.  it's the template engine itself that's throwing this error; it's saying that after compiling the template it's unable to find the sub for the fields_lhs block.  however if you examine the compiled template it's present.
Flags: needinfo?(glob)

Comment 9

2 years ago
Created attachment 8736322 [details]
curl command to reproduce the internal error

I used the 'copy as cURL' feature of the network devtools, and the result is an internal error page.
FTR, after logging out and logging in again I can't reproduce the previous error in the browser. Haven't tried cURL.
Assignee: glob → dylan
(Assignee)

Comment 11

2 years ago
Thanks glob. The problem is related to this code:

https://github.com/mozilla/webtools-bmo-bugzilla/blob/master/Bugzilla/User.pm#L769-L782

Should be able to repro by setting the referer to buglist.cgi with random non-existing list_id values.
(Assignee)

Comment 12

2 years ago
I haven't run any code yet, but looking around, check_quietly() is throwing an error at a distance --

https://github.com/mozilla/webtools-bmo-bugzilla/blob/master/Bugzilla/Search/Recent.pm#L103-L110

that should be fine, except we have a die handler (from Bugzilla::Sentry) sees an exception come through, it eventually calls ThrowTemplateError() which calls exit(), which is just another exception... it's unclear what happens from there, but I'll probably be able to fix it soon, possibly along with Bug 1216300.

Also, what the actual help. We can't just parse the referer header! WHAT WHAT WHAT.

That is all.
(Assignee)

Comment 13

2 years ago
As a note, why we see this coming from the templates is that the stack-walking code that sentry uses gets confused.
(Assignee)

Comment 14

2 years ago
Created attachment 8738830 [details] [diff] [review]
1195736_1.patch

Alright, this patch makes it so we don't install a __DIE__ handler under mod_perl. instead, we capture the error in the log_error() handler method, sent it off to sentry, and store it in a dynamically scoped variable. We then read out the exception in the main handler code. If we've already sent stuff, we do nothing (because that's the safest thing to do), otherwise we throw a generic error (ThrowTemplateError).
Attachment #8738830 - Flags: review?(dkl)
(Assignee)

Comment 15

2 years ago
It turns out this doesn't entirely address the issue -- and the root cause is a little more weird :-)

Template Toolkit is not re-entrant, we can't call into $template->process while doing a $template->process.

This bug was actually introduced by https://github.com/mozilla/webtools-bmo-bugzilla/commit/01076cee38eb2a5b806e62e62c945cd2c71bfae1. However the fix for it is applicable to upstream bugzilla as well:
during $template->process(), error_mode should always be ERROR_MODE_DIE.
(Assignee)

Comment 16

2 years ago
Created attachment 8738850 [details] [diff] [review]
1195736_2.patch

Alright, this fixes recursive calls to template->process, at least from error handling routines.
Attachment #8738830 - Attachment is obsolete: true
Attachment #8738830 - Flags: review?(dkl)
Attachment #8738850 - Flags: review?(dkl)
(Assignee)

Comment 17

2 years ago
I moved the other changes to bug 1262676.
(Assignee)

Comment 18

2 years ago
For testing this, you can use RefControl (https://addons.mozilla.org/en-US/firefox/addon/refcontrol/) to set a Refer containing an invalid list_id. Something like http://bugzilla.vm/bmo/buglist.cgi?quicksearch=%40dylan&list_id=1650000 should do
(Assignee)

Updated

2 years ago
Blocks: 1263244
Comment on attachment 8738850 [details] [diff] [review]
1195736_2.patch

Review of attachment 8738850 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl
Attachment #8738850 - Flags: review?(dkl) → review+
(Assignee)

Comment 20

2 years ago
To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   cc0c32b..33f6155  master -> master
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 21

2 years ago
Reverting due to test failures

To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   197afbe..49293a6  master -> master
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 22

2 years ago
exceptions raised inside templates are always caught, and they get stored in $template->error
to fix our recursive/re-entrancy problem with handling errors we can add a new error mode: ERROR_MODE_TEMPLATE
and before calling $template->SUPER::process() in Bugzilla::Template->process(), we set that error mode
we then add code to ThrowUserError and ThrowCodeError so those construct Template::Exception objects
it turns out (and this what made me happy) that we always pass $template->error to ThrowTemplateError
so I just have to add code to ThrowTemplateError() to match on TT2 errors from bugzilla and pass them back to ThrowUserError/ThrowCodeError for formatting
(Assignee)

Comment 23

2 years ago
This means you can catch bugzilla errors in templates as well,
[% TRY %]
[% some_janky_code %]
[% CATCH bugzilla.user.account_locked %]
...
[% END %]
(Assignee)

Comment 24

2 years ago
Created attachment 8743032 [details] [diff] [review]
1195736_5.patch
Attachment #8738850 - Attachment is obsolete: true
Attachment #8743032 - Flags: review?(dkl)
Comment on attachment 8743032 [details] [diff] [review]
1195736_5.patch

Review of attachment 8743032 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl. Fixes the issue for me.
Attachment #8743032 - Flags: review?(dkl) → review+
(Assignee)

Comment 26

2 years ago
To ssh://gitolite3@git.mozilla.org/webtools/bmo/bugzilla.git
   dab15f5..dacce0f  master -> master
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.