1646173 - No database found error for Glean Python SDK version 31.1.0 and newer on Linux

Status: RESOLVED FIXED

(Data Platform and Tools :: Glean: SDK, defect, P1)

Description

[Raphael Aurich [:raphael] UTC+02:00]

Glean Python SDK

I cannot reproduce the error in versions 29.0.0, 30.0.0, 31.0.0, but version 31.1.0 and 31.1.1 panic.

glean_sdk-31.1.0-cp35-abi3-manylinux1_x86_64.whl
glean_sdk-31.1.1-cp35-abi3-manylinux1_x86_64.whl

Log messages

I'm seeing the following error when submitting pings from Python on Linux slim-buster:

cli2        | DEBUG:burnham.missions:Starting mission 'MISSION A: ONE WARP'
cli2        | DEBUG:burnham.space_travel:Warp to abcdefgh using space-travel technology
cli2        | DEBUG:burnham.missions:Completed mission 'MISSION A: ONE WARP'
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION A: ONE WARP'
cli2        | DEBUG:burnham.missions:Starting mission 'MISSION B: TWO WARPS'
cli2        | DEBUG:burnham.space_travel:Warp to 26.2 using space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to abc using space-travel technology
cli2        | DEBUG:burnham.missions:Completed mission 'MISSION B: TWO WARPS'
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION B: TWO WARPS'
cli2        | DEBUG:burnham.missions:Starting mission 'MISSION D: TWO JUMPS'
cli2        | DEBUG:burnham.space_travel:Jump to 2016 using experimental space-travel technology
cli2        | DEBUG:burnham.space_travel:Jump to Berlin using experimental space-travel technology
cli2        | DEBUG:burnham.missions:Completed mission 'MISSION D: TWO JUMPS'
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION D: TWO JUMPS'
cli2        | DEBUG:burnham.missions:Starting mission 'MISSION E: ONE JUMP, ONE METRIC ERROR'
cli2        | DEBUG:burnham.space_travel:Jump to Starbase 46 using experimental space-travel technology
cli2        | DEBUG:burnham.missions:Error completing mission 'MISSION E: ONE JUMP, ONE METRIC ERROR': INCOMPLETE NAVIGATION SEQUENCE abcdabcdabcdabcdabcdabcd123412341234123412341234abcdabcdabcdabcdabcdabcd123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234abcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcd
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION E: ONE JUMP, ONE METRIC ERROR'
cli2        | DEBUG:burnham.missions:Starting mission 'MISSION F: TWO WARPS, ONE JUMP'
cli2        | DEBUG:burnham.space_travel:Warp to abc using space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to de using space-travel technology
cli2        | DEBUG:burnham.space_travel:Jump to 12345 using experimental space-travel technology
cli2        | DEBUG:burnham.missions:Completed mission 'MISSION F: TWO WARPS, ONE JUMP'
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION F: TWO WARPS, ONE JUMP'
cli2        | DEBUG:burnham.missions:Starting mission 'MISSION G: FIVE WARPS, FOUR JUMPS'
cli2        | DEBUG:burnham.space_travel:Jump to 1234 using experimental space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to abcd using space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to ab using space-travel technology
cli2        | DEBUG:burnham.space_travel:Jump to 8000 using experimental space-travel technology
cli2        | DEBUG:burnham.space_travel:Jump to 20 using experimental space-travel technology
cli2        | DEBUG:burnham.space_travel:Jump to 200 using experimental space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to home using space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to 4000 using space-travel technology
cli2        | DEBUG:burnham.space_travel:Warp to abc123 using space-travel technology
cli2        | DEBUG:burnham.missions:Completed mission 'MISSION G: FIVE WARPS, FOUR JUMPS'
cli2        | DEBUG:burnham.missions:Submitting ping for mission 'MISSION G: FIVE WARPS, FOUR JUMPS'
cli2        | ERROR:glean._dispatcher:Timeout sending Glean telemetry
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found
cli2        | thread '<unnamed>' panicked at 'No database found', glean-core/src/lib.rs:449:10
cli2        | [2020-06-16T20:18:31Z ERROR ffi_support::error] Caught a panic calling rust code: "No database found"
cli2        | [2020-06-16T20:18:31Z ERROR glean_ffi::handlemap_ext] Glean failed (ErrorCode(-1)): No database found

Steps to reproduce:

Clone https://github.com/hackebrot/burnham and run:

docker-compose up --build

Comment 1

[William Lachance (:wlach)]

Not sure if this is related but there has been no reported mozregression usage on mach (which I gather would be using the latest glean sdk, unlike the GUI which uses a hardcoded version-- 30.1.0 currently) since May 27:

https://sql.telemetry.mozilla.org/queries/70610#177730

Comment 2

[Michael Droettboom [:mdroettboom]]

The key thing here is this:

ERROR:glean._dispatcher:Timeout sending Glean telemetry

After that point, the Glean database gets taken down, and then all queued telemetry tasks after that complain about "No database found".

(I was only able to get this to occur when the networking between the application container and the platform container was broken by my dev machine's firewall. That doesn't seem to be happening here, but perhaps there is something else being slow in :raphael's environment above).

I think the fix here is to increase the timeout on running tasks when Glean shuts down -- now that we have an upper bound on the time waiting for network problems in the uploader anyway, I'm no longer concerned with that wait running forever.

This also exposed a smaller unrelated bug that flushing tasks could end up adding preinit tasks to run after Glean initialize runs.

:wlach -- I'm afraid this doesn't feel related to what you are seeing. Separated that out into bug 1646402.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/glean/pull/983

Comment 4

[Michael Droettboom [:mdroettboom]]

This is released in v31.1.2