1337984 - time restricted search by Bug creation date (e.g. last X days/hours) doesn't find all the latest bugs

Status: NEW

(bugzilla.mozilla.org :: Search, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://bugzilla.mozilla.org/buglist.cgi?chfield=[Bug%20creation]&chfieldfrom=48h&short_desc_type=regexp&short_desc=^[^Intermittent]&chfieldto=Now&query_format=advanced&list_id=13434740
is a search for bugs created in the last 48 hours excluding the ones about intermittent failures. It is sorted ascending by bug number, the last bugs listed are:
bug 1337969
bug 1337970
bug 1337977

Bug bug 1337971 is e.g. accessible for me and doesn't look restricted.

A search for only the bugs filed in the last hour lists more: https://bugzilla.mozilla.org/buglist.cgi?chfieldto=Now&list_id=13434739&chfieldfrom=1h&chfield=[Bug%20creation]&query_format=advanced

1337971
1337972
1337973
1337974
1337975
1337976
1337977
1337978
1337980
1337981
1337982

Comment 1

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Your 48h query returned exactly 500 bugs, which is the limit on the number of bugs we can return in a query, so there may have been more than 500 bugs matching your query filed during that window, and when you searched for bugs filed in the past hour, you got bugs that weren't in the first 500 that matched the 48 hours query. 

If you're looking to search for these bugs programmatically with the API, I recommend something similar to what I do to find untriaged bugs.

"""
    // This recursively fetches all the open bugs in Firefox related components opened since June 1st, 2016
    // which don't have a pending needinfo, and are not in the general and untriaged components
    // this does not include security filtered bugs 

    function getBugs(last) {
        var newLast;
        fetch(baseAPIRequest 
            + "&product=Core&product=Firefox&product=Firefox%20for%20Android&product=Firefox%20for%20iOS&product=Toolkit&chfieldfrom="
            + "2016-06-01&chfieldto=NOW&v4=" + last)
            .then(function(response) { // $DEITY, I can't wait for await 
                if (response.ok) {  
                    response.json()
                    .then(function(data) {
                        newLast = data.bugs[data.bugs.length - 1].id;
                        /* 
                            There are two ways we can fall out of this recursion: if the total
                            number of bugs is evenly divisible by limit (edge case) then we'll 
                            err on fetching a result set twice, but not adding it, or if the number
                            of bugs in the batch returned is less than the limit, we'll add the last
                            batch and stop 
                        */
                        if (newLast != last) {
                            completed ++;
                            console.log("completed", completed, "fetches");
                            Array.prototype.push.apply(result.bugs, data.bugs); // call push on each result.bugs
                            if (data.bugs.length === limit) {
                                console.log("calling again with last", newLast);
                                getBugs(newLast); // recursively call using the id of the last bug in the results as last                               
                            } else {
                                console.log("less bugs than limit");
                                complete();
                            }
                        } else {
                            console.log("edge case");
                            complete();
                        }
                    });
                }
            });
    }
"""

Let me know if this solves your problem or if I need to do further digging.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

> Your 48h query returned exactly 500 bugs.
I am pretty sure it listed only 471 bugs. It still can be reproduced, e.g. https://bugzilla.mozilla.org/buglist.cgi?list_id=13435562&chfield=[Bug%20creation]&chfieldfrom=53h&short_desc_type=regexp&short_desc=^[^Intermittent]&chfieldto=Now&query_format=advanced lists 494 bugs at the moment, with bug 1338116 as the latest, but bug 1338117 is already 16 minutes old.

Thank you for the script.

Comment 3

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

The error :aryx described is still present.