1966346 - Rest API returns duplicate bug IDs resulting in missing bugs

Status: ASSIGNED

(bugzilla.mozilla.org :: API, defect, P1)

Description

[Teal Dulcet [:tdulcet] (TB Council)]

Steps to Reproduce:

1. Run this Python script to download the Thunderbird related bugs with the Bugzilla Rest API: https://github.com/tdulcet/Thunderbird-Metrics/blob/main/bugzilla.py
   1. Run: python3 -m pip install requests matplotlib and python3 bugzilla.py

The script makes an API call to similar to: https://bugzilla.mozilla.org/rest/bug?product=Thunderbird&product=MailNews%20Core&product=Calendar&product=Chat%20Core&limit=1000&offset=0 and then pages through the results to download all of the matching bugs. It outputs the list of duplicate bug IDs after each API call.

Expected Results:

All Thunderbird related bugs should be downloaded, without any duplicate or missing bug IDs.

Actual Results:

Exactly 1,000 duplicate bug IDs are returned, resulting in some missing bug IDs (presumably also 1,000). The total number of duplicates is the same as the value of the limit parameter, but the duplicates are returned across two separate API calls when paging through the results. I have run the script several times over the past month and the issue is 100% reproducible.

Comment 1

[Teal Dulcet [:tdulcet] (TB Council)]

For reference, here are the 1,000 duplicate bug IDs:

        Offset 93000
Warning: Duplicate Bug ids: 784 (953, 1369, 2654, 2662, 2664, 2665, 2667, 2671, 2678, 2679, 2726, 2730, 2731, 2734, 2767, 2769, 2862, 2890, 2891, 2892, 2893, 2894, 2895, 2897, 2901, 2903, 2908, 2910, 2911, 2912, 2914, 2915, 2916, 2918, 2919, 2922, 2933, 2935, 2937, 3076, 3157, 3184, 3210, 3459, 3744, 3745, 3746, 3747, 3754, 3764, 3766, 3770, 3789, 3832, 3834, 3865, 3867, 3889, 3903, 3904, 3914, 3941, 3966, 3979, 3994, 4029, 4031, 4037, 4083, 4165, 4166, 4246, 4388, 4393, 4394, 4415, 4425, 4452, 4453, 4454, 4465, 4469, 4505, 4561, 4567, 4585, 4688, 4708, 4720, 4723, 4731, 4732, 4763, 4784, 4840, 4843, 4845, 4846, 4847, 4851, 4861, 4862, 4886, 4896, 4898, 4923, 4925, 4961, 4963, 4968, 4970, 4977, 4991, 5011, 5061, 5063, 5235, 5262, 5265, 5267, 5280, 5289, 5315, 5323, 5326, 5328, 5331, 5341, 5351, 5365, 5384, 5440, 5446, 5449, 5452, 5467, 5468, 5474, 5477, 5492, 5493, 5502, 5503, 5510, 5513, 5521, 5525, 5533, 5561, 5572, 5574, 5581, 5587, 5627, 5633, 5655, 5660, 5662, 5668, 5681, 5719, 5722, 5736, 5752, 5753, 5760, 5765, 5781, 5792, 5804, 5808, 5810, 5811, 5814, 5866, 5867, 5873, 5883, 5893, 5894, 5909, 5913, 5922, 5930, 5933, 5934, 5938, 5939, 5965, 5988, 5991, 6031, 6032, 6039, 6047, 6099, 6175, 6176, 6214, 6216, 6219, 6243, 6244, 6281, 6286, 6315, 6319, 6338, 6340, 6352, 6375, 6402, 6405, 6406, 6412, 6413, 6426, 6444, 6448, 6459, 6469, 6485, 6496, 6527, 6532, 6584, 6619, 6626, 6651, 6652, 6653, 6655, 6657, 6660, 6670, 6671, 6672, 6675, 6680, 6686, 6691, 6698, 6699, 6722, 6730, 6735, 6740, 6780, 6790, 6791, 6794, 6823, 6862, 6866, 6873, 6878, 6883, 6889, 6890, 6904, 6906, 6908, 6912, 6914, 6919, 6929, 6931, 6935, 6961, 6975, 6983, 6995, 6996, 7002, 7004, 7010, 7028, 7092, 7098, 7130, 7145, 7147, 7161, 7163, 7176, 7205, 7206, 7227, 7272, 7273, 7313, 7321, 7329, 7331, 7332, 7419, 7420, 7425, 7426, 7429, 7430, 7432, 7435, 7443, 7444, 7448, 7478, 7479, 7501, 7504, 7508, 7511, 7513, 7518, 7519, 7555, 7557, 7560, 7573, 7575, 7580, 7611, 7613, 7618, 7623, 7640, 7654, 7715, 7728, 7741, 7752, 7770, 7772, 7778, 7789, 7807, 7808, 7810, 7811, 7812, 7813, 7814, 7815, 7836, 7844, 7880, 7883, 7891, 7911, 7912, 7945, 7995, 8029, 8030, 8038, 8053, 8054, 8075, 8076, 8114, 8117, 8124, 8126, 8128, 8132, 8133, 8135, 8143, 8145, 8165, 8179, 8181, 8182, 8202, 8203, 8204, 8206, 8207, 8208, 8210, 8213, 8216, 8219, 8226, 8246, 8276, 8315, 8332, 8343, 8344, 8352, 8362, 8395, 8402, 8405, 8412, 8418, 8455, 8457, 8460, 8461, 8462, 8464, 8481, 8483, 8504, 8505, 8508, 8510, 8515, 8554, 8570, 8572, 8644, 8675, 8678, 8692, 8699, 8706, 8707, 8717, 8718, 8729, 8755, 8761, 8768, 8770, 8778, 8779, 8786, 8792, 8793, 8810, 8813, 8815, 8822, 8848, 8854, 8857, 8889, 8890, 8891, 8899, 8903, 8942, 8958, 8965, 8973, 8985, 9019, 9021, 9044, 9048, 9079, 9081, 9094, 9105, 9112, 9138, 9139, 9145, 9159, 9162, 9168, 9170, 9173, 9202, 9209, 9218, 9225, 9229, 9231, 9309, 9350, 9364, 9398, 9413, 9417, 9455, 9462, 9470, 9482, 9483, 9484, 9487, 9490, 9504, 9508, 9512, 9514, 9535, 9537, 9539, 9540, 9555, 9557, 9704, 9705, 9707, 9714, 9736, 9737, 9761, 9762, 9776, 9782, 9791, 9801, 9836, 9882, 9887, 9900, 9944, 9968, 9970, 9982, 10012, 10034, 10035, 10043, 10081, 10091, 10097, 10126, 10127, 10130, 10136, 10149, 10168, 10170, 10232, 10262, 10270, 10285, 10294, 10295, 10309, 10328, 10344, 10348, 10350, 10352, 10353, 10367, 10376, 10381, 10382, 10400, 10416, 10419, 10426, 10454, 10472, 10473, 10475, 10500, 10504, 10519, 10539, 10543, 10583, 10586, 10591, 10594, 10605, 10608, 10617, 10643, 10660, 10670, 10672, 10701, 10708, 10773, 10791, 10800, 10801, 10802, 10803, 10804, 10813, 10823, 10827, 10833, 10834, 10842, 10845, 10854, 10855, 10857, 10858, 10859, 10861, 10884, 10885, 10886, 10887, 10892, 10908, 10925, 10932, 10933, 10934, 10940, 10942, 10960, 10965, 10968, 10970, 10971, 10974, 11013, 11025, 11032, 11033, 11034, 11035, 11036, 11037, 11038, 11039, 11040, 11041, 11042, 11043, 11044, 11045, 11046, 11047, 11048, 11049, 11050, 11051, 11054, 11055, 11056, 11057, 11075, 11076, 11079, 11080, 11082, 11094, 11116, 11120, 11144, 11153, 11172, 11175, 11188, 11221, 11277, 11302, 11316, 11334, 11335, 11336, 11339, 11350, 11361, 11366, 11443, 11464, 11471, 11485, 11492, 11493, 11496, 11497, 11498, 11499, 11509, 11532, 11571, 11573, 11574, 11578, 11579, 11580, 11582, 11641, 11642, 11645, 11651, 11659, 11673, 11686, 11688, 11689, 11691, 11697, 11705, 11711, 11727, 11738, 11741, 11749, 11752, 11757, 11760, 11763, 11764, 11765, 11768, 11769, 11815, 11835, 11836, 11837, 11838, 11843, 11848, 11856, 11858, 11861, 11885, 11888, 11892, 11895, 11970, 11980, 11984, 11990, 11991, 12000, 12017, 12019, 12080, 12082, 12084, 12098, 12100, 12105, 12113, 12116, 12117, 12127, 12130, 12132, 12135, 12138, 12142, 12149, 12156, 12173, 12185, 12203, 12207, 12209, 12216, 12220, 12236, 12306, 12320, 12325, 12326, 12328, 12329, 12334, 12335, 12336, 12345, 12347, 12349, 12352, 12357, 12383, 12390, 12407, 12409, 12419, 12435, 12442, 12443, 12459, 12464, 12476, 12478, 12480, 12481, 12491, 12494, 12496, 12503)
        Offset 94000
Warning: Duplicate Bug ids: 216 (12505, 12507, 12547, 12549, 12558, 12564, 12570, 12573, 12595, 12596, 12622, 12638, 12639, 12645, 12663, 12668, 12699, 12706, 12712, 12718, 12781, 12801, 12803, 12821, 12831, 12847, 12850, 12851, 12865, 12873, 12882, 12907, 12916, 12921, 12922, 12924, 12925, 12927, 12928, 12929, 12931, 12932, 12944, 12950, 12982, 12988, 12994, 12995, 13018, 13029, 13043, 13062, 13082, 13098, 13128, 13145, 13156, 13240, 13241, 13242, 13247, 13250, 13257, 13259, 13305, 13306, 13328, 13330, 13335, 13349, 13368, 13381, 13382, 13394, 13400, 13402, 13405, 13448, 13454, 13471, 13472, 13475, 13480, 13486, 13489, 13491, 13506, 13510, 13512, 13514, 13520, 13552, 13560, 13567, 13574, 13588, 13595, 13621, 13622, 13653, 13654, 13674, 13675, 13677, 13678, 13680, 13699, 13703, 13704, 13707, 13709, 13710, 13711, 13719, 13727, 13734, 13754, 13766, 13767, 13777, 13798, 13804, 13805, 13811, 13832, 13833, 13844, 13862, 13886, 13909, 13928, 13935, 13953, 13954, 13972, 13983, 13990, 13994, 14045, 14046, 14047, 14066, 14072, 14081, 14096, 14106, 14112, 14113, 14114, 14120, 14122, 14125, 14143, 14146, 14163, 14184, 14189, 14190, 14203, 14204, 14208, 14228, 14231, 14232, 14242, 14255, 14268, 14272, 14281, 14312, 14316, 14324, 14335, 14347, 14354, 14355, 14361, 14364, 14371, 14372, 14373, 14374, 14379, 14382, 14384, 14389, 14391, 14400, 14409, 14421, 14434, 14435, 14437, 14479, 14481, 14482, 14500, 14510, 14515, 14538, 14541, 14544, 14561, 14590, 14591, 14604, 14609, 14621, 14657, 14665, 14675, 14680, 14682, 14688, 14689, 14717)

This is without providing an API key. As you can see, the duplicate bug IDs are all 3, 4 or 5 digits, making them some of the oldest bugs on BMO. The missing bug IDs I was able to identify were much larger. I also tested it when providing an API key and many more duplicates were returned, which is also reproducible and makes the API unusable due to the resulting high percentage of missing bugs.

Comment 2

[Wayne Mery (:wsmwk)]

David, what could be next step for this?

Comment 3

[David Lawrence [:dkl]]

(In reply to Wayne Mery (:wsmwk) from comment #2)

David, what could be next step for this?

I will try to replicate this and come up with a fix for it. Will take a look at this soon.