Closed Bug 1376048 Opened 9 years ago Closed 8 years ago

Mercurial/Git mapper is not working for new mercurial commits

Categories

(Developer Services :: Git, defect)

Product:
Developer Services
Component:
Git
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: marco, Assigned: dhouse)

References

Details

Attachments

(2 files)

compare_and_push_mappings.sh
1.70 KB, text/plain
Details
check_beagle_state.sh
3.39 KB, text/plain
Details
Could you look into this?
Flags: needinfo?(dhouse)
:dhouse - could be similar to bug 1193011 -- the cronjob mentioned in that bug might only recently have died, and been masking the issue.
See Also: → 1193011
We have this commit in the logs as published-to-mapper:323884:77b256dc98efb93f1d118db456f442a09bba77c2 c03f906e6579fce0b61524c4f5c8445dfa76ffe0 77b256dc98efb93f1d118db456f442a09bba77c2 d9351ba8ef375926c2ca0c98661cf1f83b5d5d4b 2017-06-23 08:34:57 -0700 No bug, Automated HPKP preload list update from host bld-linux64-spot-304 - a=hpkp-update I'm looking to see if we only *thought we published it.
:marco do you have a place I can look to see what is missing? With a list, I can republish. If you don't have a list, I can step through the commits and hit the mapper, but if you already have a place for me to look that may be faster. --- I manually removed the "published" entry for the commit, c03f906e6: build/conversion/beagle/.hg/published-to-mapper On the latest scheduled run, the commit was caught as missing and re-published. Now it is available: https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/c03f906e6579fce0b61524c4f5c8445dfa76ffe0
Flags: needinfo?(dhouse) → needinfo?(mcastelluccio)
Flags: needinfo?(mcastelluccio)
My removal of the last two weeks of mappings last night failed during the push because it timed-out on the mapper server (?). So I dropped it down to only posting 20 at a time, and that worked this morning. I pushed for only the last two weeks; I may need to set up a cron to check and push missing mappings.
Assignee: nobody → dhouse
Yesterday afternoon, I added a cron job for the vcs2vcs user on vcssync to re-push mappings from the last two hours, every hour. So let's see if that covers us.
Flags: needinfo?(dhouse)
I changed the cleanup cron to only run twice per day and not depend on modifications to mozharness (attached self-contained script): 0 11,23 * * * /home/vcs2vcs/vcssync1/compare_and_push_mappings.sh "7 days ago"
:marco, I expect the cleanup cron to catch any missing mappings. If you see problems with the mappings again please let me know.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Flags: needinfo?(dhouse)
(In reply to Marco Castelluccio [:marco] from comment #10) > I'm seeing the problem again. > https://hg.mozilla.org/mozilla-central/rev/ > 211d4dd61025c0a40caea7a54c9066e051bdde8c is at > https://github.com/mozilla/gecko-dev/commit/ > b72c4833d5497f575b9612a4ddbe33f9195afd80, but isn't found at > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/ > 211d4dd61025c0a40caea7a54c9066e051bdde8c. (This is only an example, there are also commits from yesterday missing, e.g. 53a5192b436827f667db9df036b07d245e35d97a)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Marco, when I manually ran the cleanup script on the 5th, the commit you noted was caught (with about ten others). I changed the cleanup script to run hourly and do a secondary check of the mapping (asking the mapping server specifically for each commit as well as collecting a batch). I'll check over the next week to see if we miss any after these changes.
Flags: needinfo?(dhouse)
There are others broken too: a708f80289b20da80bea42ac95777ac766ee6845, 858be731fb0135b4e6e8cc4f585537252d7efe64, 13716663a04056df2777a19082b16bbcf05654b9, 4df531710d390918c49f788d5aa57977a262fac3, b83dc364d19df20bccf84241be0a873b9c14b3c6, ...
I'm not certain what happened here. I confirmed in the logs that the broken mappings you listed were found as missing and pushed to the mapper server, but that repeated in the logs until today. So that looks like the posts fails or something on the mapper server did not keep/apply the posted mappings. I've extended the logging in the cleanup script to record when mappings are checked, and added, to hopefully give more information for next time this occurs.
We had a number of sync failures in the last 24 hours. There may be a correlation with the mappings missing, but I would expect we'd see an increase in missing mappings and we have not. The cleanup script found a few mappings missing each of the last three days. I confirmed in the logs that these were not found previously and appeared in the mapper's "worldview" since: compare_and_push_mappings/11/03/uploadaa https://hg.mozilla.org/mozilla-central/rev/0e41d07a703f19224f60b01577b2cbb5708046c9 https://github.com/mozilla/gecko-dev/commit/8e3be4c671cb5718479d95086205ad3b1b1f59aa > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/0e41d07a703f19224f60b01577b2cbb5708046c9 compare_and_push_mappings/11/11/uploadaa https://hg.mozilla.org/mozilla-central/rev/b07db5d650b7056c78ba0dbc409d060ec4e922cd https://github.com/mozilla/gecko-dev/commit/31311070d9860b24fe4a7a36976c14b328c16208 > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/b07db5d650b7056c78ba0dbc409d060ec4e922cd compare_and_push_mappings/11/17/uploadaa https://hg.mozilla.org/mozilla-central/rev/6fec4855b5345eb63fef57089e61829b88f5f4eb https://github.com/mozilla/gecko-dev/commit/d37d297687b75be21faa6aa6a930b4427b988f10 > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/6fec4855b5345eb63fef57089e61829b88f5f4eb compare_and_push_mappings/12 compare_and_push_mappings/12/09/uploadaa https://hg.mozilla.org/mozilla-central/rev/09a4282d1172ac255038e7ccacfd772140b219e2 https://github.com/mozilla/gecko-dev/commit/0edfa97e3886f672b65e70bf5604104cacdd5fa5 > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/09a4282d1172ac255038e7ccacfd772140b219e2 compare_and_push_mappings/12/16/uploadaa https://hg.mozilla.org/mozilla-central/rev/03bcd6d65af62c5e60a0ab9247ccce43885e707b https://github.com/mozilla/gecko-dev/commit/cef8389c687203085dc6b52de2fbd0260d7495bf > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/03bcd6d65af62c5e60a0ab9247ccce43885e707b compare_and_push_mappings/13 compare_and_push_mappings/13/02/uploadaa https://hg.mozilla.org/mozilla-central/rev/30ea2905130e85f9e1d8d56fa3097901eec6514b https://github.com/mozilla/gecko-dev/commit/cbd628b085ac809bf5a536109e6288aa91cbdff0 > https://api.pub.build.mozilla.org/mapper/gecko-dev/rev/hg/30ea2905130e85f9e1d8d56fa3097901eec6514b
(In reply to Dave House [:dhouse] from comment #16) > We had a number of sync failures in the last 24 hours. There may be a > correlation with the mappings missing, but I would expect we'd see an > increase in missing mappings and we have not. Is there already a bug for the sync issues? I'm seeing mozilla/gecko-dev stuck at 2 days ago (but the mappings are working).
marco: I reopened https://bugzilla.mozilla.org/show_bug.cgi?id=1369562 for the sync issue.
Attached file check_beagle_state.sh —
Current crontab: 13 * * * * /home/vcs2vcs/vcssync1/compare_and_push_mappings.sh "7 days ago" "all"> ~/compare_and_push_mappings/`date -u "+\%d"/\%H`.log 2>&1 0,15,30,45 * * * * /home/vcs2vcs/check_beagle_state.sh > ~/check_beagle_state.log Attached updated check_beagle_state.sh. I added a hex print of the name and message as there was a comparison failing this morning that appeared to be from the string having different whitespace.
We had a few times where, because a commit was older than a week, it was not caught by the cleanup script until the twice daily longer period (2 months currently). So now the hourly scan checks back 31 days. ``` [vcs2vcs@ip-172-31-30-125 ~]$ crontab -l 11 * * * * /home/vcs2vcs/vcssync1/compare_and_push_mappings.sh "31 days ago" "all"> ~/compare_and_push_mappings/`date -u "+\%d"/\%H`.log 2>&1 0,15,30,45 * * * * /home/vcs2vcs/check_beagle_state.sh > ~/check_beagle_state.log 0 1,13 * * * /home/vcs2vcs/vcssync1/compare_and_push_mappings.sh "62 days ago" "all"> ~/compare_and_push_mappings/`date -u "+\%d"/\%H_long`.log 2>&1 0 0 * * * /home/vcs2vcs/check_disk.sh > ~/check_disk.log 2>&1 1 20 22 * * /home/vcs2vcs/vcssync1/compare_and_push_mappings.sh "365 days ago" "all"> ~/compare_and_push_mappings/monthly.log 2>&1 ```
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: