1033350 - inventory errors prevent pandas from functioning correctly after panda chassis are moved

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Inventory, task)

Description

[Amy Rich [:arr] [:arich]]

it looks like system.imaging_server.0 got duplicated instead of overwritten when the k/v store was updated for all pandas. This is causing all pandas that were moved as part of move train 1 to fail.

Comment 1

[Amy Rich [:arr] [:arich]]

Three issues with this one:

1) The spreadsheet column was called keyvalue_set.system_imaging_server.0 not keyvalue_set.system.imaging_server.0 - this meant that every board had two entries in the k/v store. This is now fixed for all spreadsheets, and I removed all occurrences of all six sets of imaging server values from the mysql database directly as this was the fastest and easiest way to clean this up.  I then re-added the correct keys and values using the v2 API
2) For boards in p2, they had THREE k/v entries, two for system_imaging_server.0 and one for system.imaging_server.0.
3) One all of the entries were corrected, mozpool was unable to update it's database because there were too many changes (a precautionary measure against mass database corruption). Unfortunately there was ALSO an error in the code, so you couldn't even tell it to manually override. Mozpool has been patched (not yet shipped), one host was manually updated and the database was synced.