1767538 - Indexer update failure on prost-build

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

In bug 1762992 I introduced a "search-text" command and support for our rust code to talk gRPC to the livegrep codesearch server. Multiple runs have passed since then, but this morning's config1 run failed during the update step where it seems like a source file got corrupted or had a really bad day. This is the last run of the "morning" to start, so could be indicative of breakage due to dependencies pulling in something new.

Log excerpt:

   1739 |     unittest::TestPackedExtensionsLite* message) {
        |     ^~~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:5: error: ‘unittest’ has not been declared
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:41: error: ‘message’ was not declared in this scope
   1739 |     unittest::TestPackedExtensionsLite* message) {
        |                                         ^~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: variable or field ‘ModifyPackedExtensions’ declared void
   1776 |     unittest::TestPackedExtensionsLite* message) {
        |     ^~~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: ‘unittest’ has not been declared
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:41: error: ‘message’ was not declared in this scope
   1776 |     unittest::TestPackedExtensionsLite* message) {
        |                                         ^~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:11: error: ‘unittest’ does not name a type
   1797 |     const unittest::TestPackedExtensionsLite& message) {
        |           ^~~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected unqualified-id before ‘&’ token
   1797 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected ‘)’ before ‘&’ token
   1797 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
        |                                             )
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1796:45: note: to match this ‘(’
   1796 | void TestUtilLite::ExpectPackedExtensionsSet(
        |                                             ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:47: error: expected initializer before ‘message’
   1797 |     const unittest::TestPackedExtensionsLite& message) {
        |                                               ^~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:11: error: ‘unittest’ does not name a type
   1874 |     const unittest::TestPackedExtensionsLite& message) {
        |           ^~~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected unqualified-id before ‘&’ token
   1874 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected ‘)’ before ‘&’ token
   1874 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
        |                                             )
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1873:47: note: to match this ‘(’
   1873 | void TestUtilLite::ExpectPackedExtensionsClear(
        |                                               ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:47: error: expected initializer before ‘message’
   1874 |     const unittest::TestPackedExtensionsLite& message) {
        |                                               ^~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:11: error: ‘unittest’ does not name a type
   1894 |     const unittest::TestPackedExtensionsLite& message) {
        |           ^~~~~~~~
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected unqualified-id before ‘&’ token
   1894 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected ‘)’ before ‘&’ token
   1894 |     const unittest::TestPackedExtensionsLite& message) {
        |                                             ^
        |                                             )
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1893:50: note: to match this ‘(’
   1893 | void TestUtilLite::ExpectPackedExtensionsModified(
        |                                                  ^
  /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:47: error: expected initializer before ‘message’
   1894 |     const unittest::TestPackedExtensionsLite& message) {
        |                                               ^~~~~~~
  make[2]: *** [CMakeFiles/lite-test.dir/build.make:122: CMakeFiles/lite-test.dir/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc.o] Error 1
  make[2]: *** Waiting for unfinished jobs....
  make[1]: *** [CMakeFiles/Makefile2:351: CMakeFiles/lite-test.dir/all] Error 2
  make[1]: *** Waiting for unfinished jobs....

And that's just part of it.

This is after a bunch of rust crate compilation.

One thing we could potentially think about is that now that our provisioning process is largely automated (there's still some manual cleanup work, but I can figure out how to automate that), do we need to be running so much update logic on our indexer runs? Why not have the provisioning process be the only thing that builds our rust binaries and leave them like that? We can then use a web hook or something to poke a lambda job to trigger re-provisioning whenever a new commit lands on mozsearch, etc? The counterpoint is things have generally worked fine for a while and that sounds like it has more ways for stuff to go wrong. I think there may also be ways to check our code-generated livegrep bindings into our source tree, and that might be a way to sidestep prost-build having to do stuff.

For now I'm going to terminate the indexer and trigger a re-provisioning of the indexer and see what happens, since that will report any failures in the update process plus will also help minimize any new updates indexer runs will do. If the provisioning works, I'll re-trigger config1 and see what that does.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Both the provisioning and the indexing run completed happily. I'll reopen if this happens more in the future.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

I'm also quickly triggering a web-server re-provision because I had only triggered an indexer re-provision and I don't want to have to deal with the analysis headache of having them out of sync.