In bug 1762992 I introduced a "search-text" command and support for our rust code to talk gRPC to the livegrep codesearch server. Multiple runs have passed since then, but this morning's config1 run failed during the update step where it seems like a source file got corrupted or had a really bad day. This is the last run of the "morning" to start, so could be indicative of breakage due to dependencies pulling in something new. Log excerpt: ``` 1739 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:5: error: ‘unittest’ has not been declared /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:41: error: ‘message’ was not declared in this scope 1739 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: variable or field ‘ModifyPackedExtensions’ declared void 1776 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: ‘unittest’ has not been declared /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:41: error: ‘message’ was not declared in this scope 1776 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:11: error: ‘unittest’ does not name a type 1797 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected unqualified-id before ‘&’ token 1797 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected ‘)’ before ‘&’ token 1797 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1796:45: note: to match this ‘(’ 1796 | void TestUtilLite::ExpectPackedExtensionsSet( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:47: error: expected initializer before ‘message’ 1797 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:11: error: ‘unittest’ does not name a type 1874 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected unqualified-id before ‘&’ token 1874 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected ‘)’ before ‘&’ token 1874 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1873:47: note: to match this ‘(’ 1873 | void TestUtilLite::ExpectPackedExtensionsClear( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:47: error: expected initializer before ‘message’ 1874 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:11: error: ‘unittest’ does not name a type 1894 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected unqualified-id before ‘&’ token 1894 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected ‘)’ before ‘&’ token 1894 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1893:50: note: to match this ‘(’ 1893 | void TestUtilLite::ExpectPackedExtensionsModified( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:47: error: expected initializer before ‘message’ 1894 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ make[2]: *** [CMakeFiles/lite-test.dir/build.make:122: CMakeFiles/lite-test.dir/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [CMakeFiles/Makefile2:351: CMakeFiles/lite-test.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... ``` And that's just part of it. This is after a bunch of rust crate compilation. One thing we could potentially think about is that now that our provisioning process is largely automated (there's still some manual cleanup work, but I can figure out how to automate that), do we need to be running so much update logic on our indexer runs? Why not have the provisioning process be the only thing that builds our rust binaries and leave them like that? We can then use a web hook or something to poke a lambda job to trigger re-provisioning whenever a new commit lands on mozsearch, etc? The counterpoint is things have generally worked fine for a while and that sounds like it has more ways for stuff to go wrong. I think there may also be ways to check our compiled bindings into our source tree, and that might be a way to sidestep prost-build having to do stuff. For now I'm going to terminate the indexer and trigger a re-provisioning of the indexer and see what happens, since that will report any failures in the update process plus will also help minimize any new updates indexer runs will do. If the provisioning works, I'll re-trigger config1 and see what that does.
Bug 1767538 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
In bug 1762992 I introduced a "search-text" command and support for our rust code to talk gRPC to the livegrep codesearch server. Multiple runs have passed since then, but this morning's config1 run failed during the update step where it seems like a source file got corrupted or had a really bad day. This is the last run of the "morning" to start, so could be indicative of breakage due to dependencies pulling in something new. Log excerpt: ``` 1739 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:5: error: ‘unittest’ has not been declared /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1739:41: error: ‘message’ was not declared in this scope 1739 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: variable or field ‘ModifyPackedExtensions’ declared void 1776 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:5: error: ‘unittest’ has not been declared /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1776:41: error: ‘message’ was not declared in this scope 1776 | unittest::TestPackedExtensionsLite* message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:11: error: ‘unittest’ does not name a type 1797 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected unqualified-id before ‘&’ token 1797 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:45: error: expected ‘)’ before ‘&’ token 1797 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1796:45: note: to match this ‘(’ 1796 | void TestUtilLite::ExpectPackedExtensionsSet( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1797:47: error: expected initializer before ‘message’ 1797 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:11: error: ‘unittest’ does not name a type 1874 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected unqualified-id before ‘&’ token 1874 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:45: error: expected ‘)’ before ‘&’ token 1874 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1873:47: note: to match this ‘(’ 1873 | void TestUtilLite::ExpectPackedExtensionsClear( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1874:47: error: expected initializer before ‘message’ 1874 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:11: error: ‘unittest’ does not name a type 1894 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~~ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected unqualified-id before ‘&’ token 1894 | const unittest::TestPackedExtensionsLite& message) { | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:45: error: expected ‘)’ before ‘&’ token 1894 | const unittest::TestPackedExtensionsLite& message) { | ^ | ) /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1893:50: note: to match this ‘(’ 1893 | void TestUtilLite::ExpectPackedExtensionsModified( | ^ /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc:1894:47: error: expected initializer before ‘message’ 1894 | const unittest::TestPackedExtensionsLite& message) { | ^~~~~~~ make[2]: *** [CMakeFiles/lite-test.dir/build.make:122: CMakeFiles/lite-test.dir/home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.10.1/third-party/protobuf/src/google/protobuf/test_util_lite.cc.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [CMakeFiles/Makefile2:351: CMakeFiles/lite-test.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... ``` And that's just part of it. This is after a bunch of rust crate compilation. One thing we could potentially think about is that now that our provisioning process is largely automated (there's still some manual cleanup work, but I can figure out how to automate that), do we need to be running so much update logic on our indexer runs? Why not have the provisioning process be the only thing that builds our rust binaries and leave them like that? We can then use a web hook or something to poke a lambda job to trigger re-provisioning whenever a new commit lands on mozsearch, etc? The counterpoint is things have generally worked fine for a while and that sounds like it has more ways for stuff to go wrong. I think there may also be ways to check our code-generated livegrep bindings into our source tree, and that might be a way to sidestep prost-build having to do stuff. For now I'm going to terminate the indexer and trigger a re-provisioning of the indexer and see what happens, since that will report any failures in the update process plus will also help minimize any new updates indexer runs will do. If the provisioning works, I'll re-trigger config1 and see what that does.