Jenkins is constantly failing ES tests with a TimeoutError. It's both irritating and also totally unhelpful. Couple of ideas: 1. Talk to IT about it. 2. Raise the timeout from 5s to 30s or something absurd like that. The second item should be easy to do so I'm going to make this a 1 pointer. The first item might not be necessary. Making this a P2 because this is affecting development.
Talked about this on IRC and we decided raising the timeout beyond what it's at now is like saying, "Don't eat more than 100 hot dogs a day." to ward off health issues. So that's off the table. Updating whiteboard.
Created bug #797859 pinging IT about the ES cluster that Jenkins uses.
As I understand it, the CI machine hosts both Jenkins and the ES it uses. At some point, IT will put ES on a separate machine. That'll probably solve the problems we're seeing here. Until then, if ES-related tests start kicking up a TimeoutError, we should ping someone in IT to reboot the CI machine.
This is all set now. We have a new ES cluster that we use for CI as of a few weeks ago.