Incident is resolved. Our cloud provider had a short network partition that removed quorum from our coordination service. As a protective measure, applications started giving errors to prevent data loss. When the situation went back to normal, everything recovered and service resumed as expected.
Posted Dec 13, 2016 - 11:13 EST
Monitoring
Between 8:45 and 9:00 EST there was an issue with the coordination service supporting our queuing infrastructure. There has been some request that got answered an HTTP 5xx error and some delays in data availability for search. We addressed the issue and our monitoring the environment.