Starting at 04:25 PDT this morning (11:25 UTC) the virgo alert system stopped working.
The log file reports a bad request error:
Traceback (most recent call last):
File "/opt/rtcds/userapps/release/cal/common/scripts/vir_alert.py", line 498, in <module>
far=args.far, test=args.test))
File "/opt/rtcds/userapps/release/cal/common/scripts/vir_alert.py", line 136, in query_gracedb
'label: %sOPS %d .. %d' % (ifo, start, end))
File "/opt/rtcds/userapps/release/cal/common/scripts/vir_alert.py", line 162, in log_query
return list(connection.events(query))
File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 618, in events
response = self.get(uri).json()
File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 374, in get
return self.request("GET", url, headers=headers)
File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 488, in request
return GsiRest.request(self, method, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 356, in request
return self.adjustResponse(response)
File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 369, in adjustResponse
raise HTTPError(response.status, response.reason, response_content)
ligo.gracedb.rest.HTTPError: (400, 'BAD REQUEST / {"error":"Invalid query"}')
After discussion with Keita, we will stop monit trying to run vir_alert on h1fescript0 for now. I believe the plan is that these alerts will be added to the standard gracedb database prior to the next V1 engineering run. I have sent emails bringing sysadmins attention to a potential issue with gracedb-test in case other users are being impacted.