Reports until 16:19, Tuesday 07 May 2019
H1 TCS (TCS)
aidan.brooks@LIGO.ORG - posted 16:19, Tuesday 07 May 2019 - last comment - 08:58, Wednesday 08 May 2019(49080)
Hartmann sensors HWS-(ITMY, ETMX, ETMY) have stopped due to full disk

I received a call from Jeff Bartlett that the 3 working HWS have all stopped (ITMX was not running due to the clipping issue in the vertex).

After getting permission from Jeff, I logged into the HWS machines and found that they all stopped at the same time. Checking the disk usage confirmed my suspicion that the data disk (/data) was indeed full. We'll have to look into whether the clean-up script is still working - it might be but it's possible that it wasn't clearing out everything and this, like Thanos, was just inevitable. Once I confirm that all data more than 48 hours old is archived at CIT, we can remove all the old data and get the HWS code running again.

 

 

Images attached to this report
Comments related to this report
thomas.shaffer@LIGO.ORG - 08:58, Wednesday 08 May 2019 (49111)

I've removed the hdf5 files from the 1235600000 - 1236500000 directories in  ITMX & ITMY to bring the disk usage down to 96%. These files were backed up to LDAS so they are not lost. We should think about a long term solution to automatically remove files.

I started all four cameras back up and they appear to be taking data again.

Images attached to this comment