H1 IFO Guardian top node briefly reports OK

For maybe the first time ever, the H1 IFO top node reported ALL_NODE_OK, and indicated OK==True for the entire H1 IFO:

This is a reflection of  the fact that every node in the system is reporting OK==True, which in turn means that all nodes are operating correctly and their systems are their nominal states.  In other words,

H1 guardian is ready for O1

Of course it dropped out almost immediately as the ISS was turned off for commissioning, but still...

CDS maintenance Summary

PEM channel renaming

WP5414. Robert, Dave:

h1pemcs was modified to rename the accelerometer channels PR1, SR1, MC to PRM, SRM, IMC. Change was applied to DAQ when it was restarted this morning.

h1isiham3 phantom ini change

Hugh, Dave

After the 07:55 restart of h1isiham3 and a DAQ restart the front end was reporting an ini file mismatch. We are not sure this was a true mismatch since the ini file had not been modified since Monday. We restarted h1isiham3 which cleared the alert.

h1susey computer

WP5413. Carlos, Dave, Jim:

the original h1susey computer was re-installed to remove the IOP glitching. The ETMY model is running longer than when this machine was last in service  (was 51uS, now 55uS). We checked the new SUSETMYPI model runs well with this hardware.

Reboot digital video servers

Dave

The digital video servers h1digivideo[0,1,2] were power cycled as part of the PM reboot project. These machines have been running for 280, 280 and 251 days respectively. We saw no problems with reboots.

Digital video snapshop problem. FRS3410.

Jenne, Dave, Jim, Elli, Kiwamu:

Jenne found that certain digital cameras are producing tiff files which cannot be viewed by most graphics tools. The reboot of the video servers did not fix this. We tried power cycling the ITMX spool camera which also did not help. We tracked this down to the "Frame Type = Mono12" change in the camera configuration. Kiwamu and Elli have methods to read the 32 bit tiff files. This problem is now resovled, FRS3410 will be closed.

EPICS Gateway

Dave

Due to the extended downtime of h1ecatx1, CA clients on the H1FE network did not reconnect to this IOC (EDCU, conlog, Guardian). I restarted the h1slow-h1fe EPICS gateway which prompted the clients to reconnect to h1ecatx1.

DAQ Restarts

Jim, Dave

There were several DAQ restarts. The last restart was quite messy, details in earlier alog.

The FPGA duotone channels were added to the frame broadcaster for DMT timing diagnostics.

OMC guardian hustle

Jemie, Kiwamu,

Even though we didn't think that the change we made on the OMC model (alog 20197) would impact on locking, it actually did.

The OMC guardian kept stopping because the code tries to access an oversize index for a data array. After some investigaton, it turned out to be due to the data rate for H1:OMC-PZT1_MON_DC_OUT_DQ whose sampling rate had been changed from 16 kHz to 512 Hz in this morning. Since the OMC guardian tries to access the data with an assumption that the data sampling rate is 16 kHz, some lines in the code lead it to the bad situation. This error happened in FIND_CARRIER in which the code sweeps the OMC length and attemps to find peaks for the 45 MHz sidebands.

Jamie and I did a quick hack -- we decimated another channel, H1:OMC-DCPD_SUM_OUT_DQ, from 16 kHz to also 512 Hz using scipy.signal.deciamate() so that relavant lines are consistent for 512 Hz. We tested it once and it worked.

IOO rotation stage issues

Stefan, Sheila, Patrick

The rotation stage acted up again. It seems to be working again after power cycling the Beckhoff chassis. We did not restart the computer.

Searching for home seemed to go to home, but subsequent go to power requests did not go to the requested power. In many instances the rotation stage moved in the wrong direction.

Lock loss analysis tool

Matt E., James R., Sheila D., Terra H., Hang Y.

We are developing a tool to automate lock loss analysis, and an initial version is availabel at

/opt/rtcds/userapps/trunk/isc/common/scripts/lockloss

There is a matlab version and a python version, and right now they are essentially equivalent. An example generated using the python version is attached to this entry.

=================================================================================================================================

The code takes in a gps time around which a lock loss happens. It first looks through GUARDIAN to get a rough time of lock loss. Then it looks at 'LSC-POP_A_LF_OUT_256_DQ' to find a sudden drop, and the time of this drop will be used as our more accurate lock loss time.  It then reads in data from an user defined list of channels (or if none, all DQ channels by default) and breaks the data into three frequency bands (>0.125 Hz for low, 16 ~ 128 Hz for mid, and >128 for high). The code can identify big jumps in each channel's BLRMS and order the channels by the time of the jumps. The first 16 channels goes wrong will be plotted (as the attached one) and all channels in which big jumps are seen will be recorded into a .txt file.

In the figure attached (a lock loss at GPS time 1122672396 UTC), black lines are raw data of 'POP_A_LF_OUT_256_DQ', blue lines are the raw data of the channel in the corresponding subplot, and red lines are BLRMS for the band that first goes wrong. X-axes are time in second, with zero defined as the time of lock loss (value shown in the main title), and y-axis is an arbitary scale as we only care about trend but not absolute values. Each subplot's title consists of three parts: channel name, time of the first jump prior lock loss, and the frequency band that first goes over the critical value.

Checking all DQ channels take a considearable amount of time, so in the attached one we looked only at suspension channels. A full analysis is on going when this entry is created.

=================================================================================================================================

For future imporvement, we are working on creating a cron job which periodically check GUARDIAN and whenever a lock loss happens this analysis code will be automatically triggered. Sheila and Terra are working on a complementary code to this which looks all channels that saturates or hits some known bad zone before lock loss. More ideas/ thoughts/ comments are welcome.

Rotation stage roulette

Patrick, Jeff, Evan

We spent a few minutes cooking the IX compensation plate while trying to make the TCS rotation stage behave.

• To start with, the controller reported the measured angle as 0°. (During the previous TCS tests on Sunday, it was more like 55°.)
• When I asked for a different angle, the controller complained "Status: position warning", and the measured angle didn't change.
• So we asked it to search for home, which seemed to work. The homing position seems to be around the max power out of the TCS (5 W).
• Then I asked for 50°, but for some reason the measured angle dropped to −7° and then stopped.
• Then I asked for some other angles, but the stage kept rotating to random places.
• Then I searched for home again, and then asked for −50°. This time it seemed to work.
• Then I tuned the requested angle until the power output was 0.12 W.

This was after the Beckhoff chassis was power cycled.

Pcal line at 3001.3 Hz added

Shivaraj, Darkhan

We set a high frequency Pcal line at 3001.3 Hz using PCALX. The amplitude of the line is factor of ~5 smaller than the DARM noise spectrum at that frequency (July 21 reference).

SDF_OVERVIEW has been updated accordingly.

NOMINAL states defined for ISC_DRMI and ALIGN_IFO

Defined NOMINAL states for nodes that were missing them:

ISC_DRMI: DRMI_3F_LOCKED
ALIGN_IFO: SET_SUS_FOR_FULL_LOCK

All nodes must have NOMINAL states defined for correct state reporting by the IFO node.

Estimation of Change in Actuation Coefficient

Summary:  Less than +/-10% Change in DARM Actuation Coefficient from June 28 to present.

On July 14th, we reported in LHO alog #19163 that the DARM actuation after the vent had not changed by more than 10% from its ER7 value. Responding to a request from LeoP, we made a similar plot for the period from June 28 until now.  

The attached plot has 60s-long FFT data plotted in blue and 30-point running averages of the blue data plotted in red. The data includes all times when the Guardian State Vector value was more than 501 (DC Readout).  

The data were generated using the low frequency calibration lines (~ 35 Hz) which had low SNR before July 22nd. We increases the SNR of our calibration lines to about 100, informed by an  ER7 calibration line study, and reported in alog #19792 and the attached comments. Here we have also accounted for the fact that the response function of DARM, C/(1+G), is frequency dependent (calibration lines being few Hz apart) using the DARM refrence model for ER7.

More added to SYS_DIAG

Added a few tests today and removed ITMX/Y L2 OLDAMP Off notification.

Tests added are:

• ESD railed - if the absolute value of the average after 5 seconds stays within +/- 600 of 16200, then it will notify.
• TCS power - if the power is < 50W, notify.
• TCS Flow and IR alarms - if in alarm, notify.
• Beckhoff down - if the elasped time is not greater than the last time the test ran, then it will notify (trending showed that this channel will stop counting when Beckhoff is crashed)

As always, please be aware of these notifications and always investigate when one pops up.

h1susey, so far so good

Carlos, Jim, Dave:

after the re-install of the original h1susey front end computer we have not seen any IOP TIM and/or ADC errors, though it is too early to declare success. To show these errors if they were to appear I am no longer performing a periodic (every minute) DIAG_RESET of EY. However I am still doing this for EX since that system still gliches.

Messy DAQ restart

Dave, Jim:

at 16:20 PDT we restarted the DAQ for a new EDCU ini file. This was not a clean restart, resulting in a confusing overvew screen with some front ends not registering any DAQ status channels. After a clean shutdown and startup of monit on h1dc0 and a manual start of h1nds0 and h1nds1 we got back to a stable system. We have noticed that in the past week monit sometimes has not cleanly restarted the data concentrator daqd process. Looking at today's log files and configuration file cache it appears monit and/or start-stop-daemon got into a race condition which resulted in daqd  being restarted before the complete set of configuration files was ready. We will have to think more about how we got into this condition.

We checked that the current DAQ has the correct INI and PAR files and the frame sizes are correct.

Camera masks set for test mass IR cameras

I have set the masks (and reloaded the config files) for the ITMX spool, ITMY spool, ETMX and ETMY cameras.  Note that these are not the cameras that are used during initial alignment. 

The idea is that with an appropriately set mask to exclude extraneous light on the cameras, we can use the centroid tracking to monitor the spot positions on the test masses over long periods.  These cameras are saturating over the whole central region when the arms are on resonance, so the centering calculation won't be perfect, but hopefully it'll still give us an idea of what is going on. 

None of these had previously had a mask defined. Finding snapshots to help set the mask center and radius was what set off the confusion and investigation earlier.  Elli's script was immensely useful in plotting the images for setting the mask.

Plots below are snapshots from 28 July 2015, around 10:00utc when the IFO was locked.  Overlaid are circles (even though they look like ovals because of the non-square grid) that are equivalent to the masks that I've chosen.  We will see images from inside the circle, and exclude the portions of the image outside the circle.

The centroid values are already trended as slow channels, so we can start looking at the history tonight, particularly if we re-do some thermal tuning tests.

Ops Summary

Patrick, Cheryl, Nutsinee

Most of the logs were hand-written by Patrick. If things don't make sense that means I didn't decode his handwriting correctly.

 

- All time in UTC (PDT) -

14:58 (7:58) Peter getting something out of H2 PSL enclosure

15:02 (8:02) Jeff to LVEA colect CC Data

15:05 (8:05) Shiva and Sudarshan to EX

15:08 (8:08) Filiberto cabling started

15:18 (8:18) Joe check eye wash in LVEA

15:19 (8:19) Peter done

15:20 (8:20) Jeff done

15:34 (8:34) Ed to EY to start TCS

15:35 (8:35) Christina, Karen to EY (cleaning)

15:50 (8:50) Hannah, Elli install TCSY temp. sensor

15:58 (8:58) EX, FIl and Gerado start regen process

16:07 (9:07) Jason to optics lab

16:19 (9:19) Robert starting PEM

16:21 (9:21) Karen, Christina to EX

16:22 (9:22) Thomas, Shiva, ?

16:23 (9:23) Ed temp. back looking for drillbit

16:26 (9:26) Elli, Hannah back

16:27 (9:27) Filiberto and Andrea done, not out

16:30 (9:30) Jason start PMC transfer function

16:31 (9:31) Ed to EY

16:35 (9:35) Jim and Carlos to EY to replace H1scsey fornt end computer

16:37 (9:37) John, Elli, Hanna to LVEA

16:39 (9:39) DAQ restart

16:52 (9:52) Guardian restart full machine

16:54 (9:54) Karen, Christina leaving EX

16:59 (9:59) Robert into PSL

17:00 (10:00) Elli, Hanna to TCS ITMY

17:01 (10:01) DAC PEM CS model restart

17:09 (10:09) Cheryl to PSL

17:10 (10:10) Ed leaving EY and going to EX

17:15 (10:15) Leo about to start charge measurement at EY

17:20 (10:20) ASC-IMC, OMC, CSC model restart (Kiwamu, Keita)

17:35 (10:35) Kiwamu restart H1LSC model

17:36 (10:36) H1 ASC IMC restart (Kiwamu)

17:45 (10:45) Kyle to EY to shut off calibration gas

17:45 (10:45) Nutsinee to help Elli in the LVEA

17:51 (10:51) DAC restart digital video computers

17:53 (10:53) Karen and Christina cleaning in LVEA

17:55 (10:55) DAQ restart

18:00 (11:00) Fil to EX, fix UIM(?) drive

18:13 (11:13) DAQ restart

18:21 (11:21) Robert in PSL

18:24 (11:24) Parada water delivery

18:25 (11:25) Joe done

18:27 (11:27) Robert to LVEA to continues setting up PEM (?)

18:38 (11:38) Gerado and Kyle vack to LVEA

18:39 (11:39) measure power on photodiode

18:45 (11:45) Dave restarting camera servers

18:46 (11:46) Ed done at EX, going to LVEA

18:50 (11:50) Hugh HAM5, testing SDF

19:10 (12:10) Robert done

19:12 (12:12) Ed done

19:12 (12:12) Cheryl done

19:19 (12:19) Fil done

19:28 (12:28) Jim, Hugh, filter bugs in HEPI

19:29 (12:29) Pcal done

19:45 (12:45) Rick back to EX for laptop

19:59 (12:59) TJ, Betsy derail(?) ETMX high voltage driver

20:11 (13:11) Kyle back to EX

20:18 (13:18) Ops computer restart

20:26 (13:26) TJ, Betsy done

20:36 (13:36) Gerado to work with Elli

20:41 (13:41) EX inf. con. gage off

21:00 (14:00) ALS COMM, DIFF, XARM, and YARM nominal state is now SHUTTERED (Sheila)

21:38 (14:38) Fil to EX - try to fix ESD drive

22:13 (15:13) ESD working - Fil going to Mid Y

22:27 (15:27) Jeff to EX to restart high voltage

22:40 (15:40) Jeff coming back

0:53 (17:53) Patrick and Stefan still recovering the ifo.

 

Happy Maintenance Tuesday....

defunct LSC_CONFIGS guardian node accidentally resurected, swiftly destroyed

The old LSC_CONFIGS guardian node, which was defunct but hadn't been properly destoryed, was briefly resurected after the h1guardian0 machine reboot.  It then tried to managed some nodes, which caused a bit of confusion.

I "destroyed" the node so it won't bother us anymore:

guardctrl destroy LSC_CONFIGS

EX UIM Coil Driver - Power Switch Replaced

Replaced the power switch in the UIM coil driver S0900303 at EX. Unit was reported to be found in off position several times in the past few days. We will monitor unit to see if it switches off again.

IO on the PSL - status of the ECR, and measuring power on PDs

Per IO ECR E1300432:

 

Components to be changed in the IO path on the PSL:

1. IO_AB_M15 is changed to IO_AB_W1
2. IO_AB_PD1 is positioned on the reflected beam from the wedge
3. IO_AB_PD3 is added on the transmitted beam from the wedge, and is a large area PD, 100mm^2

Status:  not complete

1. IO_AB_M15 has been changed to IO_AB_W1 - DONE
2. IO_AB_PD1 is positioned on the reflected beam from the wedge - located there but cables are not connected
3. IO_AB_PD3 is added on the transmitted beam from the wedge, and is PDA100A, a large area PD, 100mm^2 - added but a PDA55 with only ~13mm^2 sensing area

Other information collected:

 

PD Channels:

H1:PSL-EOM_A_DC_POWER - thorlabs behind IO_AB_W1

H1:IMC-PWR_EOM_OUTPUT - thorlabs behind IO_AB_W1

PD,measured power:

IO_AB_PD3, thorlabs PDA55, 3.33mW

Incident on BS:

before the splitter, IO_AB_W1, 3.51mW

PD,measured power:

IO_AB_PD1, Newport 1811 RFPD, about 50uW - two beams, fast measurement, needs more time to set up

 

PD Channels:

H1:PSL-PERISCOPE_A_DC_POWER - bottom periscope trans

H1:IMC-PWR_IN_OUTPUT - bottom periscope trans

PD,measured power:

IO_AB_PD2, thorlabs PDA55, 83uW

Incident on steering mirror:

before steering mirror IO_AB_M9, 85uW

Incident on BS:

power in non-PD path, coming off of IO_AB_M10, 85uW

EX Beckhoff dies

After getting the ETMx ESD driver unrailed, we found the Driver state "OFF" via the red/green ESD Active indicator light.  Nice timing all day - the ESD keeps dying just ahead of us trying to use it!

So, after we returned from the railing saga, we resumed running a realignment.  A few mins in we discovered the red light indicating the driver was off.  We toggled some switches to no avail and Fil headed to EndX again.  He found the power supply tripped.  He tried to reset it and it tripped again.  After hunting around for another 20mins, it occured to us to check the Beckhoff - yep, it had crashed.  (The pressure guage that the ESD looks at lost connection and therefore shut down the power supply of the ESD for safety.)  Patrick restarted the Beckhoff, Fil then reset the ESD power supply, and the ESD finally looks healthy again.  For now.

TJ is adding these failure modes into the SYS (alarm) guardian.