PEM channel renaming
WP5414. Robert, Dave:
h1pemcs was modified to rename the accelerometer channels PR1, SR1, MC to PRM, SRM, IMC. Change was applied to DAQ when it was restarted this morning.
h1isiham3 phantom ini change
Hugh, Dave
After the 07:55 restart of h1isiham3 and a DAQ restart the front end was reporting an ini file mismatch. We are not sure this was a true mismatch since the ini file had not been modified since Monday. We restarted h1isiham3 which cleared the alert.
h1susey computer
WP5413. Carlos, Dave, Jim:
the original h1susey computer was re-installed to remove the IOP glitching. The ETMY model is running longer than when this machine was last in service (was 51uS, now 55uS). We checked the new SUSETMYPI model runs well with this hardware.
Reboot digital video servers
Dave
The digital video servers h1digivideo[0,1,2] were power cycled as part of the PM reboot project. These machines have been running for 280, 280 and 251 days respectively. We saw no problems with reboots.
Digital video snapshop problem. FRS3410.
Jenne, Dave, Jim, Elli, Kiwamu:
Jenne found that certain digital cameras are producing tiff files which cannot be viewed by most graphics tools. The reboot of the video servers did not fix this. We tried power cycling the ITMX spool camera which also did not help. We tracked this down to the "Frame Type = Mono12" change in the camera configuration. Kiwamu and Elli have methods to read the 32 bit tiff files. This problem is now resovled, FRS3410 will be closed.
EPICS Gateway
Dave
Due to the extended downtime of h1ecatx1, CA clients on the H1FE network did not reconnect to this IOC (EDCU, conlog, Guardian). I restarted the h1slow-h1fe EPICS gateway which prompted the clients to reconnect to h1ecatx1.
DAQ Restarts
Jim, Dave
There were several DAQ restarts. The last restart was quite messy, details in earlier alog.
The FPGA duotone channels were added to the frame broadcaster for DMT timing diagnostics.
Jemie, Kiwamu,
Even though we didn't think that the change we made on the OMC model (alog 20197) would impact on locking, it actually did.
The OMC guardian kept stopping because the code tries to access an oversize index for a data array. After some investigaton, it turned out to be due to the data rate for H1:OMC-PZT1_MON_DC_OUT_DQ whose sampling rate had been changed from 16 kHz to 512 Hz in this morning. Since the OMC guardian tries to access the data with an assumption that the data sampling rate is 16 kHz, some lines in the code lead it to the bad situation. This error happened in FIND_CARRIER in which the code sweeps the OMC length and attemps to find peaks for the 45 MHz sidebands.
Jamie and I did a quick hack -- we decimated another channel, H1:OMC-DCPD_SUM_OUT_DQ, from 16 kHz to also 512 Hz using scipy.signal.deciamate()
so that relavant lines are consistent for 512 Hz. We tested it once and it worked.
Somehow it still can't find the 00 mode on its own.
As for the warning message about bad filter coefficients, I changed the order of the Chebyshev from 8 to 4, and that seems to make the warning message go away.
Stefan, Sheila, Patrick The rotation stage acted up again. It seems to be working again after power cycling the Beckhoff chassis. We did not restart the computer. Searching for home seemed to go to home, but subsequent go to power requests did not go to the requested power. In many instances the rotation stage moved in the wrong direction.
Matt E., James R., Sheila D., Terra H., Hang Y.
We are developing a tool to automate lock loss analysis, and an initial version is availabel at
/opt/rtcds/userapps/trunk/isc/common/scripts/lockloss
There is a matlab version and a python version, and right now they are essentially equivalent. An example generated using the python version is attached to this entry.
=================================================================================================================================
The code takes in a gps time around which a lock loss happens. It first looks through GUARDIAN to get a rough time of lock loss. Then it looks at 'LSC-POP_A_LF_OUT_256_DQ' to find a sudden drop, and the time of this drop will be used as our more accurate lock loss time. It then reads in data from an user defined list of channels (or if none, all DQ channels by default) and breaks the data into three frequency bands (>0.125 Hz for low, 16 ~ 128 Hz for mid, and >128 for high). The code can identify big jumps in each channel's BLRMS and order the channels by the time of the jumps. The first 16 channels goes wrong will be plotted (as the attached one) and all channels in which big jumps are seen will be recorded into a .txt file.
In the figure attached (a lock loss at GPS time 1122672396 UTC), black lines are raw data of 'POP_A_LF_OUT_256_DQ', blue lines are the raw data of the channel in the corresponding subplot, and red lines are BLRMS for the band that first goes wrong. X-axes are time in second, with zero defined as the time of lock loss (value shown in the main title), and y-axis is an arbitary scale as we only care about trend but not absolute values. Each subplot's title consists of three parts: channel name, time of the first jump prior lock loss, and the frequency band that first goes over the critical value.
Checking all DQ channels take a considearable amount of time, so in the attached one we looked only at suspension channels. A full analysis is on going when this entry is created.
=================================================================================================================================
For future imporvement, we are working on creating a cron job which periodically check GUARDIAN and whenever a lock loss happens this analysis code will be automatically triggered. Sheila and Terra are working on a complementary code to this which looks all channels that saturates or hits some known bad zone before lock loss. More ideas/ thoughts/ comments are welcome.
Patrick, Jeff, Evan
We spent a few minutes cooking the IX compensation plate while trying to make the TCS rotation stage behave.
This was after the Beckhoff chassis was power cycled.
If after the beckhoff chassis is cycled and if you do not first "refind home" then you will find these issues with the rotation stage not knowing what angle to go with as it loses its mind where its at. I cant say I have noticed the waveplate acting up after this step is taken (but maybe it has without me noticing)
Also note that search for home does not take the waveplate to minimum angle necessarily. You should be pressing "go to minimum power". So after a Beckhoff restart, usually here at LLO we search for home first, so that the rotation stage finds its 0 point again, then go to minimum power and then start operating it from there. A known issue all along is that as the waveplate rotates to home it could go through a brief period where it allows maximum power into the IMC or onto the ITM CPs
Shivaraj, Darkhan
We set a high frequency Pcal line at 3001.3 Hz using PCALX. The amplitude of the line is factor of ~5 smaller than the DARM noise spectrum at that frequency (July 21 reference).
SDF_OVERVIEW has been updated accordingly.
Defined NOMINAL states for nodes that were missing them:
ISC_DRMI: DRMI_3F_LOCKED
ALIGN_IFO: SET_SUS_FOR_FULL_LOCK
All nodes must have NOMINAL states defined for correct state reporting by the IFO node.
I also changed the NOMINAL state for IMC_LOCK to correspond to the new ISS on state:
IMC_LOCK: ISS_ON
On July 14th, we reported in LHO alog #19163 that the DARM actuation after the vent had not changed by more than 10% from its ER7 value. Responding to a request from LeoP, we made a similar plot for the period from June 28 until now.
The attached plot has 60s-long FFT data plotted in blue and 30-point running averages of the blue data plotted in red. The data includes all times when the Guardian State Vector value was more than 501 (DC Readout).
The data were generated using the low frequency calibration lines (~ 35 Hz) which had low SNR before July 22nd. We increases the SNR of our calibration lines to about 100, informed by an ER7 calibration line study, and reported in alog #19792 and the attached comments. Here we have also accounted for the fact that the response function of DARM, C/(1+G), is frequency dependent (calibration lines being few Hz apart) using the DARM refrence model for ER7.
Added a few tests today and removed ITMX/Y L2 OLDAMP Off notification.
Tests added are:
As always, please be aware of these notifications and always investigate when one pops up.
Carlos, Jim, Dave:
after the re-install of the original h1susey front end computer we have not seen any IOP TIM and/or ADC errors, though it is too early to declare success. To show these errors if they were to appear I am no longer performing a periodic (every minute) DIAG_RESET of EY. However I am still doing this for EX since that system still gliches.
Dave, Jim:
at 16:20 PDT we restarted the DAQ for a new EDCU ini file. This was not a clean restart, resulting in a confusing overvew screen with some front ends not registering any DAQ status channels. After a clean shutdown and startup of monit on h1dc0 and a manual start of h1nds0 and h1nds1 we got back to a stable system. We have noticed that in the past week monit sometimes has not cleanly restarted the data concentrator daqd process. Looking at today's log files and configuration file cache it appears monit and/or start-stop-daemon got into a race condition which resulted in daqd being restarted before the complete set of configuration files was ready. We will have to think more about how we got into this condition.
We checked that the current DAQ has the correct INI and PAR files and the frame sizes are correct.
I have set the masks (and reloaded the config files) for the ITMX spool, ITMY spool, ETMX and ETMY cameras. Note that these are not the cameras that are used during initial alignment.
The idea is that with an appropriately set mask to exclude extraneous light on the cameras, we can use the centroid tracking to monitor the spot positions on the test masses over long periods. These cameras are saturating over the whole central region when the arms are on resonance, so the centering calculation won't be perfect, but hopefully it'll still give us an idea of what is going on.
None of these had previously had a mask defined. Finding snapshots to help set the mask center and radius was what set off the confusion and investigation earlier. Elli's script was immensely useful in plotting the images for setting the mask.
Plots below are snapshots from 28 July 2015, around 10:00utc when the IFO was locked. Overlaid are circles (even though they look like ovals because of the non-square grid) that are equivalent to the masks that I've chosen. We will see images from inside the circle, and exclude the portions of the image outside the circle.
The centroid values are already trended as slow channels, so we can start looking at the history tonight, particularly if we re-do some thermal tuning tests.
Patrick, Cheryl, Nutsinee
Most of the logs were hand-written by Patrick. If things don't make sense that means I didn't decode his handwriting correctly.
- All time in UTC (PDT) -
14:58 (7:58) Peter getting something out of H2 PSL enclosure
15:02 (8:02) Jeff to LVEA colect CC Data
15:05 (8:05) Shiva and Sudarshan to EX
15:08 (8:08) Filiberto cabling started
15:18 (8:18) Joe check eye wash in LVEA
15:19 (8:19) Peter done
15:20 (8:20) Jeff done
15:34 (8:34) Ed to EY to start TCS
15:35 (8:35) Christina, Karen to EY (cleaning)
15:50 (8:50) Hannah, Elli install TCSY temp. sensor
15:58 (8:58) EX, FIl and Gerado start regen process
16:07 (9:07) Jason to optics lab
16:19 (9:19) Robert starting PEM
16:21 (9:21) Karen, Christina to EX
16:22 (9:22) Thomas, Shiva, ?
16:23 (9:23) Ed temp. back looking for drillbit
16:26 (9:26) Elli, Hannah back
16:27 (9:27) Filiberto and Andrea done, not out
16:30 (9:30) Jason start PMC transfer function
16:31 (9:31) Ed to EY
16:35 (9:35) Jim and Carlos to EY to replace H1scsey fornt end computer
16:37 (9:37) John, Elli, Hanna to LVEA
16:39 (9:39) DAQ restart
16:52 (9:52) Guardian restart full machine
16:54 (9:54) Karen, Christina leaving EX
16:59 (9:59) Robert into PSL
17:00 (10:00) Elli, Hanna to TCS ITMY
17:01 (10:01) DAC PEM CS model restart
17:09 (10:09) Cheryl to PSL
17:10 (10:10) Ed leaving EY and going to EX
17:15 (10:15) Leo about to start charge measurement at EY
17:20 (10:20) ASC-IMC, OMC, CSC model restart (Kiwamu, Keita)
17:35 (10:35) Kiwamu restart H1LSC model
17:36 (10:36) H1 ASC IMC restart (Kiwamu)
17:45 (10:45) Kyle to EY to shut off calibration gas
17:45 (10:45) Nutsinee to help Elli in the LVEA
17:51 (10:51) DAC restart digital video computers
17:53 (10:53) Karen and Christina cleaning in LVEA
17:55 (10:55) DAQ restart
18:00 (11:00) Fil to EX, fix UIM(?) drive
18:13 (11:13) DAQ restart
18:21 (11:21) Robert in PSL
18:24 (11:24) Parada water delivery
18:25 (11:25) Joe done
18:27 (11:27) Robert to LVEA to continues setting up PEM (?)
18:38 (11:38) Gerado and Kyle vack to LVEA
18:39 (11:39) measure power on photodiode
18:45 (11:45) Dave restarting camera servers
18:46 (11:46) Ed done at EX, going to LVEA
18:50 (11:50) Hugh HAM5, testing SDF
19:10 (12:10) Robert done
19:12 (12:12) Ed done
19:12 (12:12) Cheryl done
19:19 (12:19) Fil done
19:28 (12:28) Jim, Hugh, filter bugs in HEPI
19:29 (12:29) Pcal done
19:45 (12:45) Rick back to EX for laptop
19:59 (12:59) TJ, Betsy derail(?) ETMX high voltage driver
20:11 (13:11) Kyle back to EX
20:18 (13:18) Ops computer restart
20:26 (13:26) TJ, Betsy done
20:36 (13:36) Gerado to work with Elli
20:41 (13:41) EX inf. con. gage off
21:00 (14:00) ALS COMM, DIFF, XARM, and YARM nominal state is now SHUTTERED (Sheila)
21:38 (14:38) Fil to EX - try to fix ESD drive
22:13 (15:13) ESD working - Fil going to Mid Y
22:27 (15:27) Jeff to EX to restart high voltage
22:40 (15:40) Jeff coming back
0:53 (17:53) Patrick and Stefan still recovering the ifo.
Happy Maintenance Tuesday....
The old LSC_CONFIGS guardian node, which was defunct but hadn't been properly destoryed, was briefly resurected after the h1guardian0 machine reboot. It then tried to managed some nodes, which caused a bit of confusion.
I "destroyed" the node so it won't bother us anymore:
guardctrl destroy LSC_CONFIGS
Replaced the power switch in the UIM coil driver S0900303 at EX. Unit was reported to be found in off position several times in the past few days. We will monitor unit to see if it switches off again.
Per IO ECR E1300432:
Components to be changed in the IO path on the PSL:
Status: not complete
Other information collected:
PD Channels:
H1:PSL-EOM_A_DC_POWER - thorlabs behind IO_AB_W1
H1:IMC-PWR_EOM_OUTPUT - thorlabs behind IO_AB_W1
PD,measured power:
IO_AB_PD3, thorlabs PDA55, 3.33mW
Incident on BS:
before the splitter, IO_AB_W1, 3.51mW
PD,measured power:
IO_AB_PD1, Newport 1811 RFPD, about 50uW - two beams, fast measurement, needs more time to set up
PD Channels:
H1:PSL-PERISCOPE_A_DC_POWER - bottom periscope trans
H1:IMC-PWR_IN_OUTPUT - bottom periscope trans
PD,measured power:
IO_AB_PD2, thorlabs PDA55, 83uW
Incident on steering mirror:
before steering mirror IO_AB_M9, 85uW
Incident on BS:
power in non-PD path, coming off of IO_AB_M10, 85uW
After getting the ETMx ESD driver unrailed, we found the Driver state "OFF" via the red/green ESD Active indicator light. Nice timing all day - the ESD keeps dying just ahead of us trying to use it!
So, after we returned from the railing saga, we resumed running a realignment. A few mins in we discovered the red light indicating the driver was off. We toggled some switches to no avail and Fil headed to EndX again. He found the power supply tripped. He tried to reset it and it tripped again. After hunting around for another 20mins, it occured to us to check the Beckhoff - yep, it had crashed. (The pressure guage that the ESD looks at lost connection and therefore shut down the power supply of the ESD for safety.) Patrick restarted the Beckhoff, Fil then reset the ESD power supply, and the ESD finally looks healthy again. For now.
TJ is adding these failure modes into the SYS (alarm) guardian.
Added H1:PSL-ISS_REFSIGNAL to the set of monitored channels in SDF. Accepted value of -2.06.
restart log for Tuesday 04Aug2015 is attached. No unexpected restarts.