My bad, after the round of SSH fixes to get the 2FA auth running again on cdsssh and cdslogin I forgot to restart the cell phone texter service on cdslogin. Gerardo noticed that we didn't get our daily 1pm vacuum summary texts today, which lead to the discovery that the service had been down since 4pm yesterday.
During troubleshooting of various stages of relock this afternoon, we noticed the ETMx ESD looked "funny" so Richard headed out to EX this time in search of a cure. By "funny" we mean the 4 QUAD signals were the same sign as the DC - normally they are opposite. As well, when Richard got to the end station, he found the HV ON/OFF switch was being toggled (audible at the rack) repeatedly so he unplugged the Binary cable which stopped it. He toggled the HV ON/OFF a few times to see about clearing the sign issue. Then, after a round of conversation, we found the BIO L3 HI/LO VOLTAGE and HI VOLT DISCONNECT were in the OFF positions on the ETMx - this is an incorrect state for acquisition. So, I have written into the SAFE.snap to restore the ETMx to the HI Voltage state after a restart. AFter all of the rack manipulations and a correct set of MEDM switches were thrown, the ETMx ESD seems healthy. On with locking attempts.
Chris S. The aluminum material for the beam tube enclosure repair arrived and Chris has started cutting it to length and punching holes for install.
Scott L. Ed P. 8/17/15 48.2 meters of tube and bellows cleaned ending 8.4 meters east of HSW-2-002. 8/18/15 20.8 meters of tube cleaned ending at the west wall of Mid-Y. The crew will stop at this point until after the O1 Run. All equipment and cords will be thoroughly cleaned and put away for storage.
I have added the CAL (Maddie) and DETCHAR (Ryan F) requested channels to the H1BROADCAST0.ini file and restarted h1broadcast0 to use the new configuration. 140 channels were added, none removed. The list of added channels are attached.
H1BROADCAST0.ini was committed to svn as version r11327
One of the things that is slow and not optimized well about our inital alingment is the lock of the laser to the Xarm. This is not a hard problem, just one that we have never taken the time to work on. This was one of the problems (annoyances) that the relocking team ran into this afternoon. I had a look at the data from a sucsessfull acquisition, and can see two problems that have simple solutions.
1) It takes too long to ramp down the feedback from the IMC locking loop. We have a 3 second pause and a 2 second ramp on this gain. While both loops are on, they fight each other and the lock is unstable (see attached screenshot). I would suggest something like a half second for each of these, or even less.
2)We had a 0.1 second delay on the FM triggering that turns on boosts. The other day I had reduced this from 0.2, but it is still too long. We should probably try a shorter FM delay or no delay.
J. Kissel, D. Barker The state of the H1PSLISS.txt foton file was in a state of confusion. A diff between the last several back ups, shows very small changes (the 14th significant digit) to the gain coefficient in about 12 filter modules. Sudarashan ensures use that he's only been making changed to the ISS_SECONDLOOP_SUM14_DC ISS_SECONDLOOP_SUM58_DC filter banks, yet there are these small changes to several filter modules in the following banks reported over the past few backups: ISS_SECONDLOOP_QPD_SUM ISS_PDB_LSD_INTEGRATION ISS_PDB_LSD_BANDPASS ISS_PDB_CALI_AC ISS_OSCILLATION_MON_LP We don't understand it, and we can't trace it. However, because the changes in the uncertain filter banks appear to all be small, we've decided to just "start from scratch" and load the whole filter en masse. So, the current archive, in /opt/rtcds/lho/h1/chans/filter_archive/h1psliss/ H1PSLISS_1123967724.txt represents the currently loaded filter coefficients. Further, the current filter file, in /opt/rtcds/lho/h1/chans/H1PSLISS.txt which is a soft link to the userapps, /opt/rtcds/userapps/release/psl/h1/filterfiles/H1PSLISS.txt has been committed to the userapps repo.
Kiwamu, Hang
We investigated the nonlinearity in the esd control signal using channel 'SUS-ETMY_L3_MASTER_OUT_UL_DQ'. In the upper panel of the attached plot, the 'linear' is the product of control signal and the bias, 'quadratic' the square of control signal, and 'total' the sum of these two. We did not do an absolute calibration but the relative magnitude should be good enough for the purpose of this study. In the lower panel we showed the coherence between the linear and quaduartic timestreams, with 512s of data and 64 averages.
From the result we can see that the nonlinear term is well below the linear component, and the coherence was low. As a result, it should be safe to use the current ESD configuration without worrying too much about the quadratic term.
Mid maintenance, the ETMy ESD tripped off. I toggled the HV ON/OFF switch on the SUS screen which tunred it on, but in the railed ~-16k state. I toggled the switch slowly a few times but the railing never cleared. Richard went to EY and did the old "Power OFF, pull DAQ plug, re-power ON, re-plug in DAQ cable" trick. It's back now.
J. Oberling, P. King
PMC Alignment
After the last week's temperature spike the transmitted power through the PMC had dropped (see alogs 20490 and 20504), today's goal was to fix it. First Peter attempted to adjust the temperature of the PMC, as it had not settled back to its original temperature from before the temp spike. See Peter's alog here for full details, but the short version is that temperature is such a slow actuator and we are so limited on time that we decided to go in and adjust the beam alignment into the PMC.
With the ISS off we tweaked mirrors M06 and M07. Pitching M06 down slightly gained us the lost PMC transmission. Walking the beam slightly up and down in pitch and left and right in yaw did not improve the transmitted power, so after re-locking the adjustment screws we had:
We checked the centering on the PMC_TRANS PD and after a very slight adjustment to mirror M36 the transmitted power increased slightly to 25.29 W. We measured the visibility of the PMC:
With the ISS on:
We then measured the transmitted PMC power and the power after the ISC EOM with a power meter.
The power out of the ISC EOM is displayed on the IMC Overview MEDM screen, in the lower left corner. On the PSL table this is IO_AB_PD3. It read 25.81 W, so this needs recalibration. We blocked the PD and saw 0.175 W on the MEDM screen, so there is some offset we need to remove as well.
O1 Prep
We turned off all the computers inside the enclosure and removed the 1 cordless phone (it is currently sitting on the desk on the west side of the H1 PSL enclosure next to the PSL electronics rack). We shut off the DBB electronics in prep for O1; these will need to be turned back on if we want to run DBB scans during Tuesday maintenance. Once done in the LVEA we asked Richard to turn off power to the clean room phone located in the PSL enclosure. All HEPA fans for the enclosure are off, as well as both air conditioners. The make-up air is set to 20%, and this was confirmed by listening to the fan (it is located in the old chiller closet on the east side of the H1 PSL enclosure).
IO_AB_PD3 Recalibration
J. Oberling, P. King, C. Vorvick
We recalibrated IO_AB_PD3. To remove the measured 0.175 W of offset we set the offset value on the filter screen to -77.0 (440 counts/W with a gain of 1). We then set the gain to 0.916 to get the MEDM window to match our power measurement. If we want more power from the PSL our only course now is to increase the front end diode currents.
just when we thought we had fixed the monit autostart of the daqd process on h1dc0 it again failed to start up after a DAQ restart this morning (following model changes). Using the monit web page, I was able to stop and start the daqd process manually. I verified there were no duplicate processes running and the PID file was correct. There is a DAQ frame gap due to this problem from 09:13 to 09:51 PDT (between GPS times 1123949568 - 1123951808).
Charge measurements was done on both ETMs. It is early enough to discuss the new tendencies, nevertheless it seems like ETMY change the sign of charging after changing the bias sign on ETMY (see 20387) while ETMX charging is the same (as well as the bias sign). Now (since Aug,10) both ETMX and ETMY Biases are -9.5V, Plots are in attachment.
After the QUAD front end restarts, we found that some of the DARM DAMP MODE loops were blasting the suspension. In these cases, the FE was restoring non-zero gains as per the safe.snap settings file. Since Jenne confirmed that the GUARDIAN sets these gains and turns these loops on, I have written all of the gains to be ZERO in the safe.snaps for future restarts.
as part of the SR3 investigation, we restarted the h1iopsush56 model to re-calibrate the 18bit DACs. The procedure was: stop user models, stop IOP model, start IOP model, check the status of the 18bit DAC autocal, clear the SWWD DACKILLs to stop the SEI SWWD countdown, start the user models one at a time.
Looking at the AUTOCAL history, this restart and the previous 3 restarts since the last reboot have all successfully calibrated all 5 DAC cards in the correct amount of time.
following the h1susetm[x,y] model changes this morning to remove the cpu intensive violin mode calculations from the models, the models are now running faster. Looking at second trends over the past few hours, the CPU max ranges before and after the change are shown below
sus | before | after |
ETMX | 57-64uS | 53-58uS |
ETMY | 57-62uS | 54-57uS |
the improvement is in the range of 3uS to 4uS. The TIM bit of the STATE_WORD triggers above 62uS, so we would not expect any more TIM errors on the CDS overview screen.
Filiberto, Richard, Elli
A filter was added to the HWS power supply. The idea is that the filter module will cut out the current spikes from the HWS that were showing up in the PEM channels. We will test this today.
The filter is in a box labeled "HWS filter for power supply" and is located inside the HWS table by HAM 4, on top on the HWS corner breakout box. The filter connects between the HWS Corner Breakout Box (refer to D1200934 and E1100892).
I can't find any evidence of 1Hz lines or glitches when the ITM HWS cameras are on. The filter module appears to be doing its job.
There is no 1Hz glitxhes in the ITMX GS13 when the HWS is on. These glitches were bothering detchar during ER7 (alog 18548). The attached spectrogram can be compared with that in alog 18548. Left spectrogram is HWS off, right plot is with HWS on.
There is also no increase in the 1Hz line in the corner station magnetometer spectrum when the HWS cameras are switched on. (Spectrum attached. "Live" spectra were taken with camera off, reference spectra were taken with camera off.)
Evan, Dan, Jeff
The SR3 M1 T2 actuator is...unwell.
We started to have trouble staying locked after power up about four hours ago. At first it looked like an ASC problem in the SRC, and since the AS36 loops are the scapegoat du jour, Evan started to retune the SRC1 loop. He observed a recurring transient in the beam spots at the AS port, and saw the same thing in the SR3 oplevs. We turned off the SR3 cage servo and the kicks kept on coming.
Eventually we looked at the SR3 M1 voltmons and found the first plot attached, which is for eight hours. The T2 OSEM and voltmon started to get ratty about three and a half hours ago. The noise is a series of slow transients with a several-second rise and a steep decay. See Fig. 2.
We're pretty convinced that it's an actuator problem, something between the DAC and the voltmon readback in the coil driver box. We have unlocked the IFO and turned off the SR3 top-stage damping and we still see the pattern of noise. We shall leave the pleasure of power-cycling the AI chassis and coil driver to others.
As an amuse bouche for the maintenance day team, we also discovered that the SR3 M1 T1 voltmon is complete nonsense. The T1 voltmon time series is a collection of step functions and spikes, 100x larger than the other M1 voltmons.
The SR3 M1 damping has been turned back on.
It would appear that the glitches come and go but tend not to be gone for too long, but enough to slow down trouble shooting. In and effort to get things moving I replaced the Triple Top driver S1001082 with S1001086 as the first effort at fixing this problem. Do to a lot of activity (it is Tuesday) it was hard to see if this fixed the problem. As can be seen from Dave B. report we also restarted the IOP in make sure a calibration occurred. Had what appeared to be glitches in the signals that turned out to be sei trips. So again not the easiest item to follow up on. After the system settled down I have not seen any excess noise on T2 for over and hour. But will continue to monitor.
Seems like we've been good for the past 2.5 hours or so.
As per LIGO-M1500250, H1 has now been put into "OBSERVATION MODE". This means:
The ODC-OPERATOR_OBSERVATION_READY bit, which is set to "Undisturbed" by the operator on the GUARD_OVERVIEW screen, will now be automatically unset by the Guardian IFO top node if any of the monitored guardian nodes drop from OK==True.
The most likely ways that this can happen are:
Operators must be aware that the INTENT bit must be manually reset after any of these changes.
The list of Guardian nodes being monitored is stored in:
USERAPPS/sys/h1/guardian/IFO_NODE_LIST.py
The current monitored node list is:
ALIGN_IFO ALS_COMM ALS_DIFF ALS_XARM ALS_YARM HPI_BS HPI_ETMX HPI_ETMY HPI_HAM1 HPI_HAM2 HPI_HAM3 HPI_HAM4 HPI_HAM5 HPI_HAM6 HPI_ITMX HPI_ITMY IMC_LOCK ISC_DRMI ISC_LOCK ISI_BS_ST1 ISI_BS_ST2 ISI_ETMX_ST1 ISI_ETMX_ST2 ISI_ETMY_ST1 ISI_ETMY_ST2 ISI_HAM2 ISI_HAM3 ISI_HAM4 ISI_HAM5 ISI_HAM6 ISI_ITMX_ST1 ISI_ITMX_ST2 ISI_ITMY_ST1 ISI_ITMY_ST2 OMC_LOCK SEI_BS SEI_ETMX SEI_ETMY SEI_HAM2 SEI_HAM3 SEI_HAM4 SEI_HAM5 SEI_HAM6 SEI_ITMX SEI_ITMY SR3_CAGE_SERVO SUS_BS SUS_ETMX SUS_ETMY SUS_IM1 SUS_IM2 SUS_IM3 SUS_IM4 SUS_ITMX SUS_ITMY SUS_MC1 SUS_MC2 SUS_MC3 SUS_OM1 SUS_OM2 SUS_OM3 SUS_OMC SUS_PR2 SUS_PR3 SUS_PRM SUS_RM1 SUS_RM2 SUS_SR2 SUS_SR3 SUS_SRM SUS_TMSX SUS_TMSY TCS_ITMX
Note to Operators: ODC-OPERATOR_OBSERVATION_READY bit, which is controlled by the button labeled "undisturbed" on the GUARD_OVERVIEW.adl screen, is distinct and different from the H1:ODC-OBSERVATORY_MODE channel, which is set from the OPS_OBSERVATORY_MODE.adl screen. The ODC-OBSERVATORY_MODE describes the activity (wind, preventative maintenance, commissioning, etc.) and is used to generate our time spent pies. These are summarized in the DetChar summary pages at:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20150817/lock/operating_mode/
The Operators' priority is to the GUARD_OVERVIEW undisturbed button, which signals the readiness for scientifically valid observing. The selection of the LHO operating mode to "OBSERVING" (in the OPS-OBSERVATORY_MODE.adl screen) can be done after the pressing of the "undisturbed" button. After a lock loss, the operator should select the appropriate LHO operating mode activity.