Between yesterday and today, Gerardo and I (well mostly Gerardo, I just spot checked and helped with calculations) successfully bonded an ear onto the S3 flat of ETM11. The ear is s/n 188. By successfully, we mean that the surface area coverage and post-bond locational dimensions meet the spec.
We had an interesting find when we opened the optic can this morning, however. Upon dismantling the cake tin container from around the ETM11 as usual, we lifted the oring plate off of the AR surface and found one of the inner handle screws was left sitting upright on the surface of the optic. It must have spun loose during shipping and jostling around the lab. SYS/COC have been alerted to this failure mode of the container. This is the first optic we have started to process which has no first contact on it after long-term storage in the container. After carefully removing the screw, I inspected the area and miraculously could not find any macroscopic marks or scratches.
Following up on the loud wandering line around 1180Hz seen in the May 3rd lock (figure 1), we see this line in DARM as coherent with similar lines in SRCL, PRCL, MICH, IMC-I, IMC-F and PSL-FSS_TPD_DC_OUT_DQ. figure 2 shows the stamp-pem coherence matrix for the ~90 minutes of lock from 1114597337-1114600937 where each cell represents coherence for a single frequency and channel. The coherent channels are labeled in the plot.
In addition, representative cross-power spectrograms (where color represents STAMP SNR, effectively a measure of coherence) for an hour of data in this time for SRCL and PSL-FSS_TPD_DC_OUT_DQ can be seen in figures 3,4 that show the nature of this line. There is also a strong wandering line between 1150 and 1160 Hz that is seen in PSL-FSS_TPD_DC_OUT_DQ (PSD, figure 5) and that is also coherent in DARM (see figure 4). This line, however, is not obvious in the DARM PSD spectrograms.
These PSD and cross-power spectrograms were made using 60s ffts and coarse-graned to 1Hz frequency bins.
Each of PRCL, SRCL, and MICH also show coherence with the 1150Hz stationary line seen in DARM, while the IMC and PSL channels do not. See (18277) for a discussion of this line.
The Sensor was locked, moved, oriented, leveled, iglooized, on concrete, repowered and masses pushed. Looks like it needs another push (manual says many pushes are sometimes required.)
There was some evidence that part of the cabling could be the problem we are seeing with the HAM2 STS. This move will allow us to test the main house field cable. We can still check the Sat Box and Cable by exchanging them for the B unit which appears good with the newly overhauled PEM Vault unit. The Chassis can be check too, one step at a time.
The noise issue that has been troubling this machine is still with us--see attached. The noise is marked.
The Satellite Cable (orange) and Box have now been swapped (HAM2 & ITMY) to see if the noise shifts.
Another look after swapping the satellite box and cable has the problem still with the seismometer. I've now swapped (1615pdt) the signals going into the Interface Chassis. Will check after a time.
Because Hugh wanted to move STS-A into the biergarten to do a huddle test, I've switched the input HAMs (1-3) to running sensor correction with STS-C. This means all corner station seismic platforms are now running off the same seismometer, as the BSC's were also moved over to C when we substituted and evaluated the biergarten STS. Will probably run like this for at least the next couple of days.
At Kiwamu's request, I have updated his PD Null script, pdOffsetNull_ver2.py, located in opt/rtcds/userapps/release/lsc/h1/scripts, to include the balance of the LSC PDs. The list of PDs zeroed by this script is now:
'LSC-POPAIR_B_RF18',
'LSC-POP_A_RF9',
'LSC-POP_A_RF45',
'LSC-POPAIR_A_RF9',
'LSC-POPAIR_A_RF45',
'LSC-POPAIR_B_RF90',
'LSC-ASAIR_B_RF18',
'LSC-ASAIR_B_RF90',
'LSC-REFLAIR_A_RF9',
'LSC-REFLAIR_A_RF45',
'LSC-REFLAIR_B_RF27',
'LSC-REFLAIR_B_RF135',
'LSC-REFL_A_RF9',
'LSC-REFL_A_RF45',
'LSC-ASAIR_A_RF45'
'LSC-X_TR_A_LF',
'LSC-Y_TR_A_LF',
'LSC-TR_X_QPD_B_SUM',
'LSC-TR_Y_QPD_B_SUM',
'LSC-POP_A_LF',
'LSC-REFL_A_LF',
'LSC-POPAIR_A_LF',
'LSC-REFLAIR_A_LF',
'LSC-ASAIR_A_LF',
'LSC-POPAIR_B_LF',
'LSC-REFLAIR_B_LF',
'LSC-ASAIR_B_LF'
The changes have been committed to the SVN.
While looking over SDF Diffs after running the script, I noticed that the offset for ASAIR_B_RF18 (both I and Q) changed from ~0 to ~300 (a few order of magnitude), whereas the other PD offsets changed little. Just a heads up.
Sheila and I found that the dark offset for LSC-TR_X_QPD_B_SUM changed from −0.9 ct to −37.2 ct at 11:41:57 local this morning. Was this when the script was run? This value is way too big.
The nds2 channel scanner is failing because a channel (H1:TCS-ETMY_HWS_PV_UPPER_THRESHOLD) has a line-feed character appended to the name. This is being propagated into the frames and makeing any list of channels unreadable.
Nutsinee, Robert, Elli
We have temporarily switched the EY HWS to a different power supply to see if this will eliminate the 57Hz coupling into DARM. The HWS is currently powered using a cable running from the big transformer box at EY. The EY HWS is currently switched on, and can't be switched off remotely in this current configuration. We plan to returm the HWS to its orgininal power supply tonight or tomorrow morning.
I have updated the safe.snaps for PEM and OAF models. These systems now have greened SDF screens.
I have "greened" up the IOP models SDF settings.
New STS-2 cable has been pulled from the CER SEI racks to the PEM area in the Beer Garden. This is to help with the huddle testing of the STS-2's in the VLEA. Work areas included HAM4, BSC2, and BSC3 where the cable was dressed in the cable tray. Work started around 8:30 and ended around 9:30.
This morning Richard and I tested the UPS used for the PSL Mephisto power supply. It is now set to log what happens. A run calibration was also done. The UPS claims that it can run for ~5-1/2 hours on the battery. We initiated a self-test, for which the UPS switched over to the battery. After a minute or so I checked the laser to find that the Mephisto power supply and frontend laser control box displayed an error condition. Interestingly the MEDM display indicated a flow sensor problem. The Beckhoff PC indicated an interlock problem and an EPICS alarm but nothing about a flow sensor problem. Both the diode and crystal chiller were still running when I entered the diode room to reset the system. Clearly the Mephisto tripped because of the UPS switching over to its batteries. Richard, Peter
Timing is pretty good, the ISIs tripped within a few seconds of each other and the HEPI tripped 16 seconds later. No issues on recovery.
So why didn't other platforms trip? Here are time series of the ITMY's sensors. The BS & ETMX tripped on Actuators. For ITMY, most of the signals show a characteristic Earthquake arrival sequence of P S & Surface waves. The ITMY actuators (first graph) maxed at maybe 14000 (trip at 30000) and the T240 signals did approach 20000cts (second graph.) Other sensors see the EQ signal but they do not exceed 1/3 of max.
Sheila, Evan
Evan struggled with the PSL tripping earlier tonight. After it stopped tripping we were able to lock long enough to get to low noise and make one DARM OLG measurement, which is attached here. Then we were hit by a large earthquake.
I made an attempt at making a filter for SRCL FF, using alpha to go the the DARM actuator For the SRCL OUT to DARM IN tf we used the noise injection from last night, screen shot of the TF is attached.
To get alpha, I took SRCLOUT-> DARM IN tf *(DARM OLG/(DARM Closed loop*DARM plant) which is (DARM IN1/SRCL OUT)*(DARM IN1/DARM IN2)/((DARM IN1/DARM EXC)*(DARM IN1/DARM OUT)). for the DARM measurements I used a cut of 0.6 on the product of all three coherences, from SRCL to DARM we have I used a cut of 0.8 on coherence. I fit the product using vecfit, the attached screenshot shows the result when using 8 zeros and 8 poles, anyone can adjust the number of zp pairs by editing line 58 in /sheila.dwyer/Noise/SRCLMICHFF/SRCL_DARM_TF.m. I did not load any filters.
Tripped around 2015-05-07 02:31:30 UTC. I reset it. Aside from the usual diode flow bit flipping issue, the diode chiller again appeared to have some unphysically fast jump in temperature for about 30 s.
PSL tripped again, around 2015-05-07 02:47:50 UTC. This time, the computer in the diode room showed a momentary dip in the diode chiller flow rate (see attached photo). I talked to Rick on the phone, and we agreed to restart it again.
PSL tripped a third time, around 04:23:00 UTC. I reset it again.
These trips are really an impediment to progress, not only because of the time it takes to restart the PSL but also the time it takes to relock the IFO and get back to low noise. If it is an option to replace the bad sensor soon, that would be good, although I think I remember from when Michael Rodruck was struggling with the same issue it was not a simple fix.
These recent PSL trips, as well as the power watchdog trips, are an example of how we have made the system less robust by being overzealous in trying to protect it.
Both the recent trips and many trips a few years ago when Michael Rodruck was still here were due to bad sensors. These sensors broke and indicated that the flow was low, when the flow was not actually low and there was no real danger to the laser.
Do we have any redunant sensors, so that we could check that mutliple sensors agree that the flow is low before we turn off the laser?
We see H1:PSL-IL_DCHILFLOW flipping for a while and then it stops flipping after 30 seconds in these recent spurious laser trips. Does this bit go to 0 and stay there if the flow is actually low? If so we could wait to trip until this bit is low for at least half a second, or when a rolling average of the bit drops below half.
The power watchdog seems like another example of overzealous protection of the hardware. (The power watch dog has tripped less often recently, I assume that is because the PSL maintence is now being done) This is a watchdog that goes off when the PSL power becomes low. It seems to me like a drop in the power is not a dangerous situation. I think it would be best to downgrade the power watchdog to an alarm or warning, or anything else that does not bring a multi million dollar facility to a standstill.
Sheila, Evan, et al.,
These laser trips are, of course, a big concern to the PSL "team." We are actively pursuing solutions.
For the past month or so we have been opeating without engaging the watchdog on the Front End laser.
We are trying to understand the functionality of the Beckhoff interlock which is appears is responsible for initiating the shutdowns. This week, we received some information from Maik Frede at LZH. He wrote the control software. It appears that the system is not functioning as designed and described in LIGO-T1000005.
Yesterday, Peter King and Jeff Bartlett replaced the flow sensors in the spare chillers with units that don't have moving parts. We have some indication that the flow sensors originally installed and currently operating (the ones that Michael Rodrick) was replacing, are the cause of the triggers. As soon as possible, we plan to either swap in the spare chillers (with the new sensors) or shut down for an hour or so to replace the sensors in the operating chillers.
Jeff, Peter, Jason, Ed Merilh and I are meeting at 9:00 this morning to discuss progress on several fronts in trying to understand and address these shutdowns.
J. Kissel I've processed the DARM OLG TF measurement Evan and Sheila took last night (LHO aLOG 18269), and was dismayed to find that the DARM coupled cavity pole (DCCP) frequency has decayed back down to 270 [Hz]. This is obvious from the attached residuals, where I show two different versions of model parameters for last nights measurement compared against the two previous measurements taken during the mini-run, where the DCCP frequency was 355 [Hz] up near the expect value. I've again used a 0.99 coherence threshold, I trust this assessment of the DCCP frequency to within 2%, especially since that's the only thing I have to change in the model (besides the overall scale factor) to arrive at this conclusion (for the skeptics of my precision, see LHO aLOG 18213). What's going on here? --- Total Blind Speculation --- As Sheila mentions in the main entry (LHo aLOG 18264), the recycling gain and initial alignment had been restored to values during the mini-run. There has been quite a lot of work on SRC alignment: maybe those SRC loops which are now higher bandwidth -- though good for stable SRCL to DARM coupling (LHO aLOG 18273) are not so good for the DCCP frequency. Perhaps because they have some bad alignment offset? Perhaps we should try very slightly altering the DC alignment of these loops to see if it has an effect on the DCCP frequency, and then optimize for it. Eventually, one might imagine using the amplitude of either the PCal or DARM calibration lines as feedback to keep the pole frequency stationary and near the design value. The good news is that the overall scale factor (what we're assuming is either the optical gain or ESD driver strength variation) changed by less than 0.5%. --------------------- The measurement template has been copied over and committed to the CalSVN here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/DARMOLGTFs/2015-05-06_H1_DARM_OLGTF_LHOaLOG18269.xml The new model parameter set can be found committed here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/H1DARMparams_1114947467.m and as usual, the model is here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/H1DARMmodel_preER7.m and all residual comparisons are done here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Scripts/CompareDARMOLGTFs.m
I guess another possibility is some effect from the lower SRCL ugf. When the guardian goes into LSC_FF, the SRCL gain is reduced by 30% to reduce the amount of control noise appearing in DARM (the SRCL ugf goes from 50 Hz to 25 Hz). I'm assuming this reduction was not done during the minirun, when the good DARM pole was measured.
I suppose we should just run two DARM OLTFs in quick succession, one with the low ugf and one without.
Evan's idea seems unlikely: how could the SRCL loop, with a bandwidth of few tens of Hz, affect what happens at mich higher frequency? This would imply huge couplings of both DARM->SRCL and SRCL->DARM. Such lareg coupling should be easily visible when measuring the SRCL open loop gain.
The most likely hypothesis is that the pole frequency depends critically on the SRC alignment. One interesting test would be to inject two DARM lines, one at low frequency (50 Hz) and one at high frequency (1000 Hz) and track their variations while aligning the ITMs. We expect the low frequency to be constant (tracking the optical gain) and the high frequency to change (tracking the pole frequency).
We already inject calibration lines at 34.7 and 538.1 [Hz] which have been on since April 1 2015 (17621). They've been at a fixed amplitude that had been tuned for a 30 [Mpc] lock stretch, so the 34.7 [Hz] line may not be visible at all times, but still -- it's on the to-do list to make this comparison. Any off-site would be much appreciated!