J. Kissel

According to this morning's measurement of the DARM OLG TF (LHO aLOG 18352), comparing model against measurement -- which recall has a ~2.5%, 1 [deg] precision (see LHO aLOG 18186) -- the DARM Coupled Cavity Pole (DCCP) frequency was 303 [Hz]. 

Just to refresh everyone's memory, since the DCCP's frequency has been constantly evolving, I list below the measurement times and the resulting DCCP frequency for all measurements thus far. As mentioned in LHO aLOG 18352, now my goal to data mine as much information as I can, similar to what Kiwamu has done for just two of these measurements, see LHO aLOG 18298.

     Date UTC               DCCP Frequency [Hz]
'Mar 10 2015 03:41:52 UTC'        320
'Apr 02 2015 08:34:20 UTC'        320
'Apr 06 2015 23:45:13 UTC'        320
'Apr 13 2015 04:15:43 UTC'        290
'Apr 13 2015 06:49:40 UTC'        290
'Apr 15 2015 07:53:56 UTC'        290
'May 01 2015 18:52:59 UTC'        355
'May 02 2015 20:35:54 UTC'        355
'May 06 2015 11:37:31 UTC'        270
'May 07 2015 07:08:12 UTC'        260
'May 11 2015 15:50:43 UTC'        303


The attached comparison only shows the latest 5 of these measurements for clarity, because they extend out to the largest frequency range, increasing the confidence in the DCCP frequency.

Note this is also the reason why we've held off on updating the calibration of the CAL-CS (and therefore the GDS low-latency pipeline). 

Last week we put the EY HWS on it's own separate power supply to see if this would reduce the 57Hz coupling of the HWS into DARM.  This workes. When the EY HWS is on, we see no coupling into DARM at 57Hz. (See attached sprectum.)  EX is still on the shared power supply, and the 57Hz line is clearly present.  The current setup at EY is only as temporary setup.  It looks like we should permanantly move all the HWS onto their own power supplies.

On Sunday evening Elli and Nutsinee logged another PSL trip here.  Taking trends over this time shows that again the chillers appear to not be bringing down the PSL (first attachment).  I've also attached trends over the same time period of all the flow rates: diode and crystal chillers, front end, HPO heads, and power meters.  All these flow rates are where they are supposed to be.  The third attachment shows all the interlock signals we capture.  Only the H1:PSL-IL_OK trips, all other interlock signals indicate no problems.

Rick covered the PSL trip early Sunday morning in alog 18346.

Seismic: No activities
Suspensions: No activities
CDS: Finish making CPS cables – Will install at BSC1, 2, & 3 during the Tuesday window.
	     Continuing with work on AA/AI boards in H2 electronics building.
PSL: Tuesday work on RefCav alignment and Laser/Chiller tripping problem.
FMC: Painting in OSB 
Vacuum: Bake-out of RGA at End-X
Software: Work on timing distribution system  

Laser Status: 
SysStat is good
Front End power is 31.8W (should be around 30 W)
FRONTEND WATCH is RED
HPO WATCH is RED

PMC:
It has been locked 0 day, 17 hr 30 minutes (should be days/weeks)
Reflected power is 2.4 Watts  and PowerSum = 24.4 Watts.
(Reflected Power should be <= 10% of PowerSum)

FSS:
It has been locked for 1 h and 28 min (should be days/weeks)
TPD[V] = .93V (min 0.9V)

ISS:
The diffracted power is around 7.8% (should be 5-9%)
Last saturation event was 1 h and 29 minutes ago (should be days/weeks)

NOTES:

Noise study recap for HAM2, STS2-A instrument:

STS2-A Noise identified at orignal location (not first time) aLog 18207.

STS2-A moved to BierGarten (on concrete & under igloo) with different field cable--still looked bad, 18305, 18307.

Satellite cable and box swapped with STS2-B, noisy Y channel still with HAM2 channel--18314 (comment to 18305.)

Field Cable swapped at Chassis end (STS2-A & B signals swapped--bad channel should now be seen on the B channel data): Noise goes away, not seen on either channel!  Although the connector at Chassis for STS2-A had already been exercised when we moved to the new cable, this connection seems the suspect.  aLog 18324.

Spent 10 minutes wiggling and straining the cable chassis connection but could not get the channel noise to come back--18326, comment to 18324.  Chassis cables swapped back.  Signals still look good.

Friday afternoon I moved the STS2-A back to its home igloo by HAM2 with its orignal Satellite cable and box.  This morning, the noise has returned to STS2-A--see attached.

J. Kissel

The last few measurements of the DARM open loop gain transfer function have revealed that the DARM coupled cavity pole has gone back down to 270 [Hz] and 260 [Hz] (not posted yet), from where it was at the mini-run at a consistent 355 [Hz]. We're at the beginning stages of tracking this change in frequency dependence "online" by comparing high and low frequency PCAL and DARM calibration lines (and considering adding a few more to increase the precision of the analysis), but until that analysis gets into full swing, I'm taking every chance I can get to measure the DARM OLG TF.

Here's another data point. We're in the "LSC_FF" IFO state, with the request input power at 10.7 [W], MICH subtraction ("FF") was ON, but the SRCL FF has a gain of zero (!! -- forgive the file names "FFon," I hadn't actually checked until writing this log).

I'll have some more complete analysis (and an answer of what today's DARM coupled cavity pole frequency is) shortly.

Nutsinee, Elli

Today we noodled around a bit, but still don't have a measuremnt of SRC length.  Once Rick had reset the PSL trip, we spent of time recovering the alignment, locked DRMI, got were ready to take a measurement when the PSL tripped again.  Just like this morning, the PSL stystem status screen only showed the interlock ok bit was tripped.  The diode chiller flow bit was green and the chiller was working fine.  After talking to Rick,  we used the PSL card in the key storage box in the control room to get into the diode room and  reset the PSL.  We are starting to get tired, and are calling it a day.

J. Kissel

5.6 Mag Earthquake near Japan at 2015-05-10 21:25:46 UTC.

For quite some time, when an ISI's watchdog trips, it runs the equivalent of a "down" script via front-end code (see E1300685, installed circa Fall 2013). One of the many things the front-end down script does is to force the isolation feedback loop filter banks' gain to zero, 0.0. However, in the era of the Celerier system of SEI guardians, the ISI nodes are using some Jedi-Level-9 combination of the python tools in 
/opt/rtcds/userapps/release/isi/common/guardian/isigiardianlib/
to run the platform's isolation stages, and somewhere along the journey to "FULLY_ISOLATED," these filter banks are set to 0.1, and then 1.0 (and I don't think ever to zero, 0.0). Thus, when a watchdog trips, the new Set Point Monitor -- who thinks it's the only thing in control of these values -- sends a notification up the ladder that these gains to zero (0.0) have been changed by something other the guardian, i.e. the front end's down script. 

No big deal here, it just caught my eye while Elli and Nutsinee were asking me "Can I reset the SEI system... now? How about now?" after the earthquake mentioned above, and I recommended bringing the SEI manager to DAMPED (such they could regain alignment, but be robust while the EQ rings down). I presume we'd never notice, since rarely if every request this interim DAMPED state of the SEI systems these days, but in the interest of non-expert clarity, it would be good to sort this one out.

I attach a screenshot of the relevant MEDM screens in this state.

Also note ETMX HPI tripped which caused the ISI and SUS to go down a few seconds later, via horizontal actuators (see second attachment), and it looks to a pretty darn quick signal causing the trip. SYS_DIAG was reporting that the EX Tidal was near the edge of its range...

The FSS was locking on a 01 mode when the resonant threshold is set to 0.55.

I raised it to 0.6 and it now locks on the 00 mode. 

However, the transmission is now only 0.93, down from about 1.4.

Apparently the alignment into the RefCav continues to drift.   The orientation of the 01 mode and the reflected spot indicat that is is off in both yaw and pitch.

Looks like the PMC input alignment could use a touchup too.

We should plan on touching up the alignments on Tuesday, if not before.

ElliK, NutsineeK, RickS

Looking at the CDS snapshots of PSL screens from home this morning, we realized that the laser had shut down.

As-found control room screens are shown in the snapshot below.

Epics channel trends indicate that the Beckhoff interlock (IL) transitioned about 6 hours ago, but none of the recorded inputs (including the chiller flow inputs) triggered (see trend plot below).

The External Shutter had not tripped on the PSL_laser.adl screen, and the chillers were still running.

In the Diode Room, the Status screen showed ony the "Interlock OK" field red, everything else under interlock green.

Because we had no indication that the source of the fault was the Diode Chiller Flow, I decided not to swap the Diode and Crystal chiller interlock cables.

We had some issues getting the system back on-line, apparently due to accidentally closing a screen on the Beckhoff computer.  Eventually, we rebooted the computer and restarted the PSL software.

Sheila, Evan

ASC

Sheila finished taking the ASC sensing matrix data, so we now have pitch and yaw data to analyze.

Last week, we were not able to turn up the gains on the AS36I→SRM loops by more than a factor of 3, even with the HSTS compensation engaged. Tonight we were mysteriously able to turn them up by a factor of 7. They have a step response time of 2 s or so.

dETM boosts (FM1 for pitch and yaw) now come on automatically in the ENGAGE_ASC state.

OMC length

We looked briefly for any evidence of DARM contamination from OMC length noise. We drove at the error point of the OMC length servo with broadband noise and watched the response in DARM. We cannot see any linear coupling between 10 Hz and 1 kHz (the coherence is <0.1). The coupling seems to be quadratic in drive strength; as the drive level is stepped up by a factor of 2, the noise in DARM increases by a factor of 4 (see attachment). If one looks at the red, green, and orange curves and extrapolates the quadratic behavior to the unexcited control level, it seems that the contamination in DARM near 80 Hz would be a little above 1×10⁻²² m/rtHz.

There is, however, high coherence between DARM and OMC length from 2 to 4 Hz, with no excitation.

Thermal (?) drift

We still seem to be battling a drift seen in POP90 (also seen in REFL_LF) which limits our locking time at 10 W to about 2 hours. This seems to be very repeatable: POP90 continually rises until, after about an hour, some of the ASC yaw loops start to show low-frequency instability (particularly the cETM and SRM loops). After another hour, the instability grows so large that the interferometer unlocks. If we want longer locks, we have to fix this.

Nutsinee, Elli

We came in today to lock DRMI and measure the SRC length.  Spent the morning locking trying to lock DRMI, we were finding initial alignment difficult.  We changed H1:LSC-REFL_A_RF9_WHITEN_GAINSTEP to 7 so that we could lock PRX.  In the afternoon we tried to take a DRMI cavity scan measurment using the auxiliary laser.    We unplugged BBPRrefl power (power to 3f PD) on ISCT1 while measuring, and plugged it back in at the end of the day.   The SNR looked poor so we have added an RF amplifier ZFL-2500VH+ to the 1611 PD on ISCT1. (This is now the same setup we were using on ISCT6.)  At this point we handed the interferometer over to Evan.  We had a chat to him about our troubles with DRMI alignment, and will try again soon.

EdM, JasonjO, JeffB, PatrickT, PeterK, RickS

As the frequency of PSL shutdowns apparently initiated by the PSL Beckhoff interlock system increased, on Thursday morning we decided to take the system down and make some changes.  Note that all Beckhoff interlock triggers were apparently generated by the Diode Chiller flow interlock input.

We have had some experience with PSL chiller flow sensors that use a rotating rotor failing.  Replacement sensors that use optical sensing of eddies induced by a vane in the water flow (no moving parts) have been tested on the PSL reference system in Hannover.  We installed these sensors in the spare PSL chillers (there are two, the Crystal Chiller, and the Diode Chiller).

Because we don't have spare 480V 3-phase circuits available in the Chiller Room, we were not able to test the spare chillers but we tested the installaiton process.  Thursday morning we shut down the PSL and installed the new flow sensors in the chillers that had been operating.  We elected to do this rather than switch to the spare chillers for several reasons, including minimizing down time.

We found that the chillers would not operate with the new flow sensors.  Further, we found that the chiller controllers would not function normally, apparently unrelated to the flow sensors.  After consultation with the Thermotek (manufacturer) reps over the phone, we decided to switch to the spare chillers.

We found that the spare chillers would not operate with the new flow sensors either, though they had been installed and tested in Hannover and detailed instllation instructions had been provided and followed.

We swapped the flow sensors in the spare chillers back to the original-style sensors.   We first found that the Diode Chiller flow sensor appeared to be faulty so we swapped it with a similar sensor.  We eventually found that the chiller-side wiring of the flow sensor in the Diode Chiller is different than in the Crystal Chiller- the black and white wires on the sensor need to be swapped.

While writing an aLog entry Thursday evening, the chiller tripped again, twice, so we swapped out the Diode Chiller sensor one more time.  We left Thursday night with the chiller apparently functioning properly.  However within a few hours, Sheila and Evan gave up commissioning efforts after several shutdowns in quick succession.

Trends from Thursday night, when the chillers were operating but the laser had shut down due to a Beckhoff IL trip seemed to indicate Beckhoff IL events that were not triggered by any of the inputs, including the Diode Chiller flow input.  Having observed Beckhoff IL events from the Diode Chiller only, even with the spare chillers and several different flow sensors (with the Crystal Chillers apparently operating without incident and no indication that there actuall was a flow interruption in the Diode Chillers), we decided to investigate functionality issues with the Beckhoff IL controller.

We attempted to swap in the spare Beckhoff IL module, but found the issues detailed in Patrick Thomas's aLog entry.  We switched back to the original Beckhoff IL controller and eventually Patrick and PeterK were able to get it back on-line late yesterday (Friday) afternoon.

With the chiller apparently operating normally, we decided to leave it running and see if some commissioning work could be accomplished.  We expect that the shutdowns will continue until we remedy the real source of the problems.  However, we removed and re-connected most relevant cables and there is a remote possibility that we could have fixed an intermittent connection.

Next steps:

If we see the shutdowns, attributed to Diode Chiller Flow errors, again, we will swap the Diode Chiller and Crystal Chiller interlock cables to see if the shutdowns follow the cables (cable or Beckhoff IL problem) or chiller (Diode Chiller problem).

We also ordered three new old-style flow sensors that will be delivered today along with two new controllers for the originally-operating chillers.  Next week we will work to get the spare chillers operating, first with the old-style sensors, then maybe with the new-style sensors, so that we will have a backup ready.

We will also discuss the issues we encountered with the Beckhoff IL module replacement with the people at NeoLase in Hannover.

Evan, Sheila

The PSL has been working all night tonight.  We got a chance to try SRCL feedforward.  It works, but it doesn't improve the noise.  We saw that we were able to reduce the noise coupling by 12 dB at first, but then later saw that the coupling was changing by about 6dB at most. The second time we tried it we did not get as good subtraction.  Evan has new measurements of the SRCL coupling and frequency noise coupling to include in the noise budget over the weekend.  

Other things:

We used the TMSY picomotors to center the beams on the QPDs, this didn't change the combination of QPDs we used for the ITM loops.  We might want to check the normalization for the Y arm QPD next time we do inital alingment. 

We can switch SRM coil drivers in full lock, PRM, PR2, SRM, SR2 are now in the guardian in the DRMI_ON_POP state. 

We started to measure the ASC sensing matrix with all the loops closed including the ITM loops, we got a reasonable measurement for pitch, the data is all on disk although we had some trouble extracting it.  We were in the middle of tuning the yaw excitations when we got another earthquake.  We were moptivated to work on ASC because we have been aligning a little by hand before turning on the ASC all night, and we are hoping to find more diagonal signals so that we don't have to do this each lock. 

Jason, Patrick, Peter, Rick

In summary:

1. We could not get the spare interlock chassis to work. The original is back in.
2. We restored the C:/TwinCAT directory to a backup that Peter took in June of 2014.
3. The PSL is running again.

The spare chassis was swapped in. The status screen no longer made sense. Pushing the interlock button made the wrong indicator turn red. I did a scan of the EtherCAT modules. It appeared to indicate that they were unchanged (that is the spare had the same modules in it as the original). Suspected maybe the safety PLC inside the interlock chassis had to be started separately? It seemed to require a username and serial number and password. Peter's attempt to enter these and start it did not appear to work. I tried restarting various things including the entire computer. No luck. I suggested resetting all of the variables including persistent. Bad idea.

Starting the PLC now gave divide by zero errors and would not run. It appeared that some of the persistent variables, which were now all zero, were used in the denominator of fractions. They appeared to be related to some calibration settings. We couldn't get by this. This confuses me. If the code does not start with them zero, and upon the first run they are by default zero... How were they initially set?

It appeared that the files for the saving of the persistent variables were located in the C:/TwinCAT/Boot directory. We tried replacing that directory with a backup that Peter had. This did not help, same divide by zero errors. We tried to delete and replace the entire TwinCAT directory. Windows would not let us delete it.

At some point scanning the EtherCAT modules started showing a whole bunch of differences and the light on the front of the interlock chassis no longer came on.

We decided to put the original interlock chassis back and try restoring the computer to a backup that Peter had taken in June of 2014. After the original chassis was put back the light on the front still did not come on. Peter tried restoring just the TwinCAT directory with the restore software. He was able to do so. We opened the link to the visual. It seemed to run but was blank. I closed it, opened the system manager and set it to run. I opened the PLC and logged in. The light on the front of the interlock chassis came on! The PLC was running again.


Remaining questions:

1. How were the persistent variables originally set? Are what they are set to now (from the backup) the same as they were before?
2. What is different about the spare interlock chassis? Could it be some programming in the safety PLC? Different wiring?

With no PSL and no one else really around, I wanted to test out the new sensor correction/blend SEI configuration nodes.

I got the blend side of it working just fine, switching between desired blends perfectly with only a few syntax errors. I watched the NXT and CUR filter banks closely to make sure that they are doing what they were suppose to just incase. The sensor correction part didn't go as smoothly though. There were a few issues that I did not fifure out before I had to go.  Wish I had time tonight to finish this but unfortunately that is not the case.

I stoped the nodes just incase they wanted to mess with anything over the weekend.

On Monday:

• I need to find a way around the added on ezca prefix for a channel names. I have one test that looks at guardian state channels to check for transitions, and the added prefix I can't seem to get around...for now.
• BRS_SC_switch() only takes one argument, Guardian says I gave two. I definitely only see one. Must be another problem somewhere for future TJ to find.

Overall it went pretty well seeing this come semi-alive. I put all of the blends back where I found them as well as the SC filters so everything should still be set for the weekend.

A remote controlled power switch has been installed at EY to allow power for the HWS camera to be turned off or on from the control room.  The IP address for the switch is 10.105.0.155, access using telnet, port 23.  Ellie King has the instructions for controlling the power.  The camera is plugged in to outlet #1.