New ISS outer loop prototype board was modifed and put in. Modification details will be in a separate alog.

PD5, 6, 7, and 8 anode and cathode cables were disconnected from the old transimpedance box under HAM2 and was connected to the PD5-8 cable for the new prototype board. One strange thing about the old setup was that the "LR C" and "LR A" cable that come from the chamber feedthrough were connected to the anode and cathode input of the old box, respectively. Since apparently they gave us a correct signal (no forward bias across the diode), I connected "LR C" chamber side cable to "LR A" rack side cable for the new setup.

I connected PD5-8 cable to PD1-4 input in the new prototype box. We can conveniently switch back and forth between the old and the new by just switching one cable that goes to the "outer loop input" connector on the 1st loop chassis.

I briefly connected the output of the new box to 1st loop, and enabled the servo, but it didn't work. It's not clear if the sign is correct, but there's no way to change the sign externally, I need to open the box. I'll make more measurements next week to see if the sign is indeed wrong.

I connected the old second loop back to the 1st loop.

While I was working on ISS outer loop at the PSL field rack, H1:PSL-MIS_FLOW_OK tripped.

Peter King walked me through the diagnostics over the phone and I was able to open the shutter.

2.5 hours into being locked at 50 W, we saw two PI ring up: ITMX 15522 Hz (Mode 2) and ETMX 15541 (Mode 17). We couldn't damp them and they broke the lock (22:22 UTC). 

We had about 10 minutes before lockloss to try different damp settings. For Mode 2, any phase/gain changes I made rang up the mode more; the existing settings were already providing the most shallow slope. For Mode 17, a slight phase change helped but instability still bounced back. Below is Mode RMS monitor after the struggle. (Green and Red modes only appear to be ringing up from bleed over).

I've also attached the trend of the two modes. 

Something looks like it really kicked on Mode 2; there's strong slope changes before we started trying to adjust damp settings. Both modes stayed rung up into the next lock aquisition which I've never seen before; we could see them on the RMS monitor before DC READOUT. I stopped at 30 W to tweak phase settings again and allow to damp. These settings have been saved to the guardian. After that we continued to 50 W and stayed locked for almost 2 hours without incident (lockloss unrelated to PI). Overall, after tweaking at 30 W for full peak minimalization, I've lowered gains (perhaps the high gains were 'kicking' things too much?). 

This is the same time scale into the lock as we saw the other night, though note that ETMX did not ring up and ITMX did not have any active damping that night. Also note the oscillation in Mode 17 ~1156310300 in the trend.

We're leaving the interferometer locked at 50 W with the tweaked PI settings so we'll see in the morning. 

Most of tonight's work was measuring the HARD Pit loops, to give them some low noise settings to match the pre-existing Yaw low noise settings.  I also tried moving around the spot position on the AS_C QPD, which is pretty much the only QPD whos offsets we haven't tried tweaking lately.

It's clear on the AS camera as well as on the ALS X green camera that something moves significantly between 2W and 50W in yaw.  I was hoping that perhaps adjusing the SRC alignment would help with this somehow, but it didn't.  What I do think it might help with though is the fact that previously, we've seen our sideband powers drop significantly when moving POP and Soft offsets to increase the carrier power recycling gain.  I haven't tested this yet though. The first two attachments show the improvement in AS90 and the AS90/POP90 ratio when the AS_C spot is moved.  The max value in yaw that I moved was to 0.94, but this was obviously too far.  Perhaps tomorrow I'll picomotor a little bit so I can go a little farther, if this indeed helps the sideband buildups.

I moved the filters in the CHARD loops around today so that I could fit in a new lowpass at 200Hz.  CHARD has been acting funny lately, and is likely why we've been tripping the quads' rms watchdogs the last day or so.  To at least help, I put in a high-freq cutoff so that we can at least limit the amount of sensor noise we send to the quads.  Probably we should add this to the DHARD loops too, since none of them have had cutoffs of any kind during the acquisition sequence for a long time now.  I modified the ISC_LOCK guardian, as well as the sensing matrix code to be aware of the new locations of the different filters.

I measured the HARD pitch loops several times while modifying the loop shapes.  Both Chard and Dhard now have 50W modifying filters that move the plant compensation frequency response to match the 50W resonant frequencies rather than the 2W freqs.  They also both have a 25Hz cutoff.  CHARD Pit I can lower the gain by a factor of 2, but DHARD pit I can't. Most of the measurements in the attached screenshots are with the BoostHBW still on.  These should have been turned off, and are turned off for the Yaw loops when we go to low noise mode, in order to win back enough phase to reasonably have our cutoffs.  I need to think more carefully about how much gain we need at what frequencies before I finalize these loop shapes, but it seems like (from the DHARD pit measurement with the boost removed) that I should have enough phase to move the cutoff lower, if I don't require all that gain at low frequency. 

The guardian is up to date with these changes - other than helping the still-cooling IFO with some alignment while trying to catch DRMI lock, you can just request Lownoise_ASC, and the IFO will go up to 50W and hang out.  (I've commented out all of the offsets in the Adjust_offset state, but other than that, things are all as they were, modulo the changes mentioned above.)

Terra is writing up our PI experiences and mysteries from today, but I expect that the IFO will stay locked for just under 3 hours, so I'm going to put it in Observe just in case the data groups want to test pipelines.  The noise is still bad though, especially the intensity noise coupling as the IFO thermalizes.  To do this, I changed the nominal IMC_LOCK state to ISS_TR_CLOSED rather than the usual ISS_ON (which implies 2nd loop on), since we can't close the 2nd loop right now - we need the new hardware so that we can have both 2nd and 3rd simultaneously.

Oooh, I almost forgot.  Sheila and I tried closing a dither loop around the PRM to keep the spot position on PR2 constant.  This seems like it was doing good things, however once she fixed the 3rd loop's SR560 we decided that it would be good to get back to a campaign of persuing low noise for a while, rather than chasing offsets around.  We should consider coming back to this though when we come back to offset moving for PRCgain improvement.

After the nearly 3hr long lock at ~50W, we were paused at Locking Arms Green due to an X-arm which started low in power and slowly recovered (probably due to cooling).

Backing up:  During the previous long lock at high power, Jenne noticed that ALS X was visibly misaligned in yaw (could easily see in camera).  

After the lockloss and attempting to get H1 back up, we could see the ALSX was slowly struggling to recover.  The theory was this is due to some sort of cooling effect after the 50W lock.  Attached is a screenshot  of trends for  oplevs, TMS pit/yaw, ALS transmission, and the PSL power.  The story here (starting from left to right) is:

1. 2-hrs of locking at low power, 
2. 2-hrs of the 50W lock
3. ALS locking for 7min after tonight's 50W lock

(For the plots, oplev pitches are the plots on the top & yaws are on the bottom)

• Notice the 7min plots' ALS-X_TR & ETMX yaw oplev (circled in blue).  Jenne hadn't seen this issue before.
• Compare the above effect with locking at 2W earlier in the day (circled in red).  Everything is flat.
• SIDE NOTE:  Also, for the BS pit oplev (circled in green), there is a cooling effect here----but this is a known effect.

Commissioners (Jenne, Terra, and Sheila) worked on H1 addressing ISS 3rd loop earlier, PRM spot servo, and then with some longer high power locks toward the end.

Notes from the evening:

• CP4 testing is causing an overfill alarm, but Chandra is aware and expects it to stop later tonight (this has been railed at 100% the entire shift).
• PRM beam spot servo work (Jenne, & Sheila)
• 16:30 4th Friday tour (Landry & Kissel)
• 17:44:  3rd Loop check (Sheila)
• Shift ended with Jenne doing ASC work at high power.
• During a nearly 3hr long shift, toward the end of it, several PI modes rung up & they could not be appeased.  

General Operataor note regarding DRMI alignment:  Due to a few DRMI locklosses, stopped at LOCK_DRMI_1F and tweaked on usual suspects:  BS, PR3, PRM, IM4, & SRM (for my session, the BS & PR3 were the only optics to help out).  Then locked up the DRMI with no issue.

J. Kissel, R. Abbott, D. Coyne, P. Fritschel, V. Sandberg

As a part of the engineering change request to remove/disable the problematic QUAD's PUM driver, RMS current, entirely analog, watchdog (ECR E1600270, and FRS Ticket 6100), I've looked into several of today's episodes of watchdog tripping. I show four examples:
18:15 UTC -- where we ride through an aquisition event where the watchdog surpasses its threshold but does not trip.
19:20 UTC -- A lock loss that we believe is caused by the PUM watchdog tripping
19:37 UTC -- A lock loss caused by watchdog tripping, 
01:00 UTC -- A "normal" lock acquisition.
These examples are plotting the time series of the RMS Current Monitor EPICS channels (e.g. H1:SUS-ITMY_L2_RMSIMON_UL_MON), which serves as the trigger signal for the Flip Flop circuit in the driver. Check out pg. 3 of T1100378 for a block diagram, and a more detailed copy-and-paste draw that shows the inter-circuit connections is in D1600332.

From the PUM driver board and monitor board schematics (D070483 and D070480, respectively), with the help of the block diagram, the calibration of these RMS IMON channels is as follows:
RMSIMON [A] = RMSIMON [ct] 
              * ADC Gain [V/ct] 
              * RMS Input Gain [V/V] 
              * RMS Output Gain [V/V] 
              * Current Monitor Gain [V/V] 
              * Output Impedance [A/V] 

            = RMSIMON [ct] 
              * 40 [V] / 2^16 [ct]
              * 1 / 10 [V/V]
              * 10 / 1 [V/V]
              * 3/2 [V/V]
              * (1 / 15) [A/V]

            = RMSIMON [ct]
              6.1035e-5 [A/ct]

As is reported in the PUM driver's design study (T0900277), user guide (T0900290), and several test reports (e.g. S1101561), the RMS watchdog level trips at ~100 [mA]. Thus, the attached examples (except for the 18:15 UTC example) works as designed. 

The StripTool isn't calibrated but after looking at the raw DTT time series and playing around with the calibration at various points: the voltage going into the Flip Flop circuits that corresponds to ~100 [mA] is ~7 [V] RMS, which is about ~2000 [ct] in the RMS I MONs.

The StripTool for the 01:00 UTC lock acquisition sequence therefore shows that quiescent current levels are a few hundred counts, or a few 10s of [mA] RMS, well below the threshold. The humps and bumps you see are transitions between states as various DARM / QUAD fed ASC loops turn on.

I'll also note, for the record that Jenne installed a low pass filter (aLOG to come later) between the DTT measurements and the StripTool, so we may be OK again. Maybe the watchdog being problematic every few months is because that's about the timescale in which we usually overhaul our ASC schemes, and the trips are just indicative of transitionary, mid-commissioning, loop outputs.

Finally, the design study (T0900277) and user guide (T0900290) mention the original motivation for the time-constant of ~10 [sec] at a current level of ~100 [mA] was entirely motivated by outgassing concerns of a hot OSEM. Research is on-going as to whether this makes sense. Preliminary investigations find an early result (T0900611) show that the time scale is more along the lines of minutes rather than ~10 [s]. 

More details to come as the picture solidifies. Follow along in the FRS ticket comments!

Keita pointed out that the ISS 3rd loop 560 was overloaded, and was in low noise when it should have been in high dynamic range.  I went to check it out and indeed, it saturates even without any input or with a terminator. 

Fil didn't have a spare working 560, but loaned me an SR650.  We are using an AC coupled low pass at 30 Hz, gain of 0 dB, positive polarity, to imitate the settings Keita used in 27895

We don't know how long the 560 has been overloaded, but this probably is the reason why we have had so much difficulty with the CSOFT instability in the last week.

Experiment on CP4 today:

Started at 90% full
Filled to well over 100% at 45% open on LLCV
At 12:10pm local, CP reached 100% full (took ~3 hr 40 min)
I sat at MY for about two hours with nothing happening (at this flow rate it takes a lot longer than 35 min. to overfill till LN2 comes out)
Finally I began to ramp the LLCV 5%, and then 10%, every 20 min. or so
At 72% open the signal from the flow meter was very noisy
After a few min. at 72% the exhaust pressure started to rapidly rise so I set to PI-mode with min. allowance set to 39% (love the new code!)
The pressure was still high and unstable and because I didn't want LN2 to spew onto the flow meter I lowered the PI-code min. to 37%
The exhaust pressure has slowly ramped back down to nominal
Leaving in PI-mode (at 37% min) over the weekend and will periodically monitor pump level and pressure

It appears that something goes wrong when zooming in that creates random lines across the plots.

3pm local

18 sec to overfill CP3 with 1/2 turn open on LLCV bypass valve. Lowered CP3 LLCV from 20% to 19%.

Attached are trends of the last month+ of relative humidity and temperature data for the 2 desiccant cabinets in the VPW.  One is used for 3IFO storage.  The last trends are posted at alog:  28526 .  The last month of data show normal trends with some bumps when the cabinet was opened for some time and the RH raised breifly.  Temp of the cabinets is the usual ~daily fluctuation of ~10deg F.

During this morning's PSL ISS code change, I tested if restarting h1pslpmc model with the FLOW IIR filter replaced with an EPICS part still closes the shutter, and indeed it did.

Looking at full data DAQ trends at the time of model startup shows what is going on. The FLOW ADC data is downsampled from the 64kHz IOP model to the 32kHz h1psliss model. The downsampling filter starts the data at zero and takes several cycles to settle to the digitized value. As an example the plot below shows an LSC channel at the time we restarted h1lsc yesterday. In the PSL PMC case, since any instantaneous FLOW value outside of the acceptible range causes a shutter close trigger, this will always close the shutter on model startup.

One solution we can think of is to add code in the FLOW_ERR C-file which triggers FLOW_OK to become False only if the flow values are out of range for several consecutive cycles. For a decision on whether the issue is of enough concern to warrent this code change we'll defer to the PSL team.

This morning the RMS watchdogs for all of the test masses have been tripping, usually during locklosses at higher ISC_LOCK states. Resetting these requires opening the tripped Quad overview, and setting H1:SUS-*TM*_BIO_L2_UL_RMSRESET to zero, then  back to 1. Since I've had all 4 trip at once, and Verbal_Alarms yells each one at you multiple times, I made an alias in my .bashrc profile to do this. Resetting all of the RMS WDs doesn't seem to effect anything, or disrupt lock, so this alias can be used any time one of these WDs trips. Entering rms_wd in a terminal now sets the WD bit to zero, sleeps for a second, then sets it to 1.

Operators can easily add this to their .bashrc file from a terminal by copy/pasting :

echo 'alias rms_wd="caput H1:SUS-ITMX_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ITMY_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ETMY_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ETMX_BIO_L2_UL_RMSRESET 0 && sleep 1 && caput H1:SUS-ITMX_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ITMY_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ETMY_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ETMX_BIO_L2_UL_RMSRESET 1"' >> .bashrc

You then need to open a new terminal to get the new alias.

J. Kissel, J. Warner

As Jim brought the instrument up this morning, we were alarmed to see a forest of lines around 20 to 25 Hz. As they did not disappear or change height as we went through the lock acquisition sequence, as ASC loops turned on, noticed them in MICH and PRCL, found a whole bunch of ASC-ADS SDF differences, and I heard Shiela suggest that she might try an LLO scheme for dithering the alignment, I began to suspect that these lines were intentional.

Indeed, after poking around, I found the dither alignment overview screen and found several oscillators pushing out excitations at 
19.1, 19.7. 20.1. and 20.8 Hz in Pitch and 21.3. 21.9, 22.3, and 23.0 Hz in yaw, being sent to PR2, PR3, and the BS.

I confirmed that these lines actually make it to the SUS by gathering an ASD of the input control request at the bottom of PR2, PR3 and the BS indeed I see the same features in PR3 and BS (why not PR2?).

*phew!*

We'll leave these on, in case the plan was to just gather data with these lines present.

See attached ASD of input request to the SUS, a screen cap of the Dither Overview screen, and the corresponding DARM ASD.

the Verbal Alarms code was logging to the ops home directory. Prior to the move of this home directory (WP5658) I have modified the code to log to a new directory:

/ligo/logs/VerbalAlarms

We restarted the program at 14:04 and verified the log files are logging correctly.