I was asked to summarize the SWWD (software watchdog) timing sequence as a reminder.

t=0: SUS IOP detects top OSEM RMS exceeds trip level, starts its 1st countdown (5 mins)

t=5mins: SUS IOP 1st countdown expired, its IPC output goes to BAD and it starts its 2nd countdown (15 mins). SEI IOP receives the BAD IPC, and starts its 1st countdown (4 mins)

t=9mins: SEI IOP 1st countdown expired, its IPC output goes to BAD and it starts its 60 second 2nd countdown. SEI user models get the 60 second warning IPC so they can cleanly shutdown before the DACs are killed

t=10mins: SEI IOP 2nd countdown expired, DAC cards associated with the chamber the SUS is located in are killed

t=20mins: SUS IOP 2nd countdown expired, SUS DAC cards are killed

For the hardware watchdog (HWWD) the times are doubled. The power to the ISI Coil Driver chassis is removed after 20 mins of continuous SUS shaking.

The power meter in the high power oscillator's external shutter was replaced.  The existing unit
ceased to function some time around the planned power outage a couple of months ago for reasons
best known to itself.

old S/N = 589365
new S/N = 627203

Jason/Peter

Sheila, Terra

Tonight we implemented the decoupled ASC SOFT matrix (first tried here but reverted a few days later). In past locks, the test mass op levs show common soft drift during power up; diagonalizing SOFT and HARD should help reduce this. 

Powering up with this new matrix rang up a CSOFT power-to-angle instabiltiy ~0.5 Hz, causing us to break lock a few times near the end of power up. Next lock, we stopped at 20 W and adjusted SOFT offsets to get the best alignment (PRG ~32) and then powered up to 50 W successfully with no instability.

We still lost PRG during the remaining power increase, so adjusted POP_A offsets then SOFT offsets again. POP_A offsets only slightly improved; far less dramatic than with old matrix and SOFT offsets. In the end we recovered quite a bit of PRG but at the cost of sideband loss. Final offsets shown in attachment.

Tomorrow we'll try applying offsets at 2 W and/or putting offsets on QPDs instead. We're leaving the new matrix implemented in the guardian (but not offsets yet).

We also had trouble with bounce modes tonight, mostly in ETMX. This was solved by turning on the EVANBR filter (which was the only test mass that hadn't had this filter turned on). We did not see the 9.4 Hz DHARD_P instability. 

We leave the interferometer at 50 W with DOWN requested.

First few hours of the shift were spent with recovery from the issues at the electronics racks out on the floor near the PSL.  As Jeff K, noted, when we tried walking through locking we found issues with the MC Common Mode Servo Board (it was swapped out per his alog). 

The rest of the evening was spent with commissioning.

There was a 6+ magnitude earthquake which shook things around (while we were still in recovery mode)---hope Virgo rode through it OK.

S. Dwyer, J. Kissel, C. Gray

After successfully recovering the IMC's VCO and recovering the IMC (29264), we were able to get up through LOCKING_ARMS_GREEN in the lock acquisition sequence. However, we found that ALS COMM failed caused lock losses during the next step (LOCKING_ALS), when the input for IMC length control in its Common Mode Chassis was switched from the IMC's PDH output to the ALS COMM PLL output. The ALS COMM PLL output is connected to IN2 of the IMC chassis that had a new daughter board installed in the star-crossed ISC rack H1-ISC-R1 today (LHO aLOG 29250). After fighting through MEDM screen confusion* at the racks, we found that OUT2 (an analog pickoff pick-off just after the input gain circuit) indicated a ~2.5 [V] offset, even with IN2 terminated with 50 [Ohms]. 

Suspecting that this symptom was indicative that the input gain circuit (circled in red in MEDM screen capture) was yet another causality of the unfortunate rack power mishap today (LHO aLOG 29253), we've replaced the entire chassis (which lives in U14 of H1-ISC-R1) with a spare we found in the EE shop -- S/N S1102627 (or Board S/N S1102627MC). 

Notably, this spare does not have one of the new daughter boards on which Chris has worked so hard.

We're not suggesting this swap be permanent, but we make the swap for tonight at least, so we can hopefully make forward progress. We suggest that IN2 and/or the input gain stage of SN S1102626 be fixed tomorrow, and the chassis restored so we can employ the new daughter board.

Other Details:
- Before removing the chassis, we powered down the entire rack using the voltage sequencer around the back at the top of the rack. 
- After installing the rack, we were sure to have all cables connected appropriately before turning the rack power on again (via the sequencer again).
- We added a few labels to the IMC's PDH output and the ALS COMM PLL output cables such that they're easier to follow and reconnect in the future.

*MEDM Screen Confusion -- whether IN1 or IN2 is fed into OUT2 of all common mode chassis is selectable on their MEDM screens. For the IMC's common mode board (at least for SN S1102626), the MEDM screen's indication of the status of that switch is exactly backwards. When the screen indicates that IN1 is feeding OUT2, IN2 is feeding OUT2, and vice versa. #facepalm

A notch at 331.9 Hz in now turned on in the LSC-DARM2, FM4. The change was accepted in SDF_OVERVIEW "safe.snap" and "OBSERVE.snap".

This filter suppresses the DARM loop at PcalY calibration line frequency that is used to track the optical gain and the coupled-cavity pole frequency.

In an earlier report we pointed out that the notch was switched off during ER9 (see LHO alog 29108).

Summary:
Repeating the Pcal timing signals measurements made at LHO (aLOG 28942) and LLO (aLOG 27207) with more test point channels in the 65k IOP model, we now have a more complete picture of the Pcal timing signals and where there are time delays.

Bottom line: 61 usec delay from user model (16 kHz) to IOP model (65 kHz); no delay from IOP model to user model; 7.5 usec zero-order-hold delay in the DAC; and 61 usec delay in the DAC or the ADC or a combination of the two. Unfortunately, we cannot determine from these measurements on which of the ADC or DAC has the delay.

Details:
I turned off the nominal high frequency Pcal x-arm excitation and the CW injections for the duration of this measurement. I injected a 960 Hz sine wave, 5000 counts amplitude in H1:CAL-PCALX_SWEPT_SINE_EXC. Then I made transfer function measurements from H1:IOP-ISC_EX_ADC_DT_OUT to H1:CAL-PCALX_DAC_FILT_DTONE_IN1, H1:IOP-ISC_EX_MADC0_TP_CH30 to H1:CAL-PCALX_DAC_NONFILT_DTONE_IN1, and H1:CAL-PCALX_SWEPT_SINE_OUT to H1:CAL-PCALX_TX_PD_VOLTS_IN1, as well as points in between (see attached diagram, and plots)

The measurements match the expectation, except there is one confusing point: the transfer function H1:IOP-ISC_EX_MADC0_TP_CH30 to H1:CAL-PCALX_DAC_NONFILT_DTONE_IN1 does not see the 7.5 usec zero-order-hold DAC delay. Why?

There is a 61 usec delay from just after the digital AI and just before the digital AA (after accounting for the known phase loss by the DAC zero-order-hold, and the analog AI and AA filters). From these measurements, we cannot determine if the delay is in the ADC or DAC or a combination of both. For now, we have timing documentation such as LIGO-G-1501195 to suggest that there are 3 IOP clock cycles delay in the DAC and 1 IOP clock cycle delay at the ADC.

It is important to note that there is no delay in the channels measured in the user model acquired by the ADC. In addition, the measurements show that there is a 61 usec delay when going from the user model to the IOP model.

All this being said, I'm still a little confused from various other timing measurements. See, for example, LLO aLOG 22227 and LHO aLOG 22117. I'll need a little time to digest this and try to reconcile the different results.

I tested a new build of the frame writer on h1fw0.  After a few minutes of testing the system was unable to establish a connection to the network share the frames are written to.  After rebooting the ldasgw machine and the frame writer Jim changed the link between the two machines from a 10Gb fiber link back to a 1Gb copper link.

While waiting for the network problems to be solved I reconfigured the system to write to local disk.  The frame write was stable while writing to local disks.  When H1FW0 & H1FW1 went unstable even writing to local disks would fail.  After the network link to the LDAS disk system was re-established Jim and I restarted the frame writer process writing to LDAS instead of local disk.  It has been running since then.

The next step is to watch h1fw0 for a week.  If h1fw0 is stable then we will move h1fw1 to use the new code so that we have access to the additional metrics is provides.

Jason, Jim, Dave:

I've been looking into why when we restart the PSL front end models the shutter controlled by the diode room Beckhoff OPC gets closed. Specifically it happens when the h1pslpmc model is started.

The h1pslpmc model controls the shutter through ezcawrite to the Beckhoff channel H1:PSL-EPICSALARM. A ZERO keeps the shutter open, a ONE closes the shutter. Settings within the pmc safe.snap file are zero, so the front end should not output a ONE value on startup and the shutter should not close.

The shutter can be instructed to close by the model if the FLOW monitor (a raw ADC input to the model) lies outside of upper or lower limits. The FLOW signal is passed through a standard IIR filter module. Trending the INMON and the OUT16 of this filtermodule when we started the model last Tuesday reveals for the first second the OUT16 is ZERO while the  input is the ADC value. We suspect this is normal for an IIR integrating filter module. The zero OUT16 triggers the shutter close because it lies outside of the upper and lower limits.

The filter module does not have any filters loaded, has no offsets and has unity gain. We propose (and Jason agrees) that the next time we restart this PSL model we replace FLOW's filter module with an EPICS-OUTPUT part and see if we can get the shutter to remain open during the model startup.

(see WP 6111) 
RGA data following recent bake shows improvement but needs extended baking - will extend bake of Vertex RGA (running pump) until commissioners are negatively impacted.  

1000 hrs. local -> Heating of Vertex RGA resumed -> will need periodic access to measure temps and make adjustments

TITLE: 08/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventitive Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Maintenace Day recovery has been halted by the IMC not locking. Investigation is ongoing.
LOG in attached txt file
 

M. Pirello, R. McCarthy, E. Castrellon

Work Permit #6116

We removed the ISC Anti-Aliasing Chassis and investigated channel 14 which was suspected to have a failure, per ALOG 29189.  To test the channels, we attached the DB9 Breakout to the front DB9 connector and used the SR785 to test the transfer function through the filter.  We applied the test leads directly to the differential test points on the PCB.  Channel 14 looked much like all of the other channels. Unfortunately, the data was corrupted on the thumb drive, but I did get one scan of channel 9 and have attached this scan.  Every scan was nearly identical to this scan.

We returned the chassis to its original location and reattached all of the cables.

Marc P, Chris W

As per Work Permit #6104, we added a 200 kHz lowpass filter (D1600314) to the fast path of the Common Mode Servo Board (D040180), located in the LVEA ISC-R1 rack. The serial number of the servo board is S1102626.

When reconnecting the power to the Common Mode Servo Board, the 17V supply was connected first (should connect 24V first or use the sequencer), which resulted in blowing diodes D4 and D5 at the power input (see D0901846). At the time, we saw LEDs of other boards on the rack turning off. The diodes were replaced and the current drawn by the Common Mode Servo Board was confirmed to be the same as drawn by an equivalent spare board. The board was then replaced in ISC-R1 rack, this time using the sequencer switch to power cycle the whole rack.

I've made a script to somewhat automate the weekly oplev trends FAMIS task. It makes 3 plots like the attached image of the oplev pit, yaw and sum channels for the test-masses, BS, PR3 and SR3. It still requires a little fiddling with the plot, you have to zoom in manually on any plots that have 1e9-like spikes, but this should still be easier than running dataviewer templates. It uses h1nds for data and a pre-release version of the python nds2 client that has gap handling, so updates in the future could break this. I'll try to maintain this script, so any changes or improvements should come to me. The script lives in the userapps/sys/h1/scripts folder.

The script is run by going to the sys/h1/scripts folder:

jim.warner@opsws0:~ 0$ cd /opt/rtcds/userapps/release/sys/h1/scripts

And running the oplev_trends.py script with python:

jim.warner@opsws0:scripts 0$ python oplev_trends.py

You will then need to do the usual zooming in on useful data, saving screen shots and posting to the alog. I'll look into automating more of this, but it works well enough for now. It would also be very easy to add this to a "Weeklies" tab on the sitemap, which I believe LLO has done with some similar tasks.