On June 17, 2016 the GDS calibration pipeline at LHO switched to ingesting double precision channels that are no longer dewhitened.  Therefore, the filters file was updated to include a unity dewhitening filter.  The new filter file can be found in the calibration svn under aligocalibration/trunkRuns/PreER9/GDSFilters/H1GDS_1150213197.npz.

The double precision channels ingested by the GDS pipeline in the current configuration on:

H1:CAL-DELTAL_CTRL_TST_DBL_DQ

H1:CAL-DELTAL_CTRL_PUM_DBL_DQ

H1:CAL-DELTAL_CTRL_UIM_DBL_DQ

H1:CAL-DELTAL_RESIDUAL_DBL_DQ

H1:CAL-DARM_ERR_WHITEN_OUT_DBL_DQ

[Jenne, Robert]

As a result of Keita's alog 28196 regarding the beam position on the BS, we wanted to move the beam splitter around in relation to the beamline, to see if that would change any clipping that we may have on the baffles.  Short answer: nope.

First, we moved ST1 by putting offsets in the isolation loops.  JeffK tells us that these are calibrated in nm, so our 5,000 count offsets correspond to about 0.5mm of motion.  We moved ST1 up and down, as well as laterally along the plane of the beam splitter (+x+y and -x-y).  No effect seen in the power recycling gain. 

Next, we moved HEPI in a similar fashion.  The thought here is that the ITM elliptical baffles are suspended from this ST0, so we weren't moving them earlier.  (By moving both ST1 and ST0 we had hoped to differentiate which set of baffles was causing us trouble.)  We moved up and down, as well as in RZ, rotation about the z-axis.  RZ is calibrated into nrad, and the baffles are order 1m away from the center of the ISI, so they were each moved on the order of 0.5mm also.  Again no effect seen in power recycling gain.

Attached is a snapshot of our striptool, with the first offsets starting at about 0:06:00 UTC, and the last ones ending around 1:00:00 UTC.  Teal is the power recycling gain.  The POP18 seems to be still relaxing from the power up to 40W for the first few minutes of our tests, but doesn't seem to be correlated with our movements.  Red trace is the vertical CPS measure of BS ST1 ISI position, and orange is superimposed with brick red measuring our lateral motion.  Light purple is vertical HEPI motion and light green is RZ HEPI motion.

We felt that if we were really dominated by clipping losses around the beam splitter, moving by 0.5mm in some direction should show us some change in recycling gain.  Since it doesn't, we conclude that the power loss must be somewhere else.

Dan, Ryan, Dave:

we worked on h1fw0 today. This writer was chosen because the nds machines have been offloaded to the cds-h1-frames file system (via h1ldasgw2) and h1fw0 is significantly more unstable than h1fw1.

Executive Summary: no matter what we did h1fw0 continues to be very unstable if it tries to write all the frame files, it is stable if it only writes the science frames. Further tests are more intrusive and will wait until later in the week.

During all the tests I monitored the status of the threads which write the frame files to disk. There are four threads: dqscifr, dqfulfr, dqstrfr, dqmtrfr (science frame, full frame, second trend frame and minute trend frame). These threads are normally in sleep state (S), transistioning to the running state when writing their frames (R) and also spending some time in the disk-sleep state (D) during the write process. 

I am interacting with the QFS file system in three ways: daqd writing the frame files, manually creating a 1GB file using the dd command and performing a directory long listing (ls -al). When everything is running correctly, the dd creation of a 1GB file takes about 5 seconds and the "ls -al" is fast. The symptom of the problem is that all the active frame writer threads all get stuck in the D uninterruptible state, forever waiting for a disk write completion signal. At this point the dd copy is still able to create a 1GB file, but it takes much longer (10-20 seconds). The long listing fails, also going into the D disk-io-sleep state. At this point data is going into daqd and none is coming out, internal buffers fill and the process dies. The long listing now completes, and the dd copy goes back to 5 seconds to complete.

Things tried today:

Monitor NFS traffic on private network between frame writer and solaris NFS server, no errors or unusual packets seen

Change the version of NFS for this mount from vers3 to vers4, no change in stability. One point of interest, to make the change on h1fw0 I rebooted the computer, and the daqd process on h1fw1 died at this time.

Dan stopped the rsync process which is backing up the raw minute trends from ldas-h1-frames. He also stopped all LDAS access to ldas-h1-frames (disk2disk copy, frame file checksumming). At this point h1fw0 hadldas-h1-frames all to itself, and it still could not write all 4 frame files and died on the hour when minute, second and full frames all being written at once.

As I have left the system, h1fw0 is only writing science frames. Dan is comparing these files with h1fw1 and will use them if h1fw1 restarts. Dan has restarted the rsync process to complete the raw minute trend backup (prior to his reconfiguring the SATABOY raids).

Later in the week Dan will reconfigure the Sataboy raid to make a more efficient file system, whose file access times should be halved.

It seems that there are some marginal increases in temp in both the OSC and the AMP that correspond to very marginal decrases in headflows. OSC_DB4_PWR seems to be doing "it's own thing" in comparison to 1,2 ad 3.

As usual, please refer further, in-depth analysis to Jason O or Peter K.

Andy, Duncan, Laura, Ryan, Josh, 

In alog 28299 Andy reported that we were seeing the ER9 range deteriorate due to glitches every 2 seconds. Figure 1 shows the glitches turning on in DARM at 2016-07-09 05:49:34 UTC. 

We think the ALS system not being shuttered and changing state in lock is to blame. Here's why. 

Excavator pointed us to a strong coupling between DARM and the ALS channels. Raw data confirmed a correlation, Figure 2 shows the ALS glitches tuning on at that same time and figure 3 shows that both DARM and ALS are glitching at the same times. 

When we investigated the Guardian ALS state for this time (figure 4), it was not in a nominal configuration to start with and that got worse around the time the glitches started. The shutter was not "Active" and at 05:49:34 UTC the ALS X state changed from "-15 locked transition" to "-19 locking WFS" and at that same time the glitches started in DARM. So at some point, ALS X decided it needed to lock the arm (looks like Y followed an hour or so later). We did not track down exactly how the glitches originated or made it into DARM because this seems non-standard enough that a configuration fix should make it go away. 

Figure 5 shows a summary page plot for nominal ALS X Guardian behavior from O1. So the shutter should be active and we don't expect to see "locking WFS" come on during an analysis ready state. 

Looked at the cameras for PR3, PR2, PRM and all 16 arm baffle PDs during a power up. Bottom line: The lost power is found on the ETM baffle PDs.

- Plot 1 and 2 show the ETMX and ETMY baffle PDs. Note the huge increase in baffle PD4 in both arms. At the same time, baffle PD 1, on the opposite side of PD4, also sees an increase, so we can't explain this with arm alignment.

- The ITM baffle PDs show much less signal - there is some on ITMY baffle PD 4, but nowwhere near as much as on the ETM. (:Plot 3 and 4)

- PRM, PR2 and PR3 roughly track the recycling gain, i.e. they increase less than linear with input power. (Plot 5)

I should also say that the Beckhof whitening and digital gain setting were the same for all 16 PD. I attached a representative snapshot for H1:AOS-ETMX_BAFFLEPD_4.

The y-axis on all these plots (labeled arbitrary) is in Microwatt/Watt input power for the Baffle PDs, and recycling gain for POP_LF.

Title:  07/11/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)
State of H1: IFO unlocked. Switched intent bit form observing to commissioning. Guardian error – "4 dead channels" & EPICs comm error 
Commissioning: 
Outgoing Operator:  None
 
Activity Log: All Times in UTC (PT)

15:00 (08:00) Start of shift.
15:00 (08:00) Guardian error
15:30 (08:30) Richard – Reset End-Y Beckhoff IOC 
15:44 (08:44) Alfredo & Elizabeth – Going to the bridge over the X-Arm to put up a sign
16:25 (09:25) Sheila – Going into Optics Lab – Looking for BS optic
16:30 (09:30) Kyle – Going to Mid-Y to run test on rotating pumps
16:35 (09:35) Alfredo & Elizabeth – Back from X-Arm Bridge
16:53 (09:53) Sheila – Finished in Optics Lab – Going to Squeezer Bay
17:20 (10:20) Kyle – Back from Mid-Y
17:29 (10:29) Alfredo & Elizabeth – Going to the bridge over the X-Arm to work on new sign
17:44 (10:44) Alfredo & Elizabeth – Back from X-Arm 
18:06 (11:06) Sheila & Haocun – Going to ISC table at HAM2 – They will open the table.
19:28 (12:28) Sheila & Haocun – Out of the LVEA. 
19:46 (12:46) Kiwamu & Nutsinee – Transitioning LVEA to Laser Hazard
20:51 (13:51) Kiwamu – Going into LVEA to adjust ITM-X and ITM-Y IR cameras
21:13 (14:13) Dick – Going into LVEA to Squeezer bay to look for parts
21:21 (14:12) Kiwamu – Back into LVEA to work on camera alignment
21:27 (14:27) Richard – Bring a tour through control room
21:32 (14:32) Dick – Out of the LVEA
21:42 (14:42) Kiwamu – Out of LVEA
23:00 (16:00) Turn over to Ed 


End of Shift Summary:

Title: 07/11/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)
Support: Jenne, Sheila, Kiwamu, Stefan, 
Incoming Operator: Ed

Shift Detail Summary: End-Y Beckhoff IOC crashed. It has been restarted. 
16:40 (09:40), IFO locked at INCREASE_POWER. IFO at 39.9W, Stefan is taking data.17:10 (10:10)
19:46 (12:46) LVEA transitioned to Laser Hazard. Will leave it Laser Hazard until Peter is finished with the hardware hunt tomorrow morning. 
IFO has been up and down all day, while commissioners are working through various issues, adjustments, and improvements.
  

 

Sheila, Haocun

Due to the satuation of the photodetectors, we added a 90:10 beamsplitter (CVI BS1-1064-90-1025-45P) in the POP beam path.

WIth the input power of 2W (1.97W) in the interferometer:

Power before adding the BS:   POPX: 5.74mW       POP Air A: 2.56mW      POP Air B: 2.52mW     (total: 10.82mW)

Power after adding the BS:      POPX: 0.852mW     POP Air A: 0.4mW        POP Air B: 0.4mW       (total: 1.652mW)

Power right before the BS: 10.88mW; Power right after the BS: 1.82mW.

PSL: 
SysStat: All Green, except VB program offline 
Frontend Output power: 34.4W  
Frontend Watch: Green
HPO Watch: Green

PMC:
Locked: 0 days, 0 hours, 21 minutes
Reflected power:    24.0W
Transmitted power: 125.5W 
Total Power:       149.3W 

ISS:
Diffracted power: 16.702%
Last saturation event: 0 days, 0 hours, 21 minutes 

FSS:
Locked: 0 days, 0 hours, 18 minutes
Trans PD: 3.350V

The watchdog and its timers suggest the Stage2 Actuators went first but the traces (attached) show the T240 are what hit the trip limit.  Has a bit of Earthquake shape to it.  An M6.3 in Ecuador is a good candidate, with an origin time of 0211utc or 18mins before the ISI tripped.  Oh yeah, there is that 3um/s excusion on the BLRMS Eq Band monitor...

No trouble reisolating the ISI this morning.  An issue that popped up is that the Operators Overview showed no indication of the platform being tripped.  It was obvious with the SDF being red with 60 differences.  FRS 5852 filed.

The smoothing of the CP level is done on the milliamps read in from the ADC before it is converted to a percentage full. Currently the smoothing parameter is set to 0.999 on each of the CPs. I have attached full data trends over an hour of the level of each CP read in milliamps before and after being smoothed. They all appear reasonable.

The code for the smoothing filter and a dataviewer template with the channels used for the trends is also attached.

0930 - 1010 hrs. local -> To and from Y-mid 

Opened exhaust check-valve bypass-valve, opened LLCV bypass valve 1/2 turn -> LN2 at exhaust in 50 seconds -> Restored valves to as found configuration.  

Next CP3 overfill to be Wednesday, July 13th. 


Also, 0945 - 1000 hrs. local -> Ran purge-air supply and QDP80 at Y-mid (maintenance task)

Jeff B.  Noted white screens on EY beckhoff channels.  We logged into the computer could not find any obvious errors besides the IOC being down.  The Beckhoff system was still running but the IOC that interfaces to EPICS had crashed.  It was restarted and connections came back.

   All subsystems reported no significant problems or issues. 

   ER9 run has finished. Vern thanked all who put in so much time and effort to make this run a success. Problems brought to light during ER9 are being addressed. 

   The Tuesday Maintenance window will be given over to a site wide effort to locate missing equipment. At this time, no other maintenance items are planned.  

J. Warner, R. Schofield, (J. Kissel remotely)

Jim called me this morning around 10a PT informing me that the ETMX ESD was unresponsive, preventing any lock re-acquisition. We eventually found that one leg of the HV power supply to the HV ESD driver at EX was off; unknown as to why. I reminded Jim how to restart it, then the ESD driver came back to live as normal after hitting the ON/OFF button on the HV ESD driver itself (no need to disconnect/reconnect any cables this time).

Diagnosis timeline:
I logged in an found that the HV monitor channels were all zeroes in the bottom right corner of the SUS ETMX overview, even though the digital request was the usual large bias and linearization voltages on the signal quadrants. I also noticed that because these monitor channels were reading zeros, the ISC_LOCK guardian -- which was in the DOWN state -- was hammering the remote ON/OFF button (I thought we fixed this??). After requesting that the ISC_LOCK just CHILL (i.e. pausing the guardian node), I waited a bit to let the HV ESD driver's internal mirco-processor relax then tried to remotely restarted it again. 
No dice. 
I advised Jim to head down to EX so we could debug together there (he'd gone with Robert). He went immediately to the VEA to try the ol' plug-un-plug cable trick, and found that the physical ON/OFF button did nothing, and that some indicator lights had shown that a leg of the 430 V input was red / dark. So, "we" went out to the entry foyer where the two HV power supplies live, and he found one off. We turned it back on --
 - Flip the rocker switch up to ON (wait for boot cycle to complete)
 - On the keypad, type
    - VSET > 430 ([V] for the amount of output voltage) > Enter
    - ISET > 80 ([mA] for the internal current limit) > Enter
    - Output ON/OFF
and then Jim went back out of the floor, the ESD HV driver looked much more lively, so he hit the button and the box came to live and I saw the voltage monitor channels go to their usual values. 

The only thing I can think of that trips the HV power supplies for the ESDs is the vacuum system (and when it does, it usually trips both), so I did a cursory vacuum system check on the EX overview screen, and didn't see anything crazy (i.e. I saw a bunch of green lights, and both vacuum Pirani gauges were sitting happily at ~1e-9 Torr. We'll debug more on Monday when I don't have to use the fragile remote connection to CDS.

Around 5:45 UTC, DARM started to glitch about every two seconds. The period is not exactly 2 seconds (I think it's just a little bit longer) but it does look pretty regular. I checked that the Pcal spectra look fine, so that's not the problem.

Attached is the change in the DARM spectrum. I wanted to use GDS-CALIB_STRAIN, but I get something that looks like it has dynamic range issues. We need to look into why that's not working. In addition, I'm attaching a Omega scan of the glitches.

Summary:

As reported before (28236), ISS 2nd loop MEDM screen appears as if it sometimes doesn't agree with reality. Turns out that it never truly agreed with reality. For example, in the first attachment, both SERVO ON/OFF indicator and additional 20 dB gain indicator are red while these are ON.

I was bothered enough to investigate and found that the MEDM screen has numerous nonsense logic and that the guardians are not written MEDM-friendly. I made a quick dirty fix for MEDM as well as guardians.

What was wrong:

Before going into details, you need to know that ALL of PSL 2nd loop I/O that are binary in nature is done via 16 bit DAC (don't ask me why). The output of DAC is sent to optical coupler via a resistor (see e.g. D1300439). To make anything ON, you want some large positive voltage to drive the LED in the coupler. To make anything OFF, you want to send sufficiently small (e.g. zero-ish) voltage or negative voltage.

As you can easily guess, one of the problems is the lack of convention. People sometimes send +32000 and other times +32700 to make the same thing ON. Sometimes -32000 and other times 0 to turn it OFF.

On top of that, there are many bugs in MEDM screen such that color indicators check some nonsense bits that never change, value set on pressing button but changed to a different number on releasing the button, that kind of things.

What was done:

I changed most everything in MEDM as well as IMC_LOCK and lsclibrary such that:

1. OFF is 0 so MEDM switch graphics using "if zero" and "if not zero" works correctly,
2. ON is 32000 for the sake of consistency but the color indicators check if the value is positive (instead of looking at some specific bit).

It seems as if it's working for now.

Madness example:

Just as an example. Manually pressing CHANGE PDs button could have made you believe that you selected PD1-4 when PD5-8 (3rd loop) was selected in reality.

In the second attachment, you can see that the CHANGE PDs "1-4" button sent 32000 (40V-ish differential) when pressed, but then 1 (less than 1 mV) when released.

PD1-4 is selected with large positive voltage (or ON or HIGH or whatever), otherwise PD5-8 is selected.

When you manually pressed and released "1-4",  the DAC output momentarilly went +40V differential, but immediately went down to some tiny voltage, so PD5-8 was selected in the end.

But the switch graphics was implemented assuming that non-zero output means PD1-4. Since the DAC output was 1 after pressing the "1-4" button, the graphics showed that PD1-4 was connected instead of 5-8. The PD indicator color box didn't help either as it checked the bit 2 of the DAC output, and it only turned to orange (CH5-8) only when DAC output was 32700.

The story was different for guardian as it used +32000 and -32000 for PD1-4 and PD5-8, respectively. When guardian did something, it looked as if PD1-4 was selected no matter what.