WP 6213
FRS 6114

Late entry for work done yesterday 10/4/2016.

FRS 6114 reports offset of 4000 counts in some of the fast I monitor channels. The AA chassis associated with these channels were powered off, no change in offset. Here is a list of the things we did.

1. The ribbon cables between the ADC card and the interface card were replaced on cards 1 and 2.
2. All ADC cards and interface cards were reseated.
3. IO SUSAUXEY was restarted, after ribbon cable swap and again after reseating the cards.

As of now, ADC1, ch19 still has an offset of ~4000 counts.

One thing to note, the adapter cards do not line up with the mounting hole screw. When the mounting screw is inserted, it lifts the back of the card up. This is something we have seen before. For now, I left the adapter cards without the mounting screw inserted.

J. Batc, F. Clara, R. McCarthy

SEI: Fixed the STS centering at End-Y; All OK
SUS: All OK
CDS: Fixed noise source at End-Y
PSL: Swapped Diode Chiller control panel; PSL has been running for 24 hours without a trip.
     Working on understanding the Crystal Chiller/Flow Sensor trips.
TCS: No report
VAC: All OK
FMC: All OK, No safety meeting today

Wednesday 15:00: Operators Guardian training session
Thursday 15:00: LIGO India Logistics kickoff meeting    	

TCS-X: No additional water required. Water level 29.0, 3.7 gpm, 20.1 deg. C.

TCS-Y: Found water level at 5.7, added 300 mL water, raised water level to 10.2, 4.0 gpm, 21.0 deg. C.

Before injections, I adjusted H1:SUS=SRM_M1_OPTICALIGN_P_OFFSET to minimize the 280 Hz jitter peak. I chose this offset because, when I looked back at 8 recent good locks, the jitter peak height was most sensitive to adjustments of this offset. I did this between about 2:10 and 2:30 UTC. The sharp jitter peak / background ratio was indeed very sensitive. The peaks were at a minimum around -680. Then I got in about an hour of injections. The first attachment shows the manipulations and a band pass around the 280 Hz peak and the second shows the basis for using this channel: the results of alignment offset manipulations since Sept. 23 on this band.

The laser power is noticeably lower since the beginning of last week.  I increased the diode currents by
0.3A and adjusted the diode temperatures.

diode currents were 50.2A, now 50.5A
diode box 1: no change
diode box 2: reduced temperature from 23 to 21.5 degC
diode box 3: reduced temperature from 23 to 22.5 degC
diode box 4: no change

Laser power went from 158.2W to 166.3W as reported by the power monitor on the MEDM laser screen.

Tonight we paused after a lockloss to look at the refl beam.  With PR2+PR3 misalinged and PRM, I turned on the refl WFS centering servos.  

First I tried to move both PRM and offsets inthe centering servos to see if there was an alingment where there is more light on the LSC refl diode.  There seems to be the maximum power on the diode when we have no offsets in the centering loops and PRM has the same alingment as durring a high power lock according to the osems.  There is no evidence for clipping here. 

Then I had a look at the RFAM, both by looking at the digital readback of REFL 9I and using a spectrum analyzer ( I lost the data from the analyzer in a floppy disk mishap).  BY unplugging the 9MHz modulation to the PSL, I got a reference for the noise that is not due to RFAM (shot noise, ADC noise, 60 Hz harmonics).  Just looking at the level of noise in the attached screenshot, and Evan's coupling measurement (29893), you would estimate that this noise is at the level of a few time 10^-21 m/rt Hz in DARM.

I also got some data for OMC length noise measurements, but will post those tomorow. 

TITLE: 10/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY:  PSL came online just as my shift was starting.  After a straightforward initial alignment, we made it to NLN with little issue (see aLog 30222 for the little issue).  Robert did PEM injections for the first half of the shift, then Sheila has been commissioning for the remainder.
LOG:

23:00-3:00 Robert to LVEA several times for injections

6:10 Filled TCSY chiller

6:40 Sheila to LVEA

6:57 Sheila out

I have reset the HEPI WD counters.  See attached screenshot for affected platforms.

T. Sadecki, J. Kissel, T. Hardwick, J. Rollins

After heroic efforts by the PSL team (LHO aLOGs 30217, 30216), with a restored laser Travis was able to get us most of the way up to nominal low noise. We got stuck going through LOWNOISE_ASC due to what we believe is a (new?) bug in the guardian code regarding the check for the input laser power. More details below, but we believe it's now fixed. 

We had paused for a while after increase power to make sure PI's were in check, but then we noticed suddenly, and without request, the laser power began to slowly decrease from ~48 W to ~42 W. Thinking (wildly guessing) that this had something to do with one of the many layers of ISS loops going awry, we decided to march the rest of the way up the acquisition sequence. That's when we hit the LOWNOISE_ASC bug. Once we fixed that bug, we were able to continue all the way up at ~42 W input power. Once we got to nominal low noise, I put the ISC_LOCK node into manual, and used the LASER_PWR node to push the input power back up to 48.1 W. Note, this required requesting the power to be as high as 58 W. Once at what we think is the ER10/02 desired power (and we got a big glitch in the IFO when requesting higher), I resigned the machine over to Robert to begin PEM injections.

LOWNOISE_ASC state problems:
After entering the LOWNOISE_ASC state, the ISC_LOCK guardian went red. The log file error:
    2016-10-05_00:53:38.914210Z ISC_LOCK LOAD REQUEST
    2016-10-05_00:53:38.914670Z ISC_LOCK RELOAD requested.  reloading system data...
    2016-10-05_00:53:40.781560Z ISC_LOCK module path: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py
    2016-10-05_00:53:40.781620Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
    2016-10-05_00:53:40.781720Z ISC_LOCK user code: /opt/rtcds/userapps/release/sys/common/guardian/ifolib/down.py
    2016-10-05_00:53:40.786580Z ISC_LOCK user code: /opt/rtcds/userapps/release/sys/common/guardian/ifolib/__init__.py
    2016-10-05_00:53:40.786700Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_library.py
    2016-10-05_00:53:40.786800Z ISC_LOCK user code: /opt/rtcds/userapps/release/als/common/guardian/alsconst.py
    2016-10-05_00:53:40.786900Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/fast_ezca.py
    2016-10-05_00:53:40.787000Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_GEN_STATES.py
    2016-10-05_00:53:45.923730Z ISC_LOCK RELOAD complete
    2016-10-05_00:53:47.826060Z ISC_LOCK W: Traceback (most recent call last):
    2016-10-05_00:53:47.826090Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/worker.py", line 461, in run
    2016-10-05_00:53:47.826100Z     retval = statefunc()
    2016-10-05_00:53:47.826100Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/state.py", line 246, in __call__
    2016-10-05_00:53:47.826110Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
    2016-10-05_00:53:47.826120Z   File "/ligo/apps/linux-x86_64/guardian-1.0.2/lib/python2.7/site-packages/guardian/state.py", line 246, in __call__
    2016-10-05_00:53:47.826120Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
    2016-10-05_00:53:47.826130Z   File "/opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py", line 3334, in run
    2016-10-05_00:53:47.826130Z     if self.low_power:
    2016-10-05_00:53:47.826140Z AttributeError: 'LOWNOISE_ASC' object has no attribute 'low_power'                                     <<< HERE'S THE PROBLEM
    2016-10-05_00:53:47.826140Z
    2016-10-05_00:53:47.862420Z ISC_LOCK ERROR in state LOWNOISE_ASC: see log for more info (LOAD to reset)
    2016-10-05_00:59:34.968070Z ISC_LOCK LOAD REQUEST
(Note for future reference, I was able to pull up the log file from the command line using guardlog --hours 2 ISC_LOCK > ISC_LOCK_log.txt, and then grep'ing for the error I knew was there. I'm not sure where guardian log files live such that you can just browse them.)

Looking into the ISC_LOCK code, we found this weirdness in the LOWNOISE_ASC state after CTRL+F searching for "low_power":
class LOWNOISE_ASC(GuardState):
    index = 507

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def main(self):
        if ezca['IMC-PWR_IN_OUT16'] > 45:             % This if statement is bonkers, and contradictory to the run state's conditional
            notify('power too Low!')
            self.low_power=True
        self.timer['LoopShapeRamp'] = 5
        self.counter = -1

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def run(self):
        if self.low_power:
            if ezca['IMC-PWR_IN_OUT16'] > 45:
                self.low_power=False
            else:
                notify('power too low!')
                self.low_power=True
        else:
            if self.counter ==-1:
                ezca.switch('ASC-CHARD_Y', 'FM6', 'ON') # Move compensation peaks for CHARD Yaw
        ...
        ...
        (etc)


Also, the real problem was that self.low_power wasn't initialized in the main portion of the code. So, Terra added the following initialization, and flipped the sign of the conditional statement in the main portion:
class LOWNOISE_ASC(GuardState):
    index = 507

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def main(self):
        self.low_power=False                # Initialize low_power attribute, 10/04/2016, see LHO aLOG 30222
        if ezca['IMC-PWR_IN_OUT16'] < 45:             # changed from > 10/04/2016, see LHO aLOG 30222
            notify('power too Low!')
            self.low_power=True
        self.timer['LoopShapeRamp'] = 5
        self.counter = -1

    @ISC_library.get_watchdog_IMC_check_decorator(nodes)
    @nodes.checker()
    def run(self):
        
    ...
    ...
    (etc)

With this bug fix, we were able to cruise through LOWNOISE_ASC. Really weird that this bug cropped up. Maybe some half-finished improvement to the state?

This is more like a preview.

Top red is DELTAL_EXTERNAL (not anti-whitened) multiplied by 2E6 to make it easier to compare the features with DBB QPDs measuring the HPO beam. Bottom is the coherence between DARM, jitter, and intensity noise measured by DBB QPDs.

As was expected a huge coherence peak is at 1kHz, but the coherence between DARM and 1QY (QPD1, PIT) is pretty high from 200 to 1kHz and at around 4.2kHz.

QPD signals look somewhat different from usual DBB scan plots (e.g. this one) in that 10-1000Hz slope is steeper in DBB scans.

It would be useful to measure coherence between DBB and various QPDs and ASC sensors (and it would be good if we can do it without going to the floor).

There's a caveat.

Jason and Peter taught me that DBB shutters could be manually opened using DBB crate front panel switches (it cannot be done via MEDM), so I went to the rack and opened the HPO DBB shutter while the IFO was locked in DC at 46W but not yet in nominal low noise (still lownoise ASC). That was a good part.

However, with manual mode, I didn't know if I can lock DBB PMC or keep it off-resonance and how, so all that was available to me was the QPD signals from the beam reflected by the DBB PMC which might or might not have been totally off-resonant.

I think it's a high priority to make DBB fully operatable.

Dave B, Joe B, Jeff K, Darkhan T,

Overview

It was discovered that a signal generated by a FixedPhaseOsc part, which is currently present in SUS-(E|I)TM(X|Y), CAL-CS, CAL-PCAL(X|Y) and some PI front-end models, is not guaranteed to have a fixed phase at GPS time 0 after a model containing the part is restarted.

See details for a suggested bug fix.

Before the bug is fixed in the code, one must update the frequency of an oscillator to some non-nominal value, say 0 Hz, and then set it back to its nominal value every time after a model with an fixed-phase oscillator is restarted. This operation should synchronize the phase of the oscillator to be 0 degrees at GPS time 0.

We restarted all of the oscillators in the SUS-ETMY, CAL-CS and CAL-PCAL(X|Y) models (cal. lines) after the issue was discovered.

Details

The FixedPhaseOsc part is expected to generate a periodically oscillating output with a requested frequency and a phase that would be 0 degrees if measured at GPS time 0. However, recent investigations revealed that the phase of a signal from FixedPhaseOsc was rather random every time a model is restarted (related alogs: 29992, 30197). We found that when a model is restarted, a FixedPhaseOsc static variable that stores the last known requested frequency does not get initialized properly. This problem can be fixed by replacing lines 100-101 in release/src/epics/util/lib/OscFixedPhase.pm with:

my $calcExp = "\L$::xpartName[$i]_freq = 0;\n";

This correction will result in a "true" condition in line 158, leading to proper initialization of the variables in the FixedPhaseOsc code.

G. Mendell, S. Karki, D. Tuyenbayev

Attached are plots showing the calibration factors (kappa_tst, kappa_pu, kappa_A, kappa_C and f_c)  for 26 Sept. 2016 generated from the Spectral Line Monitor (SLM)  data, analyzed using Matlab code from Sudarshan Karki and EPICS values from Darkhan Tuyenbayev found in

   D20160926_H1_CAL_EPICS_VALUES.m

in the calibration svn under,

  aligocalibration/trunk/Runs/PreER10/H1/Scripts/CAL_EPICS

The plots start at 1158913200 == Sep 26 2016 08:19:43 UTC

A full set of plots can be found by going here,

https://ldas-jobs.ligo-wa.caltech.edu/~gmendell/pcalmon_with_plots/daily-pcalmonNavigation.html

and clicking on the calendar in the left frame, and then any of the links to the plots in the middle frame. The plots for Sept 26, 27, and 28 have been updated using D20160926_H1_CAL_EPICS_VALUES.

(Plots and data have not shown up after Sept 28, due to a configuration problem introduced during maintenance on 27 Sept 2016. This is also now fixed and SLM should produce plots again, when we have more H1 analysis ready data.)

See also these related alogs:

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29992

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29184

The laser is gingerly up for now.  Even though it is running, it is not really fixed.
Another shake down involving the flow sensors will take place on Thursday morning.

An inquiry with Technotrans about the chiller controller yielded the information that provided the
software version the controller is running is the same, they ought to be interchangeable.  We have
always had to provide the serial number of the affected chiller in order to get a replacement
controller.

    The controller for the diode chiller (S/N 63793) was swapped with the one sitting in the
mechanical room (S/N 44806) that underwent some tests and was the chiller used before.  The
chiller started up okay.  No oscillations were observed in the chiller output temperature.





Jeff/Jason/Peter

Will need five days to run pumps in VEA, so likely will sit until O2a - O2b break?

While Daniel, Sheila and I were looking at some jitter signals, we noticed that the SUM channels that are used to normalize the DC QPDs do not have any lowpassing!!  The transmon QPDs already had these more modern parts, but they were never back-propagated to the ASC model. 

Also, while the trans QPDs had the right model parts, there aren't actually any lowpasses installed in the filter banks.  We'll want to give ourselves ~1Hz cutoffs to eliminate all the high frequency junk, while still allowing the normalization to use the overall power on the QPDs.

I have now done replaced the old QPD parts with the new ones in the ASC model, and successfully compiled the ASC model.  It has not yet been installed / restarted.

I'm almost done making the new screens, and should be done with another ~30 min of work tomorrow.

To be consistent with the transmon QPDs, rather than saving the _SUM channels, we will now be saving the _NSUM_OUT channels.  The _SUM channels will get the lowpasses, since they are used for the actual normalization of the pitch and yaw signals, so they're not what we'll want to look at for spectra and other things.  The _NSUM channels have the ability to be normalized with the PSL input power, but we are not using that capability (and haven't been with the transmon QPDs either), so really they're just the sum, with no lowpass.

The affected QPDs are:

REFL_A_DC, REFL_B_DC, AS_A_DC, AS_B_DC, POP_A, POP_B, AS_C, OMC_A, OMC_B, OMCR_A, OMCR_B, AS_D_DC(even though we don't use this last one).

Title:  10/04/2016, Day Shift 15:00 – 23:00 (08:00 - 16:00) All times in UTC (PT)
State of H1: IFO is unlocked due to ongoing laser issues. Start of maintenance day. 
Outgoing Operator: N/A
 
Activity Log: All Times in UTC (PT)

14:48 (07:48) Peter – Going to PSL electronics rack to work on PSL chiller problems
14:58 (07:48) Karen – Cleaning at End-Y 
15:00 (08:00) Chris – Escorting pest service down both arms and in Mids and Ends
15:06 (08:06) Ace Service on site
15:16 (08:16) VerbalAlarm – TCSY low water message - Vern added 0.5L water (aLOG 30199)
15:20 (08:20) Peter – Out of the LVEA
15:21 (08:21) Joe – Going into the LVEA to check batteries
15:42 (08:42) Centas on site to service matts 
15:43 (08:43) Hugh & Filiberto – Going to End-Y (WP #6211)
15:53 (08:53) Marc & Ed – In CER to replace power supplies (WP #6212)
15:55 (08:55) Chandra & Bubba – Crane work in LVEA (WP #6205)
16:23 (09:23) Kyle – Going to End-X for RGA work
16:25 (09:25) Set End-X to No BRS
16:32 (09;32) Betsy – Going into the LVEA for 3IFO
16:37 (09:37) Joe – Out of the LVEA
16:38 (09:38) Betsy – Out of the LVEA
16:42 (09:42) Coke Vending on site
16:45 (09:45) Norco - On site N2 delivery to CP1 
16:55 (09:55) Betsy – In LVEA to work on TCS-Y – Shutdown laser & Chiller work
17:03 (10:03) Karen – Finished at End-Y
17:05 (10:05) Peter & Jason – In the PSL Enclosure to check cabling
17:24 (10:24) Karen – Cleaning in LVEA near HAM5/6 to High Bay
17:40 (10:40) Pest service finished and leaving site
17:50 (10:50) Jason & Peter – Out of the PSL Enclosure
17:52 (10:52) Joe – Into LVEA to service eye wash stations
17:56 (10:56) Filiberto – Laying power cables in Biergarten
17:59 (10:59) Hugh – Finished with weekly HEPI checks
18:02 (11:02) Gerardo – Going to both end stations to check cabling
18:07 (11:07) Bubba – Out of LVEA – Finished with crane work
18:10 (11:10) Chandra – Out of the LVEA – Finished with crane work
18:14 (11:14) Norco – Leaving the site
18:18 (11:18) Joe – Out of the LVEA
18:28 (11:28) Karen – Out of LVEA – Opening the OSB receiving rollup door to remove trash
18:41 (11:41) Betsy – Going to both Mids for 3IFO stuff
18:59 (11:59) Sheila – Going into LVEA to turn on HAM2 illuminator
19:08 (12:08) Gerardo – Back from End Stations
19:09 (12:09) Kyle – Back from End-Y  
19:09 (12:09) Marc & Ed – Out of CER
19:10 (12:10) Karen – Out of the LVEA
19:12 (12:12) Sheila – Out of the LVEA
19:12 (12:12) Filiberto – Out of the LVEA
19:45 (12:45) Sheila – Going into LVEA to turn off illuminator at HAM2
19:56 (12:56) Betsy – Back from the Mid Stations
20:14 (13:14) Marc – Going to Mid-Y to get electrical parts
20:38 (13:38) Betsy – Going into the LVEA to turn on TCS-Y CO2 laser
20:50 (13:50) Betsy – Out of the LVEA
20:52 (13:52) Marc – Back from Mid-Y
20:57 (13:57) Keita – Going to PSL rack to check DBB shutter voltages
21:30 (14:30) Peter – Going into PSL Enclosure to dress up the PMC alignment
21:57 (14:57) Daniel – Going into LVEA 
22:03 (15:03) Fred – Going into the LVEA for 3IFO
22:15 (15:15) Daniel – Out of the LVEA
22:21 (15:21) Kyle – going to West Bay to retrieve tools and parts  
22:27 (15:27) Kyle – Out of the LVEA
22:38 (15:38) Peter – Out of the PSL Enclosure
22:40 (15:40) Keita – Out of the LVEA
22:45 (15:45) Fred – Out of the LVEA
23:00 (16:00) Turn over to Travis


Title: 10/04/2016, Day Shift 15:00 – 23:00 (08:00 –16:00) All times in UTC (PT)
Support: Peter, Jason,          
Incoming Operator: Travis

Shift Detail Summary: IFO has been down all shift due to laser problems and maintenance window. PSL team swapped an apparently faulty Diode chiller controller with the controller from the backup chiller. PSL has been up for almost 2 hours. Peter tweaked the alignment of the PMC. Starting an initial alignment