For the past week the IMC ODC has been reporting bad times (this problem started last tuesday) due to the WFS gain being lower than expected. You can see ODC plots on the LHO summary pages (https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/) or more specifically at https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20140421/imc/odc/

H1:IMC-WFS_GAIN is set to 0.25, while the ODC is requiring == 1. Previously this gain had always been 0 (off) or 1 (on), so if a different value is going to be commonly used, we can adapt the ODC to match. 

The h1guardian0 computer appeared to have run out of all memory, logging in remotely was impossible, and logging in to the console was difficult.  No commands could be run on the console, with insufficient memory messages.  So I rebooted the computer.  

I do not know if any processes need to be started by hand to get guardian to run.  The computer is now working again.

model restarts logged for Sun 20/Apr/2014
2014_04_20 12:00 h1hpiham2
2014_04_20 12:01 h1hpiham3
2014_04_20 12:27 h1suspr3
2014_04_20 12:29 h1suspr3
2014_04_20 12:30 h1sussr3
2014_04_20 13:32 h1susmc1
2014_04_20 13:39 h1susmc2
2014_04_20 13:41 h1susmc3
2014_04_20 13:46 h1susprm
2014_04_20 13:51 h1suspr2
2014_04_20 14:16 h1suspr2
2014_04_20 14:22 h1sussr2
2014_04_20 14:28 h1sussrm
2014_04_20 14:41 h1susomc
2014_04_20 15:44 h1susim
2014_04_20 16:14 h1sushtts
2014_04_20 16:31 h1broadcast0
2014_04_20 16:31 h1dc0
2014_04_20 16:31 h1fw0
2014_04_20 16:31 h1fw1
2014_04_20 16:31 h1nds0
2014_04_20 16:31 h1nds1
2014_04_20 16:34 h1susim

no unexpected restarts. Another busy software development day followed by a daq restart.

Overview still shows DAQ problem with HEPI BS and ETMY, perhaps those models require a DAQ reconfigure?

Arm and BS were realigned. Beam was found in ISCT1 again. 00 Transmission was about 1750 counts, 20 was about 250 or so (this is just reading the unlocked fringe peaks). Tried to lock it but we were hit by another  big EQ which made it impossible to do anything meaningful.

The chillers at all stations produce seismic noise in the 5 to 25 Hz band because of turbulence in the pipes. The chillers can be the dominant source of seismic noise in their band. The chiller signal at EY tends to be peaked at 9 Hz (e.g. here), while the chiller signal at the LVEA and EX tend to be broader band, between 10 and 25 Hz (e.g. here). Studies have shown that the seismic level can be reduced by reducing the chiller pump speed (same links). For this purpose we have installed variable frequency drives at several pumps so that we can run them at reduced frequencies (typically around 35-45 Hz instead of 60 Hz). The chiller turbulence frequencies overlap with one of the most troubling resonances in the seismic isolation system, at about 10 Hz, so it is likely that we will want these VFDs on both pumps at the end stations.

The chiller turbulence does not bear primary responsibility for the recent unusually sharp peak at just above 10 Hz that is troubling SEI commissioners at Y-end, but it makes sense to try and reduce the background for them. With this in mind John and I have upped the pressure in the chilled water lines at both end stations and I have set the VFDs properly. The Y-end VFD pump is not yet being used because I could not get the chiller to start remotely, but I hope John and Ski can get this started Monday.

The figure shows the results of the changes we made at the other end station, X-end. A week of minute trends for the 10-30 Hz band is shown. Notice the regular spikes in quiet periods prior to the 4^th day. These have a 40 minute period, correlating with the cycling of the temperature control valve. We attempted to reduce them by increasing coolant pressure at the beginning of the 4^th day. This seems to have worked, but there is a short resurgence at the beginning of the 5^th day. My switch to VFD on the 6^th day is likely responsible for the significant drop in the quiet-time background after that.

I record the keying instructions for the VFD changes I made because the manual, like at least some other Mitsubishi manuals, is extremely poor (e.g. some keys are labeled differently on the controller than in the manual).

1) Set Pr. 79 operation mode to 3 (combined operation, external and hand). This mode allows external on/off and hand unit setting of frequency. 

Switch below controller OFF, PrSET, 79, READ, 3, WRITE

2) Set the minimum frequency to 30 Hz

HAND, PrSET, down arrow to Pr.LIST, READ, down arrow to Min.F1, READ, 30, WRITE

3) Set the run frequency to 35 Hz

HAND, 35, WRITE

Robert S. John W.

Our microphones go down to about 10 Hz; this was well outside the sensitive band for iLIGO, but wont be for aLIGO, so we would like to be able to monitor lower acoustic frequencies. For this reason, the Eotvos group in Hungary has developed infrasound microphones. A prototype in S6 showed coherence between DARM and infrasound (here) suggesting that these sensors may be important. At the end of last week, Gabor, Peter, Zsolt, and I installed infrasound sensors, one in each of the three stations.

The figure shows the coherence between two of the infrasound microphones in a huddle, demonstrating that they detect infrasound with high SNR between about 3e-3 Hz and 1 Hz. Between 1 Hz and 50 Hz there are regions between peaks (the peaks are probably LVEA modes) where the SNR drops, reducing the coherence. But the infrasound range does extend high enough in frequency so that it detects in the same region as the regular microphone (note the green trace coherence between the regular microphone and the infrasound microphone between 10 and 50 Hz). The peak evident in the infrasound traces at about 38 Hz is likely the lowest resonance of the membrane. The infrasound microphones do not appear to be sensitive to vibration: the black trace shows that there is little coherence with the seismometer, except for several peaks between 20 and 30 Hz. We checked that these peaks (which come from clean rooms) were not coupling through the ground by seismically isolating one of the infrasound microphones. The clean room peaks remained in the signal from the isolated microphone, suggesting that signals from the fans were travelling through the air as well as the ground, as might be expected.

The microphones were calibrated by matching the traces to the trace from the LVEA mic in the overlap region. I calibrated the LVEA mic with a B&K 250 Hz pistonphone. The figure shows that the frequency response of the individual infrasound sensors is not completely flat (the traces diverge below the calibration frequency), so we need at least one more calibration point at low frequency to reduce the uncertainty.

Rough infrasound sensor calibration (calibrated at high f):

EY: 680 uPa/Count

LVEA: 680 uPa/Count

EX: 770 uPa/Count

CART: 910 uPa/Count

Gabor Szeifert, Péter Bojtos, Zsolt Frei, Robert Schofield

J. Kissel

Following the changes made yesterday, I've finished all H1 model changes for reducing false alarm Rate (ECR E1400102) complete in a similar fashion on the remaining HAM SUS, i.e. HLTS, HSTS (including MC_MASTERs, and unhooked pr2) OMCS, HSSS. 

Sadly all these reboots seemed to have crashed something with guardian, because it's failing to restore the ISIs, I couldn't get SR2 to recover (via guardian), the IMC is complaining that the PSL REFL PD has no light on it, and indeed there's no light seen on the REFL or TRANS cameras. Perhaps Beckhoff went down? I don't how to restart it, or I would. Regardless, I've brought all the SUS, ISIs, and HEPIs up and confirmed that all alignments are the same.

Anyways, a fun investigation to get the week started right.

However, barring bugs, this should close out ECR E1400102 for H1, and L1 can either wait for more ECRs to be complete, or update incrementally as they wish. I'll post some more details and screenshots tomorrow.

Details:
---------------------------------

h1dc0 was rebooted at 
1082070016
2014-04-20 04:30 PDT
2014-04-20 23:00 UTC
and all models touched report a clean DAQ status, with the both frame writers happily chugging.

Files afected by these changes:
${userapps}/release/sus/common/models/
M       OMCS_MASTER.mdl
M       HSSS_MASTER.mdl
M       HSTS_MASTER.mdl
M       FOUROSEM_STAGE_MASTER.mdl
M       HLTS_MASTER.mdl
M       SIXOSEM_F_STAGE_MASTER.mdl
M       SIXOSEM_T_STAGE_MASTER.mdl
M       MC_MASTER.mdl

${userapps}release/sus/h1/models/
M       h1suspr2.mdl
M       h1susim.mdl
M       h1suhtts.mdl

/opt/rtcds/userapps/release/sus/common/medm
M       medm/haux/SUS_CUST_HAUX_OVERVIEW_all.adl
M       medm/hsss/SUS_CUST_IM_DACKILL.adl
M       medm/hsss/SUS_CUST_HTTS_DACKILL.adl
M       medm/hsss/SUS_CUST_HSSS_M1_WD.adl
M       medm/omcs/SUS_CUST_OMCS_OVERVIEW.adl
A       medm/omcs/SUS_CUST_OMCS_WD.adl
M       medm/hxts/SUS_CUST_HLTS_OVERVIEW.adl
M       medm/hxts/SUS_CUST_HSTS_OVERVIEW.adl
A       medm/hxts/SUS_CUST_HXTS_WD.adl
M       medm/tmts/SUS_CUST_TMTS_WD.adl
All of the above changes have now been committed to the userapps repository.

HAM systems remain: 
SEI: HPI HAM2, HPI HAM3
SUS: IMs 1-4, MC1, MC2, MC3, PRM, PR2, PR3, SR3, SR2, SRM, OMC

Starting with HEPI reboots, which will take down the mode cleaner.

Getting started roughly around 
1082055016
Apr 20 2014 18:50:00 UTC
Apr 20 2014 11:50:00 PDT

model restarts logged for Sat 19/Apr/2014
2014_04_19 13:26 h1hpietmx
2014_04_19 13:27 h1hpietmx
2014_04_19 13:30 h1susetmx
2014_04_19 15:01 h1susetmy
2014_04_19 15:03 h1hpietmy
2014_04_19 15:18 h1susitmy
2014_04_19 15:19 h1hpiitmy
2014_04_19 15:29 h1susitmx
2014_04_19 15:32 h1hpiitmx
2014_04_19 15:45 h1sustmsx
2014_04_19 16:10 h1sustmsy
2014_04_19 17:14 h1hpibs
2014_04_19 17:19 h1susbs
2014_04_19 18:35 h1broadcast0
2014_04_19 18:35 h1dc0
2014_04_19 18:35 h1fw0
2014_04_19 18:35 h1fw1
2014_04_19 18:35 h1nds0
2014_04_19 18:35 h1nds1

A busy commissioning day followed by a DAQ restart. No unexpected restarts.

J. Kissel

Beginning to tackle the changes approved in ECR E1400102, i.e. modifying of SUS User Watchdog to reduce false alarm rate, I've made it through the three BSC-type suspensions (and their respective HEPIs), QUADs, BSFMs, and TMTSs. The final change list includes:
- Removing the OSEM DC and Actuator AC trigger paths from the watchdogs
- Removing the Optical Lever watchdog trigger paths from the QUAD and BSFMs
- Merging all stages of USER and USER DACKILL MEDM screens
- Creating a one-click, reset all button for all stages of USER and USER DACKILL watchdogs (by hooking a single reset signal to each stage of USER WDs, and copying SEI's script for resetting the USER and USER DACKILL psuedo-simultaneously.)
- Changing the SUS USER DACKILL (which informs the "PAYLOAD" trigger flag of the ISIs) to use the AND of every watchdog stage instead of an OR of just the top mass [behavior is unchanged for TMTS, but TMTS is NOT hooked up to the ISI]
- Removing the SUS USER DACKILL connection to HEPI's PAYLOAD trigger

I've exercised the SEI / SUS watchdogs (by decreasing the trigger threshold below normal values) to ensure the desired behavior is achieved and all screens are reporting the correct status, and no funky reset loops happen. Of note, if just one of the USER WDs are green, then one can reset the USER DACKILL i.e. the PAYLOAD flag for the ISI. This is fine -- those user watchdogs which are still setting off triggers cannot be reset, as has always been the case. This just makes it that much easier to begin resetting a chamber after a "catastrophic" chamber trip, which almost always is because of a front-end boot fest, not anything physically happening to the chamber.

FIXME's as a result of these changes (for BSFMs and QUADs ONLY):
- Because the L3 (fr WUADs) and M3 (for BSFMs) no longer exist, the Guardian is getting confused in its checks for whether the suspension is tripped, and throwing an error. I tried digging into the sustools.py rabbit hole, but got lost and gave up quickly. I'll leave this to Jamie / Mark / Arnaud to fix. Regardless of the guardian failure, all SUS have been turned on, damped, with their previous alignment restored.
- Because the L3 (fr WUADs) and M3 (for BSFMs) no longer exist, I've terminated where the corresponding bit is fed into the ODC bitword creator instead of shifting everything around (again). As such, bits 7 (for QUADs) and 5 (for the BSFM) should be masked out and ignored until we're motivated enough to clean this up in the front end.

Though I've tried to restore all BSC chambers to full isolation, damping and alignment, the ground has been too violently non-stationary (wind, earthquakes, trucks, etc.) to keep things up for too long. I'll try again before I leave, but I doubt it'll stay up for too long.

After all changes were complete, h1dc0 was successfully rebooted at 01:35 UTC (18:35 PDT), with both frame writers (h1fw0 and h1fw1) happily cliking up there up-time (~1500 seconds as of this log)

Those files affected / created and now committed to the USERAPPS repository:

${userapps}/release/sus/common/src/
M       WATCHDOG.c
${userapps}/release/sus/common/models/
M       TMTS_MASTER.mdl
M       QUAD_MASTER.mdl
M       SIXOSEM_F_STAGE_MASTER.mdl
M       FOUROSEM_STAGE_MASTER_OPLEV.mdl
M       BSFM_MASTER.mdl
${userapps}/release/sus/common/scripts/
A       wdreset_all.pl
${userapps}/release/sus/common/medm/
M       bsfm/SUS_CUST_BSFM_OVERVIEW.adl
A       bsfm/SUS_CUST_BSFM_WD.adl
A       quad/SUS_CUST_QUAD_WD.adl
M       quad/SUS_CUST_QUAD_OVERVIEW.adl
M       quad/SUS_CUST_QUAD_R0.adl
A       tmts/SUS_CUST_TMTS_WD.adl
M       tmts/SUS_CUST_TMTS_OVERVIEW.adl
${userapps}/release/hpi/h1/models/
M       h1hpietmx.mdl
M       h1hpietmy.mdl
M       h1hpiitmx.mdl
M       h1hpiitmy.mdl
M       h1hpibs.mdl

J. Kissel, R. Schofield,

Robert was making ground motion comparisons between PEM channels and SEI channels, and discovered that the H1:ISI-GND_STS_ETMY_[X,Y,Z]_DQ were reading ADC bit noise. Confused, having heard that the STS-2 had been re-installed about two-ish weeks ago, I took a look around and found the usual confusion between A, B, and C, channels in HEPI vs. ISI. The agreed upon, end-station, cabling configuration is to plug in the STS-2 to the B channel for both HEPI and ISIs, such that it gets routed in at the top level of the front-end model, in through the isi2stagemaster library part, then up out again to be stored in the frames once and only once. However, this means that the STS distribution chassis must have its output plugged into BOTH HEPI and ISI's "B" ADC channels. Currently, the STS is happily plugged into HEPI's B channels, but ISI's version is plugged into the A channels. It also doesn't have its calibration filters turned on. 

Queue sad trombone. *wahn*wahn*. 

We'll fix it on Tuesday (or earlier if the analog CDS crew has time).

1081907495
Apr 18 2014 18:51:19 PDT
Apr 19 2014 01:51:19 UTC
Stage 1 and Stage 2 actuators.

Found the trip after 12 hours had passed, so couldn't check the seismic FOMs, but judging by the amplitude and frequency of the actuation control before the trip, and the large amounts of earthquakes recently, that would be my guess. There's a couple of small ~ 3 [mag] EQs arounf that time, capped with a 4.7 [mag] in Northern Alaska at 01:49 UTC. Disappointingly, USGS no longer has those "time of arrival" contour maps (or I couldn't find it with a bunch of clicking around the site), so you can't use that to narrow down which earthquake was yours. 

Fire department called at 10:58a PDT, asking to check out a box at the South Mid-Station, so I said "sure, no problem." After speaking with Robert, he suggests that when they get a glitch in their alarm system, they have to send someone out to confirm its not a fire, which we suspect is the case -- no fire (confirmed by corner station camera), glitch in the system, routine check. Will post if we hear anything further/otherwise.

model restarts logged for Fri 18/Apr/2014
2014_04_18 10:21 h1susetmx
2014_04_18 10:43 h1susitmx
2014_04_18 10:43 h1susitmy
2014_04_18 10:49 h1broadcast0
2014_04_18 10:49 h1dc0
2014_04_18 10:49 h1fw0
2014_04_18 10:49 h1fw1
2014_04_18 10:49 h1nds0
2014_04_18 10:49 h1nds1
2014_04_18 13:25 h1susetmy
2014_04_18 14:21 h1suspr3
2014_04_18 14:21 h1sussr3
2014_04_18 14:37 h1susbs
2014_04_18 14:44 h1dc0
2014_04_18 14:44 h1nds0
2014_04_18 14:46 h1broadcast0
2014_04_18 14:46 h1fw0
2014_04_18 14:46 h1fw1
2014_04_18 14:46 h1nds1

no unexpected restarts.

(Sheila, Alexa)

When I was at EY taking the noise measurements, I noticed that the fiber polarization (in the wrong pol) was up to about 60%. I asked Sheila to readjust the controller box for me. Now it's back down to 18%, and the controller is off. We should keep an eye on this drift...

(Alexa posting as Sheila)

J. Kissel

In order to get started on the suspension software ECRs with top priority, I needed to first clean up the library parts and MEDM screen from the prototype optical lever damping. I've completed the needed changes as described in ECR E1400207. All changes have been installed in the front-end code, the affected models have been restarted and restored, MEDM screens have been modified, and all committed to the userapps repo. Details of the changes are spelled out below. I have *not* captured any new safe.snaps, as folks have begun to lock the Y-ARM. 

Over the weekend I'll begin making the high-priority changes, starting with removing redundant watchdog components to reduce the false alarm rate as per E1400102. 

Stuart -- you should wait to begin re-merging the LLO models with the library parts until I've completed the majority of the changes.

Details
-------------------
I've done the following to the following SUS types:
QUAD:
- Cleaned up ${userapps}/release/sus/common/models/FOUROSEM_STAGE_MASTER_OPLEV.mdl, piping the ODC state bit for the optical levers to the output.  These bits have been inserted in the middle, after the M0 and R0 damp bits, into bit 11 and 12. This means the L1, L2, and L3 LOCK state bits are now 13-15. Notably, the optical lever damping loop drive path goes *through* the DRIVEALIGN matrix, to gather any P2Y, Y2P, or Angle to Longitudinal decoupling that we install -- this is different from how the BSFM optical lever damping was originally installed.
- Added outputs to L1 and L2 optical lever state bits to ODC vector in ${userapps}/release/sus/common/models/QUAD_MASTER.mdl
- Cleaned up ${userapps}/release/sus/common/medm/quad/SUS_CUST_QUAD_OVERVIEW.adl, and added the 2 extra ODC bits in the bitword light representation in the top right corner.
- Added two extra bits to ${userapps}/release/sus/common/medm/quad/SUS_CUST_QUAD_ODC.adl.
I *have not* updated the bit representation channels, I'll that to the ODC team.

HLTS:
- Modified ${userapps}/release/sus/common/models/HLTS_MASTER.mdl to use FOUROSEM_STAGE_MASTER_OPLEV.mdl on the M2 stage, modified the M3 stage to match the optical lever damping topology of the FOUROSEM_STAGE_MASTER_OPLEV.mdl (as usual, the bottom stage of every suspension type is different in some way, so it must remain unique). Both stages now output an optical lever damping bit to the ODC. As with the QUAD, the optical lever damping status bits have been inserted just after the M1 OSEM damping status bits. As such, bits 6 - 8 are M1, M2, and M2 damping, with 9, 10, and 11 being the M1, M2, and M3 LOCK status bits.
- Modified ${userapps}/release/sus/common/medm/hxts/SUS_CUST_HLTS_OVERVIEW.adl to include optical lever damping on M2 and M3, and added two bits to the ODC word lights in the top right.
- Created new screens ${userapps}/release/sus/common/medm/hxts/SUS_CUST_HLTS_[M2/M3]_OLDAMP_ALL.adl which contain the details of the two OLDAMP filter banks, virtually identical to the BSFM and QUAD versions.
- Modified ${userapps}/release/sus/common/medm/hlts/SUS_CUST_HLTS_ODC.adl to include two extra damping bits. As with the QUAD, I *have not* updated the bit representation channels, I'll that to the ODC team.

BSFM:
- Modified ${userapps}/release/sus/common/models/BSFM_MASTER.mdl to route the OLDAMP servo through the DRIVEALIGN matrix. Regrettably, because we will eventually need to install M1 to M2 stage OSEM damping to kill the highest Roll and Vertical modes of the suspension, we cannot just copy in and use FOUROSEM_STAGE_MASTER_OPLEV.mdl. Again, also worth note -- the OLDAMP path is now routed through the DRIVEALIGN matrix. There's currently no filters in the P or Y elements of the matrix, so the loop shape / design is unaffected. (The status bit had already been routed out from the M2 stage up in to the ODC vector).
- Made a few aesthetic changes to ${userapps}/release/sus/common/medm/bsfm/SUS_CUST_BSFM_OVERVIEW.adl such that the OLDAMP path matched the QUAD and HLTS.
- Modified ${userapps}/release/sus/common/medm/bsfm/SUS_CUST_BSFM_ODC.adl to include extra damping bits. As with the QUAD, I *have not* updated the bit representation channels, I'll that to the ODC team.

nm

Arnaud & Hugh

We quickly looked at this.  We didn't expect an effect but HEPI PS control runs at ~10hz (Epics) and we could check so we did.  The attached looks at the Pier top horizontals over the ground STS2.  All the horizontsl HEPI signals looked the same wrt both horizontals of the STS2s.  First Ref is ISI OFF, HEPI position loop ON, Second Ref is ISI OFF, HEPI Position Loop OFF, current trace is with the fluid flow zeroed.  No affect evident.

Summary: moving both red and green beams to a single periscope seems to have greatly reduced the periscope contribution to the RMS. In HIFO-Y the contribution was about 7 Hz while in HIFO-X, after the move, it is roughly 0.5 Hz. About half of the current HIFO-X periscope peak contribution comes from the periscope on ISCT1 (65-75, 95-105 Hz in the spectrum) with the rest coming from the periscope inside HAM1 (the 68 Hz sharp peak). The forest of peaks between 250 and 700 Hz come from optic supports along the beam paths on ISCT1.

I investigated vibrational noise in the current HIFO-X spectrum. Figure 1 shows coherence between H1:LSC-REFL_SERVO_SLOW_OUT_DQ and the environmental sensors with the most coherence, accelerometers on the ISCTEX table at EX, and the PSL and ISCT1 periscopes at the corner station. Vibration is the source of most of the signal 60 - 900 Hz, and may also become dominant at lower frequencies as other noise sources are reduced. Of course vibrational levels in the region around 100 Hz are now roughly a factor of two or three above what we hope they will be in science mode.

I tap tested ISCT1 while the arm was locked to confirm that the broad peaks in the HIFO-X spectrum at 65-75 Hz and 95-105 Hz were due to the ISCT1 red/green periscope. They do not closely follow the shape of the periscope peak in the accelerometer on the periscope (Figure 1), probably because the features in the HIFO-X spectrum are produced by differences in periscope motion along the red and green path, not total motion. Tap testing also showed that the forest of peaks between 250 and 700 Hz is mainly due to individual optic supports along the red and green paths on ISCT1.

Tap testing did not excite the sharp 68 Hz peak that sits on top of the ISCT1 periscope peaks. Figure 2 shows that I was instead able to excite it with a frequency-sweeping shaker on a blue cross beam of HAM1. The lower plot in Figure 2 shows that the shaker could not have been exciting HAM2 or ISCT1 enough to produce the peak, supporting the conclusion that the peak is due to motion inside HAM1. A very likely source of the 68 Hz peak is the tall periscope inside HAM1 (shown in Figure 3) used to direct the beams to the red/green periscope on ISCT1. Other optic supports in HAM1 should have much higher frequencies.

Moving the two beams onto a single periscope, suggested by Stefan, seems to have worked very well. In HIFO-Y the red and the green ISCT1 periscopes added about 7 Hz to the HIFO-Y RMS (here). In Shiela’s calibrated H1:LSC-REFLBIAS_OUT spectrum, with the calibration thought to be good to a factor of a couple at this frequency, the 65-75, 95-105 Hz portion of the ISCT1 periscope peak adds about 0.20 Hz to the RMS. The 68 Hz HAM1 periscope peak adds about 0.17 Hz (it added a lot more before the clean room was turned off).