SEI Seismometer Mass Check (FAMIS#9628)

T240 Centering Check script run.  All proof masses within range.  "GREAT!"

STS Centering Check script run.  All proof masses also within range.  "GREAT!"

H1 BSC/HAM ISI CPS Sensor Noise Spectra Check (FAMIS task, #12894)

First SEI FAMIS test run post O3b, and SEI_CONF is currently in the SC_OFF_NOBRSXY state.

With that said, the only notable high-freq item is a subtle increase in frequency (1500Hz & up) for ETMy's Stage 2 V3 (see attached).

BS TF dtt template location

Given below is the location where I have stored all the dtt templates for the various measuremensts I have taken today,

/ligo/svncommon/SusSVN/sus/trunk/BSFM/H1/BS/SAGM1/Data/

Measurements taken when the optic was aligned, oplev damping was ON

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_L_0p01to50Hz_aligned.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_P_0p01to50Hz_aligned.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_R_0p01to50Hz_aligned.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_T_0p01to50Hz_aligned.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_V_0p01to50Hz_aligned.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_Y_0p01to50Hz_aligned.xml

Measurements taken when the optic was alignment was OFF, oplev damping was ON

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_L_0p01to50Hz_alignoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_P_0p01to50Hz_alignoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_R_0p01to50Hz_alignoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_T_0p01to50Hz_alignoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_V_0p01to50Hz_alignoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_Y_0p01to50Hz_alignoff.xml

Measurements taken when the optic was alignment was OFF and oplev damping was OFF

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_L_0p01to50Hz_alignoff_oplevoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_P_0p01to50Hz_alignoff_oplevoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_R_0p01to50Hz_alignoff_oplevoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_T_0p01to50Hz_alignoff_oplevoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_V_0p01to50Hz_alignoff_oplevoff.xml

2020-04-06_1500_H1SUSBS_M1_WhiteNoise_Y_0p01to50Hz_alignoff_oplevoff.xml

SUS health checks after Boise EQ - TF measurements for Beam Splitter (looks healthy)

I have attached the results (6 dof) for the transfer function for Beam Splitter suspension - it is free of any rubbing and looks healthy. These measurements were taken when the optic was in non-aligned state and the oplev damping was switched off.

 

BS ISI and HEPI RX/RY locations haven't changed before/after ID eq

I'm attaching trends of the RX/RY location monitors for the BS ISI and HEPI. The beginning of the timeseries overlaps the end of the run and ends today. The jump in the locations in the middle is the platform being put to bed for the shutdown, with the earthquake being about a third of the way from the beginning of that time. 

The main take away is the tilts of the platform aren't really any different now than they were before the shutdown. The 40 urad on/off rotation for HEPI is large, about 2x as big as the on/off shifts I've seen on the other BSCs. 

Nomachine session lists MEDM

The remote desktop nomachine system status is now available as a MEDM screen (see attachment).

The top portion shows the status of the EPICS IOC which is parsing and serving this data.

The lower area shows the current remote desktop session lists and associated statistics. The user list is a duplicate of that provided by the nxserver --list command (see below).

Emprically we found that active sessions have the remote IP field completed and inactive sessions have a dash in this field. This is used to update the active/inactive session counters.

A cronjob on opslogin0 running every minute directs output from the nxserver command and to the /var/run/nomachine_sessions file. The EPICS IOC reads this file once a minute and updates the PVs.

Example nxserver --list output:

Mon Apr  6 13:27:01 PDT 2020
NX> 127 Available sessions:

Display Username         Remote IP  Session ID                       Node           Client Type Client Version Client Platform 
------- ---------------- ---------- -------------------------------- -------------- ----------- -------------- --------------- 
1010    rahul.kumar      10.20.0.17 277A185632085B191523800F801C64DD localhost:4000 GUI         6.9.2          winnt          
1003    cheryl.vorvick   10.20.0.17 627A0180FB63A3D1B52F2BBD803F9966 localhost:4000 GUI         6.9.2          macosx         
1006    cheryl.vorvick   10.20.0.17 68E294E3A9FDCB5DA4A74BB0ACEBAB32 localhost:4000 GUI         6.9.2          macosx         
1018    jim.warner       10.20.0.17 FEF8A3C2FF8B2D967D20E7ED68130435 localhost:4000 GUI         6.9.2          winnt          
1002    david.barker     10.20.0.17 E720AC98E6DDE2BBCEFD704D4994C72E localhost:4000 GUI         6.9.2          macosx         
1005    carlos.perez     -          20CF28AD5B22FC43661265F6248E90DD localhost:4000 FF 68.0     -              notset         
1013    rahul.kumar      10.20.0.17 5FDD7010B4E60FD83A9918A6C16C8BFC localhost:4000 GUI         6.9.2          winnt          
1008    sheila.dwyer     -          F3E3D3DA4DA99924C418AEE497D0FE0D localhost:4000 GUI         6.9.2          winnt          
1004    keita.kawabe     -          D36E4093645839F3393214E9B097C1F2 localhost:4000 GUI         6.9.2          linux          
1020    jonathan.hanks   10.20.0.13 94CCCF41A2F472DAF1CDAFBD38561E5A localhost:4000 GUI         6.9.2          linux          
1014    corey.gray       10.20.0.17 3F4E6EA2CF5C3CE9A6F2E11913714A09 localhost:4000 GUI         6.9.2          macosx         
1019    jenne.driggers   10.20.0.17 F1F0489DCEEC430BFE0CBB28CEF9C291 localhost:4000 GUI         6.9.2          macosx         
1009    thomas.shaffer   -          9CA741CDB0917A072710D6E181B2B1D9 localhost:4000 GUI         6.9.2          macosx         
1015    jim.warner       10.20.0.17 DA24C6AEFC15D05BDE9CE0EE49B800C6 localhost:4000 GUI         6.9.2          winnt          
1007    corey.gray       10.20.0.17 CBF54C8F0B24F2B202A1EC1885FD1526 localhost:4000 GUI         6.9.2          macosx         
1017    david.barker     10.20.0.17 9FC1FCC06CEDD96881312DA1AF64878D localhost:4000 GUI         6.9.2          macosx         
1012    carlos.perez     10.20.0.17 D3D41731F4B82CA73F8D91844124799E localhost:4000 FF 68.0     -              linux          
1016    richard.mccarthy 10.20.0.17 B88BCCA4762EE840A6139B8EA28954B0 localhost:4000 GUI         6.9.2          winnt  

SUS health checks after Boise EQ - TF measurements for ETMY and ITMY (all healthy)

J. Kissel, Cheryl, Sheila, Jenne, Betsy, Jim, Keita, Rahul (all remotely)

I have performed the transfer function measurements for ETMY (all six dof) and ITMY (all six dof) and ITMX (only Length). Please find the plots (thirteen of them) attached below. Cheryl was with me over TS helping me with ndscope and checking if the suspension is ringing up or not.

Based on these plots I can confirm that ETMY, ITMY and ITMX (even though we have only L measurements) looks free of any rubbing and is healthy. Today’s measurements were compared to the one performed in 2018, and all dof matches up well (in terms of magnitude and coherence values). I would also like to add that while performing these TF measurements, we reduced the excitation amplitude by ¼ (from the 2018 template), since the original values were ringing-up (by looking at the inmon channels of the osems at the M0 stage, while running tf excitations) the sus chain everytime.  

J. Kissel also suggested that we switch on damping output filters for the M0 stage between 2 measurements – which we followed all the time.

I would also like to add our experience while performing ITMX transfer function in the morning. Cheryl and I started with the Length measurements. After the measurements got complete, 5 minutes later we found that the sus and ISI got tripped.  Jenne joined us remotely and along with Cheryl started damping the M0 stage. Initially, Cheryl applied a very small gain (-0.001), however on increasing it the damping got worse.

Then Jim (on Betsy’s request) noticed that we have tripped the stage 2 of the ISI – which explains why the damping got worse when Cheryl engaged it. He then restored the ISI to fully isolated (ISOLATED_DAMP) state, which includes damping.

Next, J. Kissel joined us on the TS channel and took over the damping work from Cheryl. His strategy was to engage only 1 osem at a time with a larger gain (0.5ish). He also engaged osems according to the differential mode or common mode for more effective damping. This worked and finally we were able to damp the suspension.

While on parallel J. Kissel also found out that running the TF with ISI ON is a safer way to prevent tripping. We followed his advice while doing our measurements on ETMY and ITMY.

I have saved all the data sets at the following locations,

/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMY/SAGM0/Data/

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_L_0p01to50Hz.xml

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_T_0p01to50Hz.xml

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_V_0p01to50Hz.xml

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_R_0p01to50Hz.xml

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_P_0p01to50Hz.xml

        2020-04-03_2200_H1SUSETMY_M0_Mono_WhiteNoise_Y_0p01to50Hz.xml

/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMY/SAGM0/Data/

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_L_0p01to50Hz.xml

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_T_0p01to50Hz.xml

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_V_0p01to50Hz.xml

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_R_0p01to50Hz.xml

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_P_0p01to50Hz.xml

        2020-04-03_2230_H1SUSETMY_M0_Mono_WhiteNoise_Y_0p01to50Hz.xml

 After the measurements were complete, I changed the coil output filter state back to 2.0 (from 1.0 for the test). Next I switched on the output filters for the M0_Damp and switched off the output filters for the M0_Test. I also reverted the gains for the M0_Test back to original values (these were changed for pitch and yaw only since other dof were at 1.0 as required). Finally the GUARDIAN state was set to ALIGNED (which was on DAMPED for tf).

basic comparison of Idaho 6.5 to Montana 5.8 2017 EQ

Summary: Local ground velocities at LHO were considerably larger in the Idaho EQ on March 31 55838 than in the 2017 Montana EQ, 38248  The top mass damping stayed on for all 4 quads this time, and the osems were swinging rail to rail for about half as long as they did in 2017, despite the faster ground velocities.

Ground velocity BLRMS at LHO:

	Dillon Mt July 2017	Salmon River Mountains, ID March 2020
30mHz-100mHz	19um/sec	50um/sec rms
100mHz-300mHz	61um/sec	180um/sec
300mHz-1Hz	20um/sec	120um/sec

The Montana numbers are from 43514  the attachment shows the Idaho EQ velocities. Although the local ground velocities at LHO were considerably larger for the Boise quake, our suspension top mass damping stayed on for the duration of the EQ.  Keeping the top mass damping on in these situations was a main goal of the watchdog and guardian changes made after the Montana EQ. 

I tried to make a comparison of the peak to peak motion of the top masses, but the osems are swinging out of range in both EQs.  The easiest metric to compare is how long they were swinging from rail to rail, which was about 140 seconds in the Montana EQ where top mass damping was off and about 140 65seconds in the Idaho EQ.  We could do some fitting of the actual data to estimate the peak velocities in both cases, I might try that next week.  For now we can take this as an indication that keeping the damping on helped us to swing rail to rail for a shorter time this time around.  

Usefull info for future reference:

We can use nds2 with ndscope following instructions here (Thanks!)

Nuc29 now has SUS and SEI overviews

[Jim, Jenne]

Jim and I have put overview screens for SEI and SUS on nuc29, covering up the ASC ndscopes that are currently unused.  This allows for more easily seeing the guardian status of these subsystems via the screenshots without having to actually log in.

H1SUSETMX Main Chain, Top-to-Top TFs Looks Very Healthy

J. Kissel [remotely]

Continuing health checks of the SUS after the Boise EQ (started in, e.g. LHO aLOG 55848 and LHO aLOG 55857), I took top-to-top transfer functions on the main chain (M0) of ETMX. 

This metric reports the suspension remains free of rubbing and functionally normally.

Full data set has been saved to 
    /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGM0/Data/
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_L_0p01to50Hz.xml
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_T_0p01to50Hz.xml
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_V_0p01to50Hz.xml
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_R_0p01to50Hz.xml
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_P_0p01to50Hz.xml
        2020-04-03_1939_H1SUSETMX_M0_Mono_WhiteNoise_Y_0p01to50Hz.xml

Just because it's a pain to gather screenshots, I only post three exemplary DOFs, but every other DOF equally matches the Jun 2018 reference data.

[EDIT] 
Note, while we were measuring ITMX, we ran into trouble with the IOP watchdog. This was a result of the configurations of the SEI systems at the time: 
    - Having HPI OFFLINE, ISI DAMPED is not a typical configuration of SEI system during top-to-top transfer functions, thus
    - The RMS of the raw OSEM signals -- during normal TF excitation -- exceeded the IOP watchdog threshold, triggering the start of the 5 minute countdown.
    - For some reason,  (again atypical) the residual motion after the measurement -- even with SUS damping loops turned back on -- remained large enough to *keep* the 5 minute timer counting
    - Eventually, after 5 minutes, the SEI system was forced to shutdown, tripping its watchdog.

In order to avoid this unnecessary confusion, for this measurement and all subsequent measurements, we've decided to being the SEI managers to ISOLATED_DAMPED, with HPI on an running as normal, and ST1 isolation loops running (sensor correction remains OFF though). With that configuration, and turning the SUS damping loops in between measurements, I was able to successfully complete the above suite without interference from the IOP watchdog.

Also -- as Cheryl correctly remembers, we must turn the Top Mass coil driver state from 2.0 to 1.0 during these measurements to allow for maximum DAC range.
The SUS guardian is in the "FULLY_ENABLED" state, such that M0_TEST outputs can go out, but the OPTICALIGN offsets are NOT engaged (again, to claw back DAC range).
[/EDIT]

Extended remote login list on CDS overview

The number of staff remotely logging into CDS now regularly exceeds the 6 slots I had on the CDS overview. I have now extended this to 16 slots, preferentially showing first names. 

h1pemmx running again after power cycle

Gerardo, Richard, Dave:

Gerardo power cycled the h1pemmx front end system while he was onsite this morning. The sequence was cpu off, io-chassis off, io-chassis on, cpu on. I restarted h1ioppemmx about a dozen times but always got a DAQ timing error. I then remembered we had seen this before, and it was resolved by a soft reboot after the power cycle. After soft booting h1pemmx the iop model started correctly on the second try. (the mid stations front ends do not have an IRIGB signal and get their time remotely with typically a 50% success rate).

h1pemmx is running again, attached is a 7 day minute-trend plot of the vault magnetometers

Battery Check at X2-8 and Y2-8

Y2-8
Solar panels are clear and in good condition, all cables are good.
Each battery was checked with a multimeter:
Top battery 13.06 VDC
Bottom battery 13.00 VDC
The LCD readout at the charge controller showed for both batteries 26.10 VDC

X2-8
Solar panels are clear and in good condition, all cables are good.  Removed a few tumbleweeds from on top of the panels.
Each battery was checked with a multimeter:
Right battery 13.79 VDC
Left battery 13.34 VDC
The LCD readout at the charge controller showed for both batteries 27.40 VDC

Batteries were installed on 10/2015.

SUS health checks after Boise EQ

J. Kissel, Cheryl, Sheila, Jenne, Betsy, Jim, Keita, Rahul

Since morning I have been working on the health checks for the QUADs (ETMs & ITMs) and Beam splitter, after we were stuck by 6.5 scale Boise earthquake, as reported by Corey (alog 55838). Jeff. K (and Betsy/Sheila) suggested an extensive health checklist and I am in the process of ticking them off one after other. Given below are the results from today’s measurements. These measurements were performed after coordinating/discussing with many folks remotely (due to COVID-19 Phase 3 state for LHO). 

1. I glanced over suspensions overview screen and checked for damping loops for the main chain as well as reaction chain. Nothing abnormal, the M0 damping loops were working normally for the main and reaction chain for all the ETMs/ITMs and BS.
2. CENTERING (checking the speed dials): I checked if the speed dials were within their normal range (counts: 10k – 20k approximately). I also checked if the values were fluctuating or was constant (like dead).

ETMY, L2 stage inmon values were found to be at 5k, which dropped from 10k after the earthquake. Please find attached an ndscope plot which shows the same. The speed dials for other optics looked fine.

3. OPLEV/OPTICALIGN measurements: I took a work permit (no. 8586) to perform the OPLEV measurements to check if the the bottom stage optics moves with the sldiers. Given below are the pitch and yaw numbers which I found,

Optics	Pitch values urad (HEPI OFF)	Yaw values urad (HEPI OFF)	Comments based on moving the sliders
ETMX	37.2	-6.3	movement observed as expected
ETMY	-50.4	3.9	movement observed as expected
ITMX	-27.3	-3.9	Very tiny (negligible ) movement observed for the pitch. Yaw looked fine
ITMY	-5.9	19.3	movement observed as expected
BEAMSPLITTER	15.0	11.8	movement observed as expected

  

On moving the OPTICAIGN sliders, I found that ITMX pitch did not change much (0.2 urad change in pitch values, while moving the sliders by 20 urad). Also, for the ETMX and ETMY the pitch values are higher than the typical values (+- 17 urad is highest I have seen).

The above Measurements were done while the HEPI was OFF. Hence, Jim turned the HEPI ON for ETMY and ITMX only. During this time the the Pitch and Yaw for ETMY was recorded as -52.9 and -2.9 respectively. Also, for the ITMX the pitch and Yaw numbers were -27.3 and 2.6. Hence, no significant change was observed after the HEPI was turned on. ITMX pitch was still stuck at -27.3 urad (+- 0.5 urads) on moving the OPTICALIGN sliders.

Next I will be performing a hysteresis measurements on ETM/ITMs and will alog the results. There are many more health checks still left to be performed, I will keep on posting the results as I have them.