h1lsc, h1omc, h1susitmx and h1susitmy model changes, DAQ restart

WP7120 Sheila and Dave:

The h1lsc model was modified to add two new SHMEM IPC senders. The h1omc model was modified to add two new SHMEM receivers (for the new h1lsc channels) and two new PCIE (Dolphin) sender channels. Each of the SUS ITM models were modified to add one new PCIE receiver channels to receive from h1omc.

The models were compiled and installed, the modifications to H1.ipc is shown below.

The models were restarted in the order: h1lsc, h1omc, h1susitmx, h1susitmy.

Because IPC receivers had been added, their slow channels were added to the DAQ INI file, requiring a DAQ restart.

After the DAQ was restarted, I discovered two things:

1. I forgot to check the write status of h1tw1 before restarting. I have checked that h1tw1 was writing raw minute trends every 5 minutes.

2. The EDCU did not come back to a GREEN status. The reason is that a new Guardian node ALS_DIFF_ETMY_ESD had been created (and added to the Guardian DAQ INI file) but is not being ran. We'll fix this tomorrow, for now the EDCU is RED, and it is missing these 19 guardian channels. The disconnected channel list is shown below.

ALS_DIFF_ETMY_ESD Guardian DAQ channels (disconnected):

H1:GRD-ALS_DIFF_ETMY_ESD_VERSION

H1:GRD-ALS_DIFF_ETMY_ESD_EZCA

H1:GRD-ALS_DIFF_ETMY_ESD_OP

H1:GRD-ALS_DIFF_ETMY_ESD_MODE

H1:GRD-ALS_DIFF_ETMY_ESD_STATUS

H1:GRD-ALS_DIFF_ETMY_ESD_WORKER

H1:GRD-ALS_DIFF_ETMY_ESD_LOAD_STATUS

H1:GRD-ALS_DIFF_ETMY_ESD_ERROR

H1:GRD-ALS_DIFF_ETMY_ESD_CONNECT

H1:GRD-ALS_DIFF_ETMY_ESD_EXECTIME

H1:GRD-ALS_DIFF_ETMY_ESD_STALLED

H1:GRD-ALS_DIFF_ETMY_ESD_NOTIFICATION

H1:GRD-ALS_DIFF_ETMY_ESD_NOMINAL_N

H1:GRD-ALS_DIFF_ETMY_ESD_REQUEST_N

H1:GRD-ALS_DIFF_ETMY_ESD_STATE_N

H1:GRD-ALS_DIFF_ETMY_ESD_TARGET_N

H1:GRD-ALS_DIFF_ETMY_ESD_OK

H1:GRD-ALS_DIFF_ETMY_ESD_ARCHIVE_ID

H1:GRD-ALS_DIFF_ETMY_ESD_TIME_UP

H1.ipc additional channels:

[H1:LSC-OMC_ITMX]

ipcType=SHMEM

ipcRate=16384

ipcHost=h1lsc0

ipcModel=h1lsc

ipcNum=59

desc=Automatically generated by IPCx.pm on 2017_Aug_27_13:36:33

 

[H1:LSC-OMC_ITMY]

ipcType=SHMEM

ipcRate=16384

ipcHost=h1lsc0

ipcModel=h1lsc

ipcNum=60

desc=Automatically generated by IPCx.pm on 2017_Aug_27_13:36:33

 

[H1:OMC-ITMX_LOCK_L]

ipcType=PCIE

ipcRate=16384

ipcHost=h1lsc0

ipcModel=h1omc

ipcNum=385

desc=Automatically generated by IPCx.pm on 2017_Aug_27_16:09:43

 

[H1:OMC-ITMY_LOCK_L]

ipcType=PCIE

ipcRate=16384

ipcHost=h1lsc0

ipcModel=h1omc

ipcNum=386

desc=Automatically generated by IPCx.pm on 2017_Aug_27_16:09:43

 

BS ST1 CPS trips

I came in to lock the IFO for Adam's injections which are scheduled for 2 hours from now, (38380), but have had about 4 BS ISI ST1 CPS trips in the last hour. 

Two of these might have happened while the BS was isolating stage 2 (I have not double checked) but at least one happened while we were just sitting in CHECK_IR (no feedback to the BS sus, ISI state should not have been changing).  The screenshot attached is for that most recent trip in CHECK_IR

Edit:  Made it to a fully locked interferometer and was engagning ASC when it happened again, again it was a glitch in the ST1 H3 CPS.  Second screenshot 

Edit again: This seems very similar to the situation described in 37499, 37522 the third screenshot shows the glitches getting worse over the last 1:40 minutes.  Jim describes power cycling the satlite racks which seemed to make the problem go away last time. 

Another update: I power cycled the 3 top chassis in the CER rack SEI C5, these are labeled BSC ISI interface chassis and have cables going to BSC2 CPSs, GS13s and L4C's.  The ISI has not tripped in about 20 minutes, I am relocking but paused to damp violin modes.

CDS O2 restart report: Thursday 24th - Friday 25th August 2017

Last O2 report:

model restarts logged for Fri 25/Aug/2017
2017_08_25 17:06 h1sysecatc1plc1sdf
2017_08_25 17:06 h1sysecatc1plc2sdf
2017_08_25 17:06 h1sysecatc1plc3sdf
2017_08_25 17:06 h1sysecatx1plc1sdf
2017_08_25 17:06 h1sysecatx1plc2sdf
2017_08_25 17:06 h1sysecatx1plc3sdf
2017_08_25 17:07 h1sysecaty1plc1sdf
2017_08_25 17:07 h1sysecaty1plc2sdf
2017_08_25 17:07 h1sysecaty1plc3sdf
2017_08_25 17:09 h1pslopcsdf
2017_08_25 17:13 h1hpipumpctrlsdf

50 minutes after the end of O2, h1build froze up and required a reboot. The Beckhoff SDF models were then restarted.

model restarts logged for Thu 24/Aug/2017 No restarts reported

 

Another attempt at Detchar Safety Injections

Sheila plans to come in and lock the IFO this afternoon, so that I can try to finish these detchar safety injections. I've schedule 7 injections, the first one will begin at 16:06:22 PDT, and the last at 16:51:22 PDT. Here is the update to the schedule:

1187824000 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187824450 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187824900 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187825350 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187825800 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187826250 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187826700 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt

The IFO should be left out of observation mode. Also, I'm not sure if the PCALX 1500Hz line gets turned back on, but this also needs to be turned off. Simply typing the following on the command line will do the trick:

caput H1:CAL-PCALX_PCALOSC1_OSC_SINGAIN 0
caput H1:CAL-PCALX_PCALOSC1_OSC_COSGAIN 0

% Plot hardware injections
fnm = 'detchar_27July2017_PCAL_H1.txt';
data = load(fnm);
t = 0:1/16384:(length(data)-1)*1/16384;

figure;
plot(t,data)
xlabel('time [s]')
ylabel('amplitude [h]')
title(strrep(fnm,'_','\_'))
orient landscape
saveas(gcf,'injections.pdf')

% Print times of injections
t_start = 1187824000+0.5;
times = t_start:3.0:t_start+106;
fprintf('%.2f
',times)

first sets of measurements post O2

Thomas Vo, Sheila Dwyer

We started out pre vent/discharging measurements in tonight.  

• First we ran the automated version of the ESD actuation strength measurements described using the attached script, we got data for all 4 ESDs.  
• We made an attempt to transition DARM control to the ITMX ESD but noticed that we are missing IPCs that would allow us to send DARM to the ITMs. 
• We intentionally unlocked the IFO so Thomas could take some HWS measurements for the ETMs, but has difficulty getting a reference at DC readout. While we were moving the PZT misalingment biases the interferometer became misalinged and unlocked.  We will need to investigate this before continuteing the HWS measurements. 

Restart h1build due to a kernel panic

The CA sdf systems stopped.  After a quick investigation it was found that the server running the SDF systems had locked up.  H1build had suffered a kernel panic and needed to be restarted.

I manually started the following SDF models:

h1sysecatc1plc1sdf 
h1sysecatc1plc2sdf 
h1sysecatc1plc3sdf 
h1sysecatx1plc1sdf 
h1sysecatx1plc2sdf 
h1sysecatx1plc3sdf 
h1sysecaty1plc1sdf 
h1sysecaty1plc2sdf 
h1sysecaty1plc3sdf
h1pslopcsdf 
h1hpipumpctrlsdf 

And At The Strike Of 3pm PDT, H1 (& L1 & V1) Were Done With O1!!!!

Attached are photos for our End Of O2 Toast at LHO!  :)

Ops Day Shift Summary

TITLE: 08/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY: Last shift of O2! No issues until Beckhoff SDF died during Adam's detchar safety injections. Jonathan has restarted h1build and it is currently running a file system check. Intention bit set to commissioning. Observatory mode set to commissioning. IFO at NLN.
LOG:

16:02 UTC Powercycled video2. MEDM screen updates were intermittent.
16:07 UTC Restarted the range integrand DMT that is displayed on video0. It had been running locally on video0 instead of through ssh on nuc6.
17:23 UTC Gave Adam M. remote access to cdsssh and h1hwinj1.
18:50 UTC GRB verbal alarm. Confirmed with LLO.
15:00 UTC End of O2! Set observatory mode to calibration. Changed INJ_TRANS guardian node to INJECT_SUCCESS.
22:43 UTC Adam M. starting detchar safety injections. Set intent bit to commissioning. HFD to mid X to check on RFAIR box.
~23:09 UTC Beckhoff SDF died. IFO_LOCK guardian node went into error (lost connection to SDF channels). Adam's injections interrupted. Jonathan investigating.
23:23 UTC Jonathan restarted h1build and it is running a file system check. Adam canceling remainder of injections. Transitioned observatory mode to commissioning.

15:00 UTC should be 22:00 UTC.

Detchar Safety Injection

While the LHO folk are celebrating the end of the O2 run, I decided to sneak in some detchar safety injections. Here is the update to the schedule file:

1187736700 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187737150 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187737600 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187738050 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt
1187738500 H1 INJECT_DETCHAR_ACTIVE 0 1.0 detchar/detchar_27July2017_PCAL_{ifo}.txt

End of run hardware injections

I will be carrying out coincident hardware injections into L1 and H1, right after the O2 run ends. I've updated the schedule file and uploaded it to the injection svn, and checked it out onto the hardware injection machines. Here is the update to the schedule file:

1187734000 H1L1 INJECT_BURST_ACTIVE 1 1.0 config/Burst/Waveform/{ifo}/400_400_maxspin_hwinjcbc_1126259455_{ifo}.txt
1187734300 H1L1 INJECT_BURST_ACTIVE 1 0.5 config/Burst/Waveform/{ifo}/400_400_maxspin_hwinjcbc_1126259455_{ifo}.txt
1187734600 H1L1 INJECT_BURST_ACTIVE 1 1.0 config/Burst/Waveform/{ifo}/100_20_nospin_edgeon_hwinjcbc_1126259453_{ifo}.txt
1187734900 H1L1 INJECT_BURST_ACTIVE 1 0.5 config/Burst/Waveform/{ifo}/100_20_nospin_edgeon_hwinjcbc_1126259453_{ifo}.txt
1187735200 H1L1 INJECT_STOCHASTIC_ACTIVE 1 1.0 stoch/Waveform/SB_O2A_{ifo}.txt

These are four burst injections and one stochastic injection. The first one starts at 15:06:22 PDT.

Ops SVN Check

patrick.thomas@zotws7:~$ check_sdf_snapfiles_status
WARNING: local modification on file /opt/rtcds/userapps/trunk/psl/h1/burtfiles/fss/h1pslfss_safe.snap
WARNING: local modification on file /opt/rtcds/userapps/trunk/psl/h1/burtfiles/fss/h1pslfss_safe.snap
WARNING: local modification on file /opt/rtcds/userapps/trunk/omc/h1/burtfiles/h1omc_OBSERVE.snap
WARNING: local modification on file /opt/rtcds/userapps/trunk/tcs/h1/burtfiles/h1tcscs_OBSERVE.snap


h1pslfss_safe.snap:
-Time:      Tue Aug 15 16:38:08 2017
+Time:      Fri Aug 25 00:27:45 2017
-H1:PSL-FSS_COMMON_GAIN 1 1.600000000000000e+01 1
+H1:PSL-FSS_COMMON_GAIN 1 1.602103999999996e+01 1


h1omc_OBSERVE:
-Time:      Tue Jul 18 12:07:52 2017
+Time:      Fri Aug 25 00:27:37 2017
-H1:LSC-ASAIR_A_RF45_I_OFFSET 1 -7.589000000000000e+01 1
+H1:LSC-ASAIR_A_RF45_I_OFFSET 1 -7.220000000000000e+01 1
-H1:LSC-ASAIR_A_RF45_Q_OFFSET 1 2.000000000000000e+01 1
+H1:LSC-ASAIR_A_RF45_Q_OFFSET 1 2.150000000000000e+01 1


h1tcscs_OBSERVE.snap:
-Time:      Tue Apr  4 18:11:09 2017
+Time:      Thu Aug 24 03:19:07 2017

Changes committed.

Ops Day Mid Shift Report

Have remained in observing. Adam M. has logged in remotely to prepare for hardware injections. No issues to report.

STS2 Wind/Floor Tilting Measurement now at Roam9

Thursday afternoon we had a brief but strong wind event--See first plot: 2 day's minute trends.  I had moved the roaming STS2 a week or more ago and now resides at position Roam9.  At the corner station the wind average touched on 20mph for maybe 90 minutes with a stable direction of about 170^o or ~from EndX toward the corner.  No non-wind seismic evident.

The second attachment shows the usual STS ASD comparisons between the Roaming HAM5 and the Detector used ITMY STSs, see 38184 for the Roam8 comparison and you can follow back from there.  Great news: for the X dof here at Roam9, the HAM5 STS shows several factors less floor tilting than ITMY starting at 30ishmHz; and, the Y dof is maybe no worse--is that band between 10 & 100mHz too much?

So bottom line Roam9 shows promise!

Times: RefTime is low wind, 25 Aug 0930 utc.  High wind is 0130 25 Aug.  XMLs in /ligo/home/hugh.radkins/STS_Seismometer_Studies

TMDS training session at X-end

Rai W., Mike Z., Daniel S., Jeff K., Chandra R., Kyle R. 

Today the TMDS Gas Delivery Table was transported to the X-end VEA and set it up for operation.  The actual Surface Discharge Ionizer was mounted on a stand for today's demonstration purposes.  Nominally, it would be mounted to the dedicated TMDS port of BSC9's East Door.  The Vent/Purge air supply was ran and dry air supplied to the TMDS setup.  Prior to exposing the Ionizer to the conditioned air supply, the plumbing connection at the Ionizer was decoupled and connected to a test setup that facilitated the measurement of > 0.3 micron and > 0.5 micron particulate in the conditioned air. The air line was then re-coupled to the Ionizer.  An approximate 50/50 mix of Ethylene Glycol to Alcohol along with "chunks" of dry ice was used as a cold trap to the path.  

Rai W. gave an overview of the theory and a detailed tutorial as to the operation of the setup.  Daniel S. and Jeff K. noted details related to the ion production while Kyle R. and Chandra R. noted detail related to the vacuum related issues.  

The Ionizer setup performed satisfactorily (with caveats) at making comparable amounts of positive and negative ions.  

The plan now is to connect the Ionizer to BSC9 and perform an actual discharge of the ETMx during next Tuesday's maintenance day.  

My notes from today; 

1. Oscillations were noted and attributed to electronics design -> Daniel S. will address.  
2. Rai W. requested that we provide a means of support to take the weight of the Ionizer when mounted to the 1 1/2" valve on the BSC -> Kyle R. will address.  
3. Confirmation of the preferred means to pump to rough vacuum the X-end VE between cycles of discharge gas admission -> Chandra R. will address.  

Refer to Figure 1 of T1400713, TMDS design document for elements of setup.

A couple of useful videos of the oscilloscope readouts during the experiment:

Lesson 1: Control and readback of the electrometer from the TMDS inferface chassis, D1500152.
    Rai has set up the readback of the electrometer with the square-wave input shown on the blue trace, and the electrometer readback in yellow. One is looking for an even balance in voltage from the positive and negative ions. The electrometer has a negative voltage bias at the time of recording, so it appears as though we're getting more positive than negative ions, but, rest assured, we're good. 

Lesson 2: Demonstration of ionic discharge against TMDS chamber walls, if HV supply to ion generator is too high.
    The readback of the HV current is shown on the yellow trace. The video starts out with the ion generator discharging, as is evident by the rattiness of the waveform. At the last few seconds of the video, he reduces the VARIAC HV transformer gain, bringing the ion generator back to the desired level.

Lesson 3: Demonstration of electrometer readback once HV voltage is reduced
    Once the HV is tuned with the VARIAC, with the initial max amplitude of the square wave generator, then the square wave amplitude may be reduced to ~ 3 V pkpk (assuming a flow rate of about 30-50 [mL/s ?? not sure about those units]). Once we did this, we saw evidence for a ~3 MHz oscillation on the electrometer readback. Investigating the circuit drawing for the electrometer board (D1500061), Daniel and Mike agreed to replace the readback's output resistors R9 & R10 with ~100 [ohm] resistors. 
This has been done already, and the electrometer has been reinstalled

Lesson 4: Final setup of readback / monitor system, in the "ready to discharge into chamber" configuration.
    Once again, Rai demonstrates the control of the HV VARIAC and the desire to keep the ion generator current readback from any rattiness in its waveform, which is indicative of ionic breakdown discharge against the TMDS chamber walls.

I attach a ton more pictures of the setup as well. Hopefully the filenames are a good enough caption.

Desired gas flow rate 30 - 50 Liters per Minute (LPM)

All H1 QUADs Have Complex Non-f^6 Dynamics above 50 Hz

J. Kissel

I've processed the data from LHO aLOG 38232, in which I was looking to see if the complex, high frequency dynamics found in the UIM (L1) stage of H1 SUSETMY (see LHO aLOG 31603).

Plots are attached; note that I've used the former 2016-11-17 data for ETMY. To guide the eye, I show the calibration group's dynamical model used during O2. Data shown with a coherence threshold of 0.75 or greater. I've not yet compared this against the unphysical, fitted update to the QUAD model that Brett developed for H1 SUS ETMY in CSWG aLOG 11197

It looks like 
- All suspensions see the "unknown" ~165 Hz resonance, though interestingly ITMY's mode appears to be split. ETMX and ITMY's are *strikingly* similar.
- only ETMY and ITMX show what Norna suspects is the "UIM blade bending mode" at ~110 Hz. ETMY's is indeed the ugliest.
- All but ETMX show evidence for resonances between 290 and 340 Hz, which are likely the (Suspension Point to Top Mass) wire violin mode fundamentals weakly excited by the UIM excitation. (see LHO aLOG 24917, T1300876)
- All suspensions show their (UIM to PUM) wire violin mode fundamentals between 410 and 470 Hz (see LHO aLOG 24917, T1300876)
- ETMY and ITMY's wire violin modes are *strikingly* similar; ITMX's modes are surprisingly low Q.
- We don't see the TOP to UIM wire violin modes likely because I didn't excite up to high enough a frequency; they're between ~500-510 Hz for ETMY (modeled at 495 Hz).


So now the question becomes -- what's the physical mechanism for this huge mystery resonance that's seen virtually identically in all QUADs? It seems quite a fundamental feature. Why is it split on ITMX?

It would be nice to have similarly detailed measurement of L1's QUADs, but all evidence from calibration measurements on L1's ETMY UIM stage point to this same feature (see the discrepancy between model and measurement for the UIM stage, e.g. in 2016-11-25_L1_SUSETMY_L1_actuation_stages.pdf LHO aLOG 29899). Further, it's *still there* before and after their Bounce/Roll mode Dampers (BRDs) have been installed (i.e. the same feature is seen in O1 and O2 CAL measurements; the above linked measurement was after BRD install). 

If anyone needs to fit this data to try out their FDIDENT skills, the data from the above concatenated transfer UIM to TST functions is attached, in the same format as before for ETMY. 

Analysis script lives here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O2/H1/Scripts/FullIFOActuatorTFs/
process_H1SUSQUAD_L1_HFDynamicsTest_20170814.m