After the EQ calmed down, I ran an initial alignment and then it relocked. Accepted a few ASC and LSC SDFs.
TITLE: 06/19 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 17mph Gusts, 14mph 5min avg
Primary useism: 0.11 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY: Wind has just calmed down, but we have 6.8 mag eq from Tonga on its way.
Lost lock at 05:43 (22:43) probably due to high winds. Have been buffeted by high winds and strong gusts all evening. Posted are plots of the winds over the past 4 hours. In the past 30 to 45 minutes there were several gusts of over 45mph up to one over 50mph at End-X. The high base line winds and gusts continue in the plus 40mph range. The X-Arm green has been locking (and dropping) and the Y-Arm is not locking. It appears to be off in Yaw. The camera image is clipping, so I suspect the BS is off in Yaw. Tried using the SEI_CONF MORE_WINDY state. However, this does not seem to make a difference. I am reluctant to run an Initial Alignment with the wind and microseism this elevated. I am going to remain attempting to relock, but not holding out much hope until the winds lay down.
IFO is locked and Observing. We are getting 110.2 Mpc of range from 37.2w of laser power. The wind is running at the least a Moderate Breeze (13-18mph) up to a Near Gale (32-38mph)range. Microseism is a bit rung up, as well. No other problems or issues to report.
Posted below are the HEPI pump trends for the past 45 days. The 4 CS pump pressures are steady with no noise. On 06/11 (last Tuesday) there was a small step down on EY pump pressure. The rest of the signals are a little noisy but the trends are mostly flat. Curious: Two Bad channel names, even though these channels appear in the plots. Connecting to NDS Server h1nds1 (TCP port 8088) Connecting.... done Error: Bad channel name 'H1:HPI-PUMP_EX_PS_PRESS1' Error: Bad channel name 'H1:HPI-PUMP_EX_CONTROL_VOUT' T0=19-05-05-01-14-55; Length=3888000 (s) 420 seconds worth of data was unavailable on this server 64799.0 minutes of trend displayed
[Keita, Jenne, Sheila, Georgia]
Today we had a quick look at the long trends of a few parameters before and after our 35 to 37W power up.
The attachment shows IMC input power, arm cavity circulating powers (calibration needs updating); PR gain; optical gain (kappa_C) and the DARM coupled cavity pole.
The drop in kappa_c but corresponding increase in cavity pole may suggest we need to revist our OMC ASC offsets to find an alignment that better optimised optical gain.
We also might want to revisit differential TCS settings (first guess: increase CO2X again to compensate for extra heating on ITMY).
Laser Status:
Front End Power is 31.95W (should be around 30 W)
70W Output Power is 70.23W
Front End Watch is GREEN
70W Watch is GREEN
PMC:
It has been locked 21 days, 6 hr 46 minutes (should be days/weeks)
Reflected power = 10.43Watts
Transmitted power = 54.28Watts
PowerSum = 64.71Watts.
FSS:
It has been locked for 0 days 2 hr and 57 min (should be days/weeks)
TPD[V] = 4.935V (min 0.9V)
ISS:
The diffracted power is around 2.1%
Last saturation event was 0 days 2 hours and 57 minutes ago (should be days/weeks)
Possible Issues:
GRB alert at 00:26 (17:26). LLO called to confirm. Trigger ID #582596766 with a duration of 2.048 seconds. All three sites are in Observing mode. No stand down initiated.
Contractor spent the day installing new components on all eight liquid nitrogen Dewar fill lines to bring skids up to industry standard. Components include check valve, pressure relief valve, strainer, and drain valve. At their recommendation, a support stand was installed and anchored into concrete upon hammer drilling holes. Corner and end station skids were completed before noon to minimize impact on observing.
Check valve = Nibco class 200 bronze T-453-B 1.5"
Drain valve = Rego Goddard T9454 1/2" F.NPT with BK9450R stem
Strainer = Mueller Steam Specialty LF352 1.5" with non-standard 100 mesh for cryogenic application
Pressure relief valve = rated for 350 psi
Picture of CP1 skid attached.
Note: the existing pressure relief valve just upstream of the control module safety valve on all skids, except CP1 skid, was removed to accommodate stack up length of new components
Worth noting that contractor purged pipe with gas N2 while brazing to prevent scaling on the inner surface.
The new strainer mesh is intended to prevent debris from entering the system, but does not filter ice. Contractor recommends cleaning it 2x/yr.
We could not get to Observe this afternoon because ISC_LOCK would not complete the NOMINAL_LOW_NOISE state.
Turns out the problem was ISC_LOCK was waiting for the INIT_ALIGN guardian to reach its "nominal" state, but this state was not declared. For today I added a line to the guardian naming INIT_ALIGN_COMPLETE as the nominal state, but maybe we should be sending it to an idle state once IA is complete?
I added an IDLE state to INIT_ALIGN, made it the nominal, and ISC_LOCK will request this in the PREP_FOR_LOCKING state to make sure we don't run an IA by mistake. This allows us to use this node for an initial alignment, or forgo it and do it the old way.
For LHO operators - I am currently in the middle of incorporating some additional features into the violin mode medm screen. This would mean that the medm screen would look confusing with bits and pieces of new channels added. This work should be finished by Wednesday (19 june 2019) and I will put an alog about it.
In the meanwhile, the please keep using the violin mode monitor as usual - the old channels still work and there are no changes being made to them.
TITLE: 06/18 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Lock Acquisition INCOMING OPERATOR: Jeff SHIFT SUMMARY: First day of NSF review. Praxair still working at mid stations. One lockloss from NLN. Relocking. LOG: Sheila working on PRMI 15:07 UTC Vanessa to LVEA to clean floors 15:09 UTC Christina taking item to mid X 15:16 UTC Peter to optics lab 15:22 UTC Mark C. through gate 15:22 UTC Chandra to HAM6 area 15:23 - 15:35 UTC Robert to HAM6 area 15:38 UTC Jason to PSL enclosure to move spare PMC 15:39 UTC Filiberto to end stations 15:39 UTC Georgia to optics lab 15:43 UTC Corey to end stations to retrieve laser glasses 16:02 UTC Gerardo to HAM5 16:06 UTC Vanessa to end X 16:10 UTC Bubba to LVEA to look at 3rd IFO parts 16:10 UTC Sheila done 16:12 UTC Marc to LVEA to swap baluns (WP 8244) 16:13 UTC Rahul starting charge measurements 16:20 UTC Georgia, Laurence, Dripta to end Y (WP 8246) 16:21 UTC Jason done 16:28 UTC Karen to end Y 16:28 UTC Corey back 16:31 UTC end Y to laser hazard 16:31 UTC Marc done 16:39 UTC Bubba done 16:46 UTC Unifirst through gate 16:57 - 17:06 UTC Bubba to roof 17:06 UTC Dave to end X to look at timing 17:12 UTC Restarted video2 17:15 - 17:39 UTC Daniel and Maggie to look at squeezer 17:26 UTC Robert to HAM6 and BSC2 17:29 UTC Hugh taking BS to damped 17:30 UTC Hugh powercycling CPS chassis in CER 17:33 UTC Dave back 17:35 UTC end Y charge measurement done 17:36 UTC Karen done at end Y 17:36 UTC end X charge measurement done 18:01 UTC Jim and Hugh to LVEA 18:01 UTC Gerardo done 18:04 - 18:13 UTC Bubba to LVEA to take picture of forklift 18:07 UTC Georgia locking Y arm on green 18:21 UTC NSF reviewer through gate 18:22 UTC NSF tour in control room 18:27 UTC NSF tour to LVEA 18:31 UTC Peter done in optics lab 18:34 UTC Hugh and Jim back 18:45 UTC Country Gentlemen through gate 18:48 UTC Corey to end X to sweep 19:02 UTC NSF tour out of LVEA, going to MSR 19:02 UTC Praxair running long, at end Y for another 15 min 19:12 UTC Georgia done at end Y 19:12 UTC Pepsi truck through gate 19:14 UTC Calum to VPW 19:15 UTC Corey done at end X, heading to end Y 19:16 UTC SEI_CONF to WINDY 19:24 UTC Georgia, Laurence, and Dripta back 19:41 UTC Corey done sweeping end stations 20:27 UTC Corey done sweeping LVEA 20:36 UTC Peter to optics lab 21:31 UTC Niko to optics lab 21:52 UTC Dave restarting mx stream 21:53 UTC Kyle to mid stations to check on Praxair 22:29 UTC Kyle back 22:37 UTC Chandra driving down arms to inspect dewar work
Jonathan, Dave:
In the past week we have had several occasions of DAQ-CRC error counts, primarily on h1seiex and also on h1susex. Here is what we have discovered so far:
All CRC errors are accurately reported in the DAQ data concentrator's log file
Errors appear at random times, and at random times within the second
When a front end has errors, for each model, its CRC counter increments by the same number every time
The number the CRC increments by is not the same across models. If it differs, it increments over the model position in the rtsystab file (e.g. IOP < USER1 < USER2)
The error, as seen by the data concentrator, is that the model's gps time is one second in the past
If, during the period of gps mismatch, a second boundary is crossed, both dcu_gps and gps increment within the same 16Hz cycle
The data concentrator marks the DCU DAQ-STATUS internally as 0x4000 (timing error).
This error propagates to the frame writer, which marks all channels from the model with this status
The error does not always show up on the EPICS STATUS PV (perhaps gets reset to zero before being reported?)
h1seiex:
number of CRC events since 11jun2019: 10
model order in mx_stream (num crc_errors each time): h1iopseiex(7), h1hpietmx(8), h1isietmx(8)
Error sequence: (hpi, isi) + 7*(iop, hpi, isi)
h1susex:
number of CRC events since 11jun2019: 2
model order in mx_stream (num crc_errors each time): h1iopsusex(8), h1susetmx(8), h1sustmsx(8), h1susetmxpi(9)
Error sequence: etmxpi + 8(iop, etmx, tmsx, etmxpi)
Attached plot of full data at the time of this morning's h1susex CRC glitch shows that for 8 cycles (0.5 seconds) the DAQ data is repeated. It is easy to see in a slowly varying channel (lower plot) and more hidden in the upper plot.
We reviewed the code and understand the internal mechanism of the error. There are several possible causes:
the front end models are slow in getting data into shared memory*
mx_stream is slow in reading the DAQ data from shared_memory
mx_stream is slow in processing the data (it calculates the data crc)
mx_stream is slow in sending the data
the network is slow in getting the data to the data_concentrator
* this is highly unlikely, however the mx_stream is triggered by the IOP being ready, and of the 4 computers at EX the two with problems have iop watchdogs.
Outstanding questions are;
Why is this happening only at EX?
Why did it only start last week?
Why only in h1seiex (10) and h1susex (2)?
We are reviewing our options for mitigation and/or further diagnostics.
Jonathan, Greg, Dave:
We have verified that both the full frames and the HofT frames are correctly marking the DAQ status of all impacted channels as bad (0x4000 = timing error).
Opened FRS13093 to cover this
SEI_CONF transition to EQ.
Transitioned back to WINDY around 21:00. Forgot to log the exact time since Hugh just fixed the SDF issue so we no longer drop out of Observe and I don't have a Verbal timestamp.
The SDF issue dropping us out of observing was the ETMX ISI Stage1 X Y Z CPS T240 & L4C BLND NXT filters. ETMY/ITMY & ITMX all had these filter banks Not Monitored. These ETMX channels were switched to Not Monitored.
For longer term study, I've opened FRS Ticket 13123.
Travis's 4-word aLOG claimed the 50000th aLOG and thus we honored this amazing achievement with the attached LHO Paper Plate Award! "This award goes to Travis Sadecki for his 5 words in the 50,000th aLOG (LHO aLOG 50000)." A truly astounding achievement. Who will claim the 100000th aLOG???
