TITLE: 07/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 9mph Gusts, 8mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY: No issues to report. Lock is 9+ hours old.
J. Kissel, N. Lecoeuche, C. Vorvick While clearing out SDFs prior to this observe segment, we found that PRM showed a DIFF with its OBSERVE.snap regarding the state of the switchable coil driver response. Cheryl spoke of this recently, in LHO aLOG 50273, whom was merely following Corey's lead who accepted the coil driver state response to be in State 2 LHO aLOG 50257. A bit of education: even though we do not request any use of the M2 stage of PRM while in NOMINAL_LOW_NOISE (which gives the impression that "it doesn't matter what state we're in for observation"), the DAC channels that are connected to this stage still does emit a constant level of voltage noise, which -- in some of the switchable states, not all -- the coil driver is designed filter out that non-zero voltage noise. A reminder: PRM is a "recycling cavity HSTS" whose M2 and M3 coil drivers have both been modified to increase their drive ability by reducing the output impedance (and thus allowing more of said DAC noise to be turned in to current by the driver). The state machine diagrams that reflect the response for this type of driver are found under T1100507, specifically that for the MODified Triple Acquisition driver: State 1 does "nothing," and passes through the voltage with only a fixed trans-conducance of ~3 [mA/V[. State 2 engages the switchable "acquire" circuit response of a gain-of-5x boost above 13 Hz. State 3 engages the switchable "low-pass" circuit response of a factor-of-ten reduction between 1 and 210 Hz State 4 is a weirdo state that engages both the acquire and low-pass resulting in a lumpy low-pass of 10, and a gain of 5. So -- we want state 3 when in nominal low-noise such that we get as much filtering of the DAC noise as possible. BUT -- there are several things weird here. (1) Cheryl and I've double checked her plot that showed that the state was semi-regularly changing, but we can't reproduce her data. Both ndscope and dataviewer confirm that this state *rarely* changes, and only intentionally by the commissioning team (e.g. when we need extra drive to excite bounce and roll modes a. la. LHO aLOG 49643). The last time this happened was on May 28th, and we can confirm that we *do* see that change, it was quickly changed back after the testing was complete, and it has been in State 3 since, as it should be. (2) This state is not changed during any of the lock acquisition sequence. So -- it's unclear why this is showing up in SDF, or why on June 27th -- a month later -- it shows up for Corey as an SDF DIFF. But -- the message -- We want State 3, so if anyone sees the SDF claim it should be 2 in the future please aLOG it and let me know.
Matthew Ball, Adrian Helmling-Cornell, Jim Warner, Robert Schofield
Noise around 48 Hz in DARM is often driven by variable frequency fans of chiller units; dip switches could be changed to eliminate high fan frequencies when not necessary.
The 48 Hz band in DARM is often increased by variable frequency HVAC fans that step through this band (see Figure 1). We tracked one of the intermittent signals to a particular HVAC unit CU-2A, that I think is cooling the electronics bay (Figure 1). These and many of the other units that are stepping in frequency are Mitsubishi model PUHY-P, or similar units. I think we can prevent the fans from stepping into the 48 Hz band by setting Dip SW4-4 to “off” to put it in low-noise mode (uses lower fan frequencies), and, if we are worried about capacity, set Dip SW5-5 to “off” so that "capacity priority" mode overrides "low-noise" mode in certain conditions (https://planetaklimata.com.ua/instr/Mitsubishi_Electric/Mitsubishi_Electric_PUHY-P_YJM-A_PUHY-EP_YJM-A_Service_Manual_Eng.pdf).
Laser vibrometry of the X and Y TCS 2 mirrors does not show 48 Hz resonance
We were able to point the laser vibrometer at the struts of the large TCS mirrors in BSC2 using the illuminator and camera viewports of BSC2 (for the Y-arm mirror), and the illuminator viewport of BSC3 (for the X-arm mirror). Spectrograms in Figure 2 show that the vibrometer detected fan vibrations exciting the mirror in the 48 Hz region, but not a 48 Hz resonance.
Does not appear to be related to 48 Hz peak on ISI tables
We also found a peak that was very similar to the DARM 48 Hz peak on the Z-axis of GS13s in all LHO (not LLO) ST2 ISI signals. But injections that increased the 48 Hz motion of each of the tables individually by an order of magnitude did not appear in DARM. In addition, changing the isolation gain from 1 to 0.7 on the BS ST1 ISI isolatiobn and shutting down FF01 individually on all tables did not change the 48 Hz peak in DARM.
Opened the unit control panel today and the unit is already set to Low Noise SW-4-4 off and Capacity Priority SW 5-5 off. See attached Photo
Matthew Ball, Adrian Helmling-Cornell, Robert Schofield
The brightest spot in my inspection of HAM5/6 through available viewports was a spot on the septum window seen from 3 degrees (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=48965). Anamaria does not have a viewport at the 3 degree location and did not see a spot from the viewport at LLO that gives a 28 degree view of the septum window (https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=45980). We looked from 28 degrees at LHO to see if there was a difference between LLO and LHO. Figure 1 shows that a beam spot was visible, suggesting that there is a difference between the two sites.
TITLE: 06/30 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
Wind: 10mph Gusts, 8mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY: Hi is in Observe, noise in ther 38-60Hz band has increased since last night
TITLE: 06/30 Day Shift 15:00 – 08:00 (08:00-16:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Quiet lock until a HAM3, HAM4 DAC card failed, taking us out until Fil could come in to replace it. After that we had trouble locking, so we did an initial alignment, and entered Observing at 22:00.
LOG:
15:00 (08:00) Start of shift
16:51 (09:51) Sudden lockloss, large timing glitch took the IOP into safe mode
19:43 (12:43) DAC card replaced, watchdogs untripped, slider values readjusted, ready to relock
20:06 (13:06) Having trouble with ALSX, going into initial alignment
20:32 (13:32) Finished with initial alignment
22:00 (15:00) At NLN, entering Observing
23:00 (16:00) End of shift
J. Kissel, N. Lecoeuche, w/ Moral Support from D. Gustafson FRS Ticket 13170. We've recovered the IFO to observing from the sush34 DAC card failure this morning at 09:46:04 PDT (Jun 30 2019 16:46:04 UTC) LHO aLOG 50298. We hit the observe button at 22:00:06 UTC, for a total of 5.2 hours of observation time lost. More details of the blow-by-blow of the recovery to be commented below.
D. Barker, F. Clara, J. Kissel, N. Lecoeuche Recovery blow-by-blow: Picking up from after Fil gets in and Dave gracefully shuts down the h1sus34 front-end computer: ~18:50 UTC - Fil & Niko finish replacing DAC 2-5 (SR2 M2 & M3) 18-bit DAC card in h1sush34 IO chassis (LHO aLOG 50306). They mention they had to mess with h1susaux34 timing cable (they were zip-tied together), but thankfully this did not glitch the timing on h1susau34 -- it ran happily throughout the replacement. ~19:00 UTC - Jeff hits the physical restart button on the h1sush34 computer in the Mass Storage Room, beginning automatic restart of all front-end processes ~19:10 UTC - Input-Output Processing (IOP) model process automatically restarts, successfully calibrates the DAC cards (including the new one). All user model processes are then restarted (LHO aLOG 50304). 19:22 UTC - After successful start-up of all processes on h1sush34 computer, IRIG-B timing error signal reflects that the IO chassis needs to resynchronize to the world. Dave calls to confirm success of model restarts, and that it'll take ~10 minutes for the IRIG-B system to synchronize. Until then, we won't be able to reset the IOP Software Watchdog (SWWD) because inter-process communication (IPC, needed for SUS-IOP in sush34 to talk to SEI-IOPs in seih23 and seih45) error checking requires all IO chassis' IRIG-B timing to be happy. 19:27 UTC - Jeff and Niko begin to untrip local "user" watchdogs for MC2, PR2, and SR2, and suspension portions of SWWD that don't rely on IPC. Confirmed that all SUS's drive signals are indeed going out in to the real world, and SUS are damping. - Brought IMC_LOCK guardian to "DOWN" such that the IMC doesn't starting locking (and running alignment control) while the seismic systems (namely, HEPI) are still shut down and the DC position of HAM3 / MC2 are known to be wrong. - Unmanaged IMC-LOCK (normally managed by ISC_LOCK) so we're in control and ISC_LOCK is not fighting to lock the IMC prematurely. 19:30 UTC - h1sush34 IO chassis IRIG-B timing is now synchronized, Dave hits a reset on all front-end diagnostics to clear latched indications of IPC errors (LHO aLOG 50305) - Untripped local "user" watchdogs for ISIs and HEPIs in HAM3 and HAM4 - Untripped SWWD for ISI&HPIHAM3 and ISI&HPIHAM4 DACs on SEI23 and SEI34 computers, allowing drive signals to reach the platforms finally. - SEI Manager guardians for ISI&HPIHAM3 and ISI&HPIHAM4 recognized that all watchdogs have been cleared, and automatically began the process of bringing platforms to FULLY_ISOLATED, and succeeded without problem or intervention. - Used ndscope to trend and restore MC2 alignment (used alignment sliders alone as reference; did not use any stage's OSEMs). - Brought IMC_LOCK guardian to "LOCKED." This restored resonance and alignment of the IMC, without problem or intervention. - Ran "INIT" and "DOWN" on ISC_LOCK guardian so as to re-manage IMC_LOCK and get the IFO prepped for locking. - Used ndscope to trend and restore PR2 and SR2 alignment (again -- used alignment sliders alone as reference; did not use any stage's OSEMs). 19:40 UTC - Pressed our luck and tried to just start the lock acquisition sequence with the ISC_LOCK guardian as normal. - Having trouble with the X ARM ALS -- after Green ASC system turned on, WFS DOF1 Pitch would drastically drive ETMX into the weeds and break the lock. Niko suggest he's heard that something has been done to the order of operations in the Green ASC system during the LOCKING_ARMS_GREEN state recently (must be Sheila's work on Friday June 14th LHO aLOG 49930). This is even though green flashes in the X ARM show normalized power above 0.95, and the spot on the green camera looks nice. After 4 attempts at this, we decided it prudent to just proceed with initial alignment, for starters just to see if INITIAL_ALIGNMENT guardian does any better. 20:06UTC - Requested INITIAL_ALIGNMENT of ISC_LOCK and began to run through the laminated operator checklist. We discussed running through the new automatic system (see LHO aLOG 50190), but (a) didn't remember how to get it started (I only found that LHO aLOG 50190 now while writing this log), and (b) wanted to take things slowly given our problems with the X ARM and our knowledge of the computer crash involving SUS PR2 and SR2. - While discussing, Niko tries to squeak a little bit more out of the the X ARM's green flashes and gets from 0.95 to 1.0 (max for this "normalized" signal is 1.1). - X ARM INITIAL ALIGNMENT step works well. Yes, there's still a large control transient when green ASC turns on, but it doesn't go off in to the weeds. Either (a) the change in offload sequence in ISC_LOCK's version of locking the green are different and worse than INITIAL_ALIGNMENT's version, or (b) Niko's last-ditch tweaks were enough to push the DOF1 error signal on the "other side" of the WFS error signal phase curve and the signal is now valid (where before, if the supposition is correct, the alignment was "so bad" that the WFS error signal was outside the normal linear regime, where the sign on the loop flips.) - Jeff decides to go through the rest of INITIAL_ALIGNMENT "just to see" while Niko gets lunch. Good thing -- when PRC comes up, the error / control signals for PRC1 and PRC2 are large, and it takes a while to converge. SRC_ALIGN was not so bad. Rest of initial alignment went smoothly. 20:36 UTC - Resume normal lock acquisition with ISC_LOCK. LOCKING_ARMS_GREEN succeeds, but we loose lock mid-way through CHECKING_IR. XARM is Glitching enough to cause lock-loss. After reacquiring to that point, there's glitching again, but this time not bad enough break green lock, and we hold it through CHECK_IR. 20:43 UTC - After re-acquisition and slowly finding IR, we go through to PRMI_LOCKED. Locks up nicely without any need for adjustment, so we head to OFFLOAD_DRMI_ASC We acquire pretty quickly, BUT PRC1 and PRC2 ASC error signals are huge, so we loose lock once there. 20:57 - 21:10 UTC - Upon second attempt, we request DRMI_LOCKED_CHECK_ASC, and leave it there for 10 minutes as PRC1, PRC2, and some of SRC1 and SRC2 ASC loops are slowly brought in to convergence. We wait past when the state's convergence checker thinks things have converged. 21:19 - 21:50 UTC - Here, the rest of the acquisition sequence goes well, just very slowly. We paused at each request stated state that are good stopping points for the ASC system, PREP_ASC_FOR_FULL_IFO, ENGAGE_DC_VIOLINS, INCREASE_POWER, LOWNOISE_ESD_ETMX. We waited extra long at 2W input power on DC readout (i.e. ENGAGE_DC_VIOLINS) such that the full SOFT loops and ADS system could cook for a while slowly but surely (re)finding good spot positions before the major thermalization transient of power-up. 21:50 UTC - We hit NOMINAL_LOW_NOISE, and it takes us a few minutes to clear out a few SDF differences (LHO aLOG 50307). 20:00 UTC - OBSERVING.
Sub-comment about X-ARM Problems with LOCKING ARMS GREEN (at 19:40 UTC above) I attach time-series of ALS signals during our troubles with LOCKING_ARMS_GREEN mentioned above. X ARM transmitted power is in blue in the bottom left corner, and X ARM ALS ASC signals are in the upper right corner. The first attachment shows the first 3 attempts in LOCKING_ARMS_GREEN which show ALS WFS DOF1 yanking the ETM around, and causing lock loss. The second attachment shows a zoom in on one of those lock losses. The third attachment shows success when using INITIAL_ALIGNMENT.
Sub-comment about X-ARM ALS Glitching during CHECK_IR (at 20:36 UTC above) I attach time-series of the X-ARM ALS glitch that caused a lock loss while trying to find IR. Again, X ARM transmitted power is in blue in the bottom left corner, and X ARM ALS ASC signals are in the upper right corner. The first attachment shows the X-ARM glitching that caused the lock loss, The second attachment shows glitching that occurred during the next attempt, which appears to be roughly the same, but does not cause a further lock loss. It is *not* raining, and there is only ~10 mph or less winds, so it's not obvious if this was because of exposed fiber bundles along the arms (i.e. FRS Tickets 11113 and 12209).
Activity associated with FRS Ticket 13170.
Accepting the following sdf changes to go into Observing
Fil is in transit to the site. I'm determining which card is inaccessible.
I've opened FRS13170 to cover this fault.
Last DAC (slot 2-5) is missing.
In attachment, lower left is the card locations as shown by lspci. The last 18bit DAC (slot 2-5) is missing.
I've done with the h1sush34 card analysis, I've powered the computer down in preparation for card replacment.
Fil has replaced the 18bit DAC with a spare. The IOP model now sees all six cards, here is the autocal status:
h1sush34 ~ # dmesg|grep AUTOCAL
[ 63.863055] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 69.226743] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 75.018226] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds
[ 80.381909] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
[ 86.173432] h1iopsush34: DAC AUTOCAL SUCCESS in 5342 milliseconds
[ 91.537123] h1iopsush34: DAC AUTOCAL SUCCESS in 5343 milliseconds
IOP has gone into a negative IRIG-B excursion, so DAQ and IPC data will be bad for the next few minutes.
Fil, Jeff K, Niko, Dave:
IRIG-B has caught up. I've cleared all IPC and DAQ-CRC errors. Looks like h1sush34 is back online.
Many thanks to Fil for coming out to the site and replacing the card.
DAC card replaced in h1sus34 IO chassis S1103886.
Old: PCIe-18AO8-8-40.32M S/N 101208-08
New: PCIe-18AO8-8-40.32M S/N 101208-06
For the record, sush34's last DAC card (the 6th DAC card numbered 5 because counting starts at zero), in slot 2-5, is SR2's M2 and M3 coils. Attached are some relevant fast time-series of the AS port power (as measured by H1:ASC-AS_A_DC_NSUM_OUT_DQ), the requested drive from the lowest stage of each of the SUS controlled by this chassis -- MC2, PR2, and SR2 -- (as measured by H1:SUS-$(OPTIC)_M3_MASTER_OUT_$(OSEMDOF)_DQ), and the fast current monitors from the same coils (as measured by H1:SUS-$(OPTIC)_M3_FASTIMON_$(OSEMDOF)_OUT_DQ), which are thankfully on completely different computers. 16:45:46.94 UTC -- SR2 and PR2 Coil Driver MASTER_OUT_DQ starts requesting a repetitive signal, but not MC2 16:45:47.08 UTC -- MC2 starts requesting a HUGE signal, saturating MC2's M3 stage. 16:45:47.12 UTC -- AS port registers first badness 16:45:47.15 UTC -- Power at AS port has dropped to zero So, my supposition is that the lock loss occurred because of the MC2 blast -- not because of the SR2 DAC card failure. The MC2 blast occurred, because its DAC cards are in the same IO chassis (sush34) as SR2, and something went belly up much before the reported timing system glitch. Also -- I attach a trend of how long it took for the IRIG-B to come back after recovery of the front-end.
At 09:46:04 PDT h1iopsush34 suffered a large timing glitch. All models on this front end computer stopped running.
I've captured the logs are have restarted the models with /etc/statWorld.sh
Jeff K, Nico, Dave:
h1iopsush34 is not recovering. On first startWorld try it saw all the 18bit DACs but the model did not get going. I started the IOP by hand a second time and it only autocal'ed 5 of the expected 6 18bit DACs. This was confirmed by lspci. Before calling EE out, we will try a power cycle of the IO Chassis, I powered down the front end and Jeff K is power cycling the IO Chassis.
h1sush34 ~ # lspci -v |grep 3101
Subsystem: PLX Technology, Inc. Device 3101
Subsystem: PLX Technology, Inc. Device 3101
h1sush34 ~ # lspci -v |grep 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
Subsystem: PLX Technology, Inc. Device 3357
