In a continued effort to understand nds1 restarts and crashes, we added the epics reflection of basic system metrics to h1nds1. This is the same as we had done for h1nds0 see log 47569 (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=47569). None of these are being trended in the frame as of now.
Robert starting PEM injections.
WP8119 External EDCU install
Dave, Jonathan:
I added h1edc to the DAQ (turned of EDCU on dataconcentrator, replaced EDCU ini in master by H1EDC.ini). Initially it started ok, but kept giving CRC errors at a rate of a couple every 3 or 4 seconds. For now we have backed this out of the DAQ, and reverted to the internal EDCU. h1edc continues to run on h1susauxh34.
WP8118 FE-DAQ Netgear switch upgrade to Brocade
Jonathan, Dave:
I installed the new Brocade switch sw-ey-h1daq in EY. To test it can communicate with the Allied Telesis media converter at MY this was a quick install. Surprisingly when the EY FECs went away, all FECs communicating to the second 10GE card on h1dc0 dropped out (EX and corner ISC). When the new switch was connected, everything came back correctly with no action needed.
For now I've kept the Brocade in place, temporarily sitting on top of the netgear it replaces. On the next TOO (or next TM) I'll rack it up correctly.
BRS BLRMS change to h1pemey
Jim, Dave:
Jim changed h1pemey for BRS BLRMS channel addition. I restarted the model and DAQ.
WP8075 DMT test machine upgrade
Greg, Dan, Dave:
The new DMT test machine was installed today. It is temporarily called h1dmt4, will become h1dmt3. I configured the dmt switch to provide the broadcast lan for this machine.
WP8128 DMT solaris upgrade
Dan:
h1dmtqfs0 was upgraded (OS and QFS) and its SATABOY was rebooted.
WP8115 cdsssh OS upgrade
Carlos:
cdsssh was upgraded from Deb8 to Deb9 first thing this morning.
CER workstation upgrade
Carlos:
The CDS workstation in the CER was upgraded to a portable Zotac, to replace the broken iMac.
when the models on h1lsc0 were restarted this morning at 16:25UTC, the CW hardware injection excitations stopped on h1calinj. We restarted the h1calinj model at 23:35UTC, this did not change anything.
I have restarted the psinject process on h1hwinj1 as user hinj. I'm contacting the injection team to verify this is the correct way to do this manually (we (cds) still need to get this automated).Here is the text from the startup:
[hinj@h1hwinj1 ~]$ /home/hinj/Details/bin/x_start_psinject david.barker "start after lsc restart stopped injections"
Entered psinject_common.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourcing /home/hinj/Details/psinject_config.tcl
Entering psinject_config.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourced psinject_config.tcl
Note: H1 psinject WAS running as process 21196, but has died
Starting H1 injection into H1:CAL-INJ_CW_EXC using RELEASE -> preO3_H1
Injection will start at t=1237074185 (60 seconds from now)...
Injection starting now
A quick verification that injections are running again (24-hour minute trend showing that summed injection amplitudes are where they should be).
No alogs from today yet state the situation, so here is the current status - we're still recovering from Tuesday Maint currently locked in Low Noise, however we have a bit of a mess to work through.
The LSC computer was glitched this morning at 9am when EE rack fan isolation work was taking place. CALINJ is now red out of nominal (OFF). Dave brought the LSC back up (details below eventually), then numerous activities took place as per the morning plan:
- PCAL maint checks at both EY and EX
- Fan isolation in corner LSC and ASC racks, EX and EY ISC racks
- IMC board work by Daniel
- Charge measurements of ETMX
- DAQ/EDCU work by Dave
- 1Hz comb troubleshooting - no alogs to report anything on this yet
- Minor PMC and REFCAV alignment tuneup
- PSL REFCAV EOM driver cable swap
However, once back up at ~3pm local, major noise was present and we saw numerous SDF settings changes in the OBSERVE file (which was clean during the locking last night) mostly related to the OMC (from LSC model). These were probably settings not saved correctly to the SAFE SDF file and then hung around through the locking to low noise. Sheila has just attempted to walk those back by hand but the lock blew. Back to locking now.
PEM was supposed to have some time today, however the maintenance/relocking went long, so they will attempt to work for a few more hours while yielding some hour long segments for OBS MODE starting later in the evening. Slow thermal TCS tuning may also resume while in OBS MODE overnight, similar to last night.
Yesterday we updated many OBSERVE.snaps in order to get to oberseving, but we didn't have a chance to do a similar cleaning up of safe.snaps before this morning's glitch which took down some computers as Betsy says.
Jenne and I updated a handful of things that we fixed by hand in the safe.snaps, but we need to take a bit of time in the "DOWN" state after a lockloss to update the safe.snaps to prevent these problems in the future.
We'll mobilize to get some help on this in the next day or so. No more unexpected crashes tho, right!? Famous last words.
Looking at reported 1Hz, alog 47450. The ±430, ±48V, and ±18V ESD power supply grounds were tied to rack ground in the power rack.
- with Dan Moraru and Dave Barker:
* The DMT computers have been patched.
* The GDS fileserver is now running Solaris 11.3 and the failed drive on the OS filesystem has been replaced and mirrored.
* The QFS filesystem is now at version 6.1.50
* The GDS calibration software has been updated to gstlal-calibration-1.2.9
This completes WPs 8128 and 8130.
Clocks back on:
EY: Mar. 18 15:33
EX: Mar. 19 16:22
TITLE: 03/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1:
INCOMING OPERATOR: Travis
SHIFT SUMMARY:
LOG:
15:15 Richard, Fil and Ed to CER
15:15 Karen, Vanessa to Ends
15:30 Kyle to EY
15:30 Daniel to SQZ
15:45 Niko, Rick to EY
15:45 Chris to EY
17:45 Rick, Niko to EX
17:45 Fil, Ed to EY
17:45 Phillipe to EY
18:00 Dave to EY for DAQ switch
18:45 Robert to PSL
WP 8069
In preparation of installing a modified illuminator chassis at EY, new cables were pulled from SUS-R1 to ISCTEY. Modified unit will allow to remotely turn on/off table lights and the TCS Sled LED. Fan status will also be monitored.
F. Clara, E. Merilh
The feed forward from the PSL VCO to the SQZ VCO has been setup. The feed forward is taken from the PSL VCO control signal and injected into the excitation input of the SQZ VCO. There is a factor of 20 gain difference between the tune input and the excitation input. However due to the Pomona box which implements a pole using a 15.8K resistor and a 10u capacitor and due to the difference amplifier at the excitation input the actual gain is 20/0.7793 = 25.66.
The feed forward reduced the frequency difference between the 2 VCOs by at a factor 10-20, when the squeezer is on PREP_IFO (no LO loop).
With the squeezer locked, we can compare a previous lock with a new lock. The improvement is around a factor of 15.
WP 8131
Fans for the follwing units are now being powered off external power supplies:
1. ISCEX IO Chassis
2. ISCEY IO Chassis
3. LSC IO Chassis
4. ASC IO Chassis
Part of the noise hunting seen in DARM, alog 47575.
F. Clara, R. McCarthy
The timing of h1lsc0 was glitched during its fan power upgrade. I checked the IO Chassis was seen by the computer and then restarted all the models with /etc/startWorld.sh.
Despite h1lsc0 being the timing master for h1oaf1, it did not appear that h1oaf1 needed a restart, (and so its models were not restarted).
Opened/Closed FRS Fault ticket 12567
The servo board for the IMC (S/N S1102626) was replaced with the modified spare (S/N S1102627). This implements E1800338-v3 and ECR E1900059.
When the spare is not a spare... The IMC wouldn't lock. I opened the original IMC servo and it has the fast option installed, whereas the spare has not; see alog 29250. The modifications were applied to the original board. Then, the original chassis was put back in.
Also, noticed that some of the modifications were already applied, see alog 31138. However, this was never propagated to the e-traveler.
Here is the comparison with the previous version measured in alog 47485.
There is a significant improvement in IMC_I. The error signal now looks like it is limited by converter noise below 2 kHz. Strangely, there is a set of peaks around 2.5 kHz that are not in IMC-L or IMC-F. There is some loss of coherence between IMC-F and IMC-L between 1 and 10 Hz. This seems again converter noise in IMC-F.
Full bandwidth spectrum.
h1pemcs DAC channels addition
Anamaria:
A new h1pemcs model was installed, additional channels of the 18bit DAC are being driven by PEM. No DAQ restart was needed.
DAQ Restart
Dave:
The DAQ was restarted twice at 12:09 and 12:35. The first for pemey and h1edc addition, the second for h1edc removal.
It was noticed that the system time on h1ldasgw[0,1] were off, Dan fixed this by syncing to the CDS time servers.
Reboot h1ldasgw0
Dave, Dan:
Dan noticed that h1ldasgw0 was spewing errors on the login console. I stopped h1fw0 daqd and umount /frame-0, Dan rebooted h1ldasgw0. All came back well.