Looking around I can ping the front end machines but cannot ping the boot machine, Corey is taking a look at h1boot1's console to see if it needs rebooting.
After having the issues with Guardian nodes coming up with Connection Errors, I sent a few "text pleas" to Jamie & Jenne.
Jamie was available and we eventually discovered this is NOT a Guardian issue: We Cannot Change any values for many/any systems/frontends!
Jamie was wondering if this could be an EPICS Gateway issue, but he then mentioned he's not sure of whether there was a gateway for LHO.
Jenne later chimed in (she was aligning K1) and said this sounds serious and could not help.
Have left a voicemail with Dave. Richard should be in in a few hours.
H1 is dead until this can be addressed.
(On that note, I'm going to make breakfast/lunch....well, I'll be here in case Dave happens to call within the next 30min first.)
[Edit: Originally thought this was a Guardian Issue, but we later found out it was due to the h1boot1 computer.]
New behavior to me tonight occurred when attempting an INITIAL ALIGNMENT. Everything was fine up until INIT ALIGN wanted to run the MICH BRIGHT step. This step runs a DOWN command for ALIGN_IFO, but then in an early step of the DOWN code it repeatedly gets an connection error (for PRC1_P). After this, ALIGN_IFO goes into a FAULT state & a continuous loop. Below is the log:
2019-09-11_09:24:03.821637Z ALIGN_IFO REQUEST: MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822289Z ALIGN_IFO calculating path: DOWN->MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822719Z ALIGN_IFO new target: PREP_FOR_MICH
2019-09-11_09:24:06.469708Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:24:06.824697Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:24:06.830620Z ALIGN_IFO REQUEST: DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO calculating path: DOWN->DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO new target: DOWN
2019-09-11_09:24:08.924272Z ALIGN_IFO [DOWN.main] USERMSG 0: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel ASC-PRC1_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:24:08.949899Z ALIGN_IFO EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:26:16.909062Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:17.160830Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:29.469452Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:29.721182Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:42.038308Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:42.290377Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
....
I then gave up on ALIGN_IFO (I took INIT ALIGN to DOWN a while ago), and moved back to ISC_LOCK and ran a DOWN, but now it looks like it's having the same issue. For isc lock it's listing a connection error...with a different channel & can't perform a down and is stuck in a loop.
Not sure what I can do at this point.......will keep investigating Guardian Land. :-/
Marking this DOWN TIME as CORRECTIVE MAINTENANCE since I can no longer run an alignment...or even return to locking apparently.
Here is the error I get with ISC LOCK when I try to run a DOWN: (and this was after taking OP to STOP, letting it complete, and then taking OP to EXEC)
H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:48.097459Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:50.274110Z ISC_LOCK [DOWN.main] USERMSG 3: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel SUS-ETMX_M0_LOCK_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:38:50.277456Z ISC_LOCK EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:38:50.307871Z ISC_LOCK CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:38:50.618404Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:50.870580Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.122586Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.374299Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:51.626122Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:51.888774Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.145125Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_P => OFF: INPUT, OFFSET
2019-09-11_09:38:52.396948Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CSOFT_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:52.401265Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 256
2019-09-11_09:38:52.401855Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 80
2019-09-11_09:38:52.402348Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => OFF: FM1, FM2, FM3, FM4, FM5, FM6, FM7, FM8, FM9, FM10
2019-09-11_09:38:52.402773Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW1 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS_SW2 => 0
2019-09-11_09:38:52.403250Z ISC_LOCK [DOWN.main] ezca: H1:LSC-REFLBIAS => ON: FM9, FM3
2019-09-11_09:38:54.839541Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.091397Z ISC_LOCK [DOWN.main] ezca: H1:ASC-DHARD_Y => OFF: INPUT, OFFSET
2019-09-11_09:38:55.343344Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_P => OFF: INPUT, OFFSET
2019-09-11_09:38:55.595399Z ISC_LOCK [DOWN.main] ezca: H1:ASC-CHARD_Y => OFF: INPUT, OFFSET
Sent out texts of help to Jenne & Jamie.
Jamie luckily was in Europe so it's morning time for him. He's helping me look through the issues currently.
Just for future reference, this was an issue with h1boot1, not guardian.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=51883
H1 was locked for just under 80min when it had a lockloss. Only things worth noting are about 5-7 glitchy stretch in the 17min before the lockloss.
First Locking attempt (with BS & PRM tweaks) failed at RESONANCE.
At the start of the shift with H1 in Observing I noticed that the H1 Glitches (DMT Omega) page on our Control Room wall tv is not showing anything for the last 12hrs (looks like the same situation for Omicron, too). This led me to also finding that the H1 Network Summary Page was also not showing anything for most of the last 12-hrs.
TITLE: 09/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY_NO_BRSX
Wind: 7mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.12 μm/s
QUICK SUMMARY:
TITLE: 09/10 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Sudden lockloss with SR2 saturation. Went into initial alignment after not being able to lock PRMI, but MICH_BRIGHT didn’t seem to work that well so I did MICH_DARK_LOCKED alignment and redid SRC_ALIGN. After that I lost lock from CARM_TO_TR, and then managed to reach NLN.
LOG:
23:00 (16:00) Start of shift
23:28 (16:28) Kyle back from MX
01:00 (18:00) Ethan to Optics Lab
01:19 (18:19) Ethan out of Optics Lab
04:33 (21:33) Lockloss with no warning. Alarm handler said SR2 saturated.
04:51 (21:51) Trouble locking PRMI, going to initial alignment
05:07 (22:07) Initial alignment complete, re-locking (issues with mich bright)
05:48 (22:48) Lockloss from CARM_TO_TR
06:47 (23:47) At NLN, going into Observing
07:00 (00:00) End of shift
The ALS shutters are controlled by two momentary EPICS channels, one to OPEN and one to CLOSE the shutter. Occassionally a guardian request to open/close one of these shutters is unsuccessful.
The requirement:
The status of the shutters to be added to SDF to show as a diff if they are not in their nominal state (CLOSED).
The problem:
Because the control channels are momentaries, even though these are automatically in the SDF they do not show actually the status of the shutters. Momentary records change state only for a fraction of a second and then return to their stand-by value of zero. The record which shows the shutter state is a monitor channel, and therefore not in the SDF.
I had two coding options:
1. change the existing slow controls systems for h1sysecat[x,y]1plc1sdf to make the monitor channels pseudo set-point channels and therefore in the SDF
2. create a whole new slow controls sdf just for ALS shutters
We decided on option 1.
Details:
I changed my code which generates the slow controls SDF monitor files to add the four shutter state channels for PLC1 at EX and EY. After restarting h1sysecat[x,y]1plc1sdf I ACCEPT+MON'ed the new channels.
The SDF system shows channel names, and unfortunately there is a naming issue with the EY system whereby the Green-beam and the Fiber-beam channels are reversed compared with EX. This is compounded by the fact that the names of the systems assume the channel swap has not happened, and are therefore wrong. To help operators which this possible source of confusion, I have modified the SYS_CUST_SHUTTER_SUMMARY MEDM to show any SDF differences as RED blocks. I have also removed the Green/Fiber naming confusion by hardcoding the EY names (they are colored BLUE to show I have done this).
New and Old MEDM images attached.
Ops Shift Transition: 09/10/2019, Eve Shift 23:00 – 07:00 (16:00-00:00) - UTC (PT)
State of H1: Locking
Intent Bit: Commissioning
Weather: 0-15 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.1 um/s
Outgoing Operator: Ed
Quick Summary: Post-maintenance day recovery still ongoing. In WINDY_NOBRSX until BRSX calms down.
Fil and I went to EX to look inside the BRSX enclosure in prep for the work next month. While we were there I touched up the centering of the BRS. While we had the box open, we must have disturbed the cable into the interface box inside, because when we got back to the corner station the BRS wasn't damping down. I went back down to look at it and it took a little to realize there is not retention on the ethercat cable into the interface box, so it's really easy to knock loose. It immediately started damping down when I pushed the ethernet cable into the interface chassis.
The BRS is still calming down, so we shouldn't use it for a little while yet. The operator should watch the the BRS health ndscope and wait for the BRSX RY OUT to look more like the BRSY RX OUT signal, i.e. the blue trace on the middle timeseries looks like the yellow(?) trace on that same plot.
Opened FRS Ticket 13570 to track that H1 BRS EX ethernet connector at interface box needs to be repaired.
Is it safe to transition from WINDY NO BRSX back to regular WINDY while in NOMINAL LOW NOISE? (Or do we have to wait until we are not locked to make this transition?)
The transition is handled by the SEI_CONF guardian, and is very similar to making the transition to the earthquake state, and does not affect the observation bit. If anything, putting the BRSX back into the loop should make the IFO quieter, assuming the BRS has calmed down.
Looking at minute trends of the sensor correction outputs for EX and EY for the last day, we could have transitioned back about 5pm local last night, first plot.
This is inspite of the fact that the BRSX was still not totally settled, the DC position is still slowly settling down, second plot.
And, the BRSX RY out still had something like a 5000 nrad offset (third image), it was moving slowly enough to not affect the sensor correction signal.
Ah, My Bad.
Could this be the source of my woes with locking after the h1boot1 recovery?
The LVEA was swept at 1430 local time. Kyle was still in there but promised to turn the lights off when he left.
And he informed the operator that he did so upon departure. The lights are indeed OFF.
Kara Sheila Fil Daniel
Continuing replacing cabling for the CLF AOM1, old alog 51537.
We replaced the feedthrough and the helix cable to AOM1 of teh CLF. First, we saw the RF power monitor coming back to the old value of 33.5dBm. After a minute or so it went down to 5dBm. Most likely the RF amp blew. Unclear why, but after replacing it we carefully measured the RF power delivered to the AOM. We needed an additional 3dB attenuation to get back to the previous value of 34dBm, see alog 40071. Maybe the cable was defective and we got more power than the AOM could handle? Assuming the AOM potententially has a problem, we swapped it with the spare and reconnected. A lot of alignment tweaking was required to get back to the previous difraction power. All seems to be working again.
We also installed a 1064nm wavelength filter, Thorlabs FLH1064-8, in front of the CLF launch PD. This PD was very sensitive to the table lights. The filter transmission at 1064 should be >90%.
CLF power level trend
We made the following measurements of the power on ISCT6: After 1st periscope mirror: 3.32mW Out of vacuum: 3.32mW After 2nd periscope mirror: 3.25mW Input to cube: 3.24mW Output of Cube: 3.16mW wrong polarization: .09mW Front of Detector: 3.11mW We made the follow power measurements on the Squeezer table: OPO REFL: 3.06mW and 1.28v Output monitor: 32.6dBm Drive into AOM: 33.7dBm
Yesterday, I tested, if the removed AOM still looks like a 50Ω load to make sure we didn't fry the input circuit. Indeed, it looked like a 150-250 MHz bandpass with proper 50Ω termination. However, I noticed that the SMA bulkhead thread of the AOM is tad bit too short. Out of 5 male SMA I tired 4 bottomed out, which potentially prevents a proper electric connection. This probably explains the flaky behaviour we observed.
Now that we are back to Observing, figured I'd post locking notes from the night.
Summary:
Locking Notes:
J. Kissel
Regarding Corey's comment:
NOTE: getting "IMC WFS not centered" messages for IMC LOCK. Observe this in all the space it takes up in the ISC LOCK log.
This is a known issue/problem that is not resolvable without a hardware change: check out my aLOG from tail end of July: LHO aLOG 50920, and Daniel's subsequent reference to 5109.
To quote my aLOG: "This message appears, typically, after PSL incursions (rare), site power outages (rare), or computer failures (rare) when the IMC suspensions's or PSL Periscope PT alignments get lost. Because these events are rare, instituional memory loss causes confusion for folks when they see that error message and wonder if action is needed. However, once these alignments are roughly recovered (via reseting sliders on suspensions), the WFS, again typically, eventually recover their centering all on their own as the RF loops converge [there are no "DC centering" loops on IMC WFS, as there are for many of the the WFS systems]."
However, last night, none of these things happen, and the IM alignment is all down stream of the IMC WFS. So I'm equally confused as to why the WFS got outside of their range. Worth trending.
But in short -- while these warnings are annoying -- they are reflecting of a real issue, so don't ignore them.
CAL CS differences are due to poor rounding of the installation of Calibration Model Reference Values art Calibration Line frequencies. Will work on resolving this later today.
Today we did have the problems with TR_CARM that Corey and Niko descirbed last night, and we think we have addressed the problem: 51861
We didn't have any of the locklosses from ENGAGE_SOFT_LOOPS today. I did look at one of them from last night, and it does seem that we still have low gain in SRC2 (P+Y) and INP1 P so that the loops aren't holding their error signals around 0 in these steps. Speeding these up a bit might help us out here. Keita and I started to work on speeding up some of the ASC for similar reasons last Tuesday 51715, but we didn't get a chance to try increasing the gain in these very slow loops.
Since we went about 3hrs with constant locklosses for both Niko & I (along with two Initial Alignments by Niko), I decided to back out the IM change from last night which had a data point of the one lock we had since last night in the hopes of returning to the input pointing we had been running will help with locking.
IM Alignments Restored To Sunday Night Values:
IM1, IM2, IM3 restored to where they were before the last lock. (settings restored from SDF list here).
Initial Alignment:
With the new Input Pointing, I needed to run an Initial Alignment. Took a little longer than usual, due to....
Operator (me) Errors:
Error #1) IM Slider Mistake: Noticed Xarm in IR was not locking for INPUT_ALIGN. Couldn't determine the issue, buuuuut.....
--I had a commissioner monitoring remotely (from Japan!). Jenne let me know that one of the IMs was way off! I errantly entered IM2's Yaw slider (I added an extra "0" for it's value). After this I continued with alignment
Error #2) Green Y-arm Dead: Green Y had no light flashing! Probably due to me taking multiple Nodes to DOWN
--Once again Jenne helped me here, and she noticed my ITMy was waaaay off. I had noticed ITMy was labled as MISALIGNED, so I tried taking it to ALIGNED, but it wouldn't work. So I did a MANUAL to ALIGNED & then back to AUTO, but this misses steps. So Jenne had me take it to MISALIGNED, and then to ALIGNED, and this finally gave us light flashing in the Y-arm.
Error #3) Possible Sticky Slider: During alignment, SRC was continually stuck at around PREP_FOR_SRY
--Jenne tracked this down to an optic not in state Guardian liked. I believe this was ITMx & the issue was a "sticky slider". Will make separate alog for this.
Current Status: Just had my first lock attempt after restoring IMs and doing an alignment. Had lockloss at ENGAGE SOFT LOOPS. This was fairly early on this step & did not see any obvious reasons for the lockloss.
Additional NOTE: IMC guardian node has flashes of a notification of "IMC WFS not centered". This messages flashes quick, has been occurring fairly frequently, and is new to me.
ADDENDUM: Did NOT Revert IMs to Sunday night values!
I thought I had reverted the IMs back, but when I finally made it back to OBSERVING I noticed that there were no SDF diffs for SUSIM for me to ACCEPT!
I think the problem for my brain was that after I made the error in restoring IM2's Yaw slider (adding an extra "0" which Jenne caught), my mind had me sticking with all of the new IM slider values (vs the old slider values I wanted to revert to)!!! So I ran an Initial Alignment thinking the input pointing was referted, but it really wasn't. Grrrrr! My Mistake!!
Since it has taken roughly ~5hrs of locking to get us back to OBSERVING (+ 2hrs of my aligning), I'm still thinking we want to back out the IM pointing change from Sunday night since locking has been tough---IM values can be found here.
The IM1-3 slider values were reverted today (2019-09-10). See LHO aLOG 51858.
During shift Hand-off of an H1 in NOMINAL LOW NOISE, I accepted SDF diffs which came up. We were at a nice range of 118Mpc.
SDF Diffs
The Diffs are attached and were:
1) SUSIM for IM1, IM2, & IM3
These are known and related to a change Cheryl made for relocking. She will make an alog about this.
2) ASC: RPC gains
Have a few small gain changes for ASC-RPC_DHARD & CHARD pit/yaw. These looked new to me (what's "RPC"?) & Cheryl was not familiar with them as well. I did a quick scan of ASC medms, but I could not track them down. Nevertheless, I went ahead and accepted them.
Note for OBSERVATORY MODE: The last 2.5-3hrs of downtime was listed as OBSERVING (vs. LOCK ACQUISITION or EARTHQUAKE). I switched it very briefly to LOCK ACQUISITION so we can mark that we were in fact down due to a lockloss.
The IM1-3 slider values were reverted today. See LHO aLOG 51858.
The DAQ and the slow controls systems EPICS IOCs continue to run, only the front end systems appear frozen, which points to a h1boot1 problem.
Corey reports h1boot1 console has errors, and is unresponsive to keyboard input. We are rebooting h1boot1 via the front panel RESET button.