The DMT production computers at LHO have been patched and rebooted, bringing them to Scientific Linux 7.7, and gstlal-calibration-1.2.11 has been installed. All the DMT monitors should be working again. This complete WP #8435.
Following the crash of h1susex yesterday one of the 18bit DACs was lost. Presumably this was the cause of the crash.
The missing 18bit DAC is in slot 1-7 (first dac slot after the two adc slots). Fil has replaced the unit and we are rebooting.
We have temporarily swapped the mirror which combines the HWS path and the ALS path. This is usually a polarizing beam splitter, and we have switched it with a HR 2" optic. This clobbers the HWS path (sorry!)
We wanted to check that some ROC on the beam combining optic is not the cause of the mode mismatch between the green beam and the Y arm. We've just opened gate valves and have the Y arm back now. A look at the Y arm while it is swinging after an initial alignment suggests this did not solve the problem (see attached screenshot). The higher order modes are still ~30% of the fundamental.
Some extra details:
I took the PBS out of the mount, and put it in the box of the new HR mirror and left it on the table. The mirror I installed is E1000652, an ATF 2" HR mirror for 532 at 45 degrees AOI. The new optic is thicker than the PBS, but the mount is such that the front face is at the same position relative to the beam, I removed an extra retaining ring from the mount to compensate for the extra thickness. I tweaked the alignment of the mirror until I saw light on the QPDs, and I checked that the PZTs were not railing.
I switched the optic back, so now everything is back to normal in the ISCTEY layout.
While I was out there I checked the Hartmann path and noted it was clipping on HWS-M5. I then stupidly tried to align the Hartmann path, tweaking HWS-S2, HWS-M5 and HWS-M6 while looking at the HWS live images (comparing to a time with the ring heater on), this is very slow to update. In the end I think I made everything worse so next week I will need TJ's help to fix it up to the beautiful alignment they had before, sorry TJ and Aidan :(
I created a guide to aligning the ETMY HWS to help recover the signal.
Because we have the infrastructure in place, I wanted to try using the CPS differential control on the corner station BSC ISIs. To do this I set the gains in the CPS_DIFF_FILT_MTRX screen so that ITMX X follows BS X and ITMY Y is tied to BS Y. I can then use the LSC-CPSFF channel as a measure of the differential motion seen between the CPS of these chambers. Seems to work, below 100 mhz.
Attached screenshot, the red is with the differential control off, green is with the differential control engaged. Below 100 mhz green is about a factor of 2 below red, meaning that MICH displacement witnessed by the CPS is smaller with the differential control than without, so with the differential CPS control engaged, the chamber motion is more common. Above 100mhz the motion is dominated by the microseism, which the CPS see, but the T240s would be easier to understand. It would be best to test this with a cavity, but they just opened the arms.
Jim, do you filter the CPS diff signals before feeding back, like for ham23 ?
Arnaud - yes, Jim has some 50mHz-focused bandpasses on for these.
As we try to lock for the first time since the vent, and since we're having some trouble (perhaps because we don't have end station sensor correction on??), I've turned this system on and off, and so far none of the IFO cavities seem to care, in that they still acquire nicely, but the transmissions are wobbly both with and without this system on. I haven't done an actual spectral comparison yet though.
I was able to fix the leak on the 12" CF rotatable flange on the YBM turbo this morning. All CFF joints on this unit are now leak tight (< 1.9 x 10-9 Torr*L/sec LD background/baseline).
Gerardo M., Tyler G., Kyle R.
Opened GV5 and GV7. Gerardo M. had decoupled the HAM5/HAM6 AIP pump carts earlier and I had valved-in HAM6's IP, isolated,the HAM6 turbo, spun it down and de-energized it but the connecting hose and controll cable still need to be removed. This will require climbing up on HAM6 again tomorrow. We intend to clean up our various work areas over the next couple of days.
Nice work, team!
Both BRS are still in the middle of an upgrade, we are waiting for some thermocouples to arrive on Friday. As it stands, both BRS look like they are running, but when I logged onto the beckhoffs, one seemed frozen and one was running old code, so I don't believe either is actually in a good state.
For now, we should avoid using either one, so SEI_CONF should stay in TEST_SWARM when appropriate, or SC_OFF_NOBRSXY if there is work in the LVEA or a large earthquake. LARGE_EQ should also be safe.
We repeated the Gouy phase measurement from yesterday (alog 52639) with the SR3 ring heater set to 0W (off).
We measured the (one-way) SRC Gouy phase to be 25.25deg +-0.06deg (statistical) +-1.8deg (systematic).
This is almost 4deg less than yesterday in alog 52639, i.e. very roughly, the ring heater adds about 1deg/W of one-way Gouy phase.
Fitting details:
- We used the s3 quadrant for finding the 00 peaks. It has the least contamination from higher order modes.
- We used 3 data sets to fit the 10 peaks: SUM (blue), SEG2 (red) and SEG1 (black).
- We used all 13 fringes visible in the raw data set.
- Note that compared to yesterday's measurement we have more contamination from the 00 modes, leading to a systematic shift to lower Gouy phase if we use the SUM and SEG2 data set.
- We thus reported the result from the SEG1 data set (black) for the reported value of the Gouy phase.
- But we used the maximum difference between between the 3 data sets as reported systematic error: 1.8deg.
- The statistical error is comparatively negligible: 0.063deg.
Plots:
- Plot 1: SRC scan across 13 fringes. Blue: AS_C_SUM, Red: AS_C_SEG2, Black: AS_C_SEG1 (all derived from IOP channels).
Magenta: round trip Gouy phase fit from the blue and black results. The black result is used for the reported value.
- Plot 2: Raw data strech of the 13 fringes.
The method is described here: alogs 52639 and 52504, klogs 9241 and 9246.
The code and raw data is found in /ligo/home/controls/sballmer/20191023.
The 1deg/W value for the SR3 heater is consistent with the previous Gouy phase analysis of the SRC.
Here are some tables of the SRC gouy phase from the Finesse model. I've added in what we think is happening with thermal lensing and ITM RoCs from the TCS simulation page too. I can get up to ~25 degrees too with the SR2 shift and 4W of SR3. Script to generate these numbers is attached.
Single pass gouy [deg]
Design Finesse file
----------------------
SRX x : 19.2
SRX y : 16.9
SRY x : 19.2
SRY y : 16.9
Single pass gouy [deg]
With ITM lensing
----------------------
SRX x : 18.8
SRX y : 16.5
SRY x : 18.2
SRY y : 15.8
Single pass gouy [deg]
With ITM lensing
+ 4W SR3
----------------------
SRX x : 22.4
SRX y : 20.4
SRY x : 21.8
SRY y : 19.8
Single pass gouy [deg]
With ITM lensing
+ 4W SR3
----------------------
SRX x : 23.1
SRX y : 21.1
SRY x : 22.5
SRY y : 20.6
Single pass gouy [deg]
With ITM lensing
+ 4W SR3
+ 5mm SR2 shift
----------------------
SRX x : 26.1
SRX y : 24.4
SRY x : 25.7
SRY y : 23.9
To get up to something like 29 degrees I have to start doubling the distance SR2 is shifted or doubling the SR3 heater power
Single pass gouy [deg]
With ITM lensing
+ 4W SR3
+ 10mm SR2 shift
----------------------
SRX x : 29.0
SRX y : 27.4
SRY x : 28.5
SRY y : 26.9
Single pass gouy [deg]
With ITM lensing
+ 8W SR3
+ 5mm SR2 shift
----------------------
SRX x : 29.6
SRX y : 28.0
SRY x : 29.1
SRY y : 27.5
WP 8388/Famis 12359 Corey/Hugh
Took SEI to HEPI Offline, spun down the pumps & checked and charged ar'd the accumulators: Topped up 3 of the 4 CS pump accumulator 1s. One LVEA accumulator needed charging after leaking down when trying to get the pressure reading. All others in good shape. HEPI pumps and SEI platforms all returned to nominal without issue.
Jenne, Jim, Dave:
h1seiproc was restarted, followed by a DAQ restart to add new slow channels for an IPC receiver and expanded matrix.
h1broadcast0 daqd did not restart, looks like monit lost connection to it. I manually started daqd via the monit web page.
After card replacement and power up I was able to see the full compliment of cards. However the models did not start correctly and the IOP had a negative IRIGB time. After this cleared the IOP still had a DAC error. I killed all models and started the IOP. This time it failed with the same dmesg as was seen Tuesday morning.
I am now power cycling h1susex (leaving the IO Chassis alone).
After the second power cycle of h1susex (this time leaving the IO Chassis alone) I am again able to see the full compliment of cards, and again the IOP went into a negative IRIGB excursion, but the user models this time were not in a failed state. After 10 minutes the timing came good again, and everything greened up.
We are uncertain if the particular 18bit DAC we swapped was the cause of this failure, given the repeat of the dmesg when the IOP startup failed. Note that the dmesg error references an 18bit dac card in the line
RIP [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
Here is the IOP failure dmesg for comparison with the one yesterday
638.430574] h1iopsusex: DAC I/O address=0xffffffff 0x0
[ 638.430578] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 638.445131] IP: [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[ 638.445425] PGD 81118b067 PUD 83fa9a067 PMD 0
[ 638.445715] Oops: 0000 [#1] SMP
[ 638.445998] CPU 4
[ 638.446004] Modules linked in: h1iopsusex(+) dis_irm(P) dis_ix_ntb(P) dis_ix_dma(P) dis_intel_dma(P) dis_kosif open_mx gpstime mbuf [last unloaded: h1iopsusex]
[ 638.447142]
[ 638.447417] Pid: 7840, comm: insmod Tainted: P 3.0.8 #2 Supermicro Super Server/X10SRi-F
[ 638.447982] RIP: 0010:[<ffffffffa440f772>] [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[ 638.448543] RSP: 0018:ffff88085cf5ddc8 EFLAGS: 00010282
[ 638.448825] RAX: 000000000000003e RBX: 0000000000000000 RCX: 0000000000000b43
[ 638.449111] RDX: 0000000000026b49 RSI: ffffffff81797e8c RDI: 000000000000000a
[ 638.449397] RBP: ffff88085cf5de08 R08: 0000000000000002 R09: 000000000000000a
[ 638.449683] R10: 0000000000000005 R11: 0000000000000000 R12: ffff88087bbfc000
[ 638.449970] R13: 0000000000000001 R14: ffffffffa4428320 R15: 0000000000000001
[ 638.450255] FS: 00007fa3198d8700(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000
[ 638.450810] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 638.451093] CR2: 0000000000000000 CR3: 000000083f965000 CR4: 00000000001406e0
[ 638.451379] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 638.451665] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 638.451951] Process insmod (pid: 7840, threadinfo ffff88085cf5c000, task ffff88087bdaef00)
[ 638.452505] Stack:
[ 638.452779] 00000000a4424640 00000000a4428320 0000000000000000 ffffffffa4428320
[ 638.453344] 0000000000000000 0000000000000001 0000000000000002 0000000000000000
[ 638.453909] ffff88085cf5de58 ffffffffa440fee3 ffff88085cf5de28 00000001a0000d41
[ 638.454474] Call Trace:
[ 638.454753] [<ffffffffa440fee3>] mapPciModules+0x5be/0x7f8 [h1iopsusex]
[ 638.455041] [<ffffffffa441011d>] ? mapPciModules+0x7f8/0x7f8 [h1iopsusex]
[ 638.455329] [<ffffffffa4410663>] init_module+0x546/0xd09 [h1iopsusex]
[ 638.455617] [<ffffffff8106363f>] ? load_module+0xc45/0xda8
[ 638.455902] [<ffffffffa441011d>] ? mapPciModules+0x7f8/0x7f8 [h1iopsusex]
[ 638.456190] [<ffffffff8100020a>] do_one_initcall+0x7a/0x129
[ 638.456473] [<ffffffff81063820>] sys_init_module+0x7e/0x1ca
[ 638.456757] [<ffffffff8154763b>] system_call_fastpath+0x16/0x1b
[ 638.457041] Code: 00 e8 8c 51 c1 dc 48 89 c3 48 89 c2 8b 35 27 2a 6b 02 48 c7 c7 1e a7 41 a4 b8 00 00 00 00 e8 1e ad ff ff 4a 89 1c ed 20 1f a3 a6 <8b> 33 48 c7 c7 3b a7 41 a4 b8 00 00 00 00 e8 03 ad ff ff 8b 03
[ 638.458086] RIP [<ffffffffa440f772>] gsc18ao8Init+0xe1/0x294 [h1iopsusex]
[ 638.458376] RSP <ffff88085cf5ddc8>
[ 638.458653] CR2: 0000000000000000
[ 638.459394] ---[ end trace e1932dd1eb06f959 ]---