Checked dust monitor vacuum pumps at both end stations and the corner station. All three pumps are working well. Pressures were good and the temperatures were a few degrees lower due to the cooler weather. Closing FAMIS #12998.
The cleaning crew completed the quarterly cleaning inside the PSL laser room and the Anti Room. Jason power reset the dust monitor in the Anti Room. Closing FAMIS #13788
D. Sigg, M. Pirello
The new AM-AOM Servo was installed on the green path and checked out last night. We are still assembling and testing the second servo to install on the CLF path later today or early tomorrow.
At 18:09 Mon 14th PDT h1susauxh56 cpu crashed with a kernel panic message
At 10:06 today, h1sush56 cpu crashed with an identical kernel panic message
I checked the IO Chassis look good, they do.
I'm starting the reboot process.
[Rick, Vlad, Niko]
On Thursday (10/10) we went to EY to address an alignment issue that I noticed during Tx module maintenance earlier that week. The beam spots on the Rx side had drifted significantly within the last 10 day period, possibly as the result of accidental movement of one the optics or temperature changes at the end station.
We took the alignment opportunity to simultaneously adjust the beam splitter to reduce power imbalance between the inner and outer beams. We lowered the imbalance to within 1%, and then centered the beams on the Rx integrating sphere aperture. I will be checking the alignment over the next few weeks to make sure everything's alright. (Will attach photo of new beam alignment once Rick sends it to me)
Chiara, Jenne, Jim.
Yesterday, Mon 14th Oct, we modified a number of models to prepare to try the control scheme outlined in T1900656. We hope to use differential CPS signals on corner station ISI's to improve low frequency cavity motion. This is very similar to the measurements Arnaud and Seb have been working on at LLO.
Now that we have modified, compiled and installed the models, we started to create screens: we need a filter bank matrix for CPS and one for LSC. The output of every row of a matrix will be added to the corresponding output of the other matrix, and the sum will be sent to the corresponding limiter. A draft of the CPS filter bank matrix, LSC filter bank matrix and signals to limiters are shown in screenshots 4,5,6.
Modified models are scheduled to re-start today.
For future reference this was an implementation of ECR E1900330
Attached is power point with images that show were we have IR light where we should not, fringes where we should have beams, and IR light artifacts where we never have had them before.
Since putting in the new PMC, the IO path has had 2 optic changes, both successfully installed and the input pointing restored.
The beam through the EOM and through the entire IOpath has degraded over time, see my FRS 11631 about the clipping on the output of the EOM in the IO path. The attachment shows that the beam shape at IMC_IN has gotten worse.
Given that I've never had IR artifacts, and fringes where there should be beams, and I have watched the alighnment degrade, I can only conclude that aliging the IO path is needed to correct these issues.
updated document in alog with beam proiles, alog 52499
Jonathan, Erik, Dave:
Following the replacement of the EY Dolphin switch with bad ports with a new switch, and after finding problems with the new switch re-installing the original switch only to find the RFM EY Dolphin card was not being activated, here are the hardware investigations we tried to restore the Y arm RFM (note that when swapping Adnaco chassis we reused the single-mode SFP and fiber).
(Starting with the identical hardware the system had before today's work, but now after rebooting h1cdsrfm its EY card is not being activated but is seen on the PCI bus)
Instead of just rebooting h1cdsrfm we power cycled the computer and the EY Adnaco chassis. Subsequently this was also done several times as we tried different chassis.
We did a 'big boot' and powered h1cdsrfm and both EX and EY adnaco chassis.
We noted EY was running hot, and the chassis fan was not working. After discovering that the fan power cord was loose, and was fixed by reseating it. After this EY was still running hotter than EX.
We replaced the Adnaco with a spare from the DTS (equiped with the orginal IXH611 card from sep2018 through to jul2018). The fiber link did not work.
We tried the second spare Adnaco from the DTS equiped with a brand-new out-of-the-box IXH611 card (we remembered to set the dip switches), again no fiber link.
We put the original Adnaco back (the one whose link works) now with a fixed fan and equiped with the brand-new IXH611.
We used h1iscey's switch port and X8 cable.
Final configuration at the end of the day is original Adnaco with new IXS611 card. We found a mismatch in the card serial number and the dishosts.conf and ixnodetab files, but fixing this did not change EY.
Some items to note:
Most of the times after we took h1cdsrfm out of the Dolphin fabric to reboot/power itr there were no problems with other system, except in two cases where all the EY models crashed. I would estimate 6 good restarts to each crash restart.
On one of these crashes, h1iscey could no longer see its IO Chassis. After a full system power cycle this was restored.
The CDS system as we have left it overnight:
local Dolphin working at all three locations (CS, EX and EY). Looks like h1sustmsy still has its watchdogs engaged.
long range dolphin not working, giving various IPC errors.
It appears that h1susauxh56 crashed after we left this evening? I'll restart it tomorrow.
Various CFC flags are showing following Jenne and Jim's make-install for new isi-ham, seiproc and oaf model changes.
I used /opt/DIS/sbin/dis_diag remotely to figure out what it going on. h1iscey - OK - Switch port 3 - sees ISC,SEI,SUS not RFM card h1seiey - OK - Switch port 2 - sees ISC,SEI,SUS not RFM card h1susey - OK - Switch port 1 - sees ISC,SEI,SUS not RFM card h1cdsrfm - Sees 3 cards - CTR, X OK, Y-end NOT OK - sees only RFM card -- Behavior acts as if either cable from Y-end adapter to switch is not OK, or port on Y-end Dolphin [table says #6] is disabled. I tried a manual port enable (by running /etc/dolphin_ix_port_enable.sh on h1cdsrfm), but no change. * I would suggest moving h1cdsrfm at Y-end to another port. Ports are enabled by default. If that works, then later update 'ixnodetab'.
{Bubba, Tyler, Chris, Chandra}
Removed the north door on HAM12 and currently storing on door hanger. We temporarily covered the HAM flange with a C3 soft cover. Plan to cap the septum plate with the three blanks that Gerardo found today. We will leave HAM12 stored up to air with C3 covers in place of doors. Meanwhile Bubba ordered 92.25" shipping covers for HAM11 to preserve HAM 7,8,9,10 vacuum surfaces in dry air.
This HAM12 door has some sratches along outer o-ring seal surface. Scratches are not new; we suspected an outer o-ring leak somewhere in HAM 11, 12 based on annulus ion pump behavior. Stoning technique may smooth them out.
Additional photos here.
Bubba G., Corey G., Kyle R., Tyler G.
Installed HAM6 North door.
Kyle R., Tyler G.
Installed HAM6 RGA (East Door, leftmost port)
Kyle
Rough pumped HAM6 to ~0.4 torr then switched over to turbo pump. Roughed at vent valve using turbo backing cart with dedicated "clean" maglev turbo backing flex hoses before moving cart and hoses over to back maglev turbo. These flex lines are only used to rough at the vent port or back HAM1/6 maglev turbos, i.e., are never used to vent.
Material and machinery delivery today. But, NetServices did break ground this afternoon. The attached photo shows the 8'x8'x18" cassion pad excavation. More fun tomorrow for sure!
This morning continued to pump down IP10 volume, then after lunch conbined IP10's volume with the X-Mid main volume. IP10 power cables were attached, including the ground cable.
The turbo pump will continue with the pump down, and tomorrow we'll leak test the last modified conflat.
J. Oberling, C. Vorvick
Today I began tuning up alignment and mode matching into the PMC and RefCav. I began by tweaking alignment remotely from the control room before entering the PSL enclosure; all work was done with the ISS OFF. The changes are summarized in the below table:
| Before | After | |
| PMC Trans | 51.2 W | 52.8 W |
| PMC Refl | 12.8 W | 11.3 W |
| RefCav TPD | 1.5 V | 5.4 V |
I then attempted to adjust the operating current and temperature for the pump laser diodes, but did not see any improvement so I left them as found (documented here). As can be seen, the RefCav transmission was completely recovered with a simple alignment tweak, therefore no adjustment to the FSS will be needed.
Upon entering the enclosure, we began by taking a beam propagation measurement of the 70W amp output. This is done by picking off the beam to the bullseye PD and directing it down a rail and into a WinCam beam profiler. I've attached a picture of the beam profile, which can be compared to the last profile taken in November 2018 (documented here); the picture in the linked alog is rotated by +90° w.r.t. the image attached to this alog, due to a different orientation of the WinCam). The halo now looks a little more pronounced and the beam isn't quite as round as it was (attenuation settings are different (different ND filters, don't remember what we used last November), therefore I can't really say how much of a change this is). I'll analyze the beam propagation data and attach the fit as a comment to this alog.
This done, we began adjusting the position of the PMC mode matching (MM) lenses. We started by tweaking the 2nd MM lens in both directions, looking for improvement. Once a better location was found, we then adjusted the 1st MM lens, then returned to tweaking the position of the 2nd, once again looking for improvement (if it doesn't get better, adjust the 1st MM lens in the opposite direction, then go back to adjusting the 2nd). Lather, rinse, repeat. At first, improvements were small (a few mW here, a few mW there, but nothing approaching where we left things last November). This lasted until translation stage adjustment range on the 1st MM lens ran out. To recenter the adjustment range I had to move the entire mount 8mm closer to the 70W amp. Once I had done the gross mount move, I was recentering the translation stage in stages (so as to not lose the PMC alignment) and noticed that when the lens was 4.5mm away from the center of the translation stage (i.e. 4.5mm closer to the 70W amp than before I moved the entire mount) there was a large improvement (relative to previously observed changes) in the PMC trans and refl. Since it was after 4pm at this point we called it a day; will start Round 2 tomorrow morning. When we left the enclosure the PMC was transmitting 53.1W and reflecting 11.0W. I've attached pictures of the before and after locations of the PMC MM lenses for future reference (yes, PMC MM Lens 01 looks like it hasn't moved. It has, the mount is in a different location, the translation stage being in the same spot is a coincidence).
Jonathan, Erik, Dave:
WP8354
This morning we replaced the EY Dolphin IXS600 switch (with bad ports) with a new unit. We then saw h1cdsrfm problems, it could not activate its EY IXH611 card. After various reboots/restarts (which I'll detail later) we put the original switch back in.
At this time all the hardware is identical to early this morning, but h1cdsrfm still has issues with its EY IXH611 card.
We ran dis_diag on h1cdsrfm to see if there was anything useful. One note, the temperature on the 2nd adapter (EY) is high. For a comparison we also ran dis_diag on h1iscey which is on the same rack to give a comparison.
I've attached some diagnostic information from h1cdsrfm. Output from dis_admin_commandline MACHINE NAME : h1cdsrfm SERIAL NUMBER : IXH-002348 ADAPTER TYPE : IXH611 CONNECTED : Yes ONLY ETHERNET OUT : No DISHOSTS NODEID : 104 ACTUAL NODEID : 104 ACTUAL ADAPTER STATE : Disabled NUMBER OF PORTS : 1 PREFETCH MEM SIZE : 256 CURRENT CONFIGURATION : Ok PARITY ERRORS : 0 SYSTEM ERRORS : 0 PORT link width : x8 PORT active : No PORT protected : Yes PORT operational : No PORT state : Disabled Partner information is : invalid Partner adapter number : 0 Partner port number : 0 Partner board type : Not determined Partner number of ports : 0 Output from the IX Switch's info command LINK1_STATUS : G1 x8 (1/0) LINK2_STATUS : G1 x8 (1/0) LINK3_STATUS : G1 x8 (1/0) LINK4_STATUS : NC LINK5_STATUS : G2 x8 (0/0) ---- This is cdsrfm LINK6_STATUS : NC LINK7_STATUS : NC LINK8_STATUS : NC
The temperature has stayed high for the 2nd adapter. Another interesting point is the state of the topology settings. Adapter 2 is switch topology with topology auto detect disabled. Where the other adapters where topology switch with topology auto detect off.
From the dis_nodemgr.log file. Note that there are 2 calls to dis_config for adapter 2, the last one sets the topology to 4 (not 1 which would be switch). Oct 14 2019 15:43:40 Unconfigured adapter with serialno 0x93d (is valid) found (adapterno 100) Oct 14 2019 15:43:40 Unconfigured adapter with serialno 0x928 (is valid) found (adapterno 101) Oct 14 2019 15:43:40 Unconfigured adapter with serialno 0x92c (is valid) found (adapterno 102) Oct 14 2019 15:43:40 Adapter number changed: Adapter[0] adap_conf.adapter_no 0 adap_stat.adapter_no 100 Oct 14 2019 15:43:40 Setting NEW link width 8 OLD was -1 (shadow=118) Oct 14 2019 15:43:40 Adapter 0 AdapterNo 0 Setting NEW nodeid 96 OLD was -1 Oct 14 2019 15:43:40 set dis_config: call: /opt/DIS_5_7_0/sbin/../sbin/dis_config -s 2365 -a 0 -slw 8 -n 96 -slw 8 Oct 14 2019 15:43:40 Adapter number changed: Adapter[1] adap_conf.adapter_no 1 adap_stat.adapter_no 101 Oct 14 2019 15:43:40 Setting NEW link width 8 OLD was -1 (shadow=8) Oct 14 2019 15:43:40 Adapter 1 AdapterNo 1 Setting NEW nodeid 100 OLD was -1 Oct 14 2019 15:43:40 set dis_config: call: /opt/DIS_5_7_0/sbin/../sbin/dis_config -s 2344 -a 1 -slw 8 -n 100 -slw 8 Oct 14 2019 15:43:40 Adapter number changed: Adapter[2] adap_conf.adapter_no 2 adap_stat.adapter_no 102 Oct 14 2019 15:43:40 Setting NEW link width 8 OLD was -1 (shadow=8) Oct 14 2019 15:43:40 Adapter 2 AdapterNo 2 Setting NEW nodeid 104 OLD was -1 Oct 14 2019 15:43:40 set dis_config: call: /opt/DIS_5_7_0/sbin/../sbin/dis_config -s 2348 -a 2 -slw 8 -n 104 -slw 8 Oct 14 2019 15:43:40 Adapter 0 configured with 24 reachables Oct 14 2019 15:43:40 Adapter 1 configured with 24 reachables Oct 14 2019 15:43:40 Adapter 2 configured with 24 reachables Oct 14 2019 15:44:01 Setting NEW topology 4 OLD was 0 Oct 14 2019 15:44:01 Adapter 2 AdapterNo 2 Setting NEW nodeid 104 OLD was 104 Oct 14 2019 15:44:01 set dis_config: call: /opt/DIS_5_7_0/sbin/../sbin/dis_config -s 2348 -a 2 -stt 4 -n 104 -slw 8