Rough pump down was faster than normal. Implies that the majority of the gas load comes from the O-ring sealed QF flex lines that connect the scroll pump assembly to the HAM6 connection as these were pumped overnight. Typically, these get pumped simultaneous with HAM6. In other words, it seems that the time needed to rough pump HAM6 down to the point of switching over to the turbo pump is significantly determined by how wet the connecting flex lines are. We have always suspected this but never realized to what degree.
Large units over HAM3, HAM4, HAM5, HAM6 and the Beer Garden, as well as, the small change rooms etc. on north side of OMC tube.
Filiberto C., Daniel S., Marc P.
Today we successfully applied E1600252v2 to the OMC DCPD ISC Split Whitening Chassis S1101627. This change better accommodates the violin modes of the main LIGO optics in the vicinity of 500Hz that have a tendency to saturate the whitening chain's filters. The new filter replaces a 1Hz-10Hz zero-pole, with a 50Hz-500Hz pole-zero on the second filter channel. LLO had already completed these changes August 2016, ALOG.
Updated the DCPD whitening filter at stage 2 to a nominal zpk([50],[504],-1, "n"), ie., zero at 50 Hz, pole at 504 Hz, gain of -1. This may effect the calibration.
On Chandra's request, the high alarm level for PT245 was raised from 5.0e-09 Torr to 5.0e-07 Torr.
-<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-09" description="VE gauge, MY CP4, CC">
+<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-07" description="VE gauge, MY CP4, CC">
All of H1 models are now running back at RCG3.2, some with their most recent code. Those with new code are waiting for a DAQ restart to resync the channel lists (scheduled for noon).
As of this morning there were several models which would not run: all ISI-HAM (h1isiham[2-6]) and h1susauxh34. These models got new code yesterday, but others did also and they had no reversion issues.
To verify that a clean rebuild of the new code against RCG3.2 fixes the problem, I used h1isiham6 on h1seih16 as a test case. To minimize the risk of Dolphin glitching the corner station if this did not work, h1seih16 was removed from the switch fabric prior to testing.
1. Remove h1seih16 from the Dolphin fabric and powered off the computer
2. Disconnected the Dolphin cable from the rear of h1seih16 and powered it up
3. Rebuilt/installed and restarted h1isiham6 code. At this point everything was running, SWWD was of course tripped due to the disconnected Dolphin cable.
4. Repeated step 1.
5. Plugged the Dolphin cable back into h1seih16, powered up computer, started models
Initially all was good, but after a few seconds the IOP Dolphin IPC for the SWWD went to no reception (64kHz error rate). I restarted the IOP several times and the problem appears to have resolved itself with no input from me.
I ran some diagnostics on h1susauxh34 during its crashes, then rebuilt that code against 3.2 and it is also now stable. Finally I rebuilt h1susauxe[x,y] and h1pemmx.
ISI-HAM systems now have the Ramped-Watchdog code.
The HEPI database running the Pump speed controllers monitors the differential and Pump Station output pressure; the usuals originally used to set the alarm level must have drifted. The Pump Station filters were recently changed and this may have changed the steady state.
Will investigate. FRS 10725.
The pressure okay alarm is set for the diff to be 70 +-2 and the last pressure on the Pump Station to be 80 +- 3psi. The former is the set point and the latter is just from trends:
field(INPA,"$(IFO):HPI-PUMP_$(LOC)_PS_PRESS4")
field(INPB,"$(IFO):HPI-PUMP_$(LOC)_DIFF_PRESS")
field(CALC,"B>68&&B<72&&A>77&&A<83?1:0")
The pressure sensors also have offsets in their zero readings that are hardcoded into the database but apparently these are no stationary. I saw that the zero levels were not zero so I zero'd all the offsets and the pressures were close enough to zero to leave them that way.
After getting things back up and looking at the trends, I'd say the following:
The offsets were contributing to some of the change in 'normal' and the flushing and new filters contribute with plant changes. All the pressures are lower now even out at the actuators.
The change in OFFSETs have pulled the average of Pressure4 up to about 78.5 where it had been right on the alarm edge at 77psi. This should stop the Red flashing Pressure NOT Ok.
Closing Ticket.
[Aidan, Marie, Dan, Keita, Alexei, TJ]
Yesterday we started the process of installing the new green LED source for the ETM HWS, per T1800229.
We started by making sure the ALS beam was aligned to the TMS and there was a return beam getting to the ALS photodiode. We inserted two irises, one immediately after ALS-M11 and another about half way up the periscope. These were aligned to the existing ALS beam to secure the axis. Unfortunately, subsequent analysis at the end of the day showed that one of the PZT mirrors railed at some point during or after this process of aligning the irises. We're not currently sure if they really represent the ALS axis. We will investigate this further this morning.
We inserted all the optics per T1800229 (the fiber LED source, 4 steering mirrors, a polarizer, a 50/50 beam splitter). Also, we added the polarizing beam splitter which transmits the P-polarized (horizontally polarized) HWS light and reflects the S-polarized ALS beam. This PBS has not yet been aligned to get the ALS beam back to centered on the irises.
The new HWS beam was aligned to the centers of the existing optics. We have yet to align it to the PBS or the irises.
A couple more notes:
Some photos from installation are attached.
WP7578. Hugh, Jenne, Jonathan, Dave:
After upgrading the H1 models to RCG-3.4.2 and the front ends to Gentoo 3.0.8 we ran into a Dolphin network manager issue at 3pm PDT. Given the lateness of the hour, we decided to revert back to rcg-3.2/gentoo 2.6.34. All systems are now back with the exception of the HAM Seismic systems which are crashing on start-up. We will resolve this in the morning.
During the reversion process all three of the Gen-2 front end systems (h1suse[x,y], h1lsc0) lost connection with their IO Chassis and required a full system power cycle. This is unusual as recently only h1susex has infrequently required this type of reboot.
To revert, I restored the archived target directories, the IPC file, the original INI and PAR files for those systems whose code had changed and the DHCP service on h1boot.
This is consistent with what was seen at LLO - see aLOG entries aLOG 38382, aLOG 28392, aLOG 38401. This issue snuck up on us because we had not rebooted front-ends for many, many months. It is related to PCIe expansion fiber aging and newer front-ends (10-core Intel V2 from 2015, 6-core Intel V4 from 2018). Not all fibers have the same degradation (i.e. l1susex was fine). Also newer fibers from 2015 seem OK as well (i.e. l1susey is fine) For diagnostics, I would swap fiber on h1susex or h1susey with a newer fiber, then take the old fiber to the DAQ test stand so we can try to repeat it there and see if some BIOS settings on the Intel V4 machines can get around it.
Looks like the HAM-ISI kernel objects from the latest target_archive do not match the latest in the RCG-3.2 build area. This suggests a clean make-install and restart should fix this problem. For a first try, we should think about removing the front end from the Dolphin fabric, otherwise if it were to fail it would take the corner station models down (as was happening yesterday).
PeterK, RickS
Today, we installed a periscope that allows us to sample the 35-W FrondEnd output relatively close to the laser output aperture (see attached photo) and measure the beam profile using a rail mounted on top of the FrontEnd box.
The beam parameters (waist size and location) estimated from least-squares fits to the data of gaussian beam propagation are shown in the attached plots as well as beam profiles measured with the ThorLabs BeamScan instrument both at the beginning of the rail (1 cm) and at the back of the rail (50 cm) are also attached.
Attached snapshots show the summaries for EY SEI and the OPS overview:
h1fw2 ran over the weekend with the cpu affinity settings. The retransmit rate was higher than expected. However it is still a lower rate than h1fw0, so a work permit is filed to update the other debian 8 daqd systems tomorrow. For tonight h1fw2 will run with some minor tweaks to the crc checking code that remove some runtime table generation. h1fw2 has also been configured to request per thread cpu affinity which seems to do a good job at keeping threads from migrating across all the cores. The behavior of h1fw2 tonight will impact how many of the changes are going into h1fw0 tomorrow.
Chandra, Gerardo, Kyle
Spent most of this afternoon troubleshooting the newly assembled maglev backing pump cart. Discovered that the Sentry isolation valve (redundant isolation) was faulty so it was removed for now. Also the 2-position Interlock selector switch was wired backwards etc... -> Cart is connected and operational and is pumping against a closed valve for tonight. The purge-air was decoupled and HAM6 blown down to a "slightly" positive pressure. The Kobelco compressor was also shut down.
We are set to begin pumping HAM6 tomorrow morning. Chandra R. has OKayd unattended pumping so we expect to be ready to switch over to the turbo by Wednesday.
We didn't start pumping tonight because there is some confusion about where to turn high volts off, particularly new squeezer components. M1300464 needs to be updated with clearer descriptions if VAC personnel are to verify HV off before pump down or vent.
The PSL room fans did not start up as usual this morning, which was noticeable in that after entering the PSL room, it was too quiet. The fans were toggled on/off/on from the box inside the anti-room, without any change, and then toggled off/on from the box outside of the anti-room, and the fans started up, and ran without issue.
The PSL room fan situation led to a discussion about dust, and I mentioned that the IO optics are dusty, and I mentioned that when it's windy the dust counts in the PSL room increase.
Rick suggested the tubes that go through the PSL wall, near the floor under the light pipe, as a possible source for the dust, as part of the theory that during high winds, the LVEA is at a higher pressure that the PSL room, so dusty air is pushed into the PSL room.
I've looked at channels to see if I can identify situations that support LVEA air being injected into the PSL, and have found one 12 day period last year, where there are 3 events that raise the PSL level of dust,. Two of the events are caused by wind, and one event is caused by entry into the PSL.
In the order of how much dust was measured in the PSL (highest to lowest):
event 1: 6/27/2017 ~1:19UTC - highlighted in red
event 2: 6/16/2017 ~8:50UTC - highlighted in orange
event 3: 6/20/2017 ~17:16UTC - highlighted in green
These three events are shown in the two attached files:
- Rick, Cheryl
I talked to Jeff Bartlett about the PSL dust levels, and together we looked at an old alog of his, alog 32104, and we remade the plot (attached), with some added channels, and this plot also supports the theory that LVEA air is getting into the PSL during times of higher winds. The LVEA dust at 0.3 um goes above 15000 counts, while the PSL 0.3 um increases to almost 2500 counts.
15:30 Peter JeffB to H2 diode room
16:20 Karen to EY
16:30 Tyler, Mark, ChrisS to HAM6 to torque bolts
16:45 Cheryl to PSL
17:15 Chandra to EX looking for leak checker
17:15 TJ, Aida, Keita, Dan, Alexei headed to EY
17:30 Ken at MY
17:30 Tyler, Mark moving leak checker from EX to MY
18:00 Richard to PSL
19:00 Suresh, Jason to EY
19:45 DickG to CER
19:45 Betsy, Travis, Fil to EX
21:00 Aidan, TJ, Dan, Alexei, Marie to EY
20:15 Peter to PSL
Re-terminated the pins for the ERM pigtail connector. See alog 41891 for LR and Bias failing HI-POT test. All pins were tested for the following:
1. Checked no shorts between inner pin and connector body.
2. Tested all segments (LR, UR, BIAS, UL, and LL) were not shorted to each other.
3. Performed HI-POT test to 1kV for each pin, all passed.
4. Connected all cabling in-chamber and repeated HI-POT test on air-side. All passed (UR only tested to ~800V).
R. Abbott, F. Clara, T. Sadecki, B. Weaver
Followed procedure E1800147-v2 for re-termination of connectors.
Completed continuity tests for ETMX ESD from air-side flange to optic. All air-side cabling from feedthrough to rack were labeled.
Re-energized HAM3 and HAM4 clean rooms as there still is a possibility of installing new large ion pumps