Though I did valve-in the Vertex RGA this morning I took no additional steps listed on WP #6644 as no one is now calling for the premature opening of GV5 and GV7. As such, we will continue to let the Vertex+YBM+XBM "dry out" on the turbo pumps until further notice.
Per request from Jeff K. I have raised the temperature at EX by 2 degrees F. from 64 to 66 degrees.
This afternoon while looking into some temperature issues at EY, I found a large puddle of glycol on the roof of the air handler room near the make up tank and on the floor. I traced this back to a PRV that has failed open. The PRV is on the non potable water line which is tied into the glycol return line from the chillers. This appears to be some type of auto fill system that was not used and probably has never worked. Not even sure why it was tied into this system. Anyway, I valved the PRV off and refilled the glycol lines back to normal operating pressure,~30 psi. At this point, I do not intend to replace the PRV because we will not use that system for auto fill.
[Vaishali, Jenne]
We went again to have a look at the IMC trans path on IOT2L, and we think that the main beam is just not coming out of the vacuum.
Kiwamu pointed us to alog 17310 from March 2015, where he notes that something happened in vacuum and the IMC trans beam went from somewhat clipped but still somewhat normal looking to totally abnormal looking (which we've had consistently for the last 2 years). Vaishali is currently looking into Cheryl's IMC spot measurements to see if the beam spot motion is consistent with the beam moving closer to a position where the trans beam is clipped on the IM1 suspension cage.
We still have the ghost beam on the trans camera, but no beam bright enough (by a factor of ~100, as measured by the PD) to be the main beam.
Drawing attached showing the beam on IM1 that is IMC Trans.
That beam was -6.5mm off center before the vent, and is now -7.0mm off center after the vent, making the clearance through the light pipe even worse.
The -0.5mm change in beam comes from using the MC3 OSEM changes from before to after the vent.
| 1178267478, before vent | 1179071669, after vent | diff | |
| urad | urad | urad | |
| mc1 p | -25.2 | -29.5 | -4.3 |
| mc1 y | -1030.2 | -1039.3 | -9.1 |
| mc2 p | 500.3 | 497.0 | -3.3 |
| mc2 y | -672.0 | -671.5 | 0.5 |
| mc3 p | -808.9 | -816.4 | -7.5 |
| mc3 y | -978.9 | -986.8 | -7.9 |
The root cause of this issue with IMC Trans are the positions of the beam spots on the IMC mirrors, and in particular the beam spot on MC3 that I measured recently as -5.7mm from center in yaw along the face of the optic.
The two things that we can control that effect the IMC beam spot positions are the IMC input pointing, and I have an approved ECR to add hardware to the IO beams on the PSL to monitor that beam. That beam can also be corrected from the PSL.
An emerging issue is that the steering mirror to the MC2 Trans QPD may be positioned in such a way that maintaining centering on the QPD with it's servo is not producing centered beams on the IMC mirrors.
This is my final Ops Summary alog.
Activities:
16:16 Carlos -> EY
16:57 Corey to squeezer bay and open roll up door
17:33 Corey out
17:59 Kiwamu to optics lab looking for AM laser
19:20 Richard -> roof
19:33 Richard back
19:36 Fil -> EY
19:49 Kiwamu starting measurement
19:55 Jeff measuring ITMX TF (subbing test)
20:47 Jeff B. to end stations
Kiwamu out
21:22 Jeff B. back
21:43 LVEA transitioned to LASER HAZARD
TJ(HWS table)+Jenne+Vaishali (IOT2L) -> LVEA
21:53 TJ out
22:02 TJ starting SR3 excitation (HWS measurement)
22:13 Carlos to EX
Jenne+Vaishali out
22:36 Carlos back
22:47 PEP-> EY (line hunting)
22:51 Dave -> MY
Added 250ml water to Crystal Chiller to silence low water alarm.
Richard told (or reminded) us this morning that the resonant circuit of the RFPD for locking the IMC had been adjusted for 21 MHz rather than the actual modulation frequency of 24 MHz. So I went ahead to the IOT2L optical table and measured the RFPD response using the AM diode laser. As shown in the attached figure, the RFPD is indeed configured for 21 MHz. Because the resonance at around 21 MHz is so broad, 24 MHz is well within the resonance with a slightly bigger response than 21 MHz by a few dB. The notch seems to be adjusted to 42 MHz. The ratio of the response at 48 MHz over 24 MHz was found to be about -20dB.
The raw data is also attached.
We are working several issues at the moment, here is an executive summary:
bogus second trend data: Some vacuum channels were showing bogus data in their second trends if data over Thursday lunchtime's DAQ restart is requested. Work around is to ask for data up to and beyond the restart time.
BACnet EPICS FMCS data froze around 1pm Sunday (fixed by restarting code)
CP3 overfill did not proceed, thermocouples out of -20C to +40C range (mid 40C). Code will be changed to increase upper limit to 60C. WP6649
Air compressors were turned off at both end stations, Alarm system reconfigured to ignore these channels (WP 6647)
Jeff K. reported that the FMCS channels at end Y had flatlined. I restarted the IOC and they seem to be updating again. I do not yet know why. I burtrestored to 6 AM local on May 20.
Restarted again to enable logging. Burtrestored again. Apologies for the alarms.
Jonathan, Dave
The DAQ was restarted at 19:06 Thursday 5/18 UTC (for vacuum controls IP6 work). If an NDS1 request for second trend data for certain channels is sent which spans the restart time, then the data preceding the restart is correct, but post data is bogus. In the case of H0:VAC-EY_Y3_410_PIRANI_INTLK, data at the tail end of the 19:00-19:10 second trend file is a static 2.2, and from 19:10 onwards is a noisy 2.2
We conclude that only the second trend frame from 19:00-19:10 is suspect. Both framewriters wrote the same frame, both nds servers show the same issues if spanning this frame. Investigation continues. I've saved the frame and its md5 file under h1nds1 control's homedirectory (dave subdirectory).
-rw-r--r-- 1 controls controls 434M May 22 11:27 H-H1_T-1179169200-600.gwf
-rw-r--r-- 1 controls controls 33 May 22 11:27 H-H1_T-1179169200-600.md5
This may be similar to a known Dataviewer issue when getting trend data that spans a data gap. See LLO FRS 7239.
Possible answer to Thursday's second trend issues:
Patrick noted that the interlock channel looked like it was showing a different channel's second trend data after the DAQ restart. I then remembered that the original NDS code had a performance feature built into it. When asking for second trends, it read the full channel list from the first GWF file, and then applied those channel offsets for all subsequent files it opened. In the initial system second trend files were one minute in length.
To cover the case where the channels configuration was changed during the requested time span, the code looked for any missing second trend files. At one minute per file it was not possible to restart the DAQ and not have a missing file. The file following the gap was read for the channel configuration and that was used from that point onwards.
Fast forward to today, with 10 minute trend files. It is now highly unlikely for a second trend file to be missing when the DAQ is restarted, and so the channel data pointers will be incorrect after a DAQ reconfiguration. We seem to be seeing this, with the interlock channel being replaced with a cold-cathode raw voltage channel (value of about 2.2V).
After reviewing the nds1 server code it looks like Dave's assesment is correct. The nds1 server implements a nice optimization where it reads in the frame table of contents at the start of a span of contiguous frame files. It only reads a new copy of the table of contents in when there is a gap in the frame files (thus saving a large amount of processing time). Now we can restart the daq fast enough that there need not be gaps in the trend frame files, so the nds1 server will not catch a restart of the daq (and thus the need to read a new copy of the table of contents in). I will work on correcting this on the DTS today.
After making sure all the view ports on HAM1,2, and 3 are covered (all the ones that Robert could have used), I opened the PSL light pipe for commissioning work. The LVEA is still LASER SAFE.
Starting CP3 fill. TC A error. TC B error. Fill aborted. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 2620 seconds. TC B did not register fill. LLCV set back to 36.0% open.
Dave, Patrick It appears that the temperature of the CP3 thermocouples started above 40 deg C which the script to perform this fill takes as an error. That is, the script takes any reading from the thermocouples above 40 deg C as an indication that the thermocouple is not working correctly. We have created WP 6649 to increase this upper limit.
Thanks!
Increased CP4's manual LLCV %open value from 36% to 37%. Will let new script run before altering CP3's value.
First run of updated script did not complete after an hour: Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill not completed after 3600 seconds. LLCV set back to 21.0% open. Dave is rerunning it.
Fill completed: Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 384 seconds. TC B did not register fill. LLCV set back to 21.0% open.
I looked at CP3's exhaust discharge pressure and am convinced that it did overfill following the second running of the new script. I increased CP3's LLCV %open to 22% up from 21%. It is getting warmer this might be too conservative?
On what motivated the 2 [deg F] warmer request: Two attachments: 15 day trends of (1) the FMCS temperature sensors vs. PCAL temperature sensors (2) the vertical positions of the SUS The vertical positions of the SUS show a temperature overshoot (i.e. we corrected the too hot with too cool a set point), because the vertical position has been shooting up higher that before the upgrade (about 2/3rds of the way through the graph). The PCAL temp sensors suggest that the XVEA was at 20.3 [deg C] = 68.54 [deg F] before the upgrade, and had settled to 19.3 [deg C] = 66.74 [deg F] with the recently set setpoint of 64 [deg] (which was a copy-and-paste from EY a few days ago). Hence, I requested a 68.54 - 66.74 = 2.16 ~~ 2 [deg F] increase in temperature.