patrick.thomas@LIGO.ORG - posted 14:43, Tuesday 23 May 2017 (36352)
Restarted FMCS IOCs
WP 6577 Migrated end X FMCS channels to BACNet IOC. Burtrestored to 6 AM this morning (local time).
WP 6577 Migrated end X FMCS channels to BACNet IOC. Burtrestored to 6 AM this morning (local time).
Found a pair of launderable shoe covers in the trash. When finished wearing launderable shoe covers please put them in the basket marked "Dirty Shoe Covers". The disposable shoe covers should be thrown out when you are finished using them. The disposable shoe covers are found in a tote labeled "Disposable Shoe Covers". They are made of a light solid white reinforced paper like material. The launderable show covers have a brown semi hard sole and C3 uppers. The launderable shoe covers cost between $20.00 and $35.00 each, and take 6 to 8 weeks to put into service. Please let me know if you have any questions.
model restarts logged for Mon 22/May/2017 - Fri 19/May/2017 No restarts reported
model restarts logged for Thu 18/May/2017
2017_05_18 12:07 h1broadcast0
2017_05_18 12:07 h1dc0
2017_05_18 12:07 h1fw0
2017_05_18 12:07 h1fw2
2017_05_18 12:07 h1nds0
2017_05_18 12:07 h1nds1
2017_05_18 12:07 h1tw1
2017_05_18 12:09 h1fw1
Maintenance Thursday, DAQ restart for vacuum controls IP6 change.
model restarts logged for Wed 17/May/2017 - Tue 16/May/2017 No restarts reported
model restarts logged for Mon 15/May/2017
2017_05_15 11:25 h1nds1
Unplanned restart of default NDS due to problems possibly caused by new client code.
J. Kissel, J. Warner H1 ISI ITMX ST2 (not ST1 or HEPI) tripped last night at ~9:30 UTC (1179548979, May 23 2017 04:29:21 UTC, May 22 2017 21:29:21 PDT) because Jeff forgot to turn the Reaction Chain damping back on after completing its set of transfer functions (LHO aLOG 36335). In order to help non-Jeffs diagnose the issue, Jeff has moved the link to the reaction chain on the QUAD overview screens, /opt/rtcds/userapps/release/sus/common/medm/quad/ SUS_CUST_QUAD_OVERVIEW.adl SUS_CUST_QUAD_ITM_OVERVIEW.adl to the top center (essentially swapping the positions of the never-used hierarchy switch and the reaction chain screen link). I've also added inputs and outputs to the R0 damping loops surround the button to remind people that there's a control system there which occasionally needs attention. The screens have been committed to the userapps repo. P.S. This interaction between a full isolated ISI and an undamped chain of the QUAD is not new, it's just been a while since we've regularly taken the full suite of undamped TFs on any QUAD, and I've forgotten that one should put the ISI ST2 in DAMPED. And also I haven't used the matlab automated transfer function suite that does all the settings in so long that I fear it'll do more damage than good. In the limit of infinite IFO time and person power (e.g. between O2 and O3?) we'll resurrect that code.
We have identified the source of a comb with a fundamental frequency of 11.111 Hz that appears in the DARM channel and also in the PEM magnetometer channels. The noise source appears to be a particular pressure sensor with the frequency given by the communication with the Beckhoff computer. We measured the magnetic spectrum at BPG402-SE pressure sensors both in the EY station and in the EE shop, and we saw the 11.111 Hz comb that we were looking for (see the attached plot for a spectrum). The frequency measured in the EE shop was either 11.111, when there was one PLC task cycle time of 10 ms, or 11.9 when Patrick set up two tasks with settings of 1 and 10 ms. Using a strobe light, we found that the LED on the pressure sensor also blinked at 11.1 or 11.9 Hz. As expected from the settings, the Beckhoffs in most of the racks at EY produced an 11.9 Hz signal (detected by magnetometers on the slow controls boxes, but not seen in DARM), while the one in the vacuum rack produced the 11.111 Hz signal. The coupling to DARM is likely happening because the pressure sensor used in the ESD pressure interlock is sharing the power supply with the electrostatic drive. The 24 V power cable to the ESD was a strong source of 11.111 Hz magnetic fields. The solution to this problem may be changing the power supply of the pressure sensor. This work will be done this week, and we will be able to test if the comb goes away when the detector is back and we can get spectra of the DARM channel. Pep Covas, Robert Schofield, Patrick Thomas, Richard McCarthy
looks like CER activity possibly caused a timing glitch on h1seih23. I restarted all the models, no need for reboots or power cycles.
In the past couple of days, somewhat more than the usual amount of water required to top off the chiller has been added.
This morning I added ~200 ml. Yesterday JeffB added a similar amount.
I took a look under where the air trap was installed last week and sure enough there are some drops hanging onto
the pipes and a wet spot on the floor underneath.
Found a leak between the 1/2" close nipple and the ball valve. Leak was not apparent when first installed because there was, by design, a trapped volume of air in the standpipe. After this air vented, the water leaked next. Tightening the connection between the valve and the close nipple slowed but did not stop the leak. Will replace the ball valve and the close nipple tomorrow morning, which will require the laser be shutdown.
Owing to the much warmer weather and ongoing 3 IFO work at the mid stations, I have restarted the chillers at both buildings. Temperature set points are at 74 degrees [F] for both.
J. Kissel Measurements were driven and the data was recorded successfully, but the log files that are written during the measurement to record what was driven when for offline later analysis were corrupted this time around because of some sort of missing SIStrLog /broken function and/or file access issue. I've got emails out to Dave Barker, T. Shaffer, and S. Aston in hopes that this code (written years ago over months by transient guest stars B. Sorazu, B. Barr, and L. Prokorov) can be fixed. Will try again tomorrow after some expert help debugging (or, rather, after Fil gets finished pulling cables for this new... vacuum gauge thing? I dunno, didn't see an aLOG. Maybe comes as a consequence of the new ESD driver power supplies that were replaced while I was out, and coincidentally while I was out).
Opened FRS ticket 8218.
J. Kissel
Now the that LVEA temperature is stable and normal, I've gathered transfer functions will all the "nominal" O2 offsets in place:
- Reaction Chain has +150000 V, -5000 P of drive from the test bank, originally installed to alleviate suspected rubbing during our cold January
- Main chain has normal*** P and Y alignment offsets in place.
*** Although the main chain offsets have been used to align the optic to its optical lever reference position prior to the vent, and we suspect that optical lever is a bad reference more on this in a future comment below. However, with this restoration, we're now using up ~85% of the DAC range on the F1 OSEM (where we had been using 35%).
With these offsets in place, preliminary results (i.e. from eyeballing the DTT sessions) show no signs of rubbing.
The data lives here:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMX/SAGM0/Data/
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_L_0p01to50Hz.xml
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_P_0p01to50Hz.xml
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_R_0p01to50Hz.xml
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_T_0p01to50Hz.xml
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_V_0p01to50Hz.xml
2017-05-22_2056_H1SUSITMX_M0_WhiteNoise_Y_0p01to50Hz.xml
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMX/SAGR0/Data/
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_L_0p01to50Hz.xml
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_P_0p01to50Hz.xml
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_R_0p01to50Hz.xml
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_T_0p01to50Hz.xml
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_V_0p01to50Hz.xml
2017-05-23_0047_H1SUSITMX_R0_WhiteNoise_Y_0p01to50Hz.xml
We should consider these offsets a "worst case scenario." I suspect once we get all optics re-aligned to form some sort of Michelson in the corner and/or we open to the arms that we'll find that the optical lever reference was indeed wrong, and we'll end up removing most of the main chain offsets. Plus, I think we now might be able to run with out any reaction chain offsets. Stay tuned for full matlab assessment using the exported data.
On why I think we can't trust the optical lever to restore ITMX alignment. (Even though it's been done a few times.)
Attached are 25 day trends of
(1) The Pitch of the optic as measured by
- the L3 optical lever (in [urad])
- the M0, R0, L1, and L2 OSEMs (in [urad])
- the alignment offset of the M0 stage (in [urad])
compared against the BNS range. (in [Mpc])
This shows that although the optical lever has been restored to a time around 2017-05-07 (just before the vent) none of the OSEMs agree that this is right position
(2) The Yaw of the optic as measured by
- the L3 optical lever (in [urad])
- the M0, R0, L1, and L2 OSEMs (in [urad])
- the alignment offset of the M0 stage (in [urad])
compared against the BNS range. (in [Mpc])
Similarly show that the OSEMs completely disagree with the optical lever
(3) Because the naive always suspect the ISI system, I show the Z, RX, RY, and RZ ST1 position sensors (in [nm] for Z, [nrad] for Rotation), which show > 0.5 [urad] / [um] of position change
(4) The amount of F1 (the pitch) OSEM DAC range in use (85% of the range instead of 35%) and the temperature of the optic / suspension measured in vertical displacement of the main and reaction chains (in [um]) and the FMCS temperature channels.
Another damning piece of evidence against the optical lever for ITMX -- The MICH fringes are terrible at the moment. This leads me to suspect something when wrong with the alignment restoration done to the lowest stage OSEMs by Vaishali and Jenne earlier in the week (referenced in LHO aLOG 36276). In support of this, I attach similar 25 day trends of all PRMI optic's lowest stage OSEMs in
(5) Pitch (in [urad])
(6) Yaw (in [urad])
so we're not that close with any optic according to *those* references.
Also another bit of evidence is that we cannot find a return beam from the Hartman Table (LHO aLOG 36332).
I've discussed with Betsy whether it's possible that the balance of the suspension has changed throwing off all the OSEMs, but she says essentially "no way." All that was done to the SUS was to slide in teflon rails between the test mass's lower barrel EQ stops and the optic, and then the optic was gently raised into its upper EQ stops. The process was reverse when finished. No OSEMs were touched, no balance of any mass was adjusted.
Since there is still suspicion that the IMC is in the wrong place (LHO aLOG 36329) people are reluctant to launch on a big "restoration of the alignment based on short lever arm local things" adventure before we get the arms and global alignment control back.
To clear up potential memory failure leading to confusion, it was ITMY's optical lever that was swapped a few days before the vent (see LHO aLOG 36029), not this one.
This morning Jenne brought the the ITMX suspension back to good values, according to the oplev, so that we could try a to search for a HWSX return beam. I followed Aidan's instructions and did the search just as before:
I ran the measurement from 22:03-23:30 UTC (+/- a few minutes) before I thought it was easy enough to confirm that there was still nothing showing up, and then handed the ITM back to Jeff to complete his measurements. With more time all of the gaps could have been filled, but this does not show any higher intensity points unlike previous tests (alog36179).
After talking with Aidan, he thinks that ITMX is still not within an acceptable aligned range, and the upper periscope mirror may be misaligned. If there is more time later this week we can try again with the amplitude doubled, increasing the search area, but just waiting for the green beam and his arrival might be best.
Note to myself: I left the HWSX fiber in h1hwsmsr
Though I did valve-in the Vertex RGA this morning I took no additional steps listed on WP #6644 as no one is now calling for the premature opening of GV5 and GV7. As such, we will continue to let the Vertex+YBM+XBM "dry out" on the turbo pumps until further notice.
Per request from Jeff K. I have raised the temperature at EX by 2 degrees F. from 64 to 66 degrees.
On what motivated the 2 [deg F] warmer request:
Two attachments: 15 day trends of
(1) the FMCS temperature sensors vs. PCAL temperature sensors
(2) the vertical positions of the SUS
The vertical positions of the SUS show a temperature overshoot (i.e. we corrected the too hot with too cool a set point), because the vertical position has been shooting up higher that before the upgrade (about 2/3rds of the way through the graph).
The PCAL temp sensors suggest that the XVEA was at 20.3 [deg C] = 68.54 [deg F] before the upgrade, and had settled to 19.3 [deg C] = 66.74 [deg F] with the recently set setpoint of 64 [deg] (which was a copy-and-paste from EY a few days ago).
Hence, I requested a 68.54 - 66.74 = 2.16 ~~ 2 [deg F] increase in temperature.
[Vaishali, Jenne]
We went again to have a look at the IMC trans path on IOT2L, and we think that the main beam is just not coming out of the vacuum.
Kiwamu pointed us to alog 17310 from March 2015, where he notes that something happened in vacuum and the IMC trans beam went from somewhat clipped but still somewhat normal looking to totally abnormal looking (which we've had consistently for the last 2 years). Vaishali is currently looking into Cheryl's IMC spot measurements to see if the beam spot motion is consistent with the beam moving closer to a position where the trans beam is clipped on the IM1 suspension cage.
We still have the ghost beam on the trans camera, but no beam bright enough (by a factor of ~100, as measured by the PD) to be the main beam.
Drawing attached showing the beam on IM1 that is IMC Trans.
That beam was -6.5mm off center before the vent, and is now -7.0mm off center after the vent, making the clearance through the light pipe even worse.
The -0.5mm change in beam comes from using the MC3 OSEM changes from before to after the vent.
| 1178267478, before vent | 1179071669, after vent | diff | |
| urad | urad | urad | |
| mc1 p | -25.2 | -29.5 | -4.3 |
| mc1 y | -1030.2 | -1039.3 | -9.1 |
| mc2 p | 500.3 | 497.0 | -3.3 |
| mc2 y | -672.0 | -671.5 | 0.5 |
| mc3 p | -808.9 | -816.4 | -7.5 |
| mc3 y | -978.9 | -986.8 | -7.9 |
The root cause of this issue with IMC Trans are the positions of the beam spots on the IMC mirrors, and in particular the beam spot on MC3 that I measured recently as -5.7mm from center in yaw along the face of the optic.
The two things that we can control that effect the IMC beam spot positions are the IMC input pointing, and I have an approved ECR to add hardware to the IO beams on the PSL to monitor that beam. That beam can also be corrected from the PSL.
An emerging issue is that the steering mirror to the MC2 Trans QPD may be positioned in such a way that maintaining centering on the QPD with it's servo is not producing centered beams on the IMC mirrors.
Jonathan, Dave
The DAQ was restarted at 19:06 Thursday 5/18 UTC (for vacuum controls IP6 work). If an NDS1 request for second trend data for certain channels is sent which spans the restart time, then the data preceding the restart is correct, but post data is bogus. In the case of H0:VAC-EY_Y3_410_PIRANI_INTLK, data at the tail end of the 19:00-19:10 second trend file is a static 2.2, and from 19:10 onwards is a noisy 2.2
We conclude that only the second trend frame from 19:00-19:10 is suspect. Both framewriters wrote the same frame, both nds servers show the same issues if spanning this frame. Investigation continues. I've saved the frame and its md5 file under h1nds1 control's homedirectory (dave subdirectory).
-rw-r--r-- 1 controls controls 434M May 22 11:27 H-H1_T-1179169200-600.gwf
-rw-r--r-- 1 controls controls 33 May 22 11:27 H-H1_T-1179169200-600.md5
This may be similar to a known Dataviewer issue when getting trend data that spans a data gap. See LLO FRS 7239.
Possible answer to Thursday's second trend issues:
Patrick noted that the interlock channel looked like it was showing a different channel's second trend data after the DAQ restart. I then remembered that the original NDS code had a performance feature built into it. When asking for second trends, it read the full channel list from the first GWF file, and then applied those channel offsets for all subsequent files it opened. In the initial system second trend files were one minute in length.
To cover the case where the channels configuration was changed during the requested time span, the code looked for any missing second trend files. At one minute per file it was not possible to restart the DAQ and not have a missing file. The file following the gap was read for the channel configuration and that was used from that point onwards.
Fast forward to today, with 10 minute trend files. It is now highly unlikely for a second trend file to be missing when the DAQ is restarted, and so the channel data pointers will be incorrect after a DAQ reconfiguration. We seem to be seeing this, with the interlock channel being replaced with a cold-cathode raw voltage channel (value of about 2.2V).
After reviewing the nds1 server code it looks like Dave's assesment is correct. The nds1 server implements a nice optimization where it reads in the frame table of contents at the start of a span of contiguous frame files. It only reads a new copy of the table of contents in when there is a gap in the frame files (thus saving a large amount of processing time). Now we can restart the daq fast enough that there need not be gaps in the trend frame files, so the nds1 server will not catch a restart of the daq (and thus the need to read a new copy of the table of contents in). I will work on correcting this on the DTS today.
Starting CP3 fill. TC A error. TC B error. Fill aborted. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 2620 seconds. TC B did not register fill. LLCV set back to 36.0% open.
Dave, Patrick It appears that the temperature of the CP3 thermocouples started above 40 deg C which the script to perform this fill takes as an error. That is, the script takes any reading from the thermocouples above 40 deg C as an indication that the thermocouple is not working correctly. We have created WP 6649 to increase this upper limit.
Thanks!
Increased CP4's manual LLCV %open value from 36% to 37%. Will let new script run before altering CP3's value.
First run of updated script did not complete after an hour: Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill not completed after 3600 seconds. LLCV set back to 21.0% open. Dave is rerunning it.
Fill completed: Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 384 seconds. TC B did not register fill. LLCV set back to 21.0% open.
I looked at CP3's exhaust discharge pressure and am convinced that it did overfill following the second running of the new script. I increased CP3's LLCV %open to 22% up from 21%. It is getting warmer this might be too conservative?
Kyle (Tasks part of WP #6644) ~1015 hrs local -> I valved-in the Vertex RGA to the combined vacuum volumes of the Vertex+YBM+XBM while these volumes were being pumped by their respective turbo pumps. PT120's response is attached (1 hour of second data). "From my head" 10-9 torr/10-5 L = 10-4 torr/L of gas accumulated in the unpumped ~5L RGA volume in 14 days? So, no significant air leaks in the RGA joints. ~1045 hrs. local -> I energized the Vertex RGA filament (note: I am confused with the Pfeiffer software's naming of my connect/scan session with the included phrase "Airdemo..". The other versions of this software don't do this and I makes me question as to whether I am actually connected to the Vertex RGA (h0rgacs) or am I connected to their "Simulator" or virtual RGA in this connect/scan session?
Correction to this obvious typo error: (10-9 Torr)(10+5 L) = 10-4 Torr*L of accumulated gas.
Nothing unusual to report.
Concur with Ed, all looks normal.
