This afternoon, Dave reattached the SDF status to the INTENT bit via the Guardian IFO node. Jenne, Hugh, Kissel, Sheila and I then cleaned house on the SDF in order to make sure everything was green. The operators should now be able to hit the intend bit tonight as needed. If not, figure out what SDF diff is blocking it and either determine to accept or not-monitor it. Please alog any such changes. In the coming days, we will switch all SAFE files in use on the SDF to OBSERVE files so that the full screen shows all OBSERVE files in use when the intent bit it set.
TITLE: 11/29 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventitive Maintenance
INCOMING OPERATOR: Jim
SHIFT SUMMARY:
Cheryl handed over an H1 in NLN, and we actually stayed locked until 9:10am (17:10utc). We had a fairly light Maintenance Day & ended by noon. Then we transitioned to locking up H1 for injection & other work on H1. After a few tweaks by Jenne, we made it to NLN with no issues. On DIAG_MAIN, we have the "IM2 Y out of range" note.
LOG:
I see now Jim had already plotted most of these yesterday but since I had already wrote this log, I thought i should post it anyway. My explainations and plots are better anyway.
The two attached plots show 2 months and 7 days of a number of the status channels from the BRS for diagnosis. On the 7 day plot, the large steps up on the DRIFTMON (upper left) are when Jim opened the BRS box on Thursday and when I opened it yesterday--this is a thermal step and recovery.
The CBIT indicates if the C# code is running. This code reads the camera light image.
The AMPBIT is an indication of the BRS Beam swing amplitude--looking at the 60 day image indicates the PEM crew really got the beam disturbed as it has not been in the past 60 days until last Wednesday. Jim plots 90 days and shows this happened ~21 Sept but things did not get out of whack as it did last Wednesday.
The drop outs (to zero) of the CBIT and CAMERA channels are indications of stopping the code as we attempt to get it running again.
The BOXBIT logics from 1 when the Beckoff communications to the BRS box are disrupted. Not sure why it occurred for Jim and on 8 Nov (no log spotted) but not for me yesterday.
The DAMPBIT just indicates the set point enabled damping has kicked on.
The LIGHTSRC and MODBIT are not included in these plots as they were unity for the duration indicating the light source has been fine and the ISI is communicating without problem. I expected to see a DRIFT_BIT channel but it remained ellusive.
Tagging (hopefully) Ops, so operators see this. An FRS ticket (6799) has been submitted.
Thanks Hugh, Jim, this is very helpful.
The CBIT dropping for short periods is normal. It indicates a slowdown in C# data processing due to extra cpu usage - usually because of a remote login.
The AMPBIT being low is simply showing the problem that the amplitude is large. This is an 'effect' and not the 'cause'. Other bits are similarly not showing anything unexpected.
The main problem is the BOXBIT which drops on Nov. 8th, where surprisingly it did no harm. On Nov. 25th, it dropped and I suspect this is what led to the encoder values being corrupted. After yesterday's fix, Hugh told me that the values were corrupted again and I checked the BOXBIT and I see another drop yesterday evening (see attached pic). I suspect that either the power supply going out to the BRS enclosure being intermittent, or the ethernet cable to the box being loose, is the problem. Or the Beckhoff motor controller may be failing and may need to be replaced.
If the encoder values are corrupt, the BRS-X damper may not work and people should avoid going close to BRS-X while it is being used tonight.
(Sheila Daniel)
Concerning alogs 31573 and 31679, and FRS 6787.
Increasing the thresholds didn't work because the tidal system is engaged, when we are first locked on RF at the AS port which is before the CARM reduction. At this stage the thresholds need to be small, since the arm power is still low. It won't trigger later because we loose the ALS. So, we made a change to guardian to increase the ON/OFF thresholds to 8000/4000 (from 30/10) at the same time that we change the NSUM whitening gains (which is after CARM reduction). They are reset in the down state. We also removed the 3 sec ramp time from the NSUM filter modules. They introduced a false down transient in the arm powers. Without the ramp it will now produce a false up transient.
At the moment these thresholds are optimized for a 2W lock. A further improvement could be to scale them with input power.
Work Permit | Date | Description | alog/status |
6363.html | 2016-11-29 14:14 | change IFO node to disallow SDF diffs in the decision for observation mode. Review current exceptions list prior to O2. | |
6362.html | 2016-11-29 09:54 | Replace POP90 demod chassis. See alog 31948. | 31971 |
6361.html | 2016-11-29 08:59 | Reconfigure power to the Vault equipment to provide +-12Vdc instead of just +12. Power up equipment and get signals to MidX | |
6360.html | 2016-11-29 08:16 | Connect electronics to PT180 sensor tube and energize*Open isolation valve to expose PT180 to site vacuum volume*vent locally mounted turbo, remove and replace it with 1 1/2" O-ring valve + nominal 2.75"CF-NW40 adapter*Pump to rough vacuum the volume between in-series 1 1/2" pump port valves | 31970 |
6359.html | 2016-11-29 06:41 | Tour AREVA group (~8) around site, including LVEA. | |
6358.html | 2016-11-28 17:33 | Update vacuum alarm handler at ops station: Add _ERROR channels for all existing channels. Add all pressure gauges. Remove all commented out channels except CP3. Update guidance text. | 31965 |
6357.html | 2016-11-28 16:09 | For each Vacuum channel being monitored by the cell phone texter, also monitor the corresponding ERROR channel and alarm any errors are non-zero. | |
6356.html | 2016-11-28 14:55 | Change displays in control room in preparation for O2. Add DARM BLRMS to Video0 (sharing with ASC Err sigs). Remove SDF overview screen from Video0 (information is already redundant with newly updated CDS overview screen). Ensure templates are linked to userapps repo, so that most up-to-date FOM is displayed upon computer reboot. | |
6355.html | 2016-11-28 14:10 | Install RF Amplifiers (ZHL-1A+) to add gain to the indoor PEM antenna signals from CER/LVEA. | 31928 |
6354.html | 2016-11-28 11:48 | Remove BP_IN1 channels from DAQ list in h1susprocpi for modes not being actively damped (all except modes 3, 26, 27, 28) | 31927 |
6353.html | 2016-11-28 11:36 | ECR1600351, add 41 fast DQ channels to H1BROADCAST0.ini, restart dmt-broadcaster. | 31927 |
6352.html | 2016-11-28 10:56 | Update Matlab license, there will be a 15 minutes interruption on Matlab. | WP#6352 & WP#6351 are duplicates |
6351.html | 2016-11-28 10:56 | Update Matlab license, there will be a 15 minutes interruption on Matlab. | |
6350.html | 2016-11-28 10:10 | Remove some channels from the frame to save data rate. h1alsex: Remove H1:LSC-X_EXTRA_AI channels from the commissioning frame. No need to archive. h1alsey: Same as h1alsex. h1lsc: Remove H1:LSC-JITTERFF_OUT from the commissioning frame. This feature is not used for now and no need to archive. | 31927 |
6349.html | 2016-11-28 10:01 | Investigate communication issues withing Beckhoff that began when Kyle powered down PT180 | |
6348.html | 2016-11-28 10:00 | Adjust the laser power on the ETMx and ETMy oplev lasers in an attempt to fix the ongoing laser glitches. | 31909 |
6347.html | 2016-11-28 09:54 | Forklift PSL backup chiller from Mechanical building to OSB receiving. | |
6346.html | 2016-11-28 08:21 | Update the calibration code to gstlal-calibration-1.0.9-v1, fix bugs, apply calibration factors, and apply new filters: * Update calib_state_vector calculation to better accommodate changes in kappa smoothing. * Add check for overflow input to GDS pipeline. * Apply the real parts of kappa_tst, kappa_pu, and kappa_c at LHO. * Update the calibration filter file as needed. | 31926 & 31924 |
6345.html | 2016-11-25 14:42 | Replace piezo mirror mount on PSL table. This is the implementation of E1500404 and FRS4639. The work may or may not be done: alignment and PEM injections are higher priority. Approved by Keita. | 31935 & 31974 |
The attached plots shows the power readbacks of the HPO:
The RIN calibration was obtained by dividing by the corresponding DC value.
Figure 1 shows the “before” to “after” improvement in DARM. In addition, the figure shows the large decrease in jitter in the IMC WFSs. Both are included because the coupling to DARM varies, but the jitter signal is consistent.
Figure 2 shows the original and the new mounts for the piezo mirror. The main mechanical changes are: 1) the collar is glued into the U200 mount instead of pressed by the usual single set screw. This stiffens the pitch and yaw motion of the piezo and collar around the connection to the U200 mount. 2) I added a “flying buttress”, bolted to the alternate U200 connection point on the side (inset photo). This is to stiffen and damp the pitch and yaw degrees of freedom around the single connection screw between the U200 mount and the base. 3) I added damping material to the new mount. 4) We shortened the lever arm of the massive mirror, while keeping the mirror in the same place, by moving the U200 attachment point forward to an alternate threaded hole in the base.
The main resonance moved from 260 to 350 where it overlapped with the largest resonance of the top periscope mount, so we brought it back down about 15 Hz by adding moment of inertia with damped clamps. It is now much damped and lying on top of one of the lower amplitude modes of the top mount.
In addition, we noticed that the beam was very close to the edge of the safety tube (second page of Figure 2). We moved the beam further from the edge by reducing the length of the tube by a couple of inches (the tube is angled relative to the beam) and by shimming the inner tube in the telescoping assembly. We replaced the white clean room tape, that was locking the telescoping assembly, with Kapton tape. The safety tube is a couple of inches shorter but still extends over the edge of the table (see figure).
Kiwamu I., Rick S., Robert S.
Betsy, Keita, Daniel
As part of the LVEA sweep, prior to the start of O2, this morning, we spent over an hour doing a cleanup of misc cables and test equipment in the LVEA and electronics room. There were quite a few cables dangling from various racks, here's the full list of what we cleaned up and where:
Location | Rack | Slot | Description |
Electronics Room | ISC C2 | Found unused Servo controller/cables/mixer from top of rack. Only power was connected, but lots of dangling cables. Removed entire unit and cables. | |
Electronics Room | ISC C3 | 19 | D1000124 - Port #7 had dangling cable - removed and terminated. |
Electronics Room | ISC C4 | Top | Found dangling cable from "ALS COM VCO" Port 2 of 6. Removed and terminated. |
Electronics Room | Rack next to PSL rack | Dangling fiber cable. Left it... | |
LVEA near PSL | ISC R4 | 18 | ADC Card Port stickered "AO IN 2" - Dangling BNC removed. |
LVEA near PSL | ISC R4 | 18 to PSL P1 | BNC-Lemo with restor blue box connecting "AO2" R4 to "TF IN" on P1 PMC Locking Servo Card - removed. |
LVEA near PSL | ISC R4 | 20 | T'd dangling BNC on back of chassis - removed T and unused BNC. |
LVEA near PSL | Disconnected unused O-scope, Analyzer, and extension cords near these racks. | ||
LVEA | Under HAM1 south | Disconnected extension cord running to powered off Beckhoff Rotation stage termination box thingy. Richard said unit is to be removed someday altogether. | |
LVEA | Under HAM4 NE cable tray | Turned off via power cord the TV monitor that was on. | |
LVEA | HAM6 NE corner | Kiwamu powered off and removed power cables from OSA equipment near HAM6 ISCT table. | |
LVEA | Unplugged/removed other various unused power strips and extension cords. |
I also threw the main breaker to the OFF position on both of the free standing unused transformer units in the LVEA - one I completely unplugged because I thought I could still hear it humming.
No monitors computers appear to be on except the 2 VE BECKHOFF ones that must remain on (in their stand alone racks on the floor).
We'll ask the early morning crew to sweep for Phones, Access readers, lights, and WIFI first thing in the morning.
Final walk thru of LVEA was done this morning. The following items were unplugged or powered off:
Phones
1. Next to PSL Rack
2. Next to HAM 6
3. In CER
Card Readers
1. High Bay entry
2. Main entry
Wifi
1. Unplugged network cable from patch panel in FAC Rack
Added this to Ops Sticky Notes page.
Maintenance Log: 7:20am - 12:06pm
[Richard, Fil, Kawabe, Marc]
See ALOG 31948
Replaced I&Q Demod Chassis (D0902796) S1000977 with S1001002. We removed the 0 ohm bypass resistors from channels 1 and 2 and installed filters from the old chassis into the new chassis.
Channel 1 = MCL PLP-21.4+ (21.4Mhz Low Pass Filter)
Channel 2 = MCL PHP-100+ (100MHz High Pass Filter)
This new chassis also had the power supply cable installed upside down, we fixed this. The +15V LED lead was not fully seated into the connector housing, we fixed this as well.
We do not have spare filters in stock or we would have replaced these, Fil will order some replacements.
LO monitor level increased by about 0.7dB for POPAIR_B 19 and 90, but not that much for REFLAIR_B 27 and 135, ans as a result POPAIR_B_RF18 was vommiting an error (H1:LSC-POPAIR_B_RF18_DEMOD_ERROR_CODE) and POPAIR_B_RF90 was close. I just increased the nominal LO level by 1dB and accepted in SDF for safe.
H1:LSC-POPAIR_B_RF18_DEMOD_LONOM=23 (was 22)
H1:LSC-POPAIR_B_RF90_DEMOD_LONOM=21 (was 20).
This chassis served POPAIR90, POPAIR18, REFLAIR27 and REFLAIR135, so I have re-set the demod phases for all of these. The POP 18 & 90 I adjusted the phases to maximize the I-phase signal while in DRMI_LOCKED. The REFL 27 and 135 I adjusted also in DRMI_LOCKED, so while we're locked on these 3f signals, by driving PRM in length at 132Hz and minimizing the line in the Q-phase.
I also redid the dark offsets for these channels (although the REFLair ones will get set every acquisition), as well as for AS90 which looked like it needed it.
Kyle R., Gerardo M., Richard M. (completes WPs #6332 and #6360) Initial pressure indication shortly after being energized was 3 x 10-3 torr (PT180 is a "wide-range" gauge). If real, this would be higher than expected for the ~17 hrs. of accumulation -> "burped" the accumulated gas into the connected pump setup while Gerardo monitored its pirani gauge -> it gave no indication of a change and remained steady at 1.9 x 10-3 torr which is as expected. Neither gauge is calibrated in this pressure region of interest. Noted PT180 responded as expected to being combined with local turbo -> Isolated temporary local pump setup, valved-in (exposed/combined) PT180 to site vacuum volume, vented locally mounted turbo and removed from PT180 hardware -> Installed 1 1/2" O-ring valve and 2.75" CF to NW40 adapter in place of the turbo and pumped to rough vacuum the space between the two 1 1/2" pump port valves.
I was informed that the laser had tripped this morning. Not sure exactly why the laser tripped as TwinCAT indicated that the EPICS alarm was triggered. At first glance it looks like the flow in head 3 caused the trip. It reached the trip value (0.4 lpm) roughly 2 seconds before heads 1 and 2. Attached are other trends of the laser power(s) versus the flow rate in head 3. There is some evidence to suggest that the NPRO might have tripped first, triggering the power watchdog for the front end (mis-named Head2NPROa.png).
Summary:
Average range: 70 Mpc
Duty Cycle: 35% (commissioning activities reduced observing time)
1080Hz glitching continuing
Mysterious noise at 10-30Hz for ~2 hours on Tuesday (22nd)
Hardware injections during observing time on Wednesday (1 CBC, 10 detchar safety injections)
Full DQ Shift report: https://wiki.ligo.org/DetChar/DataQuality/DQShiftH120161121
Significant exhaust temperature response (<-30C) 47 seconds after changing the manual mode %open value of the LLCV (liquid level control valve) from its as-found valve of 17 to 50 -> restored this value back to 17 following this exercise.
Here is the temp trend (in seconds). Not sure why the dip prior to fill. First I've seen that.
I trended CP3 exhaust TC temp gauges over 30 days to see if the dips noted above have always been there. Looks like it started on Nov. 15th. The only change to CP3 noted was a Dewar fill while LLCV was set high to 21%. LN2 was coming out the exhaust so I lowered to 16% while the LN2 truck was still filling Dewar. The previous day on the 14th I had filled CP3 by setting LLCV to 100% and noted this was too much flow. Is this an instability in pump reservoir where it burps every 20-40 minutes? Note that from Nov. 19-21 signal was smoother. Could be a function of LLCV setting and how full exhaust line is. Fills lately have been really fast.
CP3 exhaust is also noisy since Nov. 21st.
For completeness, here are the guardian nodes which are currently excluded from the decision of whether to go/stay in observation mode:
EXCLUDE_NODES = [
'DIAG_MAIN',
'INJ_TRANS',
'VIOLIN_DAMPER',
'BRSX_STAT',
'BRSY_STAT',
'CS_BRS',
]
Also excluded are all Seismic CONF nodes and the IFO node itself.
Code is: /opt/rtcds/userapps/release/sys/h1/guardian/IFO_NODE_LIST.py
This file is only used by the IFO guardian node.