After an Alignment, I wanted to run an INIT state for ISC_LOCK. Noticed that it was nowhere to be found. Mentioned this to TJ, and he ended up running the INIT by hand via Guardian command line.
While Corey and company were working on the TCS chiller this morning (30037), I took this opportunity to do a quick health-check on OMC with a single-bounce beam. So far things seem good.
The attached is the measured open loop from this morning.
FAMIS#7405
Laser Status:
SysStat is good
Front End Power is 34.68W (should be around 30 W)
Front End Watch is GREEN
HPO Watch is GREEN
PMC:
It has been locked 0.0 days, 1.0 hr 33.0 minutes (should be days/weeks)
Reflected power is 29.41Watts and PowerSum = 138.0Watts.
FSS:
It has been locked for 0.0 days 1.0 hr and 33.0 min (should be days/weeks)
TPD[V] = 3.913V (min 0.9V)
ISS:
The diffracted power is around 3.862% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 33.0 minutes ago (should be days/weeks)
Possible Issues:
PMC reflected power is high
Most of the meeting was summarizing Maintenance Day yesterday.
No Safety Meeting (All Hands is tommorrow).
Dicsussion about protocol for transitioning to LASER SAFE for Maintenance.
Since about 2am (about 5min after the lockloss of the night), We have been getting the following TCS Alarm:
"No TCSY chiller flow"
| Work Permit | Date | Description | alog |
| 6195.html | 2016-09-27 15:22 | Activity: add task Work Permit field to the reservation system. This is an additional argument to make_reservation and an additional column in display_reservation. Area of Activity: Reservation system | |
| 6194.html | 2016-09-27 10:25 | Activity: update GDS calibration filters | 30013, 30028 |
| 6193.html | 2016-09-27 09:05 | Activity: Power IO timing slave off alternate power supplies for the following: Corner LSC IO Chassis EX ISC IO Chassis EY ISC IO Chassis. Both front end computers and IO chassis with need to be restarted. | 30027 |
| 6192.html | 2016-09-27 06:25 | Activity: HFD to preform annual inspection of all site fire extinguishers. Joe D. will escort them. [HFD = “Hanford Fire Department”] | |
| 6191.html | 2016-09-26 12:06 | Activity: swap first and second ADC cards in h1oaf0's IO Chassis. Verify power cycle of IO Chassis and AI chassis causes TCS laster trips via chiller trip. | 30027 |
| 6190.html | 2016-09-26 11:36 | Activity: Upgrade guardian, cdsutils, gpstime packages to new releases | 30025 |
| 6189.html | 2016-09-26 11:25 | Activity: un-swap the 10GE data chain for h1fw0 and h1fw1. These were temporarily swapped when fw0 was unstable. | 30027 |
| 6188.html | 2016-09-26 11:23 | Activity: restart DAQ and Guardian with the EPICS gateway between the H1FE and H1SLOW LANs turned off. Ideally we should not need to run this gateway, it caused issues with last week's guardian reboot. | 30027 |
| 6187.html | 2016-09-26 11:20 | Activity: Mount scroll pump to Vertex Turbo Pump framing*Start and run Vertex Turbo, chilled water booster pump and both QDP80 rotating pumps (located in Mechanical Room adjacent to LVEA)*Test functionality of Turbo Pump's Safety Valve (foreline isolation) when interlocked to newly installed scroll pump*Shut down pumps when testing complete | 30005 |
| 6186.html | 2016-09-26 10:44 | Activity: Flush the TCS chillers. First we drain the old water out of the chillers, fill in the new water, then flush until all water replaced. | 30017 |
| 6185.html | 2016-09-26 09:04 | Activity: Fix broken ISS channel Fix power scaling in model | 30001, 30027 |
| 6184.html | 2016-09-26 08:23 | Activity: Update h0vaclx, h0vacly, h0vacex and h0vacey to add EPICS alarm levels to the Inficon BPG402 gauges. Will trip high voltages. No DAQ restart needed. | 29996, 29998, 30003 |
| 6183.html | 2016-09-26 07:50 | Activity: Replace the cable walkover with a larger one to relieve the congestion in the current cable walkover. Also bled air out of water line at filter under PSL table. | 30012 |
| 6182.html | 2016-09-23 17:51 | Activity: Give a tour in the LVEA to a Japanese science writer and his company who happened to be in today's public tour. | |
| 6181.html | 2016-09-23 13:50 | Activity: Update GDS software to gds-2.17.4-1 to fix CDS Bugzilla issues 812, 1036, 1041. | 30027 |
| 6180.html | 2016-09-23 10:31 | Activity: Pull chassis investigate and fix problem with switching segment 3 gains. | |
| 6179.html | 2016-09-22 11:23 | Activity: to fix fw instability, power cycle fw and solaris ldas-gw systems. Sequential restarts to preserve data. | |
| 6178.html | 2016-09-22 08:22 | Activity: Modify chassis to shape filter to better match needs. | |
| 6177.html | 2016-09-21 17:22 | Activity: Update tconvert leap seconds data file. Present version of tconvert will not do this automatically. A data file will be generated by a custom version of tconvert and copied to the proper location on the file server. | 30027 |
| 6176.html | 2016-09-21 13:31 | Activity: Connect RGA's at end stations to vacuum rack. A network and power cable will be pulled from BSC5/6 to the vacuum rack. | |
| 6175.html | 2016-09-21 12:46 | Activity: Per ECR1600260, Modify the ITM LV Low Noise Driver. A 4.7pF capacitor needs to be added to C25 (main board D1600122). Capacitor ensures stability of the HV stage, and precludes a ~175kHz oscillation. | 30000 |
| 6162.html | 2016-09-15 17:20 | Activity: upgrade h1hwsex computer from U10.04 to U14.04 | 30027 |
As this lock progressed, the 100 to 200 Hz region was improving and the range hit 80 Mpc right after 8:30 UTC. This is despite some huge noise in the 30 to 50 Hz region that came up at the same time. Around 8:49, the detector lost lock. The first attachment is the range, and the second shows the two parts of the spectrum going in opposite directions. It's not clear why the lock was lost, although the PI summary page shows something in the 3-8 Hz band growing exponentially (blue trace, third plot). Maybe it's this line at 4735 Hz (fourth plot)? It gets bigger and grows ugly sidebands as time goes on. I don't see it identified on the PI MEDM page though (I looked for 11649 and 28033 Hz as likely aliases). Edit: Actually, this could be the 10th order EY violin mode; see alog 19608. Is it possible something is going wrong with the damping and it's getting out of control? Or maybe it's nothing to worry about?
I was able to damp it with 0.1 Hz wide butterwoth +100dB filter and a gain of +0.001. And I was able to blow it up by flipping the sign of the gain. As Evan already mentioend in alog19612 this line indeed belongs to ETMY. Terra mentioned that this is not PI due to its non-exponential growth.
Title: 09/27/2016, Evening Shift 23:00 – 07:00 All times in UTC State of H1: Locked, commission Commissioning: 23:00 Daniel and Keita had the IFO when I came in, doing HP ISS measurments. 01:00 When they left, did initial alignment, then struggled with locking due to different guardians needing to be restarted following Beckhoff gateway(? I think) work earlier. OMC wouldn't lock because a switch at a rack had been flipped, HighZ something? Been locked since Stefan found and fixed that.
I did an upgrade of the following packages this morning:
The guardian and cdsutils packages are now using a new versioning scheme that is not tied to their SVN commit id. This was done in preparation of moving to git for upstream maintenance. The gpstime package is newly split out of cdsutils so that it maybe more more easily maintained and distributed.
Notable changes in these releases:
- The DCPDs whitening chassis was found to be switched to Low-Z - transitioned back to high-Z.
- We added notch filters for the 4.01Hz and5.01Hz SRM dither lines into the SRCLFF path. They are a Q of 60, 40dB deep. The high Q was picked to minimize the phase distorsion - it is actually slightly too high for the line width (which is driven by coupling fluctuation.) With this modification, the SRM alignment dither can be running in LOW_NOISE in case we need it.
- We also revived the pr2spotmove.py and pr3spotmove.py scripts (stored in /ligo/home/controls/sballmer/20160927/). They move the spot position on PR2 and PR3 respectively, leaving all other things equal. We were hoping to see a change in the auxiliary loop length noise - aspecially SRCL (they are higher than O1 - see Sheila's attachment.)
- First, we moved the pr2 spot position: (PR3 by 10urad in pit and 5urad in yaw). Next we did the same for the pr3 spot position (10urad on PRM). Unfortunately we did not see any change in the auxiliary loop noise.
Some additional notes:
Yes there is a big 1 Hz comb, presumably from the SRM dither lines. Attached plots are a 30 minute spectrum from 7:30 UTC, a little after going into undisturbed. The second plot is just zoomed in on a 10 Hz region to show the comb in more detail. There's some funny structure here that I'd like to understand. I'll also come back to this in a little while if the lock holds, to see if anything changes.
Thank you for catching this. Yes this would have been an inadvertent switch while working on the chassis. I thought I checked it. Must have looked before we were done working on it. SORRY.
The plot shows that the current vibration levels measured by the 6 accelerometers on the PSL table are about the same as on Boxing day during O1. I also checked several other O1 times, and the vibration levels were very similar to the Boxing day levels. With all of the recent work on the chiller and its lines, the recent levels, in contrast, have been changing quite a bit. This log: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29764, shows a level that was higher than O1 and, most interesting, a level on July 9 that was about a factor of 2 below O1. The vibration level is very sensitive to the flow rate and Jason thinks that the HPO crystal circuit flow rate might have been lower by a couple of l/m on July 9th than it is now (this circuit has the highest flow rate and is the predominant source of vibration, but doesn’t have a flow meter).
For O1 we set the flow rate as low as suggested by Ollie, but the actual flow rate turned out to have been higher than we thought, because of a poorly calibrated flow sensor. Thus it would not be surprising at all if we could beat the O1 vibration levels and still not be that close to the minimum flow limit - I think we should reduce the flow by at least a couple of l/m.
Another source of vibration is the flow fluctuations associated with bubbles in the system. Jeff bled quite a bit of air at the filters under the PSL table today. However, a measurement taken a couple of hours later after the fans were shut down did not show improvement in the table vibration. But I think that there is still a lot of air in the lines and repeated bleeding, along with reducing the air intake, might give us significant vibration improvement.
The observation that the vibration levels are about the same as in O1, and yet the peaks from the top mount on the periscope are about twice as large on IM4 trans diodes, suggests that the resonances of the mount are no longer as well tuned into a valley in the periscope peaks. So I think we could also get some improvement by re-tuning the resonances.
And it also seems that the jitter coupling is larger. Sheila’s thinks not by much more than two, but it seems worth expanding on Sheila’s piezo mirror injections with some table shaking injections.
Robert S., Jason O., Jeff B.
I restarted the calibration pipeline at Hanford at gps time 1159058357. Based on tests, the current latency is ~5-10 seconds. The filters file used can be found in the calibration SVN:
aligocalibration/trunk/Runs/PreER10/GDSFilters/H1GDS_ 1158989379.npz
See this aLOG for information on the filters:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=30013
The command line used to run the pipeline was gstlal_compute_strain
--data-source=lvshm
--shared-memory-partition=$LIGOSMPART
--filters-file=$filter_file
--ifo=H1
--frame-duration=4
--frames-per-file=1
--write-to-shm-partition=$HOFTSMPART
--compression-scheme=6
--compression-level=3
--control-sample-rate=4096
--expected-fcc=341.0 --partial-calibration
Note: the redundant pipeline on h1dmt2 was started at gps time 1159062755, a bit after the primary pipeline.
IO Chassis separate power for timing slave boards WP6193
Fil, Richard, Jim:
h1lsc0, h1iscex and h1iscey were powered down to re-install the separate DC power supplies for the IO Chassis timing slave assembly.
PSL-ISS model WP6185:
Daniel, Keita, Dave:
new h1psliss model was installed. DAQ was restarted. Only slow channels were added to the frame.
Frame Writer 10GE SFP-Fiber restoration WP6189
Jim:
Following the experiment to swap h1fw0 and h1fw1 10GE SFP, switch ports and fiberoptics cables to see if fw0's instability was hardware related, we undid this experiment and the systems are back to original configuration.
Deactivating internal EPICS gateways, configuring EDCU and Guardian to run without them WP6188:
Jamie, Jonathan, Dave, Jim:
We turned off the EPICS gateway h1slow-h1fe and restarted the DAQ. The EDCU did not connect to any Beckhoff channels. Jonathan made a /etc/init.d/daqd_dc0 file change to define the channel access environmental parameters and he got the EDCU to run without this gateway present. We then turned off the gateways h1aux-h1fe and h1cds-h1fe and did a final DAQ restart.
Jamie did the same on the Guardian system. We now have 6 epics gateways running (h0ve-cds, h1ve-h1fe, h1slow-cds, h1aux-cds, h0ve-cds, h1fe-cds)
h1oaf0 swap first ADC card in IO Chassis WP6191:
Nutsinee, Betsy, Jim, Dave:
Prior to powering down the IO Chassis, we powered down the 16bit DAC AI-chassis which drives the TCS chillers. Nutsinee and Betsy were on the Mech Room Mezanine and noted that the chiller did not trip, it just started driving the temperature downwards as expected. We kept the AI-chassis powered down while the water lines were being flushed.
We opened the h10af0 IO-Chassis and swapped the first ADC with its right-hand-side (as viewed from front) neighbor. This is an experiment to see if the DAC errors are related to ADC errors in this chassis.
Guardian new core code
Jamie:
new guardian core code installed today.
h1hwsex cloned from h1hwsey WP6162
Jonathan, Carlos:
h1hwsey's boot disk was downgraded to a smaller HDD, and then a clone of this was made for h1hwsex. Both systems are now running Ubuntu14 and are identical.
New GDS code release WP6181
Jim:
new GDS code release was installed.
tconvert leap seconds data file update WP6177
Jim:
In preparation for the end of year addition of a leap second to UTC, the tconvert data file was updated.
This information is based on the cumulative spectrum of recent lock stretches (9/18-9/26, computed from Fscan SFTs with spec_avg_long), plus Fscan magnetometer data.
0.997698 Hz
1.0 Hz
0.5 Hz odd harmonics (AKA 1 Hz with 0.5 Hz offset)
1.0 Hz with 0.998889 offset (new)
0.987987 Hz (new)
Attached plots:
Minor edit: offset on 4th comb mentioned has too many significant figures. Should be rounded to ~0.9989 (+/- 0.0005).
Update on the 0.996798 Hz: this comb appears in magnetometer channels between Feb 3rd and 4th*. After looking at the alogs for surrounding date range, Robert suspected the HWS as a potential source. We checked TCS-ITMX_HWS_RCXCLINKSWITCH and saw a switch on, which corresponded neatly with the date of the comb's appearance in magnetometer channels. Robert has also previously noted near-1Hz combs associated with the HWS in the CS. Richard says that the HWS can be placed on a separate power supply in the near future, which will hopefully clear this up.
*edit, typed wrong date
I removed the drivers S1600266 and 267, added capacitor 4.7pF in the C25 position and returned the drivers to service. HV was turned back on and confirmed. See Patrick's aLog entry.
Work Permit 6175
The PSL tripped around 21:37 UTC, it took us about an hour to recover from this . We called Jason and Peter, Jason called back and walked me through restarting it remotely. I added about 200mL of water to the chiller, there was water on the floor in front of the chiller. Jason confirmed that this is normal, and that the turbulence and bubbles in the chiller tube are from a known slow leak.
PSL tripped due to the power meter flow. The 1st attachment shows the power meter flow dropping just before the crystal chiller interlock trips. The 2nd attachment shows the power meter flow dropping ~1 second before the FE flow drops. Interestingly, it takes ~6 seconds after the power meter flow drops and the crystal chiller interlock trips for the crystal chiller flow to drop (3rd attachment). Not sure if this is due to the rate that the Beckhoff PC polls the crystal chiller, or if it really takes 6 seconds from interlock trip to chiller shutdown (I suspect the former). The full timeline of the trip, assuming t0 = 21:37:51 UTC (time of trip):
Filed FRS #6319.
Sheila, Keita
The 3rd whitening filter does not switch for AS_C segment 3. The readbacks look OK.
Last Tuesday we pulled the chassis and verified everything worked. This wee we pulled the chassis and verified everything worked. Finally while watching the binary switches we were able to trace it down to a pin in the cable pulling out of its socket. This was very troublesome because it would work as long as we had a breakout board inserted in between the cable and the chassis. Watching the signals while re-assembly took place we per able to narrow it down and found the problem. We have removed the back-shell of the connector and shoved the pin back in place. Next Tuesday we will crimp on a new pin. So we have not closed the work permit yet.
Work Permit 6180
This was due to a bug in the guardian MEDM states screen generation. Fixed in version 1.0.1 which was pushed yesterday.