The production DMT computer, h1dmt0, was hung. (I could not do an "ls" on the DMT monitor logs directory, and SenseMonitor_H1 and H1_llhoft frames had stopped showing up. This was perhaps related to the mornings DAQ reboot, as the SenseMonitor_H1 frames stopped showing up shortly after this. The H1_llhoft frames stopped around the time of the morning update of gds on h1dmt3, though supposedly this should not have caused any problems on h1dmt0.)
After a discussion with Dave Barker, a reboot of h1dmt0 seems to have fixed things.
TITLE: 07/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
Stopped at DC_READOUT_TRANSITION while Jeff K. works on damping the ETMX violin mode at 507.194 Hz.
TITLE: 07/11 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
Maintenance Day
LOG:
15:00 Bubba and Nicole into LVEA
15:12 Thomas and Marc to MY
15:17 Chris is with pest control people. They're going into LVEA first and then down arms.
15:23 Richard to Ends to restore ESD drivers back to normal power supplies.
15:25 Pep and Kentaro to EY: investigating line noise snd possible bird pecking
15:27 Hugh into LVEA to re-attach X Cryo accelerometer.
15:30 Marc and TVo back
15:33 Coca-cola on site
15:44 Chris and PC heading down Y arm.
15:55 Cheryl into optics lab and then squeezer bay.
15:58 rebooting nuc2&3 as per Carlos's request.
16:10 Bubba and Nicole out of LVEA. Going to both ends and both mids.
16:15 Chris and PC down X arm
16:17 Kentoro back from EY
16:35 Richard back
16:48 LN2 delivery
16:49 HFD on site to flow hydrants
16:50 Pep back into control room
16:51 Pep to EX
17:21 Cintas on site
17:23 Kiwamu says he will restart the ASC model
17:25 DAQ restart
17:32 Fil and Elizabeth going to EX for Beckhoff power
17:32 Jeff B called from PSL he's going to be cycling dust monitors
17:46 Richard informed me that the HFD is gone and the fire pumps are shut off.
17:49 Corey into squeezer bay
17:54 Corey out
17:55 Karen to EY
18:14 Jeff B out of PSL and heading to end stations to cycle dust monitors as a quarterly FAMIS task
18:28 Pest control guy leaving.
18:40 Bubba and Nicole back
18:53 Kiwamu and Nicole out to LVEA
19:03 Karen leaving EY
19:13 Nicole and Kiwamu out.
19:15 Jeff B back and into Optics lab.
19:21 Fil and Elizabeth are back
19:23 Rick, et al, are leaving EX. He reported lights and wifi turned off.
19:32 Jeff out of Optics lab and done with dust monitor checks.
19:40 restarted Alarm Handler and Verbal Alarms
19:45 Peter called. He's done in the PSL and waiting on Cheryl to finish up.
20:01 Jason to LVEA to turn out lights in H2 enclosure.
20:25 Jason tweaking PMC
20:30 Begin initial alignment
20:31 Sensor correction turned back on to WINDY state.
20:40 Jason done with PMC alignment.
locking
DQ shifter: Pep Covas (pep.covas@ligo.org)
Full summary here: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20170706
Greg is looking into problems with h1dmt0, which is causing non-reporting of H1's range.
2394 channels added and 596 channels removed. H1:ASC and H1:CDS-RACCESS channels.
Zero Count and Flow Test all installed dust monitors. All Zero Counts were were good. I adjusted the flow rate on the LVEA DM#10 (Biergarten) from 2.6l/m to 2.8l/m. All other flow rates were good.
I ran through the check list, except for the PSL environment settings because Jason and Peter were in the changing area and said that they had returned PSL settings. I didn't find any issues, Jason unplugged wifi. Seems clear.
As part of Line hunting we had put the ESD electronics on their own power supplies. WP 7044. I have now restored the system back to nominal state with the power supplies in the racks. Closed the work permit. https://services.ligo-la.caltech.edu/LHO/workpermits/view.php?permit_id=7044 https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=37165 https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=37028
Also after this morning's PSL work, I tweaked the beam alignment into the PMC (the ISS was turned OFF). With the ISS turned back ON, the PMC is now transmitting 57.7 W and reflecting 15.8 W. A screenshot of the PMC MEDM screen is attached for future reference. This completes LHO WP 7072.
After this morning's work in the PSL, I completed the weekly FAMIS tasks.
HPO Pump Diode Current Adjust (FAMIS 8430)
No adjustment was made as I could not get the laser power to increase. I also attempted to adjust the operating temperatures of the DBs, and that also did not increase the laser output power. Therefore the operating temperatures and currents of the HPO DBs remains unchanged from last week. A screenshot of the PSL Beckhoff main screen is attached for future reference. This completes FAMIS 8430.
PSL Power Watchdog Reset (FAMIS 3658)
The power watchdogs were reset during this morning's PSL work. This completes FAMIS 3658.
Per JimW's suggestion, I had a look at our beam spot positions on the test masses for now (last A2L measurement from this morning, before maintenence) versus a time on 17 June 2017 (16 June local) right after Sheila moved spots to drastically improve our range.
In the attached plot, the green squares are the old good times, and red circles are the latest measurement. Listed in the title for each optic is the total distance between the old and current spots. You can see that ETMY is basically in exactly the same place, both ITMs have moved a little bit, but ETMX has moved a lot.
So, one thing we should try today is moving the POP QPD and SOFT offsets such that we get closer to these old spots, and see if that improves our range.
Is it not possible that the change in IMC alignment / spot positions after the Montana EQ (see LHO aLOG 37363) has also decreased (or at least changed) the PSL jitter suppression, and therefore also showing up as poor range? If so, we should try moving the IMC to its pre-EQ position as as well.
I have updated GDS to version 2.18.0 (along with nds2 and gstlal packages dependent on gds) on the DMT test computer, h1dmt3, for further testing. This should have no impact on the control room or production running of the DMT.
The time of the 3 slow control machines had been off sync for some time, variation fluctuate from 1 sec to 15+ seconds between them, due to that problem I research and downloaded a third party software called NetTime, this tool allow us to resync all windows clock simultaneously with our local NTP server.
yesterday's change which unmonitored the CW GAIN and TRAMP channels was removed, we are once again monitoring these channels.
WP7066 Kiwamu, Dave:
a new h1asc model was installed, replacing RF90 with RF72. The DAQ was restarted soon afterwards.
DAQ Changes:
Fast Channels Added (2048Hz)
+: fast channel H1:ASC-AS_A_RF72_I_PIT_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_A_RF72_I_SUM_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_A_RF72_I_YAW_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_A_RF72_Q_PIT_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_A_RF72_Q_SUM_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_A_RF72_Q_YAW_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_I_PIT_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_I_SUM_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_I_YAW_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_Q_PIT_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_Q_SUM_OUT_DQ added to the DAQ
+: fast channel H1:ASC-AS_B_RF72_Q_YAW_OUT_DQ added to the DAQ
Fast Channels Removed (2048Hz)
-: fast channel H1:ASC-AS_A_RF90_PIT_OUT_DQ removed from DAQ
-: fast channel H1:ASC-AS_A_RF90_YAW_OUT_DQ removed from DAQ
-: fast channel H1:ASC-AS_B_RF90_PIT_OUT_DQ removed from DAQ
-: fast channel H1:ASC-AS_B_RF90_YAW_OUT_DQ removed from DAQ
Slow Channel Stats: 1060 channels Added, 356 channels Removed
DIAG_MAIN guardian code changed
The RF90 channels on the front end h1asc were renamed to RF72 this morning, but their Beckhoff equivalent channels retained their RF90 naming. This upset DIAG_MAIN, which checks for setting equivalency between Beckhoff whitening and h1asc anti-whitening. The code loops through channels assuming an identical core name.
I modified sys/h1/guardian/DIAG_MAIN.py to remove ASC-AS_A_RF90 from the generic loop. I added a new tuple to cover the ASC exception that RF72_AWHITEN_SET on the front end maps to RF90_WHITEN_FILTER in Beckhoff land.
Here is the SVN differences for the changes I made:
Index: DIAG_MAIN.py
===================================================================
--- DIAG_MAIN.py (revision 15816)
+++ DIAG_MAIN.py (working copy)
@@ -859,10 +859,10 @@
'ASC-POP_X_RF',
'ASC-AS_A_RF45',
'ASC-AS_A_RF36',
- 'ASC-AS_A_RF90',
+ #'ASC-AS_A_RF90', DB LHO 11july2017
'ASC-AS_B_RF45',
'ASC-AS_B_RF36',
- 'ASC-AS_B_RF90',
+ #'ASC-AS_B_RF90', DB LHO 11july2017
'ASC-POP_A',
'ASC-POP_B',
'ASC-OMC_A',
@@ -904,8 +904,10 @@
'ASC-Y_TR_B',
]
+ # DB LHO 11july2017: h1asc model changed RF90 to RF72, Beckhoff retained RF90 names. Make exception for AS_[A,B]_RF90/72 mapping
+ rf72_tups = [('ASC-AS_{}_RF90_WHITEN_FILTER_{}'.format(ab,i), 'ASC-AS_{}_RF72_AWHITEN_SET{}'.format(ab,i)) for ab in ['A', 'B'] for i in range(1,4)]
omc_tups = [('OMC-DCPD_A_WHITEN_SET_{}'.format(i),'OMC-DCPD_B_WHITEN_SET_{}'.format(i)) for i in range(1,4)]
- all_dem_tups = [('{}_WHITEN_FILTER_{}'.format(pre,i), '{}_AWHITEN_SET{}'.format(pre,i)) for pre in prefix for i in range(1,4)] + omc_tups
+ all_dem_tups = [('{}_WHITEN_FILTER_{}'.format(pre,i), '{}_AWHITEN_SET{}'.format(pre,i)) for pre in prefix for i in range(1,4)] + omc_tups + rf72_tups
for x in all_dem_tups:
if ezca[x[0]] != ezca[x[1]]:
I have made a number of medm screens for this new system. The attached are screenshots of a couple of the new medms. When we locked the interferometer with a low power (~2W) after the maintenance today, I briefly tried phase-locking the audio modulation signal at 205 Hz, but didn't really succeed. We will continue commissioning them later.
Additionally, I have edited the ISC_DRMI guardian so that it doesn't look at ASAIR_B_RF90 any more when checking whether the ASCs needs to be slowly engaged or not. With this modification in it, we succeeded in locking the interferometer multiple times today. So it seems functioning as expected. After the test today, we restored the RF hardware back to nominal -- no 72 MHz source and very small 118 MHz modulation.
CW GAIN issue has been resolved. The CW_GAIN and CW_TRAMP values were being monitored by Guardian, which forced the IFO out of observing when those values changed; those values were changed due to a loss of hardware injections and subsequent restart by the psinject script. Dave Barker and I have changed the gain and ramp to be unmonitored channels again. Notes attached.
Change undone after the decision was made that losing IFO lock was the desired behavior and issue was mitigated by sparse nature of psinject reset. See aLOG 37447.
Greg rebooted h1dmt0 and restarted the monitors. H1 range is now updating.