Add 125ml water to crystal chiller. Diode chiller level is OK.
Title: 12/02/2015, Evening Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT) State of H1: 08:00 (16:00), The IFO locked at NOMINAL_LOW_NOISE, 22.2w, 81Mpc. Outgoing Operator: TJ Quick Summary: IFO locked in Observing mode for the past 15 hours, with exceptions for commissioning mode as noted in the aLOG. Environmental conditions are good – wind is calm (0-3mph), seismic activity quiet.
Title: 12/2 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observing at 77Mpc for 16hr. Bubba still on tractor clearing snow.
Shift Summary: Boring shift (a good thing!) except for the snow.
Incoming Operator: Jeff B
Activity Log:
At 15:06 UTC Bubba told me that he was headed out to clear the parking lots of snow. Since then there has been about a 5Mpc drop in range. Coincidence?
Bubba finished using the tractor for snow removal at 17:10 (09:100
Attaching a plot that compares the DARM spectrum during Tractor activity (15:40UTC) and an earlier quiet time from the same lock (8:00UTC). It clearly shows excess noise from ~80-190Hz.
LLO lost lock, so at 12:16 UTC I brought the IFO out of Observing, cleared the timing error on H1SUSETMY, and then went back into Observing.
Observing 78 Mpc for 12 hours. Snowing.
There is a little over an inch of snow on top of a bit ice from freezing rain on site. If you are coming on site this morning please drive safe and slow. I'm not sure if Route 10 has been plowed, but I have seen a few cars heading to Hanford.
Title: 12/2 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observing at 80 Mpc for 8.5hours
Outgoing Operator: Travis
Quick Summary: Lights off, winds calm, useism 0.3um/s, CW inj running.
Title: 12/1 Eve Shift 0:00-8:00 UTC (16:00-24:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: Another very quiet shift. Only 4 ETMy saturations. ~7 hours of coincident observing time with LLO.
Incoming operator: TJ
Activity log: None
Since JeffK and Nutsinee finished their measurments, we have been in Observing.
J. Kissel I've taken new DARM Open Loop Gain and PCAL2DARM transfer functions for the bi-weekly(-ish) full-frequency-span confirmation of the time variation in calibration / IFO parameters. As a by-product of these two transfer functions, we're able to confirm that the front-end, CAL-CS, DELTAL_EXTERNAL calibration (and therefore the low-latency GDS pipeline output) is still within the desired precision. I attach screenshots of the measurements. They will be processed in greater detail tomorrow, once we have the evening's kappa values (i.e. what the calibration lines are telling us the systemaic errors are in the model parameters). Stay tuned, but things continue to look exquisite. Great job CAL team! DARM OLGTF Template: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/DARMOLGTFs/2015-12-01_H1_DARM_OLGTF_7to1200Hz.xml Corresponding Exports: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/DARMOLGTFs/ 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_coh.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_tf.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_coh.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_tf.txt PCAL2DARM TF Template: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/PCAL/2015-12-01_PCALY2DARMTF_7to1200Hz.xml Corresponding Exports: 2015-12-01_PCALY2DARMTF_7to1200Hz_A_PCALRX_B_DARMIN1_coh.txt 2015-12-01_PCALY2DARMTF_7to1200Hz_A_PCALRX_B_DARMIN1_tf.txt
The relevant kappas for this measuremnt are loctaed here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Results/CAL_PARAM/2015-12-02_LHOalog_23883_Kappas.mat
Back to Observing after Maintenance Day.
Jeff B, Ed, Kiwamu,
Today we have changed two parameters in the OMC_LOCK guardian as follows.
The guardian codes are reloaded and checked into the SVN. We tested once and worked fine.
We had an issue with the OMC_LOCK guardian where it did not succeed in finding a carrier resonance. Although this may sound familiar to some of us (for example, alog 23543, alog 23823), the symptom was different this time. The upper and lower 45 MHz rf sidebands had different peak power by approximately 11%. As the OMC_LOCK guardian stringently checks whether the power ratio is within [+25%, -10%], it was unfortunately exceeding the lower threshold and hence the failures. I have no idea why the power ratio now exceeds the power threshold. I briefly checked the TCSs but none of them seemed suspicious. Furthermore, since the interferometer was able to fully lock with the RF readout at a lower power, the change we are trying to identify may be a subtle thing.
After all, we have relaxed the lower threshold to -25% so that it matches the upper one. Also, during the investigation, we have reverted the start voltage for scanning OMC back to -24 V which had been -30 V in the past few days (alog 23823). This change might be unnecessary, but since it works fine, we will leave it as it is.
Today we energized the new ion pump mounted on beam tube port X2-8. This pump is isolated from the beam tube for now. A permanent, dedicated mounting setup and location for the controller in the X-end building is still being sorted but for now (to test functionality) we have it at a temporary site in the mechanical room of the X-end station. -> The indicated current/pressure on the controller agrees within 50% of that indicated by the independent pressure gauge which is mounted on the pump port hardware. We will make an attempt to measure any HV cable leakage current (cable is ~1,000 feet long!) and perform a few tests to convince ourselves that the pump is actually pumping etc... as opportunities arise.
Activity Log: All Times in UTC (PT) 15:55 (07:55) Jeff B. – In the LVEA to take 3IFO storage temp/humidity data 16:00 (08:00) Peter – Into the PSL for alignment work (WP #5626) 16:08 (08:08) Take over from TJ 16:15 (08:15) Gerardo – Going to End-Y to pull high voltage cable for Ion pump at Y2-8 16:20 (08:20) Lockloss – Jeff K. running charge measurements at End-X 16:21 (08:21) Jeff K – Charge measurements on ETM-X 16:24 (08:24) Filiberto – Going to End-Y for cabling work (WP #5630) 16:26 (08:26) Bubba – Going down Y-Arm to survey for rock delivery 16:50 (08:50) John & Nutsinee – Going to TCS-X table to check view ports 16:50 (08:50) American Rock – Rock delivery along Y-Arm 16:55 (08:55) Joe – Going into the LVEA 16:58 (08:58) Patrick – Starting on software side of WP #5630 17:00 (09:00) Jodi & Mitch – Going to End-Y to start garb room change (WP #5628) 17:08 (09:08) Bubba & Betsy – Going into LVEA to move Quad boxes – Using crane 17:18 (09:18) N2 delivery to End-X 18:35 (10:35) Filiberto – Finished at End-Y – Going to End-X (WP #5630) 18:41 (10:41) Joe – Going into LVEA to unplug battery chargers 18:42 (10:42) John & Nutsinee – Out of the LVEA 18:45 (10:45) Jeff K. – Going to End-Y to reset railed ESD 18:50 (10:50) Joe – Going to End-Y to check batteries 19:10 (11:10) Nutsinee – Going into the LVEA to work on HWS alignment 19:12 (11:12) Gerardo & Kyle – Back from X-Arm 19:15 (11:15) Contractor on site to meet with Bubba 19:24 (11:24) Richard and Ken out to End-X to check on some ION pump controls 19:28 (11:28) Joe is back from End-Y 19:43 (11:43) Dave & Patrick – adding FMCS temperature channels (WP 35631) 20:05 (12:05) Locked IMC at the request and approval of Kissel and Landry. 20:07 (12:07) Kyle and Gerardo back from End-X. He says Filiberto is still there, though. 20:28 (12:28) Tried relocking – Found initial alignment off 20:40 (12:40) Redo initial alignment 21:34 (13:34) FD on site 21:53 (13:53) Kyle & Gerardo – Going to X2-8 22:27 (14:27) John & Bubba – Going to Mid-X 22:37 (14:37) Kyle & Gerardo – Back from X-Arm 22:50 (14:50) Start relocking 23:49 (15:49) Locked at NOMINAL_LOW_NOISE 23:50 (15:50) In commissioning mode so Kissel can run measurements 00:07 (16:07) Turn over to Travis End of Shift Summary: Title: 12/01/2015, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT) Support: Kiwamu Incoming Operator: Travis Shift Detail Summary: 16:00 (08:00) Start Tuesday maintenance window. After maintenance window tried to relock. Green alignment bad. Realigned green and tried relocking. Found BS alignment was off. Did full initial alignment and tried to relock. Had a problem with OMC not finding carrier. Kiwamu found lower limit of carrier detection was too tight. He changed it in Guardian from 10% to 25% (which matches the upper limit). Continued with relocking and reached NOMINAL_LOW_NOISE. IFO is in observing mode while Kissel runs measurements.
These activities were recorded by Corey. He was backing me up while the I reconfigured the End-Y garbing room. 17:00 Nutsinee & John went out to LVEA (did not notify control room, doing TCS Work?) 17:14 Saw rock truck exiting gate 17:48 Hugh going to EX/EY to check HEPI fluid levels (~30min) 17:49 Ken going to EY to investigate noisy emergency light 18:00 Ion Pump high power cable (Gerardo & kyle) 18:04 Jodi & Mitch returning from EY 18:06 Joe out from LVEA (charging batteries and checking eye wash stations)....will go back out to unplug chargers later 18:07 Beckhoff work (Patrick) 18:13 Data from dessicant cabinets (Bartlett) 18:16 forklifting SUS boxes from lvea to staging (Bubba, Betsy) 30min
John, Richard, Nutsinee
Today's plan was to go out to TCSX table, take out the old IR sensor, and replace it with the new one (+VP inspection). John wanted to make sure that the sensor actually does what it was designed to do before we unwrap the light pipe and put it in so we did some tests. The comparator box was thought to be calibrated to trip at ~45 degree C. However, I forgot to write down how I did it so the number was very shady... We waved a small soldering iron in front of it, starting at low temperature (~150 deg F). The box would trip when the soldering iron was waved at half a centimeter away from the sensor. Keep in mind that the sensor is mounted at ~2.5 inches away from the center of the viewport with a small angle (the sensor is looking straight down while the optical window is about 0.6 cm deep under the viewport cover. So, the sensor must be able to trip when seeing a heat source at 2.5" away with ~22 deg viewing angle. We did a quick test by waving the soldering iron in front of the sensor with several temperature settings at several distances (At this point the potentiometer was about a quarter turn away from the room temperature tripping point. It's a 10-turn pot!). 800 deg F setting tripped the box at ~3" away, 600 deg F and 500 deg F tripped the box when waved at ~2" away. But these temperatures are impractical settings. At 800 deg F, 3" away we were able to make the box trip when placed the soldering iron rounghly 1cm above the sensor's viewing axis. This gives as a viewing angle of ~8deg, about a factor of 3 less than what's required. We did the same test with the sensor that was mounted on the TCSY viewport. The results were similar (400F, 500F soldering iron didn't trip the sensor when places at the center of the viewport but tripped the sensor when moved to the edge of the viewport. 600F at the center of the viewport did trip the sensor.)
To confirm the temperature of the soldering iron we used to do this test, we used Richard's FLIR camera and a thermocouple. Turns out the actual temperature of a certain spot on the soldering iron we pointed the IR sensor to was lower than the display. The thermocouple measured 304 F when display reads 350F, 630 F when reads 700F, and 750F when reads 850F.
Conclusion: The sensor unit isn't doing what's it supposed to do and we should rethink about installing them without an improvement to the comparator box circuit. Note that we were only using half a turn out of 10 turns available on the pot to set up this test. There's no way to fine tune this thing to where we want it to trip!
That's very helpful, thank you. Can you please link here the design documents describing this system's design and operation? This should describe the proper way to test and adjust it.
FMCS EPICS channels added
WP5631. John, Bubba, Patrick, Dave
The FMCS to EPICS system was modified to add the corner station control average temperature channel. Both the raw channel (in degF) and the degC channels were added to the IOC. The new channel names are:
H0:FMC-LVEA_CONTROL_AVTEMP_DEGF
H0:FMC-LVEA_CONTROL_AVTEMP_DEGC
These channels were added to the DAQ for trending by modifying H0EDCU_FMCS.ini
Beckhoff Vacuum Gauge channels added to EX and EY.
Patrick, Dave:
Please see Patrick's alog for full details. After the Beckhoff changes were made, the ini files were generated and copied into the DAQ chans area.
DAQ restart
Jim, Dave:
To apply the Beckhoff and FMCS changes the daq was restarted at 12:32PST. This was a very messy start and took about 25 minutes to get the DAQ back. The main issue was the monit system on h1dc0 was trying too rapidy to start the data concentrator, resulting in duplicate ini and par files in the running configuration directory. In the end we started the data concentrator manually to get the system running. We will investigate the monit settings offline.
DIAG_MAIN guardian has code change, found error and fixed it
Jeff, Dave
After the long h1nds0 downtime, Jeff was getting the guardian nodes which use NDS back by reloading them. The DIAG_MAIN node would not run, giving an error saying float and int datatypes cannot be bitwise-anded. I found the error in DIAG_MAIN.py and fixed it by casting the ezca to an INT before bitwise AND. We tested the new code by stopping and restarting the CW hw-inj. The operator saw the message on Guardian MEDM and the verbal system announced the error every second.
h1lsc stuck testpoint cleared
Dave:
we had a stuck testpoint on h1lsc for some while, I cleared them.
Test change of filter module on h1pemmx
Dave:
To test the robot updates of filter files to daqsvn, I made a blank-line change to H1PEMMX.txt and loaded it on the front end.
due to the extended DAQ downtime, there are no H1 DAQ frames between the GPS times:
1133036992 (Dec 01 2015 20:29:35 UTC)
1133038720 (Dec 01 2015 20:58:23 UTC)
These times are those of the last frame file before the outage and the first file after recovery.
In the course of the FMCS change it came to light that the svn checkout that the IOC target directory updates from is not up to date. svn checkout: /ligo/apps/linux-x86_64/epics-3.14.12.2_long_h0/iocs/fmcs target directory: /ligo/lho/h0/target/h0fmcs I made the change to the db file in the svn checkout, ran make, and updated the target directory from it. I then stopped the IOC and restarted it from the updated target. However, when I later tried to check in the modified db file, it would not accept it, because the svn checkout was not up to date, and the directory structure has changed in svn. So I copied the db file out of the way and ran svn update. I then realized that the svn checkout of the device support directory is also not up to date. At this point I am holding off on making further changes. So as it stands: The FMCS IOC target directory has been updated and the code is running out of it, but the directory it updates from is no longer compatible with it and the change to the db file is not in svn. I think I need to pause and consider how best to proceed.
The FMCS IOC was burtrestored to 11:00 AM (local time).
I have modified the FMCS Cornerstation oveview MEDM screen to show the new Control Average temperature (both degF and degC). The new chans are located in the top right corner of the screen.
Last Tuesday (24th Nov) Jim and I modified the monit on h1hwinj1 machine such that when it restarts the psinject process it smoothly ramps the excitation amplitude over a time period of 10 seconds. We manually started the new system on Tuesday and since then there have been no crashes of psinject until the last 24 hours. There have been 4 stops (with subsequent automatic restarts) in the past 24 hours, each stop was logged as being due to the error:
SIStrAppend() error adding data to stream: Block time is already past
Here are the start and crash times (all times PST). Monit automatic restarts are maked with an asterix
time of start | time of crash |
Tue 11/24 14:55:47 | Sun 11/29 17:15:56 |
Sun 11/29 17:16:00* | Mon 11/30 00:00:14 |
Mon 11/30 00:01:13* | Mon 11/30 13:09:07 |
Mon 11/30 13:09:36* | Mon 11/30 13:12:43 |
Mon 11:30 13:13:39* | still running |
Adding INJ tag so the hardware injection team sees this.
Using a script written to flag CW injection transitions (start or stop) from psinject log files on h1hwinj, I found all of the transitions below to occur during observing mode. I'll ask Laura Nuttall to insert these in the database. 1132905630 1132905633 Nov 30 2015 08:00:13 UTC 1132905689 1132905692 Nov 30 2015 08:01:12 UTC 1132952963 1132952966 Nov 30 2015 21:09:06 UTC 1132952992 1132952995 Nov 30 2015 21:09:35 UTC 1132953179 1132953182 Nov 30 2015 21:12:42 UTC 1132953235 1132953238 Nov 30 2015 21:13:38 UTC Peter Shawhan reported via e-mail that these restarts happened sporadically in iLIGO days and may be related to a busy network. Presumably, these restarts were not disruptive in those days because they were not injected via a time-domain inverse actuation filter. Looking at the first of the crashes above (see attached time-domain zooms of the CW injection channel, the IAF-filtered version of it and DARM), I'm surprised that the effect in DARM isn't larger.