O1 day 75
model restarts logged for Tue 01/Dec/2015
2015_12_01 12:32 h1dc0
2015_12_01 12:33 h1dc0
2015_12_01 12:34 h1broadcast0
2015_12_01 12:34 h1dc0
2015_12_01 12:34 h1nds0
2015_12_01 12:34 h1nds1
2015_12_01 12:34 h1tw0
2015_12_01 12:34 h1tw1
2015_12_01 12:37 h1dc0
2015_12_01 12:44 h1nds0
2015_12_01 12:46 h1dc0
2015_12_01 12:47 h1dc0
2015_12_01 12:51 h1dc0
2015_12_01 12:56 h1dc0
2015_12_01 12:58 h1nds0
2015_12_01 13:00 h1broadcast0
2015_12_01 13:00 h1nds1
2015_12_01 13:00 h1tw0
2015_12_01 13:00 h1tw1
Maintenence day. Beckhoff and FMCS changes. Initial DAQ restart failed due to monit respawning too rapidly and over-populating the running configuration directory.
O1 day 74
model restarts logged for Mon 30/Nov/2015 No restarts reported
Add 125ml water to crystal chiller. Diode chiller level is OK.

Title: 12/02/2015, Evening Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT) State of H1: 08:00 (16:00), The IFO locked at NOMINAL_LOW_NOISE, 22.2w, 81Mpc. Outgoing Operator: TJ Quick Summary: IFO locked in Observing mode for the past 15 hours, with exceptions for commissioning mode as noted in the aLOG. Environmental conditions are good – wind is calm (0-3mph), seismic activity quiet.
Title: 12/2 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observing at 77Mpc for 16hr. Bubba still on tractor clearing snow.
Shift Summary: Boring shift (a good thing!) except for the snow.
Incoming Operator: Jeff B
Activity Log:
At 15:06 UTC Bubba told me that he was headed out to clear the parking lots of snow. Since then there has been about a 5Mpc drop in range. Coincidence?
Bubba finished using the tractor for snow removal at 17:10 (09:100
Attaching a plot that compares the DARM spectrum during Tractor activity (15:40UTC) and an earlier quiet time from the same lock (8:00UTC). It clearly shows excess noise from ~80-190Hz.
LLO lost lock, so at 12:16 UTC I brought the IFO out of Observing, cleared the timing error on H1SUSETMY, and then went back into Observing.
Observing 78 Mpc for 12 hours. Snowing.
There is a little over an inch of snow on top of a bit ice from freezing rain on site. If you are coming on site this morning please drive safe and slow. I'm not sure if Route 10 has been plowed, but I have seen a few cars heading to Hanford.
Title: 12/2 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observing at 80 Mpc for 8.5hours
Outgoing Operator: Travis
Quick Summary: Lights off, winds calm, useism 0.3um/s, CW inj running.
Title: 12/1 Eve Shift 0:00-8:00 UTC (16:00-24:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: Another very quiet shift. Only 4 ETMy saturations. ~7 hours of coincident observing time with LLO.
Incoming operator: TJ
Activity log: None
Since JeffK and Nutsinee finished their measurments, we have been in Observing.
J. Kissel I've taken new DARM Open Loop Gain and PCAL2DARM transfer functions for the bi-weekly(-ish) full-frequency-span confirmation of the time variation in calibration / IFO parameters. As a by-product of these two transfer functions, we're able to confirm that the front-end, CAL-CS, DELTAL_EXTERNAL calibration (and therefore the low-latency GDS pipeline output) is still within the desired precision. I attach screenshots of the measurements. They will be processed in greater detail tomorrow, once we have the evening's kappa values (i.e. what the calibration lines are telling us the systemaic errors are in the model parameters). Stay tuned, but things continue to look exquisite. Great job CAL team! DARM OLGTF Template: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/DARMOLGTFs/2015-12-01_H1_DARM_OLGTF_7to1200Hz.xml Corresponding Exports: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/DARMOLGTFs/ 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_coh.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_tf.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_coh.txt 2015-12-01_H1_DARM_OLGTF_7to1200Hz_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_tf.txt PCAL2DARM TF Template: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/PCAL/2015-12-01_PCALY2DARMTF_7to1200Hz.xml Corresponding Exports: 2015-12-01_PCALY2DARMTF_7to1200Hz_A_PCALRX_B_DARMIN1_coh.txt 2015-12-01_PCALY2DARMTF_7to1200Hz_A_PCALRX_B_DARMIN1_tf.txt
The relevant kappas for this measuremnt are loctaed here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Results/CAL_PARAM/2015-12-02_LHOalog_23883_Kappas.mat
Back to Observing after Maintenance Day.
Activity Log: All Times in UTC (PT) 15:55 (07:55) Jeff B. – In the LVEA to take 3IFO storage temp/humidity data 16:00 (08:00) Peter – Into the PSL for alignment work (WP #5626) 16:08 (08:08) Take over from TJ 16:15 (08:15) Gerardo – Going to End-Y to pull high voltage cable for Ion pump at Y2-8 16:20 (08:20) Lockloss – Jeff K. running charge measurements at End-X 16:21 (08:21) Jeff K – Charge measurements on ETM-X 16:24 (08:24) Filiberto – Going to End-Y for cabling work (WP #5630) 16:26 (08:26) Bubba – Going down Y-Arm to survey for rock delivery 16:50 (08:50) John & Nutsinee – Going to TCS-X table to check view ports 16:50 (08:50) American Rock – Rock delivery along Y-Arm 16:55 (08:55) Joe – Going into the LVEA 16:58 (08:58) Patrick – Starting on software side of WP #5630 17:00 (09:00) Jodi & Mitch – Going to End-Y to start garb room change (WP #5628) 17:08 (09:08) Bubba & Betsy – Going into LVEA to move Quad boxes – Using crane 17:18 (09:18) N2 delivery to End-X 18:35 (10:35) Filiberto – Finished at End-Y – Going to End-X (WP #5630) 18:41 (10:41) Joe – Going into LVEA to unplug battery chargers 18:42 (10:42) John & Nutsinee – Out of the LVEA 18:45 (10:45) Jeff K. – Going to End-Y to reset railed ESD 18:50 (10:50) Joe – Going to End-Y to check batteries 19:10 (11:10) Nutsinee – Going into the LVEA to work on HWS alignment 19:12 (11:12) Gerardo & Kyle – Back from X-Arm 19:15 (11:15) Contractor on site to meet with Bubba 19:24 (11:24) Richard and Ken out to End-X to check on some ION pump controls 19:28 (11:28) Joe is back from End-Y 19:43 (11:43) Dave & Patrick – adding FMCS temperature channels (WP 35631) 20:05 (12:05) Locked IMC at the request and approval of Kissel and Landry. 20:07 (12:07) Kyle and Gerardo back from End-X. He says Filiberto is still there, though. 20:28 (12:28) Tried relocking – Found initial alignment off 20:40 (12:40) Redo initial alignment 21:34 (13:34) FD on site 21:53 (13:53) Kyle & Gerardo – Going to X2-8 22:27 (14:27) John & Bubba – Going to Mid-X 22:37 (14:37) Kyle & Gerardo – Back from X-Arm 22:50 (14:50) Start relocking 23:49 (15:49) Locked at NOMINAL_LOW_NOISE 23:50 (15:50) In commissioning mode so Kissel can run measurements 00:07 (16:07) Turn over to Travis End of Shift Summary: Title: 12/01/2015, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT) Support: Kiwamu Incoming Operator: Travis Shift Detail Summary: 16:00 (08:00) Start Tuesday maintenance window. After maintenance window tried to relock. Green alignment bad. Realigned green and tried relocking. Found BS alignment was off. Did full initial alignment and tried to relock. Had a problem with OMC not finding carrier. Kiwamu found lower limit of carrier detection was too tight. He changed it in Guardian from 10% to 25% (which matches the upper limit). Continued with relocking and reached NOMINAL_LOW_NOISE. IFO is in observing mode while Kissel runs measurements.
These activities were recorded by Corey. He was backing me up while the I reconfigured the End-Y garbing room. 17:00 Nutsinee & John went out to LVEA (did not notify control room, doing TCS Work?) 17:14 Saw rock truck exiting gate 17:48 Hugh going to EX/EY to check HEPI fluid levels (~30min) 17:49 Ken going to EY to investigate noisy emergency light 18:00 Ion Pump high power cable (Gerardo & kyle) 18:04 Jodi & Mitch returning from EY 18:06 Joe out from LVEA (charging batteries and checking eye wash stations)....will go back out to unplug chargers later 18:07 Beckhoff work (Patrick) 18:13 Data from dessicant cabinets (Bartlett) 18:16 forklifting SUS boxes from lvea to staging (Bubba, Betsy) 30min
John, Richard, Nutsinee
Today's plan was to go out to TCSX table, take out the old IR sensor, and replace it with the new one (+VP inspection). John wanted to make sure that the sensor actually does what it was designed to do before we unwrap the light pipe and put it in so we did some tests. The comparator box was thought to be calibrated to trip at ~45 degree C. However, I forgot to write down how I did it so the number was very shady... We waved a small soldering iron in front of it, starting at low temperature (~150 deg F). The box would trip when the soldering iron was waved at half a centimeter away from the sensor. Keep in mind that the sensor is mounted at ~2.5 inches away from the center of the viewport with a small angle (the sensor is looking straight down while the optical window is about 0.6 cm deep under the viewport cover. So, the sensor must be able to trip when seeing a heat source at 2.5" away with ~22 deg viewing angle. We did a quick test by waving the soldering iron in front of the sensor with several temperature settings at several distances (At this point the potentiometer was about a quarter turn away from the room temperature tripping point. It's a 10-turn pot!). 800 deg F setting tripped the box at ~3" away, 600 deg F and 500 deg F tripped the box when waved at ~2" away. But these temperatures are impractical settings. At 800 deg F, 3" away we were able to make the box trip when placed the soldering iron rounghly 1cm above the sensor's viewing axis. This gives as a viewing angle of ~8deg, about a factor of 3 less than what's required. We did the same test with the sensor that was mounted on the TCSY viewport. The results were similar (400F, 500F soldering iron didn't trip the sensor when places at the center of the viewport but tripped the sensor when moved to the edge of the viewport. 600F at the center of the viewport did trip the sensor.)
To confirm the temperature of the soldering iron we used to do this test, we used Richard's FLIR camera and a thermocouple. Turns out the actual temperature of a certain spot on the soldering iron we pointed the IR sensor to was lower than the display. The thermocouple measured 304 F when display reads 350F, 630 F when reads 700F, and 750F when reads 850F.
Conclusion: The sensor unit isn't doing what's it supposed to do and we should rethink about installing them without an improvement to the comparator box circuit. Note that we were only using half a turn out of 10 turns available on the pot to set up this test. There's no way to fine tune this thing to where we want it to trip!
That's very helpful, thank you. Can you please link here the design documents describing this system's design and operation? This should describe the proper way to test and adjust it.
Last Tuesday (24th Nov) Jim and I modified the monit on h1hwinj1 machine such that when it restarts the psinject process it smoothly ramps the excitation amplitude over a time period of 10 seconds. We manually started the new system on Tuesday and since then there have been no crashes of psinject until the last 24 hours. There have been 4 stops (with subsequent automatic restarts) in the past 24 hours, each stop was logged as being due to the error:
SIStrAppend() error adding data to stream: Block time is already past
Here are the start and crash times (all times PST). Monit automatic restarts are maked with an asterix
| time of start | time of crash |
| Tue 11/24 14:55:47 | Sun 11/29 17:15:56 |
| Sun 11/29 17:16:00* | Mon 11/30 00:00:14 |
| Mon 11/30 00:01:13* | Mon 11/30 13:09:07 |
| Mon 11/30 13:09:36* | Mon 11/30 13:12:43 |
| Mon 11:30 13:13:39* | still running |
Adding INJ tag so the hardware injection team sees this.
Using a script written to flag CW injection transitions (start or stop) from psinject log files on h1hwinj, I found all of the transitions below to occur during observing mode. I'll ask Laura Nuttall to insert these in the database. 1132905630 1132905633 Nov 30 2015 08:00:13 UTC 1132905689 1132905692 Nov 30 2015 08:01:12 UTC 1132952963 1132952966 Nov 30 2015 21:09:06 UTC 1132952992 1132952995 Nov 30 2015 21:09:35 UTC 1132953179 1132953182 Nov 30 2015 21:12:42 UTC 1132953235 1132953238 Nov 30 2015 21:13:38 UTC Peter Shawhan reported via e-mail that these restarts happened sporadically in iLIGO days and may be related to a busy network. Presumably, these restarts were not disruptive in those days because they were not injected via a time-domain inverse actuation filter. Looking at the first of the crashes above (see attached time-domain zooms of the CW injection channel, the IAF-filtered version of it and DARM), I'm surprised that the effect in DARM isn't larger.