TJ was able to troubleshoot a few nuances from the slow locking this morning in order to bring the full lock up to NOMINAL_LOW_NOISE (more details from him later). After a Kiwamu DARM filter load broke the lock, Kissel and I took the opportunity to LOAD COEFFICIENTS on a few of the FE's that showed partial loads. In order to minimize the risk, we only performed the load coefficient on:
L1LSCAUX (benign LOCKIN and readback channel stuff)
H1TCS (Aiden filter tuning from yesterday)
H1SUSITMX and Y (Nutsinee DAMP filter tuning the other day)
H1OAF
Attached shows a picture of all FEs with the handful of other FM changes which were only locally loaded. (Local loading is ok, but it masks Dave's tool which reports dates of when full loads happen.)
LVEA: Laser Safe Observation Bit: Commissioning 06:50 Richard – To all buildings to check property 08:00 Reset TIM, IPC, and ADC errors 08:35 Earnest – From Caltech on site for property checks 09:15 Peter & Earnest – Going into H2 PSL and IOT2R table for property checks 09:20 Goodwill on site for a pickup – Bubba to escort 09:30 Richard – Going into the Optics lab to check on property 09:45 Filiberto – Cabling work in electronics room
Found the H1ASC EXC bit flag enabled this morning on the CDS overview. Since locking is slow, I went ahead and assumed this was left over from midnight work, I cleared them via
diag -l
tp clear 19 *
Attached is a 6 day trend of the H1IMC-PWR_IN showing how the rotation stage finds a variety of stopping places when executing the same request. This is a long-time known issue, so I'm just reposting to make the problem visible again. In theory the asc/lsc is supposed to compensate for power-in differences, but a it's hard to hand-wave off a ~25% difference from one down state to another - it makes troubleshooting slow acquisition more muddied than it already is.
Earnest from Caltech on site for property checks SEI: Prep for Thursday maintenance FMC: A wooden post near the chiller yard was knocked down. Bubba is repairing. Beam tube cleaning on the X-Arm continues Joint sealing continues on the Y-Arm Thursday Maintenance Activities: PSL - PMC alignment FSS – Electronics upgrade BSC-ISI – Payload watchdog fix PCAL – End-X clipping investigation LYLN – Driver commissioning Cosmic ray detector work HEPI – Accumulator fix End-Y
attached is the restart report for Tuesday maintenance. The frame writers appear to be more unstable following the fiber channel change.
CDS errors reset this morning: TIM - H1IOPSUSEY, H1SUSETMY ADC - H1IOPSUSEY, H1IOPSUSEX IPC - H1SUSETMY, H1SUSTMSY, H1IOPSEIEY, H1ISIETMY, H1IOPETMX DIAG RESET command cleared all errors.
Ed P. Mark L. 7/20/15- VERY HOT AGAIN TODAY The reduced size and somewhat new crew finished moving lights, generators and support equipment to the next section to begin cleaning. We also had some cord repair to do. We changed out the U-ground style cords on the vacuum machines for a twist lock style plug. Additionally we made some 250' long AWG #10 S. O. cords with twist lock ends and installed strain relief at the generator. 33.5 meters of tube cleaned ending 14.5 meters east of HSW-2-069. 7/21/15 Ed P. Mark L. The temperature has dropped slightly and is supposed to continue a cooling trend for the next couple of days, this makes being in the enclosure much more bearable. The crew cleaned 67 meter of tube today ending at HSW-2-065. We relocated the generator and cords. The new cords are working very well. Rodney is scheduled to return on 7/22/15. The crew will then be back to a full compliment of personnel.
A2L notes:
The new A2L values hadn't been saved to the safe snapshot, so when there were boots and burts earlier today, the values were lost. This is why we re-ran A2L for the test masses at high power tonight. Those values were accepted in the SDF system. This should be done every time the script is run. Since I don't know if it's possible to do this in a scripted way, we'll have to do this accepting by hand for now.
In addition, the script a2l.py was modified to be able to handle the BS in addition to the test masses. The only change for this was an "if" statement at the beginning to define the drivealign channel names (they're in L2 for the quads, but M2 for the BS). We found that excitation amplitudes of 300 cts for pitch and 150 cts for yaw give us an SNR of about 100 in OMC_DC, so those have been added to the GetOptList.py dictionary-creator. Sheila tells me that the BS M2 is roughly 3 times stronger than the quads' L2 stages, and these excitation amplitudes are about 1/3 the quads' values, so that all makes sense.
The A2L is finishing up right now for BS, so in the next few minutes we'll have all 4 test masses and the BS all decoupled via Hang's new script. EDIT: Pitch worked, but Yaw isn't finding a happy place, so I'll have Hang look at this with me tomorrow.
I also added some lines in the "disable dither" function of a2l.py (which runs if you ctrl-c out, or when the measurements are finished) to reset the CLKGAIN excitations to zero for both the ASC oscillator and the LSC demodulators. The script already zeroed the output matrices, so the signals weren't going anywhere, but it made me unhappy to have mystery oscillators running unnecessarily.
I tried to get an OLTF of dHard pitch at 24 W, but somehow I rang up a 1.0 Hz instability and had to abort and bring the power down to 3 W. Strangely, even at 3 W this instability took a minute or so to ring down. I tried lowering the master ASC gain to 0.8 temporarily, but this did not make the ringing go away (so it is probably not a loop instability).
At 3 W I was able to get a preliminary measurement of the dHard pitch loop down to 1 Hz. More refinement required.
lowered DHARD_Y gain from 14 to 7, DSOFT_Y from 0.2 to 0.1, and updated the output matrix.
J. Kissel We restarted the h1hwinj1 and h1hwinj2 injection machines today to check whether they were contributing to the recent EPICs freeze problems we've been seeing on control room work stations. The reboot did NOT help the problem. However, since they've now been restored, I figured I'd try my luck at using Eric's instructions on how to restart them (LHO aLOG 18831, T1400349). While I was able to log on to h1hwinj, find the appropriate directory, and start_psinject, after the 60 sec start up time, I did NOT see the injection start (looking at the excitation channel in the H1:CAL-INJ_CW_EXCMON or H1:CAL-INJ_HARDWARD_OUTMON), nor does the CAL-CS GDS_TP sceen indicate that there has been a test point selected, even though the interaction with start_psinject was as follows: -bash-4.1$ ./bin/start_psinject Currently, we are running on: h1hwinj1.cds.ligo-wa.caltech.edu Result of checking injection status: H1 injection is NOT RUNNING Automatic restart should occur within 10 minutes; no human action is required H1 injection will be started with configuration ER7 if you proceed For logging purposes, please enter your name (or Ctrl-C to cancel) Jeffrey Kissel Type a BRIEF explanation (a few words) for why start_psinject is being executed Hardware Injection Machines were restarted today to see if they were affecting EPICs slowdown seen in the control room. See LHO aLOG 19809. Starting H1 injection into H1:CAL-INJ_CW_EXC on h1hwinj1.cds.ligo-wa.caltech.edu using ER7 exec /data/scirun/O1/HardwareInjection/Details/bin/exec_psinject_from_script /data/scirun/O1/HardwareInjection/Details/pulsar/ER7 /ligo/apps/sl6/gds-2.16.12 H1 H1:CAL-INJ_CW_EXC 1121573769 Injection will start at t=1121573769 (60 seconds from now)... Injection starting now -bash-4.1$ Also -- while the hardware injection machines were down, there was NO indication on the injection overview screen that the hardware injection machines were down / dead. In fact, since the EPICs records that *are* shown on the screen are all defined and stored in the front-end, they had been left in the ALL GREEN state making it deceivingly look like all is well. There should be a look-alive heartbeat EPICs variable sent from these machines, or at least a static alarm bit somewhere to indicate that these injection machines are down or up.
J. Kissel, B. Weaver, N. Kijbunchoo, T. Sadecki, E. Hall, S. Dwyer Another heavily loaded Tuesday today. For what we planned to do, see LHO aLOG 19770. We have managed to complete the IFO recovery, on the same day, and before the sun has gone down!! Most of the failures and deviations from plan today were software problems, which didn't fix the problem they'd intended to fix, were slow to recovery, or had to be reverted because of new found bugs. Towards the tail end of recovery, there were other problems that were more related to half-finished or mistakes from commissioning last night, but thankfully by the time we had reached recovery the guilty commissioned had arrived on site and confessed their sins. In summary -- not so bad -- but there was little IFO systems' software actually restarted. (All times PDT) 07:00 Richard brings SEI platforms and SUS to SAFE/OFFLINE LHO aLOG 19778 07:30 Richard, Daniel and Fil swap the internal GPS of the timing master for the external GPS unit. LHO aLOG 19782 This did *not* crash any front-ends, so the corner station models have stayed up all day, and were otherwise unaffected by the rest of maintenance. This was a huge relief. 08:00 Richard and Fil continue moving GPS Antennae on roof, no further impact on the timing system Hugh and Jim W begin ramping down HEPI for pump maintenance Jim B heads down to EY to begin timing fanout replacement and SUS fast front-end BIOS upgrade 09:15 Begin corner station SEI / SUS and IMC recovery A little bit of trouble bringing the IMC up because the WFS output had not been offloaded in a while, so with the initial alignment the WFS trigger PD had a little bit too low of light. We've since offloaded the IMC WFS DC values to the alignment sliders to alleviate this, LHO aLOG 19814 HEPI HAM2 trips during recovery, HAM5 has he usual problems with Rogue Excitation vs. Watchdog RESET, but otherwise a smooth restart of the remaining SEI and SUS. Vinny starts PEM sensor calibrations. 09:30 IMC recovery complete Jim's finished with EY Timing fanout and SUS fast FE, find that the BIOS upgrade makes no difference to ADC Timing Errors, but moves on to EX. Cable runs for PEM cosmic ray detector begin. LHO aLOG 19804 10:00 Betsy and Jeff begin EY SUS ETMY's RCG upgrade to get all improvements to TrueRMS and SDF sort-on-substring, discover that branch 2.9 (which has all improvements to TrueRMS) has some terrible BURT/SDF restore bug. End result is to compile against RCG 2.9.5 LHO aLOG 19793 Hugh and Jim finish repair of HEPI pump station, and it's brought back in loop. LHO aLOG 19794 10:15 TCSX Laser trips from PEM cable pulling, recovered shortly after by Nutisnee. Ha! We knew to expect this one! LHO aLOG 19786 10:30 Huh discovers EY HEPI fluid level is significantly lower than last week, suspects accumulator bladder has let in fluid. Executive decision is to wait until next week to assess and repair. LHO aLOG 19796. EY HEPI Trips Fil and Andres head to EX to being LVLN ESD driver install and cable pulling. The new ETMX ESD LVLN Driver was literally, merely installed in the rack, but has not been hooked up. LHO aLOG 19803 11:00 Discover that EY SUS has IOP DAC output issues after finished of recompilation against RCG 2.9.5, requires full front-end shut down and restart (including TMSX and new EXPI model) LHO aLOG 19793 Jim B. is done with BIOS upgrade for EX SUS fast-front end. Also no improvement in timing errors. 11:15 EY front-end model recovery complete EY SEI / SUS restoration begins SEI system up and running, SUS restared to damping and aligned, charge measurements begun to assess health. 11:20 EX front end model restoration begins, encountering the same problems with no IOP DAC outputs. Restart entire front-end, like EY. Begin EX SEI / SUS restoration begins Charge measurements on ETMY ESD complete, confirm charge is OK and little no change in charge since yesterday LHO aLOG 19764 EY recovery complete. 11:40 ISC EX and ISC EY front-end epics settings restored to 05:00 SUSAUX model updates installed, having been compiled against RCG 2.9.5 LHO aLOG 19780 Preventative maintenance reboots begin Conlog computer restarted, no problems LHO aLOG 19789 12:00 MX / MY PEM model changes installed LHO aLOG 19809 12:10 DAQ / Frame Builder restart, doesn't come back gracefully, needs a lot of prodding from Jim and Dave. 12:20 Frontend network switch restarted, does not fix EPICs slowdown issue EPICs Gateway restarted, does not fix EPICs slowdown issue Hardware injection machines turned OFF, does not fix EPICs slowdown issue 12:45 Guardian machine restarted. Doesn't come back up on first try, does better the second time LHO aLOG 19812 Attempt to begin ETMX charge measurements to confirm health, but EX Beckhoff PLCs crash, not necessarily unexpectedly. After restoration, settings are restored to 05:00 Beckhoff crash trips ETMX ESD HV Driver power supplies, and renders Beckhoff remote restart useless. 13:00 Fil drives to end station to turn ESD power supplies back ON. 13:15 Charge measurements on ETMX ESD begin, only one data point, but confirms EX ESD is functional EY recovery complete Begin Full IFO Recovery 13:45 Initial alignment complete for green and PRM. recovery up to this point was MUCH more smooth this week 14:00 Discover that certain visiting commissioners have removed vital MICH and SRCL lock acquisition filters last night. Filters restored from archive, and we move on. 14:20 Initial alignment complete for IFO Begin lock acquisition sequence 14:50 Reach DC Readout, increase to high power, but lose lock instantly upon transition from ETMX to ETMY as DARM actuator An ETMY ISI trip, later associated with IPC communication problems because of ADC timing issues from SUS front-end. LHO aLOG 19808, LHO aLOG 19799 several more lock-losses, a few on transition to TR CARM, a few more on transition to ETMY 16:45 Discover that certain visiting commissioners have left the ETMY hierarchical control filters in a half-commissioned state last night. Filters restored from archive. 17:00 Frame builder crashes again. LHO aLOG 19811. Evan notices that Trans QPD dark noise for X arm is huge. While the IFO continues to lose lock on TR CARM transition, Evan and Shiela investigate what possible QPD offset settings might have been lost. 18:00 Shiela just gives up, assumes that the dark noise of the QPDs has actually changed (perhaps because of ETMX electronics work earlier in the day) 19:40 Full IFO recovered to a stable 60 [Mpc]!
J. Kissel, K. Izumi, J. Driggers This morning, we had a little trouble relocking the IMC because the initial alignment of the IMC SUS causes the light on the IMC WFS trigger PD to be just a hair too low. To solve the problem this morning, Jenne temporary decreased the trigger threshold (H1:IMC-IMC_TRIGGER_THRESH_ON) from 40 to 15 [ct], such that the IMC WFS engaged, which steered the IMC to our known good alignment (and then restored the trigger threshold once the WFS had steered the optics back to a good enough alignment that the normal trigger threshold was easily surpassed). To avoid this in the future, we said "we should offload the DC value of IMC WFS control signal to the MC SUS alignment offsets," but at the time didn't know that was the right script / button / thing to call in order to do so since there have been many in the past which had suffered from bit rot. Just now, in between lock attempts, I've asked Kiwamu what the right script is, he should me, and I ran it. This changed the alignment offsets as follows: Before After H1:SUS-MC1_M1_OPTICALIGN_P_OFFSET 1207.50 1208.3750756649936 H1:SUS-MC1_M1_OPTICALIGN_Y_OFFSET -2025.30 -2024.9455029701996 H1:SUS-MC2_M1_OPTICALIGN_P_OFFSET 527.00 525.7250127471385 H1:SUS-MC2_M1_OPTICALIGN_Y_OFFSET -534.03 -535.7085973743652 H1:SUS-MC3_M1_OPTICALIGN_P_OFFSET -722.14 -721.3522559898023 H1:SUS-MC3_M1_OPTICALIGN_Y_OFFSET -2034.10 -2034.4394052459747 I've spoken with Kiwamu about the seeminly-now-prolific issue of using double-precision python scripts to produce calculated EPICs settings to ridiculous precision, and he agreed to fix it in due time. The script that does the offloading is launched from the IMC WFS MASTER screen, from the bright blue button marked "! OFFLOAD WFS" just below the output filters in the middle-bottom right. See attached screenshot.
This alog doesn't really state it, but I assume the PZT is off loaded as well. I also assume each TM is off loaded individually.
Daniel,
Travis, Nutsinee, Jeff K., Betsy, Ed, Cheryl, Hugh, Jim, Stefan, Sheila, Elli, pretty much everyone in the control room!
This log is meant for fellow operators, but commissioners might find it useful as well.
- We started relocking in the afternoon. Locking green went well. There was an argument about whether to request LOCKED NO SLOW or LOCKED WITH SLOW FEEDBACK (as stated in the Initial Alignment Wiki). I went for LOCKED NO SLOW because I knew it worked two Saturdays ago but I didn't know when was the last time wiki got updated (and I also don't know what LOCKED WITH SLOW FEEDBACK was). It is a good idea for the person who's maintaining the Ops wiki to put in the date and the content that was being updated (this was already brought up by TJ, but I just wanted to emphasize how important it is if we are going to use the wiki as the reference so we know we can trust it).
- Everything went well up until MICH DARK LOCKED. The counts on the BS output were awfully high and the AS port wouldn't go dark. The solution to this in a normal situation would be to request DOWN and wait until the beam splitter becomes settle. However, Stefan found that there was a change in the LSC MICH filter and the Guardian tried to turn on the filter that no longer existed (Guradian turning on a blank filter wasn't a problem, the problem was a filter was supposed to be there). He reverted the filter to the way it was the night before. Then everything worked again. I wasn't sure if this difference showed up in the SDF or not.
- Mode Hopping! Soon as you approach DRMI LOCKED, get the IFO_ALIGN window ready. You will have to touch the beam splitter if the RF18 (pink) is low. You have to act quick! Or the interferometer will give up on you. Use AS port camera as a reference for pitch/yaw and watch "pink" and "purple" to see if you go the right way.
- We lost lock at LOW NOISE ESD ETMY. After that we had difficulty relocking ETMY green. Power dropped after WFS was engaged. Shortly after Travis found that the TMSY slide bar was off from Kissel's screenshot in the morning (or last night?). After we put TMSY yaw back to where it was and the arm was able to lock in green. Not only the screenshot never hurt, but it has also proven to be helpful!
- We also switched ISI Windy Blend filters on for ALL Quads because the wind was reaching 35 mph (and we thought might have lost lock because of that). However, TURNING ON WINDY BLENDS TRIPPED BOTH ETMX AND ETMY ISI. Jeff demands all ISI filters are set the same while Sheila thinks it's okay to leave the Windy blend at the corner station on. Personally I would turn on the corner station windy blends iff I'm desperate. Since the wind wasn't too bad we switched the filters back to "Quite" (I'm sure it should spelled "Quiet"). This configuration worked fine the night before.
- Then we lost lock at LOW NOISE ESD agian. At least we got all the way there right after maintenance period!
- Ed was updating the Wiki during the relocking. It should be as up-to-date as it possibly can right now.
LOCKED NO SLOW: no tidal relief
LOCKED WITH SLOW FEEDBACK: tidal is feed to the ETM (will keep the green laser frequency constant)
It makes little difference, if you only spend a small amount of time in this mode. On the other hand, I am not aware of any reason why LOCKED WITH SLOW FEEDBACK wouldn't be the better option under any circumstances.
RickS, Sudarshan, Darkhan
We've adjusted PCALX, PCALY and ESD calibration line drive levels to give a signal to noise ratio of about 100 (for 8 s FFT) over the mean science mode spectra from ER7.
The following lines have been adjusted:
Line | Frequency (Hz) | Old drive level (ct) | New drive level (ct) |
PCALX | 33.1 | 38 | 700 |
PCALX | 534.7 | 6000 | 19300 |
PCALY | 36.7 | 40 | 320 |
PCALY | 540.7 | 5000 | 9900 |
ESD | 34.7 | 0.50 | 0.22 |
ESD | 538.1 | 0.20 | 2.40 |
I added pcal lines at 325.1 Hz, 3000 ct for X and 331.9 Hz, 1500 ct for Y. These are maybe not optimal because they are in the region of the PSL piezo mirror mount, but I wanted to stay close to the DARM pole and out of the bottom of the bucket.
Based on noise levels around calibration lines during ER7, to get SNR of 100 (with 8 s FFT) , drive levels adjustments needed for LLO calibration lines:
I inserted a nototch filter for each of the two lines that Evan added last night. Because of the limitation in the number of the IIR coefficients, I had to put one in suscomp (in FM7) and the other in ResG (in FM3). The previous notch that we put for the cavity pole tracker (alog 18401) was removed from FM7 since we have not been using it.