This is per the Ops Daily Check Sheet for a task to be done by a Monday Operator.
Laser Status:
SysStat is good
Front End power is 30.93W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 20.0 days, 13.0 hr 48.0 minutes (should be days/weeks)
Reflected power is 1.751Watts and PowerSum = 24.67Watts.
FSS:
It has been locked for 0.0 days 1.0 h and 21.0 min (should be days/weeks)
TPD[V] = 1.429V (min 0.9V)
ISS:
The diffracted power is around 8.255% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 21.0 minutes ago (should be days/weeks)
Intentionally dropped from Observing Mode to address SR3 Cage Servo, but then accidentally had a lockloss while addressing this servo. Out of Observing for roughly 3hrs. Here's timeline:
For Initial Alignment noticed the ALS arms were off in pitch and ugly, but after taking care of them, only other noticeable variance was the Dark Michelson was also off in pitch. Restored this by looking at the AS Camera and the AS_C medm window's QPD, and steering the SR2 in pitch quite a bit (from 1938 to 2022); but this is what was needed to bring the spot back down in the center of the camera. From here on, it was normal work.
Locking was usual time sink when we have high useism (but it's dropping). During one drop, Guardian was in the middle of DC_READOUT_TRANSITION & giving OMC not locked error. I re-requested READY_FOR_HANDOFF, but H1 dropped. Think next time, we should select DC_READOUT_TRANSITION on the ISC_LOCK guardian (instead of having NOMINAL_LOW_NOISE), so that we don't fight the Guardian while locking the OMC.
At this point, we could have opted to try the new FULLY_ISOLATED state for the BS ISI, but we had been out of Observing for so long, that we decided to hold this off until Maintenance tomorrow.
H1 made it to NLN on the next lock. There were a few known SDF diffs: BS ISI time ramps (which I returned to 0sec), and the new filters Evan added for the test mases (I ACCEPTED these).
Over last 12hrs, useism has been trending down (and maybe flattening for the last 2hrs) and is at ~0.4um/s for the LVEA & winds are hovering below 10mph.
There are two quick tasks we want to do on the way to a full lock, isolate BS ISI ST2, and measure PRCL and MICH gains in DRMI.
Isolated BS ISI stage 2, (WP 5652). We did this because of the upconversion that Robert saw in DARM when shaking the BS ISI, 24194. We did this without any problems, then the IFO broke lock durring an epics freeze while locking the OMC, so we didn't get a chance to see if it had an impact on DARM.
I also grabbed some OLG measurements with DRMI locked on 1F, as described in WP5654. (motivated by alog 24136)
The PRCL gain isn't very different, the main difference is a boost that is on in DRMI but not PRMI. The MICH gain is about 10dB high though, resulting in a UGF just above 15 Hz for DRMI and around 5.5 Hz for PRMI. Maybe we should try lock DRMI with a lower MICH gain and see if it catches better.
Since we occasionally need to adjust the ALS comm offset in order to have IR resonating in the arms, I have (finally) added the comm offset slider to the ALS overview screen. Operators are familiar already with the diff offset slider that has been there, but now if comm has an issue, we no longer have to go hunting to remember which is the right slider.
Despite turning off all violin mode damping in full lock, some of the fundamental modes appear to not ring down over long locks.
There is a very slight drive to the PUM coils from the ASC signals, which I have now notched out at 500 Hz (using FM5 in the SUS-[E|I]TM[X|Y]-L2-LOCK_[P|Y] filter modules). It's not clear that this drive is the culprit, however.
TITLE: 12/14 EVE Shift: 00:00-08:00UTC (16:00-00:00PDT), all times posted in UTC
STATE of H1: 21+hr lock?
Outgoing Operator: TJ
Quick Summary:
H1 was doing fine, but TJ/Jenne mentioned a railed value for the SR3 Servo. So we wanted to proactively address this, whenever L1 dropped out of Observing. Doug L eventually called to let us know about them dropping out to do A2L work, and we took that opportunity to address the SR3 Cage Servo......unfortunately, when we changed an OFFSET value to -16k, we immediately dropped out of lock (I'm letting Jenne address specifics in her alog). Before dropping out of lock, Jenne loaded new code to the SR3 Cage Servo Guardian to be better at notifying us when we are railed.
OK, going to look at alignment now (because the ALS arms both look ugly in pitch).
Attached are 7-day pitch, yaw, and sum trends for all active H1 optical levers.
TITLE: 12/14 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 75Mpc for 20.5hrs
SUPPORT: Patrick, Vern, Jenne
SHIFT SUMMARY: Quiet except for an EX ISI oscillation, the blends were changed and seemed to have made it better. The blends for EX only are now in the Quite_90s.
INCOMING OPERATOR: Corey
ACTIVITY LOG:
WP--5649, ECR E1500456, II 1167. See aLog 23853 and SEI log 887 and 890.
In preparation for updating the ISI per above, I'm updating our local model and src areas:
hugh.radkins@opsws1:models 0$ svn st -u
? gnd_sts_library_NEW.mdl
* 12022 isi2stagemaster.mdl
Status against revision: 12272
hugh.radkins@opsws1:models 0$ svn up isi2stagemaster.mdl
U isi2stagemaster.mdl
Updated to revision 12272.
hugh.radkins@opsws1:src 0$ svn st -u
* 11156 WD_SATCOUNT.c
? ISIWD_GPS_veryOLD.c
* BLENDMASTER_ST1.c
* 10437 .
Status against revision: 12272
hugh.radkins@opsws1:src 0$ svn up
U WD_SATCOUNT.c
A BLENDMASTER_ST1.c
Updated to revision 12272.
hugh.radkins@opsws1:bscisi 0$ svn up
U ISI_CUST_CHAMBER_ST1_BLEND_ALL.adl
Updated to revision 12272.
hugh.radkins@opsws1:bscisi 0$
I've successfully rebuilt all the ISI platforms. Will wait until tomorrow for install & restart
FYI The SATCOUNT.c update was a bug fix which applies to all ISI platforms. This is why the HAMs were also rebuilt and restarted. This fix will have the ISI saturations (seen n watchdog screen) clear immediately. Added benefit for the BSC ISIs for not tripping those platforms from stale T240 saturations. All good.
The SR3 cage servo has gone beyond its limit, and is saturating the output of the SR3 M2 coils.
Back in November (aLog 23131) I put a check into the SR3_CAGE_SERVO guardian to let the operator know when it's close to the limit, and that it needed to be offloaded to the M1 actuators. However, I only put in a check for if the servo is getting close to the positive limit. As of 12 Dec around 18:00 UTC, we've been railed at the negative limit. Ooops.
Right now, SR3 is about 13 urad from the usual value in pitch (currently the OSEMs read 935, rather than the 922 that is the cage servo's setpoint), but the ASC seems to be fine with it.
At the next opportunity, we'll offload SR3's alignment, and fix the guardian code so that it checks for both the positive and negative limits.
In the attached figure, you can see that SR3 is no longer at its nominal pointing, and that the cage servo's setting is no longer within +/-15,000 counts (which corresponds to +/-130k counts at the DAC).
Trying to fix this broke the lock :(
Even though Corey and I typed a number (-16,000) into the SR3_M2_TEST_P_OFFSET (to get the cage servo closer to zero) that should have left the coil actuators railed, 2 of the coils seem to have flipped sign. This broke the lock. I'm still looking into why typing -16,000 didn't do what I expected.
I have been looking into the lockloss that I caused earlier when Corey and I were offloading the SR3 cage servo, because I don't think that it should have caused a lockloss.
I think that I have found a mysterious sign flip in the DAC output, which I'm hoping that someone will have an idea about. Somehow, the LR and UR DAC outputs switched sign a few days ago, even though the MASTER_OUTs did not flip sign. See first attachment for a 3 day trend showing this.
When Corey typed -16,000 into the SR3_M2_TEST_OFFSET, the signs of the LR and UR DAC outputs flipped to correct themselves. See second attachment for a 20 minute trend showing this. The COILOUTFs are the same as the MASTEROUTs, and they stay whatever sign they started (LL and LR positive, UL and UR negative). However, the DAC_OUTPUTs 6 and 7 (6=UR, 7=LR) flip sign. This undoes the mysterious sign flip that happened a few days ago.
Why is the DAC output flipping sign, when the digital signals going to them are not?!?!
Jim W., Evan H., Patrick T. The end X ISI was ringing up. I took the IFO out of observing while Jim W. switched the blends. Both DOFs were on 45mHz and were switched to Quite_90. I accepted the SDF differences and returned the IFO to observing. Screenshots of the accepted SDF differences and current blends are attached. Out of observing from 19:34:07 to 19:54:02 UTC.
We've been cleaning up the scripts used to present the various calibration measurements against the model and each other. A longer description of script locations and cleanup is LLO alog 23558. This is basically the same as LHO alog 21280, except comparing against the O1 model (H1DARMparams_1124827626.m) and trying to use more generic scripts for both sites. The attached pdfs are results, and can also be found in the aligo calibration svn under: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Results/ The first 3 all measurement types (Free swinging Michelson, ALS DIFF, PCAL) from 3 different days plotted against each other and the O1 model. The second set of 3 are just the PCAL measurements against the model.
The all-method comparison plots attached here have been renamed to /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Results/ 2015-12-11_H1SUSETMY_ActuationCoeffComp_EYL1_AllMethods.pdf 2015-12-11_H1SUSETMY_ActuationCoeffComp_EYL2_AllMethods.pdf 2015-12-11_H1SUSETMY_ActuationCoeffComp_EYL3_AllMethods.pdf in order to distinguish them from the L1 results, and committed to the repo. The PCAL-only results have been moved into the PCAL subdirectory and similarly renamed, /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Results/PCAL/ 2015-12-11_H1SUSETMY_ActuationCoeffComp_PCAL_EYL1_Residuals.pdf 2015-12-11_H1SUSETMY_ActuationCoeffComp_PCAL_EYL2_Residuals.pdf 2015-12-11_H1SUSETMY_ActuationCoeffComp_PCAL_EYL3_Residuals.pdf
Attached is a plot summarising the behaviour of the front end laser pump diodes. The oscillation in the output power has been observed before and I'm not sure what it is caused by. I had thought that it might be due to an oscillation in the PID loop controlling the pump diode temperature, however it's not visible in the TEC signals (D[1-4]TEC). We still have a reasonable amount of headroom for the pump diodes. At the moment the diode current is ~51.5 A. The maximum is 65 A.
O1 days 83-87. No restarts reported for this time period.
Planning for Tuesday Maintenance:
Activity Log: All Times in UTC (PT) 08:00 (00:00) Take over from Corey 08:46 (00:46) Reset L4C WD saturations on HAM3, ITMX, and the BS 16:00 (08:00) Turn over to TJ End of Shift Summary: Title: 12/14/2015, Owl Shift 08:00 – 16:00 (00:00 – 08:00) All times in UTC (PT) Support: Incoming Operator: TJ Shift Detail Summary: Locked in Observing mode for entire shift. Power at 21.8w, range 79Mpc. Wind is light breeze (4-7mph), seismic and microseism are improving. There is an oscillation showing up in the 20 minute ASAIR_B_RF90_I_NORM, and SUS-PRM_MC3_LOCK signals. There are also oscillations showing up in the Pitch and Yaw ASC Control strip tools.
TITLE: 12/14 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 80Mpc for 12.5hrs
OUTGOING OPERATOR: Jeff b
QUICK SUMMARY: Wind 10mph max, useism 0.6um/s, CW inj running, lights off, PSL 21.8W.
Performed a run spanning the time period from 1129593617 (Oct 23 2015 00:00:00 UTC) to 1131828544 (Nov 17 2015 20:48:47 UTC)
The following parameters were used for this run:
Only two injections were found to be scheduled during this period:
However, neither of these injections were found to actually occur within the examined time period.
There were no normal injections that occurred within the examined period. There were a number CAL-INJ resets, all single-IFO with the same curious pattern to the frame flags as denoted in the immediately prior HWInj report (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23485). There was a single-IFO burst injection in H1
BURST 1129673117.000 (H1)
with the only anomaly being that it is UNSCHEDULED. There was a single UNKNOWN injection in L1
UNKNOWN 1129673117.000 (L1)
This injection was confirmed to occur only within the CAL-INJ channel of RAW frames with the TRANSIENT bit indicating an injection but none of the specific injection-type bits, CBC, BURST, DETCHAR, or STOCHASTIC, indicating an injection. It was confirmed to not occur in ODC HOFT, ODC RDS, ODC RAW, or GDS HOFT frames.
It is interesting that the single-IFO injections BURST 1129673117.000 (H1) and UNKNOWN 1129673117.000 (L1) are found to occur at the exact same time but within different IFOs. Even further, the fact the BURST injection in H1 is perfectly normal, other than being UNSCHEDULED, while the UNKNOWN injection in L1, in addition to being UNSCHEDULED, is inconsistent. HWInjReport currently does not check the HW bit or the CW bit in the CAL-INJ channel for RAW frames for the presence of injections; however, upon examining these bits directly, it was found that while the HW bit indicated an injection, along with the TRANSIENT bit, the CW bit did not indicate an injection (my first thought was that this may be a CW injection only occurring in L1). This means this was a truly UNKNOWN, UNSCHEDULED injection occurring only with L1. Given both injections occur at the exact same time and the L1 injection is a truly UNKNOWN type (practically a ghost injection, by the bits), it is likely this was intended to be a normal network injection; however either user-error or a catastrophic software error corrupted the proper setting of the bits.
There was an additional UNKNOWN injection that occurred at 1131397646 in both H1 and L1 (as noted by Peter Shawhan) with a signature similar to the one reported here. However, for some reason HWInjReport missed it. I will have to investigate why HWInjReport missed that particular injection, but, unlike the last session of bug-fixing, I am not going to cease production of reports. It is my estimation that HWInjReport is currently operating with reasonable confidence to find and report most true anomalies. If reporting errors or omissions are found, then those errors should be correctable and a new report on the relevant time period can be generated.
I didn't see any follow-up on this but I did some checks: The injection at 1129673117 is the stochastic hardware injection that Joe and Chris did. This was out of science mode. The injection at 1131397646 should be the injection in the schedule file from line: 1131397639 1 1.0 coherentbbh0_1126259455_ This CBC injection was not logged properly for the searches because tinj was setting CAL-INJ_TINJ_TYPE instead of CAL-PINJX_TINJ_TYPE. So I would guess that was the problem that you're seeing here. The issues have since been fixed though.
Now that I finally have my diagnostic tool working (it's a ripped-down version of HWInjReport called HWInjCheck that simply returns raw output from FrBitmaskTransitions to allow rapid verification of anomalies found in HWInjReport), I re-examined the missing UNKNOWN injection at 1131397646. Firstly, HWInjReport can only identify injections as UNKNOWN if they occur in the CAL-INJ. This is because, as far as I can determine, only the CAL-INJ channel has an independent excitation bit that indicates whether a transient injection has occurred, regardless of type (as well as an independent HW bit indicating the occurrence of an injection of any type). This does not seem to be true for ODC-MASTER and GDS-CALIB channels. So, in CAL-INJ, if the TRANSIENT bit is "off" (indicating the presence of a transient injection) but the CBC, BURST, DETCHAR, and STOCHASTIC bits, which indicate the type of injection, are "on", then it is clear we have an UNKNOWN type transient injection occurring. However, for ODC-MASTER and GDS-CALIB, if the type bits are not "off", there is nothing to indicate excitation of a transient injection with an unspecified type. If 1131397646 occurs with an unspecified type in the ODC-MASTER or GDS-CALIB channel but not in the CAL-INJ channel, then HWInjReport will completely miss the injection as it won't show up at all in any of the bits and channels that HWInjReport checks.
Using HWInjCheck to check the output from FrBitmaskTransitions, I have not, yet, found any indication of the occurrence of 1131397646, that is, there is no indication of excitation in the excitation bits. I will check again more carefully, but, at this point, it does seem HWInjReport is working properly; this is an expected missing injection. To be clear, there may still have actually been an excitation, there just doesn't seem to be any record of it in the bits and channels that HWInjReport checks.