During the daily run call today, Adam Mullavey mentioned that the hardware injection team is planning to do a series of coherent hardware injections over the weekend.  The software which selects event candidates and generates EM alerts, "Approval Processor", had been configured to treat hardware injections like real triggers, but that would've caused problems for the hardware injection tests.  So this evening (at 21:00 PST) I have changed the configuration of Approval Processor to make it ignore hardware injections.

This configuration change has been documented in the Bursts/EMFollow wiki at https://wiki.ligo.org/Bursts/EMFollow/SoftwareRecord .

This afternoon Annamaria and I looked again at changing alignments.  

We tried again moving the PR3 spot position back towards where I had it last night, and saw that this still improves the noise in DARM and the range although the power recycling gain drops to almost 29.  

We then tried exploring alingment of IM3, and moved it by 2000urad in pitch.  We saw the power recycling gain increase from about 32 to 32.3 at best, this alingment was with the PR3 spot in the bad jitter noise location and the IM3 pitch alignment slider at 27940.  We were able to make the jitter noise worse by moving IM3 in yaw, but not to make it better. The attached screenshot shows the extremes of the noise in the bucket while we were moving things around.  

We reverted the IM3 alignment and I was going back to the lower noise PR3 spot position when I moved to quickly and broke the lock.  

Keita, Sheila

At 11 am pacific time (19:00 UTC Nov 22nd), there was a sudden shift in alignment in the SRC that caused bad noise in DARM. The first attached screenshot shows the noise in DARM changing.

At this time, there is a shift in the alingment of SRM and SR2 in both pit and yaw, and other interferometer optics don't move significantly.  The OMs and OMC suspensions move as well, but that appears to be a result of the feedback. There is also an increase in POP90, a small drop in AS_A DC, and a small increase in AS90.  The signals on all 4 quadrants of ASA36 I change (the in loop signal, where we are using a funny matrix to take care of an offset).  On AS36 B, we see pitch signals, but not much signal in yaw. Also, the RF levels on all quadrants of AS36 WFS change by 5 to 8dB, which could be the result of the alignment shift.  

We looked at conlog for any changes at the time, there were only PI settings changing, we don't think there was anyone in the LVEA or CER at the time.  

The most obvious sign for an operator that this has happened would be that POP90 jumps suddenly in the middle of a lock stretch.  If this happens again, people could (go out of observe) open the SRC1 ASC loops by ramping the gains to zero, then try moving SRM to bring POP90 back to a normal level.  

We have known for some time that the indicated level of CP4's LN2 fluctuates too rapidly to be physically likely and is more probably due to some transient GN2 phenomenon which effects the validity of the differential transducer.  Occasionally this indicated level will deviate enough as to enter into the alarm region and will then get the attention of the CR operator (Corey just asked about this).  I guess we'll need to investigate this more seriously.  Chandra is aware of this issue but is gone until next week.  

The Vacuum Team gets notified via automated text messages of these alarms.  

Until this gets resolved, CR operators only need contact the Vacuum Team Members if CP4 stays in alarm for a prolonged period (say hours). 

(see WP #6332)

Did a few "on/off" states for Robert S. so as to compare spectra while IFO in low noise.  He OK'd the continued running of these pumps during the intended bake-out of pressure gauge PT180 located on BSC8's dome.  As is, the apparatus is ready to go with the exception of the gauge electronics which remain connected and energized.  I am ignorant as to the impact, if any, of removing this while it is energized.  I've been told that this gauge has an in-vacuum PCB that limits the bake temperature to 80C at the CFF and I can just see myself going through the bake exercise only to have to vent and replace the gauge because I had "popped" the internal electronics by removing the external electronics while energized!  Spoke with Patrick T. who is only familiar with the Beckhoff end of things, left msg with Richard M. who might know more pertaining to the hardware.  Mfgr. tech support is not available via phone during off hours etc....  

As such, I am leaving PT180 exposed to the site vacuum volume (YBM) and the pumps etc. running.  The locally mounted turbo pump is isolated from PT180 via a closed metal valve and its exhaust valve is open to the supporting pumps located on the LVEA floor.  I will attempt to start the actual bake-out on Friday - hopefully by then I will have determined how to de-energize the gauge or discounted the concern of removing the external electronics while energized.  

Until we put in a more permanent solution next week, over the holiday weekend here are the instructions to allow the operator to disallow hardware injections from running, say in the event of a GRB.

From the SITEMAP MEDM, on the X-Arm line, select the CAL pull down and press the PINJX button

You will see the CAL_INJ_CONTROL MEDM for the x-arm.

The part we are interested in is the Transient signals portion in the centre of the screen. Open the  filter modlue screen by pressing the TRANSIENT related display button.

The filtermodule can be in one of two states depending upon whether a transient hardware injection is currently runng (not a high probability occurance).

If an injection is not running (the most probable scenario), the GAIN will be 0.000. In this case, turn OFF the output ON/OFF button to the right of the LIMiT switch.

If an injection is running the GAIN will be 1.000, set the GAIN to 0.000 (it will ramp down over 2 seconds) and then turn OFF the ON/OFF button.

At this point wait for the ring-down period of time (nominally one hour).

To permit hardware injections, turn ON the ON/OFF button (in the unlikely event an injection is running, either wait for it to finish or set the GAIN to 0.000 again)

TITLE: 11/24 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 24mph Gusts, 21mph 5min avg
    Primary useism: 0.15 μm/s
    Secondary useism: 0.51 μm/s
QUICK SUMMARY:

• Came in to a NLN H1 with PEM Injectors (Anamaria/Robert) working with Kyle (who was on the floor at BSC8's VAC pumps).  They then handed over to Sheila...
• Sheila working on H1. 
• BRSx was in a rung up state & we are letting it ring down.  So, will leave SEI_CONFIG in WINDY_NO_BRSX state until Jim looks at it (tomorrow?).
• Will return to PEM Injection work after they have "lunch".

Robert, Anamaria

We did a test of the RGA at the X end station. Its peaks seem to be at around 79.5 Hz and 158.7 Hz. We were not able to see a change in DARM BUT the 80ish Hz bump was not there to begin with. We did blast a line at that frequency and saw no coupling to DARM. The 79.5 Hz peak lines up suspiciously with the center of the usual bump. Quite unlikely, but to be sure that's not it we could still do an off test when the peak is present in DARM.

Last night we did a similar test in the corner station. We didn't have sensors super close so the peaks are not as obvious but they seem to be at 75 and 158.5 Hz. This also had no effect on 80 Hz bump.

16:40: Krishna out to Biergarten

16:43 Karen into H2 building

17:03 Kyle back from EY

17:07 Green arms aligned: recentered X arm as per oplevs and started from there. Got a good alignment but still not maximum power. Got closer to maximum by tweaking TMS a µrad or two.

17:17 re-started VERBAL ALARMS

17:21 Karen out of H2 building

17:30 Keits to end stations to check on  unsed ADC channels

17:45 IA complete. Beginning lock sequence

17:46 Bubba taveling to ends and mids. X -arm first.

18:12 Bubba done with x-arm and heading to Y mid. He'll stand down from EY visit due to current locking.

18:16 Lock loss at close beam diverters. HAM6 ISI tripped.

18:30 Keita called. He's finished at EX and headed to EY

18:45 NLN - switched manually from NOISE_TUNINGS

18:53 King Soft onsite

18:55 Richard back

19:21 a2l script

20:05 Keita is back

21:52 Alarm at CP4

22:10 did remote manual CP3 overfill - 20seconds

22:15 Kyle to MY to visually inspect the remote overfill.

22:41Kyle back

23:50 Robert and Anamaria back from EX

It looks like PEM is also doing an injection on BRSX as well as the broader IFO, it's very rung up. I have disabled the damping on it temporarily, we shouldn't use it for a little while. If the damper table runs too hard for too long it can get out of position, which is kind of annoying to fix. I'll check on it tomorrow. For now, the SEI_CONF should be left in WINDY_NO_BRSX. This state is currently using some narrow band sensor correction at ETMX (that bypasses the BRSX) to help with the microseism

While I was using conlog, I got distracted by the changing ODC bits for ASC and OMC, which seem to be constantly changing because of some thresholds that probably haven't been set recently.  I quickly updated them. The attached screenshot shows all the changes I made.  

I set some things that I think should be the same so they are matching (ie, I set CHARD thresholds to match DHARD thresholds, CSOFT to match DSOFT, AS_A_DC and AS_B DC now match but I set the maximums to be just under the ADC saturation level)

20:47 at Robert's request, I have closed reamining beam diverters except for POP. 

21:06 Anamaria is at EX. BRS has been turned off.

Dr Effler reported to me that she has reverted the -.2 Pit offset that Sheila set recently back to +.2. This resulted in a 400µrad shift in IM4.

I have created a new troubleshooting guide for H1, in preparation for O2.  It is available at G1602280 (too big for alog, so won't add it here also).  Attached here is the overview page that in the document has clickable links to each section.

My vision for this is that it is a go-to document when things aren't working as smoothly as they should be.  I try to provide specific instructions on what to do, but also enough information that Detector Engineers, Operators and anyone else can start looking deeper and debug problems on their own.  Text throughout the guide is clickable, so that you can jump to the section that you need. 

I encourage others to add to the document, or point me to well-written wiki pages / alogs with detailed instructions so that I can add to the document.  For example, both the PI and violin mode damping "instructions" in the guide actually just point you to the very well written wiki pages on those topics.  There are 19 sections so far, but I'm sure that I've missed some, and that others will come up, so please let me know or modify the document yourself if you find something!