Displaying reports 65921-65940 of 85503.Go to page Start 3293 3294 3295 3296 3297 3298 3299 3300 3301 End
Reports until 15:48, Friday 31 July 2015
LHO VE
bubba.gateley@LIGO.ORG - posted 15:48, Friday 31 July 2015 (20104)
Beam Tube Washing
Scott L. Ed P. Rodney H.

7/30/15
The crew relocated lights, fans and cords to begin cleaning the next section starting at HSW-2-043. 51 meters of tube cleaned ending l2 meters east of HSW-2-041.
I repaired several of the extension cords and a cord on the vacuum used for cleaning the support tubes. 

7/31/15
Ed took today off. 
The support vehicles and all related equipment were relocated after lunch and a total of 61 meters of tube cleaned today ending 13.7 meters east of HSW-2-038.

The past ~200 meters of tube have been especially dirty with feces and urine. We have been giving these areas extra attention.

1239.5 meters of Y-Arm cleaned to date.
Non-image files attached to this report
H1 CDS (CAL, DAQ)
david.barker@LIGO.ORG - posted 15:48, Friday 31 July 2015 (20106)
IRIG-B channels added to end station cal models, DAQ and DMT

Stefan, Jim, Andres, Filiberto, Dave

WP5392

New h1calex and h1caley models were installed today, these read out ADC3-CHAN30 as the IRIGB signal. These signals were added to the commissioning frame and the frame broadcaster. The DAQ was restarted at 11:55 PDT to install this change.

H1 General (DAQ, GRD, IOO, PSL, SEI, SUS)
cheryl.vorvick@LIGO.ORG - posted 13:43, Friday 31 July 2015 - last comment - 22:50, Friday 31 July 2015(20103)
something happend coincidentally with a lock loss, and we had a number of minutes of a bad IFO state

Something happened at lock loss and the IFO did not reach the defined DOWN state.

Symptoms:

Dave, Jaime, Sheila, operators, and others are investigating.

Images attached to this report
Comments related to this report
jameson.rollins@LIGO.ORG - 22:50, Friday 31 July 2015 (20105)

Let me try to give a slightly more detailed narrative as we were able to reconstruct it:

  • At around 11:55, Dave initiated a DAQ restart.  At this point the IFO was locked in NOMINAL_LOW_NOISE.
  • The DAQ restart came with a restart of the NDS server being used by Guardian.
  • The IMC_LOCK guardian node was in ISS_ON, which utilizes cdsutils.avg(), which is an NDS call.  When the NDS server went away, the cdsutils.avg() threw a CdsutilError which caused the IMC_LOCK node to go into ERROR where it waits for operator reset (bug number 1).
  • The ISC_LOCK guardian node, which manages the IMC_LOCK node, noticed that the IMC_LOCK node had gone into ERROR and itself threw a notification.
  • No one seemed to notice that a) the IMC_LOCK node was in error and b) that the ISC_LOCK node was complaining about it (bug number 2)
  • At about 12:12 the IFO lost lock.
  • The ISC_LOCK guardian node relies on the IMC_LOCK guardian node to report the lock losses.  But since the IMC_LOCK node was in error it wasn't doing anything, which of course includes not checking for lock losses.  Consequently the ISC_LOCK node didn't know the IFO had lost lock, it didn't repond and didn't reset to DOWN, and all the control outputs were left on.  This caused the ISS to go into oscillation, and it drove multiple suspensions to trip.

So what's the take away:

bug number 1: guardian should have caught the NDS connection error during the NDS restart and gone into a "connection error" (CERROR) state.  In that case, it would have continually checked the NDS connection until it was re-established, at which point it would have continued normal operation.  This is in contrast to the ERROR state where it waits for operator intervention.  I will work on fixing this for the next release.

bug number 2: The operators didn't know or didn't repond to the fact that the IMC_LOCK guardian had gone into ERROR.  This is not good, since we need to respond quickly to these things to keep the IFO operating robustly.  I propose we set up an alarm in case any guardian node goes into ERROR.  I'll work with Dave et. al to get this setup.

As an aside, I'm going to be working over the next week to clean up the guardian and SDF/SPM situation to eliminate all the spurious warnings.  We've got too many yellow lights on the guardian screen, which means that we're now in the habit of just ignoring them.  They're supposed to be there to inform us of problems that require human intervention.  If we just leave them yellow all the time they end up having zero affect and we're left with a noisy alarm situation that everyone just ignores.

thomas.shaffer@LIGO.ORG - 16:58, Friday 31 July 2015 (20115)

A series of events lead to the ISC_LOCK Gaurdian to not understand that there was a lockloss.

  1. ISC_LOCK was brought to DOWN after realizing the confusion.
  2. A series of events lead to the ISC_LOCK Gaurdian to not understand that there was a lockloss.
  3. ISC_LOCK was brought to DOWN after realizing the confusion.
  4. DAQ restart by Dave at 11:55 PST
  5. IMC_LOCK went into Error with a "No NDS server available" from the DAQ restart
  6. This was not seen by the operator, or was dismissed as a result of the restart.
  7. Lockloss at 12:12 PST
  8. ISC_LOCK did not catch this lock because IMC_LOCK was still in Error.
  9. Since the ISC_LOCK thought it was still in full lock, it was still actuating on many suspensions and trip some watchdogs (like Daves alog20111)
  10. ISC_LOCK was brought to DOWN after realizing the confusion.

To prevent this from happening in the future, Jamie will have Guardian continue to wait for the NDS server to reconnect, rather than stopping and waiting for user intervention before becoming active again.  I also added a verbal alarm for Guardian nodes in Error to alert Operators/Users that action is required.

(If i missed something here please let me know)

H1 DetChar (DetChar, ISC)
gabriele.vajente@LIGO.ORG - posted 12:59, Friday 31 July 2015 (20102)
Coherence for latest quiet data

I ran BruCo on the quiet data period reported here. The report can be found at the following link:

https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1122369690/

I'll look into the table and provide a summary later today.

H1 DAQ (DAQ)
stefan.countryman@LIGO.ORG - posted 12:31, Friday 31 July 2015 (20101)
Recalibrated timing diagnostic Cs-III Cesium Atomic Clock Frequency and Phase in MSR
Between 11:10 and 11:25 I adjusted the frequency offset and the phase offset in the timing diagnostic system's cesium clock in MSR so that the 1PPS time difference between it and our GPS backed timing distribution system equals 0±100ns and the long-term drift rate is near zero.

Last time I adjusted the timing offset (3 weeks ago), I subtracted 68e-15 from the factory-set frequency offset of 1078e-15; this was apparently the wrong direction, as evidenced by a very visible doubling in the cesium clock's drift rate in the weeks since that adjustment. Since the last measurement of drift rate was calculated from a month's worth of data and was ostensibly accurate, I went ahead and changed the frequency offset to (1078+68)e-15. The manufacturers instructions are vague on the correct adjustment direction and the tech support reps aren't very knowledgable, so I'm going to write up a short technical paper on Cs-III calibration with specific instructions for how to go about this.

I zeroed the phase using a 1PPS signal from the MSR master. The GPS timing jitter is O(10ns), which is well within the 100ns resolution of the CsIII phase syncing utility. I used a short (3') BNC cable to minimize delay.

I also went ahead and installed the most recent Monitor3 software on the Lenovo X61 Thinkpad in MSR and put the Cs-III instruction manual on that computer's desktop, so it should be easy for someone else to make finer adjustments to the frequency offset once we have another month or two worth of 1PPS drift data. 
H1 CDS
david.barker@LIGO.ORG - posted 11:27, Friday 31 July 2015 (20098)
h1oaf0 timing error, restart of all models

at 10:37 PDT the IOP model on h1oaf0 received a timing error, all models stopped running at this time(h1pemcs, h1oaf, h1tcscs, h1odcmaster and h1calcs). The calculated IFO range went to a high value and the DARM FOM display went noisy. Only one external model receives IPC from the h1oaf0 computer, h1omc has a dolphin channel from the calcs, these channels went into 100% error.

There was PEM work in the CER around this time, this could have been the cause of the glitch. All the models were restarted and all is running correctly now. The intent bit was not set during this time.

H1 SEI
thomas.shaffer@LIGO.ORG - posted 11:04, Friday 31 July 2015 (20097)
BRS software restart

BRS software crashed on the 25th, so today I went to EX to restrart the code. I kept the damper commented out, so the damper is currently OFF. I will turn it back on when it seems like it will be calm down at EX for a bit (vac team is down there right now).

H1 SEI
thomas.shaffer@LIGO.ORG - posted 09:52, Friday 31 July 2015 - last comment - 10:19, Friday 31 July 2015(20095)
ISI_ETMY ST1/2 WD trip after lockloss, possibly Tidal?

Within a min afer a lockloss, the ISI ETMY ST1/2 WDs tripped. The medm said that it was ST1 T240 and the plots showed a slow drift to the WD trip level (plot attached). HEPI moved 170 um after lockloss before tidal began to bleed down. A further investigation is ongoing by the SEI team.

Images attached to this report
Comments related to this report
hugh.radkins@LIGO.ORG - 10:19, Friday 31 July 2015 (20096)

Looks like this is a trip we will suffer whenever the drive to the HEPI is large enough at the time of lockloss.  That doesn't necessarily mean the length of lock but it is certainly related.  If the tide turns around and the offset given to HEPI heads back to zero, we could be fine after a long lock stretch.

The attached 30 min second trend plots are around the three most recent locklosses from some low noise state.  In each case you can see the T240 tilting towards trip level but only the most recent trip was the tidal offset large enough to 'tilt' the T240 long enough to hit the trigger.  The HEPI output from ISC is in nm.

Is the problem the bleed off too fast?  It seems this is 2um/sec.  Is there horizontal to tilt coupling that needs to be addressed?  Is it a problem anyway?  The T240 will likely take a couple minutes at worse to settle and another minute for the ISI to reisolate.

Images attached to this comment
H1 General
jeffrey.bartlett@LIGO.ORG - posted 09:38, Friday 31 July 2015 (20094)
08:30 Meeting Minutes
CDS: EE shop work to prep for Ring Heater installation
PSL: Wants to tweak PR3 OpLev at first available opportunity
FAC: Continued beam tube cleaning Y-Arm   
VAC: Moving pump cart to End-X for weekend long RGA bake out

H1 ISC (DetChar, ISC)
lisa.barsotti@LIGO.ORG - posted 02:55, Friday 31 July 2015 - last comment - 08:28, Friday 31 July 2015(20089)
Undisturbed data
Evan, Matt, Lisa

We took ~25 min of undisturbed data starting at Jul 31 2015 09:21:13 UTC , after the MICH FF improvement (and  all the changes described earlier  implemented), with the interferometer locked in low noise ~ 65 Mpc (according to SensMon).

We saw a big glitch during this period (~9:28), a huge one right before (~9:18, right after the mich tuning was done), and another one at the end of the undisturbed period. 

We went back to commissioning mode as Evan is starting some noise injections for his noise budget.
Comments related to this report
evan.hall@LIGO.ORG - 05:59, Friday 31 July 2015 (20092)

I did injections into MICH (from 10:41:40 to 10:47:00) and SRCL (from 11:15:00 to 11:22:40) with all FF on.

About 30 s after turning off the SRCL injection, a number of photodiode signals associated with the AS port dropped suddenly by about 5% (in particular: AS DC, AS90, ASA sum, ASB sum, ASC sum, and AS45Q; notably, the OMC DCPDs, POP DC, MC2 sum, and IMC input power did not see this). After that, the interferometer glitched for about 30 minutes. This event happened at 11:23:31.05 and lasted for only a few milliseconds.

I did an injection into PRCL (from 12:21:50 to 12:29:00).

I did an injection into the ISS around 12:52:00. I think this measurement needs to be redone by injecting into the error point of the outer loop rather than the inner loop. With the outer loop closed it is now difficult to get good coherence between intensity and DARM below 300 Hz.

Images attached to this comment
Non-image files attached to this comment
keita.kawabe@LIGO.ORG - 08:28, Friday 31 July 2015 (20093)

PSL Peri PZT servo ON/OFF test:

IFO was still locked when I came in in the morning, so I briefly switched the IMC WFSB YAW to PSL PZT feedback path at around 7:50 AM local time.

In the attached, references are with the servo on, current traces are without the servo.

Middle finger (300Hz bump) is greatly reduced in OMC DCPD, but the coherence for right and index finger slightly increased due to gain peaking (expected), and there is another gain peaking at 620Hz (again expected). The last one could be reduced further by adding a deeper notch but that might increase the 350Hz bump gain peaking a bit.

The servo was turned on again at about 8AM+-5min.

Images attached to this comment
H1 ISC (ISC)
lisa.barsotti@LIGO.ORG - posted 01:46, Friday 31 July 2015 - last comment - 02:58, Friday 31 July 2015(20085)
Summary of changes made this week
Commissioning Team

For the meeting tomorrow, here is a list with the things that we changed this week, and they are now part of the baseline locking sequence:


1) more aggressive cut-off in MICH, to reduce coherence with DARM above 50 Hz (log 20020 );

2) more aggressive cut-offs in the  ASC_AS_A / AS_AS_B --> OM1 / OM2 centering loops (from 100 Hz to 20 Hz) to remove some coherence with those signals reported by Bruco;

3) more aggressive cut-offs in the OMC SUS alignment loops ( log 20087 ) to remove unwanted OMC length motion which was observed to cause fringe wrapping by Josh et al (log 20079 );

4) heroic modification of the DHARD loops (both PITCH and YAW) during the locking sequence, aimed to make the transition to REFL/TR and carm offset reduction more stable by increasing the phase margin of this loop, which was kind of poor and was seen to cause several lock losses in the past (log 20084 ); these loops have now high ugfs (~ 5 Hz) and >30 degrees phase margin.  Note that the ASC work is not done yet: tomorrow we will add cut off filters, and attack CHARD. Also, we haven't yet succeeded in turning off the ITMs optical levers, which are still on;

5) modification of the ISS loop to improve phase/gain margin (log 20088  and log 20088 );

6) IMC WFS --> PSL PZT loop to reduce ~300 Hz peaks in DARM ( log 20051 );

7) found and fixed a sneaky ramp time problem which was causing the switch from ESD ETMX --> ETMY to fail repeatedly (log 20076 ), Jenne got a BIG PLUS for this one;

8) added an offset to PR3 during the CARM offset reduction to counteract the wire heating ( log 20055 )

9) the Guardian usercode has been modified to speed up the locking sequence ( log 20055 ), and improve IMC_LOCK;

The interferometer has been happily locked in low noise at ~ 65 Mpc for nearly 2 hours with all the above changes implemented, and the Guardian has been updated. 

10) (Later edit) Evan just optimized the MICH feed forward, and he reduced the MICH coupling between 30 and 50 Hz; the sensitivity is now better in that region. 

Other things in our to-do-list for tomorrow:

1) More alignment work (cut-offs and CHARD)

2) Test the whole sequence again to check robustness

3) Track down the TMSX weirdness (log 20078)

4) Noise investigations and injections for noise budget



Comments related to this report
evan.hall@LIGO.ORG - 02:58, Friday 31 July 2015 (20090)

Result of MICH FF retuning attached. The tweaking filter is FM1 in LSC-MICHFF.

Images attached to this comment
H1 AOS
matthew.evans@LIGO.ORG - posted 00:47, Friday 31 July 2015 - last comment - 04:43, Friday 31 July 2015(20088)
ISS second loop filter fixed

Sudarshan, Matt

The new ISS filter was seen to oscillate at ~1MHz with no input signal.  This was fixed by adding 50pF cap into the feedback of the output opamp.  This cap, in combination with the 10k resistor, puts a pole at 300kHz, which is fine for this loop.  The second loop is currently closed (with the IFO at high power) and appears to be working properly.

Comments related to this report
kiwamu.izumi@LIGO.ORG - 04:43, Friday 31 July 2015 (20091)

Also, independently of the repair that Matt desribed above, we found that an op-amp in the very first stage kept railing at -10 V. It is AD620 stuffed in U10 of the 2nd loop servo box (D1300439-v2). We have no idea when this op-amp had died. Swapping the op-amp with a spare in the EE shop fixed the issue. We will update the e-traveler accordingly.

H1 AOS
matthew.evans@LIGO.ORG - posted 00:42, Friday 31 July 2015 (20087)
Low velocity OMC SUS ASC

Evan, Matt, Josh S. (in spirit)

The OMC ASC  loops now have 0.5Hz cut-off to remove unwanted OMC length motion (see plot).  This change is intended to reduce the OMC SUS velocity so that we don't get fringes in DARM.

The filters are in OMC-ASC_{POS, ANG}_{X, Y} loops (see screenshot), and do not appear to cause stability problems (because the UGFs are below 100mHz) with the MASTER_GAIN up to 0.2, though some gain peaking is observed at this gain.  The normal operating gain is 0.1.

Images attached to this report
H1 INJ
evan.hall@LIGO.ORG - posted 23:22, Thursday 30 July 2015 - last comment - 18:06, Friday 31 July 2015(20078)
1821 Hz TMSX sensor spikes

Matt, Evan

Why do the TMSX RT and SD OSEMs have such huge spikes at 1821 Hz and harmonics? These spikes are about 4000 ct pp in the time series. In comparison, the other OSEMs on TMSX are 100 ct pp or less (F1 and LF shown for comparison).

Also attached are the spectra and time series of the corresponding IOP channels.

Images attached to this report
Comments related to this report
evan.hall@LIGO.ORG - 02:44, Friday 31 July 2015 (20083)

On a possibly related note: in full lock, the TMSX QPDs see more than 100 times more noise at 10 Hz than the TMSY QPDs do.

From Gabriele's bruco report, the X QPDs have some coherence with DARM around 78 Hz and 80 Hz. A coherence plot is attached.

Images attached to this comment
arnaud.pele@LIGO.ORG - 12:10, Friday 31 July 2015 (20100)

It seems similar to the problem from log 12465. Recycling AA chassis power fixed the issue at the time.

keita.kawabe@LIGO.ORG - 18:06, Friday 31 July 2015 (20118)

Quenched the oscillation for now (Vern, Keita)

We were able to clearly hear some kHz-ish sound from the satellite amplifier of TMSX that is connected to SD and RT. Power cycling (i.e. removing the cable powering the BOSEM and connecting it again) didn't fix it despite many trials.

We moved to the driver, power cycled the driver chassis, and it didn't help either.

The tone of the audible oscillation changed when we wiggled the cable on the satellite amp, but that didn't fix it.

Vern gave the DB25 connector on the satellite amp a hard whack in a direction to push the connector further into the box, and that fixed the problem for now.

Images attached to this comment
Displaying reports 65921-65940 of 85503.Go to page Start 3293 3294 3295 3296 3297 3298 3299 3300 3301 End