The LHO SITEMAP MEDM was modified to add two new MAIN catagories: generic seismic (SEI) and Observing Run related screens (O-1).
I have added Hugo's generic SEI_SAT_OVERVIEW to the SEI button. I have added my new H1CDS_O1_OVERVIEW_CUST to the O-1 list. The new CDS OVERVIEW MEDM must always be GREEN. Any deviation from the O1 configuration will show as RED. Hopefully nothing will be red for long on this screen during O1.
h1nds1 (the default NDS) daqd process died this afternoon. This was a more drastic failure, the computer locked up and had to be reset. The last messages in the log files are different from yesterday to today:
Sunday 30 Aug 15:29:
Retry expired; requesting again
....
Packet buffer is full
Moday 31st Aug 15:58:
[date-time] Ask for retransmission of 15 packets; port 7097
....
Have to skip 15 packets (packet buffer limit exceeded)
The times of the restarts of h1sush2a and h1sush56 are summarized below.
2015_08_31 09:13 h1iopsush2a
2015_08_31 09:15 h1susmc1
2015_08_31 09:15 h1susmc3
2015_08_31 09:15 h1suspr3
2015_08_31 09:15 h1susprm
2015_08_31 10:09 h1iopsush56
2015_08_31 10:09 h1susomc
2015_08_31 10:09 h1sussr3
2015_08_31 10:09 h1sussrm
For the record, the above mentioned DAC recalibrations did NOT solve any of the problems that have reared up over the weekend. I can, however, report that the auto-calibration was successful for all DAC cards that were restarted. The 3rd DAC card's calibration on h1sush2a succeeded slowly, as it has done previously both on Aug 03 2015 (LHO aLOG 20165), and the time prior, Jun 09 2015 (LHO aLOG 19030). As reported before, this DAC card controls PRM M1 RT and SD. Last six channels are PR3 M1 T1, T2, T3, LF, RT, SD. Also as reported before, we don't know what this means or if it is significant. HOWEVER, according to what tests we have done, this DAC card being slow is merely coincidental with the problems we've been having with the PRM LF RT and PR3 T1 T2 noise found on those OSEM sensors. We've confirmed this by measuring the ASD of the OSEM sensors (as Evan has done in LHO aLOG 21056) with the suspension in SAFE (i.e. no requested drive) and found the noise as expected. We then switched the TEST/COIL enable switch to removed the DAC's ability to drive by removing the DAC input to the coil driver. The noise remained. The investigation continues... For h1sush2a (which houses MC1, MC3, PRM, and PR3): [8808794.054622] h1iopsush2a: DAC AUTOCAL SUCCESS in 5344 milliseconds [8808799.414955] h1iopsush2a: DAC AUTOCAL SUCCESS in 5344 milliseconds [8808806.438391] h1iopsush2a: DAC AUTOCAL SUCCESS in 6572 milliseconds [8808811.798720] h1iopsush2a: DAC AUTOCAL SUCCESS in 5344 milliseconds [8808817.586599] h1iopsush2a: DAC AUTOCAL SUCCESS in 5344 milliseconds [8808822.947012] h1iopsush2a: DAC AUTOCAL SUCCESS in 5345 milliseconds [8808828.307368] h1iopsush2a: DAC AUTOCAL SUCCESS in 5344 milliseconds the last time these DACs were recalibrated was Aug 03 2015. For h1sush2a (which houses SRM, SR3, and OMC): [5345389.136545] h1iopsush56: DAC AUTOCAL SUCCESS in 5333 milliseconds [5345394.496918] h1iopsush56: DAC AUTOCAL SUCCESS in 5344 milliseconds [5345400.284896] h1iopsush56: DAC AUTOCAL SUCCESS in 5341 milliseconds [5345405.645225] h1iopsush56: DAC AUTOCAL SUCCESS in 5344 milliseconds [5345411.433249] h1iopsush56: DAC AUTOCAL SUCCESS in 5341 milliseconds the last time these DACs were recalibrated was Aug 18 2015 (LHO aLOG 20631)
As per alog 21036, Evan dropped the absurd precision in the WFS_offset_* scripts which set the dark offsets. I reran it now since the IFO is down, confirmed that none of the 80 offsets changed by very much, and then accepted them in SDF. Now at only 4 sig didgits, SDF was able to write them into the SAFE.snap when I hit ACCEPT all.
ER8 Day 14
model restarts logged for Sun 30/Aug/2015
2015_08_30 15:30 h1nds1
2015_08_30 15:31 h1nds1
both restarts unexpected.
J. Kissel, K. Izumi, S. Dwyer, N. Kijbunchoo While trying to being the h1sush56 SUS (SR3, SRM, and OMC) to safe for DAC calibration of all cards in that chassis (see LHO aLOG 21048), we had great trouble with the SUS_SR3 guardian node. This node is not managed, but it was entire unresponsive to requests to change its state, to load, to pause, to stop, anything. Sadly, this was all true and the guadian node screen did *not* turn red indicating its in error. Looking at the guardian log, there was a message: epicsMutex pthread_mutex_unlock failed: error Invalid argument epicsMutexOsdUnlockThread _main_ (0x3e7a0f0) can't proceed, suspending. This is after requesting the ISC_LOCK guardian to DOWN. Note, we also had the SR3_CAGE_SERVO guardian still running, because, though this is managed by the ISC_LCOK guardian, it does not turn it OFF in the DOWN state. Probably not the issue, but it gathered our attention because it had gone nuts and was driving the M2 stage into constant saturation. See first attachment for screenshot of the broken situation. For the record this has happened on a smattering of guardian nodes in the past, see LHO aLOGs 17154 and 16967. ----------------------- Here's what we did to solve the problem: - Tried to restart the guardian node from a work station, guardctrl restart SUS_SR3 No success. - Tried to destroy the guardian node from a work station, guardctrl destroy SUS_SR3 No success. Both report stopping node SUS_SR3... timeout: run: SUS_SR3: (pid 3042) 23232015s, want down, got TERM (see second attachment). - Tried logging into guardian machine, h1guardian0, and began to kill process IDs sound on the machine related to SUS_SR3, (see third attachment). - Curiously, as I killed the first two processess, runsv SUS_SR3 and svlogd -ttt /var/log/guardian/SUS_SR3 as soon as I killed the latter, the guardian log came alive again, and the the SUS_SR3 node became responsive. - At Kiwamu's recommendation, I killed all processes simultaneously, destroyed the node, and restarted the node, just to make sure that all bad joojoo had been cleared up. ---------------- The problem is now solved, and we've moved on. Unsure what to do about this one in the future other than the same successively agressive restarting techniques...
This is a follow-up on alog alog20541 and alog20717
I took some time to study Robert's noise injections in more details. The result attached below. Quick conclusion: Human jumps in the change room and car sudden breaks near 2k electronics building and high-bay seems to couple into DARM.
re aLOG 20296, the bleed off of the tidal drive to the HEPIs tilts the platform enough to rile up the ISI T240s. Looked at trends for the last 30 days; the bleed down rate changed from 2 to 1u/s on 6 August.
Since then there have been many lock losses where there was some tidal buildup to be bled off. The diagonalization of the EndX HEPI must be pretty good and gives no indication of tilting the T240 much. Unlike X, the HEPI for EndY shows some fairly substantial tilting as the build up is reduced. See the attached for 15 minute plot of a recent Bleed off at EndY where the ~110um bleed off at 1u/s riled up the T240s more than half way to the trip point.
The EndY HEPI could use some diagonalization
No indication that Accumulators need checking or charging.
Trends of Pump output pressures (attached) indicate no need to restart any pumps (no DC level changes pointing to cavitation.) Yes those are actually pressures; I don't know why the trends of the L0 (corner station) don't show any more fluctuations like they do at the Ends... No change at EndY, same daily and weekly glitches.
No further HEPI maintenance this week.
Peforming some calibration measurements in the past weekend, I noticed that the red TRX and TRY QPDs had some dark offsets which were not quite cancelled. In this morning, since the interferometer was down, I adjusted the dark offset in each segment of all four qpds (ASC-X_TR_A and , and ASC-Y_TR_A and B). Subsequently I adjusted the LSC-TR offsets which receives the sum signals from these qpds -- the offsets are now zero as expected.
At the request of Richard M and Nutsinee, I restarted the models on h1sush2a to force a recalibration of the 18bit DAC cards. All cards reported successfull calibration.
Dan, Jim
The IFO was running smoothly in low-noise for almost four hours until a lockloss at 0923 UTC. Immediately prior to the lockloss, Jim and I noticed some motion of the AS beam spot. There were a few bursts of motion, a few minutes apart, and then we broke lock.
The first image is a 1-hour trend of the ASC-AS_C signals (channels 7 & 8). There are some bursts of noise in ASC-AS_C_PIT, correlated with similar noise in the M3 witness sensors of the SRC optics. SR3_WIT_P (channel 15) is the worst offender. This is the error signal used for the cage servo. The control signal is SR3_M2_TEST_P (channel 10), which has some excurions from the quiescent level at the time of the noise bursts.
So, it looks like something was kicking the SR3 OSEMs, and this got into the cage servo and broke the lock. Since nothing feeds back to SR3 it must have been an internal problem.
The second plot is a 1-hour trend of the SR3 OSEMs. M1_T2 stands out as a problem, and noise from this sensor would couple to pitch in the lower stage. Channel 16 is the T2 VOLTMON; clearly T2 is having issues. This is the same coil that Evan, Jeff and I had trouble with two weeks ago. Since then, the SR3 coil driver was swapped. The glitches don't have the same shape as the excusions back then.
Jim and I have been watching a dataviewer trend of the T2 voltmon and there are occasional large excusions that are correlated with the M3_WIT_P channel, so something is really moving. We held the ouput from the cage servo and saw a few more glitches, so it must be coming in from the top stage damping. We turned off the top stage damping and we *thought* we saw a few more, so it must be coming from the actuation electronics. The problem is very intermittent and we're not totally confident in the diagnosis.
Btw the T1 voltmon is still busted.
Vern asked for some of the pie charts I made for ER7 for the first weeks of ER8. Here they are, based on data from 15:00 UTC August 17th to 15:00 UTC on the 28th, with about 40 minutes of missing data.
A few notes:
Our duty cycle is a bit worse than it was durring ER7, although there are several reasons we would expect it to have improved. Some of the downtime can probably be explained by commsioning and calibration activities, althoug I haven't yet tried to take that into account. It will be easier to get a clear picture of things once we start running undisturbed more.
Windy for the first 2/3 of the shift, therefore not much success locking. As the wind was dying down, the locking became quicker, but typically (but not limited to) losing it at SWITCH_TO_QPDS. I have added instructions to the OPS wiki regarding turning on OMC DCPD whitening in NOMINAL_LOW_NOISE.
Activity log:
2:39 locked on DC readout for Sheila/Evan commissioning
3:24 lockloss at ENGAGE_ISS_2ND_LOOP
4:00 Dan shows up on site, starts looking into PRM and PR3 noise noted by Evan and Betsy
5:02 locked DC readout
5:11 locked NOMINAL_LOW_NOISE, Dan doing some OMC work
6:07 set to Undisturbed
Currently the OMC alignment dither lines are between 1675 and 1750Hz. We'd like to move these lines above 2kHz to avoid polluting the analysis band of the CW searches. Previously, when we moved the dither line frequencies from ~600Hz to ~1700Hz, we noticed there was a sign flip in the dither sensing matrix. We expect the tip-tilt response to be flat at these high frequencies, so before moving the lines again I thought it was prudent to check what kind of resonance structure we were dealing with.
The attached plot shows transfer functions from OM1 pitch and yaw to OMC-DCPD_NORM while in full lock. Overall the reponse of the tip-tilts is what we expect, but there are resonance features around 1900Hz and above 2.2kHz. Also the phase changes smoothly by 180degrees between 500Hz and 1600Hz, so that explains the sign flip. Note the dropout in coherence around the violin harmonics. The range of the current dither frequencies is marked by the black dashed lines.
It looks like we can move the dither lines to just above 2kHz. The phase will be a few tens of degrees different than the current frequencies, so the demod rotation phases will need to be changed. The magnitude of the tip-tilt response is roughly the same, and anyways the current dither amplitudes are only a hundred counts, so we're well below saturation. Should be a simple fix.
Nairwita Mazumder, Rich Abbott A few days back Jim noticed (alog ) that the "Bumbling line" which varies over a large frequency range is again back on ETMX seismic channels . This was first noticed on March and disappeared before ER7 and again was seen from 4th August. One can see the lines at all the horizontal and vertical sensors on ETMX. I have attached a pdf containing some follow up work done during Rich's recent visit to LHO. The first plot in the pdf is the spectrogram of ETMX GS13 on 26th August. It can be seen that there are multiple wandering lines having a fixed offset. We were suspecting that some magnetometers at the End X might be the culprit (as we could not find any correlation between temperature fluctuation with the line ). The second and third plots are the spectrum of H1:PEM-EX_MAG_EBAY_SEIRACK_Z_DQ and H1:ISI-ETMX_ST2_BLND_Z_GS13_CUR_IN1_DQ for 2nd August and 26th August respectively. The red one is for 2nd August when the bumbling line could not be found and the blue one is the recent data (26th August). It is clear that the peaks appearing on ISI-ETMX_ST2_BLND_Z_GS13 after 3rd August are correlated with the peaks of the spectrum (which also appeared around the same time) of SEIRACK magnetometer . The plots on the second page shows the coherence between GS13 and the magnetometers in the VEA and SEIRACK. It looks like the magnetometer on the SEI rack has stronger coherence with GS13 sensors than the magnetometer located at VEA . I have marked two points (blue and red cross) in the coherence plots to highlight two of the many peaks.
Adding to Nairwita's comments, the signal seen in the GS13 spectra is also present in the magnetometer data. This being the case, it's most likely that the harmonic series points to an electromagnetic artifact associated with the HEPI pump variable frequency drive. The fact that the same signature does not exist at the other end station (I assume this to be true, but have not verified) may point to an enhanced susceptibility in the X-end electronics for some reason. No reason to panic much yet, but duly noted.
I have attached the coherence plots computed between PEM-EX_MAG_SEIRACK and GS13 , ST1 CPS and ST2 CPS over the frequency range 0.4Hz-900Hz to check the following two points: (1) If there exists any coherence between CPS and the Magnetometer at frequency above 256 Hz (2) What the low frequency behavior is I can be seen that the coherence between CPS and the magnetometer above ~25Hz is pretty low compared to GS13, but them have relatively high coherence with PEM-EX_MAG_SEIRACK near 20Hz .
I have completed a set of measurements that is necessary for nailing down the ETMY actuation scale factors. This is a third round of the measurements in order to study repeatability and systematic errors. The measurement data can be found at the follwong places.
aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L3toDARM_LVLN_LPON_FullLock.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L2toDARM_FullLock.xml
/aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L1toDARM_FullLock.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMX_toDARM_FullLock.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_PCALYtoDARM_FullLock.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-08-29_PCALY2DARMTF_7to1200Hz.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-08-29_H1DARM_OLGTF_7to1200Hz.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/ALSDIFF/2015-08-29/2015-08-29_ALSDiff_ETMX_L3_HVHN.xml
/aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_MICH.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_MICH_OLGTF.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1MICH_freeswingingdata.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_XARM.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSETMX_L3_HVHN_XARM.xml
J. Kissel, K. Izumi I've process the results from the above aLOG using the same model I had used for LHO aLOGs 21015, just to get something out the door for review at today's calibration meeting. The results look encouragingly consistent with Wednesday's (LHO aLOG 20940) and Friday's (LHO aLOG 21005) results. Again, the next steps are to - functionalize the analysis scripts so we can easily process future measurements, - have those functions spit out their individual measurement results so they can be compared with each other (the comparison of which is another function to write), and - make sure that all the scripts are using the output of the latest DARM model and parameter set, instead of recreating a naive model from the matlab QUAD's [m/N] and knowledge of the electronics chain. - finish processing results from all the of the electronics chain measurements to identify if frequency-dependent systematics lie in there - Update / refine the DARM model parameters to squash the frequency-dependendent residual to as small as possible, such that we can be confident in professing number with properly quantified uncertainty. What has NOT been included in the naive actuation model: - 16k IOP up-sampling filter - Analog AA filter - Any of the un- or poorly-compensated poles and zeros of the ESD drivers - The recently found mis-match between expected and reality regarding the UIM driver's poles and zeros (LHo aLOG 20846) I have a high degree of confidence that these are the reasons for the residual frequency dependence. In other words, these be resolved once I switch to using the full DARM model's actuation chain. That being said, suggestions are welcome as to what else the discrepancies might be.
Kiwamu, Nutsinee
As we were trying to relock the ifo after several locklosses due to high wind (50mph), we noticed the sideband signals wiggled a lot before another lockloss at DC_READOUT (wind speed ~35-40 mph). We found a coherence between POP18, POP19, POP_A_LF, AS90 and PRM, SRM, BS which indicates that the DRMI was unstable. The BS ISI Windy blends weren't turned on.
One of the two lock losses seemed to be associated with PRM saturation. We heard of the saturation alarm voice pointing PRM DAC in full lock mutiple times before the lockloss in NOMINAL_LOWNOISE. I am not sure if this is the direct cause, but as shown in the attached, PRM had been experiencing ~20 sec oscillation in longitudinal which used to be a big issue in the past (alog 19850). At that point wind was around ~40 mph on average. Also, I attach spectrum of each coil on the M3 stage. It is clear that the components below 0.1 Hz are using up the DAC range when wind is high.
Just as a check, I remade Kiwamu's plot for PRM, SRM, and MC2, with all the stages that are used for actuation.
At this point, the wind ine corner station varied between 3 and 13 m/s. The 30 mHz – 100 mHz BLRMSs were about 0.02 µm/s in the CS Z (consistent with sensor noise), 250 µm/s for EX X, and 250 µm/s for EY Y.
Since this time, we have increased the offloading of PRM and SRM to M1 by a factor of 2, but we probably need an even higher crossover in order avoid saturation during these times. It may have the added benefit of allowing us to stay locked during even windier times. Additionally, MC2 does not look like it needs any work on its crossovers in order to avoid saturation.
The above comment should say 0.25 µm/s for EX X and EY Y.