Sheila, Stefan The PMC servo has been oscillating for about 2 weeks. Attached is a plot of Purple: PMC REFL power Blue: PMC REFL PDH error signal Yellow: Signal at the TEST A monitor point of the PMC locking servo board. Something is clearly wrong with the card - we suggest using a spare.
After the sus model restarts last night around 21:50 local time, the AI Chassis for h1sush2a and h1sush23 went into a locked state whereby they were not outputting the DAC drives and could not be reset.
We first worked on h1sush2a. We stopped the user models, restarted h1iopsush2a, checked the loopback duotone and saw it was now working. We started the user models and everything was driving correctly so we handed this over to the IMC group.
The we looked around and saw that h1sush34 was also in a bad state. There was less pressure to restore this system so we could do some testing.
First we went into the CER and verified that all 6 AI chassis were disabled by the IOP model.
We stopped all the user models.
First test, start h1iopsush34, reset its DACKILL, start h1sussr2, reset its DACKILL. All looks good, cannot reproduce error.
Second test, start h1iopsush34, DO NIT reset its DACKILL, start h1sussr2, reset its DACKILL, then reset IOP DACKILL. All looks good, cannot reproduce error.
For further information, we started all the models on h1sush34 and all DAC statuses are GREEN. Power down one AI Chassis, DAC0 and DAC1 5th bit went RED, it is a readback of AI status (16 channels per AI, two 18bit DACs per AI).
In Summary, code restarts caused IOP/AI lockout. Restarting IOP models cleared the problem. We cannot reproduce the problem easily. Last night the sus models were restarted at least four times, but only the last restart caused the problem.
HAM2 and HAM3 HEPI are currently locked. IO requested a survey of the shift between the current position of HEPI, and its controlled position during a recent IMC lock: October 5th 2013, 9am to 10am.
We looked at the IPS readouts to track the position of HEPI. The following script was created for calibration and coordinate transform of the signals: /ligo/svncommon/SeiSVN/seismic/HEPI/H1/Common/HEPI_Shift_Cart.m
MitchR, JimW
Per Cheryl and JoeG's request, we went into HAM2 to plug in the 5-way coax cables into the vac-side chamber feedthrus. It took 3 hours for 4 cables, but it's done. For the people planning 3IFO install stuff, this job would be much easier to do before the ISI goes in the chamber. These cables are difficult in the BSC chambers, where access is pretty good. In the HAM chamber, it's almost impossible. The reach is bad, there's no visibility and you can only use one hand. Because these cables are spring loaded, you would really like two hands, one to hold the cable in place Failiing that, a good alternative might have been to pull the feedthrus and install from the outside, but cable trays and piping would have made access on that side pretty awful, too.
The contractor removing insulation from the BT has completed the X arm and is moving his equipment to the Y arm. Photo is X2 module at the end station.
Weather has slowed the roofing work a little but still good progress has been made, Metal trim appears to be complete on the outbuildings and the electricians are restoring the lighning protection.
At the corner the upper decks are completed with membrane, Trim work is in progress and detail work around our staircase is largely done. The OSB membrane is more than 50% done. The projected finish is the 3rd week of November.
RE: 3rd photo - chalk line dust - NOT blood.
| MC2 | 5oct13 9-10am utc |
| l | -48.7 |
| p | 305.0 |
| y | 7.0 |
| pout | 802.5 |
| yout | 675.0 |
| MC1 | |
| l | 55.6 |
| p | -2042.0 |
| y | -1115.5 |
| pout | 588.8 |
| yout | -2637.7 |
| MC3 | |
| l | 35.7 |
| p | 25.0 |
| y | -1605.0 |
| pout | 444.4 |
| yout | -2865.6 |
| IM1 | |
| l | -79.0 |
| p | 2711.0 |
| y | 844.0 |
| pout | 2330.0 |
| yout | -170.0 |
| IM2 | |
| l | -89.0 |
| p | -1567.7 |
| y | -1390.0 |
| pout | -1880.0 |
| yout | -300.0 |
| IM3 | |
| l | -34.5 |
| p | 4235.0 |
| y | 279.0 |
| pout | 2850.0 |
| yout | 300.0 |
| IM4 | |
| l | -3.0 |
| p | -5071.0 |
| y | -1917.0 |
| pout | -2420.0 |
| yout | -980.0 |
| PRM | |
| l | -61.5 |
| p | 2384.0 |
| y | -588.0 |
| pout | -1538.0 |
| yout | -1153.0 |
| p drive | -820.0 |
| y drive | -430.0 |
| PR2 | |
| l | -57.0 |
| p | -307.0 |
| y | -593.0 |
| pout | 1331.0 |
| yout | 375.0 |
| p drive | 710.0 |
| y drive | 140.0 |
PR3 OpLev watchdog Sum Minimum set to 0 from 5000, in order to reset M13 level watchdog - not necessary or IO alignment, but something is preventing me from driving ANY optic on HAM2 - which is a problem.
The h1sush56 computer was restarted at 08:34 after becoming locked up during a user model start yesterday evening.
J. Kissel, D. Barker, J. Batch, A. Pele Another harrowing day of front-end code changes. In summary, I've now completed and committed all the simulink / front-end model, MEDM screen, and macro file modifications for the HLTS, HSTS, and OMCS. This involved several more recompiles, reinstalls, restarts, and restores of all 10 models involved, a couple of more crashes and restarts of the DAQ, and a whole lot of eye-crossing details. The final modification -- to the OMCS front-end code -- has crashed the h1sush56 front end again (as the rest of these installs have, save h1sush2a for some reason), and almost undoubtedly because of the usual mx_stream failure. Thankfully, if did not crash the rest of the DAQ, so data taking should still hold over night. Though I'm confident every thing I've done is legit, we should give a day to get things back up and running before we officially release everything to LLO. (Though I know they're not interested in more changes until they vent next Wednesday.) Of note -- It was not until I got to the MC_MASTER parts today, and getting some feedback from ISC folks that I realized -- for calibrated, interferometric control channel purposes (think IMC_X, PRCL, SRCL, MICH, DARM) -- we need to send out the output of each LOCK filter bank (including the top stage) to the top level to be sent over IPC to the OAF / Calibration model. For those SUS that aren't involved in global control, we'll just terminate the unused connections at the *top-level* of each model (currently only h1susmc2 is *actually* using this type of spigot to form IMC_X, the remaining IFO DOFs' control signals are pulled out for calibration in the LSC model). There are a couple of ways to do the calibrated channel thing; I'll bring up with ISC / Systems as to how we want to do it (because it's currently inconsistent across the ISC / IO models). Unfortunately, this means I've gotta go back to the QUADs and BSFM, but I'll wait for the discussion to conclude before I do so. Next on the chopping block -- tackle the severe upgrades and merging needed on the HAUX / HTTS models. Then that finally, finally, OMG finally, will close out this gosh darn ECR. Then we can do it all over again with the watchdog modifications. #facepalm.
It would be worth checking that scripts use to restart the mx_stream each front-end (such as /etc/start_streamers.sh ) are simplified to just do an mx_stream restart (i.e. /etc/init.d/mx_stream restart). The /etc/init.d/mx_stream script is customized to specify the MX slot and destination port on the the data concentrator. Old versions of /etc/start_streamers.sh did not call /etc/init.d/mx_stream but did their own restart that used slot 0, port 0 instead of unique slot/port.
CDS has brought all associated front-ends back up (see details in LHO aLOG 8424), and we've confirmed that we can get damping loops running and excitations can be sent out. As such, these modifications described above are clear of flaws, and are therefore ready for LLO to install.
Scott Shankle, Nichole Washington Cleanroom in LVEA West Bay has been staged for assembly of OFI for H1 and L1. Two tables will be moved into cleanroom via Jodi. Laminated drawings, assembly instructions and parts are in bins curbside cleanroom. All parts curbside placed in ICS storage-6963. Additional parts to be wrapped at the VPW. An OMC/OFI structure and braces are located in receiving. The OFI blades and small electropolished parts are located in Thomas Vo's office. The structure, braces, blades and small parts require transport to VPW for Class A cleaning. Mike Vargas is shipping all clean parts, hardware, tooling and tools currently at LLO to LHO.
I noted that when installing the black glass in HAM 3 that there was no plan to have a BG plate on AROM RH7 which was added after the black glass plan was made. We elected to use a spare BG plate from AROM RH5 and install it on this optic. I also noted that three optics (AROM RH3, ROM RH8, and ROM LH3) were attached to the HAM table with only one screw on one side of their base plates. These were originally planned to have clamping forks before the optic height was reduced do to the discrepancy with the sled and ISC bases were used. I installed three dog clamps (D1100640 v1 type 5) on these optics on the sides with missing screws. SNs were as follows; AROM RH3: SN 14, ROM RH8: SN 15, ROM LH3: SN 16.
Joseph Gleason, Cheryl Vorvick We've replaced the IO/ISS QPD cable with the new one provided by Peter and dressed it. I attempted to install one of the ISS PD array cables without success and will require the assistance of someone with longer arms. We've also installed irises in HAMs 2-3 in preparation for the beam re-routing.
Last friday, Betsy and Travis added weight to the middle mass of SRM currently on the test stand, in order to center the flags with the osems cf alog 8367. A first round of TF have been taken before that, showing clean measurements. A new set have been taken monday night to assess that there's no major change. The new results are showing an excellent match with the previous ones
Results in attachments are described below :
(1) H1 SUS SRM M1-M1 (top mass) comparison between modeled and measured undamped transfer functions for the 6 dofs
(2) H1 SUS SRM M1-M1 comparison between modeled and measured damped transfer function for the 6 dofs
(3) H1 SUS SRM M1-M1 undamped TF comparison before and after adding weight to the middle mass
data have been commited under the svn
I meant SRM in the title
For acceptance review I also added the spectra of SRM during phase 2b (currently plugged on the test bench).
The first 3 pdfs are showing the effect of the top mass damping on each stage. The last two pdfs are comparing SRM spectra with other hsts suspensions (PRM and SR2) during phase 2b, damping off and damping on.
Everything looks good for all osem signals, damping is working fine by significantly reducing the Qs of the resonances, and both damping on and off spectra are similar to other HSTSs at the same 2b stage of testing.
Came up again with this issue today :
When I modify an aLog from a "draft" that has been saved for a certain time (not clear how long), it won't let me post, save or preview the modified alog. It will actually delete the modification if trying to do so, and go back to the original draft.
Kyle, John We desire to repair a "legacy" leak on the 1.33" CFF on the north side of GV6. Our preference is to vent GV6's annulus volume and remove the flex line portion of the external piping (redundant) to this joint and just install blank flanges in place of the removed flex line -> Unfortunately, we weren't able to get GV6 to close enough to vent GV6's gate annulus volume to do this. Even after 70 psi was applied to the top of the pneumatic piston, GV6 never indicated "RED" or "hard-closed". Furthermore, introducing small amounts of air into the combined annulus volumes via the vent orifice of the small turbo connected to the annulus pump port resulted in PT124B and PT114B increases which verified the fact that GV6 isn't sealing when closed. We will discuss options tomorrow but for tonight we restored the nominal state with the exception of leaving the YBM turbo valved-in and IP5 valved-out
SUS offloading IPC receivers were left in HAM2-ISI and HAM3-ISI models.I removed those receivers and grounded the related master model inputs.
The new models are commited under the SVN:
- h1isiham2.mdl -r6195
- h1isiham3.mdl -r6196
Work was covered under ECR E1300578 and WP# 4180 (still open)
They will be moved in the H2 PSL on Friday, but if you want to move these before then, let me know.
Best,
Keita
Sheila, Richard, Jim, Patrick, Dave
Commissioning work on the rotation stage controller caused a 24VDC power glitch to the h1psl0 IO Chassis at 10:55 local time today. The FE computer did not crash, it allowed us to log in and verify that no ADC/DAC cards were visible via the One Stop system.
We powered down the FE computer, power cycled the IO Chassis, powered up the FE computer. When the models auto started with their safe.snap epics settings restored the PSL camera images all went blank due to a shutter closure in the Diode Room. We had seen this before and now know the recovery. On the PSL_LASER MEDM screen, the "External Shutter Request" section, the "Flow sensor" must be set to OPEN by pressing the small RESET button immediately below it. Only when all the requests are GREEN can the main RESET button be pressed. At that point a reset button has to be pressed on the Beckhoff controls system. At the moment this requires someone going into the Diode Room and pressing the button on the Beckhoff PC console. We should set up remote desktop to allow this from the control room.
The safe.snap settings do not appear to be correct. Sheila burt restored ISS, FSS and PMC from 23:00 27th Oct and the upper right video image came to life (PMC refl). But the PMC is not currently locking.
The two long cables for the IOO rotation stage were run early this week from the H1 PSL ENV Beckhoff chassis in the CER to the H1 PSL racks in the LVEA. At the same time Thomas V. and I moved the rotation stage and interface box from the CER to next to the H1 PSL racks. We reconnected them to these cables and continued testing there. Yesterday I wanted to try a different interface box. I disconnected the rotation stage from it (to avoid damaging the stage). I left the break out board on the end of the cable and removed the rotation stage from it. I then swapped the cables over to the other interface box, but did not reattach the rotation stage. I should have turned off the Beckhoff chassis first. I went back into the CER and saw the lights off on the Beckhoff chassis. I tried rocking the power switch, and the lights flashed briefly and then it wouldn't toggle anymore. It appears I tripped a breaker and took out the H1 PSL IO chassis which is in the same rack. It may be possible that the breakout board shorted out on the metal table it was sitting on, but I don't know. Richard pulled the Beckhoff chassis out, replaced something, and put it back. It appears to be working again.
As for the HSTSs, I took a spectra of PR3 from Monday. The isolation and damping loops of the ISI were closed (isolation level 3), as well as the damping loops of the suspension. The chamber was still under vacuum. Status of HEPI needs to be double checked.
The attached plot compares the spectra of LHO and LLO PR3, in a similar configuration (ISI isolated, SUS damped), for the 3 levels (M1 M2 and M3) and every degree of freedom.
Looking at the plots, we can see that LLO get more isolation at each level, even though we are using the same damping filters. That said, Jeff discovered that the suspension medm gains are set to different values, (higher values at LLO) which would explain why the resonnances of the suspension are more damped there. Other than this, spectra is very close to sensor noise. The larger signal between 0.1Hz and 1Hz is highly due to ground motion (I quickly took a spectra of the streckeisen sitting next to HAM2 chamber, which shows the same shape around 0.1Hz) More details to come next week.
Attached is the same document with the individual osems included
For acceptance review, attached is a phase 3b comparison of PR3 spectra damping on vs damping off. HAM2 ISI was in its isolated state, and HEPI was running under position control during the measurement.