Aidan, Nutsinee, Jim, Dave:
an apache web server was installed on h1hwsmsr to provide web access to the HWS camera image files. This is an internal web service with no offsite access running HTTP protocol. Its URL is http://h1hwsmsr
model restarts logged for Sun 08/Mar/2015
2015_03_08 00:26 h1fw1
one unexpected restart. Sunday reporting was given Saturday's date, possibly caused by time change to PDT. All times subsequent are now local PDT.
model restarts logged for Mon 09/Mar/2015
no restarts reported
model restarts logged for Tue 10/Mar/2015
2015_03_10 09:49 h1calcs
2015_03_10 09:54 h1calcs
2015_03_10 10:10 h1odcmaster
2015_03_10 11:30 h1pemex
2015_03_10 11:36 h1alsex
2015_03_10 11:36 h1calex
2015_03_10 11:37 h1calex
2015_03_10 11:38 h1iscex
2015_03_10 11:41 h1pemex
2015_03_10 11:45 h1iscex
2015_03_10 11:50 h1iopiscex
2015_03_10 11:50 h1iscex
2015_03_10 11:52 h1alsex
2015_03_10 11:52 h1pemex
2015_03_10 11:56 h1calex
2015_03_10 11:56 h1pemex
2015_03_10 11:58 h1alsex
2015_03_10 11:58 h1calex
2015_03_10 11:58 h1iscex
2015_03_10 11:59 h1iscex
2015_03_10 12:07 h1asc
2015_03_10 12:16 h1iscex
2015_03_10 12:21 h1iscex
2015_03_10 12:26 h1iscex
2015_03_10 12:47 h1omc
2015_03_10 13:41 h1omc
2015_03_10 13:46 h1lsc
2015_03_10 14:41 h1odcmaster
2015_03_10 15:01 h1alsey
2015_03_10 15:01 h1iopiscey
2015_03_10 15:01 h1pemey
2015_03_10 15:03 h1caley
2015_03_10 15:05 h1iscey
2015_03_10 15:09 h1broadcast0
2015_03_10 15:09 h1dc0
2015_03_10 15:09 h1fw0
2015_03_10 15:09 h1fw1
2015_03_10 15:09 h1nds0
2015_03_10 15:09 h1nds1
2015_03_10 15:09 h1odcmaster
2015_03_10 15:11 h1calcs
2015_03_10 15:13 h1omc
2015_03_10 16:26 h1pemex
2015_03_10 16:34 h1pemey
2015_03_10 16:36 h1broadcast0
2015_03_10 16:36 h1dc0
2015_03_10 16:36 h1fw0
2015_03_10 16:36 h1fw1
2015_03_10 16:36 h1nds0
2015_03_10 16:36 h1nds1
2015_03_10 16:56 h1pemex
2015_03_10 16:56 h1pemey
2015_03_10 17:05 h1dc0
2015_03_10 17:06 h1dc0
2015_03_10 17:07 h1broadcast0
2015_03_10 17:07 h1fw0
2015_03_10 17:07 h1fw1
2015_03_10 17:07 h1nds0
2015_03_10 17:07 h1nds1
no unexpected restarts. Please refer to slow controls alog for Beckhoff restarts. FE and DAQ details in CDS maintenance alog.
CDS maintenance summary, Tuesday 10th Mach 2015.
ODC work, WP5095
TJ, Ryan, Stefan, Daniel, Dave:
All three ODC models were slowed from 32kHz to 16kHz. Additional shared memory ODC IPC transmitter channels were added to the end station PEM, CAL and ALS models. Further ODC changes were made, please see the ODC team's alogs for details.
Shuffle of end station ISC models WP5095
Daniel, Jim, Dave:
The PEM and CAL models, which were sharing a single core, were split back into their own cores. The ODC model, which had its own core, was combined with the ISC model and given an RFM IPC sender. The original CAL DCU-IDs were used, and the new ones assigned to ALS were returned to the pool. The CAL core number was changed as it was conflicting with ISC
In the following table, each model's dcuid and core number is shown. Models which share the same core are color coded.
| model | was | is now |
| h1pemex | 84,2 | 84,2 |
| h1calex | 84,2 | 124,4 |
| h1iscex | 126,5 | 86,5 |
| h1odcx | 86,5 | 86,5 |
| h1alsex | 85,3 | 85,3 |
| h1pemey | 94,2 | 94,2 |
| h1caley | 94,2 | 125,4 |
| h1iscey | 127,5 | 96,5 |
| h1odcy | 96,4 | 96,5 |
| h1alsey | 95,3 | 95,3 |
The dcuids 126 and 127 are no longer used.
The data from the PEM models were slow in recovering, I eventually remembered that the ADC part in the model should be changed from ADC1 to ADC0 and all the internal bus selectors modified accordingly.
PCAL model changes
Rick, Sudarshan, Dave:
The PCAL common model was modified to give a more logical ordering of its inputs (in order of ADC channel number, PCAL signals first, Timing signals last). h1calex and h1caley were modified accordingly.
Attempt to pre-load the PSL ODC DAQ channel type change
Daniel, Stefan, Ryan, Dave:
It was suggested that the fix of the PSL ISS ODC DAQ data type problem (it is FLOAT, should be 32bit UINT) could be preloaded and applied whenever the PSL is next restarted. Well, in hindsight this is not possible since the new INI file was taken by the DAQ on restart. I backed out this change and verified that only the ODC channel was corrupted for a few hours when this was tried.
DAQ restarts were necessary to support the above work, a few more than originally intended due to PEM and PSL changes.
Stefan, Elli
After maintenance day today there seemed to be a lot of changes to the interferometer confirguration, so we decided to try to lock the full interferometer to check this still works. When requesting Check_IR in LSC_Lock guardian, the ALS_COMM was not finding the comm beat note. The IR buildup in the x-arm (LSC-TR_X_A_LF_OUTPUT) was 0.7 rather than 1, and guardian would not recognise the IR resonance. We suspected a bad alignment but we did the initial alignment twice (getting an x-arm buildup of 1 at the input_align step) and repeating the initial alignment didn't help. In order to push on to ALS_DIFF, Stefan was lowering the LSC power scale from 3 to 2W (IMC input power stayed at 3W) so that the guardian would rocognise the comm beatnote.
Once IR comm and diff beat notes were found, we brought the LSC power scale back 3W and let DRMI lock. DRMI would catch lock but soon after the ASC loops engaged they would break the lock. PRC1 broke the lock twice, so Stefan turned off the PRC1 loop, and could keep DRMI locked for a bittle longer, but then SRC2 took it down. These are the two pointing loops.
Nutsinee, Elli
This morning we visited end-x and end-y to get the HWS working there. We can remote login to the h1hwsey and h1hwsex computers and the relevant software is installed. Daniel connected camera and frame grabber enable switches in the Beckhoff system (alog 17176). We confirmed cameras and frame grabbers turn on when requested by epics. We can run serial_cmd and take. On Thursday we will check alignment of the green beam onto the cameras.
J. Kissel, S. Dwyer
Since we now believe the end station ISC and ALS front-end models and their respective settings are pretty stable, we've turned on the SDF monitoring systems. So far, the only channels I've *removed* from the monitoring system are
${IFO}:ALS-${ARM}_CAM_ITM_PIT_POS
${IFO}:ALS-${ARM}_CAM_ITM_YAW_POS
${IFO}:ALS-${ARM}_CAM_ITM_SUM
${IFO}:ALS-${ARM}_WFS_SWITCH
where the first three are actually the camera's software output being constantly jammed into an EPICs setting (so it's not really a setting, just a monitor); and the the on-off switch for the green WFS, which should be controlled by Guardian. I would post a screen shot of the DIFF screens, but there are no DIFFs, so it's not very exciting.
I've also edited (and committed) the
/opt/rtcds/userapps/release/sys/common/medm/SDF_OVERVIEW.adl
screen to (a) include the new ALSEX and ALSEY, and (b) change the DCUIDs of ISCEX and ISCEY, which are now one DCUID higher to make room for the new ALS models.
Details:
Steps to get these set up:
(1) Burt restore these systems to 12p PDT last night to ensure that we have good -- or at least something that might be good -- settings to use as our starting point.
(2) Create / update safe.snaps in the userapps repo using makeSafeBackup, e.g.
makeSafeBackup als h1alsex
For the record, these files live in:
/opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscex_safe.snap
/opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscey_safe.snap
/opt/rtcds/userapps/release/als/h1/burtfiles/h1alsex_safe.snap
/opt/rtcds/userapps/release/als/h1/burtfiles/h1alsey_safe.snap
(3) Ensure the safe.snap in the burt folder of the target area for each model is a softlink to that repo file.
]$ ls -1l --color=auto h1{als,isc}e*/*epics/burt/safe.snap
Feb 3 10:42 h1alsex/h1alsexepics/burt/safe.snap -> /opt/rtcds/userapps/release/als/h1/burtfiles/h1alsex_safe.snap
Feb 3 10:43 h1alsey/h1alseyepics/burt/safe.snap -> /opt/rtcds/userapps/release/als/h1/burtfiles/h1alsey_safe.snap
Mar 10 17:36 h1iscex/h1iscexepics/burt/safe.snap -> /opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscex_safe.snap
Mar 10 17:36 h1iscey/h1isceyepics/burt/safe.snap -> /opt/rtcds/userapps/release/isc/h1/burtfiles/h1iscey_safe.snap
Note, this took some effort, because for some strange reason the ALS auto-generated target folders had different permissions (controls only writeable) than the ISC auto-generated target folders (which had controls and controls group writeable). A little logging in as controls and chmod g+w on the folders did the trick.
(4) Use /ligo/cds/userscripts/sdf_set_monitor.py to add a "1" to the end of each burt setting line,
]$ sdf_set_monitor 1 /opt/rtcds/userapps/release/als/h1/burtfiles/h1alsex_safe.snap
(5) Load the new safe.snap into the front end as the accepted table of good values but hitting "LOAD TABLE ONLY" on the SDF_RESTORE overview screen for each model.
(6) Find a few channels that you want to ignore, and remove them from the table,
]$ sdf_set_monitor -c - 0 h1alsex_safe.snap
(channel name, hit return)
(channel name, hit return)
(hit ctrl+d to send'er)
I recall Jonathan / Dave / Betsy working on a different way to use this script, or another script entire that accepts the chanels themselves instead of a file, but we couldn't find a log, and Jamie the python pro was sitting next to me, so he made a few changes to the script and we did it this way. The changes aren't committed, because the script doesn't live in a repo.
(7) Have Sheila test your ability to update the database to accept new changes by having her change a few now-monitored settings, or have Stefan test you by burting an older file and changing vitrually all the settings from what you've stored. Do so, from the SDF OVERVIEW for that model, and at the bottom, for
FILE TYPE SELECTION chose EPICS DB AS BURT
FILE OPTIONS SELECTION chose OVERWRITE
hit SDF SAVE FILE
(wait a second, watch for the "TIME" channel, H1:FEC-${DCUID}_SDF_RELOAD_TIME, to report a modified SDF file)
hit LOAD TABLE ONLY
(8) commit your newly minted SDF monitored safe.snaps to the userapps repo.
Note, I know this will be fixed in the RCG 2.9.1 bug fix / upgrade, but the H1:FEC-${DCUID}_SDF_DIFF_CNT -- the number of differences before the loaded SDF table and what the front end has as settings has a hard limit at 40, even though there may be more than 40 differences. I experienced this when Stefan pushed a burt restore of iscex and iscey. Many channels were different than when Sheila told me was a good time -- many (or at least one) more than 40 channels were different. We didn't know whether we liked midnight last night or an hour ago, so we just went with last night.
Last night we saw a familiar bit of excess noise return -- a double-hump feature between 500Hz and 1400Hz, see first plot. This noise had been intermittently observed up until March 3 (see Gabriele's discussion here) -- after this we were successful in closing the corner station ASC loops in low noise, and we thought we had driven it away. But last night it was back.
The noise decays over a timescale of tens of minutes, which makes us think of a slow alignment loop that drags the IFO towards an operating point where the noise that is causing the bump doesn't couple to DARM. Also the intermittent appearance of the noise makes us think that the coupling depends on the initial alignment, or at least something that changes from lock to lock.
The only alignment loops that were changing on the timescale of the noise last night were the YAW loops in the IMC. The second plot attached shows a bandpassed trend of DARM_IN1 (top) and a trend of IMC-DOF_3_YAW, for the two hours as the noise disappears.
The third and fourth plots are a two-hour trends of other ASC signals (same timescale as the second plot). The corner station ASC (third plot) and the ETM loops (fourth plot) arrive at their preferred output values fairly quickly, much faster than the decay rate of the noise.
Summary:
It's the BS butterfly thing at 2449Hz, down converted by the OMC dither at 3300Hz to make 851Hz line, and then the OMC LSC feedback shakes the OMC length at that frequency finally upconverting low frequency OMC length signal into the OMC transmission at around 851Hz.
Details:
The twin peaks are centered at 851Hz (top). Blue is when it was with a hump, red is when it died down. You can also see that the line at 2449Hz was a factor of 30 or so larger when there was a hump.
According to Dan, 2449Hz is the BS butterfly mode, and 851 Hz is the difference between the BS butterfly and 3300Hz OMC length dither.
On the bottom plot, the OMC PZT2 monitor shows a huge 851Hz line, and this line was also about a factor of 30 larger when there was a hump. This is the PZT used for OMC length control.
On the second plot, I shifted the top plot such that the frequency axis starts at 851Hz, and on the bottom plot I put the OMC length error signal. This shows that the features match up, and the width of the half-hump roughly agrees with that of the OMC length.
These things all mean that we're unnecessarilly shaking the OMC length at 851Hz with too large an amplitude, and that the noise will probably go away if we notch this line somewhere in the OMC length control path. We can also more aggressively low-pass the OMC length feedback.
There is now a 6th-order butterworth bandstop filter in the input to the OMC LSC demodulation for 2.4k-2.5k. There are also additional butterworth rolloff filters 1k in the demodulation products (OMC-LSC_X_SIN and COS) in case any of the other high frequency bumps in the spectrum are due to the OMC length drive.
Ryan Fisher, TJ Massinger: We added an ODC channel to the h1omc.mdl, requiring changes to the library parts: omclsc.mdl and omc.mdl We added an ODC channel to the h1asc.mdl, requiring changes to ASC_MASTER.mdl (With Kiwamu and Stefan) We also added a matrix to the ASC_MASTER.mdl to allow for selection between AS_A and AS_B for the AS45Q PIT signal used for test mass damping. The output matrix is called OUTMATRIX_TESTMASS_DAMP. We changed the configuration of the LSC ODC to account for the LSC/OMC model restructuring from alog 16893. These changes were made to the lsc.mdl common library part. Channels that no longer exist in the LSC model were terminated inside of the ODC block and the bits were set to default to 1. We also added the ODC_CHANNEL_OUT to the annotation block to automatically record the ODC channel to the frames. We made several changes to h1odcmaster (with Duncan Macleod): All SIM (simulated) inputs were removed as they are no longer needed for testing. The IPC links for ASC, OMC and LSC ODC channels were added. The RFM links from h1iscex/ey for the X and Y ODC channels were added. cdsEzCaRead links were used to read in Guardian states from ISC, IMC and ODC. The model was converted to using a common library part for the ODC MASTER channel calcuation. This library part includes the use of the new cdsEpicsOutLong parts to record the EPICs values as 32 bit unsigned integers. Duncan M. updated the ODC MASTER MEDM screens to agree with the changes made to h1odcmaster.
Cleared HAM2, ITMX, BS, ITMY, ETMY
Scott L. Ed P. Chris S. Cleaned 57 meters of tube towards X-1-5 double doors. Problems with the generator slowed the start this morning, John and I will investigate more tomorrow morning. Busy maintenance day and 3rd IFO duties prevented us from looking into the generator issues. Fortunately we are at the only location that we have electrical power in the X arm enclosure and were able to relocate the extension cords to continue cleaning. Continuous monitoring of the beam tube pressures by the control room.
Beam tube cleaning scheduled from 7:30 - 4 07:18 Cris into LVEA 07:30 Karen into LVEA 07:30 Peter into H1 PSL enclosure ~8:20 Corey to squeezer bay for 3IFO work 08:26 Sudarshan to end Y for PCAL work 08:57 Bubba and Jodi to move 3IFO parts from corner station to mid stations 08:58 Jim W. running measurements at HAM6 09:01 Filiberto to Y arm spool to remove electronics from test stand 09:05 Hugh and TJ cycling HEPI pumps 09:07 Richard to end stations 09:08 Rick, Ed and Jeff B. to H1 diode room to reset watchdogs 09:08 Betsy to LVEA west bay for 3IFO work 09:13 Daniel changing Beckhoff code 09:20 Peter transitioned LVEA to laser safe 09:21 Jim B. to LVEA to help Filiberto 09:27 Travis to LVEA west bay for 3IFO work 09:34 Jeff B. and Andres moved dust monitor in diode room off floor 09:39 Hugh and TJ done cycling HEPI pumps 09:39 Jamie restarting guardian machine 09:46 TJ to end stations to restock cables 09:46 Jim B. back 09:47 Peter K. to H1 enclosure 09:47 Rick to end Y 09:49 Elli to end stations to work on HWS cameras 09:53 Hugh to HAM1 to read HEPI load cells 10:09 Dave restarting H1 ODC master 10:10 Hugh done at HAM1 10:14 Peter and Matt done at H1 PSL 10:15 Betsy and Travis done 10:17 Keita to squeezer bay to talk with Corey 10:18 Travis and Betsy starting TFs on SRM, SR2 and SR3 10:38 Richard to squeezer bay to talk to Corey 11:03 Karen to end Y, Cris to end X 11:15 Gary and Jodi at mid X 11:27 Dave restarting end X ISC model 11:38 LN2 truck went through gate 11:51 Mayflower Metal delivering metal recycling container 11:57 Informed by Kyle that pumps are running on HAM1, HAM2 and BSC3 12:02 Peter transitioning the LVEA to laser hazard 12:12 Peter done transitioning the LVEA to laser hazard 12:20 ASC model changes done, Ryan starting on OMC, LSC, ODC master 12:24 Beckhoff end Y PLC3 EPICS invalid, Daniel restarted IOC 12:32 Karen leaving end Y 12:53 Praxair truck is ~ 15 minutes away 12:42 Corey to ISCT1 to take pictures 12:44 Jeff K. recompiling, restarting H1 OMC model for Ryan 12:52 Corey done at ISCT1 13:11 Cristina driving forklift from LSB to warehouse and staging building 13:11 Filiberto to mid Y 13:13 Rick and Sudarshan to end Y to work on PCAL 13:39 Jeff K. and Ryan resuming work on model changes for ODC 13:39 Betsy to LVEA 13:45 TJ to LVEA to hang cables in racks 13:48 Corey to squeezer bay 3IFO 13:48 Greg to LVEA to put parts on racks by HAM6 and look for filters to reuse from eligo chillers 14:10 Hugh taking HAM1 HEPI down to take safe.snap 14:14 Filiberto back from mid Y 14:19 Peter and Ed to H2 PSL enclosure to work on Livingston PMC 14:29 Travis and Betsy done running TFs 14:37 TJ done 14:57 Rick and Sudarshan done, Dave restarting models at end Y 15:07 DAQ restart 15:56 Corey out of LVEA
In order to expedite their removal, I am letting them run 24/7 until not needed (rest of this week?)
J. Kissel I've compiled, installed, and restarted the h1calcs front-end code to absorb the changes described in LHO aLOG 17114 this morning. Most of the changes require brand-new custom MEDM screens (yaaaayyyy), so I'll post more of how the infrastructure has been filled in once I have some screens to show. For now, I've confirmed that the DARM calibration stuff is still ON and the input for the slow optical gain correction is turned OFF, and NOT changing the DARM calibration.
A couple of EtherCAT software problems were address today:
The h1asc awgtpman is a special version which will allow the full 64 test points to be set. In order to do this, the awgtpman_h1asc.cmd script (generated by RCG) was replaced with a hand-edited version which starts up awgtpman-2.9-asc-special (instead of awgtpman) and also includes a -A option to the command line. If the h1asc model is rebuilt, this will be undone by the RCG. Coordinate model rebuilding and restarting with Jim Batch in order to ensure the modified awgtpman is running. This is a temporary solution to allow commissioning to continue where more testpoints are needed. We will decide on a complete solution for the next RCG release.
The two plots show the vacuum gauges at either end of the X1 module for the past 100 days.
The features shown are due to large gate valve closures at the corner station. PT144 is next to the corner station while PT343 is at the Mid station.
We left the IFO on a DC lock last night. Attached is the spectrum. Two nights were notable: 1) The hump around 800Hz (apparently known as "twin peaks") was bigger than ever. Interestingly it was NOT there at all during earlier locks during the day. The only thing we know we did between the locks was a complete initial alignment. Also, over the period of about 1h, the hump completely disappeared. 2) The low-frequency noise (20Hz to 100Hz) is significantly less stationary now. (This did not improve over time.)
ASC showing up in displacement / roll mode ringup a) The first plot, middle graph, shows coherence between ASC loops and DARM. CHARD shows up very prominently - it's narrow stop-band get imprinted on the coherence. The same noise is also seen in IMP1_P, but it is likely just a witness, since it shares sensors with CHARD. Bottom lime: CHARD needs a proper low-pass. b) The first plot, bottom graph shows some coherence between the 20Hz-100Hz excess and ASC_MICH_P. ASC_SRC2_P also shows it, but might again just be a witness (it uses AS_C as error signal). That excess BTW is all glitches. c) The roll mode was seen to ring up over the 2h lock. Interestingly both ETMX and ETMY modes seem to be ringing. Since DARM feedback only goes to ETMX, but CHARD goes to both, I again suspect CHARD. Again: CHARD needs a proper low-pass.
The bounce and roll mode ringups were due to human intervention -- Chris and I stayed a bit later last night working on the violin mode damping, and we couldn't resist changing the gains on the feedback loops, enough that things became unstable. At 09:10 UTC when Stefan took his DTT spectrum, the gains on the feedback were nonzero, which impresses bounce & roll mode noise on the M0 stage OSEMs. When the feedback gain is zero you can't see these modes in the top stage OSEMs -- this makes identifying the optic that's bouncing or rolling a challenge. It may have been only one ETM was rolling, or an ITM (although I suspect both ETMs thanks to the ASC loops, as Stefan says.)
The attached images from the summary pages show the roll mode increasing around 09:40 UTC, when we realized our feedback was hurting more than helping. After this I zeroed the gain on the DARM --> ETM M0 feedback and the mode stayed the same height for the rest of the night. (When the noise got very bad after 11:00 UTC the bounce mode stayed pretty much the same.)
That said I would not argue against some roll-off filters for CHARD!
The attached screencaps show the roll mode damping setup that has worked well for us, tonight. (ETMX gain 10000/ETMY gain -600)
The corner whitening filters were not switching, with an error message invalid data chn and readback different for Refl 9 and refl air 9, and for POP we had an error message invalid data for channels 1, 2, 3 and 4.
Daneil fixed this by going to the system manager for the corner, finding the X gateway under device 1, corner MSR L0, Corner MSR L8_9 Vertex, going to the Online tab, and clicking on several of the buttons under state machine (Init, pre-op, safe op, op) in an order that seemed random to me.
I am just writing down the solution to this problem so that we can look it up next time it happens.
This happened again, we repeated what we did, on the Corner Chassis 2 L-2 (X-gateway) under Online the current state was safeop. We hit a bunch of buttons (including clear error) until it arrived in OP. We repeated this for L-1, which was also in safe OP.
An alternative solution would be to restart h1ecatc1