We noticed a strange 0x2000 daq status of the h1isiham3 which should not be the case (models were restarted, then the DAQ). So to be safe I have just restarted the h1isiham2,3 models again. I cleared the Dolphin IPC errors this raises on the suspension systems in ham2,3.

BTW, the safe.snap files for h1isiham2,3 are out of date and generate many errors when ran manually.

This morning I asked the question: 

"Who is OSEM[*] in the Alarm Handlder??"     (  *  can be 1-6 in the Alarm Handler)

Dr. Kissel handed over a crumpled Post-It note which decrypted the OSEM mystery.  Currently, to understand what a SUS alarm means, one needs this Post It note because the Alarm handler names SUS Watchdog channels as OSEM1, OSEM2, ...OSEM6 (which isn't really useful since the SUS Expert screens don't use the OSEM names; they use F1, F2, F3, Left, Right, Side).  I posted the Post It notes to the CDS SUS wiki.  [As for the actual alarm in question, it was related to OSEM5&6 (Left & Right) "Centering"; and this is most likely due to not being pumped down].

I did actually do something here.  Jeff showed me the IOP Watchdog screen, and how some of the custom medm screens on here have been changed from using "OSEM" to "useful" names.  But we noticed some sub-screens weren't updated (namely EY BSC6).  So, I updated this screen from the OPS Station.  Here are steps taken:

• Open Terminal
• Since we were logged in as ops, we had to log  in as controls with graphical capabilities to make medm changes to custom screens
  • ssh -X controls@operator0
• cd /opt/rtcds/userappsrelease/cds/h1/medm
• Open medm screen you want to edit
  • medm H1CDS_SOFTWARE_SUS_WATCHDOGS_EY.adl &       | this is screen I changed |
• Make changes to screen (here I changed the OSEM names to actual names)
• Save (on medm)
• Back on Terminal, one will want to commit this modified file to SVN
• Check SVN status & then commit
  • svn st
  • svn commit -m "changed names for OSEMs" H1CDS_SOFTWARE_SUS_WATCHDOGS_EY.adl
    • To commit one will need to login with your ligo.org account
• End

(I also changed names on the H1CDS_SOFTWARE_SUS_WATCHDOGS_TST.adl screen as well)

I added temperature/relative humidity sensors to the dust monitors in the optics lab, vacuum prep lab and bake oven room.

We restarted the h1isiham2 and h1isiham3 models. The new models have ODC parts added. The DAQ was restarted.

This closes out WP3768

Re 5807 I did another hands on inspection of all the welded and threaded joints at HAM3 and find no indication of fluid.  The servo is still running at 26psi and the reservoir fluid level has not dropped.  We will run at substantially higher pressure once the Actuators are introduced to the system but I do believe we are OK here.  I'll continue to monitor this closely.

[Deepak N., Rodica M., Chris M.]

* After Besty's realignment of the MC3 BOSEMs we found that the mode cleaner was not aligned.  Deepak checked the top stage earthquake stops, and they all seemed to be fine.  After noticing that the RT BOSEM was mis-centered by ~4000 counts we decided to recenter it.  This brought the mode cleaner much closer to flashing, but we still had to make some small tweaks afterwards.  The moral of the story is that the meaning of the alignment numbers is rather sensitive to the BOSEM centering.  I guess this isn't terribly surprising because, as Jeff K. pointed out, the magnetic field of the BOSEMs is designed to roll off quickly outside of the opening.  

* We attempted to set the polarizaiton of the recycled MC Refl beam using the HWP as mentioned in yesterday's post, but ran into a confusing inability to do so. Hmm...

J. Kissel, B. Bland

Betsy took another crack at de-rubbing the TMSY this morning. Her words which I've paraphrased: "I definitely readjusted the stiffness of the cables between the optical table and top mass, and around the top mass. But that aside, it really does look totally free -- I softly bounced, rocked and rolled it and it's definitely has a good range of motion before hitting its stops."

Since, I've taken another set of transfer functions (see attached). It looks looks significantly better than previous measurements:
- Vertical Modes are no longer double-peaked
- The most prominent Roll feature between 0.65 and 0.8 Hz has changed shape to be less complicated, lower in frequency, and dare-I-say fewer modes
- Whatever high frequency noise that appeared in yesterday's Tranverse TF has gone away again
- The second L/P mode at 0.8 Hz has moved down to 0.75 Hz
- TFs in general are a little bit cleaner
But until we get a similarly detailed measurement under vacuum, or a model we can trust, I can't definitively say "we're good." But I certainly can be convinced that what noise that's left that we see is from air-currents and ambient noise. I'll let team-TMS make the final call.

Each ODC channel has a Summary Bit that is used to indicate whether each system is in a nominally good operating state.  This bit is generated by checking that the necessary bits in the ODC channel are ON, as specified by the subsystem commissioners.  With the input from Jeff Kissel and Sebastien Biscans, we have set all of the EPICS variables needed to correctly populate the Summary bits for all active ODC channels in H1.  The EPICS variable Bitmasks are decimal numbers that correspond to a bitmask, where for each bit in the ODC channel that should be on when the system it is characterizing is in a good working state.  Here is an example:
H1:SUS-BS_ODC_COMBINE_ODC_BITMASK = 126
In binary, this is 0111 1110, which indicates that bits 6, 5, 4, 3, 2, and 1 should all be ON when the SUS-BS is operating nominally.  If you are curious what those bits mean, you can do the following:

controls@opsws1:~ 0$ caget H1:SUS-BS_ODC_BIT1
H1:SUS-BS_ODC_BIT1             Master Switch ON/OFF Status
controls@opsws1:~ 0$ caget H1:SUS-BS_ODC_BIT2
H1:SUS-BS_ODC_BIT2             DACKILL Status

These BITMASK EPICS records were set using scripts located in /ligo/home/ryan.fisher, and attached to this entry (with modified file extensions so the alog will post them).

With this completed, the EPICS description strings for the bits set and the MEDM screens reworked, we believe we have most of the active SUS and SEI subsystem ODC channels operational.  

The remaining systems for which ODCs need to be implemented under SUS and SEI are: the HAUX (HAM-Auxilliary) front ends and the HAM-ISI front ends.  The latter should be implemented with the next update from the cds_user_app svn (anticipated Mar 15).

We will be generating new safe.snap files specifically for the updated ODC EPICS settings for all models and sending them to CDS-announce for incorporation into the SVN and live systems.

To check for possible spectral lines in the H2 One Arm Test due purely to the aLIGO DAQ / electronics, I picked a couple of channels that appeared very quiet in Greg Mendell's Fscan plots for SUS channels and applied the same spectral averaging procedure as used in previous studies of the  OAT feedback channel.

Many detailed spectra are posted here, with a sampling attached to this report. As can be seen in the first column of plots at the above link, the first channel examined, H2_SUS-ETMY_L1_OSEMINF_LL_IN1_DQ, has an unsurprising comb of 60-Hz harmonics (indicated with 'A' in the plots), with odd harmonics much stronger than even harmonics. In addition, though, it has an even stronger comb of 56.8407-Hz harmonics (indicated by 'B' in the plots, with two aliased reflections in the 900-1000 Hz band indicated by 'R' in the plots). This latter comb was previously seen in the OAT feedback channel and showed correlation with EY seismometer, accelerometer, magnetometer and microphone channels.

I tried looking at the corresponding ITMY chanel, H2_SUS-ITMY_L1_OSEMINF_LL_IN1_DQ, and found the 60-Hz harmonic comb to be lower and the other comb to be absent. The 2nd column of plots at the above link shows the spectra for the ITMY channel using the same scale as for the ETMY channel. The 3rd column of plots shows the ITMY channel with a zoomed-in vertical scale to look for evidence of other spectral lines. With the exception of a few lines that appear only on August 25, no other sharp structure above 10 Hz is apparent (although there are broad bumps here and there, for example, near 400 Hz).

In the  table of plots, each spectrum for each channel is shown for three cases: 1) individually for the days of July 16, 20; August 4, 22, 25; 2) individually for the days of August 28, 31; September 7, 13; and 3) averaged together over all the days of July 16, 20; August 4, 22, 25, 28, 31; and September 7, 13. These are all the days for which enough "science mode" OAT data was produced to warrant Fscan SFT production. Click on thumbnails to get pdfs.

Jeff Kissel confirmed that both of these channels were indeed connected to OSEM sensors during the One Arm Test, but nominally the full band above 0.5 Hz should have been dominated by sensor noise, as shown in this plot he provided.

The spectra attached to this report show some examples of the above for the full 0-1000 Hz band, where approximately 200 half-hour SFTs (0.5 mHz binning) have been averaged together with inverse noise weighting. 

I found the distribution manifold valves closed at HAM3 this afternoon while checking the lines.  I doubt anyone would have operated those valves so I must accept the responsibility for leaving them closed.  This does put HAM3 at some level of risk.  If there were a leak there, it would be below the leak found on HAM4 which at ~120psi was losing about 1/2psi per day assuming I only failed to open them at my last log book entry when the system then endured a week long pressure dwell at 125psi.

Upon opening the valves, I got an air bubble out which tripped the Pump Stations.  Pumped a little more fluid into the system and got it back stable at 26 psi.  The Servo is On and this pressure pushes the drive to 55hz.  Max is 60.  I've done a thorough hands on inspection of HAM3 and see/feel no leaks.  The Mezzinine fluid level is stable.

J. Batch, J. Kissel

After the plug got kicked out of the wall (see LHO aLOG 5794), we had to turn off all front-end computers and power cycle the IO chassis. Upon restart and restoration of the front end computers, I launched off a DTT session transfer function hoping to resume transfer functionon the TMS. However, as soon as I started, I noticed that my excitation would drop out intermittently. UH OH. All lights on the GDS_TP screen showed green. 

Jim to the rescue!! He immediately recognized the problem (from my verbal story only!) to be that there are too many awgtpman processes running on the front end. Like it's happened before or something.

He confirmed the problem by logging into the h1susb6 frontend (on which hsustmsy lives), and running
controls@h1susb6 ~ 0$ ps aux | grep awgtpman
This revealed duplicated invocations of awgtpman *for each model* (not just TMS).

This was quickly and easily resolved, with a 
sudo pkill awgtpman
which killed all of the awgtpman processes. Monit -- the program running on every front end which monitors various important processes, ensuring they're up and running -- then restarted only one process properly.
This was confirmed by another grep of aux,
controls@h1susb6 ~ 0$ ps aux | grep awgtpman
root     28813  0.2  3.1 279784 190428 ?       Ssl 16:04   0:04 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1susetmy -1 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1susetmy.log
root     28823  0.3  3.1 279540 190132 ?       Ssl 16:04   0:05 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1sustmsy -1 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1sustmsy.log
root     28835  0.2  3.1 279408 189904 ?       Ssl 16:04   0:04 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1iopsusb6 -4 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1iopsusb6.log
controls 31462  0.0  0.0   6156   412 pts/0    S+   16:32   0:00 grep --colour=auto awgtpman
controls@h1susb6 ~ 0$

We logged into h1seib6, h1pemey, ns h1susauxb6, also showed the same symptoms -- and we rectified the problem.

WHY DID THIS HAPPEN?

/etc/rc.local is a local start up script that's run only when the computer is hard-booted/power-cycled, like this afternoon -- which we rarely happens, believe it or not (typically it's just the front-end process that gets restarted). This very low-level shell script is hosted on the h1boot server, so a change to it immediately gets propagated to every front end. This script had recently been modified to invoke all models' front-end-process startup script on that given front end BEFORE Monit is turned on. The problem is that both the startup scripts and Monit start awgtpman processes, but they do it in *different*, *independent* ways. Regardless of the order in which monit or the model start scripts are invoked, two awgtpman processes would get started.

The change to the /etc/rc.local script is a temporary fix. The motivation for the fix is unknown to Jim. *COUGH*. What needs to happen is a permanent change to the model start scripts,such that they call awgtpman in the same way that Monit does. Then, because Monit checks for the existence of an awgtpman started by its own method, it will not fire off a new process. This requires a change to the RCG code generator, which writes the front-end model startup scripts. Such a change should then be tested extensively on the DAQ Test Stand (or some other non-observatory location), then released to the sites as a tagged version of the RCG code, which is then installed at a well-determined time that is known not interfere with current activities.

Finally, I also updated the HAMISI ODC screens to include the bit label read-back.

Note that the HAMISIs don't have the ODC code in the front end - this still needs to be one.


This concludes my round of medm screen updates. 

I updated the HPI ODC channel screens to include the bit description read-backs. I also added a bit-bar to the overview screen.

While doing this I noticed that the sitemap sets different userapps variables for HPI HAM1 vs the remaining HAMs. Look at the userapps directory name.

HAM1: SITE=LHO,site=lho,ifo=h1,IFO=H1,chamber=ham2,CHAMBER=HAM2,IOP=SEIH23,iop=seih23,DCUID=54,IOPDCUID=53,IOP_NAME=SEI_H23,USERAPPSDIR=/opt/rtcds/userapps/release
HAM2: SITE=LHO,site=lho,IFO=H1,ifo=h1,CHAMBER=HAM1,chamber=ham1,IOP=SEIH16,iop=seih16,DCU_ID=49,IOP_DCU_ID=48,IOP_NAME=SEI_H16,USERAPPS_DIR=/opt/rtcds/userapps/release

I added ODC channel screens, including bit label readbacks, to all SUS overview screens. The only suspensions that still don't have ODC bits are the HAUX (Ham auxiliary).

Jason and Travis set the IAS PLX in the HAM3 South spool.  After coarsely adjusting yaw and pitch of the reflected beam back into the PLX, we are now fine tuning those DOFs via the pushers and the pitch masses.

Ryan Fisher, Stefan Ballmer

We have set the EPICS records for the labels that describe the ODC bits for the SUS and ISI models currently running in H1.*  The labels provide a short description of what every bit in each ODC channel represents, for each ODC channel.  The higher order bits in the ODC channels that do not have labels are currently unused, but may be used in the future.

These records were set using scripts located in /ligo/home/ryan.fisher, and attached to this entry.  

* Excluding H1:SUS-TMSY_ODC_BIT* for now.