To check for possible spectral lines in the H2 One Arm Test due purely to the aLIGO DAQ / electronics, I picked a couple of channels that appeared very quiet in Greg Mendell's Fscan plots for SUS channels and applied the same spectral averaging procedure as used in previous studies of the  OAT feedback channel.

Many detailed spectra are posted here, with a sampling attached to this report. As can be seen in the first column of plots at the above link, the first channel examined, H2_SUS-ETMY_L1_OSEMINF_LL_IN1_DQ, has an unsurprising comb of 60-Hz harmonics (indicated with 'A' in the plots), with odd harmonics much stronger than even harmonics. In addition, though, it has an even stronger comb of 56.8407-Hz harmonics (indicated by 'B' in the plots, with two aliased reflections in the 900-1000 Hz band indicated by 'R' in the plots). This latter comb was previously seen in the OAT feedback channel and showed correlation with EY seismometer, accelerometer, magnetometer and microphone channels.

I tried looking at the corresponding ITMY chanel, H2_SUS-ITMY_L1_OSEMINF_LL_IN1_DQ, and found the 60-Hz harmonic comb to be lower and the other comb to be absent. The 2nd column of plots at the above link shows the spectra for the ITMY channel using the same scale as for the ETMY channel. The 3rd column of plots shows the ITMY channel with a zoomed-in vertical scale to look for evidence of other spectral lines. With the exception of a few lines that appear only on August 25, no other sharp structure above 10 Hz is apparent (although there are broad bumps here and there, for example, near 400 Hz).

In the  table of plots, each spectrum for each channel is shown for three cases: 1) individually for the days of July 16, 20; August 4, 22, 25; 2) individually for the days of August 28, 31; September 7, 13; and 3) averaged together over all the days of July 16, 20; August 4, 22, 25, 28, 31; and September 7, 13. These are all the days for which enough "science mode" OAT data was produced to warrant Fscan SFT production. Click on thumbnails to get pdfs.

Jeff Kissel confirmed that both of these channels were indeed connected to OSEM sensors during the One Arm Test, but nominally the full band above 0.5 Hz should have been dominated by sensor noise, as shown in this plot he provided.

The spectra attached to this report show some examples of the above for the full 0-1000 Hz band, where approximately 200 half-hour SFTs (0.5 mHz binning) have been averaged together with inverse noise weighting. 

I found the distribution manifold valves closed at HAM3 this afternoon while checking the lines.  I doubt anyone would have operated those valves so I must accept the responsibility for leaving them closed.  This does put HAM3 at some level of risk.  If there were a leak there, it would be below the leak found on HAM4 which at ~120psi was losing about 1/2psi per day assuming I only failed to open them at my last log book entry when the system then endured a week long pressure dwell at 125psi.

Upon opening the valves, I got an air bubble out which tripped the Pump Stations.  Pumped a little more fluid into the system and got it back stable at 26 psi.  The Servo is On and this pressure pushes the drive to 55hz.  Max is 60.  I've done a thorough hands on inspection of HAM3 and see/feel no leaks.  The Mezzinine fluid level is stable.

J. Batch, J. Kissel

After the plug got kicked out of the wall (see LHO aLOG 5794), we had to turn off all front-end computers and power cycle the IO chassis. Upon restart and restoration of the front end computers, I launched off a DTT session transfer function hoping to resume transfer functionon the TMS. However, as soon as I started, I noticed that my excitation would drop out intermittently. UH OH. All lights on the GDS_TP screen showed green. 

Jim to the rescue!! He immediately recognized the problem (from my verbal story only!) to be that there are too many awgtpman processes running on the front end. Like it's happened before or something.

He confirmed the problem by logging into the h1susb6 frontend (on which hsustmsy lives), and running
controls@h1susb6 ~ 0$ ps aux | grep awgtpman
This revealed duplicated invocations of awgtpman *for each model* (not just TMS).

This was quickly and easily resolved, with a 
sudo pkill awgtpman
which killed all of the awgtpman processes. Monit -- the program running on every front end which monitors various important processes, ensuring they're up and running -- then restarted only one process properly.
This was confirmed by another grep of aux,
controls@h1susb6 ~ 0$ ps aux | grep awgtpman
root     28813  0.2  3.1 279784 190428 ?       Ssl 16:04   0:04 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1susetmy -1 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1susetmy.log
root     28823  0.3  3.1 279540 190132 ?       Ssl 16:04   0:05 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1sustmsy -1 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1sustmsy.log
root     28835  0.2  3.1 279408 189904 ?       Ssl 16:04   0:04 /opt/rtcds/lho/h1/target/gds/bin/awgtpman -s h1iopsusb6 -4 -l /opt/rtcds/lho/h1/target/gds/awgtpman_logs/h1iopsusb6.log
controls 31462  0.0  0.0   6156   412 pts/0    S+   16:32   0:00 grep --colour=auto awgtpman
controls@h1susb6 ~ 0$

We logged into h1seib6, h1pemey, ns h1susauxb6, also showed the same symptoms -- and we rectified the problem.

WHY DID THIS HAPPEN?

/etc/rc.local is a local start up script that's run only when the computer is hard-booted/power-cycled, like this afternoon -- which we rarely happens, believe it or not (typically it's just the front-end process that gets restarted). This very low-level shell script is hosted on the h1boot server, so a change to it immediately gets propagated to every front end. This script had recently been modified to invoke all models' front-end-process startup script on that given front end BEFORE Monit is turned on. The problem is that both the startup scripts and Monit start awgtpman processes, but they do it in *different*, *independent* ways. Regardless of the order in which monit or the model start scripts are invoked, two awgtpman processes would get started.

The change to the /etc/rc.local script is a temporary fix. The motivation for the fix is unknown to Jim. *COUGH*. What needs to happen is a permanent change to the model start scripts,such that they call awgtpman in the same way that Monit does. Then, because Monit checks for the existence of an awgtpman started by its own method, it will not fire off a new process. This requires a change to the RCG code generator, which writes the front-end model startup scripts. Such a change should then be tested extensively on the DAQ Test Stand (or some other non-observatory location), then released to the sites as a tagged version of the RCG code, which is then installed at a well-determined time that is known not interfere with current activities.

Finally, I also updated the HAMISI ODC screens to include the bit label read-back.

Note that the HAMISIs don't have the ODC code in the front end - this still needs to be one.


This concludes my round of medm screen updates. 

I updated the HPI ODC channel screens to include the bit description read-backs. I also added a bit-bar to the overview screen.

While doing this I noticed that the sitemap sets different userapps variables for HPI HAM1 vs the remaining HAMs. Look at the userapps directory name.

HAM1: SITE=LHO,site=lho,ifo=h1,IFO=H1,chamber=ham2,CHAMBER=HAM2,IOP=SEIH23,iop=seih23,DCUID=54,IOPDCUID=53,IOP_NAME=SEI_H23,USERAPPSDIR=/opt/rtcds/userapps/release
HAM2: SITE=LHO,site=lho,IFO=H1,ifo=h1,CHAMBER=HAM1,chamber=ham1,IOP=SEIH16,iop=seih16,DCU_ID=49,IOP_DCU_ID=48,IOP_NAME=SEI_H16,USERAPPS_DIR=/opt/rtcds/userapps/release

I added ODC channel screens, including bit label readbacks, to all SUS overview screens. The only suspensions that still don't have ODC bits are the HAUX (Ham auxiliary).

Jason and Travis set the IAS PLX in the HAM3 South spool.  After coarsely adjusting yaw and pitch of the reflected beam back into the PLX, we are now fine tuning those DOFs via the pushers and the pitch masses.

Today, with the approval of the resident IO rep Chris, I swapped in the MC3 top mass blade tip EQ stops (a necessary retrofit with an ECR (has a number, yada yada).  To date, the MC1 and MC2 still need them.  Other HSTSes have the correct screw.  These are on the to-do list for next week.

Kiwamu had a DC Power Supply box (D1002932) he wanted installed inside the IOT2L.  After ordering a few parts, I went about installing the box; this required installing support brackets.  These support brackets required moving the entire internal cable tray a few inches toward the doors (which isn't trivial with all the cables cable-tied down).  After a bit of grief and sweat, the box is installed and ready to go. 

Pictures for IOT2L are in ResourceSpace here.

15:08 DAQ restart to add back the EX DUST monitor channels.

All front-end computers were restarted at the end Y after someone tripped over the power cord for the I/O chassis power supply, AGAIN.

crashed last night

Ryan Fisher, Jeff Kissel, Stefan Ballmer

We started added the epics bit label strings to the ODC medm screen. Ultimately these strings are intended to be initialized my the front end model to keep track of long-term ODC bit assignment changes. For now we populated them by hand (see Ryan's elog).

So far we modified the medm screens for the BSCISI (BSCISI/ISI_CUST_CHAMBER_ODC.adl) and added the SUS quads (SUS_CUST_QUAD_ODC.adl). For the quads we modified the OVERVIEW screen to include a button to the ODC screen, and reduced the number of ODC bit indicators to the bits 0 to 12 that are currently used.

Many more screens need to be done: SUS triples and auxiliaries, HAMISI and HEPI.

Ryan Fisher, Stefan Ballmer

We have set the EPICS records for the labels that describe the ODC bits for the SUS and ISI models currently running in H1.*  The labels provide a short description of what every bit in each ODC channel represents, for each ODC channel.  The higher order bits in the ODC channels that do not have labels are currently unused, but may be used in the future.

These records were set using scripts located in /ligo/home/ryan.fisher, and attached to this entry.  

* Excluding H1:SUS-TMSY_ODC_BIT* for now.