Jenne told me this afternoon that the laser had tripped. The crystal chiller indicated an error with flow sensor 1. All the flow sensors in the chiller are the vortex style and have no moving parts and are not likely to fail in the same manner as the old turbine ones. My guess is that after ~31k hours of service, it's time to start scheduling a time where we switch the chiller out for the spare and send this unit back to Termotek for servicing - who recommend that the servicing be done annually. This is the second time in about 4 days that the laser has tripped because of this problem.
h1pemey
Dave:
I modified h1pemey to receive the muxed OMC DCPD demodulated channels from h1omc, and demux them into 8 dolphin channels. These will be received by the h1susetmypi model in a later mod. The h1pemey model now has the same mods as were applied to h1pemex last week.
Vacuum Controls
Carlos, Patrick, Dave:
Patrick made new vacuum controls changes today. Some LLCV channels changed names, and new INI files were generated. I copied the running minute trend files from old-name to new-name on both h1tw0 and h1tw1. I created the new edcu file, which was renamed from H0EDCU_VE.ini to H0EDCU_VAC.ini
Target files, autoburts and snap files are being renamed from h0veXX to h0vacXX.
Beckhoff computers were renamed from h0veXX to h0vacXX. The NTP settings were verified, but it looks like the clocks remain unsynced.
Slow Controls Beckhoff Computers
Carlos, Jim, Dave:
The NTP settings on h1ecat[c1,ex,ey] were changed from name based to IP address. This caused a resync of the time on these machines (were 4 seconds out).
DAQ Restart
Dave:
DAQ restart to incorporate the h1pemey new model and the VAC new ini.
For some reason the high voltage supply for the EX ESD was off this afternoon. Tega and I turned it back on. It looks like the same might be true for EY, but we havent' gone out there yet.
Ryan, Carlos, Jim, Dave:
The new ZFS file system on h1fs0 which is NFS exported as /opt/rtcds became 97% full recently. This is a fast (SSD) but relatively small (1TB) file system. The reason for being fast is to prevent front end epics freez-ups which we saw last year when the disk was busy syncing to its backup machine.
With this being a snapshotted ZFS file system, freeing up space is not as easy as removing and/or compressing files. First all the snapshots must be destroyed to free up the file blocks and allow file modifications.
Process on both h1fs0 (primary) and h1fs1 (backup) was:
1 destroy all snapshots
2 remove all old adl files in the rcg generated dirs /opt/rtcds/lho/h1/medm/model-name/
3 tar and compress old build areas in /opt/rtcds/lho/h1/rtbuild*
4 delete all non-rcg generated target_archive dirs (mostly from 2011-2012)
5 delete all rcg generated target_archive dirs with the exception of the most recent four dirs
2&3 freed up only a modest amount of disk space, they were mainly to clean up clogged directories.
4&5: prior to doing this we verified these old target archives were backed up to tape. Stop 5 freed the lion's share of disk space, about 600GB.
The end result is the amount of free space went from 53GB to 685GB, disk usage went down from 97% to 28%.
At the end of the procedure, a full ZFS-snapshot sync of h1fs0 to h1fs1 was done manually, and the hourly sync was turned back on.
Transition Summary: Title: 05/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) State of H1: IFO unlocked. Work on PSL is ongoing. Tuesday Maintenance day Commissioning: Outgoing Operator: N/A Activity Log: All Times in UTC (PT) 14:00 (06:00) Peter – In the H1 PSL enclosure 14:30 (07:30) Jeff – Replace network cable to PSL dust monitor #102 15:00 (08:00) Start of Tuesday maintenance day 15:00 (08:00) Christina & Karen – Cleaning in the LVEA 15:14 (08:14) Sprague – on site to remove pest control boxes 15:30 (08:30) Hugh – Restarting HAM models (WP #5874) 15:48 (08:48) Filiberto, Leslie, & Manny – Going to pull cables for the access system 15:49 (08:49) Hugh – Finished with model restarts 15:54 (08:54) N2 delivery at both end stations 16:00 (09:00) Patrick – Update Beckhoff code for vacuum controls computer (WP #5875) 16:04 (09:04) Kyle – Going to both end stations VEAs. 16:10 (09:10) Jim B – Restarting H0EPICS to apply security patch. 16:10 (09:10) Bubba – Replacing faulty check valve air comp at Mid-X (WP #5876) 16:15 09:15) Peter – Out of the H1 PSL enclosure 16:17 (09:17) Filiberto – Power off Vac rack LX, & LY to check grounds 16:35 (09:35) Christina & Karen – Cleaning at End-X 16:35 (09:35) Chandra – Going to Mid-X to work on CP4 16:36 (09:36) Kiwamu – Going to HAM6 area to check S/Ns on electronics 16:41 (09:41) Kiwamu – Out of LVEA 16:43 (09:43) Jim B. – Finished with H0EPICS restart 16:45 (09:45) Cintas – On site to service rugs and garb 16:50 (09:50) Ed – Going to End-Y to terminate cables in the VEA 16:59 (09:59) Bubba – Back from Mid-X 17:05 (10:05) Christina & Karen – Finished at End-X and are going to End-Y 17:06 (10:06) Kyle – Back from the End Stations 17:10 (10:10) Filiberto – Out of the LVEA 17:18 (10:18) Hugh – Going to both End Stations to check HEPI pumps 17:45 (10:45) Hugh – Finished with HEPI checks at End Stations 17:50 (10:50) Hugh – Checking CS HEPI pumps 17:58 (10:58) Hugh – Finished with CS HEPI pump checks 18:04 (11:04) Nutsinee – Going to LVEA to take pictures near HAM4 18:13 (11:13) Nutsinee – Out of the LVEA 18:44 (11:44) Vendor on site to deliver drinks 18:54 (11:54) Gerardo – Going to End-Y VEA to measure for EitherCat cables 19:29 (12:29) Gerardo – Back from End Stations 19:49 (12:49) Manny & Leslie – Going into the LVEA to brackets for access sys on IO cabinets 20:27 (13:27) Bubba – Going into LVEA to look at N2 LTS system 20:28 (13:28) Gerardo – Going to Mid-Y to shim CP3 and CP4 20:58 (13:58) Bubba – Out of the LVEA 21:04 (14:04) Manny & Leslie – Finished in the LVEA 21:11 (14:11) Sheila & Jeff – Adjusted the ALS wrong polarization from 43% to 7% 22:10 (15:10) Sheila – Going to End-X 22:20 (15:20) Manny & Leslie – Going to both End Stations to install access controls bracket 22:39 (15:39) Chandra – Going into the LVEA to the walkover bridge 22:45 (15:45) Chandra – Out of the LVEA 22:48 (15:48) Chandra – Going to both end stations to make measurements 22:48 (15:48) Gerardo – Going into the LVEA – climbing around HAM6 End of Shift Summary: Title: 05/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) Support: Incoming Operator: Cheryl Shift Detail Summary: Maintenance Tuesday. Work continues on recovery of the PSL.
Attached are measurements of the frequency stabilisation taken from this morning. Noise spectra and the transfer functions are plotted for common gains of 20 dB and 22 dB, and fast gains of 22 dB and 8 dB respectively. At lower frequencies the noise is lower with the common gain at 20 dB. However this results in a slight decrease in the unity gain frequency. With these settings the UGF is ~450 kHz. With the common gain at 22 dB, the UGF is ~650 kHz. Anecdotally it seems that the frequency servo is more glitchy with the common gain at 22 dB. Improving the cross-over by looking at the spectrum and flattening it and adjusting the fast gain, improves the flatness of the spectrum but then increasing the common gain does not push the noise floor as low as the two cases illustrated.
Related log(s): 21131
Summary. - Today, I have updated the first anti-whitening filters of the OMC DCPDs in the OMC digital front end model in order to make them more accurately match the actual analog circuits. The attached show screenshots of the foton filters before and after the update.
Impact. - The new OMC DCPD responses are different from what they used to be by 1% at 10 Hz in magnitude and almost no change at 100Hz and above. The O2 DARM model must take this update into account. Note that even though the qualitative behavior of the mismatched anti-whitenings is similar to the large bias we have been seeing in the DARM response (for example 24569), they do not explain the bias (which is about 10% at 10 Hz in magnitude).
DCPD balancing. - According to my calculation, the balancing should be good at a 0.1% level without introducing an artificial imbalance in the OMC front end. So I removed the existing artificitial imbalace (-0.3%) and update the OBSERVE and DOWN SDFs. However, the imbalance should be experimentally double-checked and possibly re-adjusted.
Today, while Jenne and Sheila were re-tuning the OMC dither loops, I looked at the balance of two OMC DCPDs. The attached shows the null and sum spectra, and ratio between DCPD A and B.
The isolation between null and sum is as good as 66 dB, according to a injected line at 12 Hz (OM3 pitch by Jenne and Sheila). The two PDs match each other with a 0.1% level accuracy in magnitude and 0.1 deg level for the phase. This seems good enough for the moment. Though, we should check the responses above 100 Hz at some point.
I have removed the power stabilisation photodiode box from the table for testing/debugging. Please do not attempt to engage the power stabilisation.
The h0epics computer was patched and restarted. Dust IOC's for the diode room, lab, lvea, psl, ex, and ey were restarted. The dewpoint monitor IOC was restarted. The protocol file for the psl IOC was updated to handle three digit ID numbers more reliably.
WP 5874
SEI update summarized in T1600090. This includes :
o GS13 signals storing rate changed. ECR E1600077
o STS signals at the senscor channel point not recorded anymore -> ECR E1600077
o blend glitch-> ECR E1500456
o 10 secs delay on coil driver status before triggering watchdog-> ECR E1600076
o remove user dackill-> ECR E1600042
o Cumulative saturation count->ECR E1500325
Thanks to Arnaud for this round up.
Required updating the isihammaster.mdl, guardian isi manager.py and the HAM ISI Overview medm.
See attached for full log.
State of H1: DRMI locked! Then DRMI lost lock, FSS went into oscillation
H1 Configuration:
Initial Alignment / Locking:
Activities:
Other H1 activities / configuration:
The fast gain of the FSS was only set to -8 dB in order for Jason and I to proceed with other laser related work without having to worry about it re-acquiring and oscillating. It is not the default value. I will re-measure the cross over today and update the log.
CP4 is back on scale and in PID mode. Experimented with PID loops using CP2.
[Sheila, Jenne, Cheryl]
We found that many optics' drivealign sliders were in the wrong place after last week's RCG upgrade - these are things that we specifically keep out of the SDF system, since they are often changing by small amounts. But, this means that they aren't kept up to date in the safe and down SDF files. Anyhow, they have been restored, and we were able to lock and align the green arms, and were also able to lock the Xarm in IR for the input align step of initial alignment.
Cheryl will attempt the rest of the alignment sequence, so we have a nice IFO for tomorrow.
Sheila, Jenne, Cheryl
Xarm IR locking was difficult again today, and we made two observations.
We have a more aggresive low pass filter in MC2 M3 than we really need, CLP300. We locked the arm twice sucsesfully with this filter off and CLP500 on instead. screenshots are attached with the digital gain set to 0.15. We didn't add this to the guardian yet, but I think its a good idea (third screenshot attached shows the trial configuration).
We are in no danger of saturating M3, but M2 saturates when we are trying to acquire. This might be the root of some of our troubles accquiring the xarm.
I've tested this new arrangement about 10 times in the last 15 minutes, and it does seem much better than what we had before, so I've put it in the guardian and committed them.
The noise eater was bad in these periods: [17:34, 17:52], [18:10, 18:39] and [19:38, now-ish] UTC, or [10:34, 10:52], [11:10, 11:39], and [12:38, now-ish] local time.
I went to the LVEA and reset it.
This probably doesn't have anything to do with Peter's dark noise measurements, but this might have caused some confusion when he was measuring the intensity noise.
Following on from last week's entry about the power stabilisation photodiodes https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27054 It took advantage of the laser being down to measure the output of the photodiodes when laser was off. The two measurements are quite different. Not just in level but also spectral shape. The spectrum posted earlier is observed when one tees the signal off and inputs it into an oscilloscope. One observation is that the isolated LEMO connectors on the photodiode box are not isolated from the table surface. Neither are the photodiode housings within the box.
For what it's worth, attached is the free running RIN of high power laser measured by the first loop diodes (blue, green) and FSS transmission (red) as of now.
Even though Peter and Jason decreased the light on the PDs, I had to increase the diffraction to 15+%, otherwise the PDA and PDB "AC" still rails once in a while and ruin the spectrum.
Also plotted is the dark noise of PDA projected onto RIN. This "dark" spectrum was from last evening when the laser was shut down. Since the only PDA and PDB signals we record on the frame are REL channels (i.e. dewhitened AC divided by DC), I had to look at the dark offset of PDA from last evening:
RIN projection = REL spectrum * dark offset / 2.3V
where 2.3V is the "DC" output when the red, blue and green traces were taken. I couldn't do this to PDB as that guy has almost zero dark offset.
Anyway, at least when the laser was shut down and nobody was in the room, the dark noise looked fine.
Set CP4 to manual mode at 25% LLCV open. Then installed flow meter at exhaust. Will leave as-is for a couple hours to observe changes in LL or exhaust pressure like we saw last Friday (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27057). The LL spikes may have been driven by the LLCV PID. Switching to manual mode should exploit that variable.
Just switched back to PID. Will leave the flow meter connected overnight.
I just pushed a minor upgrade to guardian to fix a small issue with the guardlog client when following node logs. The new version is r1542.
The client was overly buffering stream data from the server, which was causing data from the stream to not be output in a timely manner. This issue should be fixed in the version I just pushed.
As always make sure you have a fresh session to get the new version:
jameson.rollins@operator1:~ 0$ which guardlog /ligo/apps/linux-x86_64/guardian-1542/bin/guardlog jameson.rollins@operator1:~ 0$
Sorry, Sheila. This issue has been fixed now. Just needed to tweak the lockloss script to account for some updated guardlog arguments.
Yes, its working, thank you Jamie.
Tagged this with FRS#5469.