TITLE: 07/18 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
LOG:
15:00 Chris & Bubba working on fans
15:00 Pep to EY
15:00 Richard to CER mezzanine
15:00 HFD on site
15:15 SEI_CONF take to SC_OFF_NOBRSXY, IFO still locked
15:30 Richard Marc to LVEA to move rack near HAM6
15:30 JeffB to both ends compressor rooms, done 16:00
15:45 Cheryl & Sheila to PSL to remove gig-e camera
15:45 Hugh and Richard to TCSY to look at laser trip, out 16:00
16:15 Chandra to VEAs for viewport tabs
16:30 Marc to CER
17:00 Chris to VEAs
18:00 Mike to LVEA with tour
18:00 Pep to EY
20:15 Sheila, Jian to LVEA comm excitations
20:30 Travis to LVEA
20:30 Richard to CER
21:30 Sheila, Thomas doing CARM measurement
We lubed all of the supply fans on site today and as part of the quarterly lube, I switch fans and chillers at the end stations. This was done this morning during maintenance and all settings on the chillers and air handlers currently running are duplicated. There may be a slight lag in the temperature catch up. Should be < .75 F.
Keita and Daniel pointed out that one way to check if all of our range loss is due to subtractable jitter noise or something else, is to compare cleaned data from before the earthquake to after.
In the attached plot, the blue trace labeled "Orig" is the raw data from the 15th of July, before we (accidentally) switched to using only one DCPD. The red-orange trace labeled "CleanPostEQ" is from that same 15 July time, but has been cleaned. The yellow trace labeled "CleanPreEQ" is cleaned data from the 5th of July, just before the big Montana earthquake hit us.
The ratios in the bottom portion of the plot are the 2 cleaned spectra compared to the raw post-EQ blue spectrum. Because of this, the % improvements in the title of the plot are misleading.
The conclusion is that we definitely have some noise below 90Hz or so that is not witnessed by the IMC WFS, the PSL bullseye, the ASC Hard loops, or the vertex length loops.
I attach here another set of spectra, where in each figure the blue is cleaned data from before the earthquake (6 July 2017 04:00:00 UTC), the red is cleaned data from after the earthquake (21 Juy 2017 14:00:00 UTC), and yellow is the difference between those 2 spectra. These traces are the same on all 3 figures.
The other two traces, purple and green, are lines overlaid to help decide what kind of slope our residual is. The figure with "f2" in the name has 1/f^2 lines, "f3" in the name has 1/f^3 lines, and "f4" in the name has 1/f^4 lines.
I'm not sure what would give us 1/f^3 noise, but it looks to me like that might be the best fit. It's definitely not 1/f^4.
Richard, Fil, Dave, Carlos, Cheryl
The two digital GIGE cameras Cheryl installed on the PSL table are now producing images.
The cameras are called h1iogige1 (10.106.0.52) and h1iogige2 (10.106.0.53).
I have reconfigured h1digivideo2's VID-CAM28 and VID-CAM29 to be IO GIGE 1 and 2. For some reason monit had VID-CAM28 commented out, I reactivated this camera. I changed the MEDM screen to show the correct names (image attached).
GigEs were not connected to medm's until after the installation was complete, so the cameras were aligned without being able to see the images, and further alignment and more attenuation are needed.
Keita pointed out that we have only been using OMC DCPD B for at least the last lock. I trended the relative gains of DCPD A and DCPD B, and it looks like we've been in this situation since 16July2017 at around 05:30 UTC.
The attached is a 5 day trend of the gains. The 2 spikes earlier are when we were either adding or removing a layer of whitening, due to the violin modes being too high. On the 16th, it looks like the guardian was started to switch the whitening, but then maybe got stopped halfway. This explains why it has looked like the shot noise was too high the last few days.
Clearly we need to write into the READY_FOR_HANDOFF OMC lock state to ensure that the 2 gains are both at their nominal values. Also, it looks like someone accepted the wrong values into the Observe SDF file, so we've been able to go to Observing with the wrong values :( No good. The safe SDF file was correct. I'll fix the Observe file.
EDIT: Looking through the guardian log file, it looks like the code gets a bit confused when you try to use these states before we're using the DCPDs for DARM. So, now if we're at DC_Readout_Transition or before (and don't need to do any gain ramping), we skip the gain changes and just change the whitening. If we're on DC_Readout or after, the change will happen as before. Either way, at the end of the state is now a check to see if the DCPD gains are equal (they should both be 1000). The new code is in the svn and has been reloaded, although not actually used.
Tagging CAL and DetChar.
The exact time of departure from this configuration is Jul 16 2017 05:24:00 UTC (1184217858) and return to normal is Jul 18 2017 18:58:00 UTC (1184439498) We (the calibration group) have recommended that any observation time within the period be marked with a CAT1 veto, citing that the detector was in a non-O2-standard configuration and the calibration is suspect. Yes, it is plausible to reconstruct the calibration during these times, but given our limited person-power we have elected to abandon the effort at this time.
No problem, I made a flag and inserted it in to the segment database. H1:DCH-CAL_NON_O2_STANDARD_CONFIG:1 captures this time.
| optic | absolute distance from IO_AB_L4 | focal length and beam diam. |
| IO_AB_L4 | 0 | fl = 1145.6mm |
| IO_lens3 | 43.4cm | fl = -618.2mm |
| IO_lens4 | 123.3cm | fl = 720.2mm |
| IOGigE1 | 163.1cm | beam diameter = 1.9mm |
| IOGigE2 | 245.5cm | beam diameter = 1.9mm |
Beam spot measurements: according to these measurements, the beam spots from before to after the 5.8M EQ, and all changes are 0.14mm or less.
| alpha | beam dist from center, 17 July 2017 | diff from 17 May 2017 | |
| mc1 p | -0.057 | 2.40 | 0.10 |
| mc1 y | 0.092 | 3.87 | 0.08 |
| mc2 p | -0.183 | 7.67 | 0.04 |
| mc2 y | -0.020 | -0.85 | -0.04 |
| mc3 p | -0.054 | 2.26 | 0.10 |
| mc3 y | -0.138 | -5.84 | -0.14 |
Dave and TJ:
we rebooted h1guardian0 because its loading has been regularly in the 50's. TJ recovered the GRD nodes following the reboot.
Looking over the trends, since the interface chassis was restarted and power cable secured, I see no large glitches, especially on corner3 only, that aren't seen on all CPSs and that are not related to ground motion. Can't access FRS to give a number but I'll keep an eye on this for a few more days.
I've updated FRS Ticket 8517.
This morning I performed the weekly PSL FAMIS tasks.
HPO Pump Diode Current Adjust (FAMIS 8431)
I adjusted the HPO pump diode currents, changes summarized in the table below. All adjustments were done with the ISS OFF. I have attached a screenshot of the PSL Beckhoff main screen for future reference.
| HPO Diode Currents (A) | ||
| Old | New | |
| DB1 | 48.9 | 49.0 |
| DB2 | 51.8 | 52.0 |
| DB3 | 51.8 | 52.0 |
| DB4 | 51.8 | 52.0 |
I then tweaked the DB operating temperatures. The only change was to DB1, all diodes changed from 29.0 °C to 28.5 °C; no changes were made to the operating temps of DBs 2, 3, or 4. The HPO is now outputting 154.9 W and the ISS has been turned back ON. This completes FAMIS 8431.
PSL Power Watchdog Reset (FAMIS 3659)
I reset both PSL power watchdogs at 16:26 UTC (9:26 PDT). This completes FAMIS 3659.
Tried two different browsers;
This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.
This is due to a big Grouper update - see LLO aLOG 34921. It should not affect local logins to CDS workstations or the main aLOGs, but does affect all remote logins and web site accesses. This was not as intended and they are working hard to fix it.
IFO in Observing for past 4 hours. Environmental conditions are good. A2Ls are both below reference. Range is holding around 48Mpc.
J. Kissel
As we're getting slammed by another earthquake, I wonder if there can be a few things that team SEI can change in order to get the SEI platforms back to DAMPED (HEPI robust isolated on L4Cs, HAMs Damping with GS13s, BSC ISIs damping with L4Cs and GS13s) and optics back to DAMPED as soon as possible. Here're the flaws I see at the moment:
(1) SEI_CONFIG manager doesn't turn off BSC-ISI Z sensor correction (at least in EARTHQUAKE_v2), so giant EQs like the current still push HEPI in Z enough that it makes the T240s angry and trips the ISI's watchdog, even though the T240s aren't yet involved in any active control. The SEI manager should have a state where it turns off ALL sensor correction, not just horizontal sensor correction.
EDITNevermind => I had not read Jim's updates to the SEI_CONFIG Guide. We should be in LARGE_EQ_NOBRSXY, and that does indeed turn off ALL sensor correction.
(2) This may be hard, but it would be best if the T240s weren't in the watchdog system until they're used.
Just planting the seed for discussion, no need to pull out ECR pens just yet.
Re #2, My recollection was that it was ignored or else while it was riled up it would trip which it does not. Regardless, I reviewed the model and indeed I recalled correctly that the T240s are ignored in the WD as long as all the Isolation Gains are zero. Not sure the situation that Corey was experiencing during the EQ ring-down that suggested the T240s were the problem.
J. Oberling, E. Merilh, J. Warner
The PSL tripped this morning, likely due to the flow in laser head 3. As can be seen in this alog, the flow in head 3 (and only head 3) dropped from 0.52 lpm to ~0.47 lpm sometime on the morning of Saturday, 7/15/2017. It is likely this was the cause of the laser trip. I will do more forensics when I am onsite tomorrow. When restarting the cyrstal chiller, the flow in laser head 3 was back up around 0.54 lpm. Maybe something worked its way through the system, causing the laser trip?
When restarting the laser, the Beckhoff software appeared to lose communication with the hardware of the PSL, requiring a reboot of the PSL Beckhoff PC. Once this was done, everything worked fine and the laser came back up. I had difficulty injection locking the 35W FE to the HPO; I believe this is due to the lower power out of the HPO. I engaged the injection locking by turning the RAMP OFF, and monitoring the power circulating in both directions of the HPO. When the circulating power favored the forward direction, I manually engaged the injection locking. This worked the first time and the laser is now back up and running. Ed re-engaged the PMC, FSS, and ISS, and Jim reset the NPRO noise eater. By this time ~30 minutes had elapsed, so I engaged the LRA and the power watchdogs. The laser is now up and running.
J. Kissel filed FRS 8539 for this trip.
Looking at some trends it appears that it was Laser Head 4 that tripped the PSL, not Laser Head 3 as previously surmised. The first attachment shows the 4 laser head flows around the time of the trip. The second shows the flow through Head 4 and the Head 1-4 Flow Interlock, while the third attachment shows the flow through Head 3 and the interlock. It is clear from this that the flow through Head 4 was at the trip point of 0.4 lpm at the time the interlock tripped, while the flow through Head 3 remained above the trip point. It is unclear why the flow through Laser Head 4 fell so fast, possibly something moving through the system causing a glitch with the flow sensor?
SudarshanK, RickS
To study the downconversion of Pcal excitation we used a DB9 breakout at the back of the Pcal chassis and routed the excitation channel through H1:CAL-PCALX_WS_PD for now. This will be returned to its original configuration once the investigation is complete.
The PcalX is still fully functional and hardware injection can be carried out if needed. The WS_PD channel through which the excitation channel is routed is a spare channel that is only used during the end-station PD calibration.
Summary:
We have established that most of these down converted lines (the one that hurts us) reported in alog 36959 arise from the aliasing of the harmonics of the injected high frequency calibration lines. The lines that have the possibility to show up in the DARM are the ones that lie between 10 Hz to few hundred Hz. We can avoid these lines by selecting the high frequency calibration lines such that the aliased lines are not in the given low frequency region.
Details:
We ran into issues with low frequency lines (in 100 Hz range) showing up in DARM when we were running high frequency calibration lines in the region of 5.5 - 6 kHz. This was first noticed in this LHO alog 36959 and LLO alog 34346. To understand this problem we wanted to find where these lines were being produced, photon calibrator itself or the data acquisition system. We disconnected the cable that inputs the excitation to the Pcal electronics and used a DB9 breakout box to acquire the excitation signal so that we were getting these signal before it enters the Pcal system. We rerouted this signal through the H1:CAL-PCALX_WS_PD channel which was one of the unused Pcal channel.
The first plot shows the spectrum with a 5036 Hz excitation line (in blue) and without an excitation (in red). Some of the extra lines seen in the blue spectra are the result of aliasing of the higher order harmonics of the injected as well as the imaged lines (imaging happens when going from 16khz to 64 kHz). In this particular case the 68 Hz line is the aliasing of the 13th order harmonic of 5036 Hz injected line. The second plot shows the predicted aliased lines (based on harmonics and upsampling) that would be produced for 5036 Hz injected line. Some of the predicted lines show up in the actual spectrum but not all and not all line that are present in the spectra are predicted. However, the lines that show up above few hundred Hz will be well below the DARM signal as the actuation transfer function goes as 1/f^2. Shivaraj, at LLO has taken some additional measurement to see if we can find the source of all the aliased lines. Report to follow.
Tagging DetChar and CDS on this, so that the CW group can follow up with them to understand how to flag the data for this known issue.
RickS, SudarshanK,
The breakout box used to acquire the analog excitation signal on X-end Pcal has been removed and the Pcal configuration is now back to normal. During the visit, Rick repeated some measurement that reproduced the 86 Hz line when a Pcal line was injected at 5950 Hz with an excitation amplitude of 25000 cts. This measurement was made at the DAC (analog) output using SR785. The 86 Hz line is about 70dB below the injected line. This still does not explain why we don't see 86 Hz line in analog output in the measurement (LLO alog 34495) that Shivaraj made using Agilent signal analyzer. :(
On Jeff's request, I am putting down the empirical formula that predicts the lowest frequency down-converted lines for each high frequency excitation.
DCF = 2^16 - EF * n, where DCF is the down converted frequency, EF is the excitation frequency and n is the harmonic number. For frequencies around 4-6 kHz, the harmonic number that produces the line in the bucket are between 11 and 13.
I don't have a physical intuition on why this happens. If we were looking at the resampled 16 kHz signals, the presence of these lines would not be surprising because these would be produced because of aliasing but we see these low frequency lines on the DAC output, measured using the analog spectrum analyzer. Plots showing the same are attached.