Gerardo, Kyle Today we had to shut down the LVEA purge-air supply for a few hours in order to replace a failing pneumatic valve. In-chamber work in HAM6 was impacted.
Update on what I did with ZM1 today.
Summary: the UL flag now looks much better, but I dropped a screw down into the ISI and I topped out some adjustment screws.
After talking with Jeff K this week about what might be going on, I remembered that the UL OSEM flag was slightly askew from the other flags. Jeff's data may also point to a problem coming from that area so I wanted to see if I could straighten or align the flag today. I pulled the flag and rolled it on the table and you could see that the tip of the flag was not straight. I replaced it with another that I had brought with me and it looked slightly better, but still not great. Roll tested this one as well and all looked good, hmm. So I put it back in and then tried to move the OSEM in the desired direction with some success. All the flags now seemed to sit a bit lower, so I adjusted the height but I found that the screw to lock down this adjustment couldn't reach anymore. I'll have to get some longer ones tomorrow. Overall, the UL flag is now much more centered and the others are good as well.
As I was putting the earthquake stops back on I fumbled one of the 4-40 screws and dropped it right down into one of the through holes on the table (I blame the gloves). I could not see a way to get to it without removing one of the HAM side panels, so I will consult with SEI team and take action tomorrow.
TITLE: 03/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
LOG:
15:00 Rick, Travis and Niko to EX for in-chamber Pcal alignment.
15:02 Peter ad Jeff moving stuff into H2 Chiller enclosure
15:40 Peter and Jeff back from LVEA15:51 Fil out to EY for HV testing
16:01 Marc ro CER to repair squeezer Ethercat chassis WP# 7413
16:02 Hugh to EX mechanical room
16:09 Dan(HVAC) out of optics lab and heading to end stations.
16:10 Jason heading to EX to swap OpLev laser
16:18 Kim and Betsy to EY. Pulling first contact and if all goes well, locking ISI.
16:38 Rick, Travis and Niko back. Travis to EY to join Betsy.
16:40 Norco on site delivering LN2 to dewar 76 (CS-X)
16:57 Elizabeth heading out to outbuildings - delivering net switches/dressing cables/checking connectivity
17:00 Second LN2 delivery on site to dewar 72 (MY)
17:05 Jason back
17:26 Cheryl headed into the PSL
17:33 Marc out
17:35 N out to ISCT6.
17:49 Paradise water with delivery
17:50 M2 and Chris out of optics lab and out to EY to put BSC door on.
18:03 Karen to MY
18:20 Betsy ad Travis back
18:42 Corey out to optics lab/LVEA for tools
19:05 DAQ resart. Took OPO, ZMs ad OFI to SAFE state
19:07 Cheryl out for lunch
19:11 Corey and Elizabeth out for lunch
19:10 Returned all suspensions back to previous states
19:33 M2 done at EY and back. Door is on
19:49 Pepsi on site
19:55 Elizabeth back out to previous out-building tasks
19:56 Purge air work is over. Fil ad TJ out to HAM6
20:26 Jason out to PSL
20:29 Richard out to HAM6
20:30 Me to EX to center the OpLev
20:33 Richard back
20:34 Hugh out to EX
20:36 Cheryl back out to PSL
20:39 Chris out to ends moving SUS equipment
21:28 Betsy and Travis heading to EX
21:47 Hugh back
22:22 Jim out to EX
22:56 Jim back
I restarted the DAQ at 12:05 PDT. About 90 minutes later h1fw0 crashed and was auto-restarted by monit. This was followed quickly with another restart. At this point I started monitoring it carefully and noticed it was very slow in writing its frame files, to the extent that it was sometimes over 64 seconds behind. Most of the time it managed to almost catch up.
I verified that h1fw0's E18 RAID was not reporting any errors. On h1fw0 I noticed that looking at the /trend mount froze the terminal. So as a first try, I rebooted h1fw0 (leaving h1ldasgw0 running). It came back up correctly and the problem appears to have been resolved. The /trend issue was an old mount in /etc/fstab trying to mount h1tw0, a machine which was turned off several weeks ago. I have corrected the fstab file, but for the current run it was still in the file and /trend is resolving as an empty mount point (as it should be) and not freezing the terminal.
Attached plot show 30 mins of second trend, showing the full file size for h1fw0 (red), h1fw1 (green) and h1fw2 (blue). fw2 and fw3 are in lock step, fw0 is one cycle behind initially and catches up later.
I spoke too soon, h1fw0 restarted itself 93 minutes later.
I performed a full power cycle of both h1fw0 and h1ldasgw0 (solaris QFS server) which has fixed these issues in the past. System has been back for 15 minutes, I'm monitoring it closely.
Looks like it is fixed now, has been running for 18 hours.
Today, I went in and pulled the 2nd First Contact sheet after the reapplication yesterday afternoon. Unfortunately, the 2 small glint blemishes that I didn't like yesterday were still there. So, they are either in the coating, scratches, or are not coming off. So, I then completed the closeout steps of:
1) N2 blow, measuring charge (details below)
2) Placing the 1" witness optic on the side of the QUAD structure
3) Jim unlocking ISI
4) Placing 3" horizontal wafer below ETMY QUAD on floor
5) Mounting 3" vertical wafer on lower front of QUAD structure
6) Wiping the floor on the way out
7) Removing tools
8) Kissel running TFs - 3 per suspended chain - all good
9) Launching door crew
Door on by lunch, started work at ~9:30am.
Details of the N2 deionization during this closeout:
Measuring at the back of the AERM surface, the measured voltage started at ~8v, measured in the Center, UL, LR positions. We then pulled the FC and blew everywhere around the ETM and AERM for many minutes. Measuring again, the charge at these locations was down to ~2.5v. Another round of blowing, with a check of the zero of the meter in between, and the charge then read 1.3v in the Center and UL and 2.1v at the LR position.
5 minutes later we remeasured and found the charge at 0.4 in the center and -0.6v at UL.
WP7410 susopo
Jeff K, Dave:
A new h1susopo model was installed sans DACKILL in the OFI part.
Beckhoff Code Change
Daniel, Dave:
A new h1ecatc1plc4 was installed. New INI file was added to the DAQ. New autoBurt.req file added to target. New monitor.req added to SDF. I restarted h1sysecatc1plc4sdf on h1build. I accepted and monitored all the new channels in the safe.snap file.
WP7399 h1tw1 offload
Dave:
I have completed the first part of h1tw1's raw minute trend files offload. The files have been copied and verified from h1tw1 to h1fw1 (served by h1nds1). The files have been deleted from h1tw1, reducing its /trend RAID disk usage from 96% to 10%. The final part of the process is to copy the files from "h1fw1's SATABOY" to "h1tw0's SATABOY" so that h1nds0 can see the files. There is no rush to do this since h1nds1 is the default nds.
DAQ Restart
We only needed one DAQ restart, performed at 12:05 PDT, which:
using new h1susopo ini
using new h1ecatc1plc4 ini
restarted h1nds1 with new daqdrc (gets past 6 months of minute trends from SATABOY)
restarted h1nds0 with new daqrc (no longer has past 6 months of minute trends from h1tw1)
restart log is attached.
The above mentioned h1susopo model restart closes WP 7410
I re-arranged the Beckhoff SDF part of the CDS Overview MEDM to add the missing PLC4
The previous ROC was only showing rate-of-change for the unity time periods (1MIN, 1HOUR, 1DAY). For the other periods it was just showing the value differences.
My new system (rolled out yesterday) has three EPICS channels per time period: R, D and V.
V is the value acquired n-minutes ago. E.g for the 3HOUR period, it is the value acquired 180 minutes ago.
D is the value difference between the current value and V
R is the rate of change (D/period) in units appropriate to the period (either per minute, per hour or per day).
The second CP4 thermocouple was added. For performance reasons, the dewar channel was removed.
The bias slider values that were stored in the SDF SAFE file were old. When there was a boot fest last Thur of ETMY SUS (alog 40913), these values got restored and the ETMY no longer pointed to ~0 on the Oplev. Today I trended those values an dataviewer in order to restore them. The latest values which we mechanically pointed the ETMY to, used a little bit of bias:
PIT 34.4
YAW -105.0
However these values don't put it on the Oplev in PIT now. Today it wants bias slider values below in order to be well centered on the OL:
PIT -135.6
YAW -105.0
Whatever. So many mysteries, so little time. Door closed.
I have loaded these newest bias values into the ETMY SUS SDF SAFE file.
In light of the recent oplev issues (primarily the increased signal noise, which is an indication of laser glitching), I swapped the oplev laser for a spare so there is not an issue with the oplev during the upcoming ETMx work (as the oplev is being used as our alignment reference). The old laser is SN 130-1 and the new laser is SN 197-2. The new laser has not been tweaked for glitch-free operation, it will have to be replaced before the start of O3 with one that has.
Note, the current SUM on this OPLEV is now ~23,800.
ESD cables were tested with HIPOT tester. Cables tested to 1KV, all passed.
While working in the Chiller room, the PSL Chillers were serviced. 100ml water was added to the Crystal chiller. No water added to the Diode chiller. No changes in the filters for either chiller.
As I was crawling into bed......... I noticed that the temperature of the regenerating GN2 flow leaving the heater was 68C -> it should have been much higher. After 5 minutes, I refreshed the screen and it was then 55C -> I realized that the heating of the regeneration flow had stopped, sent Chandra R. and Gerardo M. a text and set off to the Y-mid. Upon arrival, I noted that the dewar vapor pressure was as expected (18 psig) and the flow of rengenerating GN2 was also as expected at 50 SCFH. In the VEA, I noted that the temperature of the GN2 leaving the heater was now 16C? 18C? Next, I noticed that one of the RED indicator lamps on the Regeneration Control Panel was illuminated -> I reset this before noting which of the two lamps was lit (they are side-by-side). Upon reseting, the GN2 temperature began to rise too quickly as the sepoint was 150C and the proportional gain was 7. -> I lowered the SETPOINT to a value slightly above the current value and also lowered the proportional gain value but the PID was too slow and the GN2 temperature continued to rise too quickly. Next, I turned off the heating at the control panel (small switch - not main Panel Power) and let the temperature stabilize. Once stable, I set the proportional gain to 1.00 and the SETPOINT value to 2C above the currently displayed value. Then, I turned on the Regeneration Heater. Now the "SHORTED SCR" lamp lit up - reset - goes off then back on etc... I can hear excess current (60 Hz hummmm) before trip. It is broken. I closed the regeneration and pressure build manual valves and shut down the Regeneration Control Panel. Also the 3-phase CB. So for tonight, only the air duct heater associated with the bake enclosure will be heating CP4. 2350 hrs. local -> Leaving site now.
Last Thur at 4pm local, the ETMX Oplev took another jump - this time ~70uRad in PITCH, negligable in YAW. At the same time, the PIT, YAW, and SUM signals became noisy and have been ever since. According to the THUR Mar 08 Ops summary alog 40914, the only crew was the in-chamber PCAL baffle crew, who were not around the OPLEV piers outside of the chamber and down the tube a bit. There apparently were no reboots. The hash on the signals is an apparent sign that the oplev laser is dying however. Jason reports that the PIT and YAW signals are normalized by the sum. There is a very small shift on the SUM signal at the jump on THUR but I'm not sure how it adds up to the PIT-only 70uR shift.
Due to this mystery and the PCAL misalignment (alog 40968), we launched Keita to check the ALS pointing at the table at EX. He found things somewhat well aligned using ~50uRad of bias on the ETMX/TMSX. So, at least the ALS beam and the hashy, twitchy OPLEV beam pointing monuments agree that the ETMX is still pointed to within ~50-100uR of the last good arm pointing (recall that the vent to air contributes to some pitch mis-pointing as well, see alog 40714). The PCAL reports a much larger magnitude of ~1mR of alignment error.
In the morning, Travis and Rick intend to remount the PCAL target on the front of the ETMX and see where the incoming PCAL beam is - if it's off, the whole periscope has been misaligned. If it's hitting the ETM at the right place, Travis will look closer at the ETMX itself. (Note, we have not yet touched the ETMX at all since the vent from last week - only the PCAL baffle crew took parts past it and into the manifold to work. The ISI has been locked however, but the PCAL was checked for good alignment before the start of the baffle work and after the ISI lock. alog 40892)
Also, Jason plans to swap the laser for a healthy one first thing tomorrow.
Hmmm... looking closer at the attached plot I see that at the end of the plot when we asked Corey to return the ETMX to zero mid-morning, he only had to use the ETMX PIT slider but I see a step in both the PIT and YAW trends of the OPLEV...
TBC... Anyone volunteer to sort this mystery?
Today, Hugh pointed out that the EX vea temp has been on the rise for the last few weeks due to all of the cleanrooms running. This isn't a huge surprise but it got us wondering if the steps in pointing of the ETMX as viewed by the OPLEV PIT are actually the suspension sagging and brushing something mechanical. (The steps account for ~100uR of unaccounted mispointing.) Indeed the top BOSEM sensors see the steps. However, a couple hours of staring at trends gets us no closer to this theory as the actually problem, for the following reasons:
1) Bringing the BSC9 volume up to air lifted the suspension, while the temperature increase is dropping the suspension, restoring it towards its original vertical position.
2) The temperature is hotter now than at the time of to step events (purple lines on the plot), yet transfer functions of ETMX V and P look healthy to me.
So, the agreed apon sentence from late yesterday still holds:
We will rezero the OPLEV now, with the biases from the arm lock - this should be good enough to find the arm again after pump down to within ~100uRad.
Recall, ALS pointing on the table yesterday with these slider settings looked pretty good.
Bias sliders:
PIT 7.3
YAW 40.2
Richard, Fil, Dave:
at MY we powered down: h1pemmy front end computer, its IO Chassis, its AA Chassis and both fiber-eth-converters (h1fe and h1daq).
These will remain powered off for the duration of the bake-out.
MY weather station is also down, suspect turning off DC power supplies in the DAQ rack did this.
NikoL, Rick S
We switched on the the Pcal laser to check the beam pointing after the vibration damper and baffles installation work last week.
We (with Corey's help) pointed the ETM to center the optical level signals.
With this alignment the Pcal spots are way off, more than 1 cm (see attached photo, both beams should be (and were earlier last week) centered on the integrating sphere aperture).
Keita is going to check on the TMS alignment so we can try to discriminate between the possibility that the Pcal periscope shifted during the alignment work and the possibility that the ETM pointing (or OptLev) has changed.
If the ETM pointing is off, it would be on order 10 mm over 10 m, or about 1 mrad.
TravisS, NikoL, RickS
It appears that the Pcal periscope structure moved (top pitched back, away from the ETM) during the "Baffles and Shields" installation.
The attached photos show the spot positions on the ETM target both before (last Wed.) and after (this morning) the installation of the vibration absorbers, Mt. Brackets, and barrel baffle panels.
Also attached is a detail of the Pcal ETM target (D1301014) showing the dimensions of the features on the target.
It appears that the upper beam moved up about 3.5 mm and the lower beam moved right about 3.5 mm.
The gaps between the centers of the upper flexures and the A7 adapter wall were slightly more than 0230". Our goal was to keep them below 0.220". We set them significantly below 0.220" last week (maybe 0.210") but apparently the additional mass of hte hardware added reduced the compression (as expected, see LIGO-T1800047-v2).
After talking with Hugh and Jim it seems that we might leave this screw where it is unless it starts to cause some issues.
As a future reminder, I dropped a 4-40x.5" SHCS over the South corner of HAM6. It fell through one of the helicoiled through holes and dropped inside the table. The hole it fell through is above spring post #1, I am told. I did not see or hear it fall through to stage 0, and I could not locate it through the holes in the wall panels on the South-East side of the table. During their CPS investigation, Hugh and Corey had one of the access panels removed on that corner, but I also did not see or feel anything.