WP7420 New SQZ guardian nodes and DAQ restart

Daniel, TJ, Dave:

TJ created the three new SQZ guardian nodes (SQZ_CLF, SQZ_OPO, SQZ_LO).

We also re-created the INJ_TRANS node after fixing the NFS mount issue between h1guardian0 and h1hwinj1.

The DAQ was restarted to use the new H1EDCU_GRD.ini This was urgent, the loss of the INJ_TRANS channels from the frame files was causing Det Char issues with their summary page generation.

   Check chiller water levels and filters. I added 100ml on Tuesday (aLOG #40984); no additional water was needed today. No change in the filters. The Diode chiller filters are clean. The Crystal chiller filters need to be replace when the 70W work is finished.

As we get back to commissioning, we are again actively changing IPCs in front-end models.  This can cause the number actually in use to vary from those in the static lists, that only get pruned manually or when re-building all models. At present, we are limited to 512 IPCs per type (Shared memory, PCIe, RFM-X, RFM-Y).  I updated a script 'find_used_ipc_channels.py' started by Joe Betzwieser to go over all the in-use models (using the rtsystab file) and list all the IPCs in use in files.

After running it at LLO (see aLOG 38202 ). I have run this for H1 today with the following:

 Found 197 SHMEM IPCs - list in ipc_shmem_channels.txt
 Found 414 PCIE IPCs - list in ipc_pcie_channels.txt
 Found 31 RFM0 IPCs - list in ipc_rfm0_channels.txt
 Found 31 RFM1 IPCs - list in ipc_rfm1_channels.txt

I have attached the channel lists.  These are somewhat lower that LLO usage.

   Posted are the weekly ISI CPS noise spectra. Did not see any egregious anomalies. Closing FAMIS #6941  

Marc fixed the thermistor readbacks in the EtherCAT chassis. All 3 are now working.

Marc also fixed the OpAmp in the slow output path of the TTFSS.

Fil fixed the in-vac wiring problem for DCPD1.

An updated TwinCAT code and medm screen was installed to control the OPO TEC.

We think CP4's regen heater SCR failed last night, so Kyle and I swapped it out with CP3's this afternoon. Will test tomorrow morning with LN2/GN2 flow.

Gerardo, Kyle 

Today we had to shut down the LVEA purge-air supply for a few hours in order to replace a failing pneumatic valve.  In-chamber work in HAM6 was impacted.  

Update on what I did with ZM1 today.

Summary: the UL flag now looks much better, but I dropped a screw down into the ISI and I topped out some adjustment screws.

After talking with Jeff K this week about what might be going on, I remembered that the UL OSEM flag was slightly askew from the other flags. Jeff's data may also point to a problem coming from that area so I wanted to see if I could straighten or align the flag today. I pulled the flag and rolled it on the table and you could see that the tip of the flag was not straight. I replaced it with another that I had brought with me and it looked slightly better, but still not great. Roll tested this one as well and all looked good, hmm. So I put it back in and then tried to move the OSEM in the desired direction with some success. All the flags now seemed to sit a bit lower, so I adjusted the height but I found that the screw to lock down this adjustment couldn't reach anymore. I'll have to get some longer ones tomorrow. Overall, the UL flag is now much more centered and the others are good as well.

As I was putting the earthquake stops back on I fumbled one of the 4-40 screws and dropped it right down into one of the through holes on the table (I blame the gloves). I could not see a way to get to it without removing one of the HAM side panels, so I will consult with SEI team and take action tomorrow.

Looked at in-vacuum cabling for the swapped pins for DCPD 1, see alog 40881. After pins were swapped we followed the pin layout and verified pins to SQZ-R1 rack were correct. Documents we referred for the pin layout are D1700384, D1700308, and D1700185.

F. Clara, T. Shaffer

TITLE: 03/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
LOG:

15:00 Rick, Travis and Niko to EX for in-chamber Pcal alignment.

15:02 Peter ad Jeff moving stuff into H2 Chiller enclosure

15:40 Peter and Jeff back from LVEA15:51 Fil out to EY for HV testing

16:01 Marc ro CER to repair squeezer Ethercat chassis WP# 7413

16:02 Hugh to EX mechanical room

16:09 Dan(HVAC) out of optics lab and heading to end stations.

16:10 Jason heading to EX to swap OpLev laser

16:18 Kim and Betsy to EY. Pulling first contact and if all goes well, locking ISI.

16:38 Rick, Travis and Niko back. Travis to EY to join Betsy.

16:40 Norco on site delivering LN₂ to dewar 76 (CS-X)

16:57 Elizabeth heading out to outbuildings - delivering net switches/dressing cables/checking connectivity

17:00 Second LN₂ delivery on site to dewar 72 (MY)

17:05 Jason back

17:26 Cheryl headed into the PSL

17:33 Marc out

17:35 N out to ISCT6.

17:49 Paradise water with delivery

17:50 M² and Chris out of optics lab and out to EY to put BSC door on.

18:03 Karen to MY

18:20 Betsy ad Travis back

18:42 Corey out to optics lab/LVEA for tools

19:05 DAQ resart. Took OPO, ZMs ad OFI to SAFE state

19:07 Cheryl out for lunch

19:11 Corey and Elizabeth out for lunch

19:10 Returned all suspensions back to previous states

19:33 M² done at EY and back. Door is on

19:49 Pepsi on site

19:55 Elizabeth back out to previous out-building tasks

19:56 Purge air work is over. Fil ad TJ out to HAM6

20:26 Jason out to PSL

20:29 Richard out to HAM6

20:30 Me to EX to center the OpLev

20:33 Richard back

20:34 Hugh out to EX

20:36 Cheryl back out to PSL

20:39 Chris out to ends moving SUS equipment

21:28 Betsy and Travis heading to EX

21:47 Hugh back

22:22 Jim out to EX

22:56 Jim back

I restarted the DAQ at 12:05 PDT. About 90 minutes later h1fw0 crashed and was auto-restarted by monit. This was followed quickly with another restart. At this point I started monitoring it carefully and noticed it was very slow in writing its frame files, to the extent that it was sometimes over 64 seconds behind. Most of the time it managed to almost catch up.

I verified that h1fw0's E18 RAID was not reporting any errors. On h1fw0 I noticed that looking at the /trend mount froze the terminal. So as a first try, I rebooted h1fw0 (leaving h1ldasgw0 running). It came back up correctly and the problem appears to have been resolved. The /trend issue was an old mount in /etc/fstab trying to mount h1tw0, a machine which was turned off several weeks ago. I have corrected the fstab file, but for the current run it was still in the file and /trend is resolving as an empty mount point (as it should be) and not freezing the terminal.

Attached plot show 30 mins of second trend, showing the full file size for h1fw0 (red), h1fw1 (green) and h1fw2 (blue). fw2 and fw3 are in lock step, fw0 is one cycle behind initially and catches up later.

Today, I went in and pulled the 2nd First Contact sheet after the reapplication yesterday afternoon.  Unfortunately, the 2 small glint blemishes that I didn't like yesterday were still there.  So, they are either in the coating, scratches, or are not coming off.  So, I then completed the closeout steps of:

1) N2 blow, measuring charge (details below)

2) Placing the 1" witness optic on the side of the QUAD structure

3) Jim unlocking ISI

4) Placing 3" horizontal wafer below ETMY QUAD on floor

5) Mounting 3" vertical wafer on lower front of QUAD structure

6) Wiping the floor on the way out

7) Removing tools

8) Kissel running TFs - 3 per suspended chain - all good

9) Launching door crew

Door on by lunch, started work at ~9:30am.

WP7410 susopo

Jeff K, Dave:

A new h1susopo model was installed sans DACKILL in the OFI part.

Beckhoff Code Change

Daniel, Dave:

A new h1ecatc1plc4 was installed. New INI file was added to the DAQ. New autoBurt.req file added to target. New monitor.req added to SDF. I restarted h1sysecatc1plc4sdf on h1build. I accepted and monitored all the new channels in the safe.snap file.

WP7399 h1tw1 offload

Dave:

I have completed the first part of h1tw1's raw minute trend files offload. The files have been copied and verified from h1tw1 to h1fw1 (served by h1nds1). The files have been deleted from h1tw1, reducing its /trend RAID disk usage from 96% to 10%. The final part of the process is to copy the files from "h1fw1's SATABOY" to "h1tw0's SATABOY" so that h1nds0 can see the files. There is no rush to do this since h1nds1 is the default nds.

DAQ Restart

We only needed one DAQ restart, performed at 12:05 PDT, which:

using new h1susopo ini

using new h1ecatc1plc4 ini

restarted h1nds1 with new daqdrc (gets past 6 months of minute trends from SATABOY)

restarted h1nds0 with new daqrc (no longer has past 6 months of minute trends from h1tw1)

restart log is attached.

Last Thur at 4pm local, the ETMX Oplev took another jump - this time ~70uRad in PITCH, negligable in YAW.  At the same time, the PIT, YAW, and SUM signals became noisy and have been ever since.  According to the THUR Mar 08 Ops summary alog 40914, the only crew was the in-chamber PCAL baffle crew, who were not around the OPLEV piers outside of the chamber and down the tube a bit.  There apparently were no reboots.  The hash on the signals is an apparent sign that the oplev laser is dying however.  Jason reports that the PIT and YAW signals are normalized by the sum.  There is a very small shift on the SUM signal at the jump on THUR but I'm not sure how it adds up to the PIT-only 70uR shift.

Due to this mystery and the PCAL misalignment (alog 40968), we launched Keita to check the ALS pointing at the table at EX.  He found things somewhat well aligned using ~50uRad of bias on the ETMX/TMSX.  So, at least the ALS beam and the hashy, twitchy OPLEV beam pointing monuments agree that the ETMX is still pointed to within ~50-100uR of the last good arm pointing (recall that the vent to air contributes to some pitch mis-pointing as well, see alog 40714).  The PCAL reports a much larger magnitude of ~1mR of alignment error.

In the morning, Travis and Rick intend to remount the PCAL target on the front of the ETMX and see where the incoming PCAL beam is - if it's off, the whole periscope has been misaligned.  If it's hitting the ETM at the right place, Travis will look closer at the ETMX itself.  (Note, we have not yet touched the ETMX at all since the vent from last week - only the PCAL baffle crew took parts past it and into the manifold to work.  The ISI has been locked however, but the PCAL was checked for good alignment before the start of the baffle work and after the ISI lock.  alog 40892)

Also, Jason plans to swap the laser for a healthy one first thing tomorrow.

Hmmm... looking closer at the attached plot I see that at the end of the plot when we asked Corey to return the ETMX to zero mid-morning, he only had to use the ETMX PIT slider but I see a step in both the PIT and YAW trends of the OPLEV... 

TBC...  Anyone volunteer to sort this mystery?

Richard, Fil, Dave:

at MY we powered down: h1pemmy front end computer, its IO Chassis, its AA Chassis and both fiber-eth-converters (h1fe and h1daq).

These will remain powered off for the duration of the bake-out.