J. Kissel
The issues we found with the IM's binary IO motivated me to update the MEDM screen interface to the system -- namely making it more like other HAM Single Stage Suspensions (HSSS) and the top masses of HAM X Double Suspensions (HPDS, HDDS, HTDS, HSDS) which clearly indicate that the HAM-A drivers they use have had their remote binary IO control / readbacks disabled.
At LHO the MEDM screens that call the interface are
/opt/rtcds/userapps/release/sus/common/medm/haux/
SUS_CUST_HAUX_MONITOR_OVERVIEW_all.adl
and
/opt/rtcds/userapps/release/sus/common/medm/hsss
SUS_CUST_HSSS_OVERVIEW.adl
which now *both* call the same BIO screen, which I've moved
FROM
/opt/rtcds/userapps/release/sus/common/medm/
haux/SUS_CUST_HAUX_BIO_ALL.adl
TO
hsss/SUS_CUST_HAUX_BIO_ALL.adl
then modified and committed the svn location change. See BEFORE and AFTER.
I then edited the following macros to change the SUSTYPE or sustype variables from "IM" or "im" to "HAUX" or "haux."
/opt/rtcds/userapps/release/sus/common/medm/
susim1_overview_macro.txt
susim2_overview_macro.txt
susim3_overview_macro.txt
susim4_overview_macro.txt
susimall_overview_macro.txt
These have *not* been committed to the svn as the macros also contain all of the ADC / DAC / AI / AA / front-end configurations that have diverged for suspensions at L1 and H1.
It turns out I never committed some medm SUS overview screens to svn after we removed the USER DACKILL back a long time ago, so I just committed them. They are /userapps/sus/common/medm/:
J. Kissel, O. Patane, Oli and I have moved on to recovering the SUS IMs post their migration from the h1sush2b computer into the newly merged h1sush12 computer / IO chassis system yesterday (see 88519 and 88527), but immediately noticed that the MEDM OVERVIEW screen reported that the test/coil enable bits were in apparently non-functional state (see "before" OVERVIEW and BIO screens). Thinking through it, and remembering that the IM's binary IO switching had been rendered "defunct" back in July 2022 during their upgrade of HAM-A coil drivers (LHO:64171 manifesting ECRs E2200307, E2400048, and E2500295), we release that the digital representations of the sush2b BIO card read/write had never been removed from the h1susim model. Prior to the sush2b + sush2a = sush12 merge, talking to the sush2b card didn't matter. But now that this model is running on the merged chassis, the digital representation of talking to binary card 0 and card 1 means that that both the h1susmc1 and h1susim models are talking to the same BIO card (card 0 and card 1, the lower an upper halves of the physical BIO0 card) in the sush12 chassis! No bueno. This double comms is confirmed by comparing the top level and BINARY DECODE block of h1susmc1 (userapps svn rev r33947) model's Binary IO communications to the "before" communications within the h1susim model top level (userapps rev r33974). As such, I've modified the top level of the h1susim model to remove all digital representation of the cards, and sending 85 = 01010101 ("the Test/Coil switch is set to COIL enabling drive from the DAC, and the LP is ON" for each of the 4x suspensions) into the EPICs readback to indicate / match reality, and terminating the top level control request. See top level of the model now committed as rev r34204.
D. Barker, E. von Ries, J. Kissel This h1susim front-end model change has been installed as of 2025-12-18 18:25 UTC. Dave will likely post an official aLOG eventually. All signs point toward "success!"
J. Kissel, O. Patane
We've recovered all damping and alignment functionality for H1SUSMC1, H1SUSMC3, H1SUSPRM, and H1SUSPR3 (including turning on L, P. and Y Estimators for PR3); the HAM {Small, Large} Triple Suspensions (HXTS, HSTS, HLTS) on HAM2 after their host computer changed from h1sush2a to h1sush12 and their DACs were upgraded to a 32CH 28-bit LIGO DAC (see analog changes in LHO:88519 and software changes in LHO:88527).
The recovery was relatively simple for these suspensions:
- Untripped the software watchdogs for these HAM2 suspensions (H1:IOP-SUS_MC1_DACKILL_RESET, H1:IOP-SUS_MC3_DACKILL_RESET, H1:IOP-SUS_PRM_DACKILL_RESET, H1:IOP-SUS_PR3_DACKILL_RESET) and the HAM2 seismic system beneath it (H1:IOP-SEI_HAM2_DACKILL_RESET).
- Used SEI_HAM2 guardian to bring HAM2 SEI system to ISI_DAMPED_HEPI_OFFLINE (from them having had their watchdogs trip overnight).
- Left / made sure that SUS Guardians in SAFE and unmanaged
- Added a gain of 2^10 = 1024x in the COILOUTF banks in FM10 to compensate for the 18-bit DAC to 28-bit
- Turned off inputs to the COILOUTF, untripped watchdogs, turned MASTER SWITCH ON
- Added small 5 count offsets in COILOUTF, one-at-a-time, and sorted out the MEDM overview macro files***
- lower masses needed adjusted due to the consolidation of DACs
- Computer name changed from h1sush2a to h1sush12
- Tested functionality of binary IO by switching H1:SUS-{MC1,MC3,PRM,PR3}_BIO_{M1,M2,M3}_STATEREQ into various states confirming that the read backs switch correctly (and at this point, we just assume that the binary OUT is actually switching the coil drivers).
- With the SUS guardian still in SAFE, and the MASTER SWITCH still ON, and watching the top mass OSEMs on ndscope ^^^,
- Turned on damping loops one DOF at a time, to confirm that resonant motion exponentially decays as expected without any saturation (reminding ourselves that the DAC saturation limit is now +/-2^27 = +/- 134217728 [DAC ct]_{peak})
- Turned on alignment offsets to see a significant shift in top mass
- Seeing no issue, used the SUS guardian to bring the suspension to ALIGNED, then back to SAFE, then back to ALIGNED.
*** The MEDM overview macro files are
/opt/rtcds/userapps/release/sus/common/medm/
susmc1_overview_macro.txt
susmc3_overview_macro.txt
susprm_overview_macro.txt
suspr3_overview_macro.txt
^^^ Handy ndscope command line to quickly switch between 'scopes of the top mass EULER basis report of OSEM motion
$ optic=MC1;ndscope -k H1:SUS-${optic}_M1_DAMP_{L,P,Y}_IN1_DQ &
Closes FAMIS 27622, last checked in alog 88380
Laser Status:
NPRO output power is 1.86W
AMP1 output power is 70.35W
AMP2 output power is 139.5W
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PDWD watchdog is GREEN
PMC:
It has been locked 6 days, 14 hr 45 minutes
Reflected power = 24.87W
Transmitted power = 105.8W
PowerSum = 130.7W
FSS:
It has been locked for 0 days 16 hr and 45 min
TPD[V] = 0.5086V
ISS:
The diffracted power is around 4.5%
Last saturation event was 0 days 18 hours and 22 minutes ago
Possible Issues:
PMC reflected power is high
TITLE: 12/16 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 15mph Gusts, 10mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 1.00 μm/s
QUICK SUMMARY:
IFO is in IDLE for PLANNED ENGINEERING
A look at today's planned activities:
Workstations were updated and rebooted. This was an OS packages update. Conda packages were not updated.
Jennie W, Masayuki,
Today Masayuki and I went into the optics lab for some JAC prep.
M. Todd, G. Vajente, L. Dartez
Another disk showed problems on cdsfs0, so I replaced it was a spare.
Cdsfs1 has numerous problems. Zpool status is noting that sdl, sdh, sdp, sdo are bad. It looks like the io slot for sdl may be bad (I tried replacing the disk and the system could not see it). I replaced the sdh disk and got it resilvering. Then sdp started resilvering (which was unexpected). I'm going to wait and see how this resilvering goes before I proceed. The activity lights do not match my model of what zpool status is telling me. This is an old box, and replacement/retirement may just be the better path for it.
As a follow-up note, this cdsfs0/1 are old file servers. They used to host and backup /ligo. Now cdsf0 is just used for vm disks, and fs1 backs up fs0.
The eventual goal is to retire them.
After doors came off this afternoon, I went to HAM1 and locked the ISI. Mitch then helped with rearranging a lot of the mass on the table to match the new layout in v6 of D1200524 : https://dcc.ligo.org/DocDB/0089/D1200524/006/D1200524-v6.PDF
We left the two stacks off of the +X side of the table. None of the stacks are currently on viton, I didn't want to leave any loose stacks while JAC was installed. We will rectify that when they are done. I will try to get pictures of the current table layout tomorrow.
(Jordan, Travis, Randy, Jim, Tyler, Gerardo)
Volume for HAM1 was vented this morning, no issues encountered during the process. HAM1 ion pump was isolated before the vent.
Annulus system for both HAM1 and HAM2 was vented with nitrogen gas, both ion pumps were turned off before the annulus system was vented.
Bolts were removed for both doors, we left 4 bolts on each door and then broke for lunch.
After lunch both doors were removed, -Y door had a stubborn O-ring, but we managed to place it back into its dove tail groove, good O-ring. Similar issue for the +Y door, inner O-ring was stuck to the door and it just did not want to stay put, we used the O-ring protectors to persueade it to stay in the groove.
Jim locked the table, then with his help the septum viewport protectors were installed.
Jim and Mitch have the chamber now.
Merge h1sush2a/2b to form h1sush12
Jeff, Oli, Fil, Marc, Erik, Jonathan, EJ, Dave
h1sush2a was upgraded to become h1sush12, with h1sush2b being powered down for now.
In the MSR, the supermicro W2245 h1sush2a was removed from the rack and in its place the new W3323 h1sush12 was installed. The IX Dolphin card and cable was transferred to the new machine.
In the CER H1SUSH2A IO Chassis was upgraded to become H1SUSH12:
The ADC was removed from sush2b to become the third ADC in sush12.
All the 18bit DACs were removed.
3 LIGO-DACs were installed, along with a new 4th ADC.
The 3 Binary cards were left in place. In fact their cables were long enough to remain attached while the chassis was pulled out from the rack.
New models for h1iopsush12, h1sus[mc1,mc3,prm,pr3,im,ham1] were added. Note that h1sushtts is now called h1susham1. A DAQ restart was needed.
More details will follow.
New ASCIMC, LSC models
Jennie, Dave:
New h1ascimc and h1lsc models were installed. A DAQ restart was needed.
Beckhoff changes
Daniel, Dave:
Daniel installed new CS-ISC beckhoff PLC. A DAQ restart was needed.
DAQ Restart
Dave, Jonathan.
The DAQ was restarted for:
retire iopsush2a, iopsush2b. Add new iopsush12
h1sush12 user model changes
new h1ascimc and h1lsc models
EDC restart for ECAT CS ISC.
When we did the daq restart, we did the 1 leg first as that is what the control room is pointed at. The daqd systems did not come back properly. It exposed an error in the checkdaqconfig script that runs before daqd on the data concentrator machine. If the daqd process is restarted too quickly it may only do a partial copy of the ini files to a safe stable location. We had to remove the directory in the daq channel list archive and let checkdaqdata re-run. Then the daq 1 leg came up as expected.
When the daq 0 leg was restarted we rebooted the data concentrator (gds0) to make sure all its interfaces came up properly after a boot. This was done as some of the machines (including at least one of the gds machines) had not come back with all its interfaces enabled.
FW0 has been running out of memory overnight and restarting itself many times. Jonathan found the process which was taking the additional memory and killed it. We expect FW0 to be stable from now onwards.
Mon Dec 15 10:07:40 2025 INFO: Fill completed in 7min 37secs
F. Clara, J. Figueroa, J. Kissel, M. Pirello ECR E2400409 and E2500296 (IIET 35739 and 35706, respectively) WP 12901 DWG D0902810-v12 DCN E2500341 Today we've gone forward with merging the sush2a and sush2b IO chassis, facilitated by upgrading those SUS's DAC cards to 32 CH, 28- bit DACs (D2200368). This entry covers the analog electronics and cabling that were impacted by the change, covering that all of the suspensions in, or soon to be in HAM1 and HAM2 -- MC1, MC3, PRM, PR3, IM1, IM2, IM3, IM4, RM1, RM2, PM1, JM1, and JM3. We followed changes outlined in E2500341 which highlights the changes in the sush12 SUS electronics system drawing from D0902810-v11 to D0902810-v12. That required - disconnecting all affected AI output cables in SUS-C3 and SUS-C4, - Removing the existing 6x 2x 8CH DAC AI chassis (D1000305), - modifying them to become D2500353 1x 32CH AI chassis . replacing the 2x 8CH AI rear panel with WD relays (D1000551) with 1x 32CH AI rear panel without WD relays (D2500097) . replacing the 2x 8CH back panel (https://dcc.ligo.org/LIGO-D1000552 with 1x 32CH back panel (D2400308) - re-installing, then - cabling everything up according to the D0902810-v12 version of the wiring diagram. Here's the list of modified AI chassis serial numbers and assignment: D1000305 > D2500353 D2500097 SUS Chain Rack / Position Chassis S/N Rear Board S/N 32CH DAC Card / IO Slot Channels (Counting from 0) MC1, MC3, PRM TOPs SUS-C4 / U30 S1104370 S2501311 DAC0 / #2 0-15 PRM, PR3 TOPs MC1 MID BOT SUS-C4 / U29 S1104374 S2501315 DAC0 / #2 16-31 MC3, PRM MID/BOT SUS-C4 / U22 S1104375 S2501310 DAC1 / #4 0-15 PR3 MID/BOT, RM1, RM2 SUS-C4 / U21 S1104378 S2501312 DAC1 / #4 16-31 IM1, IM2, IM3, IM4 SUS-C4 / U10 S1104377 S2501314 DAC2 / #5 0-15 PM1, JM1, and JM3 SUS-C4 / U2 S1102760** S2501313 DAC2 / #5 16-31 ** Technically, the (now) PM1, JM1, and JM3 AI chassis S1102760 was a 16-bit DAC AI chassis (D1101521) instead of a D1000305 at the start of today, and it didn't *need* the additional 16CH pass-through SCSI connection, but in the spirit of making everything the same, we elected to make it a full "new normal" D2500353 chassis.
Pictures of the now-D2500353 AI chassis rear boards and their connections. They're attached in the order of rear board serial numbers; the corresponding chassis number is listed above.
Updated H1-SUSH12 Timing FPGA code to latest firmware version 1589 V5
Updated H1-SUSH2AUX timing FPGA code to latest firmware version 1589 V5
Rebooted both chassis and they are now reporting correctly according to Dave.
For reference: The 16-chn/16-bit AI chassis with interface boards D070101 are not compatible with the new 32-chn DACs. Both have a 68-pin SCSI connector but the pinout isn't compatible.