REMOVE SEARCH FILTER
SEARCH AGAIN
Search criteria
Section: H1
Task: DAQ
Dave, Jonathan,
We restarted the daqds today, daq1 leg at 15:56, and the daq0 leg at 16:01 localtime.
This was in support of two workpermints
12945 - monitoring the daq kafka connector as we push data into the NGDD system in ldas (NGDD = next generation data delivery)
12946 - TJ added two guardian nodes to help manage JM1 and JM2
Added the cabling for the new whitening concentrators. This also required a software upgrade.
The following QPD whitening channels were affected:
Instead of 4 x 8-bits, only 1 x 8 bits are used and all 4 segments are switched simulatanously. There is no change in the whitening interface for these QPDs, since the software already tied the segments together. The only change is in the readnback bits where only the last 8-bits are still significant.
This allowed us to add the whitening for the JAC WFS as well as dewhitening for the JAC PZT driver.
We have been seeing a low rate of data drops with the frame writer serving as both the frame writer and the data concentrator for the DAQD 0 systems. This work is to try an alternate configuration. Moving the gds broadcaster to be a data concentrator and making the frame writer single purpose again.
The data drops are due to messages from teh front end arriving too late and being discarded. We are not seeing them on the DAQD 1 leg, only the 0. This leads us to tend towards the issue be the load (and thus responsivness to input messages) of the FW machine.
Today I adjusted the system such that GDS0 became the data concentrator and the broadcaster.
I've attached a diagram of the current layout.
The main changes:
The final migration followed this rough order
The control room monitoring and medm screens still expect a DC, FW, TW, NDS, and GDS. So gds0 as the DC is exporting DAQ-DC0 variables, and we are running a epics proxy ioc which maps DAQ-GDS0 channels to DAQ-DC0 for now. This will change in the future. Most of the DC0 variables will probably be taken over by the cps_recv process (this is in testing on the large test stand in LLO).
There were a few daqd restarts to make sure everything was working. Data is flowing and fw0 and fw1 are producing identical frames.
The plan is to let this run though til January and evaluate the error rate.
A quick summary of the current TwinCAT setup:
Any change of hardware needs to be reflected in the Altium workflow which serves as the basis for the system project using the provided scripts.
The Altium script will generate H1EcatC1_NetList.xml. ProcessTcNetList.ps1 will then use the netlist as an input to generate H1EcatC1_BoxList.xml and H1EcatC1_Mapping.xml.
(All located in C:\SlowControls\TwinCAT3\Source\Interferometer\H1EcatC1\Configure).
We have updated the spare Beckhoff computer to this version. The upgrade of the main Beckhoff computer is pending.
Marc Daniel
We upgared the EtherCAT Corner Station Chassis 5 according to D1200132-v4 and E1200077-v4. The corresponding software changes were also comitted. This now includes all necessary upgrades to support JAC and most of the ones needed for BHD.
The picomotor controllers were not working. The software side looked ok, but there was no physical drive signal. The TwinCAT system showed an error message about "nonsensical priority order of the PLC tasks". In the past, we ignored these messages without any problems. After fixing this issue and re-activating the system, it started working again. Not usre if it just needed a restart, or if the priority order has now become important. More investigation needed.
The atomic clock has been resynchronized with GPS. The tolerance has been reduced to <1000ns again.
Marc Daniel
We finished the software chanegs for EtherCAT Corner Station Chassis 4.
We found 2 issues related to the power outage:
Baffle PD chassis in EX has a bench suppy that needs to be turned on by hand.
The system is back up and running.
WP 12915
Corner Station Controls Chassis 4 Wiring Diagram - E1101222
EtherCAT Corner Station Chassis 4 - D1101266
Modifications to the Beckhoff chassis - E2000499
The EtherCAT Corner Station Chassis 4 was modified per E2000499. New Beckhoff terminals were installed to support JAC and BHD. A new rear panel was installed to accommodate the new rear adapter boards and shutter control connectors. The din rail power terminal blocks were relocated making the exising power cables too short. New power cables installed to the EtherCAT couplers and power terminals. All field cabling was reconnected, expect for ISC_313. The Beckhoff software will need to be updated.
EtherCAT Corner Station Chassis 4 - Serial Number S1107450
F. Clara, D. Sigg
We switched to the spare slow controls computer that is running Windows 10 IoT Enterprise LTSC, 21H2 (OS build 19044.4780), and the most recent TwinCAT 3.1 (4026.19.0). This TwinCAT version uses a packet manager and has changed the install directory to C:\Program Files (x86)\Beckhoff\TwinCAT. It supports Visual Studio 2022 and the new Altium workflow.
One new "feature" is that the PLC boot project needs to be activated in a separate step (previous versions would do this automatically after a build). We updated the install scripts to accommodate this.
We also tried a new TcIoc that was linked against EPICS 7, but it crashed during SDF restore repeatedly. We reverted back to the previous version linked against 3.15.9.
Jonathan and Yuri
This is a continuation of WP 12886. This is the last physical step in the control room/MSR for the reconfiguration. We ran a fiber from the warehouse patch pannel to h1daqdc0 and replaced the network card with a newer card. We will be renaming h1daqdc0, likely to h1daqkc0 (daqd kafka connector), and converting it to send the IFO data into the LDAS as part of the NGDD project.
Next steps on this work is to work on the software setup, and to install the corresponding fiber patch in the LDAS room.
Jonathan, Dave, Erik, Richard,
As per WP 12886 we reconfigured the DAQD 0 computers. The goal was to combine the functionality of the data concentrator (DC0) and frame writer (FW0) into one machine, consolidating them to one machine. This then frees up the other machine for use with pushing data into the NGDD (Next Generation Data Delivery) kafka brokers in LDAS.
At this point h1daqdc0 is no longer the data concentrator. H1daqfw0 is the data concentrator + frame writer.
This required a few other changes:
Once this work was done and h1daqfw2 was shown to be producing identical frames to h1daqfw[01] work was able to move onto the consolidation.
The basic steps:
the H1:DAQ-DC0 epics variables are used in many places, so Dave and Jonathan configured h1daqfw0 to output H1:DAQ-DC0 epics variables, and put a small IOC together to output the set of H1:DAQ-FW0 variables we need. This is an area we need to revisit. One likely approach is to make use of a feature in cps_recv that outputs the daqd STATUS, CRC_CPS, CRC_SUM variables and then to move the daqd on FW0 back to outputing the FW0 variables.
This work validates that we can combine the DC and FW computers into one.
The next step is to turn the old DC machine into the producer for NGDD data. We will run the fiber to connect it through to LDAS later this week and work on setting that system up.
My checklist for this is in the DCC https://dcc.ligo.org/T2500385
I have updated the MEDM screens for the OSEM Estimator and committed them to Userapps. This captures the name changes made to the models for SR3 and PR3 at LHO. These name changes just make the names more sensible, and do not change any functionality. LLO can pull the updates and it will make no difference because nothing at LLO uses these screens.
userapps/trunk/sus/common/medm$ svn ci -m"Mods to estimator screens to match name update in the estimator models, BTL"
Sending estim/CONTROL_6.adl
Sending estim/ESTIMATOR_OVERVIEW.adl
Sending estim/FADE_CONTROL.adl
Sending hxts/SUS_CUST_HLTS_OVERVIEW_W_EST.adl
Transmitting file data ....done
Committing transaction...
Committed revision 32544.
Note - the indicator lights for the filters on the Estimator Overview screen are not working correctly, and I plan to fix this in the next few days. the indicators in the control_6 screen have been fixed.
(They should be grey=output off, green = output on, settings as expected, red = output on, settings not correct.)
There are many settings to capture in the SDF file. Most of these are the filter settings. Those will come up in the OFF setting, and that is good, and fine to capture.
A few things do need to be updated. These include:
In the Estimator Overview screen
set the ramp time to what it was before - 5 seconds is fine
set the Initial channel to 1 (this is OFF)
set the next channel to 1 (this is also off, and it should match the initial channel. This channel will change during operation, so maybe you don't want to monitor it?)
click the OSEM select and then set the correct channel to 1 (the 'hint' at the bottom says which one it is. for Yaw estimator, pick yaw and for pitch estimator, choose pitch)
As per WP 12455 Dave and Jonathan replaced the power supply on h1daqframes-1. Due to the available space in the rack we decided to power off the system so that we could move it safely and replace the power supply.
We did some double checking to verify which machines where hooked together. We followed the physical connections and cross checked them with the names and numbers for interfaces on each machine. This was good as the labels on the front of the system where wrong. This is the frame disk associated with h1daqfw1.
We powered down h1daqfw1, verified that the link to h1daqframes-1 had gone dark. After that we powered off h1daqframes-1 pushed it forwared enought to replace the power supply, pushed it back in th place and restarted it.
After the disk server was back we restarted h1daqfw1 as well.
Dave will fix the labeling.
Labels are now good. MSR-RACK09 U02-U03 (bottom unit) is h1daqframes-1. MSR-RACK09 U04-U05 (next to bottom unit) is h1daqframes-0.
Closed FRS27399
h1daqfw2 is now producing identical raw, second trend, and minute trend frames to h1daqfw[01]. The issues mentioned in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=84132 have been resolved. There are times where we expect there to be mismatches. These are generally when data is missing, the daqd may replay recent data, where the new frame writer will write zeros. We will continue tests on the frame writer, including software updates. The next test is to watch the behavior of h1daqfw2 when h1daqfw0 & h1daqfw1 have a restart. It is designed to automatically adjust its channel list and have no gaps in the frames and need no restart to accommodate the changes.
Jonathan, Dave, We replaced the hardware that is running h2daqfw2 today. This was done to put in a system that had more memory. We pulled a spare daq unit out of the test stand (most recently named x6dc0). While doing that we swapped the memory with a broken daqd system, so that the new h2daqfw2 now has 256GB of ram. The system is up and running again, I'm tracking the setup in puppet so that we have a low rebuild time. This was done to support investigating differences between the frames generated on h1daqfw[01] and hqdaqfw2. I need to be able to run modified versions of daqd to see some of the internal state that is not exposed to be able to understand and fix differences in encoding of the frames. When I ran daqd on the old hardware, it ran out of memory, even after reducing buffer sizes.
The TwinCAT slow controls software was updated to incorporate LSC-REFL_B. This includes controls and readbacks for a new demod chassis and a new delay line.
Today I setup h1daqfw2 as a platform to test a new frame writer for use after O4. For the fw hardware I repurposed h1digivideo3 which is an older Xeon server with 10 cores and 64GB of ram. I added 2x2TB old hard disks in a RAID 0 config (to improve the write performance). At this point I am not looking to do any mid to long term storage of frames. I did not connect this up to the data stream via dolphin. Instead I am running a new instance of cps_xmit on h1daqnds0 and using that over a new dedicated 1g link to h1daqfw2. I've updated the puppet config for h1daqnds0 to make this a persistent change. At this point I am running the new frame writer on h1daqfw2, and it is producing frames. I need to do some more configuration (mainly around the run number server) so the frames will be identical to those output from the other frame writers (the difference should be in metadata in the frame headers, not the recorded data). In simulated data setups I have produced frames that are byte for byte identical to daqd frames so it is fairly likely that after I get that working I will see identical frames. The point of this frame writer is to move towards a auto-reconfiguring/restartless system that is able to adjust on the fly to channel changes, remove some other limitations in the daqd, and to become the ngdd projects frame writer for downstream derived data products. The first things I will look at with this is memory and cpu requirements under the H1 load. This testing will be ongoing.
