REMOVE SEARCH FILTER
SEARCH AGAIN
Search criteria
Section: H1
Task: DAQ
Since enabling irqbalance on dc0 we seen less CRC errors (only around restart times for the last few days).
Today we enabled irqbalance on dc1. Unlick on dc0 there was a small (but visible) dip in the average cpu use.
We will continue to watch this.
Per WP 13472 updated h1daqfw0 to use the new auto-reconfiguring frame writer. This is the same code has been running on h1daqfw2 for months in a test mode.
I also did an inplace upgrade of h1daqfw0 to Debian 12, which is what h1daqfw2 is running and what we have tested as.
Erik helped with package wrangling for Debian 12. I had been running h1daqfw2 from the experimental repository (which is a snapshot of the continous integration system). We were able to move this from using the experimental repository to the unstable repository. We will move it up to production later. We don't have a full Debian 12 release of advligorts as we are not planning on supporting the rcg and real time code on Debian 12.
Dave and I will be watching the timings and checksums to make sure everything looks good. The IO setup is a bit different between h1daqfw2 and h1daqfw0 (fw2 writes to local disk, fw0 to a NAS).
Started on WP13402. The 1 leg of the daqd system is down for upgrade. The control room workstations and FOM displays were pushed to using the h1daq0 nds last night.
Dave, Tony, and Jonathan
Supporting WP 13402 we did some prep work.
* Reviewed the work plan.
* Added the new daqd switches to the DNS as sw-msr-daqd[0,1,2].
* Configured sw-msr-daqd0 and moved it to its final position.
* Reviewed the inventory of fiber, nics, and optics.
* Put a hostname override into puppet the workstations and display machines to point them towards h1daqnds0 so that tomorrow we can take h1daqnds1 down.
As per WP 13401. In preparation for upcoming daqd upgrades I'm working to move the run number server off of h1daqscript0 and onto the HA container cluster. I have the new instance running with the current run number database. I just need to configure the framewriters to point to it. I will do that next week. The run number server helps keep the run number that is put into the frames consistent between the two frame writers.
This is a step to retiring the h1daqscript0 machine once we remove the daqd's px dolphin.
WP 13326
Dave noted we lost Beckhoff channels yesterday around 6PM. Daniel logged in and confirmed terminals missing in Corner Chassis 4. The third EtherCAT coupler was replaced.
Chassis Serial Number S1107450
D. Barker, F. Clara, D. Sigg
I updated and restarted the frame writer on h1daqfw2 yesterday at 1:37pm and today at 3:04pm localtime.
I updated and restarted the experimental framewriter on h1daqfw2 today around 2:20pm localtime today. This was to verify some new code still produced identical frames.
Upgraded the main slow controls software to add some calibration features for CHETA.
Restarted the slow controls system to fix an issue with the CHETA motor stages.
Updated the slowcontrols SDF for h1syscs[aux,isc,tcs]sdf, restarted these processes on h1ecatmon0 and accepted+monitored the new not-init channels.
The update revealed that there are 12 CS AUX channels which are pending to be added to the DAQ (H1EPICS_ECATAUXCS.ini).
+[H1:SYS-MOTION_C_PZTSHUTTER_B_COUNT]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_ERROR_CODE]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_ERROR_FLAG]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_HIGH]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_LIMITS]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_LOW]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_NULL]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_RANGE]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_STATE]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_TRIG]
+[H1:SYS-MOTION_C_PZTSHUTTER_B_VOLTS]
We had some hiccups in our end of day daqd restart.
1. There were duplicate channels between the h1susb13 and h1susb2h34 models. We are not sure how these got through, the daqd should have crashed on startup with an error message. We found this as it caused issues on the test framewriter on h1daqfw2.
2. After Oli removed the old block from h1susb13 we found that there were some GDS broadcaster channels that were dropped that probably shouldn't be. We removed them from the broadcast list in order to get the daqd to run. Listing the channels here so that they can be added back in tommorow:
After looking through the code, this is an old bug. It has been fixed for a while. The daqd had not been updated with the rcg for a long time due to no feature changes.
We should do a daqd package upgrade next week.
Dave, Jonathan,
We restarted the daqds today, daq1 leg at 15:56, and the daq0 leg at 16:01 localtime.
This was in support of two workpermints
12945 - monitoring the daq kafka connector as we push data into the NGDD system in ldas (NGDD = next generation data delivery)
12946 - TJ added two guardian nodes to help manage JM1 and JM2
Added the cabling for the new whitening concentrators. This also required a software upgrade.
The following QPD whitening channels were affected:
Instead of 4 x 8-bits, only 1 x 8 bits are used and all 4 segments are switched simulatanously. There is no change in the whitening interface for these QPDs, since the software already tied the segments together. The only change is in the readnback bits where only the last 8-bits are still significant.
This allowed us to add the whitening for the JAC WFS as well as dewhitening for the JAC PZT driver.
We have been seeing a low rate of data drops with the frame writer serving as both the frame writer and the data concentrator for the DAQD 0 systems. This work is to try an alternate configuration. Moving the gds broadcaster to be a data concentrator and making the frame writer single purpose again.
The data drops are due to messages from teh front end arriving too late and being discarded. We are not seeing them on the DAQD 1 leg, only the 0. This leads us to tend towards the issue be the load (and thus responsivness to input messages) of the FW machine.
Today I adjusted the system such that GDS0 became the data concentrator and the broadcaster.
I've attached a diagram of the current layout.
The main changes:
The final migration followed this rough order
The control room monitoring and medm screens still expect a DC, FW, TW, NDS, and GDS. So gds0 as the DC is exporting DAQ-DC0 variables, and we are running a epics proxy ioc which maps DAQ-GDS0 channels to DAQ-DC0 for now. This will change in the future. Most of the DC0 variables will probably be taken over by the cps_recv process (this is in testing on the large test stand in LLO).
There were a few daqd restarts to make sure everything was working. Data is flowing and fw0 and fw1 are producing identical frames.
The plan is to let this run though til January and evaluate the error rate.
A quick summary of the current TwinCAT setup:
Any change of hardware needs to be reflected in the Altium workflow which serves as the basis for the system project using the provided scripts.
The Altium script will generate H1EcatC1_NetList.xml. ProcessTcNetList.ps1 will then use the netlist as an input to generate H1EcatC1_BoxList.xml and H1EcatC1_Mapping.xml.
(All located in C:\SlowControls\TwinCAT3\Source\Interferometer\H1EcatC1\Configure).
We have updated the spare Beckhoff computer to this version. The upgrade of the main Beckhoff computer is pending.
Marc Daniel
We upgared the EtherCAT Corner Station Chassis 5 according to D1200132-v4 and E1200077-v4. The corresponding software changes were also comitted. This now includes all necessary upgrades to support JAC and most of the ones needed for BHD.
The picomotor controllers were not working. The software side looked ok, but there was no physical drive signal. The TwinCAT system showed an error message about "nonsensical priority order of the PLC tasks". In the past, we ignored these messages without any problems. After fixing this issue and re-activating the system, it started working again. Not usre if it just needed a restart, or if the priority order has now become important. More investigation needed.
As the new framewriter is a different piece of software, we track different metrics. So we will need an updated edc capture list and a daqd restart.
A note about what this system brings in.
* It automatically changes its configuration when there is a channel list. It is a step towards not having daqd restarts (those don't go away while we have the daqd in other places).
* When we restart the daqd we usually have a gap of several frames. We hide this from the outside world by having two frame writers and only restarting one at a time. This reconfigures the channel list and does not have gaps unless you actually restart the program.
* It is also being used in the ngdd work to write derived frame data, so that we have a common frame writer (we just change the input type).