Spliting the End Station ISC Models Into Two
Daniel, Jim, Dave:
WP5034. The problem: end station isc models are running long, so their RFM IPC to the corner station do not make it in time. The solution: split the ISC model up into ALS and ISC, with the latter running faster and able to send RFM IPC to the corner in the allocated time (60uS).
Because the h1iscex,ey computers did not have any spare cores (iop, pem, isc, odc, cal) for now we added CAL to PEM to free up a core. Hopefully a better long term solution is to combine isc and odc.
Daniel had done a lot of early legwork getting the models into shape. He hand edited the H1.ipc file to change the source model name for channels moving from ISC to the new ALS models. Daniel also handled the filter module files.
We identified all models which receive IPC channels (including Dolphin) from the new ISC and ALS models. These were also recompiled and restarted.
First thing this morning we moved the new H1.ipc file into place and did a round of make and make-install on the related models. The testpoint.par file needed editing during the "make install" process due to the model-dcuid change for the isc models.
The rtsystab file was modified for the new model layout in the end station isc front end.
Then all the related models were stopped:
h1lsc, h1asc, h1susetmx, h1susetmy, h1hpietmx, h1hpietmy, h1pemex, h1iscex, h1odcx, h1calex, h1pemey, h1iscey, h1odcy, h1caley
The h1iopiscex and h1iopiscey were restarted. We saw an awgtpman autostart problem with these models.
The end station models were restarted (h1pemex, h1iscex, h1odcx, h1alsex, h1pemey, h1iscey, h1odcy, h1alsey). We had to play around with the safe.snap files for the ISC and ALS models to get them to startup.
The receiving models were restarted (h1lsc, h1asc, h1susetmx, h1susetmy, h1hpietmx, h1hpietmy)
The IPC from the end station ISC models to the LSC are now without any errors. The ALS model still has some IPC to the LSC and this retains its error rate (this model was not substantially sped up).
The DAQ master file was reconfigured for the new model layout, and H1EDCU_DAQ.ini was resynced.
I checked the DAQ data for the PEM and found it was corrupted. This was tracked to the change of the ADC part name in the model when PEM and CAL were combined, and we forgot to remap the bus selector parts attached to the ADC. This was fixed, there is a two hour gap in PEM data.
CDS OVERVIEW MEDM
Dave:
The H1CDS_STATE_WORD_CUSTOM.adl overview MEDM screen was updated:
Another reboot of h1alsey was needed to fix a wrong ADC channel assignment. If memory serves right, there is a busted ADC channel in EX but then this "fix" was propagated to EY when the new ey model was templated from ex.
Current max CPU times:
The IPC errors from the red transmitted power and the red QPDs have disappeared. We still have the channels H1:ALS-X/Y_ARM with IPC errors. These are sent from the als models. They are no longer needed and can be eliminated.
IPC errors from the corner to the ETM SUS have not changed and are around 10-20 Hz. This is not surprising, since the corner models were not updated.
Noticed that the ODC models in EX/EY run at 32kHz. Since the end stations only run at 16kHz, this is unnecessary.
All safe.snap and filter files are up to date. medm screens updated where needed. All changes in svn. Looks like this change is a success.
Still to do:
Here is the link to the busted iscex ADC channel which prompted the software fix.