Reports until 16:36, Monday 06 May 2019
H1 CDS
david.barker@LIGO.ORG - posted 16:36, Monday 06 May 2019 - last comment - 19:18, Monday 06 May 2019(49046)
h1edc change to stop accessing the file system

WP8197 h1edc DAQ-CRC mitigation

Jonathan, Dave:

During this afternoon's earthquake downtime, I made the h1edc code change to stop it checking H1EDC.ini file's checksum. There is clear evidence that the /opt/rtcds file system slows between 4am and 5am local, and for the past two mornings h1edc's DAQ-CRC counter has incremented around these times.

Remember that h1edc is unique in that it builds out of RCG-3.5.0 area. I changed the file

/opt/rtcds/rtscore/advLigoRTS-3.5.0/src/epics/seq/edcu.c and commented out the code block:

        // Check file CRCs every 5 seconds.
        // DAQ and COEFF file checking was moved from skeleton.st to here RCG V2.9.
        /* %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
           Do not check H1EDC.ini checksums. This may be causing DAQ-CRC errors at LHO during O3
                                                                              D.Barker LHO 06may2019
        if(!fivesectimer) {
            status = checkFileCrc(daqFile);
            if(status != daqFileCrc) {
                daqFileCrc = status;
                status = dbPutField(&daqmsgaddr,DBR_STRING,modfilemsg,1);
                logFileEntry("Detected Change to DAQ Config file.");
            }
        }
        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% */

On h1susauxh34, in the /opt/rtcds/lho/h1/rtbuild/rt-3.5.0 build area I did the  compile sequence

Note that the install overwrites the H1EDC.ini with a 'fec version' of this file. This generated a 0x2000 DAQ error, though the actual data was unchanged. I cleared this error by regenerating  H1EDC.ini  using my script.

I restarted h1edc at 15:35 PDT, and then cleared the DAQ-CRC counters.

To test that h1edc is no longer checking H1EDC.ini, I edited this file (added a trailing blank line) and no 0x2000 error ensued.

Comments related to this report
keith.thorne@LIGO.ORG - 19:18, Monday 06 May 2019 (49052)
We have seen occasional CRC errors on l1edc, but not at this level.  Perhaps we will trend to see if a pattern persists

At LLO, a separate SSD-based file server pair (1lfs1/l1fs2) handles the front-end file system, with a larger spinning-media server pair (cdsfs1/cdsfs2) for workstation-specific file systems