[Halld-online] [New Logentry] DAQ Testing

[davidl at jlab.org]

Logentry Text:
--

- Initial setup
  - Used hdops account already logged into gluon02
  - rcm.sh issues command:  daq_setups.sh -rcm 50 -s SoftROC
    This fails to open any windows. Maybe it does not understand
    the “-rcm gluon50” argument (it’s not listed in the usage statement)
  - Run “daq_setups.sh -s SoftROC” manually
  - Some windows open, but others fail due to “Maximum number of clients reached”
    - Did quick web search. Found this page:
      https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/910539
    - The file /etc/ati/atiapfuser.blb on gluon02 had 528 open file descriptors
      I renamed it to /etc/ati/atiapfuser.blb.bak, but this didn’t help.
    - Closed all windows that were left open and still ran into same error
    - Logged out and back in to hdops account on gluon02
    - Windows opened OK

  - Started SoftROC configuration using rcm.sh. 
  - Configure succeed
  - Download failed due to undefined symbol for evio::evioException
    in the /gluex/coda/3.01/Linux/lib/libcodaChannels.so file
  - This was due to building mcROL.so with a new Makefile that did 
    not include the evioxx (or expat) libraries
  - Fixed mcROL.so link problems

  - Started SoftROC run with internal trigger loop at 10000
    - Event rate was 200-300Hz
    - Starting hd_ana did not affect event rate as it did last time
    - ET event rate was only about 2Hz
    - hd_ana event rate also about 2Hz (should have been higher)

 - Started SoftROC run with internal trigger loop at 1000
    - Event rate was 2kHz
    - Starting hd_ana did not affect event rate
    - ET event rate was still only about 2Hz
    - hd_ana event rate also about 2Hz (should have been higher)
    - further debugging needed

- Moving to 50 crate test
  - Goals:
    * Run on hdops
    * Use multi-cast instead of direct
    * Run CODA components on different nodes

- Ran 50 ROC configuration with 1k events per chunk in softROCcontroller
  - Still using direct mode
  - All components running on gluon53
  - Event rate ~2kHz
  - No change in event rate when hd_ana connected.
  - Problem with ET rates appears fixed

- Ran 50 ROC configuration with 5k events per chunk in softROCcontroller
  - Event rate was more erratic and within a few seconds, event rate dropped to zero
  - No error messages observed indicating exact source of problem

- Ran 50 ROC configuration with 2k events per chunk in softROCcontroller
  - Event rate was rock solid at 2kHz. No deviation was observed whatsoever
  - Ran for several minutes and ended run

- Switching to multi-cast configuration
  - BCAL Data Concentrator dies prior to Configure
  - Restarted again and this time it stays alive
  - noticed multiple instances of coda_roc running on ROCs (Sergey will fix the bug later)
  - Stop all processes and restart

  - Need to run on multiple machines when using multi-casting. Reconfigure
    to start all EMUs on different hosts. Reconfigure hosts file

  - Started all components
  - BCAL DC ended again with timeout

  - Started all components
  - All components connected OK this time
  - Configure succeeded
  - Download succeeded
  - Prestart failed  because SEB could not connect to ER’s ET system
  - The ET system appears to be setup correctly and can be accessed via
    tcp port using hd_ana and et_monitor. Unclear why SEB can’t connect 
    using multi-cast

 - Try again. Shut down all processes and restart
  - Configure failed when roctrig1 died
  - Restarted shmem_serv on roctrig1
  - Killed all components and restarted them
  - Configure failed due to same issue with failure to connect to ET system via multi-cast

- Change configuration to use direct for the SEB to ER connection. All others left 
  as multi-cast
  - BCAL DC dies again shortly after startup
  - Restart everything once more. BCAL DC connects this time
  - SEB to ER direct connection succeeds
  - DC to SEB connections using multi-cast fail 





---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/3289375
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-online/attachments/20140710/4f2dd8ba/attachment.html