[Halld-online] [New Logentry] DAQ Testing

[davidl at jlab.org]

Logentry Text:
--

We met to continue the program of large scale DAQ testing. 

Participants:  Sergey F., Vardan G., Carl T., Dave A., David L.


- Setup to run from hdops account by installing from hdsys account

  - Installed new DAQ using hdsys into /gluex/daq/daq_dev_vers.
    To use this, one must do the following after logging into the hdops
    account:

      source /gluex/daq/daq_dev_vers/work/online_setup.cshrc

  - Changed permissions on “log” and “ddb” directories in cool so
    that hdops can write to them
  - Changed gluon50-daq to just gluon50 in ~/.rcm.ini so that ethernet
    interface would be used instead of IB. This was done because the
    IB interface is currently down
  - Removed the setup.xml file installed by default in $COOL_HOME/gluex
    since it was causing the platform to try and use “furletov” and “gluon50”
    instead of defaulting to the local machine (gluon02)
  - Built the mcROL.so in $DAQ_HOME/vme/src/rol_soft. Also modified
    the online_setup.cshrc script to set CODA_ROL to this directory

- 2 crate configuration: SoftROC

  - Started up rcm.sh and selected:
      RUN: TEST
      SETUP: SoftROC
      TRIGGER: PULSER

  - All components started OK (platform, PEB, ER, ROCSTPSC1, ROCSTPSC2, rcgui)
  - Configure failed. RTVs not set. Message is buried in exception from platform.
  - Tried resting and configuring again. RC again reports error “Control Supervisor is no registered”
  - Stop all components via rcm and restart
  - Noticed message from platform: “tee: log.platform: Permission Denied”. Both log
    and lock files appear to be in place however in the $COOL_HOME/gluex/log directory
    with correct timestamps and owned by hdops
 - setup.xml needs to be writable by hdops
 - rcm config. is in both .rcm.ini and a file in work. The /work file saves last settings
   and is used for storing last settings used. These may need to be consolidated.


At this point we switched over to testing the 50 crate system from Sergey's account.

- Testing 50 crate system fast startup and shutdown

  - Attempt 1
    - All crates except roctrig1 started. roctrig1 was not responding to ping. Crate
      was power cycled remotely via:

      rocs.sh trig:1 

     (syntax is: basename:num1,num2,num3,..NumN  or basename:num1:numN

    - roctrig1 came up OK and restarted all components. started is much faster (few seconds)
    - tested fast shutdown of components. It worked within a few seconds as well

  - Attempt 2
    - Configure failed due to some components connecting to the wrong gluon
      due to the direct connection being set to a different node than we’re on.
      (we’re using gluon50 and configuration used gluon53). Switching to using
      gluon53

  - Attempt 3 (using gluon53)
    - Configure succeeded
    - Download succeeded
    - Prestart succeeded
    - Go succeeded (forgot to start softROCcontroller)

  - Attempt 4
    - Reset and try again. (don’t kill any components)
     - Configure succeeded
    - Download succeeded
    - Started softROCcontroller
    - Prestart succeeded
    - Go succeeded
    - Data flowed for 1million events
    - Started hd_ana connecting to ER’s ET system. Very odd behavior:
      Event rate went up by a significant amount (to 1.5kHz to 2kHz from
      around 200Hz) when hd_ana was reading events from ET
      This propagated all of the way back to the ROCs.
      Stopping hd_ana caused rate to drop back down.
      Current theory is that there are too few events in the ET system created
      by the ER. The SEB is trying to grab 10 at a time, but the whole system
      has only 20. The addition of hd_ana which reads events 1 at a time 
      smoothes out the flow resulting in fewer failed requests and therefore
      fewer sleeps waiting for events to appear.

  - Attempt 5
    - Changing number of events CODA starts ET system with by editing
      COOL “Options/ERsoftROC.xml” file
    - Configure, Download, Prestart, and Go succeeded
    - Event rate behavior is the same.

This event speed-up phenomenon exposed an issue that will be addressed by the CODA group that
should help smooth out and possibly speed up the data rate.

- Testing fast starts and stops




---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/3289248
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-online/attachments/20140708/3852b4fa/attachment.html