[Clas12_software] A new queue is created for your jobs

[Vardan Gyurjyan]

Chip,

I think that the main operational mode is going to be the off-line data processing mode, where each farm node will process data using local IO services. So, in this mode we will have N data streams (coordinated by the physics data processing application orchestrator, running on the ClaRA executive node), where N = number of farm nodes at the active ClaRA expansion. 
The online mode tests are the proof that ClaRA can be used to process data in real time on the JLAB computing farm, and for Hall-B online data processing farm hardware estimate, in case Hall-B resources will permit purchasing one in the future. 

Sincerely,
Vardan  

----- Original Message -----
From: "Chip Watson" <watson at jlab.org>
To: "Vardan Gyurjyan" <gurjyan at jlab.org>
Cc: "Ying Chen" <ychen at jlab.org>, scicomp at jlab.org, "clas12 software" <clas12_software at jlab.org>
Sent: Monday, June 10, 2013 2:36:22 PM
Subject: Re: A new queue is created for your jobs

Vardan, 

Do yo u mean that the mode of running the data stream for multiple farm nodes through one I O node only app lies to live data, and not to stored data? Is it your plan to have analysis done by single nodes reading their own data directly? 

Can you describe how this will be done? What is the operational plan for looking at the online data using farm nodes? 

Chip 


On 6/10/13 2:08 PM, Vardan Gyurjyan wrote: 


Thank you Ying,

We run our single data-stream test using infiband. The test shows that ClaRA scales linearly over 10 nodes, and is able to keep up with 2KHz DAQ rate. However, we would like to scale more than 10 nodes to see when we will face a bottleneck. I would like to mention that this mode is for the online/real-time processing of the experimental data. 

The next step will be to start tests using multiple data-streams, i.e. separate IO for each node (off-line data processing/cooking mode). Currently I am writing so called staging service that is responsible for the data-file management (copying data-files to and from the local persistent storage). We will need, again, more than 10 farm nodes in the queue for the off-line data processing mode test.

Sincerely,
Vardan