[Halld-offline] Offline Software Meeting Minutes, April 6, 2018

[Mark Ito]

Folks,

Please find the minutes below and at

https://halldweb.jlab.org/wiki/index.php/GlueX_Offline_Meeting,_April_6,_2018#Minutes 
.

   -- Mark

___________________________________


    GlueX Offline Meeting, April 6, 2018, Minutes

Present:

  * *JLab: * Alex Austregesilo, Amber Boehnlein, Thomas Britton, Mark
    Ito (chair), David Lawrence, Simon Taylor, Beni Zihlmann

There is a recording of this meeting <https://bluejeans.com/s/meuB8/> on 
the BlueJeans site. Use your JLab credentials to access it.


      Announcements

 1. *New tag of sim-recon for simulation: recon-2017_01-ver02-sim_ver01
    <https://mailman.jlab.org/pipermail/halld-offline/2018-March/003137.html>.*
    This tag is meant for use by simulations.
 2. *Meeting on Containers, 11:30 today, A110.* Note that future
    containers meetings will be in A110 as well.
 3. *Launches.* Alex A. is starting a new analysis launch on 2016 data
    for Lubomir Pentchev.
 4. *Software Review, Summer 2018.* Mark reported that there are plans
    for another software review, perhaps early in the Summer.
 5. *SQLiteCpp and SQLite libraries.* Mark reports that there are still
    some residual problems with these libraries in the builds. He is
    working on a solution and a new version should be coming out soon
    with fixes. One holding item is to see if the recent fix to the
    memory leak problem in tracking does the job.


      Review of minutes from the March 9 meeting

We went over the minutes 
<https://halldweb.jlab.org/wiki/index.php/GlueX_Offline_Meeting,_March_9,_2018#Minutes> 
without much comment.


      Report from the GlueX Containers meeting on March 30

Mark reviewed his notes from the meeting 
<https://halldweb.jlab.org/wiki/index.php/GlueX_Containers_Meeting,_March_30,_2018>. 
Two major topics from that meeting:

 1. Mark reviewed the talk he gave at the OSG All-Hands meeting on
    March ??? He emphasized the new user-friendly access to the OSG for
    GlueX collaborators and raised the idea of doing raw data
    reconstruction on the OSG.
 2. Richard reported on work he has done using XROOTD to stream data
    over the network as input to jobs running on the OSG. This solves
    the problem of getting our data to a remote grid node when the
    identity/location of the node is unknown until the job starts. It is
    the solution that Atlas and CMS have been using for years and relies
    on mature technology. We hope to set up a system on the Submit Host
    in the near future.

For all of the details, see the notes at the link given above.


      Progress on Running Simulation on the OSG

Thomas described the latest go-around submitting jobs to the OSG using 
MCwrapper.

  * There was no wait for jobs to start up. They were running minutes
    after they were submitted.
  * We observed use of a large fraction (30-40%) of the 1 Gb interface
    on the Submit Node at multi-job-start-up time.
  * Amber mentioned that in the interim report to the Office of Nuclear
    Physics, the ability to plan large simulation campaigns, outside of
    peak demand relief was not mentioned. Neither was the ability to
    reconstruct raw data.
  * Amber reported that NERSC is committed devoting resourced to GlueX
    reconstruction as part of their strategic agenda. They need to have
    specific deliverables and so we need to identify appropriate
    data-sets/code-versions for running there. We hope to do this this
    summer.
  * Alex mentioned bookkeeping as an issue when running on the OSG.
    Indeed when the data challenges were run, the OSG component was not
    tracked with a database, only the JLab-resident jobs were. That used
    a database developed by Mark. On the batch farm, using SWIF, Alex is
    tracking jobs with the SWIF database (although we do not have
    SQL-query level access to the SWIF database). Mark thought that
    bookkeeping is actually where most of the GlueX development manpower
    will be spent in getting these large-scale campaigns under control.
    Functions include basic job info, resource use, job status,
    evaluation of job success, re-submission on failure, and permanent
    storage and cataloging of output files. Amber mentioned a Fermilab
    project that we might want to look at. She remarked that the need is
    global to all groups at the lab and beyond and a common solution
    would avoid duplication of effort.


      HDvis Update

Thomas has added new features to HDvis:

 1. Scrubber bar: used to click and drag time-in-event.
 2. Camera snap-to's: Fixed views of the detector, such as "barrel top"
    or "TOF".
 3. Context menus for choosing mass hypothesis: right click and choose
    particle type for charged tracks. Tracks redrawn with chosen hypothesis.

He is working on a method for implementing a "next event" that will work 
for multiple users running independent browsers on the same node. See 
the demo displaying a J/ψ event at ??? in the recording.


      Monitoring Dashboard

Sean sent email proposing a monitoring dashboard (with colors) that 
would display all relevant monitoring results at a glance with links for 
digging deeper when necessary.

  * Alex would like to see things like ρ yield.
  * We might want to add Monte Carlo data to the recon_test.
  * There is simulation test that runs twice a week. Not everyone is on
    the email list for results.
  * David is looking at InfluxDB as a way to look at selected quantities
    as a function of time or run number.
  * We will meet with Sean to flesh out issues when he is here next week.


      Action Items

 1. Put MC into recon_test.
 2. Schedule a monitoring dashboard meeting.
 3. Put together a new release that fixes SQLite issue, MC treatment of
    the CDC, and the memory leak.

Retrieved from 
"https://halldweb.jlab.org/wiki/index.php?title=GlueX_Offline_Meeting,_April_6,_2018&oldid=86331"

  * This page was last modified on 10 April 2018, at 17:54.

-- 
Mark Ito, marki at jlab.org, (757)269-5295

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20180410/3d6ca47d/attachment.html>