[Clas_offline] clasdb
Vardan Gyurjyan
gurjyan at jlab.org
Wed Nov 8 15:13:49 EST 2017
I presented an example in my previous e-mail.
According to the Clara log of the presented example, for a single thread there were 4 mySql open statements and only 2 close statements. Most likely there are also bugs (or improper thread enabled operations) in the CalibrationConstantsLoader and/or GeometryLoader classes (just a guess).
This is the code that presumable accesses the database and it is in the processDataEvent() method:
// Load the constants
//-------------------
int newRun = bank.getInt("run", 0);
boolean T2DCalc = false;
if(Run!=newRun) {
//if(newRun>751 && newRun<912) {
if(newRun>99) {
T2DCalc = true;
Constants.setT0(true);
Constants.setUseMiniStagger(true);
}
T2DCalc = true;
Constants.setUseMiniStagger(true);
System.out.println(" SETTING RUN-DEPENDENT CONSTANTS, T0 = "+Constants.getT0()+ " use ministagger "+Constants.getUseMiniStagger());
CalibrationConstantsLoader.Load(newRun, "default");
TableLoader.Fill();
GeometryLoader.Load(newRun, "default");
}
---------------------------------------------------
Vardan H. Gyurjyan, Ph.D.
Staff Scientist
Thomas Jefferson Accelerator Facility
Newport News, VA, 23606
E-mail: gurjyan at jlab.org
757-269-5879 (JLAB)
> On Nov 8, 2017, at 3:00 PM, Gagik Gavalian <gavalian at jlab.org> wrote:
>
> Hi,
>
> Which services are connecting to database in processEvent method ?
> That would be wrong.
>
> Gagik
>
>> On Nov 8, 2017, at 2:43 PM, Vardan Gyurjyan <gurjyan at jlab.org <mailto:gurjyan at jlab.org>> wrote:
>>
>> Hi,
>> Rebooting will not solve the problem. The problem is in the engine design.
>>
>> I think the problem is a that service configuration is done in the processEvent() method of the engine, instead of init() method. Since processEvent() is scaled by Clara over multiple threads, we are getting multiple database accesses for a single service (higher scaling, higer the number of database accesses). Also, not always database is properly closed. Below is an example of a production using a single service. Note that this is service independent and the solution would be to move service configuration to Clara configure() transaction (user engine init() method).
>>
>> Best,
>>
>> -vardan
>>
>>
>> YAML
>>
>> io-services:
>>
>> reader:
>>
>> class: org.jlab.clas.std.services.convertors.HipoToHipoReader
>>
>> name: HipoToHipoReader
>>
>> writer:
>>
>> class: org.jlab.clas.std.services.convertors.HipoToHipoWriter
>>
>> name: HipoToHipoWriter
>>
>> services:
>>
>> data-processing:
>>
>> chain:
>>
>> - class: org.jlab.service.dc.DCHBEngine
>>
>> name: DCHB
>>
>> mime-types:
>>
>> - binary/data-hipo
>>
>>
>>
>> Single thread execution.
>>
>> vem:log gurjyan$ cat 129.57.75.136_gurjyan_clara_fe_dpe.log | grep mysql
>>
>>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> vem:log gurjyan$ cat 129.57.75.136_gurjyan_clara_fe_dpe.log | grep disconnect
>>
>> [DB] ---> database disconnect : success
>>
>> [DB] ---> database disconnect : success
>>
>>
>>
>> 2 thread execution
>>
>> vem:log gurjyan$ cat 129.57.75.136_gurjyan_clara_fe_dpe.log | grep mysql
>>
>>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> [DB] ---> open connection with : mysql://clas12reader@clasdb.jlab.org/clas12 <mysql://clas12reader@clasdb.jlab.org/clas12>
>> vem:log gurjyan$ cat 129.57.75.136_gurjyan_clara_fe_dpe.log | grep disconnect
>>
>> [DB] ---> database disconnect : success
>>
>> [DB] ---> database disconnect : success
>>
>> [DB] ---> database disconnect : success
>>
>> [DB] ---> database disconnect : success
>>
>> ---------------------------------------------------
>> Vardan H. Gyurjyan, Ph.D.
>> Staff Scientist
>> Thomas Jefferson Accelerator Facility
>> Newport News, VA, 23606
>> E-mail: gurjyan at jlab.org <mailto:gurjyan at jlab.org>
>> 757-269-5879 (JLAB)
>>
>>
>>
>>> On Nov 8, 2017, at 2:12 PM, Harout Avakian <avakian at jlab.org <mailto:avakian at jlab.org>> wrote:
>>>
>>> Dear All,
>>>
>>> We were having problems with the mysql server. Marty will reboot the system.
>>>
>>> Harut
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: RE: clasdb
>>> Date: Wed, 8 Nov 2017 14:08:13 -0500 (EST)
>>> From: Marty Wise <wise at jlab.org> <mailto:wise at jlab.org>
>>> To: Harout Avakian <avakian at jlab.org> <mailto:avakian at jlab.org>
>>> CC: Nathan Baltzell <baltzell at jlab.org> <mailto:baltzell at jlab.org>
>>>
>>> OK. Proceeding with the reboot now. Will let you know when it's back.
>>>
>>> Marty Wise (wise at jlab.org <mailto:wise at jlab.org> <mailto:wise at jlab.org> <mailto:wise at jlab.org> )
>>> JLab IT/CNI
>>>
>>> >>> On Nov 8, 2017, at 14:00, Marty Wise <wise at jlab.org> <mailto:wise at jlab.org> wrote:
>>> >>>
>>> >>> Nathan and Harout,
>>> >>>
>>> >>> I was working on an email to you with some details and a suggestion
>>> >>> that we should reboot the system as soon as possible. It had been
>>> >>> seeing arodun 1000 connections. When I check now, though, it has only
>>> >>> 9 active connections. Did you do something to kill jobs/connections?
>>> >>>
>>> >>> Does this lull give us a chance at a reboot?
>>> >>>
>>> >>> Marty Wise (wise at jlab.org <mailto:wise at jlab.org> <mailto:wise at jlab.org> <mailto:wise at jlab.org> ) JLab IT/CNI
>>> >>>
>>> >>>
>>>
>>> _______________________________________________
>>> Clas_offline mailing list
>>> Clas_offline at jlab.org <mailto:Clas_offline at jlab.org>
>>> https://mailman.jlab.org/mailman/listinfo/clas_offline <https://mailman.jlab.org/mailman/listinfo/clas_offline>
>> _______________________________________________
>> Clas_offline mailing list
>> Clas_offline at jlab.org <mailto:Clas_offline at jlab.org>
>> https://mailman.jlab.org/mailman/listinfo/clas_offline
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas_offline/attachments/20171108/3980501c/attachment-0001.html>
More information about the Clas_offline
mailing list