[Halld-cal] Minutes from yesterday's meeting
Fernando Barbosa
barbosa at jlab.org
Tue Nov 19 13:19:10 EST 2019
Hello Alex,
I will make sure we also keep track of this other module. Certainly, we will need to have a better assessment of the protocol followed by the Winner crates upon interlock.
Thanks and best regards,
Fernando
From: Alexander Somov <somov at jlab.org>
Sent: Tuesday, November 19, 2019 11:17 AM
To: Fernando Barbosa <barbosa at jlab.org>; Hovanes Egiyan <hovanes.egiyan at gmail.com>; Timothy Whitlatch <whitey at jlab.org>; halld-cal at jlab.org
Subject: Re: [Halld-cal] Minutes from yesterday's meeting
Hello Fernando,
Thanks for taking care of this.
There are 5 modules with the DAC problems, not 4, see log 3741916
(one module was released later by RadCon, it's in 118).
There is another module (from fadc spares) with the suspicious
channel.
The voltage spike could be the problem. Though, the spike could
occur during the crate interlock trip (see Hovanes email), caused
by temperature, i.e., this accident can be indirectly related to
heat.
(potentially it could have happened during the general power
outage in the hall, which is out of our control).
Modules were replaced with the crate switched off unless calorimeter
guys did something before. Modules were identified dead before
replacing, the sequence is: identify the problem and replace
the module.
We should try to prevent it from happening again, replacing
modules is an expensive procedure.
Cheers,
Alex
On 11/19/19 9:27 AM, Fernando Barbosa wrote:
Hello Hovanes,
That's good news as I also had a suspicion that heat was not the issue here.
We had 5 fADC250s that were replaced. The symptoms were:
1) One had one channel that baseline (DAC setting) could not be adjusted.
2) On the other four, the baseline setting for all 16 channels in a module reported a timeout error.
Bill Gunning just tested all of them and here is what he found:
1) The one with the one "faulty" channel is fully operational. He tested it multiple times and it is still operating correctly. We will re-test again just to be sure.
2) The other 4 had their fuses blown on the 12 V supply (all supply feeds are fused on each module) and their input filter capacitors shorted, likely caused by voltage spikes. This supply powers the 16-channel DAC among other sections of the board. This fault is unusual and it could be caused by hotswap, i.e. replacing modules with the crate on. I don't think this has been the case but I will check the logs and with Alex further.
The conclusion is that these faults are not heat related but power related (the 4 units). My recommendation is still to keep the fans fixed at 4,000 RPM, which is more than adequate, and between 3,000 RPM and 5,000 RPM for those crates in the PID loop.
Thanks and best regards,
Fernando
From: Hovanes Egiyan <hovanes.egiyan at gmail.com><mailto:hovanes.egiyan at gmail.com>
Sent: Monday, November 18, 2019 1:58 PM
To: Fernando Barbosa <barbosa at jlab.org><mailto:barbosa at jlab.org>; Timothy Whitlatch <whitey at jlab.org><mailto:whitey at jlab.org>; Alexander Somov <somov at jlab.org><mailto:somov at jlab.org>; halld-cal at jlab.org<mailto:halld-cal at jlab.org>
Subject: Re: [Halld-cal] Minutes from yesterday's meeting
Hi Fernando, Tim.
not only the crates are set to turn off at high temperatures in the hall, but
the crates that were online were actually turned off by EPICS during that
"open door" incident. See log entry https://logbooks.jlab.org/entry/3734190
The current high temperature limit in EPICS is 82F
as measured by PLC in front and behind the solenoid magnet, not 80F.
Note that some crates might shut off power to the bus after (or before) EPICS does using
crate's internal interlock. Also, EPICS interlock only shuts power to the basket, the crates
internal interlock will most likely still get engaged if temperature keeps rising after EPICS
shuts the power off, and will take the crate's fan-tray offline. I do not know what exactly
triggers the crates internal interlock, probably input or output temperature to the fans. Therefore ,
I do not know the order these interlock trip, it might be crate-dependent based on the
environment in the hall and around the crate.
Hovanes .
On 11/18/19 9:55 AM, Fernando Barbosa wrote:
Hello Tim,
I am not certain but I will talk to Hovanes.
Best regards,
Fernando
From: Timothy Whitlatch <whitey at jlab.org><mailto:whitey at jlab.org>
Sent: Monday, November 18, 2019 9:17 AM
To: Fernando Barbosa <barbosa at jlab.org><mailto:barbosa at jlab.org>; Alexander Somov <somov at jlab.org><mailto:somov at jlab.org>; halld-cal at jlab.org<mailto:halld-cal at jlab.org>
Cc: Hovanes Egiyan <hovanes.egiyan at unh.edu><mailto:hovanes.egiyan at unh.edu>
Subject: Re: [Halld-cal] Minutes from yesterday's meeting
Hi Fernando,
I was under the impression that the crates already are set to turn off above 80F
Cheers,
Tim Whitlatch
Hall D Engineer
Jefferson Lab
757-269-5087
________________________________
From: Fernando Barbosa <barbosa at jlab.org<mailto:barbosa at jlab.org>>
Sent: Monday, November 18, 2019 8:56 AM
To: Timothy Whitlatch <whitey at jlab.org<mailto:whitey at jlab.org>>; Alexander Somov <somov at jlab.org<mailto:somov at jlab.org>>; halld-cal at jlab.org<mailto:halld-cal at jlab.org> <halld-cal at jlab.org<mailto:halld-cal at jlab.org>>
Cc: Fernando Barbosa <barbosa at jlab.org<mailto:barbosa at jlab.org>>
Subject: RE: [Halld-cal] Minutes from yesterday's meeting
Hello Tim,
Thanks for reviewing the logs. This certainly could explain the increased failure rate of these modules. I will follow up closely on the diagnostics of these 5 fADC250s that failed to see if there is anything that points to an overheating condition.
It would also be very helpful if Alex could check the DAQ logs for the temperature readout of the FPGAs for this same period.
One area that needs to be addressed in this regard is to implement a protocol that turns crates off when the temperature in the hall reaches some set point: the crates would stay off until the temperature in the hall would decrease below the set point and someone would then need to re-enable them (an administrative control). I will talk to Hovanes and Nick.
Best regards,
Fernando
From: Timothy Whitlatch <whitey at jlab.org<mailto:whitey at jlab.org>>
Sent: Monday, November 18, 2019 7:54 AM
To: Alexander Somov <somov at jlab.org<mailto:somov at jlab.org>>; Fernando Barbosa <barbosa at jlab.org<mailto:barbosa at jlab.org>>; halld-cal at jlab.org<mailto:halld-cal at jlab.org>
Subject: Re: [Halld-cal] Minutes from yesterday's meeting
Hi All,
In looking back at the archive data, there is only 1 time the hall reached above 80F this spring through fall. October 12 the temp reached 84F and was above 80F for about 12 hours. This is when the overhead door was left open after a tour.
Cheers,
Tim Whitlatch
Hall D Engineer
Jefferson Lab
757-269-5087
________________________________
From: Halld-cal <halld-cal-bounces at jlab.org<mailto:halld-cal-bounces at jlab.org>> on behalf of Alexander Somov <somov at jlab.org<mailto:somov at jlab.org>>
Sent: Friday, November 15, 2019 6:57 PM
To: Fernando Barbosa <barbosa at jlab.org<mailto:barbosa at jlab.org>>; halld-cal at jlab.org<mailto:halld-cal at jlab.org> <halld-cal at jlab.org<mailto:halld-cal at jlab.org>>
Subject: Re: [Halld-cal] Minutes from yesterday's meeting
Hi Fernando,
As we have discussed, the temperature accident happened
not the first time. After the first accident, when we
melted a CTP module by running a fan speed at ~2000 rpms,
I took many trigger-related (and other) crates from the
PID control and ran them at 4500+ rpm.
After the overheating problems during last year, Hai Dong
suggested to run the fan at about 5000 rpm for the sake of
CTPs, and I agree with him. There should be enough margin
to cool the crate down if the temperature in the hall
increases.
The CCAL crate, positioned on the same FCAL platform
runs at about 5000 rpm for more than a year, I didn't
have any problems with it so far.
I belive that the PID fan threshold should be set to at
least 4500 rpm for all crates (indeed it's a choice of
sub-detector groups, who should be responsible for
damaged electronics as well).
Cheers,
Sasha
P.S. Indeed, there could be other not temperature-related
issues, but this is the most likely one.
Rad damage can be an issue as well, but then modules should
fail "continuously" during the run, likely with various symptoms,
which seems to be not the case. We got 5 modules dead after
the run. The rad dose in the tagger hall is also significantly
larger than in Hall D; I haven't seen such problems with fadcs
there.
On 11/15/19 3:10 PM, Fernando Barbosa wrote:
Hello Elton, all,
Some notes regarding the paragraph below:
a) The overheating is not an issue unless the hall temperature increases significantly above the specs. For a full description, refer to GlueX-doc-2752. My recommendation has always been to operate the fans fixed at 4,000 RPM or in the 3,000 RPM to 5,000 RPM range under PID control.
b) During the Summer, we experienced some unexplained failures. Nick recently informed me that there were two instances when the temperature in the hall was excessive duw to HVAC failures or door open for long periods of time. Nick has suggested some additional interlock controls. It would be instructive if someone could check the environmental logs and others since the end of the Spring run.
c) The last sentence is not correct. Spreading the modules apart will not provide better cooling as the thermal radiative losses are insignificant compared to the convective cooling efficiency.
d) There is also the possibility of radiation damage on the D2 platform due to changing experimental conditions (prototype tests of the COMCAL and others). This was also brought up by CAEN during maintenance of failed HV controllers. I have Alexandre D. to look into this in detail from logs of radiation detectors he manages.
Best regards,
Fernando
[cid:image001.png at 01D59EDB.EC421D80]
-----Original Message-----
From: Halld-cal <halld-cal-bounces at jlab.org><mailto:halld-cal-bounces at jlab.org> On Behalf Of Elton Smith
Sent: Friday, November 15, 2019 1:25 PM
To: halld-cal at jlab.org<mailto:halld-cal at jlab.org>
Subject: [Halld-cal] Minutes from yesterday's meeting
Dear collaborators,
I have posted the minutes from yesterday's meeting on the wiki at https://halldweb.jlab.org/wiki/index.php/Nov_14,_2019_Calorimeter#Minutes.
Cheers, Elton.
--
Elton Smith
Jefferson Lab MS 12H3
12000 Jefferson Ave STE 4
Newport News, VA 23606
(757)269-7625
(757)269-6331 fax
_______________________________________________
Halld-cal mailing list
Halld-cal at jlab.org<mailto:Halld-cal at jlab.org>
https://mailman.jlab.org/mailman/listinfo/halld-cal
_______________________________________________
Halld-cal mailing list
Halld-cal at jlab.org<mailto:Halld-cal at jlab.org>
https://mailman.jlab.org/mailman/listinfo/halld-cal
_______________________________________________
Halld-cal mailing list
Halld-cal at jlab.org<mailto:Halld-cal at jlab.org>
https://mailman.jlab.org/mailman/listinfo/halld-cal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-cal/attachments/20191119/8e6990cd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 20174 bytes
Desc: image001.png
URL: <https://mailman.jlab.org/pipermail/halld-cal/attachments/20191119/8e6990cd/attachment-0001.png>
More information about the Halld-cal
mailing list