[Halld-cal] Minutes from yesterday's meeting
Hovanes Egiyan
hovanes.egiyan at gmail.com
Mon Nov 18 13:57:41 EST 2019
Hi Fernando, Tim.
not only the crates are set to turn off at high temperatures in the
hall, but
the crates that were online were actually turned off by EPICS during that
"open door" incident. See log entry https://logbooks.jlab.org/entry/3734190
The current high temperature limit in EPICS is 82F
as measured by PLC in front and behind the solenoid magnet, not 80F.
Note that some crates might shut off power to the bus after (or before)
EPICS does using
crate's internal interlock. Also, EPICS interlock only shuts power to
the basket, the crates
internal interlock will most likely still get engaged if temperature
keeps rising after EPICS
shuts the power off, and will take the crate's fan-tray offline. I do
not know what exactly
triggers the crates internal interlock, probably input or output
temperature to the fans. Therefore ,
I do not know the order these interlock trip, it might be
crate-dependent based on the
environment in the hall and around the crate.
Hovanes .
On 11/18/19 9:55 AM, Fernando Barbosa wrote:
>
> Hello Tim,
>
> I am not certain but I will talk to Hovanes.
>
> Best regards,
>
> Fernando
>
> *From:*Timothy Whitlatch <whitey at jlab.org>
> *Sent:* Monday, November 18, 2019 9:17 AM
> *To:* Fernando Barbosa <barbosa at jlab.org>; Alexander Somov
> <somov at jlab.org>; halld-cal at jlab.org
> *Cc:* Hovanes Egiyan <hovanes.egiyan at unh.edu>
> *Subject:* Re: [Halld-cal] Minutes from yesterday's meeting
>
> Hi Fernando,
>
> I was under the impression that the crates already are set to turn off
> above 80F
>
> Cheers,
>
> Tim Whitlatch
>
> Hall D Engineer
>
> Jefferson Lab
>
> 757-269-5087
>
> ------------------------------------------------------------------------
>
> *From:*Fernando Barbosa <barbosa at jlab.org <mailto:barbosa at jlab.org>>
> *Sent:* Monday, November 18, 2019 8:56 AM
> *To:* Timothy Whitlatch <whitey at jlab.org <mailto:whitey at jlab.org>>;
> Alexander Somov <somov at jlab.org <mailto:somov at jlab.org>>;
> halld-cal at jlab.org <mailto:halld-cal at jlab.org> <halld-cal at jlab.org
> <mailto:halld-cal at jlab.org>>
> *Cc:* Fernando Barbosa <barbosa at jlab.org <mailto:barbosa at jlab.org>>
> *Subject:* RE: [Halld-cal] Minutes from yesterday's meeting
>
> Hello Tim,
>
> Thanks for reviewing the logs. This certainly could explain the
> increased failure rate of these modules. I will follow up closely on
> the diagnostics of these 5 fADC250s that failed to see if there is
> anything that points to an overheating condition.
>
> It would also be very helpful if Alex could check the DAQ logs for the
> temperature readout of the FPGAs for this same period.
>
> One area that needs to be addressed in this regard is to implement a
> protocol that turns crates off when the temperature in the hall
> reaches some set point: the crates would stay off until the
> temperature in the hall would decrease below the set point and someone
> would then need to re-enable them (an administrative control). I will
> talk to Hovanes and Nick.
>
> Best regards,
>
> Fernando
>
> *From:*Timothy Whitlatch <whitey at jlab.org <mailto:whitey at jlab.org>>
> *Sent:* Monday, November 18, 2019 7:54 AM
> *To:* Alexander Somov <somov at jlab.org <mailto:somov at jlab.org>>;
> Fernando Barbosa <barbosa at jlab.org <mailto:barbosa at jlab.org>>;
> halld-cal at jlab.org <mailto:halld-cal at jlab.org>
> *Subject:* Re: [Halld-cal] Minutes from yesterday's meeting
>
> Hi All,
>
> In looking back at the archive data, there is only 1 time the hall
> reached above 80F this spring through fall. October 12 the temp
> reached 84F and was above 80F for about 12 hours. This is when the
> overhead door was left open after a tour.
>
> Cheers,
>
> Tim Whitlatch
>
> Hall D Engineer
>
> Jefferson Lab
>
> 757-269-5087
>
> ------------------------------------------------------------------------
>
> *From:*Halld-cal <halld-cal-bounces at jlab.org
> <mailto:halld-cal-bounces at jlab.org>> on behalf of Alexander Somov
> <somov at jlab.org <mailto:somov at jlab.org>>
> *Sent:* Friday, November 15, 2019 6:57 PM
> *To:* Fernando Barbosa <barbosa at jlab.org <mailto:barbosa at jlab.org>>;
> halld-cal at jlab.org <mailto:halld-cal at jlab.org> <halld-cal at jlab.org
> <mailto:halld-cal at jlab.org>>
> *Subject:* Re: [Halld-cal] Minutes from yesterday's meeting
>
> Hi Fernando,
>
> As we have discussed, the temperature accident happened
> not the first time. After the first accident, when we
> melted a CTP module by running a fan speed at ~2000 rpms,
> I took many trigger-related (and other) crates from the
> PID control and ran them at 4500+ rpm.
>
> After the overheating problems during last year, Hai Dong
> suggested to run the fan at about 5000 rpm for the sake of
> CTPs, and I agree with him. There should be enough margin
> to cool the crate down if the temperature in the hall
> increases.
>
>
> The CCAL crate, positioned on the same FCAL platform
> runs at about 5000 rpm for more than a year, I didn't
> have any problems with it so far.
>
>
> I belive that the PID fan threshold should be set to at
> least 4500 rpm for all crates (indeed it's a choice of
> sub-detector groups, who should be responsible for
> damaged electronics as well).
>
>
>
> Cheers,
> Sasha
>
>
> P.S. Indeed, there could be other not temperature-related
> issues, but this is the most likely one.
>
>
> Rad damage can be an issue as well, but then modules should
> fail "continuously" during the run, likely with various symptoms,
> which seems to be not the case. We got 5 modules dead after
> the run. The rad dose in the tagger hall is also significantly
> larger than in Hall D; I haven't seen such problems with fadcs
> there.
>
> On 11/15/19 3:10 PM, Fernando Barbosa wrote:
>
> Hello Elton, all,
>
> Some notes regarding the paragraph below:
>
> a)The overheating is not an issue unless the hall temperature
> increases significantly above the specs. For a full description,
> refer to GlueX-doc-2752. My recommendation has always been to
> operate the fans fixed at 4,000 RPM or in the 3,000 RPM to 5,000
> RPM range under PID control.
>
> b)During the Summer, we experienced some unexplained failures.
> Nick recently informed me that there were two instances when the
> temperature in the hall was excessive duw to HVAC failures or door
> open for long periods of time. Nick has suggested some additional
> interlock controls. It would be instructive if someone could check
> the environmental logs and others since the end of the Spring run.
>
> c)The last sentence is not correct. Spreading the modules apart
> will not provide better cooling as the thermal radiative losses
> are insignificant compared to the convective cooling efficiency.
>
> d)There is also the possibility of radiation damage on the D2
> platform due to changing experimental conditions (prototype tests
> of the COMCAL and others). This was also brought up by CAEN during
> maintenance of failed HV controllers. I have Alexandre D. to look
> into this in detail from logs of radiation detectors he manages.
>
> Best regards,
>
> Fernando
>
> -----Original Message-----
> From: Halld-cal <halld-cal-bounces at jlab.org>
> <mailto:halld-cal-bounces at jlab.org> On Behalf Of Elton Smith
> Sent: Friday, November 15, 2019 1:25 PM
> To: halld-cal at jlab.org <mailto:halld-cal at jlab.org>
> Subject: [Halld-cal] Minutes from yesterday's meeting
>
> Dear collaborators,
>
> I have posted the minutes from yesterday's meeting on the wiki at
> https://halldweb.jlab.org/wiki/index.php/Nov_14,_2019_Calorimeter#Minutes.
>
> Cheers, Elton.
>
> --
>
> Elton Smith
>
> Jefferson Lab MS 12H3
>
> 12000 Jefferson Ave STE 4
>
> Newport News, VA 23606
>
> (757)269-7625
>
> (757)269-6331 fax
>
> _______________________________________________
>
> Halld-cal mailing list
>
> Halld-cal at jlab.org <mailto:Halld-cal at jlab.org>
>
> https://mailman.jlab.org/mailman/listinfo/halld-cal
>
> _______________________________________________
>
> Halld-cal mailing list
>
> Halld-cal at jlab.org <mailto:Halld-cal at jlab.org>
>
> https://mailman.jlab.org/mailman/listinfo/halld-cal
>
>
> _______________________________________________
> Halld-cal mailing list
> Halld-cal at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-cal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-cal/attachments/20191118/e936c941/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 20174 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld-cal/attachments/20191118/e936c941/attachment-0001.png>
More information about the Halld-cal
mailing list