[Dsg-hallb_magnets] Torus Fast-DAQ time gap debug

[Nicholas Sandoval]

Hi Tyler,

My previous response on this issue is below in quotes, the biggest issue was the inability to find the exact combination of VT samples sent to the PLC due to each individual sample having a different time stamp, not the gaps.  Now, if we did have the ability to stack each sample to form a comparator and we still did not find one that went over the threshold then we may find that it happened to align with one of the time gaps.  

First we need the ability to recreate exactly what the PLC sees.  If you could determine the exact order the 24 taps + current measurement are sent to EPICS and if that order stays the same we could then use the first time stamp of each set as the real time stamp for each 24 sample set.  This would then allow us to align the data so that we could then add/subtract the correct sums of taps to recreate the comparator values.  


 "Every 1/2000 set of 24 VT samples should have a single time stamp that allows tordaqgui to align them.  This is not the case when reviewing the raw data in Excel for example.

The investigation should determine if in fact the cRIO is sampling each channel at different times or if this is just a time stamp that gets added after FPGA or some phenomena in the buffering.

This seems to be the reason why Probir and I were unable to find the exact combination of voltage taps that stacked up resulting in the 3000A fast dump tripping the PLC comparator even though both streams of data should be the same."

Best Regards,

Nick
----- Original Message -----
From: "Tyler Lemon" <tlemon at jlab.org>
To: "dsg-hallb magnets" <dsg-hallb_magnets at jlab.org>, "Nathan Baltzell" <baltzell at jlab.org>, "Dave Kashy" <kashy at jlab.org>
Sent: Tuesday, January 31, 2017 8:51:00 AM
Subject: Re: [Dsg-hallb_magnets] Torus Fast-DAQ time gap debug

Hello Nathan, 
Where would be the EPICS recorded data from the Fall 2016 full current commissioning? I'm curious to see if the frequency of the time gaps was the same then as it is now since Nick mentioned that they could not find what caused the fast dump from 3000A because of time gaps. 

Ruben, 
Changes have been made to the Fast-DAQ LabVIEW to hopefully lower cRIO CPU usage. The controlled ramp down interlock for cRIO FastDAQ Comm will trip when deploying these changes. Please let me know if there are plans to energize the Torus so a controlled ramp down can be avoided. 

Best regards, 
Tyler 



From: "Nathan Baltzell" <baltzell at jlab.org> 
To: "Tyler Lemon" <tlemon at jlab.org> 
Cc: "dsg-hallb magnets" <dsg-hallb_magnets at jlab.org>, "Dave Kashy" <kashy at jlab.org> 
Sent: Monday, January 30, 2017 7:14:37 PM 
Subject: Re: [Dsg-hallb_magnets] Torus Fast-DAQ time gap debug 

Hi Tyler, 

These fixed 200 ms gaps are likely from missed PV updates between the cRio IOC and the softIOC (update rate of 5 Hz --> 200 ms) and not due to incorrect time stamps. 

We first saw this issue when the PV update rate was increase from 1 Hz to 5 Hz, and, if I remember correctly, Christiana found it to be related to heavy cpu load on the cRio and found some way to resolve it. 

-Nathan 


On Jan 30, 2017, at 4:13 PM, Tyler Lemon <tlemon at jlab.org> wrote: 

> Hello Nathan, Nick, and Wesley, 
> 
> This afternoon, I tested a few different configurations of the timing loop on the Fast-DAQ cRIO that I thought would fix the 200ms time gap we are seeing. The 200ms time gap still appeared for all tests. 
> 
> Also, I made a subVI that uses the local variables that make up the EPICS IOC to display the Fast-DAQ data arrays. I cannot see any time gaps in the data using LabVIEW. 
> 
> This leads me to think that there is something happening on the EPICS end (whether EPICS server on cRIO or elsewhere) where incorrect time stamps are assigned. I can see in the LabVIEW that there is nothing that assigns a time stamp to data passed to EPICS. Maybe adding any sort of time stamp or counter would allow the data to be stitched together correctly. The new time stamp could be added as an additional PV sent to EPICS. 
> 
> Best regards, 
> Tyler 
> 
> From: "Ruben Fair" <rfair at jlab.org> 
> To: "Tyler Lemon" <tlemon at jlab.org> 
> Cc: "dsg-hallb magnets" <dsg-hallb_magnets at jlab.org>, "Dave Kashy" <kashy at jlab.org> 
> Sent: Monday, January 30, 2017 1:41:07 PM 
> Subject: Re: [Dsg-hallb_magnets] Torus Fast-DAQ time gap debug 
> 
> Hi Tyler, 
> 
> There will be no energization of the Torus today. 
> 
> Regards 
> 
> Ruben 
> 
> From: "Tyler Lemon" <tlemon at jlab.org> 
> To: "dsg-hallb magnets" <dsg-hallb_magnets at jlab.org> 
> Sent: Monday, January 30, 2017 1:38:52 PM 
> Subject: [Dsg-hallb_magnets] Torus Fast-DAQ time gap debug 
> 
> Hello, 
> 
> Since the Torus is not presently powered, I am going to try to implement a possible solution to the 200ms EPICS time gaps. I will record data with the EPICS data logger before and after to see if the change stops the 200ms time gaps. 
> 
> The change will momentarily stop the Fast-DAQ cRIO and trip the "cRIO Fast-DAQ Comm" interlock. The interlock will be able to be reset after the program is deployed. 
> 
> Please let me know if any upcoming tests are planned that will require the Torus to be energized since Fast-DAQ cRIO communication loss will cause a controlled ramp downs. 
> 
> Best regards, 
> Tyler 
> 
> _______________________________________________ 
> Dsg-hallb_magnets mailing list 
> Dsg-hallb_magnets at jlab.org 
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_magnets 
> 


_______________________________________________
Dsg-hallb_magnets mailing list
Dsg-hallb_magnets at jlab.org
https://mailman.jlab.org/mailman/listinfo/dsg-hallb_magnets