[G14_run] Systematic uncertainties on asymmetries - (Tsuneo's question to K Sigma analysis note)

Wed Nov 8 17:39:12 EST 2017

The best way to avoid confusion is to call it systematic scale uncertainty. 

Sent from Galaxy S8

On Wed, Nov 8, 2017 at 5:07 PM -0500, "Nicholas Zachariou" <nicholas at jlab.org> wrote:

Hi Andy, 
That makes perfect sense now!
Thanks, Nick
On 8 Nov 2017, at 21:03, Dr. A.M. Sandorfi <sandorfi at jlab.org> wrote:

Re: [G14_run] Systematic uncertainties on asymmetries - (Tsuneo's question to K Sigma analysis note)

Nick,

Pardon my cavalier interchange of the words “error” and “uncertainty”. But that’s not the point. There is a systematic uncertainty in the beam polarization, for example, and its variation from the nominal value is approximated by a gaussian distribution with a 1 sigma of 3.4%. So, someone else using the data set in some calculation is allowed to consider different values of that polarization within this distribution. BUT, if an overall fit to some model, a PWA, ...whatever, prefers a value for the beam polarization that is different from our quoted nominal, he/she must shift ALL of our data points together by exactly the same factor, because that’s how the measurement responds to a change to that parameter. {For example, If the assumed beam polarization is lowered by X%, then the deduced asymmetry values in kinematic bins at W=1800 and cosA= -0.3, and at W=2100 and cosA= +0.6,  go up by exactly the same X%.} That’s the point! For this reason one does not combine uncertainties in the overall scale of an entire data set in quadrature with point-to-point uncertainties.

Andy

On 11/8/17, 5:20 PM, "Nicholas Zachariou" <nicholas at jlab.org> wrote:

Hi Andy, 

Thank you for the explanation. I do agree that that would be the correct way to do so if we found a systematic error (and not uncertainty). In that case we would need to shift our points to account for the systematic offset. In most cases (and in the analysis we have done for the E observable on K+Sigma-) the systematic studies yield uncertainties (i.e. the systematic effect of specific sources can produce higher or lower in magnitude observables). The same goes for the photon polarization. The 3.4% is a systematic uncertainty and not an error. This means that the true value of the photon polarization can vary from the nominal value by 3.4% in either direction. This effects our observable E directly in magnitude, so I think its more proper to include that in the total uncertainty by adding it to the statistical in quadrature. Maybe I am missing something and would love to discuss more on this during the meeting. 

Best regards, 

Nick

On 8 Nov 2017, at 17:35, Dr. A.M. Sandorfi <sandorfi at jlab.org> wrote:

Re: [G14_run] Systematic uncertainties on asymmetries - (Tsuneo's question to K Sigma analysis note) 

Nick,

A systematic scale uncertainty represents a shift to the entire data set as a hole. For example, imagine you find out after publication that a polarization (beam or target) was to high by 5%. How do you correct that – you multiply every data point by 0.95. Essentially, that just changes the scale of the axes in a plot of the data. The point-to-point statistical fluctuations do not change. That is not the equivalent to increasing the size of the point-to-point fluctuations.

Now there can be some components of the systematic uncertainty that change with kinematic bins, and these parts could be added in quadrature with statistical errors. But those do not usually dominate the total systematic error. For this reason, when it comes time to publish, point-to-point systematics and scale uncertainties should be clearly quoted separated – as we have done in our recent pi-p PRL.

Andy

On 11/7/17, 11:02 PM, "Nicholas Zachariou" <nicholas at jlab.org <x-msg://22/nicholas@jlab.org> > wrote:

Hi Andy,

Can you elaborate why its not correct to add the systematic in quadrature with the statistical? 

I am not sure I understand why this "uses data that dont represent the experiment". Maybe we can talk more during the g14 meeting.

Nick

On Nov 7, 2017, at 22:22, "Dr. A.M.  Sandorfi" <sandorfi at jlab.org <x-msg://22/sandorfi@jlab.org> > wrote:

Hi Eugene,

Yes, your point is well taken - for those data points where the asymmetry is

very close to zero. One has two choices: we could either give an overall

fractional (%) uncertainty, while quoting the absolute value as an exception

when the asymmetry vanishes, or just quote only an absolute uncertainty for

all points. 

The trouble comes at the subsequent stage when the data is used by the

various PWA groups. All PWA groups have their fitting routines set up to

float the scale of a data set while including a chi^2 penalty that is

weighted by a fractional systematic error. If we give them a systematic

uncertainty that is absolute, they will combine it in quadrature with the

statistical error to create an inflated point by point uncertainty, and set

the fitting scale to 1. I can guarantee that this will happen and it is a

completely incorrect way to use the data that doesn't represent the

experiment. So it is better to use the first approach - quote the systematic

uncertainty as a fractional (%) error, while explicitly noting the absolute

value of the systematic uncertainty for those asymmetry points with nearly

zero value. The later qualifying statement will probably be ignored in PWA

analyses, but at least most of the data will have been included properly.

Andy

On 11/4/17, 11:37 PM, "Eugene Pasyuk" <pasyuk at jlab.org <x-msg://22/pasyuk@jlab.org> > wrote:

 Any asymmetry can be anything between -1 and +1, 0 included. For any

 observable which may be equal to 0 relative uncertainty does not make sense.

 Only absolute uncertainty must be used.

 The second term in Nick's equation is equal to 0 if ObservableValue is always

 equal to 0 regardless of sigma_sys_relative. This is incorrect but good

 illustration why one must not use relative uncertainty for asymmetries.

 -Eugene

 ----- Original Message -----

 From: "Tsuneo Kageya" <kageya at jlab.org <x-msg://22/kageya@jlab.org> >

 To: "Nicholas Zachariou" <nicholas at jlab.org <x-msg://22/nicholas@jlab.org> >

 Cc: "g14 run" <g14_run at jlab.org <x-msg://22/g14_run@jlab.org> >

 Sent: Saturday, November 4, 2017 10:57:28 PM

 Subject: Re: [G14_run] G14_run Digest, Vol 74, Issue 3 (Tuneo's  question to

 K Sigma analysis note)

 Nick,

 thank you for the response.

 I would like to know why the absolute is more appropriate.

 I will look forward the statements.

     Regards, Tsuneo Kageya.

 ----- Original Message -----

 From: "Nicholas Zachariou" <nicholas at jlab.org <x-msg://22/nicholas@jlab.org> >

 To: "Tsuneo Kageya" <kageya at jlab.org <x-msg://22/kageya@jlab.org> >

 Cc: "g14 run" <g14_run at jlab.org <x-msg://22/g14_run@jlab.org> >

 Sent: Saturday, November 4, 2017 3:23:46 PM

 Subject: Re: [G14_run] G14_run Digest, Vol 74, Issue 3 (Tuneo's  question to

 K

 Sigma analysis note)

 Hi Tsuneo,

 The absolute error is propagated directly to the total uncertainty

 (sigma_tot=sqrt(sigma_sys_absolute^2+(ObservableValue*sigma_sys_relative)^2 +

 sigma_statistical^2).

 I can elaborate more if you like on why thats the case (why absolute are more

 appropriate in my case). I will include some statements in the note to

 reflect

 this.

 Let me know if you would like to discuss this more.

 Best regards,

 Nick

 On Nov 4, 2017, 19:12, at 19:12, Tsuneo Kageya <kageya at jlab.org <x-msg://22/kageya@jlab.org> > wrote:

 Hi Nick,

 Sorry to be late to make a question.

 I have a question about the systematic error calculations.

 At page 35, on the table 4, you calculated the total absolute

 systematic error

 to be 0.10.   How this is reflected into the total relative systematic

 error ?

 On the pi-p analysis, I think we calculated the systematic errors from

 cuts in

 the similar way and they are combined to the other errors (target and

 beam polarizations).

 Is this number 0.10 means 10 % or 0.1 % ?

 I may misunderstand this issue.  Please let me know.

    Regards, Tsuneo Kageya.

 ----- Original Message -----

 From: "g14 run-request" <g14_run-request at jlab.org <x-msg://22/g14_run-request@jlab.org> >

 To: "g14 run" <g14_run at jlab.org <x-msg://22/g14_run@jlab.org> >

 Sent: Saturday, November 4, 2017 12:00:03 PM

 Subject: G14_run Digest, Vol 74, Issue 3

 Send G14_run mailing list submissions to

 g14_run at jlab.org <x-msg://22/g14_run@jlab.org> 

 To subscribe or unsubscribe via the World Wide Web, visit

 https://mailman.jlab.org/mailman/listinfo/g14_run

 or, via email, send a message with subject or body 'help' to

 g14_run-request at jlab.org <x-msg://22/g14_run-request@jlab.org> 

 You can reach the person managing the list at

 g14_run-owner at jlab.org <x-msg://22/g14_run-owner@jlab.org> 

 When replying, please edit your Subject line so it is more specific

 than "Re: Contents of G14_run digest..."

 Today's Topics:

   1. Re: Updated Analysis Note (Reinhard Schumacher)

   2. Re: Updated Analysis Note (Nicholas Zachariou)

 Message: 1

 Date: Fri, 3 Nov 2017 22:19:24 -0400

 From: Reinhard Schumacher <schumacher at cmu.edu <x-msg://22/schumacher@cmu.edu> >

 To: g14_run at jlab.org <x-msg://22/g14_run@jlab.org> 

 Subject: Re: [G14_run] Updated Analysis Note

 Message-ID: <a5780718-56ac-1dc0-44e8-6b991dbf849b at cmu.edu <x-msg://22/a5780718-56ac-1dc0-44e8-6b991dbf849b@cmu.edu> >

 Content-Type: text/plain; charset="windows-1252"; Format="flowed"

 Hi Nick,

 Indeed, noticeably improved.?? I recommend that you put the horizontal

 error bars on Figs.? 29 - 32, too.? They are just as important there

 since the model curves can vary a lot across one bin.

 Reinhard

 On 11/3/2017 7:13 PM, Nicholas Zachariou wrote:

 Dear all,

 I am attaching the updated note that incorporates and addresses all

 comments made. I have noticed that I have previously forgotten to

 include the systematic uncertainty associated with the

 photon-selection, and is now estimated and included.

 I would like to thank again Shumacher for his time and valuable

 insight, and everybody for the comments and feedback. If there is no

 other comments, I will be submitting the note early next week.

 Best regards,

 Nick

 G14_run mailing list

 G14_run at jlab.org <x-msg://22/G14_run@jlab.org> 

 https://mailman.jlab.org/mailman/listinfo/g14_run

 --

 Reinhard Schumacher         Department of Physics, 5000 Forbes Ave.

 Carnegie Mellon University, Pittsburgh, PA 15213, U.S.A.

 phone: 412-268-5177         web: www-meg.phys.cmu.edu/~schumach <http://www-meg.phys.cmu.edu/~schumach>  <http://www-meg.phys.cmu.edu/~schumach> 

 -------------- next part --------------

 An HTML attachment was scrubbed...

 URL:

 <https://mailman.jlab.org/pipermail/g14_run/attachments/20171103/651ca313/at

 tachment-0001.html>

 Message: 2

 Date: Sat, 04 Nov 2017 07:10:14 +0000

 From: Nicholas Zachariou <nicholas at jlab.org <x-msg://22/nicholas@jlab.org> >

 To: Reinhard Schumacher <schumacher at cmu.edu <x-msg://22/schumacher@cmu.edu> >

 Cc: G14 Run <g14_run at jlab.org <x-msg://22/g14_run@jlab.org> >

 Subject: Re: [G14_run] Updated Analysis Note

 Message-ID: <d21ed42f-6b13-482d-8c8b-4ea087ad31e1 at jlab.org <x-msg://22/d21ed42f-6b13-482d-8c8b-4ea087ad31e1@jlab.org> >

 Content-Type: text/plain; charset="utf-8"

 Hi Reinhard,

 The figures looked a bit busy when I did that (too many lines) and

 thats why i left the x-uncertainties out in those, but its takes me 2

 minutes to incorborate them. I think that will be more relevant when we

 decide exactly how to present our results in the publication.

 In the meantime I was wondering if its OK with the group to share our

 preliminary results with the theorists and see if we can get any

 insights from them.

 Best regards,

 Nick

 On Nov 4, 2017, 02:19, at 02:19, Reinhard Schumacher

 <schumacher at cmu.edu <x-msg://22/schumacher@cmu.edu> > wrote:

 Hi Nick,