[Halld_src] [EXTERNAL] SRC-CT ALP Paper Reviewer Comments

[Jackson Reeves Pybus]

Hello all,

See forwarded the comments sent in this weekend by the reviewers of the ALP manuscript submitted in August. The reviewers are requesting some changes and clarifications in the paper, but generally seem positively inclined toward eventual publication in PLB following revision. We will work on addressing these comments and putting together a revised and clearer manuscript in the coming weeks, which we will circulate to the author list.

Best regards,
Jackson

> Reviewer #1: 
> This paper by the GlueX collaboration presents an experimental search for axion-like particles 
> (ALPs) of masses m_a=200--450 MeV, produced through the Primakoff process in photon-nucleus 
> collisions, gamma+A -> ALP(gammagamma)+A, at the JLab facility. The search is based on normalizing
> the ALP search to the similar eta meson production from the Primakoff process and on the use of 
> data from different nuclear targets, which allows to reduce backgrounds and systematic uncertainties. 
> The measurement is interesting, the topic of ALP searches is timely, and the study is well 
> motivated for future searches with larger data samples. However, the manuscript is confusing 
> in various sections, and needs several clarifications before it can be accepted for publication. 
> My comments follow below ordered by paper section:
> 
> * Generic:
> 
> The use of symbols is not always clear/consistent. A couple of examples are given 
> (but more of them are indicated below):
> - The current text mixes the use of symbols "X" and "a" to refer to ALPs. It would be
> appropriate to clearly define a generic pseudoscalar "X = eta,a" diphoton system in the
> beginning, and then consistently use "a" or "X" as needed. In this sense, for 
> illustration purposes of the key idea of the measurement, the Primakoff process of Fig. 1 
> would probably be clearer if it used "X=a,eta" in the propagator.
> - The authors use "Lambda" in various contexts and in a non-consistent/confusing way. 
> Around Eq. (4), it appears (often undefined) as "Lambda", "Lambda_X", "Lambda_eta" 
> (but not "Lambda_a", although they use mu_a just before), "Lambda_95",... 
> I'm sure this can be better explained and simplified.
> - Expressions such as "the eta" (and similar) are jargonic. One should write "the eta meson".
> 
> * Abstract:
> 
> - The collision process considered, gamma A -> a A, should be explicitly mentioned in
> the first sentence, as well as its average c.m. energy, and even a mention of the JLab 
> facility would be appropriate. The authors (wrongly) take for granted that all readers 
> know that the GlueX experiment uses GeV-photon beams from JLab.
> - Paper consistence: "(ALP)" should be "(ALPs)"
> - "200 < m_a < 450 GeV via the decay X -> gamma gamma" mixes "X" and "a" symbols (see above).
> - "limits on the coupling on the scale of O(1 TeV)" --> The coupling is 1/Lambda, so it 
>   should be O(1 TeV^{-1}), and on the scale" (which "scale"?) should be "of the order".
> - "to estimate and subtract these backgrounds" (repetitive) should read "to estimate and subtract it".
> 
> * INTRODUCTION:
> 
> - "Hierarchy" should not be capitalized
> - "Such ALPs could predominantly couple to photons," should be "Such ALPs predominantly decay 
>   into two photons," as the key point here is that, kinematically, MeV--GeV-mass ALPs can only 
>   (or mostly) decay to 2 gamma.
> - "probe beyond-SM physics using SM probes" (repetitive) -> "beyond-SM physics using SM probes"
> - "known kinematic functionS"
> 
> * EXPERIMENT:
> 
> - The reader has no clue what typical photon beam energies are used for this experiment.
> The values of E_gamma (a variable used further down) should be given. Also, in particular,
> because this value determines the amount of Lorentz boost, and associated displaced decay, 
> that a potential ALP would get (see below).
> - What is the magnetic field strength of the SC magnet?
> - Given the use of ToF variables in the analysis (and potential issues with the ALP lifetime
> in future measurements, see below), and the discussion of off-vertex backgrounds, it would 
> be relevant to quote the distance between the used detectors (TOF, FCAL) and the target. 
> In particular, because the schema of Fig. 5 does not seem to be to-scale, dimensions-wise, 
> and may be misleading.
> - The photon energy and angular resolutions of the calorimeters is an important piece of 
> information for the measurement and should be given for the range of energies of interest 
> for this analysis so that the reader understands the 5-MeV binning chosen for the diphoton 
> invariant mass distributions. 
> 
> * EVENT SELECTION:
> 
> - The assumption "4-momentum of the photons p_gamma,i was determined by assuming a reaction 
> vertex in the center of the target" is valid for the range of (m_a,Lambda) values considered
> in this paper, for which the decay length L \propto (boost)*Lambda^2/m_a^4 is small, but
> not if Lambda is much larger (and/or for lower m_a) in which case the ALP is long-lived 
> and decays away from the production vertex. (See Section RESULTS below).
> - "the mass of the 12C nucleus means that recoil nuclei cannot be detected" is not proper 
> English.
> - "The total 4-momentum ... is further inferred by adding the momentum of the 2-photons"
>   should say "adding the 4-momenta of the two photons..."
> - Unclear "angle of the "diphoton" system": angle with respect to what? (also unclear why 
> "diphoton" is written in quotation marks).
> - "Monte-Carlo simulation of signals": The authors should explain which MC simulation is
> used to generate the signal.
> - "Events with additional showers in either the FCAL or BCAL were rejected in order to 
> suppress non-Primakoff events with additional particles.": I would say "in order to 
> suppress non-exclusive events with more than two photons produced.", because technically
> the process gamma-gamma -> eta -> 3pi is also a Primakoff process, except with a 
> multihadron decay, that would be also rejected by the cut.
> - There is no mention to signal efficiency losses. What are the leading sources of signal 
> reconstruction inefficiencies for this analysis? (e.g. what is the impact, if any, of 
> photon conversion(s) in the material in front of the FCAL?). Even if those are cancelled 
> out in the ratio ALP/eta yields, it's important to to discuss them so that the reader has
> an idea of how good the experimental setup is for this measurement. Also, possible electron 
> backgrounds in the calorimeters (which would generate a signal identical to photons) are 
> not mentioned either. I assume that those are removed by the magnet and charged-particle 
> cuts, but it would be worth to mention it.
> - In the discussion of Fig. 2, there is no mention of the origin of the non-resonant background
> between the diphoton meson peaks. This is the key continuum background on top of which one 
> searches for an ALP bump, and the reader needs an explanation of it at this point (rather
> than waiting for later sections of the paper).
> 
> * STATISTICAL ANALYSIS:
> 
> - "2-photon mass spectrum in the diphoton invariant mass range" is redundant
> - "the (450 MeV) upper bound is proximate to the eta peak": The eta signal starts at 500 MeV,
> so one wonders why the authors leave a 50-MeV gap off the diphoton mass search range.
> - "the measured 2-photon mass spectrum was considered in a window of width Delta m = 20sigma_m":
> Why 20 times the resolution, and not 10 times, or any other value? In this mass range, the 
> physical ALP width is much narrower than the experimental sigma_m~6--18 MeV resolution.
> - The relative 3--4% diphoton mass resolution quoted implies absolute mass resolutions of 
> Delta-m = 6--18 MeV in the considered 200-450 MeV range? The search is said to be done
> over a mass window of 450-180=220 MeV in 400 bins of width sigma_m/20=0.55 MeV? Is such
> a narrow width (20 times better than the resolution) reasonable? Also, given that the 
> absolute resolution spans a factor of ~3 (6 to 18 MeV), shouldn't the binning be 
> mass-dependent? (Maybe I'm missing something, but the text is not very clear in general).
> 
> * NORMALIZATION:
> 
> - "Primakoff production cross section for X = pi0, eta, a": The "pi0" can be dropped as
> it's outside the analysis cuts.
> - The authors define sigma_X = 1/Lambda_X^2*sigma_0(m_X). The use of the cross-section 
> symbol "sigma_0" for the term on the r.h.s. of the expression is not very clear, 
> because the  product sigma*Lambda^2 has no (natural) units, right?
> - "eta -> gamma gamma also includes contributions from incoherent nuclear production, 
> coherent nuclear production, and interference between coherent and Primakoff production"
> The reader needs more information about all such production processes. The terms "coherent"
> (target photon emitted from nucleus?) and "incoherent" (target photon emitted from 
> nucleon?) are jargon that depends on the topic/physics studied. Are those only 
> exclusive (diffractive) photoproduction processes? Or also inclusive events that pass 
> the cuts? Other processes?
> - "we also see substantial contributions of _other_ events at larger angles"?
> - Is the multicomponent fit of the data shown in Fig. 4 a "template" fit with individual 
> contributions curves constrained from models, and only their normalization floating?
> There is a lack of details in the text on this fitting procedure.
> 
> * BACKGROUNDS:
> 
> - "It is important to explore the primary source of background for this measurement, which 
> is the photoproduction of eta -> gamma gamma and omega -> pi0+gamma outside of the target"
> Are those the contributions that populate the nonresonant diphoton counts in Figs 2 and 3?
> Nothing has been said so far, so the reader finds this statement here somehow out of the blue.
> Do the authors have simulations of both processes that confirm this statement (and the 
> full discussion in the first part of this section)?
> - The caption of Fig. 5 should indicate "(not to scale)" as the dimensions are clearly not
> those of the experimental setup.
> - Typo "While with it is"
> - Unclear "complex background structures are no longer present". The reader has no clue at 
> this point which "structures" they refer to. There has been no apparent "complex background"
> so far discussed in the diphoton mass distributions.
> - Eq. (5): Is "L" luminosity or integrated luminosity? It was "\mathcal{L}" in Eq. (2).
> - Fig. 6: The invariant 2-photon mass spectrum for carbon data after subtraction of helium data
> is mostly negative in the search region 200--450 MeV. Why this oversubtraction? How do the 
> authors deal with negative numbers of counts when searching for an ALP signal excess?
> 
> * RESULTS:
> 
> - "limits set on the coupling on the scale of O(1 TeV)" --> "limits set on the coupling 
> at the level of O(1 TeV^{-1})".
> - The authors propose to remove the FDC to reduce backgrounds: Wouldn't this have the drawback 
> of a potential increase in the rejection of charged-particle (e+/e-) backgrounds?
> - Since the authors plan to set future ALP limits for couplings Lambda~10^-4 GeV and masses 
> m_a~100 MeV, where the ALPs start to have long lifetimes (ctau \propto Lambda^2m_a^-3) and are 
> forward boosted, the possible impact of displaced vertices should be mentioned.
> - Fig. 8: Why is there a gap in the m_a region between the Empty-target and Pb(1 fb-1) limits?
> Does this correspond to the pi0 region? If so, can the Primakoff production of pi0 be fully
> removed, so that an ALP can be searched for in this mass region too?
> - Fig. 8: The authors show Belle-II expected limits for ALPs based on the 2017 work of [49]. 
> However, the Belle-II collaboration has now a direct search [https://arxiv.org/abs/2007.13071]
> with 0.45 fb-1 of collected data, which reaches Lambda~10^3 GeV. Scaling these recent limits 
> by sqrt(50e3/0.45)~300 to account for the L_int = 50 ab-1 expected eventually at Belle-II, 
> would reach limits at the level of Lambda~3 10^5 GeV, i.e. ~100 times stronger than those 
> shown in Fig. 8.
> 
> * Biblio:
> 
> - The JHEP refs. miss the publication year.
> - Ref. [41] has a wrong Collaboration name.
> 
> 
> 
> 
> 
> Reviewer #2: Dear Authors, 
> 
> Thank you for this interesting contribution showing a proof-of-principle result that can be improved in future iterations to constrain ALP parameter space. I would like to see this analysis published. That said, I have some points that should be addressed. 
> 
> 1)  Main concern:  The originally planned physics analysis was found to have some large fluctuations with respect to the background model, as they say, "perhaps due to more complex resolved background structure."  These fluctuations lead to some excesses and deficits larger than 3 Sigma.  The authors believe they understand the problem but that solving it is beyond the scope of this paper. Therefore, the authors present a second analysis, with a different background estimate, that shows almost no fluctuations, but a slightly worse sensitivity.  Both analyses are presented in Fig. 7, and both curves are presented in Fig. 8, the main result of the paper.  It is pointed out that the observed limits are excluded by other experiments, and it is also pointed out that this measurement is a proof-of-principle, and there are several techniques that can be used to improve the measurement in the future to become world leading.  
> 
> The problem is that the original approach to evaluating the limits (shown in black in Fig. 8) leads to unexpectedly good limits, and unexpectedly large excesses in the same plot.   It also leads to confusion to the reader, or to whoever is presenting the results: which result is the correct one ? 
> 
> Since the original analysis was chosen a priori, and has a better sensitivity, they present this one. But since the results were unexpected, they also present another analysis with less sensitivity. 
> 
> I would argue that, while it is good to present Figure 7, in order to be completely transparent about what happened in the analysis a posteriori, the authors should choose the second analysis (using the background-subtracted approach) as their main result.  The second analysis does not have huge excesses and deficits with respect to the background model that are known to be incorrect.  It is also the more conservative choice.  Therefore, I suggest that the original analysis limits, known to be fluctuating excessively, *not* be shown in Figure 8.  This more conservative choice should not be a serious problem, considering that the authors are busy upgrading their experiment, and are planning to put out improved results in the future.  Presumably they will overcome this background modeling problem in the next analysis.
> 
> 
> 2)  In the normalization section IV and V, the text is written as if the goal of this search was to do an ALP exclusion, rather than an ALP discovery. 
> There are several parts of the analysis that sound like they are designed just for excluding: 
> 	"was used to calculate the limits on the signal yield"
> 	"we derive the relationship between the ALP exclusion and measurement of Primakoff eta" 
> 	"we may relate the excluded ALP-photon coupling ..."
> 	...
> 
> It should be pointed out that until you look at the unblinded data and check to see that there is no excess, you should hold off on setting limits.  Your goal is presumably to discover something.  (We don't do these experiments with the goal of setting limits :)  Here you need to reconsider how you present your results. 
> 
> And again, at the end of the section, it says: 
> "The total normalization uncertainty on the excluded ALP cross sec- tion is found to be 17%."
> 	-> We haven't even unblinded the data and we are already talking about uncertainties on the excluded cross section.  
> 
> In the results section, here, the section starts with "The upper limits".  Again, we should be evaluating the data for an excess, before discussing setting limits. 
> 
> And in the projected results, you talk about "background-only projections for the limits" rather than possible signal sensitivity. You could instead talk about improvements in the sensitivity from the current value of XXX to an m_a down to some 1/Lambda value.  Choose a benchmark such as 0.325, somewhere in the middle of the mass range.  
> 
> Finally, in the conclusions, the statement is : "which could provide world-leading limits over a range of possible ALP masses."  
> as if the ultimate goal is just to set limits.  Aim higher !
> 
> Please go through the paper, and rewrite it so that it does not sound like you are designing an analysis with the goal of setting limits. 
> 
> 
> 3) I understand that the electron beam is continuous, not bunched.  
> Currently, it is not so clear what is meant by RF time in Table I and first used in the text at "electron-beam RF time for the event".  Can you add an explanation in the text on what is meant by "electron-beam RF time" and "t_RF" ?  Where is this timing measurement made, and what is the resolution on this value.  What is the time resolution on the shower time in FCAL and BCAL ? (Most important is the time resolution on the difference).  It would also be helpful to add a sentence somewhere describing some relevant beam parameters, such as the rate of electrons or photons, pileup and dead time, etc. 
> 
> 
> 4) Figure 2:   Do you have MC simulation shapes to compare to these curves to see if these contributions match what you expect ?  It is implied that these simulations exist, in order to fit the diphoton mass region, but the plots are not included. 
> 
> 
> 5) Discussion on blinding:  It is mentioned that "a blinded analysis of the complete data set. Blinding was achieved by analyzing 10% of the complete data set."
> 	-> These two sentences appear to be in conflict.  Was it a blinded analysis of the complete dataset or only 10% of the data ?
> 
> 6) The omega background is stated to be caused by three photon events where one of the photons is undetected.  It is not so clear why the peak is then so close to 782 MeV, given that a photon is missing.  
> 
> 6a) There is a statement that "in a sizable fraction of these events, at least one photon is undetected."  Is this because of an ineffiency of the detector or because of detector acceptance ?
> 
> 6b) Could it be the case that, in fact, there are 3 measured photons: it is just that two of them are overlapping, and are reconstructed as one photon?  This would then give you a peak closer to the expected one.   Since you require a small angle of deflection, I wonder if you are more susceptible to photon merging in 3-photon events, and whether this will have an effect on the analysis, especially in terms of background expectation and signal acceptance.  Perhaps you can check some variables like transverse shower size to see?
> 
> 6c) Do you have an estimate of the fraction of ALP signal that is reconstructed as 2 photons as compared to a different number of photons ?  Does your simulation consider these possibilities, or is it just a parametrization ?
> 
> 7) It looks a bit like there could be leakage of the omega decay into the eta mass window, which would effect the denominator of your ratio. I say this because of the slight asymmetry in the omega-mass distribution, which has a different slope on the left side than the right side.  Do you have some estimate of this effect that is backed up by MC simulation ?  Is it a systematic uncertainty to consider leakage from Omega into the eta mass window ?
> 
> 8) The signal is stated to be modeled by a Gaussian.  Then in the text it says that for a given ALP signal mass hypothesis, the window is 20 Sigma around the mass ("in a window of width delta m = 20 sigma m") .  Isn't this much too large of a window ?  With a 3-sigma window, you would get 99.7% of the signal, so why consider 20 Sigma ? This also doesn't seem to fit with what you say later that the "search region was filled into 400 bins, giving a bin width of sigma m/20".  I think probably this is not what you are doing and the text needs to be cleared up.   (Or perhaps this is the problem that later arises as sharp features in the Figure 7 limits). 
> 
> 9) Figure 4 and explanation in the text: when Figure 4 is introduced in the text, it is not explained where you get the contributions from. Only in the next paragraph do you mention that the angular distributions are fit, modeled with the calculations from Ref. 35.  You should reorder this a bit.  Perhaps you can introduce Figure 4 in the second paragraph when you have explain what the curves are. 
> 
> 10) Also, in Figure 4, it seems that these curves for Primakoff, coherent, interference, and incoherent are just "generator-level" distributions, not including detector response, and efficiencies and acceptances. Or, at least, none of these corrections are mentioned to the theoretical calculations of [35].  Are there any acceptance or efficiency corrections applied ?
> 
> 11) In Figure 4, is the blue line the sum of the scaled contributions of the red, orange, purple, and green ?  Or is it some other type of fit ?  The blue line has some jagged corners that don't appear to fit with the jagged lines of the colored contributions. 
> 
> 12a) Figure 5:  I appreciate this plot, which helps to make sense of the experiment.  It would be helpful if the components of the detector were labeled. The reader is not sure what the various blocks are, and which components are active detector layers:  FVAL, TOF, BCAL, SC, CDC, FDC, ...  It would be nice if the active detector layers had different colors than passive components. 
> 
> 12b) There is a statement in the text that the interaction vertex cannot be isolated due to lack of charged tracks, but if so, then can't there be a charged track requirement in the selection ?  It looks like the true vertex is in one of the FDCs, so not sure why it can't be identified here. 
> 
> 13) For the non-target related background estimate using Helium, are we sure that the flat part of the spectrum has the same slope as that of C.  Do you have a plot of the comparison of the Helium and Carbon data shapes to see that they are similar enough to subtract ?
> 
> 15) The discussion of getting rid of the FDC in favor of a bag of helium is a bit confusing, considering the FDC are an active drift chamber, and helium gas is passive.  Are the FDC not used for any measurement that is necessary for identifying signals ?  
> 
> 
> ----------  Grammar, wording ----------
> 
> "physics events in the detector are recorded if sufficient ..." 
> 	->  It is not clear what a "physics" event is - presumably every event is produced by some physics process.  Perhaps you mean: 
> 	"Events are recorded for further analysis if sufficient ..."
> 
> "up to known kinematic function" 
> 	->
> "up to a known kinematic function"  ???
> 
> "for photo-produced ALP" -> "for a photo-produced ALP" 
> 
> "that recoil nuclei" -> "that the recoiling nuclei" 
> 
> "allowing us to infer the angle of the photon momentum"
> 	->
> "allowing the angle of the photon momentum to be inferred."
> 
> "2-photons" -> "two photons" 
> 
> "and the physics cuts were tuned" ->
> "and the selection criteria for signal events was tuned" 
> 
> 
> "Several physics cuts" -> "Several kinematical requirements" 
> 	"cut" is an ambiguous word.
> 
> ""elasticity" cut" -> "elasticity requirement"
> Perhaps : "An elasticity requirement that the total energy ... was applied in order to ..."
> 
> "An additional cut was" 
> 	-> "An additional requirement was" 
> 	or 
> 	"An additional selection was" 
> 
> "These latter two cuts" -> "These latter two requirements" 
> 
> "to cut on the missing mass" -> "to select events based on the missing mass"
> 
> "vetoes and cuts" -> "vetoes and requirements" 
> 
> "The mass of the diphoton" -> "The mass of the two photons"
> 
> "means that photons originating from low-mass diphotons"
> 	->
> "means that low mass photon pairs"
> 
> "do not impact sufficiently far from the beamline to fall within the FCAL."
> ->
> "do not impact sufficiently far from the beamline to reach the FCAL acceptance."
> 
> 
> "a sharp loss of signal below an invariant mass of ..." 
> 	-> do you mean a loss of the ALP signal, or a loss of diphoton events in general ? I think you mean to say: 
> 	"a sharp loss of diphoton events below an invariant mass of ..."
> 
> "The background 2-photon combinations was modelled by a polynomial of 4th order"
> 	-> 
> "The 2-photon combinations of the background process was modelled by a polynomial of 4th order."
> 
> "ALP-photon 1/Lambda coupling" 
> ->
> "1/Lambda coupling expected for an ALP-photon interaction" 
> 
> "eta, a. epsilon =" -> "eta, and a, epsilon =" 
> 
> "which depends on mass"
> ->
> "which depends on the mass"
> r 
> "of X = a, eta, we derive" -> "of X = a and eta, we derive"
> 
> "ALP exclusion and measurement of Primakoff eta" 
> 	-> Presumably you are not looking to exclude ALPs, but actually to discover them.   I don't think you want to say that you
> 
> 
> "measurement of Primakoff eta" 
> ->
> "measurement of the Primakoff eta" 
> 
> "that normalization to the eta meson yield" 
> 	->
> 
> "that the normalization to the eta meson yield" 
> 
> "taking ratio between" -> "taking the ratio between"
> 
> "While with it is possible to address" 
> ->
> "While it is possible to address" 
> 
> "that mass spectrum is " -> 'that the mass spectrum is"
> 
> "spectrum to ALP signal" -> "spectrum to the ALP signal"
> 
> Fig. 6:  "The invariant 2-photon mass spectrum"
> 	->
> 	"The 2-photon invariant mass spectrum" 
> 
> Eq 5:  add a comma after "0.02"
> 
> "for nucleus A." -> "for nucleus A that is either He or C."
> 
> "to avoid dominating the statistical uncertainties" 
> 	-> 
> "to avoid statistical uncertainties being dominant" 
> 
> "running at higher photon flux"
> ->  (not sure what you mean here ?)
> 
> "range would naturally be" -> "range could naturally be" 
> 
> "mases"  (check spelling "masses")
> 
> "considerably lower than photon detection possible in this analysis."
> 	->  unclear what you mean, perhaps: 
> "considerably lower than the detection angle of this analysis."
> 
> "Using the background-only projections ... helium bag."
> ->  This sentence is a bit too long.  Please break it down into 2 or 3 sentences. Perhaps : 
> 	"projected limits also shown in Figure 8. including" 
> 	-> 
> 	"projected limits.  These are also shown in Figure 8, including"
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1872 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld_src/attachments/20231030/91738693/attachment-0001.p7s>