<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Manav,<div><br></div><div> Thanks for the plot and explanation.</div><div><br></div><div>I’m trying to understand your description of the error bars. There are configurations where</div><div>the error bars stretch from 0% to 100%. Does that mean for one of the trials there was</div><div>a trial with 0% accuracy and another trial with 100% accuracy?</div><div><br></div><div><br></div><div><div>
<meta charset="UTF-8"><div>Regards,<br>-David<br><br>-------------------------------------------------------------<br>David Lawrence Ph.D.<br>Staff Scientist - - EPSCI Group Lead<br>Thomas Jefferson National Accelerator Facility<br>Newport News, VA<br>davidl@jlab.org<br>(757) 269-5567 W<br>(757) 746-6697 C</div>
</div>
<div><br><blockquote type="cite"><div>On Feb 8, 2024, at 2:51 PM, Manav Bilakhia <manav.mitesh@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><font face="arial, sans-serif">Hi,</font><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I wanted to share the plot I was trying to show this morning.<br><br>Here's the description of the that is attached below<br><br clear="all"></font><div><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">This plot has a lot of information in it. Let's break it
down.<br>
<br>
Important note: I generated 1866 knockout plots over the summer.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif"> Full
dataset means all natural plots + knockout plots.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif"> Natural
plots are the plots that I did not generate.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif"> Each
trial group on the x-axis had five trials each</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">The y-axis represents the accuracy when the model was tested
on just the knockout plots I created over the summer via a different training
script.<br>
The x-axis:<br>
Example label1: train: 60 validation: 40
knockout in train: 0.<br>
<br>
This means that 60% of the natural dataset was used for training. 40% was set
aside for validation, and there were 0 knockout plots in the training.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">Example label 2: train: 95 validation: 5 knockout in train:
20</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">This means that 95% of the FULL DATASET was used for
training, which includes 20% of all knockout plots. <br>
5% of the full dataset was set aside for validation.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">Example label 3: Hydra original</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">This means that I did not do anything different. I ran the training
script without altering how many knockout or natural plants are used where. Hydra original has knockouts as it was given the full dataset.</font></p><div style="margin: 0in 0in 8pt; line-height: 107%;"><font face="arial, sans-serif"> </font><br class="webkit-block-placeholder"></div><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif"><b>Interpretation:</b> In this plot, we see no particular trend. This is because some plots in the natural dataset look like the knockout
plots I produced. This was also suggested in the meeting this morning.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif"> <br>Why the big lower error bars?<br>
This is basically because of how the models are trained. The training script
figures out the location of each plot from the database. After which, it's put
in a pandas data frame and shuffled before being split into training and
validation sets. Sometimes, the shuffle
is just unlucky for the model with very little knockout or knockout-like plots
that never end up in the training. I tested all these models on bad plots only. Every time a plot was not predicted as bad, it was predicted as cosmic. There were
only a handful of times when the plots were predicted led or good or no data. <br>
The model is confused between cosmic and bad with these unlucky shuffles.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">How are the error bars calculated?</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">Lower error is the mean of trials in the trial group – min of
trials in the trial group. Upper error is max of trials in the trial group – the mean of
the trials in the trial group.</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">Best,</font></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%"><font face="arial, sans-serif">Manav</font></p></div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p class="MsoNormal" style="margin-bottom:0cm;color:rgb(34,34,34)"><font face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin-bottom:0cm;color:rgb(34,34,34)"><br></p></div></div></div></div>
<span id="cid:f_lsdmu3si0"><Figure_1.png></span>_______________________________________________<br>Hydrateam mailing list<br>Hydrateam@jlab.org<br>https://mailman.jlab.org/mailman/listinfo/hydrateam<br></div></blockquote></div><br></div></body></html>