Model-RAOB Time Series Statistics Change Details
- 27-Feb-2019 (beta)
- Corrected the autocorrelation calculations used to calculate
standard error. We now
- Limit the autocorrelation to 0.95
- Insert the appropriate number of gaps (one for each
pressure level) when there is a gap in the time
series. (This may possibly slow down processing time for
long time windows.)
- Correctly set the lag for time when there are multiple
pressure levels.
Routine changed: error_library.pl (linked from
../../surface/beta/error_library.new.pl
- 19-Oct-2018 (beta)
- Changed a buffer from 200 bytes to 2000 bytes to account for very
long experiment names. Routines changed:
DisplayFrame.
- 8-March-2017 (production, on new server)
- All beta changes moved to production.
- 7-March-2017 (beta)
- (On new web server:) The java applet itself now gets the list of
models from getModels.cgi on the server, and gets default start
and end dates. This to facilitate a java web start version.
Routines changed: StatsDisplay,
getModels.cgi (new), UA.RAOBS.TimeSeries.jnlp.
- The close_icon.gif image is now stored in the jar file. Routines changed: CurveParameters, makejar.
- 4-Jan-2017 (production)
- All beta changes moved to production.
- 29-Dec-2017 (beta)
- Zooming fixed for the "Dieoff" option. Also, for the "Dieoff"
option, we now plot all points, even those with "too few" entries
in the average. Routines changed: DisplayFrame,
getStats.cgi.
- 7-Dec-2016 (beta)
- "Dieoff" added as a "projection" option. Routines
changed: DisplayFrame, getStats.cgi.
- 28-Oct-2016 (production)
- All beta changes moved to production.
- Query times added to the query log. Routines
changed: getStats.cgi
- 21-July-2006 (beta)
- The ability to not show outliers (data more than
3 standard deviations from the mean of each curve) was
added. Also, the format for the input form was changed a little
bit. Routines changed: getStats.cgi,
CurvesControl, CurveParameters.
- 21-July-2016 (production)
- All beta changes moved to production.
- 26-May-2016 (beta)
- Included some extra class files in the jar file so we don't get
security errors when the user chooses to change the label of a
curve. Routines changed: makejar.
- 30-Sept-2015 (beta)
- Gap-filling now made selectable.
- Problem with labels in the legend fixed.
- Ability to edit labels in the legend added.
- Routines changed: getStats.cgi, CurvesControl,
DisplayFrame.
- 27-Sept-2015 (beta)
- WARNING! turned off the test that plots a gap in
the averaging period isn't filled with enough data. Routine changed: getStats.cgi
- 2-Sept-2015 (beta)
- Major change -- adapted the DisplayFrame code from the Model-RAOB
profile plots that used Patrick's java code that avoids calls to
'JPlotLayout'. This gives us much more control over layers and
grids, and potentially allows us to add error bars such as we have
in the Model-RAPB profile plots. Also, changed the color of the
Y-Axis grid to very light gray (new Color(220,220,220)). Routine changed: DisplayFrame, CurveParameters,
getStats.cgi.
- 24-Aug-2015 (beta and production)
- Changed color of the Y-Axis grid to lightGray. Routine changed: DisplayFrame.
- 2-June-2015 (beta)
- Changed that the 'too few points' or 'too few valid times' does
not apply is the requested stat is 'N'. Routine changed: getStats.cgi.
- 18-Mar-2015 (production)
- All beta changes moved to production
- 23-Jan-2015 (beta)
- Fixed a bug that caused the scaling to not use the full dynamic
range. This didn't affect data values, except perhaps at the
round-off level. Routine changed:
getStats.cgi.
- 19-Jan-2015 (beta and production)
- All beta changes moved to production
- Added 60D, 90D and 180D averages. Routine changed:
CurveParamerers.
- Added line breaks to difference curves whenever the time gap
between ponts in greater than the minimum time gap between points
(just as we do for non-difference curves). Routine
changed: getStats.cgi
- 16-Jan-2015 (beta)
- For averaged data (e.g. 7D averages), we do not plot points if
the number of valid times in the averaging period is too
small--indicating missing model results. For each curve, we note
the maximum number of valid times in an averaging period (most
likely this occurs when a model produced results for all
expected valid times). If the number of distinct valid times in
a particular period is less than 75% of that maximum, then that
value isn't plotted.
- We have cleaned up when we plot a continuous curve and when we
break a curve, and we treat unevenly-spaced (due to missing
runs) data better. We now plot all valid times for
which there is data. We note the minimum time gap between good points
on each curve. Whenever two good points on a curve are separated by
more than the minimum time gap, we break the curve.
- We have cleaned up difference curves. Now we only calculate
difference data for times when both of the differrenced curves
have data.
- 17-Oct-2014 (beta)
- Worked around an apparent bug in the latest version of java that
causes the codeBase to include arguments if the filename is left
out of the URL. I.e., URL's ending in /?debug=true include the
"?debug=true" in the codeBase. We use string substitution to
correct for this. Routine changed:
StatsDisplay.
- Added infrastructure for requiring a password for access to some
model results (currently for the model 'MPAS', statistics for which
don't exist yet). Routines changed or new:
StatsDisplay, CurvesControl, DisplayFrame, ...lib.PasswordFrame
(new), ...lib.PasswordReader (new).
- 12-Sep-2014 (beta and production)
- re: the change of 16-July-2014: we have relaxed the criteria for
complete averaging intervals from 90% to 75%,
because we were
getting too many gaps in, e.g., 7-day averages. If
the actual data within an averaging interval fills (contiguously)
less than 75% of the averaging interval, that averaging interval
is set to 'missing'. Routines changed:
getStats.cgi.
- 28-Aug-2014 (beta and production)
- All beta changes moved to production
- Made some changes to allow the java code to be run as an
application to generate png's. Shouldn't cause any change to the
behavior of the applet. Routines changed:
DisplayFrame, StatsDisplay.
- 13-Aug-2014 (beta)
- Fixed problem with detecting incomplete average intervals for 1D
and 3D averaging.
- Allowed negative forecast lengths (persistence forecasts).
- Routines changed: CurveParameters,
getStats.cgi.
- 16-July-2014 (beta and production)
- Fixed to not put out data from incomplete averaging intervals. If
the actual data within an averaging interval fills (contiguously)
less than 90% of the averaging interval, that averaging interval
is set to 'missing'. Routines changed:
getStats.cgi.
- 12-Jun-2014 (beta and production)
- Corrected an error in the production of 'RAOB average' and
'model average' statistics. We forgot to divide by 100 for wind
and temperature.
- Added missing statistics for 'RAOB average' and 'model average'
when the region is set to an individual RAOB's wmoid. We now
generate statistics for winds, temperature, rh, and
height. Previously, only winds were available, and they were off
by a factor of 100.
- 7-Oct-2013 (production)
- All beta changes moved to production.
- 9-July-2013 (beta)
- Found an error in the calculation of sum_model_ws in the summary
tables. The sum included times/locations when the RAOB wind was
null. Since the model wind is subsequently calculated as
sum_model_ws/N_dw, and since N_dw only includes times/locations
when the RAOB wind is non-null, the divisior is inappropriate. So,
now the sum_model_ws only includes times/locations when the RAOB
wind data also exists. This didn't impact North American RAOBs,
because they apparently report winds at all levels. But elsewhere
in the world they do not.
- 20-May-2013 (beta)
- Found and corrected an error when the statistic 'model' is used. It was
previously calculated for T and RH as ob + bias. But since bias is
defined as 'ob minus model', to get the model value correctly, we
must use ob - bias. This is now corrected. Routine changed: getStats.cgi
- 1-Apr-2013 (beta, no fooling)
- Corrected units on the plots for RH and RHob. Routine changed: DisplayFrame.
- Made the web page a little narrower for ease of reading on mac
laptops. Routines changed: CurvesControl,
CurveParameters.
- 28-Mar-2013 (beta)
- Set the
model
field to editable, so the user can
enter a model name without finding it int he long dropdown list.
Routines changed: CurvesParameters.
- Extended the possible forecast projections out to 240 hours to
accommodate the FIM and related models. Routines
changed: Welcome.cgi.
- 22-Mar-2013 (beta)
- Changed 'humidity' variable to 'RH' and added 'RHobT' (Model RH
w.r.t, observed T) as a variable for some models.
- Added a warning message whenever there's an error reading a curve,
so folks don't get the 'wait cursor of death'.
- Cleaned up less-often-used stats such as RAOB and model averages.
- Routines changed: getStats.cgi, DisplayFrame,
CurvesParameters.
- 27-Feb-2013 (beta)
- Brought back the zero line when differences are displayed (it
vanished due to a bug). Also,
changed the order of colors, so the zero line shows in gray when
two curves are differences. Also, linked the colors of the curves
with those of the curve-parameter forms. Routine
changed: getStats.cgi, DisplayFrame,
CurvesControl.
- 7-Dec-2012 (production and beta)
- Changed the dp_to_rh_calculator for isoBak13 from Fan-Whiting to
'FW-to-Wobus' to match the other RUC runs.
- 11-Oct-2012 (beta)
- For matched curves we now do a joint query to ensure that the model
events are matched. (There was no problem identified with the old
way, but its logic was cumbersome and hard to maintain.). Also, we
send out the mean and range of each curve, and data scaled in the
range +/- 32766, to provide slightly better accuracy, and the
ability to plot the number of raob's and levels for each data
point. Routine changed: getStats2.cgi,
CurvesControl, DisplayFrame.
- We now allow a comma-delimited series of RAOB wmoid's to be
entered in the 'Region' field. Routine changed:
getStats.cgi, CurveParameters.java.
- 26-Sep-2012 (beta)
- Cloud cover ("cvr") option added. If you select "cloudy",
statistics will be generated by comparing RAOBS only with model
soundings that have cloud hydrometers at some level. But the
comparison will be done at all levels. Clear is the
converse. WARNING hydrometeor values are not
loaded into the database for all models. If the number of 'cloudy'
comparisons is zero, this is probably one of those models. The
only models for which this is currently implemented are
HRRR and RR1h.
- Average RAOB T and RH and average model T and RH are now
available statistics. The only models for which this is currently
implemented are HRRR and RR1h.
- Routines changed:
getStats.cgi,CurveParameters.
- 5-Mar-2012 (beta)
- The default curve weight is now HEAVY, rather than SOLID, per
Stan's request. Routine changed
DisplayFrame.
- 22-Feb-2012 (beta)
- Adjusted the web page to correctly list models that have tables
that end in "_Areg0" as well as "_reg0". The tables that end in
"_Areg0" (or other region numbers) include average model wind
speed and average RAOB wind speed, so wind speed bias can be
calculated. Routine changed:
Welcome.cgi.
- 9-Feb-2012 (beta)
- Added region '14' -- the HRRR domain -- so comparisons between the
HRRR and models in larger regions will be accurate. (HRRR does not
include 14 active RAOBs that are in the RUC domain.)
- The regions are now provided as a parameter to the applet, rather
than hardwired in the code.
- Routine changed:StatsDisplay, CurveParameters,
CurvesControl.
- 5-Jan-2012 (production and beta)
- The upper limit of the drop-down box for years now updates
automatically to the current year +1. Routine
changed: CurvesControl, lib.JYearBox.
- 17-Nov-2010 (production and beta)
- The list of models is now generated by looking at the database,
for tables that match "ruc_ua_sums2.*_reg0". If a table of that
name exists, the model name implicit in the table name is included
in the list of models. Routine changed:
Welcome.cgi.
- 4-Nov-2010 (production and beta)
- All beta changes moved to production.
- Made 'projections' (aka 'forecast_lens') an input parameter to the
applet.Made 'projections' (aka 'forecast_lens') an input parameter to the
applet.
- Fixed the repainting so that closed CurveParameter panes truly disappear.
- Routines changed: Welcome.cgi,
StatsDisplay, CurveParameters, CurvesControl.
- 2-Nov-2010 (beta)
- Corrected order of colors in curves. Routines
changed: getStats.cgi.
- Made the zero line HEAVY so it shows better. Routines
changed: DisplayFrame.
- 28-Oct-2010 (beta)
- Added 18h forecasts to the list of available 'proj'. Routine(s) changed: CurveParameters.
- 22-Oct-2010 (beta)
- Can now eliminate any curves by clicking on the 'x' in the upper
right of each curve's parameter specification. (Previously, you
could only eliminate the last curve.) Routine(s) changed: DisplayFrame, CurveParameters,
CurvesControl, StatsDisplay, SimpleLine2, getStats.cgi.
- 21-Oct-2010 (beta)
- Corrected for a bug in Java 6 update 22 that breaks JButtons with
html text. Routines changed: CurvesControl.
- 22-Feb-2010 (beta and production)
- For matched curves, we now require that the number of items in the
averaged or summed 'group by' group are the same. This is in response to some RR
soundings that did not include all levels, so the group average
over multiple levels were not always consistent between
curves. (We believe we've also fixed the RR processing that
allowed incomplete files to generate soundings.)
Routines changed: getStats.cgi.
- 15-Feb-2010 (beta, and production on 22-Feb-2010)
- If difference curves are not asked for, the zero
curve is not put out. Routines changed:
getStats.cgi.
- 12-Feb-2010 (beta)
- Text window summary of curve statistics shows the last
(difference) curves first, for ease or reading. Also, the location
of the time series plot is a little lower so that it doesn't
obscure the summary statistics. Routines changed:
DisplayFrame, NewFrameThread.
- 10-Feb-2010 (beta)
- Text window now shows a summary of curve statistics at the top.
Routines changed: DisplayFrame.
- 27-Jan-2009 (production and beta)
- All beta changes moved to production
- Difference curves are now optional. Routines
changed: CurvesControl, getStats.cgi
- 8-Oct-2009 (beta)
- Added a text window option.
- Standard error calculation now is made even for data sets with
missing data. Mean, standard error, standard deviation, and
other statistical summaries of each curve are available in the
text window.
- Routines changed: DisplayFrame, StatsDisplay,
CurvesControl.
- 3-Oct-2009 (beta)
- Added a date range, which defaults to the past 30 days.Routines changed: CurvesControl, StatsDisplay.
- 21-Sep-2009 (beta)
- Added standard error calculation (only works for data sets without
missing values). Moved the calculation of the mean and standard
deviation to the new statistics-processing routine. New routine: gov.noaa.fsl.frd.lib.Stat.
- 1-May-2009 (beta)
- The ability to include a date range has been added.
- Plot pairwise has been added so that various
pairs of models can be compared. This can be useful to compare
pairs of models that don't generally overlap in time, such as
dev/dev2 and NoTAM13/Bak13.
- Routines changed: CurveParameters,
CurvesControl, getStats.cgi, Welcome.html.
- 20-Jan-2009 (production and beta)
- A little more rearrangement, and moved beta changes to production.
- 2-Jan-2009 (beta)
- Rearranged the order of fields to use screen space more
efficiently. Also, removed the 'matched' attribute for each curve
and replaced that function with 'plot unmatched' and 'plot
matching' buttons. Also, renamed the web page from 'inter.html' to
'Welcome.html'. Routines changed:
CurveParameters, CurvesControl,Welcome.html.
- 16-June-2008 (production and beta)
- Region names instead of region numbers are now put on the output
plots. Routines changed: CurveParameters,
DisplayFrame, getStats.cgi.
- 3-June-2008 (production )
- The sign of the bias statistic has been changed: it is now 'model
minus RAOB', to match what is done at NCEP and by RTVS. Routines changed: CurveParameters, getStats.cgi.
- When two or more curves are plotted, a 'zero' curve is also
plotted. Routine changed: getStats.cgi.
- 5-May-2008 (beta and production)
- Order of curve legends is reversed. First curve plotted is at the
bottom, subsequent curves are higher up. This was done so that
difference curves can more easily be clipped off screen captures.
Routine changed: DisplayFrame.
- 28-Mar-2008 (production)
- All beta changes moved to production
- 27-Mar-2008 (beta)
- Added the ability to input a single RAOB wmoid as the
'region'. (This avoids using the region summary tables in database
ruc_ua_sums2, which is useful for debugging.) Routines changed: CurveParameters, getStats.cgi.
- Corrected an error in database loading that caused model soundings
to be missing for many RAOB sites. Fortunately, the sounding
information was available in text files, so we were able to
restore the database with the correct data and regional
summaries. The problem persisted from about 29-Feb-2008 until
today. (This fix also effects the production site.) Routines changed: Verify, UpdateAllSums.
- 11-Mar-2008
- We've become more strict on loading soundings into the
database. If any
line in the sounding is bad, we reject the entire thing. This
primarily affects RAOBs. In addition, we've added several models
over the past year. We also now print out a warning if the
location reported in a sounding differs from the location
specified in the database. Routines changed:
SoundingLoader2, Verify, UpdateAllSums.
- 28-March-2007
- Forecast projection '12' added. Routine changed:
CurveParameters.
- 1-March-2007 (beta and production)
- Standard RUC menu added to left side.
- January-2007 (beta and productions)
- Plot generation speeded up by pre-calculating daily sums.
- November-2006 (beta and production)
- Full RAOB soundings, interpolated to every 10 mb, are compared
with model soundings
- Model soundings, interpolated to every 10 mb, are generated directly
from RUC native files
- Comparisons are made every 10 mb up from the surface.
- No RAOB data are automatically eliminated. (About a dozen
obviously erroneous RAOBs have been eliminated by hand since 23
February 2006).
Last modified: Wed Feb 27 23:07:20 GMT 2019