Appendix B.3: Investigating Prediction Uncertainty Using 5-year Moving Windows
Megan Higgs - Critical Inference LLC
Last update 11/7/2020
Original data files
The data file used for this analysis was created by combining data from the following spreadsheets provided by technical consultant Dr. Bruce Johnson. Data for US-41 and CR-846 are from Copy of retrospective_traffic_data_updated_20201021.xlsx and data from SR-29, SR-82, and Church Road are from Copy of PTMS AADT history_20200921.xlsx.
Assess prediction errors and variability in predictions
In this investigation, non-overlapping 5-year windows are used as baseline and prediction time periods for PVM and the FREM equation is applied using the midpoint year AADT for each time period. This is a best case scenario because using observed/calculated AADT for given segment’s future AADT rather than predicted, thus removing a substantial source of uncertainty associated with predicting AADT in the future.
Predictions based on non-overlapping 5 year intervals
Combine all segments into one faceted plot
Summarize the predictions by future year
This section provides summaries of different summary statistics calculated for each year predictions are available for a particular road segment.
Summaries of standard deviations of predictions for a given year:
## CR-846 FDOT SR-29 CC-MS2 SR-29 FDOT SR-82 FDOT US-41 FDOT
## Min. 0.1309582 0.1589509 0.1866975 0.00000000 0.004863649
## 1st Qu. 0.1359724 0.1674849 0.2713015 0.00000000 0.145813675
## Median 0.1580090 0.1760190 0.3222695 0.00000000 0.307095958
## Mean 0.1742065 0.1760190 0.3440253 0.01296257 0.247762295
## 3rd Qu. 0.1962431 0.1845530 0.4006227 0.00000000 0.325040362
## Max. 0.2498497 0.1930870 0.5596634 0.10096658 0.475262985
## NA's 15.0000000 17.0000000 1.0000000 1.00000000 3.000000000Summaries of the proportion of predictions greater than or equal to 0.8:
## $`CR-846 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.00 0.00 0.00 0.32 0.60 1.00 14
##
## $`SR-29 CC-MS2`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 16
##
## $`SR-29 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.3618 0.5294 0.4736 0.6500 0.8182
##
## $`SR-82 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0 0 0 0 0 0
##
## $`US-41 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.0000 0.0000 0.2000 0.1703 0.2727 0.5625 2Summaries of the proportion of predictions greater than or equal to 0.8 excluding predictions of 0:
## 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
## CR-846 FDOT NA NA NA NA NA NA NA NA NA NA
## SR-29 CC-MS2 NA NA NA NA NA NA NA NA NA NA
## SR-29 FDOT 0 0 0 0.25 0.2 0.5 0.7142857 0.75 0.7777778 0.7
## SR-82 FDOT NA NA NA NA NA NA NA NA NA NA
## US-41 FDOT NA NA 0 0.00 0.0 0.0 0.0000000 0.00 0.0000000 0.0
## 2009 2010 2011 2012 2013 2014
## CR-846 FDOT NA NA NA NA 1.00 0.0000000
## SR-29 CC-MS2 NA NA NA NA NA NA
## SR-29 FDOT 0.8181818 0.5833333 0.5384615 0.5000000 0.60 0.6000000
## SR-82 FDOT NA NA NA NA NA NA
## US-41 FDOT 0.2500000 0.2222222 0.3000000 0.2727273 0.25 0.3076923
## 2015 2016 2017
## CR-846 FDOT 0.0000000 0.0000 0.6000000
## SR-29 CC-MS2 NA 0.0000 0.0000000
## SR-29 FDOT 0.6000000 0.5625 0.5294118
## SR-82 FDOT 0.0000000 0.0000 0.0000000
## US-41 FDOT 0.4285714 0.6000 0.5000000Summaries of the proportion of predictions greater than or equal to 1.0:
## $`CR-846 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.00 0.00 0.00 0.08 0.00 0.40 14
##
## $`SR-29 CC-MS2`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 16
##
## $`SR-29 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.1071 0.3684 0.3222 0.4837 0.6364
##
## $`SR-82 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0 0 0 0 0 0
##
## $`US-41 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.00000 0.00000 0.00000 0.05398 0.00000 0.31250 2Visualize prediction errors
These plots are similar to the above plots, but plot prediction errors rather than raw predicted 5-year average PVM. Vertical variability within a year displays prediction errors for a given future year given different baseline intervals and years. Horizontal variability with a connection of points of the same color, connected by a line provides variability in predictions errors for a given baseline year as the future year changes.
Summarize the predictions by future year
These provide similar summaries as provided for the predictions.
Summaries of the standard deviation of prediction errors (same as for predictions):
## CR-846 FDOT SR-29 CC-MS2 SR-29 FDOT SR-82 FDOT US-41 FDOT
## Min. 0.1309582 0.1589509 0.1866975 0.00000000 0.004863649
## 1st Qu. 0.1359724 0.1674849 0.2713015 0.00000000 0.145813675
## Median 0.1580090 0.1760190 0.3222695 0.00000000 0.307095958
## Mean 0.1742065 0.1760190 0.3440253 0.01296257 0.247762295
## 3rd Qu. 0.1962431 0.1845530 0.4006227 0.00000000 0.325040362
## Max. 0.2498497 0.1930870 0.5596634 0.10096658 0.475262985
## NA's 15.0000000 17.0000000 1.0000000 1.00000000 3.000000000Summaries of the standard deviation of prediction errors (same as for predictions):
Summaries of the proportion of prediction errors with absolute value greater than or equal to 0.8:
## $`CR-846 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 14
##
## $`SR-29 CC-MS2`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 16
##
## $`SR-29 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.0000 0.1765 0.2397 0.3875 0.8182
##
## $`SR-82 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.0000 0.0000 0.2016 0.0000 1.0000
##
## $`US-41 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.00000 0.06667 0.17647 0.28193 0.50000 0.70000 2Summaries of the proportion of prediction errors with absolute value greater than or equal to 0.8 for predictions that are not zero:
## 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
## CR-846 FDOT NA NA NA NA NA NA NA NA NA NA
## SR-29 CC-MS2 NA NA NA NA NA NA NA NA NA NA
## SR-29 FDOT 0 0 0 0 0 0.00 0.1428571 0.375 0.5555556 0.6
## SR-82 FDOT NA NA NA NA NA NA NA NA NA NA
## US-41 FDOT NA NA 0 0 0 0.25 0.4000000 0.400 0.0000000 0.0
## 2009 2010 2011 2012 2013 2014 2015
## CR-846 FDOT NA NA NA NA 0.0 0.0000000 0.0
## SR-29 CC-MS2 NA NA NA NA NA NA NA
## SR-29 FDOT 0.8181818 0.5833333 0.3076923 0.2142857 0.4 0.0000000 0.2
## SR-82 FDOT NA NA NA NA NA NA 0.0
## US-41 FDOT 0.0000000 0.6666667 0.6000000 0.5454545 0.5 0.4615385 0.0
## 2016 2017
## CR-846 FDOT 0.00000000 0.0000000
## SR-29 CC-MS2 0.00000000 0.0000000
## SR-29 FDOT 0.25000000 0.1764706
## SR-82 FDOT 0.00000000 1.0000000
## US-41 FDOT 0.06666667 0.1875000Summaries of the proportions from above across years:
## $`CR-846 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 14
##
## $`SR-29 CC-MS2`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0 0 0 0 0 0 17
##
## $`SR-29 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0000 0.0000 0.2000 0.2433 0.3875 0.8182
##
## $`SR-82 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.0000 0.0000 0.0000 0.3333 0.5000 1.0000 16
##
## $`US-41 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.0000 0.0000 0.1875 0.2399 0.4615 0.6667 2Summaries of the average prediction errors for predictions that are not zero:
## 1999 2000 2001 2002 2003
## CR-846 FDOT NA NA NA NA NA
## SR-29 CC-MS2 NA NA NA NA NA
## SR-29 FDOT -0.08888889 0.2833333 0.2703704 0.42106481 0.2325589
## SR-82 FDOT NA NA NA NA NA
## US-41 FDOT NA NA -0.4152368 -0.02457909 0.3948186
## 2004 2005 2006 2007 2008
## CR-846 FDOT NA NA NA NA NA
## SR-29 CC-MS2 NA NA NA NA NA
## SR-29 FDOT -0.01482387 -0.3585967 -0.6805026 -0.8107435 -0.7854514
## SR-82 FDOT NA NA NA NA NA
## US-41 FDOT 0.64407889 0.6896618 0.6856015 0.4742554 0.4585968
## 2009 2010 2011 2012 2013
## CR-846 FDOT NA NA NA NA -0.4461647
## SR-29 CC-MS2 NA NA NA NA NA
## SR-29 FDOT -1.0803528 -0.8265004 -0.5865132 -0.5102202 -0.6273711
## SR-82 FDOT NA NA NA NA NA
## US-41 FDOT 0.3159746 0.7062341 0.8693869 0.8880040 0.8597788
## 2014 2015 2016 2017
## CR-846 FDOT -0.09624894 0.3913300 0.3136533 0.02834891
## SR-29 CC-MS2 NA NA 0.3269337 0.32508379
## SR-29 FDOT -0.22737113 -0.3193013 -0.3366926 -0.24253622
## SR-82 FDOT NA 0.5775510 0.7575510 0.93053741
## US-41 FDOT 0.61432449 0.1175624 -0.1729002 -0.39768926## $`CR-846 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## -0.44616 -0.09625 0.02835 0.03818 0.31365 0.39133 14
##
## $`SR-29 CC-MS2`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.3251 0.3255 0.3260 0.3260 0.3265 0.3269 17
##
## $`SR-29 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## -1.08035 -0.65394 -0.33669 -0.33098 -0.05186 0.42106
##
## $`SR-82 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.5776 0.6676 0.7576 0.7552 0.8440 0.9305 16
##
## $`US-41 FDOT`
## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## -0.4152 0.1176 0.4743 0.3946 0.6897 0.8880 2Observed vs. Predicted values by “future” year
Compare SLR to FREM predictions informally
This is not included in the report is not a recommendation for an alternative method for prediction. It is simply meant as a comparison to show how different the FREM predictions are from those based on regression lines estimated from the data. This also demonstrates the magnitude of prediction intervals based on regression and there is no reason to assume FREM predictions should have much less uncertainty unless the formula has a strong theoretical foundation and need not be based on data and estimation.
#Need to fit the lm to every segment and collect all the predictions
# Should write a function, but will probably just refit for the 5 segments given time
# Fill in the segment_source of interest:
# Options are: "SR-29 CC-MS2", "SR-29 FDOT", "SR-82 FDOT", "CR-846 FDOT", "US-41 FDOT", "Church Rd FDOT"
lm_data <- dplyr::filter(seg_df, segment_source=="SR-29 FDOT")
data_forpred <- dplyr::filter(future_data, road_segment=="SR-29")
ll_slr_out <- lm(ln_MA5_PVM ~ ln_AADT_1000, data=lm_data)
ll_newdata_df <- data.frame(YEAR=2060, ln_AADT_1000=log((data_forpred$AADT_2060/1000)), ln_MA5_PVM=NA)
pred_test <- predict(ll_slr_out, newdata=ll_newdata_df, se.fit=TRUE, interval="confidence")
pred_2060_lm1 <- predict(ll_slr_out, newdata=ll_newdata_df, se.fit=TRUE, interval="prediction", level=0.95)
pred_2060_lm1## $fit
## fit lwr upr
## 1 -0.9828912 -3.326022 1.360239
##
## $se.fit
## [1] 0.8847963
##
## $df
## [1] 22
##
## $residual.scale
## [1] 0.7026081exp(pred_2060_lm1$fit) #backtransformed prediction and 99% prediction interval## fit lwr upr
## 1 0.3742276 0.03593579 3.897125pred_2060_lm2 <- predict(ll_slr_out, newdata=ll_newdata_df, se.fit=TRUE, interval="prediction", level=0.99)
pred_2060_lm2## $fit
## fit lwr upr
## 1 -0.9828912 -4.167615 2.201832
##
## $se.fit
## [1] 0.8847963
##
## $df
## [1] 22
##
## $residual.scale
## [1] 0.7026081exp(pred_2060_lm2$fit) #backtransformed prediction and 99% prediction interval## fit lwr upr
## 1 0.3742276 0.01548916 9.041566#pred_int95 <- exp(pred_2060_lm1$fit)[-1]
#pred_int99 <- exp(pred_2060_lm2$fit)[-1]
#pred_2060_lm <- pred_int95[1]
##Results for different segments
# SR-29 - CC-MS2
# fit lwr upr
#1 391.692 2.573587 59614.31
# SR-29 - FDOT
#exp(pred_2060_lm1$fit)
# fit lwr upr
#1 0.3742276 0.03593579 3.897125
#US-41
# exp(pred_2060_lm1$fit)
# fit lwr upr
# 1 0.01050003 0.0009161512 0.1203411
#SR-82
# exp(pred_2060_lm1$fit)
# fit lwr upr
#1 1.474867 0.1783914 12.1936
#CR-846
#exp(pred_2060_lm1$fit)
# fit lwr upr
#1 0.6478387 0.2494381 1.682562
#Church Rd
# exp(pred_2060_lm1$fit)
# fit lwr upr
#1 0.2 9.619061e-09 4158410Check Poisson model for the counts
Not included in report. Code below is just for one road segment at a time.
Compare the 5 year moving averages to what would be obtained from a Poisson model of independent counts. This is just a quick and informal check of reasonableness of the simple Poisson model, but does not include any temporal autocorrelation.
The below plots are only for SR29FDOT_df - need to change dataframe to see others.
