Judgmental Forecasting. Three components of a time series influence the degree of difficultythat is associated withthe judgmental forecasting task, namely:· the complexityof the underlying signal, comprising factors such as itsseasonality,cyclesandtrends,and autocorrelation;:thelevel of noisearoundthesignal; and: the stability of the underlying signal.Commonmistakes·Damping thetrend: Confusing noise with signal:Underestimationofuncertainty
Judgmental Forecasting • Three components of a time series influence the degree of difficulty that is associated with the judgmental forecasting task, namely: • the complexity of the underlying signal, comprising factors such as its seasonality, cycles and trends, and autocorrelation; • the level of noise around the signal; and • the stability of the underlying signal. • Common mistakes • Damping the trend • Confusing noise with signal • Underestimation of uncertainty
? Skill Score(MSEy)(H)E(Yi- 0,)MSEs= (H)(0- 0.)2SS=1MSEy=MSEB: Murphy's decomposition (Murphy, 1988)ss=(ro)-[ro-(]"-{-]where rxo is thecorrelation between theforecast and the observed event;sy and so are thestandard deviations of the forecast and the observed event, respectively; and Y and O are themeans of theforecastandtheobserved event.Murphycalledthesecondterm'conditionalbias.Murphycalledthethirdterm‘unconditionalbias
• Skill Score • Murphy’s decomposition (Murphy, 1988)
·3-stepdecomposition(Steward&Lusk,1994):Step1-Murphy'sdecomposition:Step2-Usethelensmodel equation (LME)tofurtherdecomposethecorrelationcomponentoftheMurphy'sdecomposition.TheLMEshowsthatthecorrelationisdeterminedbypropertiesoftheenvironmentalsystem,thecognitivesystemandtherelationsbetweenthem.·Step3-Thetwocomponents(unreliabilityofsubjectiveinterpretationofcues and unreliability of information processing)are further decomposed
• 3-step decomposition (Steward & Lusk, 1994) • Step 1 – Murphy’s decomposition • Step 2 - Use the lens model equation (LME) to further decompose the correlation component of the Murphy’s decomposition. The LME shows that the correlation is determined by properties of the environmental system, the cognitive system and the relations between them. • Step 3 – The two components (unreliability of subjective interpretation of cues and unreliability of information processing) are further decomposed
SS=Skil Score=1-(MSE/MSE,)ConditionalUnconditionalSquaredcomelation(regression)(base rate)biasbiasStep 1:(rmo)"SS=Murphy[0 -(5v/80)]"-[(Y-0)/s0]?(1988)Step 2:GSSi(RoxRyx)Tucker[ro-(sy1s0)[(Y.0)1s0](1964)Step3:ExpandedRu)"- [vo -($1s0)]"-[(Y.0)/s0]?VixGVuxSSE(Rotlensmodel0?OOOO①Components of skill:1.Environmentalpredictabity2.Fidellty of theinfomation system3.Matchbetweenenvironnentandforecaster4.Reliabilityofinfomationacquisition5.Reliabilityof infomationprocessing6.Conditiona/regressionbias7.Unconditional/base ratebias
Table I.Components of skill addressed by selected methods for improving forecastsComponent of skill"-23456MethodforimprovingforecastsxAIdentifynewdescriptorsthroughresearchxBDevelopbettermeasuresoftruedescriptorsxxxxcTrainforecasteraboutenvironmental systemxXDExperiencewithforecastingproblemECognitivefeedbackFTrainforecastertoignorenon-predictivecuesXXXGDevelopcleardefinitionsof cuesHTraining to improve cue judgmentsx1ImproveinformationdisplaysXxxxxJBootstrapping-replaceforecaster with modelKCombine several forecastsxLRequire justification of forecastsMDecomposeforecastingtaskNMechanical combination of cuesXxxxX0Statistical trainingPxFeedback about nature of biases in forecastQSearch fordiscrepant informationxRStatistical correction for bias
Feedback type:Outcomefeedback:Performancefeedback:Cognitiveprocessfeedback:Taskpropertiesfeedback
• Feedback type • Outcome feedback • Performance feedback • Cognitive process feedback • Task properties feedback
Feedback-Based Rolling Training(Petropoulos et al., 2017)·Focusonperformancefeedback,distinguishingtwotypes:feedbackonthebiasassociatedwiththeforecastssubmitted,andfeedbackontheiraccuracy·Participantsare10o5undergraduatestudents.Eachparticipantprovidedjudgmentalestimates(4-periodahead)followingbothapproaches(unaidedandafterrollingtraining),usingafullysymmetricexperiment·16quarterlyserieswereselectedmanuallyfromtheM3-Competitiondataset.Foranalysispurposes,the16seriesweresplitagainintotwosetsofequalsizeintermsof noise (lowand high).The required lengthof all series was setto28points(sevenyears),withTongerseriesbeingtruncated: 28 in-sample points and 4 out-of-sample points. For rolling training, 3 blocks of 4-periodareused (12in-samplepointsforfirstestimate).Aftercompletingeachofthelattertworounds,theparticipantsfilledinaquestionnaireforsubsequentanalysis
Feedback-Based Rolling Training (Petropoulos et al., 2017) • Focus on performance feedback, distinguishing two types: feedback on the bias associated with the forecasts submitted, and feedback on their accuracy • Participants are 105 undergraduate students. Each participant provided judgmental estimates (4-period ahead) following both approaches (unaided and after rolling training), using a fully symmetric experiment. • 16 quarterly series were selected manually from the M3-Competition data set. For analysis purposes, the 16 series were split again into two sets of equal size in terms of noise (low and high). The required length of all series was set to 28 points (seven years), with longer series being truncated. • 28 in-sample points and 4 out-of-sample points. For rolling training, 3 blocks of 4- period are used (12 in-sample points for first estimate). • After completing each of the latter two rounds, the participants filled in a questionnaire for subsequent analysis
Questionsposedtotheparticipants.QuestionsAfterbothUJand RT roundsHow confidentareyou thattheforecasts you submitted in this round,on average,wouldbewithin 10% ofthe actual values?Please,rateyour expectedforecasting performance in the series of thisroundDidyouexaminecarefullythetimeseriesgraphs?Didyoutakeintoaccountanyhistoricpatterns intheserieswhenmakingyourforecastsduringthis round?Howmuchtime(onaverage)didyouspendforeachseriesofthisround?How likely it is that takingmoretime would changeyourforecasts?AftercompletionoftheexperimentHowfamiliarareyouwithsuchforecastingexercises?Howwouldyoudescribeyourlevelofexpertise?Please,ratethe effectiveness of rolling trainingas a tool to increase youraccuracy.Please,indicatehowmotivated youwereto provideaccurateestimates
·PercentageimprovementinaccuracyMAERT100MAE,=ZPmedian(%)二MAED·Majorfindings.Overall,theRTapproach results in statisticallysignificant betterforecastingperformances(3.78%performancegain):RTresultsinimprovementsforseriesbothwithhighnoise(5.18%)andwhenlongerhorizonsareexamined(4.17%):Biasfeedbackdemonstratesthemostsignificantimprovements(4.89%overall)whiletheimprovementsforaccuracyfeedbackaregenerallysmallerandnotconsistent.·RTleadsparticipantstobemorecautious.Theforecastingperformancesachievedwithboth UJand RT areassociated withthetimethattheparticipants reportedspendinginproducingtheforecasts
• Percentage improvement in accuracy • Major findings • Overall, the RT approach results in statistically significant better forecasting performances (3.78% performance gain). • RT results in improvements for series both with high noise (5.18%) and when longer horizons are examined (4.17%). • Bias feedback demonstrates the most significant improvements (4.89% overall), while the improvements for accuracy feedback are generally smaller and not consistent. • RT leads participants to be more cautious. The forecasting performances achieved with both UJ and RT are associated with the time that the participants reported spending in producing the forecasts
Judgmental Hierarchical Forecasting(Kremeret al.,2016)? Forecast directly on aggregate data vs. summing up indirectly fromlower-levelforecasts·Lowlevel demand series arenonstationaryand correlated· Subjects: undergraduate, graduate, and professional·One-period aheadforecast·Conclusions:.Whetherbottom-upor direct-topforecasting is advantageous fromajudgmentalforecastingperspectivedependstoalargedegreeontheunderlying correlation structureat the lower level
Judgmental Hierarchical Forecasting (Kremer et al., 2016) • Forecast directly on aggregate data vs. summing up indirectly from lower-level forecasts • Low level demand series are nonstationary and correlated • Subjects: undergraduate, graduate, and professional • One-period ahead forecast • Conclusions: • Whether bottom-up or direct-top forecasting is advantageous from a judgmental forecasting perspective depends to a large degree on the underlying correlation structure at the lower level