Chapter6TheMultilayerPerceptron6.1Introduction6.2Feed-forwardNetworkMapping6.3SigmoidUnits2.“tanh”ActivationFunction:6.4MethodofTraining:BPTheminimizationofthecostfunctionbyusinggradientmethod:(1)Thegradientforthehidden-to-outputweights:(2)Thegradientfortheinput-to-hiddenweights:(3)Summaryofthegradientcomputing:3.ImplementingBPAlgorithm(Cont’d)(Cont’d)Step3:Accumulategradientsovertheinputpatterns4.Networkswithmorethan2LayersBackpropagationofError:AnExample(Cont’d)(Cont’d)(Cont’d)(Cont’d)(Cont’d)(Cont’d)Demonstrations6.5TrainingAspects:LocalMinima(Cont’d)(Cont’d)2.TrainingwithNoiseSolutions:3.MomentumandLearningRateAdaptationWithmomentumµm,theweightupdateatagiventimetbecomeswhere0<µm<1isanewglobalparameterwhichmustbedeterminedbytrialanderror.Momentumsimplyaddsafractionµmofthepreviousweightupdatetothecurrentone.Whenthegradientkeepspointinginthesamedirection,thiswillincreasethesizeofthestepstakentowardstheminimum.Itisthereforeoftennecessarytoreducethegloballearningrateµwhenusingalotofmomentum(µmcloseto1).Ifyoucombineahighlearningratewithalotofmomentum,youwillrushpasttheminimumwithhugesteps!(Cont’d)(Cont’d)(2)LearningRateAdaptation(Cont’d)(Cont’d)6.6TrainingAspects:OverTraining(Cont’d)2.Bias-VarianceTrade-off(Cont’d)3.GeneralizationAbility4.PreventingOverfitting/Overtraining(1)Earlystopping(Cont’d)(2)Weightdecay(3)Trainingwithnoise6.7TrainingAspects:OptimizingNetSizeBywayofanexample,thenonlineardatawhichformedourfirstexamplecanbefittedverywellusing40tanhfunctions.Learningwith40hiddenunitsisconsiderablyharderthanlearningwith2,andtakessignificantlylonger.Theresultingfitisnobetter(asmeasuredbythesumsquarederror)thanthe2-unitmodel.Themostusualanswerisnotnecessarilythebest:weguessanappropriatenumber(aswedidabove).Anothercommonsolutionistotryoutseveralnetworksizes,andselectthemostpromising.Neitherofthesemethodsisveryprincipled.Twomorerigorousclassesofmethodsareavailable,however.Wecaneitherstartwithanetworkwhichweknowtobetoosmall,anditerativelyaddunitsandweights,orwecant