Logistic classification model (logit or logistic regression)

The logistic model (or logit) is a classification model used to predict variables that can take only two values. StatLect Index>Fundamentalsof statistics Logisticclassificationmodel(logitorlogisticregression) byMarcoTaboga,PhD Thelogisticmodel(orlogit)isa classification modelusedtopredictvariablesthatcantakeonlytwo values. Tableofcontents OverviewInterpretationofthepredictedoutputClassificationvsregressionSampleConditionalprobabilitiesThelogisticfunctionExplanationAlternativesThelogitmodelasalatentvariablemodelEstimationbymaximumlikelihoodHypothesistesting Overview Thelogisticclassificationmodelhasthefollowingcharacteristics: theoutputvariable canbeequaltoeither0or1; thepredictedoutput isanumberbetween0and1; asinlinearregression,weuseavectorofestimatedcoefficients tocompute , alinearcombinationoftheinputvariables ; unlikeinlinearregression,wetransform usinganonlinearfunction , tomakesurethatthepredictions are between0and1. Interpretationofthepredictedoutput Inalogitmodel,thepredictedoutput hastwointerpretations: theestimatedprobabilitythat willbeequalto1; ourbestguessofthevalueoftheoutputvariable . Classificationvsregression Alogitmodelisoftencalledlogisticregressionmodel. However,weprefertosticktotheconvention(widespreadinthemachine learningcommunity)ofusingthetermregressiononlyformodelsinwhichthe outputvariableiscontinuous. Therefore,weusethetermclassificationherebecauseinalogitmodelthe outputisdiscrete. Sample Supposethatweobserveasampleofdata for . Eachobservationhas: anoutputvariabledenotedby ; a vectorofinputs,denotedby . Conditionalprobabilities Theoutput cantakeonlytwovalues,either0or1(itisa Bernoullirandom variable). Theprobabilitythattheoutput isequalto1, conditional ontheinputs , isassumedto bewhere is thelogisticfunctionand isa vectorofcoefficients. Theprobabilitythat isequalto0 is Thelogisticfunction Itisimmediatetoseethatthelogisticfunction isalwayspositive. Furthermore,itisincreasingand so thatit satisfies Thus, isawell-definedprobabilitybecauseitliesbetween0and1. Explanation Whyisthelogisticclassificationmodelspecifiedinthismanner? Whyisthelogisticfunctionusedtotransformthelinearcombinationof inputs ? Thesimpleansweristhatwewouldliketodosomethingsimilartowhatwedo inalinear regressionmodel:usealinearcombinationoftheinputsasourprediction oftheoutput. However,ourpredictionneedstobeaprobabilityandthereisnoguarantee thatthelinearcombination isbetween0and1. Thus,weusethelogisticfunctionbecauseitprovidesaconvenientwayof transforming andforcingittolieintheintervalbetween0and1. Alternatives Wecouldhaveusedotherfunctionsthatenjoypropertiessimilartothe logisticfunction. Asamatteroffact,otherpopularclassificationmodelscanbeobtainedby simplysubstitutingthelogisticfunctionwithanotherfunctionandleaving everythingelseinthemodelunchanged. Forexample,bysubstitutingthelogitfunctionwiththecumulative distributionfunctionofastandardnormaldistribution,weobtainthe so-called probit model. Thelogitmodelasalatentvariablemodel Anotherwayofthinkingaboutthelogitmodelistodefinealatentvariable (i.e.,anunobserved variable)where isarandomerrortermthataddsnoisetotherelationshipbetweentheinputs andthevariable . Thelatentvariable isthenassumedtodeterminetheoutput as follows: Fromtheseassumptionsandtheadditionalassumptionthat hasasymmetricdistributionaround , itfollows thatwhere isthecumulativedistribution functionoftheerror . Itturnsoutthatthelogisticfunctionusedtodefinethelogitmodelisthe cumulativedistributionfunctionofasymmetricprobabilitydistribution calledstandardlogisticdistribution. Therefore,thelogitmodelcanbewrittenasalatentvariablemodel, specifiedbyequations(1)and(2)above,inwhichtheerror hasalogisticdistribution. Bychoosingdifferentdistributionsfortheerror , weobtainotherbinaryclassificationmodels. Forexample,ifweassumethat hasastandardnormaldistribution,thenweobtaintheprobitmodel. Estimationbymaximumlikelihood Thevectorofcoefficients isoftenestimatedby maximum likelihoodmethods. Assumethattheobservations inthesampleareIIDanddenotethe vectorofalloutputsby andthe matrixofallinputsby . Thelatterisassumedtohavefull rank. Itispossibletoprove(seethelectureon Maximum likelihoodestimationofthelogitmodel)thatthemaximumlikelihood estimator (whenitexists)canbeobtainedbyperformingsimple Newton-Raphson iterationsasfollows: startfromaguess (e.g., ); recursivelyupdatethe guess:where:and isan diagonalmatrix(i.e.,havingalloff-diagonalentriesequalto ) suchthattheelementsonitsdiagonalare stopwhennumericalconvergenceisachieved,thatis,whenthedifference between and issosmallastobenegligible; setthemaximumlikelihoodestimator equaltothelastupdate (denotethelastiterationby ). Theasymptoticcovariancematrixofthemaximumlikelihoodestimator canbeconsistentlyestimatedby so thatthedistributionoftheestimator isapproximatelynormalwithmeanequalto andcovariance matrix . Hypothesistesting Ifthelogitmodelisestimatedwiththemaximumlikelihoodprocedure illustratedabove,anyoneoftheclassical tests basedonmaximumlikelihoodprocedures(e.g., Wald, Likelihood Ratio,Lagrange Multiplier)canbeusedto testan hypothesisaboutthevectorofcoefficients . Othertestscanbeconstructedbyexploitingtheasymptoticnormalityofthe maximumlikelihoodestimator. Forexample,wecanperformaztesttotestthe nullhypothesis where isthe -th entryofthevectorofcoefficients and . Theteststatistic iswhere isthe -th entryof and isthe -th entryonthediagonalofthematrix . Asthesamplesize increases, convergesindistributiontoa standardnormal distribution.Thelatterdistributioncanbeusedto derivecriticalvaluesandperformthe test. Proof We haveBy theasymptoticnormalityofthemaximumlikelihoodestimator,thenumerator convergesin distributiontoanormalrandomvariablewithmean . Furthermore,theconsistencyofourestimatoroftheasymptoticcovariance matriximplies thatwhere denotesconvergence inprobability.Bythe ContinuousMapping theorem, and, bySlutsky'stheorem, convergesindistributiontoastandardnormalrandomvariable. Howtocite Pleaseciteas: Taboga,Marco(2021)."Logisticclassificationmodel(logitorlogisticregression)",Lecturesonprobabilitytheoryandmathematicalstatistics.KindleDirectPublishing.Onlineappendix.https://www.statlect.com/fundamentals-of-statistics/logistic-classification-model. Thebooks Mostofthelearningmaterialsfoundonthiswebsitearenowavailableinatraditionaltextbookformat. ProbabilityandstatisticsMatrixalgebra Featuredpages Hypothesistesting Meansquareconvergence Bernoullidistribution Gammadistribution Likelihoodratiotest Pointestimation Explore Gammafunction Multivariatenormaldistribution Waldtest Mainsections Mathematicaltools Fundamentalsofprobability Probabilitydistributions Asymptotictheory Fundamentalsofstatistics Glossary About AboutStatlect Contacts Cookies,privacyandtermsofuse Glossaryentries Continuousmappingtheorem Nullhypothesis TypeIIerror Meansquarederror Distributionfunction Probabilitydensityfunction Share Toenhanceyourprivacy, weremovedthesocialbuttons, butdon'tforgettoshare.