HadoopFairSchedulerDesignDocumentAugust15,2009Contents1IntroductionTheHadoopFairSchedulerstartedasasimplemeanstoshareMapReduceclusters.Overtime,ithasgrowninfunctionalitytosupporthierarchicalscheduling,preemption,andmultiplewaysoforganizingandweighingjobs.ThisdocumentexplainsthegoalsandfeaturesoftheFairScheduleranditsinternaldesign.2FairSchedulerGoalsTheFairSchedulerwasdesignedwithfourmaingoals:1.Runsmalljobsquicklyeveniftheyaresharingaclusterwithlargejobs.UnlikeHadoop’sbuilt-inFIFOscheduler,fairschedulingletssmalljobsmakeprogressevenifalargejobisrunning,withoutstarvingthelargejob.2.Provideguaranteedservicelevelsto“production”jobs,toletthemrunalongsideexperimentaljobsinasharedcluster.3.Besimpletoadministerandconfigure.Theschedulershoulddosomethingreasonable“outofthebox,”andusersshouldonlyneedtoconfigureitastheydiscoverthattheywanttousemoreadvancedfeatures.4.Supportreconfigurationatruntime,withoutrequiringaclusterrestart.3SchedulerFeaturesThissectionprovidesaquickoverviewofthefeaturesoftheFairScheduler.Adetailedus-ageguideisavailableintheHadoopdocumentationinbuild/docs/fairscheduler.html.13.1PoolsTheFairSchedulergroupsjobsinto“pools”andperformsfairsharingbetweenthesepools.EachpoolcanuseeitherFIFOorfairsharingtoschedulejobsinternaltothepool.ThepoolthatajobisplacedinisdeterminedbyaJobConfproperty,the“poolnameproperty”.Bydefault,thisisuser.name,sothatthereisonepoolperuser.However,differentpropertiescanbeused,e.g.group.nametohaveonepoolperUnixgroup.Acommontrickistosetthepoolnamepropertytoanunusedpropertynamesuchaspool.nameandmakethisdefaulttouser.name,sothatthereisonepoolperuserbutitisalsopossibletoplacejobsinto“special”poolsbysettingtheirpool.namedirectly.Themapred-site.xmlsnippetbelowshowshowtodothis:<property><name>mapred.fairscheduler.poolnameproperty</name><value>pool.name</value></property><property><name>pool.name</name><value>${user.name}</value></property>3.2MinimumSharesNormally,activepools(thosethatcontain