±¾ÎÄתÔØ×Ô΢ÐŹ«Öںš¸Java´óÊý¾ÝÓëÊý¾Ý²Ö¿â¡¹£¬×÷ÕßÀ϶¡£×ªÔر¾ÎÄÇëÁªÏµJava´óÊý¾ÝÓëÊý¾Ý²Ö¿â¹«Öںš£
Ŀ¼
1. Yarnµ÷¶ÈÆ÷½éÉÜ
2.Yarnµ÷¶ÈÆ÷ÅäÖÃ
ÀíÏëÇé¿öÏ£¬ÎÒÃÇÓ¦ÓöÔYarn×ÊÔ´µÄÇëÇóÓ¦¸ÃÁ¢¿ÌµÃµ½Âú×㣬µ«ÏÖʵÇé¿ö×ÊÔ´ÍùÍùÊÇÓÐÏ޵ģ¬ÌرðÊÇÔÚÒ»¸öºÜ·±Ã¦µÄ¼¯Èº£¬Ò»¸öÓ¦ÓÃ×ÊÔ´µÄÇëÇó¾³£ÐèÒªµÈ´ýÒ»¶Îʱ¼ä²ÅÄܵĵ½ÏàÓ¦µÄ×ÊÔ´¡£ÔÚYarnÖУ¬¸ºÔð¸øÓ¦Ó÷ÖÅä×ÊÔ´µÄ¾ÍÊÇScheduler¡£Æäʵµ÷¶È±¾Éí¾ÍÊÇÒ»¸öÄÑÌ⣬ºÜÄÑÕÒµ½Ò»¸öÍêÃÀµÄ²ßÂÔ¿ÉÒÔ½â¾öËùÓеÄÓ¦Óó¡¾°¡£Îª´Ë£¬YarnÌṩÁ˶àÖÖµ÷¶ÈÆ÷ºÍ¿ÉÅäÖõIJßÂÔ¹©ÎÒÃÇÑ¡Ôñ¡£YARN¼Ü¹¹ÈçÏÂ:
1. Yarnµ÷¶ÈÆ÷½éÉÜ
1.1. FIFO (ÏȽøÏȳöµ÷¶ÈÆ÷)
FIFO Scheduler°ÑÓ¦Óð´Ìá½»µÄ˳ÐòÅųÉÒ»¸ö¶ÓÁУ¬ÕâÊÇÒ»¸öÏȽøÏȳö¶ÓÁУ¬ÔÚ½øÐÐ×ÊÔ´·ÖÅäµÄʱºò£¬Ïȸø¶ÓÁÐÖÐ×îÍ·ÉϵÄÓ¦ÓýøÐзÖÅä×ÊÔ´£¬´ý×îÍ·ÉϵÄÓ¦ÓÃÐèÇóÂú×ãºóÔÙ¸øÏÂÒ»¸ö·ÖÅ䣬ÒÔ´ËÀàÍÆ¡£FIFO SchedulerÊÇ×î¼òµ¥Ò²ÊÇ×îÈÝÒ×Àí½âµÄµ÷¶ÈÆ÷£¬Ò²²»ÐèÒªÈκÎÅäÖ㬵«Ëü²¢²»ÊÊÓÃÓÚ¹²Ïí¼¯Èº¡£´óµÄÓ¦ÓÿÉÄÜ»áÕ¼ÓÃËùÓм¯Èº×ÊÔ´£¬Õâ¾Íµ¼ÖÂÆäËüÓ¦Óñ»×èÈû¡£ÔÚ¹²Ïí¼¯ÈºÖУ¬¸üÊʺϲÉÓÃCapacity Scheduler»òFair Scheduler£¬ÕâÁ½¸öµ÷¶ÈÆ÷¶¼ÔÊÐí´óÈÎÎñºÍСÈÎÎñÔÚÌá½»µÄͬʱ»ñµÃÒ»¶¨µÄϵͳ×ÊÔ´¡£ÏÂÃæ“Yarnµ÷¶ÈÆ÷¶Ô±Èͼ”չʾÁËÕ⼸¸öµ÷¶ÈÆ÷µÄÇø±ð£¬´ÓͼÖпÉÒÔ¿´³ö£¬ÔÚFIFO µ÷¶ÈÆ÷ÖУ¬Ð¡ÈÎÎñ»á±»´óÈÎÎñ×èÈû¡£
1.2. Capacity (ÈÝÁ¿µ÷¶ÈÆ÷)
yarn-site.xmlÖÐĬÈÏÅäÖõÄ×ÊÔ´µ÷¶ÈÆ÷¡£¶ø¶ÔÓÚCapacityµ÷¶ÈÆ÷£¬ÓÐÒ»¸öרÃŵĶÓÁÐÓÃÀ´ÔËÐÐСÈÎÎñ£¬µ«ÊÇΪСÈÎÎñרÃÅÉèÖÃÒ»¸ö¶ÓÁлáÔ¤ÏÈÕ¼ÓÃÒ»¶¨µÄ¼¯Èº×ÊÔ´£¬Õâ¾Íµ¼Ö´óÈÎÎñµÄÖ´ÐÐʱ¼ä»áÂäºóÓÚʹÓÃFIFOµ÷¶ÈÆ÷ʱµÄʱ¼ä¡£ÓÃÕâ¸ö×ÊÔ´µ÷¶ÈÆ÷£¬¾Í¿ÉÒÔÅäÖÃyarn×ÊÔ´¶ÓÁУ¬Õâ¸öºóÃæºó½éÉÜÓõ½¡£
1.3. Fair (¹«Æ½µ÷¶ÈÆ÷)
Fairµ÷¶ÈÆ÷µÄÉè¼ÆÄ¿±êÊÇΪËùÓеÄÓ¦Ó÷ÖÅ乫ƽµÄ×ÊÔ´(¶Ô¹«Æ½µÄ¶¨Òå¿ÉÒÔͨ¹ý²ÎÊýÀ´ÉèÖÃ)¡£ÔÚÉÏÃæµÄ“Yarnµ÷¶ÈÆ÷¶Ô±Èͼ”չʾÁËÒ»¸ö¶ÓÁÐÖÐÁ½¸öÓ¦ÓõĹ«Æ½µ÷¶È;µ±È»£¬¹«Æ½µ÷¶ÈÔÚÒ²¿ÉÒÔÔÚ¶à¸ö¶ÓÁм乤×÷¡£¾Ù¸öÀý×Ó£¬¼ÙÉèÓÐÁ½¸öÓû§AºÍB£¬ËûÃÇ·Ö±ðÓµÓÐÒ»¸ö¶ÓÁС£µ±AÆô¶¯Ò»¸öjob¶øBûÓÐÈÎÎñʱ£¬A»á»ñµÃÈ«²¿¼¯Èº×ÊÔ´;µ±BÆô¶¯Ò»¸öjobºó£¬AµÄjob»á¼ÌÐøÔËÐУ¬²»¹ýÒ»»á¶ùÖ®ºóÁ½¸öÈÎÎñ»á¸÷×Ô»ñµÃÒ»°ëµÄ¼¯Èº×ÊÔ´¡£Èç¹û´ËʱBÔÙÆô¶¯µÚ¶þ¸öjob²¢ÇÒÆäËüjob»¹ÔÚÔËÐУ¬ÔòËü½«»áºÍBµÄµÚÒ»¸öjob¹²ÏíBÕâ¸ö¶ÓÁеÄ×ÊÔ´£¬Ò²¾ÍÊÇBµÄÁ½¸öjob»áÓÃÓÚËÄ·ÖÖ®Ò»µÄ¼¯Èº×ÊÔ´£¬¶øAµÄjobÈÔÈ»ÓÃÓÚ¼¯ÈºÒ»°ëµÄ×ÊÔ´£¬½á¹û¾ÍÊÇ×ÊÔ´×îÖÕÔÚÁ½¸öÓû§Ö®¼äƽµÈµÄ¹²Ïí¡£ÔÚFairµ÷¶ÈÆ÷ÖУ¬ÎÒÃDz»ÐèÒªÔ¤ÏÈÕ¼ÓÃÒ»¶¨µÄϵͳ×ÊÔ´£¬Fairµ÷¶ÈÆ÷»áΪËùÓÐÔËÐеÄjob¶¯Ì¬µÄµ÷Õûϵͳ×ÊÔ´¡£µ±µÚÒ»¸ö´ójobÌύʱ£¬Ö»ÓÐÕâÒ»¸öjobÔÚÔËÐУ¬´ËʱËü»ñµÃÁËËùÓм¯Èº×ÊÔ´;µ±µÚ¶þ¸öСÈÎÎñÌá½»ºó£¬Fairµ÷¶ÈÆ÷»á·ÖÅäÒ»°ë×ÊÔ´¸øÕâ¸öСÈÎÎñ£¬ÈÃÕâÁ½¸öÈÎÎñ¹«Æ½µÄ¹²Ïí¼¯Èº×ÊÔ´¡£
a) ¹«Æ½µ÷¶ÈÆ÷£¬¾ÍÊÇÄܹ»¹²ÏíÕû¸ö¼¯ÈºµÄ×ÊÔ´
b) ²»ÓÃÔ¤ÏÈÕ¼ÓÃ×ÊÔ´£¬Ã¿Ò»¸ö×÷Òµ¶¼Êǹ²ÏíµÄ
c) ÿµ±Ìá½»Ò»¸ö×÷ÒµµÄʱºò£¬¾Í»áÕ¼ÓÃÕû¸ö×ÊÔ´¡£Èç¹ûÔÙÌá½»Ò»¸ö×÷Òµ£¬ÄÇôµÚÒ»¸ö×÷Òµ¾Í»á·Ö¸øµÚ¶þ¸ö×÷ÒµÒ»²¿·Ö×ÊÔ´£¬µÚÒ»¸ö×÷ÒµÒ²¾ÍÊÍ·ÅÒ»²¿·Ö×ÊÔ´¡£ÔÙÌá½»ÆäËûµÄ×÷ҵʱ£¬Ò²Í¬Àí¡£¡£¡£¡£Ò²¾ÍÊÇ˵ÿһ¸ö×÷Òµ½øÀ´£¬¶¼Óлú»á»ñÈ¡×ÊÔ´¡£
1.4. Fair SchedulerÓëCapacity SchedulerÇø±ð
2.Yarnµ÷¶ÈÆ÷ÅäÖÃ
yarn×ÊÔ´µ÷¶ÈÆ÷ÊÇÔÚyarn-site.xmlÖÐÅäÖá£
2.1. Fair Scheduler
Fair SchedulerµÄÅäÖÃÑ¡Ïî°üÀ¨Á½²¿·Ö:
Ò»²¿·ÖÔÚyarn-site.xmlÖУ¬Ö÷ÒªÓÃÓÚÅäÖõ÷¶ÈÆ÷¼¶±ðµÄ²ÎÊý
Ò»²¿·ÖÔÚÒ»¸ö×Ô¶¨ÒåÅäÖÃÎļþ(ĬÈÏÊÇfair-scheduler.xml)ÖУ¬Ö÷ÒªÓÃÓÚÅäÖø÷¸ö¶ÓÁеÄ×ÊÔ´Á¿¡¢È¨ÖصÈÐÅÏ¢¡£
2.1.1 yarn-site.xml
yarn-site.xml½éÉÜ
- <!– scheduler start –>
- <property>
- <name>yarn.resourcemanager.scheduler.class</name>
- <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
- <description>ÅäÖÃYarnʹÓõĵ÷¶ÈÆ÷²å¼þÀàÃû£»Fair Scheduler¶ÔÓ¦µÄÊÇ£ºorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</description>
- </property>
- <property>
- <name>yarn.scheduler.fair.allocation.file</name>
- <value>/etc/hadoop/conf/fair-scheduler.xml</value>
- <description>ÅäÖÃ×ÊÔ´³ØÒÔ¼°ÆäÊôÐÔÅä¶îµÄXMLÎļþ·¾¶£¨±¾µØ·¾¶£©</description>
- </property>
- <property>
- <name>yarn.scheduler.fair.preemption</name>
- <value>true</value>
- <description>¿ªÆô×ÊÔ´ÇÀÕ¼,default is True</description>
- </property>
- <property>
- <name>yarn.scheduler.fair.user-as-default-queue</name>
- <value>true</value>
- <description>ÉèÖóÉtrue£¬µ±ÈÎÎñÖÐδָ¶¨×ÊÔ´³ØµÄʱºò£¬½«ÒÔÓû§Ãû×÷Ϊ×ÊÔ´³ØÃû¡£Õâ¸öÅäÖþÍʵÏÖÁ˸ù¾ÝÓû§Ãû×Ô¶¯·ÖÅä×ÊÔ´³Ø¡£default is True</description>
- </property>
- <property>
- <name>yarn.scheduler.fair.allow-undeclared-pools</name>
- <value>false</value>
- <description>ÊÇ·ñÔÊÐí´´½¨Î´¶¨ÒåµÄ×ÊÔ´³Ø¡£Èç¹ûÉèÖóÉtrue£¬yarn½«»á×Ô¶¯´´½¨ÈÎÎñÖÐÖ¸¶¨µÄ䶨Òå¹ýµÄ×ÊÔ´³Ø¡£ÉèÖóÉfalseÖ®ºó£¬ÈÎÎñÖÐÖ¸¶¨µÄ䶨ÒåµÄ×ÊÔ´³Ø½«ÎÞЧ£¬¸ÃÈÎÎñ»á±»·ÖÅäµ½default×ÊÔ´³ØÖС£,default is True</description>
- </property>
- <!– scheduler end –>
2.1.2 fair-scheduler.xml
¼ÙÉèÔÚÉú²ú»·¾³YarnÖУ¬×ܹ²ÓÐËÄÀàÓû§ÐèҪʹÓü¯Èº£¬production¡¢spark¡¢default¡¢streaming¡£ÎªÁËʹÆäÌá½»µÄÈÎÎñ²»ÊÜÓ°Ï죬ÎÒÃÇÔÚYarnÉϹ滮ÅäÖÃÁËËĸö×ÊÔ´³Ø£¬·Ö±ðΪproduction,spark,default,streaming¡£²¢¸ù¾Ýʵ¼ÊÒµÎñÇé¿ö£¬ÎªÃ¿¸ö×ÊÔ´³Ø·ÖÅäÁËÏàÓ¦µÄ×ÊÔ´¼°ÓÅÏȼ¶µÈ,defaultÓÃÓÚ¿ª·¢²âÊÔÄ¿µÄ.
ResourceManagerÉÏfair-scheduler.xmlÅäÖÃÈçÏ£º
- <?xml version="1.0"?>
- <allocations>
- <queue name="root">
- <aclSubmitApps></aclSubmitApps>
- <aclAdministerApps></aclAdministerApps>
- <queue name="production">
- <minResources>8192mb,8vcores</minResources>
- <maxResources>419840mb,125vcores</maxResources>
- <maxRunningApps>60</maxRunningApps>
- <schedulingMode>fair</schedulingMode>
- <weight>7.5</weight>
- <aclSubmitApps>*</aclSubmitApps>
- <aclAdministerApps>production</aclAdministerApps>
- </queue>
- <queue name="spark">
- <minResources>8192mb,8vcores</minResources>
- <maxResources>376480mb,110vcores</maxResources>
- <maxRunningApps>50</maxRunningApps>
- <schedulingMode>fair</schedulingMode>
- <weight>1</weight>
- <aclSubmitApps>*</aclSubmitApps>
- <aclAdministerApps>spark</aclAdministerApps>
- </queue>
- <queue name="default">
- <minResources>8192mb,8vcores</minResources>
- <maxResources>202400mb,20vcores</maxResources>
- <maxRunningApps>20</maxRunningApps>
- <schedulingMode>FIFO</schedulingMode>
- <weight>0.5</weight>
- <aclSubmitApps>*</aclSubmitApps>
- <aclAdministerApps>*</aclAdministerApps>
- </queue>
- <queue name="streaming">
- <minResources>8192mb,8vcores</minResources>
- <maxResources>69120mb,16vcores</maxResources>
- <maxRunningApps>20</maxRunningApps>
- <schedulingMode>fair</schedulingMode>
- <aclSubmitApps>*</aclSubmitApps>
- <weight>1</weight>
- <aclAdministerApps>streaming</aclAdministerApps>
- </queue>
- </queue>
- <user name="production">
- <!-- ¶ÔÓÚÌض¨Óû§µÄÅäÖÃ:production×î¶à¿ÉÒÔͬʱÔËÐеÄÈÎÎñ -->
- <maxRunningApps>100</maxRunningApps>
- </user>
- <user name="default">
- <!-- ¶ÔÓÚĬÈÏÓû§ÅäÖÃ×î¶à¿ÉÒÔͬʱÔËÐеÄÈÎÎñ -->
- <maxRunningApps>10</maxRunningApps>
- </user>
- <!-- users max running apps -->
- <userMaxAppsDefault>50</userMaxAppsDefault>
- <!--ĬÈϵÄÓû§×î¶à¿ÉÒÔͬʱÔËÐеÄÈÎÎñ -->
- <queuePlacementPolicy>
- <rule name="specified"/>
- <rule name="primaryGroup" create="false" />
- <rule name="secondaryGroupExistingQueue" create="false" />
- <rule name="default" queue="default"/>
- </queuePlacementPolicy>
- </allocations>
²ÎÊý½éÉÜ:
ÕâÑù£¬Ã¿¸öÓû§×éϵÄÓû§Ìá½»ÈÎÎñʱºò£¬»áµ½ÏàÓ¦µÄ×ÊÔ´³ØÖУ¬¶ø²»Ó°ÏìÆäËûÒµÎñ¡£¶ÓÁеIJã´ÎÊÇͨ¹ýǶÌ×
ÔªËØʵÏֵġ£ËùÓеĶÓÁж¼ÊÇroot¶ÓÁеĺ¢×Ó£¬¼´Ê¹Ã»ÓÐÅäµ½ÔªËØÀï¡£Fairµ÷¶ÈÆ÷ÖеĶÓÁÐÓÐÒ»¸öȨÖØÊôÐÔ(Õâ¸öȨÖؾÍÊǶԹ«Æ½µÄ¶¨Òå)£¬²¢°ÑÕâ¸öÊôÐÔ×÷Ϊ¹«Æ½µ÷¶ÈµÄÒÀ¾Ý¡£ÔÚÕâ¸öÀý×ÓÖУ¬µ±µ÷¶ÈÆ÷·ÖÅ伯Ⱥ7.5,1,1,0.5×ÊÔ´¸øproduction,spark,streaming,defaultʱ±ãÊÓ×÷¹«Æ½,ÕâÀïµÄȨÖز¢²»Êǰٷֱȡ£×¢Ò⣬¶ÔÓÚÔÚûÓÐÅäÖÃÎļþʱ°´Óû§×Ô¶¯´´½¨µÄ¶ÓÁУ¬ËüÃÇÈÔÓÐȨÖز¢ÇÒȨÖØֵΪ1¡£Ã¿¸ö¶ÓÁÐÄÚ²¿ÈÔ¿ÉÒÔÓв»Í¬µÄµ÷¶È²ßÂÔ¡£¶ÓÁеÄĬÈϵ÷¶È²ßÂÔ¿ÉÒÔͨ¹ý¶¥¼¶ÔªËؽøÐÐÅäÖã¬Èç¹ûûÓÐÅäÖã¬Ä¬ÈϲÉÓù«Æ½µ÷¶È¡£¾¡¹ÜÊÇFairµ÷¶ÈÆ÷£¬ÆäÈÔÖ§³ÖÔÚ¶ÓÁ춱ð½øÐÐFIFOµ÷¶È¡£Ã¿¸ö¶ÓÁеĵ÷¶È²ßÂÔ¿ÉÒÔ±»ÆäÄÚ²¿µÄÔªËظ²¸Ç£¬ÔÚÉÏÃæÕâ¸öÀý×ÓÖУ¬default¶ÓÁоͱ»Ö¸¶¨²ÉÓÃfifo½øÐе÷¶È£¬ËùÒÔ£¬¶ÔÓÚÌá½»µ½default¶ÓÁеÄÈÎÎñ¾Í¿ÉÒÔ°´ÕÕFIFO¹æÔò˳ÐòµÄÖ´ÐÐÁË¡£ÐèҪעÒ⣬spark,production,streaming,defaultÖ®¼äµÄµ÷¶ÈÈÔÈ»Êǹ«Æ½µ÷¶È¡£Ã¿¸ö¶ÓÁпÉÅäÖÃ×î´ó¡¢×îС×ÊÔ´Õ¼ÓÃÊýºÍ×î´ó¿ÉÔËÐеÄÓ¦ÓõÄÊýÁ¿¡£
Fairµ÷¶ÈÆ÷²ÉÓÃÁËÒ»Ì×»ùÓÚ¹æÔòµÄϵͳÀ´È·¶¨Ó¦ÓÃÓ¦¸Ã·Åµ½Äĸö¶ÓÁС£ÔÚÉÏÃæµÄÀý×ÓÖУ¬ÔªËض¨ÒåÁËÒ»¸ö¹æÔòÁÐ±í£¬ÆäÖеÄÿ¸ö¹æÔò»á±»Öð¸ö³¢ÊÔÖ±µ½Æ¥Åä³É¹¦¡£ÀýÈ磬ÉÏÀýµÚÒ»¸ö¹æÔòspecified£¬Ôò»á°ÑÓ¦Ó÷ŵ½ËüÖ¸¶¨µÄ¶ÓÁÐÖУ¬ÈôÕâ¸öÓ¦ÓÃûÓÐÖ¸¶¨¶ÓÁÐÃû»ò¶ÓÁÐÃû²»´æÔÚ£¬Ôò˵Ã÷²»Æ¥ÅäÕâ¸ö¹æÔò£¬È»ºó³¢ÊÔÏÂÒ»¸ö¹æÔò¡£primaryGroup¹æÔò»á³¢ÊÔ°ÑÓ¦Ó÷ÅÔÚÒÔÓû§ËùÔÚµÄUnix×éÃûÃüÃûµÄ¶ÓÁÐÖУ¬Èç¹ûûÓÐÕâ¸ö¶ÓÁУ¬²»´´½¨¶ÓÁÐת¶ø³¢ÊÔÏÂÒ»¸ö¹æÔò¡£µ±Ç°ÃæËùÓйæÔò²»Âú×ãʱ£¬Ôò´¥·¢default¹æÔò£¬°ÑÓ¦Ó÷ÅÔÚdefault¶ÓÁÐÖС£
µ±È»£¬ÎÒÃÇ¿ÉÒÔ²»ÅäÖÃqueuePlacementPolicy¹æÔò£¬µ÷¶ÈÆ÷ÔòĬÈϲÉÓÃÈçϹæÔò£º
- <queuePlacementPolicy>
- <rule name="specified" />
- <rule name="user" />
- </queuePlacementPolicy>
ÉÏÃæ¹æÔòÒâ˼Êdzý·Ç¶ÓÁб»×¼È·µÄ¶¨Ò壬·ñÔò»áÒÔÓû§ÃûΪ¶ÓÁÐÃû´´½¨¶ÓÁС£»¹ÓÐÒ»¸ö¼òµ¥µÄÅäÖòßÂÔ¿ÉÒÔʹµÃËùÓеÄÓ¦Ó÷ÅÈëͬһ¸ö¶ÓÁÐ(default)£¬ÕâÑù¾Í¿ÉÒÔÈÃËùÓÐÓ¦ÓÃÖ®¼äƽµÈ¹²Ïí¼¯Èº¶ø²»ÊÇÔÚÓû§Ö®¼ä¡£Õâ¸öÅäÖõĶ¨ÒåÈçÏ£º
- <queuePlacementPolicy>
- <rule name="default" />
- </queuePlacementPolicy>
ʵÏÖÉÏÃ湦ÄÜÎÒÃÇ»¹¿ÉÒÔ²»Ê¹ÓÃÅäÖÃÎļþ£¬Ö±½ÓÉèÖÃyarn.scheduler.fair.user-as-default-queue=false£¬ÕâÑùÓ¦Óñã»á±»·ÅÈëdefault ¶ÓÁУ¬¶ø²»ÊǸ÷¸öÓû§Ãû¶ÓÁС£ÁíÍ⣬ÎÒÃÇ»¹¿ÉÒÔÉèÖÃyarn.scheduler.fair.allow-undeclared-pools=false£¬ÕâÑùÓû§¾ÍÎÞ·¨´´½¨¶ÓÁÐÁË¡£
µ±Ò»¸öjobÌá½»µ½Ò»¸ö·±Ã¦¼¯ÈºÖеĿնÓÁÐʱ£¬job²¢²»»áÂíÉÏÖ´ÐУ¬¶øÊÇ×èÈûÖ±µ½ÕýÔÚÔËÐеÄjobÊÍ·Åϵͳ×ÊÔ´¡£ÎªÁËʹÌá½»jobµÄÖ´ÐÐʱ¼ä¸ü¾ßÔ¤²âÐÔ(¿ÉÒÔÉèÖõȴýµÄ³¬Ê±Ê±¼ä)£¬Fairµ÷¶ÈÆ÷Ö§³ÖÇÀÕ¼¡£ÇÀÕ¼¾ÍÊÇÔÊÐíµ÷¶ÈÆ÷ɱµôÕ¼Óó¬¹ýÆäÓ¦Õ¼·Ý¶î×ÊÔ´¶ÓÁеÄcontainers£¬ÕâЩcontainers×ÊÔ´±ã¿É±»·ÖÅäµ½Ó¦¸ÃÏíÓÐÕâЩ·Ý¶î×ÊÔ´µÄ¶ÓÁÐÖС£ÐèҪעÒâÇÀÕ¼»á½µµÍ¼¯ÈºµÄÖ´ÐÐЧÂÊ£¬ÒòΪ±»ÖÕÖ¹µÄcontainersÐèÒª±»ÖØÐÂÖ´ÐС£¿ÉÒÔͨ¹ýÉèÖÃÒ»¸öÈ«¾ÖµÄ²ÎÊýyarn.scheduler.fair.preemption=trueÀ´ÆôÓÃÇÀÕ¼¹¦ÄÜ¡£´ËÍ⣬»¹ÓÐÁ½¸ö²ÎÊýÓÃÀ´¿ØÖÆÇÀÕ¼µÄ¹ýÆÚʱ¼ä(ÕâÁ½¸ö²ÎÊýĬÈÏûÓÐÅäÖã¬ÐèÒªÖÁÉÙÅäÖÃÒ»¸öÀ´ÔÊÐíÇÀÕ¼Container)£º
- minSharePreemptionTimeout
- fairSharePreemptionTimeout
Èç¹û¶ÓÁÐÔÚminimum share preemption timeoutÖ¸¶¨µÄʱ¼äÄÚδ»ñµÃ×îСµÄ×ÊÔ´±£ÕÏ£¬µ÷¶ÈÆ÷¾Í»áÇÀÕ¼containers¡£ÎÒÃÇ¿ÉÒÔͨ¹ýÅäÖÃÎļþÖеĶ¥¼¶ÔªËØ</defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout></defaultminsharepreemptiontimeout>ΪËùÓжÓÁÐÅäÖÃÕâ¸ö³¬Ê±Ê±¼ä;ÎÒÃÇ»¹¿ÉÒÔÔÚÔªËØÄÚÅäÖÃÔªËØÀ´ÎªÄ³¸ö¶ÓÁÐÖ¸¶¨³¬Ê±Ê±¼ä¡£</defaultminsharepreemptiontimeout>
ÓëÖ®ÀàËÆ£¬Èç¹û¶ÓÁÐÔÚfair share preemption timeoutÖ¸¶¨Ê±¼äÄÚδ»ñµÃƽµÈµÄ×ÊÔ´µÄÒ»°ë£¨Õâ¸ö±ÈÀý¿ÉÒÔÅäÖã©£¬µ÷¶ÈÆ÷Ôò»á½øÐÐÇÀÕ¼containers¡£Õâ¸ö³¬Ê±Ê±¼ä¿ÉÒÔͨ¹ý¶¥¼¶ÔªËØ<defaultfairsharepreemptiontimeout style="font-size: inherit;color: inherit;line-height: inherit;">ºÍÔªËؼ¶ÔªËØ·Ö±ðÅäÖÃËùÓжÓÁкÍij¸ö¶ÓÁеij¬Ê±Ê±¼ä¡£ÉÏÃæÌáµ½µÄ±ÈÀý¿ÉÒÔͨ¹ý<defaultfairsharepreemptionthreshold style="font-size: inherit;color: inherit;line-height: inherit;">(ÅäÖÃËùÓжÓÁÐ)ºÍ(ÅäÖÃij¸ö¶ÓÁÐ)½øÐÐÅäÖã¬Ä¬ÈÏÊÇ0.5¡£</defaultfairsharepreemptionthreshold></defaultfairsharepreemptiontimeout>
ÐèҪעÒâµÄÊÇ£¬ËùÓпͻ§¶ËÌá½»ÈÎÎñµÄÓû§ºÍÓû§×éµÄ¶ÔÓ¦¹Øϵ£¬ÐèҪά»¤ÔÚResourceManagerÉÏ£¬ResourceManagerÔÚ·ÖÅä×ÊÔ´³Øʱºò£¬ÊÇ´ÓResourceManagerÉ϶ÁÈ¡Óû§ºÍÓû§×éµÄ¶ÔÓ¦¹ØϵµÄ£¬·ñÔò¾Í»á±»·ÖÅäµ½default×ÊÔ´³Ø¡£ÔÚÈÕÖ¾ÖгöÏÖ”UserGroupInformation: No groups available for user”ÀàËƵľ¯¸æ¡£¶ø¿Í»§¶Ë»úÆ÷ÉϵÄÓû§¶ÔÓ¦µÄÓû§×éÎ޹ؽôÒª¡£
ÿ´ÎÔÚResourceManagerÉÏÐÂÔöÓû§»òÕßµ÷Õû×ÊÔ´³ØÅä¶îºó£¬ÐèÒªÖ´ÐÐÏÂÃæµÄÃüÁîË¢ÐÂʹÆäÉúЧ.
yarn rmadmin -refreshQueues yarn rmadmin -refreshUserToGroupsMappings
¶¯Ì¬¸üÐÂÖ»Ö§³ÖÐÞ¸Ä×ÊÔ´³ØÅä¶î£¬Èç¹ûÊÇÐÂÔö»ò¼õÉÙ×ÊÔ´³Ø£¬ÔòÐèÒªÖØÆôYarn¼¯Èº.
Fair Scheduer¸÷×ÊÔ´³ØÅäÖü°Ê¹ÓÃÇé¿ö£¬ÔÚResourceManagerµÄWEB¼à¿ØÒ³ÃæÉÏÒ²¿ÉÒÔ¿´µ½: http://ResourceManagerHost:8088/cluster/scheduler
2.2 Capacity ÅäÖÃ
hadoop2.7ĬÈÏʹÓõÄÊÇCapacity SchedulerÈÝÁ¿µ÷¶ÈÆ÷
yarn-site.xml
- <property>
- <name>yarn.resourcemanager.scheduler.class</name>
- <value>org.apache.hadoop.yarn.server.resourcemanager.capacity.CapacityScheduler</value>
- </property>
Capacity µ÷¶ÈÆ÷ÔÊÐí¶à¸ö×éÖ¯¹²ÏíÕû¸ö¼¯Èº£¬Ã¿¸ö×éÖ¯¿ÉÒÔ»ñµÃ¼¯ÈºµÄÒ»²¿·Ö¼ÆËãÄÜÁ¦¡£Í¨¹ýΪÿ¸ö×éÖ¯·ÖÅäרÃŵĶÓÁУ¬È»ºóÔÙΪÿ¸ö¶ÓÁзÖÅäÒ»¶¨µÄ¼¯Èº×ÊÔ´£¬ÕâÑùÕû¸ö¼¯Èº¾Í¿ÉÒÔͨ¹ýÉèÖöà¸ö¶ÓÁеķ½Ê½¸ø¶à¸ö×éÖ¯Ìṩ·þÎñÁË¡£³ý´ËÖ®Í⣬¶ÓÁÐÄÚ²¿ÓÖ¿ÉÒÔ´¹Ö±»®·Ö£¬ÕâÑùÒ»¸ö×éÖ¯ÄÚ²¿µÄ¶à¸ö³ÉÔ±¾Í¿ÉÒÔ¹²ÏíÕâ¸ö¶ÓÁÐ×ÊÔ´ÁË£¬ÔÚÒ»¸ö¶ÓÁÐÄÚ²¿£¬×ÊÔ´µÄµ÷¶ÈÊDzÉÓõÄÊÇÏȽøÏȳö(FIFO)²ßÂÔ¡£
Ò»¸öjob¿ÉÄÜʹÓò»ÁËÕû¸ö¶ÓÁеÄ×ÊÔ´¡£È»¶øÈç¹ûÕâ¸ö¶ÓÁÐÖÐÔËÐжà¸öjob£¬Èç¹ûÕâ¸ö¶ÓÁеÄ×ÊÔ´¹»Óã¬ÄÇô¾Í·ÖÅä¸øÕâЩjob£¬Èç¹ûÕâ¸ö¶ÓÁеÄ×ÊÔ´²»¹»ÓÃÁËÄØ?ÆäʵCapacityµ÷¶ÈÆ÷ÈÔ¿ÉÄÜ·ÖÅä¶îÍâµÄ×ÊÔ´¸øÕâ¸ö¶ÓÁУ¬Õâ¾ÍÊÇ“µ¯ÐÔ¶ÓÁД(queue elasticity)µÄ¸ÅÄî¡£
ÔÚÕý³£µÄ²Ù×÷ÖУ¬Capacityµ÷¶ÈÆ÷²»»áÇ¿ÖÆÊÍ·ÅContainer£¬µ±Ò»¸ö¶ÓÁÐ×ÊÔ´²»¹»ÓÃʱ£¬Õâ¸ö¶ÓÁÐÖ»ÄÜ»ñµÃÆäËü¶ÓÁÐÊͷźóµÄContainer×ÊÔ´¡£µ±È»£¬ÎÒÃÇ¿ÉÒÔΪ¶ÓÁÐÉèÖÃÒ»¸ö×î´ó×ÊԴʹÓÃÁ¿£¬ÒÔÃâÕâ¸ö¶ÓÁйý¶àµÄÕ¼ÓÿÕÏÐ×ÊÔ´£¬µ¼ÖÂÆäËü¶ÓÁÐÎÞ·¨Ê¹ÓÃÕâЩ¿ÕÏÐ×ÊÔ´£¬Õâ¾ÍÊÇ”µ¯ÐÔ¶ÓÁДÐèҪȨºâµÄµØ·½¡£
¼ÙÉèÎÒÃÇÓÐÈçϲã´ÎµÄ¶ÓÁУº
- root
- ©À©¤©¤ prod
- ©¸©¤©¤ dev
- ©À©¤©¤ eng
- ©¸©¤©¤ science
ÏÂÃæÊÇÒ»¸ö¼òµ¥µÄCapacityµ÷¶ÈÆ÷µÄÅäÖÃÎļþ£¬ÎļþÃûΪcapacity-scheduler.xml¡£ÔÚÕâ¸öÅäÖÃÖУ¬ÔÚroot¶ÓÁÐÏÂÃ涨ÒåÁËÁ½¸ö×Ó¶ÓÁÐprodºÍdev£¬·Ö±ðÕ¼40%ºÍ60%µÄÈÝÁ¿¡£ÐèҪעÒ⣬һ¸ö¶ÓÁеÄÅäÖÃÊÇͨ¹ýÊôÐÔyarn.sheduler.capacity..Ö¸¶¨µÄ£¬´ú±íµÄÊǶÓÁеļ̳ÐÊ÷£¬Èçroot.prod¶ÓÁУ¬Ò»°ãÖ¸capacityºÍmaximum-capacity¡£
- <?xml version="1.0"?>
- <configuration>
- <property>
- <name>yarn.scheduler.capacity.root.queues(/&eae)
- <value>prod,dev</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.dev.queues</tta*e>
- <value>eng,science</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.prod.capacity</name>
- <value>40</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.dev.capacity</name>
- <value >60</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.dev.maximuin-capacity</name>
- <value>75</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.dev.eng.capacity</name>
- <value >50</value>
- </property>
- <property>
- <name>yarn.scheduler.capacity.root.dev.science.capacity</name>
- <value >50</value>
- </property>
- </configuration>
ÎÒÃÇ¿ÉÒÔ¿´µ½£¬dev¶ÓÁÐÓÖ±»·Ö³ÉÁËengºÍscienceÁ½¸öÏàͬÈÝÁ¿µÄ×Ó¶ÓÁС£devµÄmaximum-capacityÊôÐÔ±»ÉèÖóÉÁË75%£¬ËùÒÔ¼´Ê¹prod¶ÓÁÐÍêÈ«¿ÕÏÐdevÒ²²»»áÕ¼ÓÃÈ«²¿¼¯Èº×ÊÔ´£¬Ò²¾ÍÊÇ˵£¬prod¶ÓÁÐÈÔÓÐ25%µÄ¿ÉÓÃ×ÊÔ´ÓÃÀ´Ó¦¼±¡£ÎÒÃÇ×¢Òâµ½£¬engºÍscienceÁ½¸ö¶ÓÁÐûÓÐÉèÖÃmaximum-capacityÊôÐÔ£¬Ò²¾ÍÊÇ˵eng»òscience¶ÓÁÐÖеÄjob¿ÉÄÜ»áÓõ½Õû¸ödev¶ÓÁеÄËùÓÐ×ÊÔ´(×î¶àΪ¼¯ÈºµÄ75%)¡£¶øÀàËƵģ¬prodÓÉÓÚûÓÐÉèÖÃmaximum-capacityÊôÐÔ£¬ËüÓпÉÄÜ»áÕ¼Óü¯ÈºÈ«²¿×ÊÔ´¡£CapacityÈÝÆ÷³ýÁË¿ÉÒÔÅäÖöÓÁм°ÆäÈÝÁ¿Í⣬ÎÒÃÇ»¹¿ÉÒÔÅäÖÃÒ»¸öÓû§»òÓ¦ÓÿÉÒÔ·ÖÅäµÄ×î´ó×ÊÔ´ÊýÁ¿¡¢¿ÉÒÔͬʱÔËÐжàÉÙÓ¦ÓᢶÓÁеÄACLÈÏÖ¤µÈ¡£
¹ØÓÚ¶ÓÁеÄÉèÖã¬ÕâÈ¡¾öÓÚÎÒÃǾßÌåµÄÓ¦ÓᣱÈÈ磬ÔÚMapReduceÖУ¬ÎÒÃÇ¿ÉÒÔͨ¹ýmapreduce.job.queuenameÊôÐÔÖ¸¶¨ÒªÓõĶÓÁС£Èç¹û¶ÓÁв»´æÔÚ£¬ÎÒÃÇÔÚÌá½»ÈÎÎñʱ¾Í»áÊÕµ½´íÎó¡£Èç¹ûÎÒÃÇûÓж¨ÒåÈκζÓÁУ¬ËùÓеÄÓ¦Óý«»á·ÅÔÚÒ»¸ödefault¶ÓÁÐÖС£
×¢Ò⣺¶ÔÓÚCapacityµ÷¶ÈÆ÷£¬ÎÒÃǵĶÓÁÐÃû±ØÐëÊǶÓÁÐÊ÷ÖеÄ×îºóÒ»²¿·Ö£¬Èç¹ûÎÒÃÇʹÓöÓÁÐÊ÷Ôò²»»á±»Ê¶±ð¡£±ÈÈ磬ÔÚÉÏÃæÅäÖÃÖУ¬ÎÒÃÇʹÓÃprodºÍeng×÷Ϊ¶ÓÁÐÃûÊÇ¿ÉÒԵģ¬µ«ÊÇÈç¹ûÎÒÃÇÓÃroot.dev.eng»òÕßdev.engÊÇÎÞЧµÄ¡£
2.3 FIFO Scheduler
yarn-site.xmlÎļþ
- <property>
- <name>yarn.resourcemanager.scheduler.class</name>
- <value>org.apache.hadoop.yarn.server.resourcemanager.fifo.FifoScheduler</value>
- </property>
Ç°ÑÔ Ç×£¬ÏÔ´æÕ¨ÁË£¬ÄãµÄÏÔ¿¨¿ìðÑÌÁË£¡ torch.FatalError:cudaruntimeerror( 2 ...
Gopher China×÷Ϊ¹úÄÚ×îȨÍþºÍ×îʵÁ¦¸É»õµÄGo´ó»á£¬ÖÂÁ¦ÓÚΪ¹ã´óµÄGopherÌṩһ...
ΪÁË·ÀÖ¹2016Äê6Ô£8ÔÂÖÐÑ®´´½¨µÄ Windows ÔÆ·þÎñÆ÷¼«¶ËÇé¿öϳöÏÖ¶ÏÍøµÄÎÊÌ⣬...
Á½ÖÖ²¢Ðз¢Õ¹µÄÇ÷ÊÆÕýÔڸıäËùÓÐÐÐÒµµÄÐÅÏ¢¼¼Êõ¸ñ¾Ö:ÔƺÍÈ˹¤ÖÇÄÜ(AI)ºÍ»úÆ÷ѧϰ...
ÿ¸ö JavaScript ³ÌÐòÔ±¶¼±ØÐëÖªµÀ±Õ°üÊÇʲô¡£ÔÚ JavaScript ÃæÊÔÖУ¬ÄãºÜ¿ÉÄÜ...
¿ªË½·þ¿ÉÒÔÓà ÔÆ·þÎñÆ÷ Âð£¿ËäÈ»´Ó ÔÆ·þÎñÆ÷ µÄʹÓÃÐÔÄÜÉÏÀ´½²£¬¿ªÉè˽·þÊÇ¿ÉÒÔ...
TOPÔÆ 1ÔÂ22ÈÕѶ£¬¾ÝIDCÆÀÊöÍø(idcps.com)½ñÈÕ±¨µÀ£º¾Ýntldstats.com×îÐÂÊý¾ÝÏÔ...
´ó¼ÒºÃ£¬ÎÒÊdzÌÐòß÷£¬½ñÌìÎÒÃÇ·ÅËÉÏ£¬¿´Ò»Ð©ÀàÐÍת»»µÄ֪ʶµã£¡ ÖÚËùÖÜÖªC++¹Ø...
SASµÄÓÅÊÆ £¨1£©SAS±È½ÏÈÝÒ×ÈëÃÅ£¬Ð½Ë®¸£ÀûºÃ Ñ¡ÔñSAS²»¿É·ñÈÏÊÇËüµÄ¸ßн·Ç³£µÄ...
ÓÐ ÓòÃû ÔõôÉêÇë ÆóÒµÓÊÏä £¿ÉêÇëÆóÒµÓÊ֮ǰ£¬ÐèҪע²á¹ºÂòÒ»¸öÓòÃû£¬Èç¹ûÉêÇë...