¶ÔÓÚ¼ôÖ¦¼¼Êõ£¬ÄãÁ˽â¶àÉÙ£¿ÕâÀïÓÐÒ»·ÝÃؼ®£¬ÕûÀíÁË 2019 Äê¶ÈµÄ 6 ƪÂÛÎÄËùÌáµ½µÄ×îмôÖ¦·½·¨¡£
¼ôÖ¦ÊÇÒ»ÖÖ°ïÖúÉñ¾ÍøÂçʵÏÖ¹æÄ£¸üС¡¢Ð§Âʸü¸ßµÄÉî¶Èѧϰ·½·¨¡£ÕâÊÇÒ»ÖÖÄ£ÐÍÓÅ»¯¼¼Êõ£¬Ëüɾ³ýȨÖØÕÅÁ¿Öв»±ØÒªµÄÖµ£¬´Ó¶øʹµÃѹËõºóµÄÉñ¾ÍøÂçÔËÐÐËٶȸü¿ì£¬ÍøÂçѵÁ·¹ý³ÌÖеļÆËã³É±¾Ò²ÓÐËù½µµÍ¡£ÔÚ½«Ä£ÐͲ¿Êðµ½ÊÖ»úµÈ±ßÔµÉ豸ÉÏʱ£¬¼ôÖ¦µÄ×÷Óøü¼ÓÃ÷ÏÔÏÖ¡£
±¾Æª¾«Ñ¡ÁËÉñ¾ÍøÂç¼ôÖ¦ÁìÓòµÄһЩÑо¿ÂÛÎÄ£¬¹©´ó¼Òѧϰ²Î¿¼¡£
ÂÛÎÄ 1£ºPruning from Scratch (2019)
×÷ÕßÌá³öÁËÒ»ÖÖ´ÓÍ·¿ªÊ¼¼ôÖ¦µÄÍøÂç¼ôÖ¦Á÷³Ì¡£ËûÃÇÔÚ CIFAR10 ºÍ ImageNet Êý¾Ý¼¯É϶Զà¸öѹËõ·ÖÀàÄ£ÐͽøÐÐÁËÊÔÑ飬½á¹û±íÃ÷¸ÃÁ÷³Ì½µµÍÁËÕý³£¼ôÖ¦·½·¨µÄԤѵÁ·¿ªÏú£¬Í¬Ê±Ìá¸ßÁËÍøÂçµÄ׼ȷÂÊ¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1909.12579.pdf
ÏÂͼչʾÁË´«Í³¼ôÖ¦Á÷³ÌµÄÈý¸ö½×¶Î£ºÔ¤ÑµÁ·¡¢¼ôÖ¦ºÍ΢µ÷¡£
ÕâƪÂÛÎÄÌá³öµÄ¼ôÖ¦¼¼Êõ°üÀ¨¿É»ùÓÚËæ»ú³õʼ»¯È¨ÖØѧµÃµÄеļôÖ¦Á÷³Ì¡£Í¨µÀÖØÒªÐÔ£¨channel importance£©Ôò¿Éͨ¹ý¹ØÁª±êÁ¿ÃÅ¿Ø£¨scalar gate£©ÖµºÍÿ¸öÍøÂç²ãÀ´Ñ§µÃ¡£
ÓÅ»¯Í¨µÀÖØÒªÐÔ£¬¿ÉÔÚÏ¡ÊèÐÔÕýÔò»¯µÄÇé¿öÏÂÌá¸ßÄ£ÐÍÐÔÄÜ¡£Ôڴ˹ý³ÌÖУ¬Ëæ»úȨÖز¢Î´µÃµ½¸üС£È»ºó£¬»ùÓÚ¸ø¶¨×ÊÔ´Ô¼Êø£¬Ê¹Óöþ·ÖËÑË÷²ßÂÔÈ·¶¨¼ôÖ¦ºóÄ£Ð͵ÄͨµÀÊýÅäÖá£
ϱíչʾÁËÄ£ÐÍÔÚ²»Í¬Êý¾Ý¼¯ÉϵÄ׼ȷÂÊ£º
ÂÛÎÄ 2£ºAdversarial Neural Pruning (2019)
ÕâƪÂÛÎÄÖ÷Ҫ̽ÌÖÔÚÓöµ½¶Ô¿¹ÈŶ¯Ê±ÍøÂçÒþÌØÕ÷µÄʧÕæÎÊÌâ¡£¸ÃÂÛÎÄÌá³öµÄ·½·¨ÊÇ£ºÑ§Ï°±´Ò¶Ë¹¼ôÖ¦ÑÚÂ룬À´ÒÖÖƽϸ߼¶µÄʧÕæÌØÕ÷£¬´Ó¶ø×î´ó»¯ÆäÃæ¶Ô¶Ô¿¹ÈŶ¯µÄÎȽ¡ÐÔ¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1908.04355.pdf
×÷Õß¿¼ÂÇÁËÉî¶ÈÉñ¾ÍøÂçÖÐÒþÌØÕ÷µÄ´àÈõÐÔ¡£¸Ã·½·¨Ìá³ö¼ô³ý´àÈõµÄÌØÕ÷£¬Í¬Ê±±£ÁôÎȽ¡µÄÌØÕ÷¡£ÕâÒ»¹ý³Ì¿Éͨ¹ýÔÚ±´Ò¶Ë¹¿ò¼ÜÖжԿ¹µØѧϰ¼ôÖ¦ÑÚÂëÀ´Íê³É¡£
¶Ô¿¹Éñ¾¼ôÖ¦£¨Adversarial Neural Pruning£¬ANP£©½áºÏÁ˶Կ¹ÑµÁ·ºÍ±´Ò¶Ë¹¼ôÖ¦·½·¨¡£¸ÃÂÛÎÄÌá³öµÄÐÂÄ£Ðͼ°Æä»ùÏßÄ£ÐÍÊÇ£º
ϱíչʾÁËÄ£Ð͵ÄÐÔÄÜ£º
ÂÛÎÄ 3£ºRethinking the Value of Network Pruning (ICLR 2019)
ÕâƪÂÛÎÄÌá³öµÄÍøÂç¼ôÖ¦·½·¨·ÖΪÁ½À࣬Ŀ±ê¼ô֦ģÐ͵ļܹ¹ÓÉÈËÀà»ò¼ôÖ¦Ëã·¨À´¾ö¶¨¡£ÔÚʵÑéÖУ¬×÷Õ߶ԱÈÁË´ÓÍ·¿ªÊ¼ÑµÁ·¼ô֦ģÐͺͻùÓڼ̳ÐȨÖؽøÐÐ΢µ÷µÃµ½µÄ¼ô֦ģÐ͵Ľá¹û£¬¸Ã¶Ô±ÈÕë¶ÔÔ¤¶¨Òå·½·¨ºÍ×Ô¶¯»¯·½·¨¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1810.05270v2.pdf
ÏÂͼչʾÁËʹÓûùÓÚ L1 ·¶ÊýµÄÂ˲¨Æ÷¼ôÖ¦µÄÔ¤¶¨Òå½á¹¹»¯¼ôÖ¦ËùµÃµ½µÄ½á¹û¡£Ã¿Ò»²ã¶¼Ê¹ÓýÏСµÄ L1 ·¶Êý¼ôµôÒ»¶¨±ÈÀýµÄÂ˲¨Æ÷¡£¡¸Pruned Model¡¹ÁÐÊÇÓÃÓÚÅäÖÃÿ¸öÄ£Ð͵ÄÔ¤¶¨ÒåÄ¿±êÄ£ÐÍÁÐ±í¡£ÎÒÃÇ¿ÉÒÔ¿´µ½£¬Ã¿Ò»ÐÐÖУ¬´ÓÍ·¿ªÊ¼ÑµÁ·µÄÄ£ÐÍÐÔÄÜÖÁÉÙÓë΢µ÷Ä£ÐͳÖƽ¡£
ÈçϱíËùʾ£¬ThiNet Ì°À·µØ¼ôÈ¥Á˶ÔÏÂÒ»²ãµÄ¼¤»îÖµÓ°Ïì×îСµÄͨµÀ¡£
ϱíչʾÁË»ùÓڻعéµÄÌØÕ÷Öؽ¨·½·¨µÄ½á¹û¡£¸Ã·½·¨×îС»¯ÁËÏÂÒ»²ãµÄÌØÕ÷ͼÖؽ¨Îó²î£¬´Ó¶øʵÏÖ¶ÔͨµÀ¼ôÖ¦¡£¸ÃÓÅ»¯ÎÊÌâ¿ÉÒÔͨ¹ý LASSO »Ø¹é½â¾ö¡£
ÖÁÓÚ Network Slimming£¬ÔÚѵÁ·¹ý³ÌÖУ¬¶ÔÅú¹éÒ»»¯²ãÖеÄͨµÀ¼¶Ëõ·ÅÒò×ÓÊ©¼Ó L1 Ï¡ÊèÐÔ¡£Ö®ºó£¬ÀûÓýϵ͵ÄËõ·ÅÒò×Ó¶ÔͨµÀ¼ôÖ¦¡£ÓÉÓÚͨµÀËõ·ÅÒò×Ó¾¹ý¿ç²ã¶Ô±È£¬Òò´Ë¸Ã·½·¨Äܹ»µÃµ½×Ô¶¯·¢ÏÖµÄÄ¿±ê¼Ü¹¹¡£
ÂÛÎÄ 4£ºNetwork Pruning via Transformable Architecture Search (NeurIPS 2019)
ÕâƪÂÛÎÄÌá³öÁËÖ±½Ó¶Ô¾ß±¸Áé»îͨµÀÊýºÍ²ãÊýµÄÍøÂçÓ¦ÓÃÉñ¾¼Ü¹¹ËÑË÷¡£ÊµÏÖ¼ôÖ¦ÍøÂçµÄËðʧ×îС»¯ÓÐÀûÓÚѧϰͨµÀÊý¡£¼ôÖ¦ÍøÂçµÄÌØÕ÷ͼÓÉ»ùÓÚ¸ÅÂÊ·Ö²¼²ÉÑùµÄ K ¸öÌØÕ÷ͼƬ¶Î×é³É£¬Í¨¹ý·´Ïò´«²¥½«Ëðʧ´«Êäµ½ÍøÂçȨÖغͲÎÊý»¯·Ö²¼¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1905.09717v5.pdf
¼ôÖ¦ÍøÂçµÄ¿í¶ÈºÍÉî¶ÈÊÇ»ùÓÚÿ¸ö·Ö²¼¹æÄ£µÄ×î´ó¸ÅÂʵÃÀ´µÄ£¬È»ºóͨ¹ý´ÓÔʼÍøÂç½øÐÐ֪ʶǨÒÆÀ´»ñÈ¡ÕâЩ²ÎÊý¡£ÂÛÎÄ×÷ÕßÔÚ CIFAR-10¡¢CIFAR-100¡¢ImageNet Êý¾Ý¼¯ÉÏÆÀ¹ÀÁ˸ÃÄ£ÐÍ¡£
¸Ã¼ôÖ¦·½·¨°üº¬Èý¸ö²½Ö裺
ϱí¶Ô±ÈÁ˲»Í¬ ResNet Ä£Ð;¹ý²»Í¬¼ôÖ¦Ëã·¨ºó£¬ËùµÃµ½µÄÄ£ÐÍÔÚ ImageNet Êý¾Ý¼¯Éϵĸ÷×Ô±íÏÖ£º
ÂÛÎÄ 5£ºSelf-Adaptive Network Pruning (ICONIP 2019)
ÕâƪÂÛÎÄÌá³öͨ¹ý×ÔÊÊÓ¦ÍøÂç¼ôÖ¦·½·¨£¨SANP£©½µµÍ CNN µÄ¼ÆËã³É±¾£¬Í¨¹ý¶Ôÿ¸ö¾í»ý²ãÒýÈë Saliency-and-Pruning Module (SPM) À´ÊµÏÖ£¬SPM Ä£¿é¿ÉÒÔѧϰԤ²âÏÔÖøÐÔ·ÖÊý£¬²¢¶Ôÿ¸öͨµÀ¼ôÖ¦¡£SANP »á¸ù¾Ýÿ¸ö²ãºÍÿ¸öÑù±¾¾ö¶¨¶ÔÓ¦µÄ¼ôÖ¦²ßÂÔ¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1910.08906.pdf
¸ù¾ÝÏÂÃæµÄ¼Ü¹¹Í¼£¬SPM Ä£¿éǶÈëÔÚ¾í»ýÍøÂçµÄÿ¸ö²ãÖС£¸ÃÄ£¿é¿ÉÒÔ»ùÓÚÊäÈëÌØÕ÷È¥Ô¤²âͨµÀµÄÏÔÖøÐÔ·ÖÊý£¬È»ºóΪÿ¸öͨµÀÉú³É¶ÔÓ¦µÄ¼ôÖ¦¾ö²ß¡£
¶ÔÓÚ¼ôÖ¦¾ö²ßΪ 0 µÄͨµÀ£¬ÔòÌø¹ý¾í»ýÔËË㣬ȻºóÀûÓ÷ÖÀàÄ¿±êºÍ³É±¾Ä¿±êÁªºÏѵÁ·¹Ç¸ÉÍøÂçºÍ SPM Ä£¿é¡£¼ÆËã³É±¾È¡¾öÓÚÿһ²ãµÄ¼ôÖ¦¾ö²ß¡£
ϱíչʾÁ˸÷½·¨µÄһЩ½á¹û£º
ÂÛÎÄ 6£ºStructured Pruning of Large Language Models (2019)
ÕâƪÂÛÎÄÌá³öµÄ¼ôÖ¦·½·¨»ùÓÚµÍÖÈ·Ö½âºÍÔöÇ¿À¸ñÀÊÈÕ L_0 ·¶ÊýÕýÔò»¯£¨augmented Lagrangian 10 norm regularization£©µÄÔÀí¡£L_0 ÕýÔò»¯·Å¿íÁ˽ṹ»¯¼ôÖ¦´øÀ´µÄÔ¼Êø£¬¶øµÍÖÈ·Ö½âÔò±£ÁôÁ˾ØÕóµÄÃܼ¯½á¹¹¡£
ÂÛÎÄÁ´½Ó£ºhttps://arxiv.org/pdf/1910.04732.pdf
ÕýÔò»¯ÈÃÍøÂçȥѡÔñ´ýÒƳýµÄȨÖØ¡£È¨ÖؾØÕó±»·Ö½âΪÁ½¸ö½ÏСµÄ¾ØÕó£¬È»ºóÉèÖÃÕâÁ½¸ö¾ØÕóÖ®¼äµÄ¶Ô½ÇÏßÑÚÂ루diagonal mask£©¡£ÔÚѵÁ·¹ý³ÌÖУ¬Ê¹Óà L_0 ÕýÔò»¯¶Ô¸ÃÑÚÂëÖ´ÐмôÖ¦¡£ÔöÇ¿À¸ñÀÊÈÕ·½·¨ÓÃÓÚ¿ØÖÆÄ£Ð͵Ä×îÖÕÏ¡Êè³Ì¶È, ÂÛÎÄ×÷Õß½«¸Ã·½·¨½Ð×ö FLOP (Factorized L0 Pruning)¡£
ÂÛÎÄʹÓõÄ×Ö·û¼¶ÓïÑÔÄ£ÐÍÓÃÔÚ enwik8 Êý¾Ý¼¯µÄʵÑéÖУ¬¸ÃÊý¾Ý¼¯°üº¬Ñ¡È¡×Ôά»ù°Ù¿ÆµÄ 100M Êý¾Ý¡£×÷ÕßÔÚ SRU ºÍ Transformer-XL Ä£ÐÍÉÏÆÀ¹ÀÁË FLOP ·½·¨¡£Ï±íչʾÁ˲¿·Ö½á¹û£º
ÒÔÉϾÍÊDZ¾´ÎΪ´ó¼Ò½éÉܵļ¸ÖÖ¼ôÖ¦¼¼Êõ£¬±¾ÎĽéÉܵÄÂÛÎÄÒ²ÓдúÂëʵÏÖ£¬´ó¼Ò¿ÉÒÔÇ××Ô²âÊÔ¡£
ÁìȡרÊô 10ÔªÎÞÃż÷ȯ
˽Ïí×îР¼¼Êõ¸É»õ