±¾ÎÄ×î³õ·¢±íÓÚ Medium ²©¿Í£¬¾Ô×÷Õß Jerry Wei ÊÚȨ£¬InfoQ ÖÐÎÄÕ¾·Òë²¢·ÖÏí¡£
ELMo£¬ÊÇÀ´×ÔÓïÑÔÄ£Ð͵ÄǶÈ루Embedding fromLanguageModels£©µÄËõд£¬ÕâÊÇÒ»ÖÖ×îÏȽøµÄÓïÑÔ½¨Ä£Ë¼Ïë¡£ÊÇʲôÈÃËüÈç´Ë³É¹¦£¿
2018 Äê·¢±íµÄÂÛÎÄ¡¶Éî¶ÈÉÏÏÂÎĵĴÊǶÈë¡·£¨Deep Contextualized Word Embeddings£©£¬Ìá³öÁËÓïÑÔÄ£ÐÍǶÈ루ELMo£©µÄ˼Ï룬ÔÚÎÊ´ð¡¢Çé¸Ð·ÖÎöºÍÃüÃûʵÌåÌáÈ¡µÈ¶àÏîÈÈÃÅÈÎÎñÉ϶¼ÊµÏÖÁË×îÏȽøµÄÐÔÄÜ¡£ÊÂʵ֤Ã÷£¬ELMo ¿ÉÒÔ´øÀ´¸ß´ï 5% µÄÐÔÄÜÌáÉý¡£µ«ÊÇ£¬ÊÇʲôÈÃÕâ¸öÏë·¨Èç´Ë¾ßÓиïÃüÐÔÄØ£¿
Elmo ²»½ö½öÊÇÒ»¸öľżÈËÎïÍæ¾ßµÄÃû×Ö£¬¶øÇÒ ELMo »¹ÊÇÒ»¸öÇ¿´óµÄ¼ÆËãÄ£ÐÍ£¬¿ÉÒÔ½«µ¥´Êת»»³ÉÊý×Ö¡£ÕâÒ»ÖØÒªµÄ¹ý³ÌÔÊÐí»úÆ÷ѧϰģÐÍ£¨ÒÔÊý×Ö¶ø²»Êǵ¥´Ê×÷ΪÊäÈ룩¿ÉÒÔÔÚÎı¾Êý¾ÝÉϽøÐÐѵÁ·¡£
µ±ÎÒͨ¶ÁÔʼÂÛÎÄʱ£¬Óм¸¸öÖ÷Òª¹ÛµãÒýÆðÁËÎÒµÄ×¢Ò⣺
ÈÃÎÒÃÇÏêϸÌÖÂÛÒ»ÏÂÕâЩҪµã£¬²¢ÌÖÂÛËüÃǵÄÖØÒªÐÔ¡£
Ïñ GloVe¡¢´Ê´üºÍ Word2Vec Ö»ÊǸù¾Ýµ¥´ÊµÄ×ÖÃæƴдÉú³ÉǶÈë¡£ËüÃDz»»á¿¼Âǵ¥´ÊµÄʹÓ÷½Ê½¡£ÀýÈ磬ÔÚÏÂÃæµÄʾÀýÖУ¬ÕâЩÓïÑÔÄ£Ðͽ«Îª¡°trust¡±·µ»ØÏàͬµÄǶÈ룺
I can¡¯ttrustyou. They have notrustleft for their friend. He has atrustfund.
È»¶ø£¬ELMo »á¸ù¾ÝÆäÖÜΧµÄµ¥´Ê£¬·µ»Ø²»Í¬µÄǶÈ룺ÆäǶÈëÊÇÉÏÏÂÎÄÏà¹ØµÄ¡£ÔÚÕâЩʾÀýÖУ¬Ëüʵ¼ÊÉϻ᷵»Ø¡°trust¡±²»Í¬µÄ´ð°¸£¬ÒòΪËü»áʶ±ð³ö¸Ã´ÊÔÚ²»Í¬µÄÉÏÏÂÎÄÖÐʹÓá£ÕâÖÖ¶ÀÌصÄÄÜÁ¦±¾ÖÊÉÏÒâζ×Å ELMo µÄǶÈëÓиü¶àµÄ¿ÉÓÃÐÅÏ¢£¬Òò´ËÐÔÄÜ¿ÉÄÜ»áÌá¸ß¡£Ò»ÖÖÀàËƵĽâÊÍÉÏÏÂÎĵÄÓïÑÔ½¨Ä£·½·¨ÊÇBERT¡£
ÎÞÂÛÄãÊÇÒ»Ãû×ÊÉîµÄ»úÆ÷ѧϰÑо¿Õߣ¬»¹ÊÇÒ»ÃûÆÕͨµÄ¹Û²ìÕߣ¬Ä㶼¿ÉÄÜÊìϤ´óÊý¾ÝµÄÍþÁ¦¡£×î³õµÄ ELMo Ä£ÐÍÊÇÔÚ 55 ÒÚ¸öµ¥´ÊµÄÓïÁÏ¿âÉϽøÐÐѵÁ·µÄ£¬¼´Ê¹ÊÇ¡°Ð¡¡±°æ±¾Ò²ÓÐ 10 ÒÚ¸öµ¥´ÊµÄѵÁ·¼¯¡£Õâ¿ÉÊÇÒ»¸öºÜ´óµÄÊý¾ÝÁ¿°¡£¡ÔÚÈç´Ë¶àÊý¾ÝÉϽøÐÐѵÁ·£¬Òâζ×Å ELMo ÒѾѧϰÁ˺ܶàÓïÑÔ֪ʶ£¬ÔÚ´ó·¶Î§µÄÊý¾Ý¼¯É϶¼»áÓв»´íµÄÐÔÄÜ¡£
Íƶ¯»úÆ÷ѧϰÁìÓò·¢Õ¹µÄÒòËØÊǽ«Ñо¿ÓèÒÔ¿ªÔ´µÄÎÄ»¯¡£Í¨¹ý½«´úÂëºÍÊý¾Ý¼¯¿ªÔ´£¬Ñо¿ÈËÔ±¿ÉÒÔÈøÃÁìÓòµÄÆäËûÈËÇáËɵØÓ¦Óò¢¹¹½¨ÏÖÓеÄÏë·¨¡£ÓëÕâÖÖÎÄ»¯ÏàÒ»Ö£¬ELMo Êǹ㷺¿ªÔ´µÄ¡£ËüÓÐÒ»¸öÍøÕ¾£¬²»½ö°üÀ¨ËüµÄ»ù±¾ÐÅÏ¢£¬»¹°üÀ¨ÁËÄ£Ð͵ÄС°æ±¾¡¢Öа汾ºÍÔʼ°æ±¾µÄÏÂÔØÁ´½Ó¡£ÏëʹÓà EMLo µÄÈËÒ»¶¨Òª¿´¿´Õâ¸öÍøÕ¾£¬À´»ñÈ¡Ä£Ð͵ĿìËÙ¿½±´¡£´ËÍ⣬¸Ã´úÂ뻹·¢²¼ÔÚGitHubÉÏ£¬²¢°üÀ¨Ò»¸öÏ൱Ï꾡µÄ×ÔÊöÎļþ£¬ÈÃÓû§ÖªµÀÈçºÎʹÓà ELMo¡£Èç¹ûÓÐÈË»¨Á˼¸¸öСʱµÄʱ¼ä²ÅÄÜÈà EMLo Ä£ÐÍÕý³£ÔËÐУ¬ÎÒ»á¸Ðµ½ºÜ¾ªÑÈ¡£
ELMo ÔÚÖøÃûµÄÈÎÎñÉÏÈ¡µÃÁË×îÏȽøµÄÐÔÄÜ£¬Èç SQuAD¡¢NER ºÍ SST
×÷ΪÉÏÏÂÎĸÐÖª´ÊǶÈëºÍ´óÊý¾ÝµÄÇ¿´ó×éºÏ£¬ELMo ÔÚ×ÔÈ»ÓïÑÔ´¦ÀíÖеĴóÊý¾Ý¼¯£¨°üÀ¨ SQuAD¡¢NER ºÍ SST£©ÉÏÈ¡µÃÁË×îÏȽøµÄÐÔÄÜ¡£ELMo ³¹µ×¸Ä±äÁËÎÒÃÇ´¦Àí¼ÆËãÓïÑÔѧÈÎÎñµÄ·½Ê½£¬±ÈÈçÎÊ´ðºÍÇé¸Ð¼ì²â£¬ÕâÏÔÈ»ÊǸÃÁìÓòµÄÒ»¸ö¹Ø¼ü½øÕ¹£¬ÒòΪÔʼÂÛÎÄÒѱ»ÒýÓÃÁË 4500 ¶à´Î¡£´ËÍ⣬ÔÚ·¢±í ELMo µÄÂÛÎĺó£¬Ìá½»¸ø×î´óµÄ¹ú¼Ê×ÔÈ»ÓïÑÔ´¦Àí»áÒ顪¡ª¼ÆËãÓïÑÔѧлᣨAssociation for Computational Linguistics£¬ACL£©»áÒéÊÕµ½µÄͶ¸åÁ¿·ÁËÒ»·¬£¬´Ó 2018 ÄêµÄ 1544 ·ÝÔö¼Óµ½ 2019 ÄêµÄ 2905 ·Ý£¨²»¹ýÕâÒ²¿ÉÄÜÊÇÓÉÓÚ 2019 Äê³õ·¢±íµÄ BERT£©¡£
ÎÒ»¹ÒªÖ¸³öµÄÊÇ£¬ELMo ºÍ BERT ·Ç³£ÏàËÆ£¬ÒòΪËüÃǶ¼À´×ÔÖ¥Âé½Ö£¨Òë×¢£ºElmo ºÍ Bert ¶¼ÊÇÃÀ¹ú¾µä¶¯»Æ¬¡¶Ö¥Âé½Ö¡·µÄ½ÇÉ«£©£¡ºÃ°É£¬ËüÃǶ¼ÊǽâÊ͵¥´ÊÉÏÏÂÎĵÄÓïÑÔÄ£ÐÍ£¬¶¼ÊÇ»ùÓÚ´óÐÍÊý¾Ý¼¯½øÐÐѵÁ·µÄ£¬²¢ÇÒÕýÈç¸øÎÒÃÇËùÖªµÄ×ÔÈ»ÓïÑÔ´¦ÀíÁìÓò´øÀ´Á˸ïÃüÐԵıä¸ï¡£ÎÒ»¹Ð´ÁËһƪ¹ØÓÚ BERT µÄ²©ÎÄ£¬Èç¹ûÄãÓÐÐËȤµÄ»°£¬¿ÉÒÔ¶ÁÒ»¶Á¡£
ELMo ÊÇ×ÔÈ»ÓïÑÔ´¦Àí×î´óµÄ½ø²½Ö®Ò»£¬ÒòΪËü±¾ÖÊÉÏÊǵÚÒ»¸ö¹Ø×¢ÉÏÏÂÎĵÄÓïÑÔÄ£ÐÍ£¬ÔÊÐíÔÚ¶àÖÖÈÎÎñÖÐʵÏÖ¸üºÃµÄÐÔÄÜ¡£
ÑÓÉìÔĶÁ
Jerry Wei£¬¶ÔÈ˹¤ÖÇÄÜ£¬ÓÈÆä¶ÔҽѧͼÏñ·ÖÎöºÍ×ÔÈ»ÓïÑÔ´¦Àí¸ÐÐËȤ¡£
ÔÎÄÁ´½Ó£º
https://towardsdatascience.com/elmo-why-its-one-of-the-biggest-advancements-in-nlp-7911161d44be
ÁìȡרÊô 10ÔªÎÞÃż÷ȯ
˽Ïí×îР¼¼Êõ¸É»õ