ICLR vs arxiv-sanity
ICLR vs arxiv-sanity
我觉得交叉对比 ICLR 2017（一个备受欢迎的深度学习大会）的论文决策（分为四类：Oral、Poster、Workshop 和 Reject）与所有论文在 arxiv-sanity上 被添加到某些人的库的次数是一件有趣的事情。ICLR 2017 决策的制定取决于在某段时间内区域主席和评论者的数量，以此来决定某篇论文的命运，而 arxiv-sanity 是一些像我一样每月工作两小时，并能熟练使用大量论文的人来决定的。这是一场自上而下与自下而上之间的斗争。让我们看看发生了什么。
访问网址可查看 ICLR 2017 的所有论文决策：https://openreview.net/group?id=ICLR.cc/2017/conference。总计提交的论文一共有491篇，其中15篇是 Oral 的（约3%），180篇是 Poster 的（约37.3%），48篇是 Workshop 建议的（约9.8%），另外有245篇是 Reject 的（约49.9%）。被采纳的论文将会呈现在今年四月24-27日于法国 Toulon 举行的 ICLR 大会上，Toulon是一个我非常想去的城市。大家一起来欣赏一下，如此的迷人：
抱歉，跑题了。
另一方面，我们所接触到的 arxiv-sanity 有库的功能。简单来说，任何的注册用户都可以在他们的库里添加论文，arxiv-sanity 将在论文的全文中基于双字母组TFIDF（term frequency-inverse document frequency）特征训练一个个性化的 SVM（Support Vector Machine），并基于论文内容推荐给用户。例如，在我的库里有很多关于 RL/生成模型/CV 的论文，当这些主题有新的论文的时候，会顶置到我的推荐。Arxiv-sanity 的审阅池现在有3195位用户，这是指在其账号下的库中至少添加过一篇论文的用户。从总体上来看，这些用户共添加了55,671篇论文到他们库里，平均每人添加过17.4篇论文。
Arxiv-sanity 的一个重要特征是用户做的不单单是没有反应的投票。添加到你的库里的论文是有权重的，因为那些论文会影响到给你的推荐。这样一来，你就有动力仅仅去添加那些真正和你有关系的东西。这么做很聪明不是吗？是不是？好吧就先这样吧。
实验
长话短说，我把 ICLR 上所有的论文都搜集起来，然后使用标题在 arxiv-sanity 上做准确的匹配。有些 ICLR 上的论文在 arxiv-sanity 上没有，有些不能匹配的原因是作者修改了名字，或者是因为包含特殊字符，等等。
例如，我们看看从 ICLR 上找到的 oral 论文。结果如下：
for oral, found 10/15papers on arxiv with library counts:
64 Reinforcement Learningwith Unsupervised Auxiliary Tasks
44 Neural ArchitectureSearch with Reinforcement Learning
38 Understanding deep learningrequires rethinking generalizatio...
28 Towards PrincipledMethods for Training Generative Adversaria...
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
22 Learning End-to-EndGoal-Oriented Dialog
19 Q-Prop:Sample-Efficient Policy Gradient with An Off-Policy C...
13 Learning to Act byPredicting the Future
12 Amortised MAP Inferencefor Image Super-resolution
8 Multi-Agent Cooperationand the Emergence of (Natural) Langua...
8 End-to-end OptimizedImage Compression
从结果上看，15篇 oral 论文中有10篇在 arxiv-sanity 上匹配成功，数量接近每个注册者添加到他们库里文档的数量。例如：“无监控辅助任务的强化学习”在 arxiv-sanity 上有64个用户把它加入到库里了。我不得不把有些论文的名字截短，因为 medium.com 的结构不太灵活，也不允许你去改变字体的大小。
现在我们来看一下 poster：
for poster, found 113/183papers on arxiv with library counts:
149 Adversarial FeatureLearning
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
147 HierarchicalMultiscale Recurrent Neural Networks
140 Recurrent BatchNormalization
80 HyperNetworks
79 FractalNet: Ultra-DeepNeural Networks without Residuals
73 Zoneout: RegularizingRNNs by Randomly Preserving Hidden Acti...
62 Unrolled GenerativeAdversarial Networks
52 Adversarially LearnedInference
49 Quasi-Recurrent NeuralNetworks
48 Do Deep ConvolutionalNets Really Need to be Deep and Convolu...
46 Neural Photo Editingwith Introspective Adversarial Networks
43 An Actor-CriticAlgorithm for Sequence Prediction
41 A LearnedRepresentation For Artistic Style
37 Structured AttentionNetworks
33 Mollifying Networks
30 DeepCoder: Learning toWrite Programs
28 SGDR: StochasticGradient Descent with Warm Restarts
27 Learning to Navigate inComplex Environments
27 GenerativeMulti-Adversarial Networks
26 Soft Weight-Sharing forNeural Network Compression
25 Pruning Filters forEfficient ConvNets
24 Why Deep NeuralNetworks for Function Approximation?
24 Mode RegularizedGenerative Adversarial Networks
24 Dialogue Learning WithHuman-in-the-Loop
24 Designing NeuralNetwork Architectures using Reinforcement Le...
23 PGQ: Combining policy gradientand Q-learning
22 Frustratingly ShortAttention Spans in Neural Language Modeli...
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
21 Tracking the WorldState with Recurrent Entity Networks
21 Deep ProbabilisticProgramming
20 Density estimationusing Real NVP
20 Adversarial TrainingMethods for Semi-Supervised Text Classif...
19 Semi-SupervisedClassification with Graph Convolutional Netwo...
19 PixelVAE: A LatentVariable Model for Natural Images
19 Learning to Optimize
19 Learning a NaturalLanguage Interface with Neural Programmer
19 Entropy-SGD: BiasingGradient Descent Into Wide Valleys
19 Dynamic CoattentionNetworks For Question Answering
18 PixelCNN++: Improvingthe PixelCNN with Discretized Logistic ...
18 Generalizing Skillswith Semi-Supervised Reinforcement Learni...
18 Deep Learning withDynamic Computation Graphs
18 Automatic RuleExtraction from Long Short Term Memory Network...
18 Adversarial MachineLearning at Scale
17 Learning through DialogueInteractions by Asking Questions
16 Learning to PerformPhysics Experiments via Deep Reinforcemen...
16 CategoricalReparameterization with Gumbel-Softmax
15 Sample EfficientActor-Critic with Experience Replay
14 Variational LossyAutoencoder
14 Identity Matters inDeep Learning
14 Bidirectional AttentionFlow for Machine Comprehension
13 Towards a NeuralStatistician
13 Recurrent MixtureDensity Network for Spatiotemporal Visual A...
13 On DetectingAdversarial Perturbations
pdfcrowd.com open in browserPRO version
Are you a developer? Tryout the HTML to PDF API
12 Trained TernaryQuantization
12 Improving PolicyGradient by Exploring Under-appreciated Rewa...
12 Capacity andTrainability in Recurrent Neural Networks
11 SampleRNN: AnUnconditional End-to-End Neural Audio Generatio...
11 Machine ComprehensionUsing Match-LSTM and Answer Pointer
11 Latent SequenceDecompositions
11 CalibratingEnergy-based Generative Adversarial Networks
10 UnsupervisedCross-Domain Image Generation
10 Learning to RememberRare Events
10 Highway and ResidualNetworks learn Unrolled Iterative Estima...
9 TopicRNN: A RecurrentNeural Network with Long-Range Semantic...
9 Steerable CNNs
9 Query-Reduction Networksfor Question Answering
9 Lossy Image Compressionwith Compressive Autoencoders
9 Learning to ComposeWords into Sentences with Reinforcement L...
8 Stick-BreakingVariational Autoencoders
8 Deep VariationalInformation Bottleneck
8 Batch Policy GradientMethods for Improving Neural Conversati...
7 Discrete VariationalAutoencoders
7 Data Noising asSmoothing in Neural Network Language Models
6 Variable Computation inRecurrent Neural Networks
6 Sigma Delta QuantizedNetworks
6 Dropout withExpectation-linear Regularization
6 Delving intoTransferable Adversarial Examples and Black-box ...
6 A CompositionalObject-Based Approach to Learning Physical Dy...
5 Towards the Limit ofNetwork Quantization
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
5 Tighter bounds lead toimproved classifiers
5 Pointer Sentinel MixtureModels
5 On the QuantitativeAnalysis of Decoder-Based Generative Mode...
5 Neuro-Symbolic ProgramSynthesis
5 Lie-Access Neural TuringMachines
5 Learning tosuperoptimize programs
5 Learning Features ofMusic From Scratch
5 Improving NeuralLanguage Models with a Continuous Cache
5 Deep Biaffine Attentionfor Neural Dependency Parsing
4 Temporal Ensembling forSemi-Supervised Learning
4 Diet Networks: ThinParameters for Fat Genomics
4 DeepDSL: ACompilation-based Domain-Specific Language for Dee...
4 DSD: Dense-Sparse-DenseTraining for Deep Neural Networks
4 A recurrent neuralnetwork without chaos
3 Trusting SVM forPiecewise Linear CNNs
3 The Neural Noisy Channel
3 Revisiting ClassifierTwo-Sample Tests
3 Regularizing CNNs withLocally Constrained Decorrelations
3 Optimal BinaryAutoencoding with Pairwise Correlations
3 Loss-aware Binarizationof Deep Networks
3 Learning RecurrentRepresentations for Hierarchical Behavior ...
3 EPOpt: Learning RobustNeural Network Policies Using Model En...
3 Deep InformationPropagation
2 Words or Characters?Fine-grained Gating for Reading Comprehe...
2 Topology and Geometry ofHalf-Rectified Network Optimization
pdfcrowd.com open inbrowser PRO version
Are you a developer? Try outthe HTML to PDF API
2 Maximum Entropy FlowNetworks
2 Incorporating long-rangeconsistency in CNN-based texture gen...
2 Hadamard Product forLow-rank Bilinear Pooling
1 Multi-view RecurrentNeural Acoustic Word Embeddings
1 Inductive Bias of DeepConvolutional Networks through Pooling...
1 Geometry of Polysemy
1 Autoencoding VariationalInference For Topic Models
1 A STRUCTUREDSELF-ATTENTIVE SENTENCE EMBEDDING
0 Deep Multi-taskRepresentation Learning: A Tensor Factorisati...
0 A Compare-Aggregate Modelfor Matching Text Sequences
有些得到人们的青睐（149次添加），有些却非常少（0）。
再来看看 workshop 建议的结果：
for workshop, found 23/48papers on arxiv with library counts:
60 Adversarial examples inthe physical world
31 Learning in ImplicitGenerative Models
16 Surprise-Based IntrinsicMotivation for Deep Reinforcement Le...
14 Multiplicative LSTM forsequence modelling
13 Efficient SoftmaxApproximation for GPUs
12 RenderGAN: GeneratingRealistic Labeled Data
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
12 Generalizable FeaturesFrom Unsupervised Learning
10 Programming With aDifferentiable Forth Interpreter
8 Gated Multimodal Unitsfor Information Fusion
8 Deep Learning with Setsand Point Clouds
7 Unsupervised PerceptualRewards for Imitation Learning
5 Song From PI: AMusically Plausible Network for Pop Music Gen...
5 Modular MultitaskReinforcement Learning with Policy Sketches
5 A Differentiable PhysicsEngine for Deep Learning in Robotics
4 Exponential Machines
4 Dataset Augmentation inFeature Space
3 Semi-supervised deeplearning by metric embedding
2 Adaptive FeatureAbstraction for Translating Video to Languag...
1 Modularized Morphing ofNeural Networks
1 Learning ContinuousSemantic Representations of Symbolic Expr...
1 Extrapolation andlearning equations
0 Online StructureLearning for Sum-Product Networks with Gauss...
0 Bit-Pragmatic DeepNeural Network Computing
我没有把200多篇被拒的论文一一列出来，然而其中的一些论文在 arxiv-sanity 上深受某些用户的喜爱，但是取没有得到 ICLR 的区域主席和评论者的青睐：
for reject, found 58/245papers on arxiv with library counts:
46 The Predictron:End-To-End Learning and Planning
39 RL^2: FastReinforcement Learning via Slow Reinforcement Lear...
35 Understandingintermediate layers using linear classifier pro...
33 Hierarchical MemoryNetworks
31 An Analysis of DeepNeural Network Models for Practical Appli...
20 Low-rank passthroughneural networks
19 Higher Order RecurrentNeural Networks
18 Adding Gradient NoiseImproves Learning for Very Deep Network...
16 UnsupervisedPretraining for Sequence to Sequence Learning
16 A Joint Many-TaskModel: Growing a Neural Network for Multipl...
15 Adversarial examplesfor generative models
14 Gated-Attention Readersfor Text Comprehension
13 Extensions andLimitations of the Neural GPU
12 Warped Convolutions: EfficientInvariance to Spatial Transfor...
11 Neural CombinatorialOptimization with Reinforcement Learning
11 Memory-augmentedAttention Modelling for Videos
10 GRAM: Graph-basedAttention Model for Healthcare Representati...
9 Wav2Letter: anEnd-to-End ConvNet-based Speech Recognition Sy...
9 Understanding trainedCNNs by indexing neuron selectivity
9 The Power of Sparsity inConvolutional Neural Networks
9 Improving StochasticGradient Descent with Feedback
8 TowardsInformation-Seeking Agents
8 NEWSQA: A MACHINECOMPREHENSION DATASET
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
8 LipNet: End-to-EndSentence-level Lipreading
7 Generative AdversarialParallelization
7 Efficient Summarizationwith Read-Again and Copy Mechanism
6 Multi-task learning withdeep model based reinforcement learn...
6 Multi-modal VariationalEncoder-Decoders
6 End-to-End Answer ChunkExtraction and Ranking for Reading Co...
6 Boosting ImageCaptioning with Attributes
6 Beyond Fine Tuning: AModular Approach to Learning on Small D...
5 Structured SequenceModeling with Graph Convolutional Recurre...
5 Human perception incomputer vision
5 Cooperative Training ofDescriptor and Generator Networks
这是未删节的完整版本（https://cs.stanford.edu/people/karpathy/iclr2017.txt）。这份表单中位于顶部的论文很可能是由于不公平的原因而被拒绝掉的。
另外一个问题，如果单纯是由 arxiv-sannity 的用户用投票的形式来决定（在 arxiv-sanity 上能够找到的论文），那么 ICLR2017 将会是什么样子？如下是摘要：
oral:
149 Adversarial FeatureLearning
147 HierarchicalMultiscale Recurrent Neural Networks
140 Recurrent BatchNormalization
80 HyperNetworks
79 FractalNet: Ultra-DeepNeural Networks without Residuals
73 Zoneout: RegularizingRNNs by Randomly Preserving Hidden Acti...
64 Reinforcement Learningwith Unsupervised Auxiliary Tasks
62 Unrolled GenerativeAdversarial Networks
60 Adversarial examples inthe physical world
52 Adversarially LearnedInference
-------------------------------------------------
poster:
49 Quasi-Recurrent NeuralNetworks
48 Do Deep ConvolutionalNets Really Need to be Deep and Convolu...
46 The Predictron:End-To-End Learning and Planning
46 Neural Photo Editingwith Introspective Adversarial Networks
44 Neural ArchitectureSearch with Reinforcement Learning
43 An Actor-CriticAlgorithm for Sequence Prediction
41 A LearnedRepresentation For Artistic Style
39 RL^2: FastReinforcement Learning via Slow Reinforcement Lear...
38 Understanding deeplearning requires rethinking generalizatio...
37 Structured AttentionNetworks
35 Understandingintermediate layers using linear classifier pro...
33 Mollifying Networks
33 Hierarchical MemoryNetworks
pdfcrowd.com open inbrowser PRO version
Are you a developer? Tryout the HTML to PDF API
31 Learning in ImplicitGenerative Models
31 An Analysis of DeepNeural Network Models for Practical Appli...
30 DeepCoder: Learning toWrite Programs
这里是完整的列表（https://cs.stanford.edu/people/karpathy/as2017.txt）。特别标注一下，有些被 ICLR 拒掉的论文，如果单纯是看 arxiv-sanity 的添加情况，应该是属于 oral 的，特别是 the Predictron、RL?、“理解中间层”和“层级记忆网络”。相反，有些被采纳的论文却鲜有 arxiv-sanity 用户的青睐。下面是混淆矩阵：
这是混淆矩阵的文本版（https://cs.stanford.edu/people/karpathy/iclr_conf.txt），每一个单元都把对应的论文的标题集合到了一起。结果看起来不太坏，两个组织在 oral 这个类型上完全不能达成一致，但是在poster这个类型上达到了惊人的一致。最重要的是，在 oral/poster 和 rejection 之间混淆非常少。同时，也恭喜Max等人，因为其所著的《无监控辅助任务的强化学习》（Reinforcement Learningwith Unsupervised Auxiliary Tasks，https://arxiv.org/abs/1611.05397）是两个组织唯一共同认可的 oral 类型论文。
最后，我在几天前读到了媒体所发表的文章，是 Carlos E. Perez 所著的《被 ICLR2017 所拒绝的10篇最应上榜的论文》（Ten Deserving Deep Learning Papers that were Rejected at ICLR 2017，https://medium.com/intuitionmachine/eight-deserving-deep-learning-papers-that-were-rejected-at-iclr-2017-119e19a4c30b）。看起来 arxiv-sanity 的用户非常赞同这篇博客，该博客所列的所有文章（包括 LipNet）（同样该文章我们也能在 arxiv-sanity 上找到）已经被 arxiv-sanity 的用户接受了。
讨论
有很多因素会影响这些结果。例如，随着时间的推移，arxiv-sanity 的用户数会逐渐变多，因此这些结果可能会让用户稍微喜欢一点的论文在 arxiv-sanity 上发表的时间比之前稍晚一点，用户也会更加关注它们，就像关注网站上的新论文一样。同样，论文的发表也不是等频的，举例来说，如果有些论文是通过推特来传播的，将会有更多的人看到它，也会有更多的人把它加到自己的库里。最后，arxiv-sanity 所带来的一个最好的论据是“富者越来越富”，因为 arxiv-sanity 的论文都是实名的，名誉会引来更多的关注。在这种特殊情况下，因为 ICLR 采用的是单盲法，所以这不是造成差异化的因素。
总体来说，从这个实验我得出来的结论是，信号已经非常明显了。自下而上的，我们当下正免费获得这些资源，不用花费诸多人力物力和时间就能完成。不管是在提交阶段还是在审核阶段，有些人都花费了大量的时间，痛苦的、紧张的而又反复地论证着，可能会持续数周甚至数月。但我认为，或许我们不需要这么做。或者说，至少还有很大的改进的空间。
校正：有人提出了一个有趣的想法，把 ICLR 2018 的提交或接受的论文的引文数量进行一下加合，以此为指标看看哪个会“win”。对此，我们拭目以待。
本文由 AI100 编译，转载需得到本公众号同意。
编译：AI100
原文链接：https://medium.com/@karpathy/iclr-2017-vs-arxiv-sanity-d1488ac5c131