From one piece Frankle & Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" (arXiv) 5 beliefs, in the piece's order there
-
korrents.com
A small enough winning ticket learns faster than the network it was cut out of, and ends up more accurate than it.Their words
Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
-
Their words
The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
-
Their words
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively.
+ 2 more
-
Their words
However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
-
Their words
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy.