Weight normalization: A simple reparameterization to accelerate training of deep neural networks
OpenAI introduces weight normalization, a simple reparameterization that accelerates training of deep neural networks. The technique decouples weight magnitude and direction, reducing gradient covariance and improving convergence.