Encoding videos as neural networks is a recently proposed approach that allows new forms of video processing. However, traditional techniques still outperform such neural video representation (NVR) methods for the task of video compression. This performance gap can be explained by
the fact that current NVR methods: i) use architectures that do not efficiently obtain a compact representation of temporal and spatial information; and ii) minimize rate and distortion disjointly (first overfitting a network on a video and then using heuristic techniques such as post-training
quantization or weight pruning to compress the model). We propose a novel convolutional architecture for video representation that better represents spatio-temporal information
and a training strategy capable of jointly optimizing rateand distortion.