Pix2Pix GAN distillation
Implementation of "Teachers Do More Than Teach. Compressing Image-to-Image Models" paper
Teacher generator is being pruned using binary search over BatchNorm scaling factors to meet certain computational budget requirements and trainer using Kernel Alignment distillation loss and GAN losses.
Computational budget is computed through the calculation of MAC operation and dividing it by the desired fraction.
As a result student model learns to imitate the outputs of the teacher model but with 10x smaller compute.
Link to github for more details.