This article was first published in Proceedings of the ACM on Computer Graphics and Interactive Techniques.
We show how to transform a nondifferentiable rasterizer into a differentiable rasterizer with minimal engineering efforts and no external dependencies (no PyTorch* or TensorFlow*). We rely on stochastic gradient estimation, a technique that consists of rasterizing after randomly perturbing the scene's parameters so their gradient can be stochastically estimated and descended. This method is simple and robust but does not scale in dimensionality (number of scene parameters). Our insight is that the number of parameters contributing to a given rasterized pixel is bounded. Estimating and averaging gradients on a per-pixel basis hence bounds the dimensionality of the underlying optimization problem and makes the method scalable. Furthermore, it is simple to track per-pixel contributing parameters by rasterizing ID- and UV-buffers, which are trivial additions to a rasterization engine if not already available. With these minor modifications, we obtain an in-engine optimizer for 3D assets with millions of geometry and texture parameters.
