Receptive Field in Temporal Conv Block

The importance & how-to of receptive field, by configuring # of convolutional layers or the kernel dimension

Temporal Convolution blocks are typically used after the backbone module. If we have a model which makes use of temporal context, to f.ex assist in occluded objects, then such a block is very handy.

Its purpose is essentially to temporally fuse information from neighboring frames into the central frame. If we didn't include this module, then our forward pass would be quite a lot heavier, especially in an attention-based architecture that increases quadratically with the amount of tokens.

As mentioned, with is a convolutional operation. Meaning we will have a kernel with 3 dimensions. The temporal and spatial dimension. I'll setup a concrete example..

We are working with frames, each in spatial resolution. We want this fused into a singular frame.

We pass the 7 frames through our backbone, and from there we are given feature maps. Ther architecture is drawn below:

We want to ensure that our temporal block, allows all frames to communicate with each other temporally. Meaning: it's receptive field covers the entire context window, in this case, being 7 frames in time.

We have the following equation, that allows us to ensure, we cover the entire receptive field.

Very simple. In our case, our kernel's temporal dimension of 3, and 3 convolutional layers covers our receptive field perfectly. The operation can also be visualized as follows:

From there we simply fuse the frames into 1 via mean.

Now here is what happens if we reduced the kernel's temporal dimension to 2:

There is no longer full context, which will be fused into our final output once we mean. And our equation matches our receptive field being only 4 frames:

But we don't necessarily need to increase the temporal dimension, we can also increase the number of convolutional layers, which in some cases could actually assist learning, as there are more learnable parameters in the block, but this of course also means there will be more compute. The schematics will then look as follows:

And once again, our formula agrees:

Conclusion: Full receptive field communication can be achieved by tinkering our temporal kernel dimension and/or our number of convolutional layers.

More convolutional layers = more compute but more learning params

Higher kernel temporal dimension = less compute but fewer learning params