Efficient VIS ; tracklet queries & proposals

A paper breakdown of: https://arxiv.org/pdf/2203.01853

VIS: Video Instance Segmentation The point of this paper is to come with a method where VIS can be performed more efficiently, while still preserving high accuracy, and allowing the model to be fully end to end.

The paper accomplishes that via the following bullet points:

These will also be the main topics which will be explained at a deeper level in this article.

VIS is either processed at frame-level or clip-level. With frame level, our model with perform segmentation, then links current masks with historical masks, via manual data association (f.x. NMS). Frame-level also limits itself in its use of temporal data.

However clip-level makes use of the temporal object information, which typically leads to greater accuracy. VisTR, which is built of DETR did a fine job of performing clip level instance segmentation, however the issue comes to slow convergence speeds, and the architecture not truly being end to end. As if the clip was too big to fit on the GPU then it would have to be split and stitched together via data association.

EfficientVIS allows for quicker convergence speeds, and a fully end-to-end pipeline by making use of its "tracklet queries & proposals."

Tracklet Query : A latent embedding of the object information

Tracklet Proposal : Proposal of where the object resides

The way we fetch information from the frame, is via our TDC. And then exchange this information between the queries through our FTSA block.

1. EfficientVIS is the first RoI-wise clip-level VIS framework that runs in real-time Where it really gains its convergence speed is its ROI-wise design, which allowed it to be the first real time VIS architecture. An issue VisTR had, is that every query attended to one another.

2. EfficientVIS is the first fully end-to-end neural net for VIS. This is done, by feeding our tracklet queries from one clip onto the next. Preserving the object information through time.

In each forward pass, EfficientVIS takes in T frames , and outputs our m tracklet masks.

Our dimensions look as follows:

We have T input frames, and N instances, where our tracklet masks are per object across all T frames.

Below you can see the entire architecture. To gain some intuition before we get into the details.

efficientvis-architecture-overview

You can see how we start with our randomly initialised tracklet queries & proposals. The dimensions of the tracklets are as follows:

Where we have a latent embedding per instance i for every frame T that has a channel dimension of C. Our Tracklet proposal is of dimension 4 instead of C, as those are the 4 coordinates of our proposal box. Or the center coordinate (x,y) and its respective (h,w).

As you can see in the animation, the very first step is the CNN backbone to extract the features.

From there we have our main block which runs M iterations. We have our queries & proposals which are randomly intialized. First step is to perform FTSA (factorised temporal spatial self attention) between the queries. Which is visualized below:

trackletqueriesproposalsftsa

This is just regular attention, but first across our dimension T per object N (temporal attention), then across our dimension N per frame T (spatial attention).

Now our tracklet queries have deduped, making sure they dont cluster onto a POI. And have also learnt more about itself through the temporal span.

Now that we have performed FTSA on our queries, we can move onto TDC. TDC receives our queries, proposals and base feature frames as input (Whereas FTSA only operated on our queries as input).

TDC has the following equation:

Let's break it down. is simply our region of interest. We are taking our currently iterated on frame t', and extracting our tracklet proposal region . Now we will be using our dynamically generated kernel to extract the features from the region. Our kernel , is generated off our tracklet query . Which as you recall, contains the content embedding, the semantic information of our object. Which allows this kernel to know what features it is interested in collecting.

So shortly put, : the extracted object instance features from ROI .

Now onto: . This is our attention matrix. is the following function:

Simply put: we are taking the extracted features from our region of interest, given our base frame t: AND the currently iterated on frame t': . Then we perform cosine similarity, the higher the similarity the higher the value. Then at last perform softmax across the 3 cosine similarity matrices. The softmax is performed, so the cell (i,j) across all 3 cosine similarity matrices will add to 1.

I say "softmax across the 3" as we are iterating across 3 frames. The base frame t, and the neighboring frames. Below I will visualize this more clearly:

tdc

Now we have our output tracklet feature: . Now we move onto our Head network.

Now we have already covered the 2 main topics which makes this model converge quicker than f.x VisTr, being: FTSA & TDC.

Now let's quickly skim over our Head network, which isn't as exciting. I will already visualize it below so you can gain a mental model of it:

headnetworks

The key takeaway from our head networks is how we keep refining the tracklet proposal.

The network is structured as:

Classificaiton Head

This head first does global average pooling to collapse the dimension from:

Then we perform a linear projection, to project our $N \times C $ matrix to . being the number of classes. Then we perform softmax to create a probability distribution per object instance on which class it is most likely to be.

Box Head

Box Head is quite similiar to our Classification head. Where we perform average pooling to collapse the spatial dimension as follows:

Then we perform a linear projection to get the following dimension: . Our 4 coordinates, are the delta coordinates, meaning our predicted offset shift to our current proposal box.

Mask Head

The mask head is quite simple. We already have our tracklet feature, which contains the feature maps of every object N at every frame T. Our tracklet feature maps live in a ROI align. We already have our tracklet feature , which holds a feature map for every object at every frame . These feature maps live at a fixed RoIAlign resolution that is identical across all proposals.

We first apply regular convolutions to the feature map, then a deconvolution that upsamples it to the mask resolution (e.g. ). We then apply a sigmoid to every pixel, giving a soft probability in . Thresholding this with (e.g. ) yields the binary mask; pixel is foreground when .

Finally, we bilinearly interpolate this fixed mask onto the tracklet proposal box and paste it into the frame. Since varies with how large the detected object is, the same small mask scales up for an elephant and down for a cricket, producing the full-frame mask . which is the same across all proposals. We first perform regular 3x3 convolutions to the feature map, then deconv it up to its mask output dimension. From there we perform sigmoid on every pixel. Now we apply a mask depending on if the pixel sigmoid value is over or under a threshold we set.

Hungarian Matching

Hungarian matching is performed one to one. Where we assign every N instance to their corresponding K class, which allows for the global optimum in regards to achieving lowest loss, where the objects that are not assigned a category, is assigned the background class. This ensures that queries are not clustering over the same object.

Loss

Calculate loss for that iteration, and aggregate it as we only perform back propogation at the last M'th iteration.

Proposal Update

Now we take our updated proposal from our box regression head, and make use of it for the next iteration, this will then allow our TDC to become more accurate at every iteration.

Correspondence Learning

Correspondence learning allows this architecture to be fully end to end. Requiring no manual data association between the input clips to the model. This correspondence is performed via the passing of our tracklet queries from one clip to another. As you (hopefully) can recall, our tracklet query contains the semantic information of an object across all T frames. What we do, is that we average the semantic information through the T frames before passing it on to the next clip.

However our tracklet proposal is not sent forward, instead its initiated to attend to the whole screen to begin with, then iteration by iteration its refined via its box regression head, allowing for a more refined TDC extraction, which leads to a semantically richer tracklet feature, leading to better predictions, at quicker convergence speeds.