NOOUGAT - Towards Unified Online & Offline MOT

Paper breakdown of: https://arxiv.org/pdf/2509.02111

1. Introduction

NOOUGAT proposes learnt tracklet association across non-overlapping windows, replacing the overlap-and-stitch approach used by SUSHI f.ex, where windows share frames and the resulting trajectories are matched with the Hungarian algorithm. This leads to plenty of redundant computation, with a stride of T/2 most frames get processed twice, and earlier methods like MPNTrack process each frame up to 15 times.

Here is a notebook covering SUSHI which would give some intuition to this extension:

Read next: Unifying Short and Long-Term Tracking with Graph Hierarchies

More importantly, the subclip size T is a tunable knob. Set T=1 and the model tracks frame-by-frame like an online tracker; set it to 256 and it reasons over long temporal context like an offline one. Same weights, and same architecture. That's the "novelty" of this paper, being the first tracker designed to operate with arbitrary temporal horizons.

Both regimes have different requirements. F.ex, autonomous driving requires online processing, where the model needs to act on frame-by-frame information, only using past information. Whereas video analysis of "tour de france" can be performed offline, allowing access to future information, which can help with occlusion, and disappearance/reappearance of the object detections (cyclists).

Online tracking depends heavily on hand-crafted heuristic association, meaning it isn't end-to-end. End-to-end trackers do exist (MOTR, MeMOTR), but they jointly learn detection and tracking, which makes them heavy and data-hungry — MeMOTR has 51M trainable parameters against NOOUGAT's 27K, since NOOUGAT takes off-the-shelf detections and only learns association.

Offline methods have been using learned approaches, with GNNs (Graph Neural Networks), where they show good performance by learning the associations from the data, instead of it being hand-stitched. There is still a hurdle however, and that is the manual stitching between the trajectories from those overlapping subclips which I mentioned at the start.

This paper addresses the need for separate models, for online and offline tracking, as well as addressing the need for the subclip association heuristics. They call it their Neural Online and Offline Unified Graph Architecture for Tracking : NOOUGAT.

• Online trackers: They work on past information, then the singular incoming frame with new detections. Then typically, some manual heuristics are performed between the past tracks and the new detections, which are then merged into the global tracks.
• Offline trackers: Have plenty of past and future information, with learnt association, which are then merged to tracks. However, as you can see, the yellow triangles symbolize the subclips, which need overlapping. And this overlapping is where the heuristic stitching occurs.
• Unified tracker: Learnt association inside the subclips, and learnt association across the non-overlapping subclips too. With T=1 this is an online tracker — no separate model needed.

1.1 ALT layer proposal

This learnt association between the non-overlapping subclips is performed via ALT (Autoregressive Long-term Tracking), a GNN layer applied autoregressively. NOOUGAT is built on SUSHI: each subclip is processed by a GNN hierarchy that outputs local tracklets. ALT first merges the tracklets from subclips 1 and 2; that result becomes the past tracks, which ALT then associates with subclip 3's tracklets, and so on until the sequence is covered.

Note that the hierarchy shares weights across its levels and with the ALT layer, which the paper finds cuts convergence from roughly 90k iterations down to 40k. This can be seen in the below flow diagram.

2. Autoregressive Graph Tracking

The current approach to graph tracking right now, is via a sliding window inference. Given a subclip with fixed size of frames, then the sliding window needs a stride that respects . This then gives us a set of subclips where each clip is processed independently by the graph solver.

Given a video sequence of frames, where we have overlapping subclips, our would look as follows:

The overlap is a necessary condition here as it's what lets the tracker stitch the outputs of neighbouring subclips together, usually with linear programming. But it leads to a lot of redundant computation. SUSHI sets , so the majority of frames are processed twice; MPNTrack uses with , meaning each frame is processed up to 15 times.

NOOUGAT partitions the sequence into non-overlapping subclips, simply:

Every subclip is processed by the GNN hierarchy, and then merged autoregressively with the ALT (autoregressive long-term tracking) module. Each frame is therefore processed exactly once.

Now some notations the paper has:

: The set of trajectories produced by ALT up to subclip

: The set of trajectories produced by the hierarchical GNN for the subclip . Meaning, the local tracklets that are formed BEFORE the autoregressive merging step.

We begin the architecture with:

Then at each step , we update the trajectories with:

Essentially, is formed by associating the previous trajectories with the incoming tracklets . After steps, we get the final set of tracks covering the whole sequence.

As already mentioned, the nodes in the graph represent trajectories, whereas the edges connect them. So builds a graph over the union of past and incoming tracks. In the original SUSHI formulation, every node can connect to every other, past or future:

Real-time applications are stricter: associations must use past information only, and once a decision is made it can't be revised. This imposes a bipartite constraint, where past nodes connect only to incoming nodes, so prior associations can never be modified. In that case ALT builds :

The ALT layer supports both formulations, which is what lets NOOUGAT cover real-time and batch processing with one architecture which makes the comparison against existing online trackers a fair one.

3. Unified Graph Architecture

3.1 Unified Node and Edge features

Since the architecture is fully learnable, the model can pick the most relevant association cues itself. To keep that consistent, NOOUGAT uses a fixed set of input features across all sequences and datasets. Following SUSHI, the initial edge features are geometric, motion and appearance cues.

On top of those, they add two cues borrowed from recent frame-by-frame trackers: trajectory velocity (VDC) and detection confidence.

VDC (Velocity Direction Consistency) was introduced in OC-SORT, and captures the intuition that an incoming detection aligned with the direction of a past tracklet is more likely to be a correct match. It does this by computing the angle between the tracklet's velocity vector and the vector connecting it to a candidate detection. But since the nodes in both the GNN hierarchy and ALT represent trajectories rather than single detections, NOOUGAT extends VDC in the following manner.

For an edge connecting two trajectories , they compute the forward velocity of and the backward velocity of . The forward velocity of is the average displacement over the last detections of the track, normalized to unit length; for , the backward velocity is computed from the first detections. They set across all configurations and datasets. Then, the VDC is given by:

The negation on is what makes the two vectors comparable: flipping the backward velocity points it in the direction of travel, so a perfectly consistent pair gives an angle of 0.

VDC is then added to the initial set of edge features. For detection confidence, they follow SPAMming Labels and add bounding box dimensions and confidence scores as initial node embeddings, originally proposed for node classification, but a clean way to get confidence into the message-passing framework.

Neither cue is the point of the paper, but VDC is worth 1.4 IDF1 in their ablation, even alongside strong appearance and motion cues.

4. Training

ALT has a harder job at train time than a normal graph tracker. Graph-based trackers usually operate on fixed-size clips, so tracklet lengths and occlusion patterns stay fairly uniform. ALT has to generalize across entire sequences, which means training needs to cover the diversity of temporal contexts it'll actually see at inference.

Their recipe works as follows:

1. Sample a random subclip of incoming frames, used to train the GNN hierarchy as in SUSHI.

1. Call the first frame of that subclip . Then sample a random window of past frames , where is the last past frame, and take the ground-truth tracks over it.

ALT is trained to associate those ground-truth past tracklets with the output of the hierarchy.

Three things make this harder on purpose:

• Randomized past — the start time and the duration of the past tracklets are both randomized, which they find improves generalization.
• Skip parameter — a small temporal gap is introduced between and , simulating short occlusions. They set the max gap to 8 frames.
• Track dropout — random past tracks and detections are dropped as augmentation.

Training itself is progressive, again following SUSHI: unfreeze one hierarchy level every 500 iterations, and only once the hierarchy is fully unfrozen does ALT start training. As mentioned in section 1, ALT shares weights with the hierarchy, which brings convergence down from roughly 90k iterations to 40k.

The rest of the setup: focal loss with , Adam, learning rate , weight decay . The online model trains for 100k iterations at batch size 32, the offline one for 50k at batch size 8. For graph construction, each past track is connected to its top 10 nearest neighbours among the incoming tracks by geometry, appearance and motion similarity.

4. Conclusion

NOOUGAT allows for a flexible archictecture, for real time tracking which heavily depends on past information, and offline tracking, equal amounts of past & future info. Where its foundation comes from the SUSHI architecture, essentially adding a layer on top, called ALT (Autoregressive Long-term Tracking), which then associates tracklets between our windows, versus perfomring "manual" stitching, which could for example be making use of the Hungarian Loss.