ReID API

The encoder protocol, feature bank, and NumPy helpers are included in the base package. For model loading and plotting, install the reid extra (pip install "trackers[reid]"; see the install guide).

This page covers ReIDConfig, the ReIDEncoder protocol, FeatureBank, appearance association helpers, and the threshold-selection plots in trackers.reid. For enabling appearance on BoT-SORT and for benchmark results, see the ReID appearance guide. Model loading and gallery evaluation are in the standalone reid package.

ReIDConfig

trackers.reid.config.ReIDConfig dataclass

Settings for appearance ReID association.

Holds only the values that steer fusion, so any tracker that accepts a ReID encoder can reuse it. The encoder itself is a separate argument, since it is an object rather than a setting.

Attributes:

Name	Type	Description
ema_alpha	float	EMA momentum for track appearance features. Default 0.9.
appearance_threshold	float	Maximum appearance distance, 0.5 * (1 - cosine similarity), allowed to contribute with either fusion method. Default 0.25 (BoT-SORT appearance_thresh). The proximity check still applies in both cases.
proximity_threshold	float	IoU distance gate applied before appearance is used. Default 0.5 (BoT-SORT proximity_thresh; requires IoU >= 1 - proximity_threshold).
fusion	ReidFusionMethod	How appearance combines with box overlap. "botsort" keeps the better of the two scores, so a close appearance match can stand in for weak overlap. "adaptive", from Deep OC-SORT, adds appearance on top of overlap and weights it more when one candidate clearly looks most alike, less when several look alike.
appearance_weight	float	Base appearance weight for adaptive fusion.
adaptive_weight_cap	float	Maximum adaptive appearance bonus.

Raises:

Type	Description
ValueError	If a setting is outside its valid range.

ReIDEncoder

trackers.reid.encoder.ReIDEncoder

Bases: Protocol

Encoder with extract_embeddings(detections, frame).

extract_embeddings(detections, frame)

Return appearance embeddings for each detection box.

Parameters:

Name	Type	Description	Default
detections	Detections	Boxes to embed (xyxy).	required
frame	ndarray	BGR frame the detections were produced on.	required

Returns:

Type	Description
ndarray	Float32 array of shape (N, D), or (0, 0) when empty.

FeatureBank

trackers.reid.feature_bank.FeatureBank

Per-track EMA unit embedding (L2 before and after blend).

Follows the per-track feature update in BoT-SORT (https://github.com/NirAharon/BoT-SORT/blob/main/tracker/bot_sort.py).

Parameters:

Name	Type	Description	Default
alpha	float	EMA momentum, the weight kept on the stored embedding at each update. Default 0.9, as in BoT-SORT.	0.9

feature property

Current stored unit embedding, or None if never updated.

update(embedding, *, skip_normalize=False)

Blend an embedding into the stored unit feature.

Parameters:

Name	Type	Description	Default
embedding	ndarray	Raw embedding, or a unit embedding when skip_normalize is true.	required
skip_normalize	bool	Skip input normalization for validated output from extract_detection_embeddings.	False

Raises:

Type	Description
ValueError	If the embedding is empty, non-finite, or changes shape.

appearance_similarity

trackers.reid.appearance.appearance_similarity(track_features, det_embeddings, *, are_embeddings_normalized=False)

Compute cosine similarities between track and detection embeddings.

Parameters:

Name	Type	Description	Default
track_features	Sequence[ndarray | None]	Sequence of T track features, each with shape (D,). Entries may be None when a track has no appearance feature.	required
det_embeddings	ndarray	Detection embedding matrix with shape (N, D).	required
are_embeddings_normalized	bool	Whether detection rows are already validated unit embeddings from :func:extract_detection_embeddings.	False

Returns:

Type	Description
ndarray	Float32 cosine similarities in [-1, 1] with shape (T, N), where
ndarray	higher means more similar. A track without a feature yet gets a row of
ndarray	zeros, so appearance neither helps nor hurts its matches.

Raises:

Type	Description
ValueError	If detection embeddings are not a finite 2-D matrix, or a track feature is empty, non-finite, or has the wrong dimension.

Example

tracks = [np.array([1.0, 0.0], dtype=np.float32), None] detections = np.array([[1.0, 0.0], [0.0, 1.0]], dtype=np.float32) appearance_similarity(tracks, detections) array([[1., 0.], [0., 0.]], dtype=float32)

extract_detection_embeddings

trackers.reid.appearance.extract_detection_embeddings(model, frame, boxes)

Extract appearance embeddings for detection boxes.

Parameters:

Name	Type	Description	Default
model	ReIDEncoder	Encoder that returns one embedding per detection.	required
frame	ndarray	BGR image with shape (H, W, C).	required
boxes	ndarray	Detection boxes in xyxy format with shape (N, 4).	required

Returns:

Type	Description
ndarray	Float32 embedding matrix with shape (N, D). Returns shape (0, 0)
ndarray	when boxes is empty without calling model.

Raises:

Type	Description
ValueError	If the encoder output is not a finite 2-D matrix or its row count does not match the number of boxes.

Example

class Encoder: ... def extract_embeddings(self, detections, frame): ... return np.ones((len(detections), 2), dtype=np.float32) frame = np.zeros((8, 8, 3), dtype=np.uint8) boxes = np.array([[0.0, 0.0, 4.0, 4.0]], dtype=np.float32) extract_detection_embeddings(Encoder(), frame, boxes) array([[0.70710677, 0.70710677]], dtype=float32)

Choosing a threshold

Measure your own encoder on your own footage instead of inheriting a threshold from a paper. These helpers embed a labeled dataset, sample the distances a tracker actually sees, plot them, and report separability.

Both plot functions take their reference lines as ThresholdLines, either a sequence of values or a mapping from value to annotation.

from trackers.reid import (
    extract_ground_truth_embeddings,
    plot_appearance_distances,
    plot_frame_gap_sweep,
    sample_appearance_distances,
    sweep_frame_gap,
)

embeddings, ids, frame_ids, sequence_ids = extract_ground_truth_embeddings(
    "fastreid_mot17_sbs50", "mot17/val", keep_classes=(1,)
)
distances = sample_appearance_distances(embeddings, ids, frame_ids, sequence_ids)
same_id_rate, different_id_rate = distances.rates_at(0.25)
plot_appearance_distances(distances, thresholds={0.20: "selected", 0.25: "default"})
plot_frame_gap_sweep(sweep_frame_gap(embeddings, ids, frame_ids, sequence_ids))

extract_ground_truth_embeddings

trackers.reid.appearance.extract_ground_truth_embeddings(model, dataset_root, *, sequences=None, keep_classes=None, frame_stride=1)

Embed every ground-truth crop in a MOT-format dataset.

Walks {dataset_root}/{sequence}/gt/gt.txt against the frames in {dataset_root}/{sequence}/img1. Rows flagged ignore (confidence 0) are always dropped. Identities are renumbered across sequences, so the same track number in two videos stays two identities.

The four returned arrays are what :func:trackers.reid.sample_appearance_distances expects, which is how an appearance_threshold gets calibrated on a new dataset.

Parameters:

Name	Type	Description	Default
model	ReIDEncoder | str	Encoder to embed crops with, or a model name to load.	required
dataset_root	str | Path	Directory holding one folder per sequence.	required
sequences	Sequence[str] | None	Sequences to read. Defaults to every one found.	None
keep_classes	Sequence[int] | None	MOT class ids to keep, e.g. (1,) for MOT17 pedestrians. Defaults to every class, which suits single-class datasets such as SoccerNet.	None
frame_stride	int	Embed every Nth frame. Raising it trims a dense dataset, at the cost of the smallest frame gaps.	1

Returns:

Type	Description
tuple[ndarray, ndarray, ndarray, ndarray]	(embeddings, ids, frame_ids, sequence_ids), aligned row-wise.

Raises:

Type	Description
FileNotFoundError	If no sequence under dataset_root has a gt/gt.txt, or a named sequence is missing one.
ValueError	If no crop survived the filters.

AppearanceDistances

trackers.reid.threshold_selection.AppearanceDistances dataclass

Sampled appearance distances for one frame-gap band.

Distances are 0.5 * (1 - cosine_similarity), the term both fusion methods compare with ReIDConfig.appearance_threshold.

Attributes:

Name	Type	Description
same_id	ndarray	Distances between two crops of the same identity.
different_id	ndarray	Distances between crops of two different identities.
minimum_frame_gap	int	Lower bound of the band the pairs were drawn from.
maximum_frame_gap	int	Upper bound of the band the pairs were drawn from.

label property

Gap band as an axis label, e.g. "1" or "6-15".

roc_auc property

Threshold-free separability of the two classes.

See :func:roc_auc.

rates_at(threshold)

Return the match rate of both classes at one candidate threshold.

A pair counts as accepted at or below the threshold, matching both fusion methods. Appearance is excluded only once the distance exceeds ReIDConfig.appearance_threshold.

Parameters:

Name	Type	Description	Default
threshold	float	Candidate ReIDConfig.appearance_threshold.	required

Returns:

Type	Description
float	(same_id_rate, different_id_rate): the fraction of same-ID pairs
float	accepted, to be maximised, and the fraction of different-ID pairs
tuple[float, float]	accepted, to be minimised.

sample_appearance_distances

trackers.reid.threshold_selection.sample_appearance_distances(embeddings, ids, frame_ids, sequence_ids, *, same_id_pairs=5000, different_id_pairs=5000, minimum_frame_gap=1, maximum_frame_gap=30, seed=0)

Sample same-ID and different-ID crop pairs inside one frame-gap band.

Anchors are filtered to those that have a candidate in the active gap band before sampling, and every sequence holding any gets an equal quota. Within a sequence, same-ID pairs pick an identity uniformly so that long tracks cannot dominate, different-ID pairs pick an anchor crop uniformly, and both then pick a candidate uniformly inside the band.

Parameters:

Name	Type	Description	Default
embeddings	ndarray	Appearance embeddings, shape (N, D). Normalised here, so either raw or unit-length input works.	required
ids	ndarray	Hashable scalar identity label per embedding, shape (N,).	required
frame_ids	ndarray	Frame number per embedding, shape (N,).	required
sequence_ids	ndarray	Hashable scalar sequence or video label per embedding, shape (N,).	required
same_id_pairs	int	Same-ID pairs to draw across all sequences.	5000
different_id_pairs	int	Different-ID pairs to draw across all sequences.	5000
minimum_frame_gap	int	Smallest allowed frame gap. Must be at least 1: a gap of 0 lets a crop pair with itself, which is what puts the spike at distance 0 in the original BoT-SORT figure.	1
maximum_frame_gap	int	Largest allowed frame gap, i.e. the association horizon being measured.	30
seed	int	Seed for the pair-drawing generator.	0

Returns:

Type	Description
AppearanceDistances	The sampled distances for this band.

Raises:

Type	Description
TypeError	If identity or sequence labels are not hashable.
ValueError	If the arrays disagree in length, the gap band is invalid, or either class yielded no pairs at all.

sweep_frame_gap

trackers.reid.threshold_selection.sweep_frame_gap(embeddings, ids, frame_ids, sequence_ids, *, gap_bands=DEFAULT_FRAME_GAP_BANDS, pairs_per_class=5000, seed=0)

Measure separability inside each frame-gap band in turn.

A threshold tuned on consecutive frames says nothing about re-finding a track after an occlusion, so this reports how far the same threshold carries as the gap widens. Bands that hold no pairs are skipped rather than raising, since a short dataset legitimately has no 240-frame gaps.

Parameters:

Name	Type	Description	Default
embeddings	ndarray	Appearance embeddings, shape (N, D).	required
ids	ndarray	Identity label per embedding.	required
frame_ids	ndarray	Frame number per embedding.	required
sequence_ids	ndarray	Sequence or video label per embedding.	required
gap_bands	Sequence[tuple[int, int]]	(minimum, maximum) frame gaps to measure.	DEFAULT_FRAME_GAP_BANDS
pairs_per_class	int	Pairs drawn per class within each band.	5000
seed	int	Seed for the pair-drawing generator.	0

Returns:

Type	Description
list[AppearanceDistances]	One entry per band that yielded pairs, in gap_bands order.

Raises:

Type	Description
ValueError	If the input arrays or a requested gap band are invalid.

roc_auc

trackers.reid.threshold_selection.roc_auc(same_id, different_id)

Return the probability that a same-ID pair scores closer than a different-ID pair.

Ties count as half. Equivalent to the area under the curve traced by sweeping the threshold from 0 to 1 and plotting the two rates from :meth:AppearanceDistances.rates_at against each other, which is why it summarises every threshold instead of one chosen operating point.

Parameters:

Name	Type	Description	Default
same_id	ndarray	Distances between crops of the same identity.	required
different_id	ndarray	Distances between crops of different identities.	required

Returns:

Type	Description
float	1.0 when every same-ID pair is closer than every different-ID pair,
float	0.5 when appearance carries no information, and 0.0 when the two
float	classes are ordered the wrong way round.

Raises:

Type	Description
ValueError	If either distance array is empty.

plot_appearance_distances

trackers.reid.plots.plot_appearance_distances(distances, *, thresholds=(0.25,), title=None)

Plot the two distance distributions with candidate thresholds marked.

Where the two histograms overlap is where no threshold can separate them.

Parameters:

Name	Type	Description	Default
distances	AppearanceDistances	Output of :func:sample_appearance_distances.	required
thresholds	ThresholdLines	Candidate thresholds to draw as vertical reference lines. Pass a mapping to annotate them, e.g. {0.20: "selected", 0.25: "default"}. The first is drawn dashed black and the rest recede into grey.	(0.25,)
title	str | None	Figure title. Defaults to naming the gap band that was sampled.	None

Returns:

Type	Description
Figure	The figure the distances were drawn on.

plot_frame_gap_sweep

trackers.reid.plots.plot_frame_gap_sweep(sweep, *, thresholds=(0.25,), percentiles=(10, 90), title=None)

Plot how separability degrades as the frame gap widens.

The upper panel tracks each class's median and percentile band against the gap; the lower panel tracks :func:roc_auc, which answers the same question without committing to a threshold. Note that the two panels do not measure the same thing: the shaded bands can sit clear of each other while the AUC is still short of 1.0, because percentile ranges ignore where the mass sits.

Parameters:

Name	Type	Description	Default
sweep	Sequence[AppearanceDistances]	Output of :func:sweep_frame_gap.	required
thresholds	ThresholdLines	Candidate thresholds to draw as horizontal reference lines. Pass a mapping to annotate them, e.g. {0.20: "selected", 0.25: "default"}.	(0.25,)
percentiles	tuple[int, int]	(low, high) bounds of the band shaded around each class's median. Keep it symmetric so both classes are read the same way.	(10, 90)
title	str | None	Figure title.	None

Returns:

Type	Description
Figure	The figure the sweep was drawn on.

Raises:

Type	Description
ValueError	If sweep is empty.