Detect Anything via Next Point Prediction

Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, Lei Zhang

International Digital Economy Academy (IDEA)

Object detection has long been dominated by traditional coordinate regression-based models, such as YOLO, DETR, and Grounding DINO. Although recent efforts have attempted to leverage MLLMs to tackle this task, they face challenges like low recall rate, duplicate predictions, coordinate misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a 3B-scale MLLM that achieves state-of-the-art object perception performance. On benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot setting. This is enabled by three key designs: 1) Task Formulation: we use special tokens to represent quantized coordinates from 0 to 999, reducing the model's learning difficulty and improving token efficiency for coordinate prediction; 2) Data Engines: we construct multiple data engines to generate high-quality grounding, referring, and pointing data; 3) Training Pipelines: we employ a two-stage training process, combining supervised fine-tuning on 22 million data with GRPO-based reinforcement post-training. Beyond conventional detection, Rex-Omni's inherent language understanding enables versatile capabilities such as object referring, pointing, visual prompting, GUI grounding, spatial referring, OCR and key-pointing, all systematically evaluated on dedicated benchmarks.

Rex-Omni teaser

Introduction

We argue that the performance disparity in MLLM-based object detection primarily arises from two fundamental challenges: (1) MLLMs typically treat coordinate prediction as a discrete classification task, facing significant learning difficulty in mapping discrete tokens to continuous pixel space; (2) teacher-forced SFT creates a mismatch between training and inference, leading to duplicate predictions and coordinate drift during free-form generation.

To overcome these limitations, we propose Rex-Omni, a 3B-scale MLLM that achieves competitive performance with traditional detectors while excelling in language understanding. We introduce a coordinate-token task formulation, design multiple data engines for grounding/referring/pointing, and adopt a two-stage pipeline (SFT on 22M data followed by GRPO-based reinforcement post-training with geometry-aware rewards) to improve localization accuracy and mitigate undesirable behaviors such as duplicate predictions.