LocateAnything-3B: Fast Visual Grounding with Parallel Box Decoding
LocateAnything-3B is a 3‑billion‑parameter vision‑language model released by NVIDIA (part of the Eagle VLM family) for high‑quality visual grounding. It accepts an image and a natural‑language prompt and returns text that encodes semantic labels together with structured bounding‑box or point coordinates. The model uses a novel Parallel Box Decoding (PBD) approach that predicts complete boxes in a single parallel step, delivering up to 2.5× higher throughput than autoregressive decoders. It is built on the Qwen2.5-3B-Instruct language model and a MoonViT vision encoder, integrated via a multimodal MLP projector.
The model is trained on a multi‑domain dataset of 12 M images, 138 M+ queries, and 785 M bounding boxes covering natural scenes, robotics, driving, GUI interaction, and document understanding. Consequently it supports a wide range of tasks: open‑set object detection, dense multi‑object detection, referring‑expression grounding, GUI element grounding, OCR/layout localization, and point‑based spatial reasoning. Inference can run in fast, slow, or hybrid modes, with a recommended hybrid setting (max_new_tokens=8192) for a balance of speed and accuracy. It runs on NVIDIA GPUs (Ampere, Hopper, Blackwell, Lovelace) using the Transformers library, with optional MagiAttention for further speed gains.
LocateAnything-3B is released under an NVIDIA non‑commercial license, permitting academic and non‑profit research use only. The model’s code and demo are available via a Hugging Face Space and a GitHub repository (NVlabs/Eagle/Embodied). With over one million downloads and a high trending score, it is rapidly becoming a go‑to foundation for developers building multimodal perception systems that require precise and efficient visual grounding.
Project Ideas
- Create an interactive GUI assistant that receives a screenshot and a natural‑language command (e.g., "click the Submit button") and returns the bounding box of the target UI element for automated clicking.
- Build an automated dataset annotation pipeline that takes raw images and textual prompts to generate bounding‑box labels for object detection training data.
- Develop a real‑time robotics perception module that uses LocateAnything-3B in hybrid mode to locate objects or landmarks from camera feeds based on spoken instructions.
- Implement a document layout analysis tool that grounds queries like "show the table on page 2" by returning box coordinates of tables, figures, and headings within scanned PDFs.
- Design a multimodal image search engine that returns image regions matching user queries such as "find all red cars" by extracting and ranking the predicted bounding boxes.