Text-prompted segmentation and page OCR from TII
Falcon-Perception is the inference code for two vision-language models from Abu Dhabi's Technology Innovation Institute (TII). Falcon Perception (0.6B parameters) takes an image and a plain-language query and returns bounding boxes and pixel masks for whatever the query describes. Falcon OCR (270M parameters) uses the same architecture to turn a page into text, LaTeX formulas or HTML tables, and its engine is built on top of the Perception one.
The open source OCR tools comparison puts Falcon OCR next to classic OCR engines and document-to-Markdown converters.
Paged KV cache on CUDA, MLX on a Mac
- Plain-language queries: "the cat on the left" comes back as boxes and pixel-level masks.
- Apache 2.0 for code and weights: both models are on Hugging Face under the same license as the repo.
- Paged engine: paged KV cache, continuous batching, CUDA graphs, and preemption that pauses running sequences when memory runs short. Attention goes through PyTorch FlexAttention, with no hand-written kernel.
- MLX backend: on a Mac, it needs neither PyTorch nor transformers.
- End-to-end OCR by default: the whole page goes to the model. The layout mode, which the README keeps for very dense pages such as multi-column newspapers, also downloads PP-DocLayoutV3, a layout model that comes from PaddleOCR.
Track objects, transcribe receipts, serve an OpenAI-compatible API
- Find an object described in words in a photo, or track it through a video (one of the repo's notebooks does this).
- Transcribe receipts, handwritten notes or scans, formulas and tables included.
- Serve Falcon OCR behind an OpenAI-compatible API with the Docker image, which also takes PDFs.
Unauthenticated REST server, NVIDIA or Apple Silicon hardware
Falcon-Perception expects an NVIDIA GPU on Linux or an Apple Silicon Mac. On Linux the recommended uv install pulls PyTorch built for CUDA 12.8, which needs NVIDIA driver 570 or newer. The README says nothing about CPU or AMD GPUs. The server and the paged engine run on CUDA; on a Mac, MLX only provides the simpler batch engine. The code sizes its cache from the card's memory in tiers starting at 15 GiB, yet one user on two RTX 3090s saw the server ask for 56 GiB at startup (issue #5, open since April).
The REST server has no documented authentication and binds to 0.0.0.0 by default. On September 30, 2026, a reporter publicly announced an unauthenticated, remotely reachable flaw in the HTTP API (issue #37) and held back the details because the repo has no SECURITY.md or private reporting channel. As of October 7, nobody has replied.
TII pitches Falcon Perception as a single Transformer for image and text, instead of a vision encoder bolted onto a text decoder. Its launch post admits Falcon Perception still trails SAM 3 at deciding whether the queried object is in the image at all. On the OCR side, the README puts Falcon OCR 1.5 at about a third the size of PaddleOCR-VL and about a twentieth of Chandra 2, the model behind Chandra, whose weights ship under a modified OpenRAIL-M license that restricts commercial use.
The audience is people who want to read or change a VLM inference engine. The README calls the engine "minimal, readable yet performant". For OCR-only production work it points to the vLLM Docker image instead. The Perception model can't run on vLLM, according to the maintainer.
The project is in beta and has shipped one release: 1.0.0, on PyPI on April 7, 2026. The Falcon OCR 1.5 changes (September 11, 2026) live only on the main branch, and the README installs from source. Fine-tuning code, which the maintainer said was in progress on April 8, 2026, is still unpublished.