We're excited to announce that our research paper on pico-type has been published on ArXiv (2608.14658): "pico-type: A 1.5M-Parameter Byte-Level Multi-Head Content Classifier". This paper presents our work on creating an ultra-compact, tokenizer-free neural architecture that runs in under 6ms on consumer CPUs.
The Problem with Tokenizer-Based Classifiers
Standard language models and text classifiers rely on subword tokenizers (WordPiece, BPE, SentencePiece). While effective for large models, tokenizers introduce significant overhead for edge and high-throughput pipelines: dictionary lookups, out-of-vocabulary artifacts, and multi-megabyte vocabulary files that dwarf the model itself.
The Byte-Level Architecture
pico-type eliminates tokenization entirely. Operating directly on raw UTF-8 byte streams, the model processes text through a compact embedding layer and multi-head representation blocks, outputting predictions across multiple task heads simultaneously:
- Language Detection: 62 programming languages
- MIME Classification: 90 file/content MIME types
- Binary vs. Text: Robust binary stream differentiation
Latency and Resource Efficiency
With only 1.5 million parameters and a total ONNX binary size of under 200KB, pico-type executes inference in less than 6ms on a single CPU core without GPU acceleration or heavy dependencies. This makes it drop-in ready for git pre-commit hooks, edge gateways, high-speed ingestion pipelines, and air-gapped embedded environments.
Availability
The paper is open on ArXiv (2608.14658), and the model checkpoints are open-sourced on HuggingFace under the Apache 2.0 license.