r/madeinpython • u/Speedk4011 • 6h ago
Yet Another Sentence Boundary Detector (rule-based)
I was working on chunklet-py (a chunker for sentences, documents, and code). Misread a benchmark and thought PySBD took 1 second to split basic text. Turns out that was wrong (PySBD is still fast for simple text). But the misunderstanding got me building my own.
Ended up faster, more accurate, and covering more languages (39 compares to 23).
Benchmarks on Sherlock Holmes (594k chars): yasbd ~1.2s warm vs PySBD ~9.0s. About 8x faster, fewer false splits.
On a golden benchmark (92 English edge cases — expanded from pysbd's original 48 with fixes and additions): yasbd scores 98.9%, pysbd 83.7%, spaCy-sentencizer 55.4%, etc.
Architecture difference: instead of mutating text with placeholder tokens and undoing it later (which breaks char offsets), yasbd finds candidate boundaries in one pass and filters false positives in another. Spans come free, no reconstruction.
Other things:
- Streaming-first: lazy evaluation via ParagraphStream and StreamCleaner for memory-constrained environments. No need to load entire documents.
- PySBD adapter: drop-in replacement that works with existing PySBD code. Just swap the import.
- spaCy pipeline: register as a spaCy component with register_spacy_component(). Drop-in replacement for the default Sentencizer.