Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features by @abyste.bsky.social & @jlibovicky.bsky.social aclanthology.org/2026.finding... TL;DR: Morpho-syntax features replace gold segmentation data for tokenization eval.

👉 What do we do? We use the good old IBM1 model to align subwords with morphological features from Unimorph and we show it captures the same thing as morpheme boundary recall. 👉 Why it matters? For many languages good segmentation data is missing. Morphological features are more widely available.

We (= mostly @abyste.bsky.social) developed a way to evaluate how morphological a #tokenization is w/o gold segmentation labels. arxiv.org/abs/2601.18536 The key: align subword tokens with morphological features from UniMorph using IBM Model 1. To appear in EACL 2026 Findings.

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentati...

arxiv.org