UD_Persian-PerDT ships entity annotations in not-to-release/Dadegan with NER
tag/ that nothing in this project had looked at: 29,107 sentences, 484,312
tokens, 15,833 entities, under the treebank's own CC BY-SA 4.0. That removes the
reason core was withheld. The previous commit split dep from ent because the only
redistributable Persian NER corpus known then was ParsTwiNER, a Twitter corpus
scoring 67.22 F against 85-98 for the UD components, and hiding that genre and
quality gap behind one package name was not acceptable. A NER layer from the same
corpus has none of those problems: one genre, one tokenization, one licence, one
provenance chain.
Two shipping packages, both from PerDT alone:
fa_dep_news_sm 7.5 MB TAG 95.96 LEMMA 97.91 LAS 85.15
fa_core_news_sm 13 MB the above plus ENTS_P/R/F 77.67 / 66.87 / 71.87
Per label: LOC 80.24, DAT 74.45, MON 73.68, ORG 68.77, TIM 66.67, PER 65.29,
PCT 57.14. PER scoring below LOC on comparable data is the silver labels showing:
PerDT starts 6.24% of PER spans with a title against ParsTwiNER's 1.41%, so the
boundaries are less regular than the count suggests. MON, TIM and PCT rest on 4
to 11 test entities and are indicative only.
The labels are silver, from Beheshti-NER (Taher et al. 2020) with manual
corrections per the treebank README. meta.json notes say so. A human-annotated
test set is the outstanding work, tracked in ../ner_dataset.
Alignment: the NER files use the original Dadegan tokenization, matching the
released UD tokenization in only 57 to 62% of sentences (dropped copulas and
auxiliaries, one honorific corrupted to a comma). scripts/transfer_perdt_ner.py
realigns with difflib at 99.86% train / 99.74% dev / 99.51% test; spans whose
tokens do not all map contiguously are dropped rather than guessed.
Also fixed a silent metadata corruption. Folding both benchmark reports over one
key set gave core tag_acc 0.00, because the NER corpus has no gold tags so its
report carries tag_acc: 0.0, which overwrote the real 95.96; sents_f was wrong
the same way. finalize_pipeline.py now takes --ud-metrics and --ner-metrics
separately, each restricted to the keys its corpus can evidence. Its variant
guards work both directions now: a dep pipeline may not contain ner, a core one
must.
ParsTwiNER moves out of the shipping repo to a future mixed-genre package, with
the ablation that justifies it: prose-trained NER scores 45.72 F on tweets,
tweet-trained scores 55.59 on prose, and mixing gives 72.11 / 66.49, so roughly
20 F of cross-genre robustness for under 1 F on prose.
Adds LICENSE recording the split: MIT for the code, CC BY-SA 4.0 for the trained
pipelines as Adapted Material, with the attribution, modification notice and
warranty disclaimer that CC BY-SA 4.0 section 3 requires. Verified the treebank's
LICENSE.txt is unmodified CC BY-SA 4.0 with no carve-out for not-to-release/.
Author metadata filled in; both packages rebuilt and installed from their wheels.
No trained Persian pipeline exists for spaCy: spacy.load("fa_core_news_sm") has
never worked. This builds one from openly-licensed data so the result can actually
be redistributed.
Test scores (held-out splits): TAG 95.96, POS 96.24, MORPH 96.29, LEMMA 97.91,
UAS 89.69, LAS 85.15, ENTS_F 67.22. ~9,250 words/s, 13 MB wheel. 1h27m on 4 CPU
cores, no GPU.
Corpus choices, with evidence:
- UD_Persian-PerDT (CC BY-SA 4.0) over Seraji: 3.7x more tokens (452k vs 121k) and
Seraji has no PROPN tag at all. hazm's own spaCy parser used PerDT too.
- ParsTwiNER (MIT) for NER. ARMAN/PEYMA/NSURL are research-only; spaCy never
shipped the 2018 Persian models precisely because of corpus licensing. Cost: NER
is trained on tweets, so it is the weak component.
- --merge-subtokens: measured token F 0.9887 vs 0.9823 split. Without it, 1.5% of
gold token boundaries are unreachable by the tokenizer we ship.
- morphologizer + trainable_lemmatizer instead of the English attribute_ruler +
rule lemmatizer, because UD gives gold UPOS/FEATS/lemmas to train and measure on.
The ner component embeds its own tok2vec rather than using a Tok2VecListener, so it
can be sourced into the core pipeline after being trained on a separate corpus.
Also documents an upstream bug: spacy/lang/fa/syntax_iterators.py matches ClearNLP
labels (dobj, pobj, nsubjpass, attr, dative) that do not exist in UD, so
doc.noun_chunks returns bare head nouns (1.31 vs 2.77 tokens/chunk). Patch and
tests in docs/upstream/fa-noun-chunks.md.
Mohamad Fazeli