New table: 200k rows x 300d, full Persian Wikipedia dump, 5 epochs,
vs md's 50k rows / 400k documents.
- configs/fa_ner_lg.cfg, project.yml ent-lg workflow (vectors-lg
through smoke-ent-lg)
- scripts/compare_tiers.py: generalized sm/md pair to N tiers; ent
NER test now includes lg; fixed sm baseline to the file that's
actually scored (perdt-ner-test.json, not the missing ent-test.json)
- scripts/finalize_pipeline.py: FLORET_LG source and vectors_note_lg
corrected to full Wikipedia, 5 epochs (were a generic Wikipedia +
OSCAR placeholder)
- docs/MODELS.md §7: PerDT NER test ENTS_F 75.94 (sm 71.87, md
74.71), full per-label table, cost (217 MB wheel)
Not built: fa_dep_news_lg / fa_core_news_lg.
- New ## Named entity recognition section: labels, silver-provenance,
per-label breakdown table, package options. Moved out of ## Results
(which now just points to it) and out of ## Caveats (silver-label and
thin-label bullets folded into the new section's prose instead of
repeating them).
- Added fa_ent_news_sm to the package table (built, was undocumented).
- Design decision 4 no longer repeats the NER transfer methodology now
that it has a dedicated home.
- Dropped the warning-emoji decoration on the comparability note.
- project.yml: new md workflow (vectors-md, train-dep-md, train-ner-md,
finalize-dep-md, assemble-core-md, evaluate-md, finalize-meta-md,
compare-md, package-md, smoke-md), same corpus/architecture as sm plus
the fa_floret static vector table (50k rows x 300d, 400k Persian
documents).
- configs/fa_dep_news_md.cfg, configs/fa_ner_md.cfg: byte-identical to the
sm configs except include_static_vectors, isolating what the vectors buy.
- scripts/unpack_vectors.py: extracts a floret wheel's vectors-only
pipeline into a directory --paths.vectors can point at.
- scripts/finalize_pipeline.py: --size now accepts md (floret source +
vectors note), plus lg/trf (used by the pending Colab notebook on
colab-lg-trf-training; lg's vectors note is generated from the trained
model's actual vector table shape since that tier is still being
iterated on).
- scripts/compare_tiers.py: sm vs md metrics diff.
- docs/MODELS.md, README.md, README.fa.md: md tier results, fa_ent_news_md
package row, and the analysis of why floret helps NER recall.
- docs/CONTRIBUTING-GUIDE.md, README.md, README.fa.md: fixed every
documented pip install URL. spacy huggingface-hub push names the
uploaded wheel '<name>-any-py3-none-any.whl'; 'any' is not a valid PEP
440 version, so current pip rejects it. Re-uploaded a correctly
versioned copy of every published wheel (fa_core_news_sm, fa_dep_news_sm,
fa_ent_news_md) to the Hub and repointed the docs at that filename.
Described the shipped component as trained on ParsTwiNER (67.22 F); it has
trained on PerDT's own NER layer (71.87 F) since the treebank's NER layer was
found. Also updated the fa_core_news_sm reference from future tense to the
assemble-core step that already sources this component into it.
- Title now 'Persian (Farsi) pipelines for spaCy'
- Section order: intro, Results, Install, Caveats, Build, Design decisions,
Why not hazm, More -- numbers before any provenance argument
- Install points at published HF wheels instead of local packages/*.whl
paths, verified against a clean venv
- Dropped ## Roadmap (mirrored in gitignored TODO.md) and
## Licensing drove most decisions here (superseded by docs/MODELS.md)
- Trimmed provenance/build-trivia prose; kept every score, the hazm
comparison, and the four design decisions verbatim
UD_Persian-PerDT ships entity annotations in not-to-release/Dadegan with NER
tag/ that nothing in this project had looked at: 29,107 sentences, 484,312
tokens, 15,833 entities, under the treebank's own CC BY-SA 4.0. That removes the
reason core was withheld. The previous commit split dep from ent because the only
redistributable Persian NER corpus known then was ParsTwiNER, a Twitter corpus
scoring 67.22 F against 85-98 for the UD components, and hiding that genre and
quality gap behind one package name was not acceptable. A NER layer from the same
corpus has none of those problems: one genre, one tokenization, one licence, one
provenance chain.
Two shipping packages, both from PerDT alone:
fa_dep_news_sm 7.5 MB TAG 95.96 LEMMA 97.91 LAS 85.15
fa_core_news_sm 13 MB the above plus ENTS_P/R/F 77.67 / 66.87 / 71.87
Per label: LOC 80.24, DAT 74.45, MON 73.68, ORG 68.77, TIM 66.67, PER 65.29,
PCT 57.14. PER scoring below LOC on comparable data is the silver labels showing:
PerDT starts 6.24% of PER spans with a title against ParsTwiNER's 1.41%, so the
boundaries are less regular than the count suggests. MON, TIM and PCT rest on 4
to 11 test entities and are indicative only.
The labels are silver, from Beheshti-NER (Taher et al. 2020) with manual
corrections per the treebank README. meta.json notes say so. A human-annotated
test set is the outstanding work, tracked in ../ner_dataset.
Alignment: the NER files use the original Dadegan tokenization, matching the
released UD tokenization in only 57 to 62% of sentences (dropped copulas and
auxiliaries, one honorific corrupted to a comma). scripts/transfer_perdt_ner.py
realigns with difflib at 99.86% train / 99.74% dev / 99.51% test; spans whose
tokens do not all map contiguously are dropped rather than guessed.
Also fixed a silent metadata corruption. Folding both benchmark reports over one
key set gave core tag_acc 0.00, because the NER corpus has no gold tags so its
report carries tag_acc: 0.0, which overwrote the real 95.96; sents_f was wrong
the same way. finalize_pipeline.py now takes --ud-metrics and --ner-metrics
separately, each restricted to the keys its corpus can evidence. Its variant
guards work both directions now: a dep pipeline may not contain ner, a core one
must.
ParsTwiNER moves out of the shipping repo to a future mixed-genre package, with
the ablation that justifies it: prose-trained NER scores 45.72 F on tweets,
tweet-trained scores 55.59 on prose, and mixing gives 72.11 / 66.49, so roughly
20 F of cross-genre robustness for under 1 F on prose.
Adds LICENSE recording the split: MIT for the code, CC BY-SA 4.0 for the trained
pipelines as Adapted Material, with the attribution, modification notice and
warranty disclaimer that CC BY-SA 4.0 section 3 requires. Verified the treebank's
LICENSE.txt is unmodified CC BY-SA 4.0 with no carve-out for not-to-release/.
Author metadata filled in; both packages rebuilt and installed from their wheels.
spaCy's naming scheme encodes contents: dep = tagger+parser+lemmatizer,
ent = NER only, core = both. Shipping a single fa_core_news_sm implied the NER
was held to the same standard as the rest of the pipeline. It is not, and the
numbers are not close:
UD components (PerDT, edited prose): TAG 95.96 LEMMA 97.91 LAS 85.15
ner (ParsTwiNER, tweets): ENTS_F 67.22
One package name and one version number would paper over that. So the treebank
components ship as fa_dep_news_sm (CC BY-SA 4.0, 7.5 MB) and NER ships as an
opt-in fa_ent_news_sm (MIT, 5.6 MB). Users now choose the weak component
deliberately instead of inheriting it.
Counting ParsTwiNER settles what "weak" means. It is not a small corpus --
232,917 tokens and 16,250 entities, the same order as the restricted ARMAN and
PEYMA. But the labels are skewed: PER 6258, LOC 5478, ORG 2694, NAT 939,
EVE 482, POG 399. The two starved labels are exactly the two scoring worst
(EVE 30.0, POG 41.2). Head labels suffer genre mismatch, tail labels suffer data
starvation -- two problems needing two different fixes. Recorded in README and
docs/MODELS.md.
fa_core_news_sm is reserved, not abandoned. ../ner_dataset/PLAN.md targets
prose-genre NER data that must beat ParsTwiNER on a human-annotated test set
before the name gets used. Nothing technical blocks the merge: the ner component
already embeds its own tok2vec instead of a Tok2VecListener, so
dep.add_pipe("ner", source=spacy.load("fa_ent_news_sm"))
reassembles a core-equivalent pipeline at runtime -- verified, not assumed.
Mechanics:
- scripts/assemble_core.py -> scripts/finalize_pipeline.py, now variant-aware
(dep|ent) and refusing to publish a dep pipeline containing an ner component,
so the split cannot silently regress.
- published meta.json["performance"] now comes from a strict whitelist. It was
inheriting raw *_loss values and a bogus tag_micro_f: 0.0 from training meta.
- project.yml split into two independent workflows, `all` and `ner`; all 16
commands dry-run clean.
- configs/fa_core_news_sm.cfg -> configs/fa_dep_news_sm.cfg.
- smoke_test.py degrades gracefully on pipelines lacking DEP/MORPH/ner.
Also folded into ../ner_dataset/PLAN.md: PerDT's XPOS encodes animacy on proper
nouns (N_ANM 6752, N_IANM 12682). Animate PROPN is a strong free prior for PER,
shrinking the annotation task to splitting inanimate PROPN into LOC/ORG, and
giving a gold-grounded cross-check that beats the model's self-reported
confidence for routing items to human review.
Committed unsigned: the OpenPGP smartcard holding 05E227BF4D6736DE is not
present (gpg: selecting card failed: No such device).
No trained Persian pipeline exists for spaCy: spacy.load("fa_core_news_sm") has
never worked. This builds one from openly-licensed data so the result can actually
be redistributed.
Test scores (held-out splits): TAG 95.96, POS 96.24, MORPH 96.29, LEMMA 97.91,
UAS 89.69, LAS 85.15, ENTS_F 67.22. ~9,250 words/s, 13 MB wheel. 1h27m on 4 CPU
cores, no GPU.
Corpus choices, with evidence:
- UD_Persian-PerDT (CC BY-SA 4.0) over Seraji: 3.7x more tokens (452k vs 121k) and
Seraji has no PROPN tag at all. hazm's own spaCy parser used PerDT too.
- ParsTwiNER (MIT) for NER. ARMAN/PEYMA/NSURL are research-only; spaCy never
shipped the 2018 Persian models precisely because of corpus licensing. Cost: NER
is trained on tweets, so it is the weak component.
- --merge-subtokens: measured token F 0.9887 vs 0.9823 split. Without it, 1.5% of
gold token boundaries are unreachable by the tokenizer we ship.
- morphologizer + trainable_lemmatizer instead of the English attribute_ruler +
rule lemmatizer, because UD gives gold UPOS/FEATS/lemmas to train and measure on.
The ner component embeds its own tok2vec rather than using a Tok2VecListener, so it
can be sourced into the core pipeline after being trained on a separate corpus.
Also documents an upstream bug: spacy/lang/fa/syntax_iterators.py matches ClearNLP
labels (dobj, pobj, nsubjpass, attr, dative) that do not exist in UD, so
doc.noun_chunks returns bare head nouns (1.31 vs 2.77 tokens/chunk). Patch and
tests in docs/upstream/fa-noun-chunks.md.
Mohamad Fazeli