spacy-fa-pipeline/docs/MODELS.md

The Persian pipelines: what to build, from what, and why

1. What the four English pipelines are

One pipeline design at four embedding budgets. Every one has the same components; the axis of variation is where token representations come from.

	en_core_web_sm	en_core_web_md	en_core_web_lg	en_core_web_trf
Size on disk	12 MB	31 MB	382 MB	436 MB
Embeddings	hash embeddings only	685k keys / 20k vectors (300d)	685k keys / 343k vectors (300d)	roberta-base, 768d contextual
Components	tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner	same	same	transformer, tagger, parser, attribute_ruler, lemmatizer, ner
TAG_ACC	0.97	0.97	0.97	0.98
DEP_UAS / LAS	0.92 / 0.90	0.92 / 0.90	0.92 / 0.90	0.95 / 0.94
ENTS_F	0.84	0.85	0.86	0.90
Training data	OntoNotes 5 (+ ClearNLP dep conversion, WordNet 3.0)	+ Explosion vectors (OSCAR 2109 + Wikipedia + OpenSubtitles + WMT News Crawl)	same	OntoNotes 5 + roberta-base

Static vectors buy nothing measurable for tagging and parsing (identical to two decimals) and 1 to 2 F on NER. The transformer buys about 4 LAS and 6 NER F, at 36x the size and a GPU requirement. That ordering sets the roadmap below.

Source: https://spacy.io/models/en.

2. Target: the Persian pipelines

Naming follows [lang]_[type]_[genre]_[size] (https://spacy.io/models#conventions). The type slot carries real information: dep = tagger + parser + lemmatizer, ent = NER only, core = both. Genre is news, after the dominant genre of UD_Persian-PerDT, whose README lists "news fiction nonfiction academic web blog". spaCy labels comparable treebank-trained pipelines such as de_core_news_sm as news.

Pipeline	Components	Embeddings	Status
fa_dep_news_sm	tok2vec, tagger, morphologizer, trainable_lemmatizer, parser	hash embeddings	built, shipping
fa_core_news_sm	the above plus ner	hash embeddings	built, shipping
fa_ent_news_sm	ner (own internal tok2vec)	hash embeddings	built, optional
fa_core_web_sm	same as core, mixed-genre training data	hash embeddings	not built; would add ParsTwiNER to cover social media
fa_core_news_md	+ static vectors	floret, 50k rows	vectors must be trained first (CPU-days on fa Wikipedia + OSCAR)
fa_core_news_lg	same	floret, 200k rows	same as md, bigger table
fa_core_news_trf	transformer instead of tok2vec	HooshvareLab/roberta-fa-zwnj-base (Apache-2.0)	not on this hardware; 2 GB VRAM cannot fine-tune a 125M-param encoder

Why core is honest here

core promises that the NER belongs to the same pipeline, built to the same standard and versioned together. That holds because PerDT ships its own entity layer in not-to-release/Dadegan with NER tag/: 15,833 entities over the same 29,107 sentences, under the same CC BY-SA 4.0, in the same genre, aligned to the same tokenization.

An earlier revision of this project refused to ship core, and was right to given what it knew then. The only redistributable Persian NER corpus found at that point was ParsTwiNER, a Twitter corpus scoring 67.22 F against 85-98 for the UD components, and folding that into one package would have hidden a genre and quality gap behind a single name and version number. Finding the treebank's own layer removed the objection rather than answering it.

fa_dep_news_sm still ships alongside core, for users who want a 7.5 MB syntax-only model or who would rather not depend on silver entity labels.

Why morphologizer + trainable_lemmatizer

The English pipelines use attribute_ruler because OntoNotes gives them PTB tags and they derive UPOS by rule. UD treebanks give UPOS and FEATS as gold data, so a morphologizer trained on them has strictly more information, and Persian morphology (Number, Person, Tense, Mood, Voice, Polarity, PronType) is worth predicting.

Persian rule-lemmatizer tables do exist in spacy-lookups-data (fa_lemma_exc.json 1.68 MB, fa_lemma_index.json, fa_lemma_rules.json, derived from Seraji's treebank), but a rule lemmatizer's accuracy cannot be measured against the corpus it was extracted from, and it needs token.pos to run at all. trainable_lemmatizer learns edit trees from PerDT's gold lemmas and reports a real lemma_acc. PerDT yields 1,908 edit trees at 100% lemma coverage.

senter is omitted from v1. It is a separately trained component that ships disabled by default in the English pipelines, and the parser already produces sentence boundaries. Adding it later takes one training run.

3. Resource inventory, with licences

3.1 Treebanks, for tagger / morphologizer / lemmatizer / parser

Treebank	Sents	Tokens	License	LEMMA	FEATS	XPOS	PROPN	Verdict
UD_Persian-PerDT (PerUDT v1.0)	29,107	~509k	CC BY-SA 4.0	converted + corrections	converted + corrections	manual native (34 types)	yes	chosen
UD_Persian-Seraji	5,997	~152k	CC BY-SA 4.0	manual native	manual native	manual native (30 types)	no	secondary, cross-eval
UD_Persian-PUD	1,000		CC BY-SA 4.0			none		test-only, parallel corpus
UD_Persian-IPerUDT	tiny		CC BY-SA 4.0			none		grammar examples, unusable

Measured locally with scripts/inspect_treebanks.py (train splits):

file                             sents    tokens    MWT  empty  lemma%  feats%  UPOS  XPOS  DEP
fa_perdt-ud-train.conllu         26196    452496   6508      0   100.0    57.7    16    34   34
fa_seraji-ud-train.conllu         4798    121067   1117      0   100.0    65.0    15    30   39

PerDT over Seraji for three reasons. It has 3.7x more training tokens (452k against 121k), and at sm size data is the binding constraint. Seraji has no PROPN: proper nouns are tagged NOUN, which breaks the downstream tasks people use spaCy for. And hazm independently chose PerDT for its own spaCy dependency parser, whose config.cfg names modified_fa_perdt-ud-train.spacy, which makes the numbers comparable.

Seraji's richer manual FEATS and fully manual lemmas make it the natural cross-evaluation set and a candidate for a future concatenated-corpus run. The two use different XPOS inventories, so naive concatenation would corrupt the tag label space.

3.2 NER

Dataset	Labels	Size	License	Usable
PerDT's own NER layer	PER, LOC, ORG, DAT, MON, TIM, PCT	29,107 sentences / 484k tokens / 15,833 entities	CC BY-SA 4.0, same as the treebank	chosen
ParsTwiNER	PER, ORG, LOC, NAT, POG, EVENT	7,667 tweets / 233k tokens / 16,250 entities	MIT (verified via GitHub API on overfit-ir/parstwiner)	usable, wrong genre
ARMAN (PersianNER)	6 classes	250k tokens	academic research only	no
PEYMA	7 classes	302k tokens	"free for research purposes", no OSS licence	no
NSURL-2019 Task 7	PEYMA tagset	~1M tokens	no explicit licence	no
HooshvareLab merged ParsNER	10 classes	ARMAN+PEYMA+WikiANN	inherits ARMAN/PEYMA restrictions; HF repo gated (401)	no

spaCy's maintainers state that the Persian models trained in 2018 were never published because of corpus licensing (spaCy discussion #8233, following PR #2797, which added only spacy.blank("fa") tokenizer support).

The PerDT layer lives in not-to-release/Dadegan with NER tag/{train,dev,test}_with_NER_tag.txt as two-column IOB2. In UD convention not-to-release/ means "excluded from the official UD release build", normally working and source data, and the directory is public on GitHub under the repo's LICENSE.txt. Confirm that reading with the PerDT authors before publishing anything derived from it, since redistribution rights are the whole point.

Three properties of that layer decide how it gets used:

1. The labels are silver. The treebank README states they came from the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. The published ents_f is therefore measured against a silver test split and partly reflects agreement with that tagger. A human-annotated test set is the outstanding work.
2. Tokenization differs. The NER files use the original Dadegan tokenization, which matches the released UD tokenization exactly in only 57 to 62% of sentences: the NER files drop some copulas and auxiliaries, and at least one honorific is corrupted (ص written as ،). Entities sit on content words present in both, so scripts/transfer_perdt_ner.py aligns them with difflib, transferring 99.86% of train entities, 99.74% of dev and 99.51% of test. Spans whose tokens do not all map contiguously are dropped rather than guessed.
3. The label sets do not line up with ParsTwiNER. PerDT has DAT, MON, TIM and PCT; ParsTwiNER has NAT, EVE and POG. The intersection is PER, LOC, ORG. Since spaCy assigns one label per token, concatenating the two raw would teach the model that dates are O in half the corpus. Any mixed-genre variant has to solve that first.

Genre matters more than any of this. Measured with one config on the three shared labels, a PerDT-trained NER scores 72.80 F on prose and 45.72 on tweets, while a ParsTwiNER-trained one scores 68.59 on tweets and 55.59 on prose. Training on both gives 72.11 and 66.49, so mixing costs under 1 F on prose and 2 F on tweets while recovering roughly 20 F off-genre. That is the argument for a future fa_core_web_sm, and the reason the current news packages stay prose-only.

3.3 Vectors, for md and lg

Option	License	Note
floret vectors trained via spacy-vectors-builder (MIT tooling) on fa Wikipedia + OSCAR	corpus-dependent	recommended: subword + Bloom-hash embeddings, bounded table size, zero OOV
fastText cc.fa.300	CC BY-SA 3.0	fallback; classic word table, large and OOV-prone

Persian surface forms multiply through suffixation and through inconsistent ZWNJ (U+200C) usage: the same word appears as می‌رود, میرود and می رود in real text. A classic word-vector table misses every variant it did not see, while floret's subword hashing covers them. spaCy ships floret vectors for Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian for the same reason.

3.4 Transformer encoders, for trf

Model	Arch	License	Note
HooshvareLab/roberta-fa-zwnj-base	RoBERTa-base	Apache-2.0	recommended: licensed, ZWNJ-aware, smallest credible option
FacebookAI/xlm-roberta-base	XLM-R base, 278M	MIT	licensed but larger
PartAI/TookaBERT-Base	BERT-base	Apache-2.0	licensed
m3hrdadfi/albert-fa-base-v2	ALBERT-base-v2	Apache-2.0	licensed, smallest
HooshvareLab/bert-base-parsbert-uncased	BERT-base, ~162M	no licence on the card	what hazm used; redistribution risk
sbunlp/fabert	BERT-base, 124M	no licence on the card	redistribution risk

All are BERT, RoBERTa, XLM-R or ALBERT, so all work with spacy-transformers and with spacy-curated-transformers, which supports exactly ALBERT, BERT, CamemBERT, RoBERTa and XLM-RoBERTa.

hazm's pipelines use ParsBERT, which carries no licence statement. Copying that choice would reintroduce the redistribution problem that stopped the 2018 attempt.

4. What already exists, and why it is not enough

No trained spaCy Persian pipeline exists. There are zero persian, farsi or fa_ hits in spaCy's website/meta/universe.json, and spacy.load("fa_core_news_sm") has never worked.

hazm ships three single-task spaCy pipelines on the HF Hub: hazm-parsbert-postagger (tag_acc 0.9862, hazm's own EZ-augmented tagset), hazm-bert-dependency-parser (dep_uas 0.9246, dep_las 0.8934, trained on PerDT) and hazm-parsbert-chunker (tag_acc 0.9618). Each is transformer + one component, version: 0.0.0, with empty license, author and sources, pinned to spaCy 3.6. Using all three costs three BERT forward passes over the same text and gives no shared Doc.

hazm's training code is not reusable: its trainable models are pycrfsuite CRFs (hazm/sequence_tagger.py), and the repo contains no config.cfg or spacy train. Its Spacy* classes only download the HF pipelines above.

hazm's tokenizer is incompatible with UD gold tokenization. Normalizer's AFFIX_SPACING_PATTERNS fuse ZWNJ affixes, and WordTokenizer.join_verb_parts() glues multi-word verb chains into single underscore-joined tokens, so training against PerDT with it would misalign tokens systematically.

DadmaTools (Apache-2.0 code) emits spaCy-compatible Doc objects but is not a loadable spaCy pipeline package, and its NER wraps ARMAN and PEYMA.

What hazm contributes here is one validated design decision, that PerDT is the corpus, and the stop-word list already vendored into spacy/lang/fa.

5. Decision: train the sm tier first

Not md or lg: they need floret vectors trained from scratch on Wikipedia and OSCAR, costing CPU-days, and the English reference numbers show no tag or dep accuracy gain. Not trf: 2 GB of VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait until the CPU pipeline proves the data plumbing. sm is also the tier the others are validated against, since md, lg and trf reuse the same corpus conversion, config skeleton and evaluation harness.

Tokenization, settled by measurement

spaCy has no multi-word-token layer, so CoNLL-U MWT ranges (Persian pronominal clitics and copulas, پدرم = پدر + م) must either be merged into one token or kept split. Measured on the PerDT dev set with scripts/tokenization_report.py, comparing gold boundaries against spacy.blank("fa")'s tokenizer:

spacy convert mode	token P	token R	token F	XPOS types	merge artefacts
--merge-subtokens	0.9860	0.9914	0.9887	68	34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space
plain (clitics split)	0.9875	0.9772	0.9823	35	none

Chosen: --merge-subtokens, which is also what Explosion's tagger_parser_ud template does. Without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never produce, so those tokens are permanently unlearnable and unpredictable at runtime. With it, every gold token is reachable, and the cost is limited to rare composite XPOS tags such as N_IANM_PR_JOPER (noun + enclitic pronoun) and 1.5% of lemmas that come out as two words.

The better long-term fix is Persian clitic-splitting suffix rules in spacy/lang/fa, which is an upstream PR rather than a model change. Recorded in docs/CONTRIBUTING-GUIDE.md §5.

Final composition

Shared trained components, in both fa_dep_news_sm (7.5 MB) and fa_core_news_sm (13 MB), both CC BY-SA 4.0:

Component	Trained on	Metric	Test score
tok2vec	shared, PerDT
tagger (XPOS)	PerDT, 90 labels	tag_acc	95.96
morphologizer (UPOS + FEATS)	PerDT, 298 labels	pos_acc / morph_acc	96.24 / 96.29
trainable_lemmatizer	PerDT, 1,908 edit trees	lemma_acc	97.91
parser	PerDT, 34 deprels	dep_uas / dep_las	89.69 / 85.15

fa_core_news_sm adds, and fa_ent_news_sm ships alone:

Component	Trained on	Metric	Test score
ner (own internal tok2vec)	PerDT NER layer, 7 labels	ents_p/r/f	77.67 / 66.87 / 71.87

Per label, F against training examples: LOC 80.24 (4,954), DAT 74.45 (1,323), MON 73.68 (205), ORG 68.77 (2,643), TIM 66.67 (135), PER 65.29 (4,847), PCT 57.14 (121).

PER scoring below LOC and ORG on nearly the same amount of data is the silver labels showing through. PerDT includes titles and honorifics inside PER spans inconsistently: 6.24% of its PER spans start with one (دکتر, مهندس, آقای), against 1.41% in the human-annotated ParsTwiNER, so the boundaries are less regular than the count suggests. MON, TIM and PCT are thin enough that their scores rest on 4 to 11 test entities each and should be treated as indicative only. Persian money, times and percentages are regular enough that an EntityRuler may beat the statistical model for those three.

Training cost on a 4-core i5-7200U with no GPU: 1h27m for the UD components (early-stopped at step 10,800, best checkpoint step 9,200) and 17 min for NER (best at step 6,000). Both runs are single-threaded and can run concurrently. Inference runs at about 9,250 words/s.

The --merge-subtokens artefacts show up in the shipped model as predicted: کتاب‌هایش ("his/her books") is one token tagged N_IANM_PR_JOPER with lemma کتاب او. Check this before consuming token.lemma_ downstream.

Sources are recorded in each meta.json with their licences, per https://spacy.io/api/data-formats#meta, including the treebank's NER layer as its own entry crediting Beheshti-NER. CC BY-SA 4.0 on PerDT means every package carries attribution and a share-alike notice, handled in scripts/finalize_pipeline.py, which also enforces the shape of each variant: it refuses to publish a dep pipeline containing ner, or a core one without it.