commit c2ad4d00222f2bb5fdfb87e988521aa6c036faed Author: Mohamad Fazeli Date: Wed Jul 29 20:52:13 2026 +0330 fa_core_news_sm: Persian spaCy pipeline from UD_Persian-PerDT + ParsTwiNER No trained Persian pipeline exists for spaCy: spacy.load("fa_core_news_sm") has never worked. This builds one from openly-licensed data so the result can actually be redistributed. Test scores (held-out splits): TAG 95.96, POS 96.24, MORPH 96.29, LEMMA 97.91, UAS 89.69, LAS 85.15, ENTS_F 67.22. ~9,250 words/s, 13 MB wheel. 1h27m on 4 CPU cores, no GPU. Corpus choices, with evidence: - UD_Persian-PerDT (CC BY-SA 4.0) over Seraji: 3.7x more tokens (452k vs 121k) and Seraji has no PROPN tag at all. hazm's own spaCy parser used PerDT too. - ParsTwiNER (MIT) for NER. ARMAN/PEYMA/NSURL are research-only; spaCy never shipped the 2018 Persian models precisely because of corpus licensing. Cost: NER is trained on tweets, so it is the weak component. - --merge-subtokens: measured token F 0.9887 vs 0.9823 split. Without it, 1.5% of gold token boundaries are unreachable by the tokenizer we ship. - morphologizer + trainable_lemmatizer instead of the English attribute_ruler + rule lemmatizer, because UD gives gold UPOS/FEATS/lemmas to train and measure on. The ner component embeds its own tok2vec rather than using a Tok2VecListener, so it can be sourced into the core pipeline after being trained on a separate corpus. Also documents an upstream bug: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels (dobj, pobj, nsubjpass, attr, dative) that do not exist in UD, so doc.noun_chunks returns bare head nouns (1.31 vs 2.77 tokens/chunk). Patch and tests in docs/upstream/fa-noun-chunks.md. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3bcf9af --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +# Downloaded corpora — reproduce with `spacy project assets` (checksums in project.yml) +assets/ +# Generated artifacts +corpus/ +training/ +metrics/ +packages/ +.venv/ +__pycache__/ +*.pyc diff --git a/.omp/AGENTS.md b/.omp/AGENTS.md new file mode 100644 index 0000000..c7be7d6 --- /dev/null +++ b/.omp/AGENTS.md @@ -0,0 +1,57 @@ +# Repo context for agents + +## Remotes + +| Name | URL | Purpose | +|---|---|---| +| `gitea` | `https://gitea.cap.nlogn.ir/fazel/spacy-fa-pipeline.git` | Primary self-hosted Gitea | + +## Pushing to Gitea + +Credentials come from `rbw` (Bitwarden CLI) — no token in git config or any tracked file. + +**rbw entry:** `api/gitea_token_AllExceptAdmin` + +> Note: the older `site_farsi` convention used `pass gitea/token_AllExceptAdmin`. That entry +> exists but its GPG agent is not unlocked on this machine — `pass` blocks for ~60s and +> returns empty. Use `rbw`. + +The credential helper is set locally in `.git/config` (never committed): + +``` +credential.https://gitea.cap.nlogn.ir.helper= + !f() { echo username=fazel; echo password=$(rbw get "api/gitea_token_AllExceptAdmin" | head -1); }; f +``` + +**Push command** (bypass proxy — Gitea is on the local network): + +```bash +NO_PROXY="*" no_proxy="*" http_proxy="" https_proxy="" HTTP_PROXY="" HTTPS_PROXY="" \ + git push gitea main +``` + +If `rbw` is locked, `rbw unlock` first. To rotate: update in the Gitea UI, then +`rbw edit "api/gitea_token_AllExceptAdmin"`. No git config change needed. + +## What is NOT committed + +`.gitignore` excludes `assets/ corpus/ training/ metrics/ packages/ .venv/`. The repo holds +only source: configs, scripts, `project.yml`, docs. Everything else is regenerated: + +```bash +python -m venv .venv && .venv/bin/python -m pip install -r requirements.txt +.venv/bin/python -m spacy project assets # checksummed downloads +.venv/bin/python -m spacy project run all # ~2h on 4 CPU cores +``` + +The trained wheel (13 MB) is a build artifact, not source. Publish it to the HF Hub instead — +see `docs/CONTRIBUTING-GUIDE.md` §2. + +## Environment + +The venv is Python 3.12.2, created from the conda env at `/home/fazel/anaconda3/envs/p12` +(the anaconda base python is 3.7 and cannot run spaCy 3.8). The IPython kernel available to +agents is bound to that 3.7 base and will fail on this project — shell out to `.venv/bin/python`. + +GPU (GTX 940MX, 2 GB) is unused: too small for a transformer, not worth the transfer overhead +for an `sm` pipeline. All training is CPU, `--gpu-id -1`. diff --git a/README.md b/README.md new file mode 100644 index 0000000..4e6c83f --- /dev/null +++ b/README.md @@ -0,0 +1,164 @@ +# fa_core_news_sm — a Persian pipeline for spaCy + +There is no trained Persian pipeline for spaCy. `spacy.load("fa_core_news_sm")` has never +worked; `spacy.blank("fa")` gives you a tokenizer and stop words and nothing else. This +project builds the missing pipeline from openly-licensed data, using spaCy's own tooling, so +the result can actually be redistributed. + +- **What the four English pipelines are, and what the four Persian equivalents should be:** + [`docs/MODELS.md`](docs/MODELS.md) +- **How models get contributed/published in the spaCy ecosystem, and what upstream `fa` + already has:** [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md) +- **The build itself:** [`project.yml`](project.yml) + +## The short version + +| | | +| --- | --- | +| Pipeline | `fa_core_news_sm` — tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner | +| Syntax/morphology data | [UD_Persian-PerDT](https://github.com/UniversalDependencies/UD_Persian-PerDT) (PerUDT v1.0) — 29,107 sentences, **CC BY-SA 4.0** | +| NER data | [ParsTwiNER](https://github.com/overfit-ir/parstwiner) — 7,667 tweets, **MIT** | +| Language data | `spacy/lang/fa` upstream (its stop word list comes from hazm) | +| Licence of the result | CC BY-SA 4.0 (inherited from the treebank) | +| Hardware | 4-core CPU. No GPU needed. | + +### Results + +Trained and evaluated on this laptop (4-core i5-7200U, CPU only, 1h27m for the UD +components, ~25 min for NER). Scores are on the **held-out test splits**, produced by +`spacy benchmark accuracy` and stored in `metrics/`. + +| Metric | `fa_core_news_sm` | reference | +| --- | --- | --- | +| `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | | +| `TAG_ACC` (XPOS) | **95.96** | | +| `POS_ACC` (UPOS) | **96.24** | | +| `MORPH_ACC` | 96.29 | | +| `LEMMA_ACC` | **97.91** | | +| `SENTS_F` | 99.25 | | +| `DEP_UAS` | **89.69** | hazm+ParsBERT: 92.46 | +| `DEP_LAS` | **85.15** | hazm+ParsBERT: 89.34 | +| `ENTS_P` / `ENTS_R` / `ENTS_F` | 74.77 / 61.06 / **67.22** | | +| Speed | ~9,250 words/s (CPU) | | +| Wheel size | 13 MB | `en_core_web_sm`: 12 MB | + +Per-entity F: `LOC` 73.9, `PER` 69.1, `NAT` 63.2, `ORG` 59.3, `POG` 41.2, `EVE` 30.0. + +Read these honestly: + +- **Parsing is 4.2 LAS behind hazm's parser**, which is the expected gap between a 13 MB + CPU model with hash embeddings and a fine-tuned ParsBERT. It is the same corpus and the + same spaCy parser architecture, so the comparison is fair, and it sets the target for the + future `trf` tier. +- **NER is the weak component.** 67 F reflects three compounding handicaps: an `sm` model + with no static vectors, a 233k-token training corpus, and a genre mismatch (trained on + tweets, most users will run it on prose). `EVE` and `POG` are near-useless. This is the + price of using the only MIT-licensed Persian NER corpus that exists. +- **Everything else is competitive with the English `sm` pipeline** (`en_core_web_sm`: + TAG 97, LAS 90, ENTS_F 84 — on a much larger and cleaner corpus). + +Reproduce: `.venv/bin/python -m spacy project run all`. + +### Install the built pipeline + +```bash +.venv/bin/python -m pip install packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl +``` + +```python +import spacy +nlp = spacy.load("fa_core_news_sm") +doc = nlp("دانشگاه تهران در سال ۱۳۱۳ تأسیس شد.") +print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc]) +print(doc.ents) # (دانشگاه تهران, ORG) +``` + +### Why not hazm's own models + +hazm is the reference Persian NLP toolkit and it *does* publish spaCy-format pipelines on the +HF Hub, so it was the obvious starting point. It does not survive contact: + +- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). There is no + `config.cfg` and no `spacy train` anywhere in the repo — the `Spacy*` classes only download + pretrained pipelines. +- Those pretrained pipelines are three *single-task* models (`transformer + tagger`, + `transformer + parser`, `transformer + chunker`), each `version: 0.0.0` with an empty + `license` field, pinned to spaCy 3.6. Using all three means three ParsBERT forward passes + over the same text and no shared `Doc`. +- Its tokenizer is deliberately incompatible with UD tokenization: the normaliser fuses ZWNJ + affixes and `join_verb_parts()` glues multi-word verb chains into single tokens. +- Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) are gated behind + `peykaregan.ir` / `dadegan.ir` under research-only terms. + +What hazm *does* give us: confirmation of the corpus choice — hazm's own spaCy parser was +trained on `modified_fa_perdt-ud-train.spacy`, i.e. the same treebank we use — plus the stop +word list already vendored into `spacy/lang/fa`. Full analysis in +[`docs/MODELS.md`](docs/MODELS.md) §4. + +### Why licensing is the load-bearing constraint + +spaCy's maintainers state that Persian models trained back in 2018 were never published +*because of corpus licensing* (spaCy discussion #8233, after PR #2797 added `fa` tokenizer +support). The standard Persian NER corpora — ARMAN, PEYMA, NSURL — are all "research use +only", and wrapping them in an Apache-2.0 toolkit does not launder that. ParsTwiNER (MIT) is +the only redistributable Persian NER corpus we could verify, which is why the NER component is +trained on tweets. That trade-off is recorded in the model's `meta.json["notes"]`. + +## Setup + +```bash +# Python 3.12 +python -m venv .venv +.venv/bin/python -m pip install -U pip +.venv/bin/python -m pip install "spacy>=3.8,<3.9" spacy-lookups-data +``` + +## Build + +Everything is driven by [`project.yml`](project.yml): + +```bash +.venv/bin/python -m spacy project assets # download + checksum the corpora +.venv/bin/python -m spacy project run all # inspect -> convert -> train -> assemble -> evaluate -> package +``` + +Individual steps: + +| Command | What it does | +| --- | --- | +| `inspect` | annotation coverage of the treebanks (`scripts/inspect_treebanks.py`) | +| `convert-ud` | CoNLL-U → `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report | +| `convert-ner` | unpack ParsTwiNER, IOB2 → `DocBin` | +| `debug-data` | `spacy debug data` on both corpora before spending CPU | +| `train-core` | tagger + morphologizer + trainable_lemmatizer + parser on PerDT | +| `train-ner` | standalone `ner` with its own embedded tok2vec on ParsTwiNER | +| `assemble` | source `ner` into the core pipeline, write full `meta.json` (`scripts/assemble_core.py`) | +| `evaluate` | `spacy benchmark accuracy` on both held-out test sets | +| `package` | build the wheel + sdist | +| `smoke` | run the pipeline over real Persian text and print every annotation layer | + +The two training runs are independent and can run concurrently — each is single-threaded. + +## Design decisions worth knowing before you touch anything + +1. **`--merge-subtokens`.** spaCy has no multiword-token layer, and PerDT splits pronominal + clitics (`پدرم` → `پدر` + `م`). Measured on dev: merging gives token F 0.9887 vs 0.9823 for + the split version, at the cost of 34 composite XPOS tags on 1.5% of tokens. Merging wins + because otherwise 1.5% of gold tokens are boundaries the shipped tokenizer can never + produce. Numbers: `scripts/tokenization_report.py`. +2. **`ner` carries its own tok2vec.** A `Tok2VecListener` only resolves inside the pipeline it + was trained in, so a listener-based component cannot be sourced into another pipeline. + `configs/fa_ner_sm.cfg` embeds the tok2vec instead — the same design as `en_core_web_sm`. +3. **`morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer.** + The English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD + gives us gold UPOS, FEATS and lemmas, so we train on them and get real `pos_acc`, + `morph_acc` and `lemma_acc` numbers instead of unmeasurable rule coverage. +4. **PerDT, not Seraji.** 3.7× more tokens, and Seraji has no `PROPN` tag at all. + +## Roadmap + +`md`/`lg` need floret vectors trained on Persian Wikipedia + OSCAR (see +`spacy-vectors-builder`); floret rather than classic fastText because Persian's ZWNJ usage is +inconsistent and explodes the surface vocabulary. `trf` needs a rented GPU and should use +`HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) rather than ParsBERT, whose model card +carries no licence. `senter` is one extra training run away. Details in `docs/MODELS.md` §2. diff --git a/configs/fa_core_news_sm.cfg b/configs/fa_core_news_sm.cfg new file mode 100644 index 0000000..5ba810b --- /dev/null +++ b/configs/fa_core_news_sm.cfg @@ -0,0 +1,231 @@ +# fa_core_news_sm — UD components (tagger, morphologizer, trainable_lemmatizer, parser). +# +# Generated with: +# spacy init config configs/fa_core_news_sm.cfg --lang fa \ +# --pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency +# +# Deviations from the generated defaults: +# - eval_frequency 200 -> 400 (dev set is only 146 docs; 200 wastes CPU on a 4-core box) +# The `ner` component is trained separately (configs/fa_ner_sm.cfg) from a different +# corpus and merged in by scripts/assemble_core.py. + +[paths] +train = null +dev = null +vectors = null +init_tok2vec = null + +[system] +gpu_allocator = null +seed = 0 + +[nlp] +lang = "fa" +pipeline = ["tok2vec", "tagger", "morphologizer", "trainable_lemmatizer", "parser"] +batch_size = 1000 +disabled = [] +before_creation = null +after_creation = null +after_pipeline_creation = null + +[corpora] + +[training] +dev_corpus = "corpora.dev" +train_corpus = "corpora.train" +seed = ${system.seed} +gpu_allocator = ${system.gpu_allocator} +dropout = 0.1 +accumulate_gradient = 1 +patience = 1600 +max_epochs = 0 +max_steps = 20000 +eval_frequency = 400 +frozen_components = [] +annotating_components = [] +before_to_disk = null +before_update = null + +[initialize] +vectors = ${paths.vectors} +init_tok2vec = ${paths.init_tok2vec} +vocab_data = null +lookups = null +before_init = null +after_init = null + +[components] + +[pretraining] + +[nlp.tokenizer] +@tokenizers = "spacy.Tokenizer.v1" + +[nlp.vectors] +@vectors = "spacy.Vectors.v1" + +[corpora.train] +@readers = "spacy.Corpus.v1" +path = ${paths.train} +max_length = 0 +gold_preproc = false +limit = 0 +augmenter = null + +[corpora.dev] +@readers = "spacy.Corpus.v1" +path = ${paths.dev} +max_length = 0 +gold_preproc = false +limit = 0 +augmenter = null + +[training.optimizer] +@optimizers = "Adam.v1" +beta1 = 0.9 +beta2 = 0.999 +L2_is_weight_decay = true +L2 = 0.01 +grad_clip = 1.0 +use_averages = false +eps = 1e-08 +learn_rate = 0.001 + +[training.batcher] +@batchers = "spacy.batch_by_words.v1" +discard_oversize = false +tolerance = 0.2 +get_length = null + +[training.logger] +@loggers = "spacy.ConsoleLogger.v1" +progress_bar = false + +[training.score_weights] +tag_acc = 0.25 +pos_acc = 0.12 +tag_micro_p = null +tag_micro_r = null +tag_micro_f = null +morph_acc = 0.12 +morph_per_feat = null +lemma_acc = 0.25 +dep_uas = 0.12 +dep_las = 0.12 +dep_las_per_type = null +sents_p = null +sents_r = null +sents_f = 0.0 + +[initialize.tokenizer] + +[initialize.components] + +[components.tok2vec] +factory = "tok2vec" + +[components.tagger] +factory = "tagger" +label_smoothing = 0.05 +overwrite = false +neg_prefix = "!" + +[components.morphologizer] +factory = "morphologizer" +label_smoothing = 0.05 +overwrite = true +extend = false + +[components.trainable_lemmatizer] +factory = "trainable_lemmatizer" +backoff = "orth" +min_tree_freq = 3 +overwrite = false +top_k = 1 + +[components.parser] +factory = "parser" +moves = null +update_with_oracle_cut_size = 100 +learn_tokens = false +min_action_freq = 30 + +[training.batcher.size] +@schedules = "compounding.v1" +start = 100 +stop = 1000 +compound = 1.001 +t = 0.0 + +[components.tok2vec.model] +@architectures = "spacy.Tok2Vec.v2" + +[components.tagger.model] +@architectures = "spacy.Tagger.v2" +nO = null +normalize = false + +[components.tagger.scorer] +@scorers = "spacy.tagger_scorer.v1" + +[components.morphologizer.model] +@architectures = "spacy.Tagger.v2" +nO = null +normalize = false + +[components.morphologizer.scorer] +@scorers = "spacy.morphologizer_scorer.v1" + +[components.trainable_lemmatizer.model] +@architectures = "spacy.Tagger.v2" +nO = null +normalize = false + +[components.trainable_lemmatizer.scorer] +@scorers = "spacy.lemmatizer_scorer.v1" + +[components.parser.model] +@architectures = "spacy.TransitionBasedParser.v2" +state_type = "parser" +extra_state_tokens = false +hidden_width = 128 +maxout_pieces = 3 +use_upper = true +nO = null + +[components.parser.scorer] +@scorers = "spacy.parser_scorer.v1" + +[components.tok2vec.model.embed] +@architectures = "spacy.MultiHashEmbed.v2" +width = ${components.tok2vec.model.encode.width} +attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] +rows = [5000, 1000, 2500, 2500] +include_static_vectors = false + +[components.tok2vec.model.encode] +@architectures = "spacy.MaxoutWindowEncoder.v2" +width = 96 +depth = 4 +window_size = 1 +maxout_pieces = 3 + +[components.tagger.model.tok2vec] +@architectures = "spacy.Tok2VecListener.v1" +width = ${components.tok2vec.model.encode.width} +upstream = "*" + +[components.morphologizer.model.tok2vec] +@architectures = "spacy.Tok2VecListener.v1" +width = ${components.tok2vec.model.encode.width} +upstream = "*" + +[components.trainable_lemmatizer.model.tok2vec] +@architectures = "spacy.Tok2VecListener.v1" +width = ${components.tok2vec.model.encode.width} +upstream = "*" + +[components.parser.model.tok2vec] +@architectures = "spacy.Tok2VecListener.v1" +width = ${components.tok2vec.model.encode.width} +upstream = "*" \ No newline at end of file diff --git a/configs/fa_ner_sm.cfg b/configs/fa_ner_sm.cfg new file mode 100644 index 0000000..263c104 --- /dev/null +++ b/configs/fa_ner_sm.cfg @@ -0,0 +1,156 @@ +# fa NER component, CPU size (sm). +# +# Trained on ParsTwiNER (MIT) — a different corpus from the UD treebank that trains +# the rest of fa_core_news_sm, so this is a standalone run whose `ner` component is +# later sourced into the core pipeline by scripts/assemble_core.py. +# +# Deliberate deviation from `spacy init config --pipeline ner`: the tok2vec is +# EMBEDDED inside components.ner.model instead of being a separate `tok2vec` +# component with a Tok2VecListener. A listener can only resolve inside the pipeline +# it was trained in; embedding makes the component self-contained and therefore +# sourceable. This is the same design as en_core_web_sm, whose `ner` "has its own +# independent internal tok2vec" (https://spacy.io/models#design). + +[paths] +train = null +dev = null +vectors = null +init_tok2vec = null + +[system] +gpu_allocator = null +seed = 0 + +[nlp] +lang = "fa" +pipeline = ["ner"] +batch_size = 1000 +disabled = [] +before_creation = null +after_creation = null +after_pipeline_creation = null + +[nlp.tokenizer] +@tokenizers = "spacy.Tokenizer.v1" + +[nlp.vectors] +@vectors = "spacy.Vectors.v1" + +[components] + +[components.ner] +factory = "ner" +moves = null +update_with_oracle_cut_size = 100 +incorrect_spans_key = null + +[components.ner.model] +@architectures = "spacy.TransitionBasedParser.v2" +state_type = "ner" +extra_state_tokens = false +hidden_width = 64 +maxout_pieces = 2 +use_upper = true +nO = null + +[components.ner.model.tok2vec] +@architectures = "spacy.Tok2Vec.v2" + +[components.ner.model.tok2vec.embed] +@architectures = "spacy.MultiHashEmbed.v2" +width = ${components.ner.model.tok2vec.encode.width} +attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] +rows = [5000, 1000, 2500, 2500] +include_static_vectors = false + +[components.ner.model.tok2vec.encode] +@architectures = "spacy.MaxoutWindowEncoder.v2" +width = 96 +depth = 4 +window_size = 1 +maxout_pieces = 3 + +[components.ner.scorer] +@scorers = "spacy.ner_scorer.v1" + +[corpora] + +[corpora.train] +@readers = "spacy.Corpus.v1" +path = ${paths.train} +max_length = 0 +gold_preproc = false +limit = 0 +augmenter = null + +[corpora.dev] +@readers = "spacy.Corpus.v1" +path = ${paths.dev} +max_length = 0 +gold_preproc = false +limit = 0 +augmenter = null + +[training] +dev_corpus = "corpora.dev" +train_corpus = "corpora.train" +seed = ${system.seed} +gpu_allocator = ${system.gpu_allocator} +dropout = 0.1 +accumulate_gradient = 1 +patience = 1600 +max_epochs = 0 +max_steps = 20000 +eval_frequency = 400 +frozen_components = [] +annotating_components = [] +before_to_disk = null +before_update = null + +[training.optimizer] +@optimizers = "Adam.v1" +beta1 = 0.9 +beta2 = 0.999 +L2_is_weight_decay = true +L2 = 0.01 +grad_clip = 1.0 +use_averages = false +eps = 1e-08 +learn_rate = 0.001 + +[training.batcher] +@batchers = "spacy.batch_by_words.v1" +discard_oversize = false +tolerance = 0.2 +get_length = null + +[training.batcher.size] +@schedules = "compounding.v1" +start = 100 +stop = 1000 +compound = 1.001 +t = 0.0 + +[training.logger] +@loggers = "spacy.ConsoleLogger.v1" +progress_bar = false + +[training.score_weights] +ents_f = 1.0 +ents_p = 0.0 +ents_r = 0.0 +ents_per_type = null + +[initialize] +vectors = ${paths.vectors} +init_tok2vec = ${paths.init_tok2vec} +vocab_data = null +lookups = null +before_init = null +after_init = null + +[initialize.tokenizer] + +[initialize.components] + +[pretraining] diff --git a/docs/CONTRIBUTING-GUIDE.md b/docs/CONTRIBUTING-GUIDE.md new file mode 100644 index 0000000..76303ec --- /dev/null +++ b/docs/CONTRIBUTING-GUIDE.md @@ -0,0 +1,239 @@ +# Contribution guidelines: getting a Persian pipeline into the spaCy ecosystem + +Everything below is sourced from spaCy's own primary docs/repos (URLs inline). Read this +before writing code, because **where** a contribution goes decides how it must be shaped. + +## 0. The one thing to internalise + +spaCy splits Persian support into two *completely different* contribution surfaces: + +| Surface | What it is | Where it lives | How you contribute | +| --- | --- | --- | --- | +| **Language data** (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` | +| **Trained pipeline** (`fa_core_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | **You cannot.** Publish it yourself (PyPI / HF Hub) and get it listed in spaCy Universe | + +`spacy/lang/fa` **already exists upstream**. What does not exist is any trained `fa` pipeline. +So this project is a *publishing* project, not an upstream-PR project — with optional +upstream PRs for the language-data gaps we find along the way. + +## 1. Policy, verbatim + +From : + +- **No CLA.** There is no contributor licence agreement section; contribution is an ordinary + GitHub PR governed by the Contributor Covenant Code of Conduct v1.4. +- Inclusion philosophy — this is the sentence that decides everything: + > "Our philosophy is to prefer a smaller core library. […] If you're looking to implement a + > new spaCy feature, starting with a custom component package is usually the best strategy. + > […] And if it works well, we can always integrate it into the core library later." +- The only sanctioned route for non-trivial additions is the *Publishing spaCy extensions and + plugins* section: + > "An extension or plugin should add substantial functionality, be well-documented and + > open-source. It should be available for users to download and install as a Python package + > – for example via PyPi." + > "Once your extension is published, you can open a PR to suggest it for the Universe page." +- `CONTRIBUTING.md` contains **zero** mention of submitting trained models. Neither does + — its README only documents `compatibility.json` + as "the source of spaCy's internal compatibility check, performed when you run the download + command". There is no PR template, issue label, or documented process for a third party to + land a pipeline in `spacy download`. + **Conclusion: the `spacy download` index is an Explosion-only release channel.** + (`[INFERENCE]` from the absence of any process across all three primary sources.) +- `https://spacy.io/usage/adding-languages` no longer exists; it redirects to + . The `BaseDefaults` / + `Language` contract is documented at . + +### Code rules that apply to a `spacy/lang/fa` PR + +- `black` formatting, `flake8` clean, type hints where practical. +- Tests go in `spacy/tests/lang/fa/`; regression tests use `@pytest.mark.issue(N)`. +- Touching `.pyx` means the reviewer must rebuild: `python setup.py build_ext --inplace`. +- Language data must be *pure data + rules*: no model downloads, no network, no heavy deps. + +## 2. Three publishing routes for the trained pipeline + +1. **Hugging Face Hub** — Explosion's own tool, the path of least resistance + (): + ```bash + pip install spacy-huggingface-hub + huggingface-cli login + python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build wheel + python -m spacy huggingface-hub push packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl --org + ``` + Users then `pip install https://huggingface.co//fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl`. +2. **PyPI / self-hosted wheel** — `spacy package … --build sdist,wheel` then `twine upload`, + or attach the wheel to a GitHub Release. See . +3. **spaCy Universe listing** — lists the package on spacy.io but hosts nothing. Per + : fork `explosion/spaCy`, + append an entry to `website/meta/universe.json`, open a PR. Required fields: + `id`, `title`, `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`, + `url`, `thumb`, `image`, `author`, `author_links`, `category`, `tags`. + Checklist: must be **open-source with a user-friendly license** and "at least somewhat documented". + +**Route we take:** HF Hub for the artifact + a Universe PR once metrics are respectable. + +## 3. Naming and versioning conventions (non-negotiable if you want to look official) + +From and the `spacy-models` README: + +``` +[lang]_[type]_[genre]_[size] e.g. fa_core_news_sm +``` + +| Slot | Allowed values | Meaning | +| --- | --- | --- | +| `type` | `core` | tagger + parser + lemmatizer + NER | +| | `dep` | tagger + parser + lemmatizer, **no NER** | +| | `ent` | NER only | +| | `sent` | sentence segmentation only | +| `genre` | `news`, `web`, `wiki` | text domain of the training corpus | +| `size` | `sm` | no static vectors | +| | `md` | ~20k unique vectors / ~500k keys (or 50k floret rows) | +| | `lg` | ~500k vectors (or 200k floret rows) | +| | `trf` | transformer, no static vectors | + +**Version numbers are `a.b.c` = spaCy-major . spaCy-minor . model-revision.** Trained against +spaCy 3.8 → the package version starts at `3.8.0`. Retraining on new data bumps `c`. +Do **not** invent `0.1.0`-style versions; that is exactly how hazm's HF pipelines ended up with +`version: 0.0.0` and `spacy_version: >=3.6.0,<3.7.0`, which is unusable metadata. + +### `meta.json` fields that matter () + +`lang`, `name`, `version`, `spacy_version` (range), `description`, `author`, `email`, `url`, +`license`, **`sources`** (list of `{name, url, author, license}` — this is where treebank +provenance and its CC BY-SA obligation must be recorded), `requirements` (extra pip deps +injected into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance` +(auto-filled by `spacy train`), `speed`, `spacy_git_version`. + +Note: in v3, `meta.json` "isn't used to construct the language class and pipeline anymore" — +`config.cfg` is the single source of truth for loading. `meta.json` is metadata + packaging. + +## 4. Canonical training workflow (what Explosion actually runs) + +Template: +(`project.yml` + `configs/default.cfg`). Directory layout `assets / corpus / configs / +training / metrics / packages` is the convention — this repo follows it. + +```bash +# 1. assets: clone the UD treebank +git clone https://github.com/UniversalDependencies/UD_Persian-PerDT assets/UD_Persian-PerDT + +# 2. corpus: CoNLL-U -> binary DocBin +python -m spacy convert assets/.../fa_perdt-ud-train.conllu corpus/ \ + --converter conllu --n-sents 10 --merge-subtokens + +# 3. config +python -m spacy init config configs/default.cfg --lang fa \ + --pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency +python -m spacy init fill-config partial.cfg configs/default.cfg # completes defaults + +# 4. (md/lg only) vectors +python -m spacy init vectors fa cc.fa.300.vec vectors/ --mode floret --prune 200000 + +# 5. train +python -m spacy train configs/default.cfg --output training/ --gpu-id -1 \ + --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy + +# 6. evaluate on held-out test +python -m spacy benchmark accuracy training/model-best corpus/test.spacy --output metrics/test.json +# ('spacy evaluate' is now just an alias for 'benchmark accuracy') + +# 7. package +python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build sdist,wheel +``` + +`spacy assemble` builds a pipeline from a config **without training** — useful for a +rule-only artifact (tokenizer + lookup lemmatizer + stop words). + +### Flags worth knowing on `spacy convert` + +- `--converter conllu` — explicit is better than `auto`. +- `--n-sents N` — how many sentences per `Doc`; 10 is the Explosion default and gives the + parser/senter cross-sentence context. +- `--merge-subtokens` — spaCy has **no multi-word-token layer**. Persian UD treebanks use MWT + ranges for clitics (`پدرم` → `پدر` + `م`), so this flag decides whether gold tokens are + clitic-split or fused. See `docs/MODELS.md` for the tokenisation decision. +- `--morphology` — appends morph features to the tag (only if you *aren't* using a morphologizer). + +### How the four size tiers differ mechanically + +| Tier | Embedding source | Config difference | +| --- | --- | --- | +| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, everything else a `spacy.Tok2VecListener` on the shared `tok2vec` | +| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = `; `lg` just has a bigger table | +| `trf` | contextual | replace `tok2vec` with a `transformer` component; every other component listens via `TransformerListener` instead of `Tok2VecListener` | + +`spacy pretrain` (Tok2Vec LM-style pretraining on raw text → `[initialize] init_tok2vec`) is +orthogonal to the tier and optional. + +## 5. Upstream `spacy/lang/fa` — current state and gaps + +Contents of : + +| File | Size | What it gives us | +| --- | --- | --- | +| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a **`lemmatizer` factory defaulting to `mode="rule"`** | +| `tokenizer_exceptions.py` | 64.9 KB | Large generated compound-verb / enclitic exception table | +| `generate_verbs_exc.py` | 14.8 KB | Dev script that generates the above | +| `stop_words.py` | 3.8 KB | ~500 stop words, comment says **"Stop words from HAZM package"** | +| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals + `ام`/`ین` suffixes) | +| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only — **no prefixes, no infixes** | +| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, needs a trained parser to do anything | + +Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer, or +stop-word tests. + +`spacy-lookups-data` (MIT) **already ships Persian rule-lemmatizer tables**: + +| File | Size | +| --- | --- | +| `fa_lemma_exc.json` | 1.68 MB | +| `fa_lemma_index.json` | 171 KB | +| `fa_lemma_rules.json` | 882 B | +| `fa_source.txt` | "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" | + +Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`, +`fa_license.txt` (Catalan has one; Persian's Seraji-derived provenance is undocumented). + +**Gap list for a real `fa_core_news_*`:** + +1. No trained artefacts at all — no weights, no `config.cfg`, no `meta.json`, no entry in + `compatibility.json`. `spacy.load("fa_core_news_sm")` fails today. +2. No word vectors for `fa` → `md`/`lg` need vectors built from scratch. +3. No NER data or labels anywhere in `spacy/lang/fa` → `core` (which implies NER) needs an + external annotated corpus. +4. Rule lemmatizer needs `token.pos` from a tagger/morphologizer → chicken-and-egg until the + tagger exists (or use a `trainable_lemmatizer`, which learns edit trees and needs no tables). +5. No Persian-specific prefix/infix rules — ZWNJ (U+200C) is only handled implicitly through + the verb-exception table. +6. Zero tokenizer test coverage for a 65 KB exception table. + +Items 5–7 are legitimate upstream PR material; items 1–4 are this project's job. + +## 6. Prior art we must not duplicate badly + +hazm publishes **spaCy-format** Persian pipelines on the HF Hub (MIT-ish, but `license` field +left empty), each a single-task pipeline built on ParsBERT: + +| HF repo | Pipeline | Reported | Trained on (`[paths]` in its `config.cfg`) | +| --- | --- | --- | --- | +| `roshan-research/hazm-parsbert-postagger` | `transformer, tagger` | `tag_acc` 0.9862 | `data_train_98_rs10.spacy` (hazm's own EZ-augmented tagset: `NOUN,EZ`, `ADJ,EZ`, …) | +| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy` — **UD_Persian-PerDT** | +| `roshan-research/hazm-parsbert-chunker` | `transformer, tagger` (IOB chunk tags) | `tag_acc` 0.9618 | hazm chunk data | + +Both use `spacy-transformers` `TransformerModel.v3` on +`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 / stride 96, +`spacy_version >= 3.6.0,<3.7.0`. + +What they get wrong, and what we fix: + +- Three separate pipelines instead of one `core` pipeline → users pay for three BERT forward + passes and cannot share a `Doc`. +- `version: 0.0.0`, empty `license`/`author`/`sources`, `name: "pipeline"` → violates every + convention in §3 and makes redistribution legally murky. +- Transformer-only, so no CPU-friendly tier at all. +- No lemmatizer, no morphologizer, no NER, no vectors. +- Pinned to spaCy 3.6; unusable on 3.8 without retraining. + +The valuable, reusable finding: **hazm's own parser recipe is "UD_Persian-PerDT + spaCy +`TransitionBasedParser`"**, which independently confirms the corpus choice in `docs/MODELS.md`. diff --git a/docs/MODELS.md b/docs/MODELS.md new file mode 100644 index 0000000..3611d3c --- /dev/null +++ b/docs/MODELS.md @@ -0,0 +1,219 @@ +# The Persian pipelines: what to build, from what, and why + +## 1. What "the 4 English pipelines" actually are + +They are **one pipeline design at four embedding budgets**, not four different products. +Every one of them has the same components; the only real axis of variation is where token +representations come from. + +| | `en_core_web_sm` | `en_core_web_md` | `en_core_web_lg` | `en_core_web_trf` | +| --- | --- | --- | --- | --- | +| Size on disk | 12 MB | 31 MB | 382 MB | 436 MB | +| Embeddings | hash embeddings only | 685k keys / 20k vectors (300d) | 685k keys / 343k vectors (300d) | `roberta-base`, 768d contextual | +| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | **transformer**, tagger, parser, attribute_ruler, lemmatizer, ner | +| `TAG_ACC` | 0.97 | 0.97 | 0.97 | 0.98 | +| `DEP_UAS` / `LAS` | 0.92 / 0.90 | 0.92 / 0.90 | 0.92 / 0.90 | 0.95 / 0.94 | +| `ENTS_F` | 0.84 | 0.85 | 0.86 | 0.90 | +| Training data | OntoNotes 5 (+ ClearNLP dep conversion, WordNet 3.0) | + Explosion vectors (OSCAR 2109 + Wikipedia + OpenSubtitles + WMT News Crawl) | same | OntoNotes 5 + roberta-base | + +Read the accuracy table honestly: **static vectors buy almost nothing for tagging and +parsing** (identical to two decimals) and ~1–2 F on NER. The transformer buys ~4 LAS and +~6 NER F, at 36× the size and a GPU requirement. That ordering dictates the roadmap below. + +Sources: . + +## 2. Target: the four Persian pipelines + +Naming follows `[lang]_[type]_[genre]_[size]` (). +`core` = tagger + parser + lemmatizer + NER. Genre is `news`, after the dominant genre of +UD_Persian-PerDT (its README lists "news fiction nonfiction academic web blog", and spaCy +labels comparable treebank-trained pipelines such as `de_core_news_sm` as `news`). + +| Pipeline | Components | Embeddings | Trainable on this hardware? | +| --- | --- | --- | --- | +| **`fa_core_news_sm`** | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner | hash embeddings | **yes — this is what we train now** | +| `fa_core_news_md` | same | floret vectors, 50k rows | yes, but vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) | +| `fa_core_news_lg` | same | floret vectors, 200k rows | same as md, bigger table | +| `fa_core_news_trf` | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | **no** — 2 GB VRAM (GTX 940MX) cannot fine-tune a 125M-param encoder; needs rented GPU | + +One deviation from the English design, deliberate: Persian gets a **`morphologizer`** +(UPOS + morphological features) and a **`trainable_lemmatizer`** instead of English's +`attribute_ruler` + rule `lemmatizer`. Reasons: + +- The English pipelines use `attribute_ruler` because OntoNotes gives them PTB `tag`s and + they *derive* UPOS from tags by rule. UD treebanks give UPOS and FEATS as gold data — + training a morphologizer on them is strictly more information, and Persian morphology + (Number, Person, Tense, Mood, Voice, Polarity, PronType) is worth predicting. +- Persian rule-lemmatizer tables *do* exist in `spacy-lookups-data` (`fa_lemma_exc.json` + 1.68 MB, `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but + a rule lemmatizer's accuracy is unmeasurable against the corpus it was extracted from and it + needs `token.pos` to work at all. `trainable_lemmatizer` learns edit trees from PerDT's gold + lemmas and reports a real `lemma_acc`. PerDT yields **1,908 edit trees** with 100% lemma + coverage — plenty. + +`senter` is intentionally omitted from v1: it is a separately trained component that ships +*disabled by default* in the English pipelines, and the parser already produces sentence +boundaries. Adding it later requires only one extra training run. + +## 3. Resource inventory (everything checked for license) + +### 3.1 Treebanks — the tagger / morphologizer / lemmatizer / parser data + +| Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN? | Verdict | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **UD_Persian-PerDT** (PerUDT v1.0) | 29,107 | ~509k | **CC BY-SA 4.0** | converted + corrections | converted + corrections | manual native (34 types) | **yes** | **CHOSEN** | +| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | **no** | secondary / cross-eval | +| UD_Persian-PUD | 1,000 | — | CC BY-SA 4.0 | — | — | none | — | test-only, parallel corpus | +| UD_Persian-IPerUDT | tiny | — | CC BY-SA 4.0 | — | — | none | — | grammar examples, unusable | + +Measured locally with `scripts/inspect_treebanks.py` (train splits): + +``` +file sents tokens MWT empty lemma% feats% UPOS XPOS DEP +fa_perdt-ud-train.conllu 26196 452496 6508 0 100.0 57.7 16 34 34 +fa_seraji-ud-train.conllu 4798 121067 1117 0 100.0 65.0 15 30 39 +``` + +Why PerDT over Seraji: + +1. **3.7× more training tokens** (452k vs 121k). At `sm` size, data is the binding constraint. +2. **Seraji has no `PROPN`** — proper nouns are tagged `NOUN`. A pipeline that cannot mark + proper nouns is crippled for exactly the downstream tasks people use spaCy for. +3. hazm independently chose PerDT for its own spaCy dependency parser — its + `config.cfg` names `modified_fa_perdt-ud-train.spacy` as the training set. Converging on + the same corpus makes our numbers comparable to theirs. + +Seraji's advantages (richer manual FEATS, fully manual lemmas) are real; it is the natural +cross-evaluation set and a candidate for a future concatenated-corpus run. The two use +different XPOS inventories, so naive concatenation would corrupt the `tag` label space. + +### 3.2 NER — the one place with a licensing minefield + +| Dataset | Labels | Size | License | Usable? | +| --- | --- | --- | --- | --- | +| **ParsTwiNER** | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens | **MIT** (verified via GitHub API on `overfit-ir/parstwiner`) | **CHOSEN** | +| ARMAN (PersianNER) | 6 classes | 250k tokens | academic research only | no | +| PEYMA | 7 classes | 302k tokens | "free for research purposes", no OSS licence | no | +| NSURL-2019 Task 7 | PEYMA tagset | ~1M tokens | no explicit licence | no | +| HooshvareLab merged ParsNER | 10 classes | ARMAN+PEYMA+WikiANN | inherits ARMAN/PEYMA restrictions; HF repo gated (401) | no | + +This is not pedantry. **spaCy's own maintainers state that Persian models trained back in +2018 were never published precisely because of corpus licensing** (spaCy discussion #8233, +following PR #2797 which added only `spacy.blank("fa")` tokenizer support). Repeating that +mistake would waste the whole exercise: a pipeline you cannot legally redistribute is not a +pipeline, it is a local file. + +Consequence to state plainly in the model card: **the NER component is trained on Twitter +text while the rest of the pipeline is trained on edited prose.** Expect NER to degrade on +formal news text relative to what an ARMAN/PEYMA-trained model would score. That is the +price of a redistributable artifact. + +### 3.3 Vectors (for md / lg) + +| Option | License | Note | +| --- | --- | --- | +| **floret vectors** trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | **recommended.** Subword + Bloom-hash embeddings: bounded table size, zero OOV | +| fastText `cc.fa.300` | CC BY-SA 3.0 | quick fallback; classic word table, large and OOV-prone | + +Floret is the right call for Persian specifically. Persian surface forms explode through +suffixation *and* through inconsistent ZWNJ (U+200C) usage — the same word appears as +`می‌رود` / `میرود` / `می رود` in real text. A classic word-vector table has a miss for every +variant; floret's subword hashing covers all of them. spaCy already ships floret vectors for +Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian for the same reason. + +### 3.4 Transformer encoders (for trf) + +| Model | Arch | License | Note | +| --- | --- | --- | --- | +| **`HooshvareLab/roberta-fa-zwnj-base`** | RoBERTa-base | **Apache-2.0** | recommended: licensed, ZWNJ-aware, smallest of the credible options | +| `FacebookAI/xlm-roberta-base` | XLM-R base, 278M | MIT | licensed but larger | +| `PartAI/TookaBERT-Base` | BERT-base | Apache-2.0 | licensed | +| `m3hrdadfi/albert-fa-base-v2` | ALBERT-base-v2 | Apache-2.0 | licensed, smallest | +| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | **no licence on the card** | what hazm used; redistribution risk | +| `sbunlp/fabert` | BERT-base, 124M | **no licence on the card** | redistribution risk | + +All of these are BERT/RoBERTa/XLM-R/ALBERT, so all are supported by both +`spacy-transformers` and `spacy-curated-transformers` (the latter supports exactly +ALBERT / BERT / CamemBERT / RoBERTa / XLM-RoBERTa). + +Note the trap: hazm's own pipelines use ParsBERT, which has **no license statement**. Copying +that choice would reintroduce the redistribution problem that sank the 2018 attempt. + +## 4. What already exists (and why it is not enough) + +- **No trained spaCy Persian pipeline exists.** Zero `persian` / `farsi` / `fa_` hits in + spaCy's `website/meta/universe.json`. `spacy.load("fa_core_news_sm")` has never worked. +- **hazm** ships three *single-task* spaCy pipelines on the HF Hub: + `hazm-parsbert-postagger` (`tag_acc` 0.9862, hazm's own EZ-augmented tagset), + `hazm-bert-dependency-parser` (`dep_uas` 0.9246 / `dep_las` 0.8934, trained on PerDT), + `hazm-parsbert-chunker` (`tag_acc` 0.9618). Each is `transformer + one component`, + `version: 0.0.0`, empty `license`/`author`/`sources`, pinned to spaCy 3.6. Using all three + means three separate BERT forward passes over the same text and no shared `Doc`. +- **hazm's training code is not reusable.** Its trainable models are pycrfsuite CRFs + (`hazm/sequence_tagger.py`); there is no `config.cfg` or `spacy train` anywhere in the repo. + Its `Spacy*` classes only *download* the HF pipelines above. +- **hazm's tokenizer is actively incompatible** with UD gold tokenization: `Normalizer`'s + `AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes and `WordTokenizer.join_verb_parts()` glues + multi-word verb chains into single underscore-joined tokens. Training against PerDT with + hazm's tokenizer would misalign tokens systematically. +- **DadmaTools** (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable + spaCy pipeline package, and its NER wraps ARMAN/PEYMA — the restricted data again. + +So what hazm genuinely contributes to this project is **one validated design decision** +(PerDT is the corpus) and **the stop-word list already vendored into `spacy/lang/fa`**. +Everything else is built with spaCy-native tooling. + +## 5. Decision: train `fa_core_news_sm` first + +Not md/lg: those need floret vectors trained from scratch on Wikipedia+OSCAR (CPU-days) and +buy ~0.00 tag/dep accuracy in the English reference numbers. +Not trf: 2 GB of VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait +until the CPU pipeline proves the data plumbing is right. +`sm` is also the tier every other tier is validated against: md/lg/trf reuse the exact same +corpus conversion, config skeleton and evaluation harness. + +### Tokenization decision, settled with a measurement + +spaCy has no multi-word-token layer, so CoNLL-U MWT ranges (Persian pronominal clitics and +copulas: `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on +the PerDT dev set with `scripts/tokenization_report.py`, comparing gold boundaries against +`spacy.blank("fa")`'s tokenizer: + +| `spacy convert` mode | token P | token R | token F | XPOS types | merge artefacts | +| --- | --- | --- | --- | --- | --- | +| `--merge-subtokens` | 0.9860 | 0.9914 | **0.9887** | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space | +| plain (clitics split) | 0.9875 | 0.9772 | 0.9823 | 35 | none | + +**Chosen: `--merge-subtokens`** (also what Explosion's `tagger_parser_ud` template does). +Rationale: without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never +produce, so those tokens are permanently unlearnable and unpredictable at runtime. With it, +every gold token is reachable; the cost is confined to rare composite XPOS tags such as +`N_IANM_PR_JOPER` (noun + enclitic pronoun) — which are, at least, informative — and 1.5% of +lemmas that come out as two words. + +The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which +would be an upstream PR, not a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5. + +### Final composition of `fa_core_news_sm` + +| Component | Trained on | Metric | Test score | +| --- | --- | --- | --- | +| `tok2vec` | shared, PerDT | — | — | +| `tagger` (XPOS) | PerDT, 90 labels | `tag_acc` | 95.96 | +| `morphologizer` (UPOS + FEATS) | PerDT, 298 labels | `pos_acc` / `morph_acc` | 96.24 / 96.29 | +| `trainable_lemmatizer` | PerDT, 1,908 edit trees | `lemma_acc` | 97.91 | +| `parser` | PerDT, 34 deprels | `dep_uas` / `dep_las` | 89.69 / 85.15 | +| `ner` (own internal tok2vec) | ParsTwiNER, 6 labels | `ents_p/r/f` | 74.77 / 61.06 / 67.22 | + +Training cost on the target hardware (4-core i5-7200U, no GPU): **1h27m** for the UD +components (early-stopped at step 10,800; best checkpoint step 9,200) and ~25 min for NER +(best at step 4,000). Both runs are single-threaded, so they were run concurrently. +Inference: ~9,250 words/s. Wheel: 13 MB. + +The `--merge-subtokens` artefacts predicted above are visible in the shipped model exactly as +expected — `کتاب‌هایش` ("his/her books") comes out as one token tagged `N_IANM_PR_JOPER` with +lemma `کتاب او`. Worth knowing before you consume `token.lemma_` downstream. + +Sources recorded in `meta.json`, both with their licenses, per +. CC BY-SA 4.0 on PerDT means the packaged pipeline +must carry attribution and share-alike notice — handled in `scripts/assemble_core.py`. diff --git a/docs/upstream/fa-noun-chunks.md b/docs/upstream/fa-noun-chunks.md new file mode 100644 index 0000000..9e5e978 --- /dev/null +++ b/docs/upstream/fa-noun-chunks.md @@ -0,0 +1,223 @@ +# Upstream PR candidate: `spacy/lang/fa/syntax_iterators.py` uses non-UD dependency labels + +## The bug + +`spacy/lang/fa/syntax_iterators.py` matches these dependency labels as noun-phrase heads: + +```python +labels = ["nsubj", "dobj", "nsubjpass", "pcomp", "pobj", "dative", "appos", "attr", "ROOT"] +``` + +`dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They are not +Universal Dependencies relations, and **every Persian treebank is UD** (`UD_Persian-PerDT`, +`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline must +therefore emit UD labels, so five of the nine labels are dead code and `doc.noun_chunks` +silently returns bare head nouns. + +Compare `spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py`, which +use UD labels (`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their +treebanks are UD too. `spacy/lang/de` legitimately uses TIGER labels (`sb`, `oa`, `nk`, …) +because the German pipelines are trained on TIGER. Persian has no such excuse. + +Second, smaller problem: the iterator yields `word.left_edge.i` → `word.i + 1`, i.e. it only +ever expands **left**. Persian noun phrases expand **right** through ezafe: +`رئیس انجمن جراحان قلب ایران` ("the head of the Iranian society of heart surgeons") is one NP +whose head is the leftmost token. Left-only expansion truncates it to `رئیس`. + +## Evidence + +Measured with `scripts/check_noun_chunks.py` on the `UD_Persian-PerDT` dev split (146 docs) +parsed by this project's trained pipeline: + +``` +shipped noun_chunks: 1503 chunks, 1.31 tokens/chunk +proposed noun_chunks: 5300 chunks, 2.77 tokens/chunk + +deprels on NOUN/PROPN/PRON tokens in dev (top 15): + nmod 2894 - + obl 1401 - + nsubj 1326 shipped + compound:lvc 1294 - + obl:arg 1095 - + obj 973 - + conj 583 shipped + flat:name 389 - + ROOT 101 shipped + xcomp 89 - + appos 51 shipped + nsubj:pass 38 - +``` + +1.31 tokens per chunk is the tell: the shipped iterator is returning single head nouns. +`nmod` (2894), `obl` (1401), `obl:arg` (1095) and `obj` (973) — the four most common +noun-bearing relations after `nsubj` — are all unreachable. + +Live text: + +``` +رئیس انجمن جراحان قلب ایران تأکید کرد که امکانات پیشرفته در ایران وجود دارد. + shipped : ['رئیس', 'امکانات'] + proposed: ['رئیس انجمن جراحان قلب ایران', 'امکانات پیشرفته', 'ایران'] + +دانشگاه تهران بزرگ‌ترین دانشگاه ایران است. + shipped : ['دانشگاه'] + proposed: ['دانشگاه تهران', 'ایران'] +``` + +## Proposed patch + +Swap in UD labels, expand right through modifier chains, and drop a leading `ADP`/`CCONJ` +exactly as `fr`/`es` already do: + +```diff +--- a/spacy/lang/fa/syntax_iterators.py ++++ b/spacy/lang/fa/syntax_iterators.py +@@ + def noun_chunks(doclike: Union[Doc, Span]) -> Iterator[Tuple[int, int, int]]: + """ + Detect base noun phrases from a dependency parse. Works on both Doc and Span. + """ +- labels = [ +- "nsubj", +- "dobj", +- "nsubjpass", +- "pcomp", +- "pobj", +- "dative", +- "appos", +- "attr", +- "ROOT", +- ] ++ # Persian pipelines are trained on Universal Dependencies treebanks ++ # (UD_Persian-PerDT, UD_Persian-Seraji), so these are UD relations. ++ labels = [ ++ "nsubj", ++ "nsubj:pass", ++ "obj", ++ "iobj", ++ "obl", ++ "obl:arg", ++ "nmod", ++ "appos", ++ "vocative", ++ "ROOT", ++ ] ++ # Persian noun phrases grow to the right through ezafe constructions. ++ post_modifiers = [ ++ "nmod", ++ "nmod:poss", ++ "amod", ++ "det", ++ "nummod", ++ "flat", ++ "flat:name", ++ "flat:num", ++ "fixed", ++ "compound", ++ ] + doc = doclike.doc # Ensure works on both Doc and Span. + + if not doc.has_annotation("DEP"): + raise ValueError(Errors.E029) + + np_deps = [doc.vocab.strings.add(label) for label in labels] ++ np_modifs = {doc.vocab.strings.add(label) for label in post_modifiers} + conj = doc.vocab.strings.add("conj") + np_label = doc.vocab.strings.add("NP") ++ adp_pos = doc.vocab.strings.add("ADP") ++ cconj_pos = doc.vocab.strings.add("CCONJ") + prev_end = -1 + for i, word in enumerate(doclike): + if word.pos not in (NOUN, PROPN, PRON): + continue + # Prevent nested chunks from being produced + if word.left_edge.i <= prev_end: + continue +- if word.dep in np_deps: +- prev_end = word.i +- yield word.left_edge.i, word.i + 1, np_label +- elif word.dep == conj: ++ if word.dep == conj: + head = word.head + while head.dep == conj and head.head.i < head.i: + head = head.head +- # If the head is an NP, and we're coordinated to it, we're an NP +- if head.dep in np_deps: +- prev_end = word.i +- yield word.left_edge.i, word.i + 1, np_label ++ # If the head is an NP, and we're coordinated to it, we're an NP ++ if head.dep not in np_deps: ++ continue ++ elif word.dep not in np_deps: ++ continue ++ # Expand right through the ezafe / modifier chain. ++ right = word ++ for child in word.rights: ++ if child.dep in np_modifs: ++ right = child.right_edge ++ else: ++ break ++ start, end = word.left_edge.i, max(word.i, right.i) + 1 ++ # A leading preposition or coordinator is not part of the NP. ++ while start < word.i and doc[start].pos in (adp_pos, cconj_pos): ++ start += 1 ++ if end <= prev_end: ++ continue ++ prev_end = end - 1 ++ yield start, end, np_label +``` + +The working implementation lives in `scripts/check_noun_chunks.py::proposed_noun_chunks`. + +## Test to add + +`spacy/tests/lang/fa/test_noun_chunks.py` currently has one test (a hand-built `Doc`, 296 B). +Add UD-label coverage: + +```python +def test_fa_noun_chunks_ezafe(fa_vocab): + # رئیس انجمن جراحان — "head of the surgeons' society" + words = ["رئیس", "انجمن", "جراحان", "آمد"] + heads = [3, 0, 1, 3] + deps = ["nsubj", "nmod", "nmod", "ROOT"] + pos = ["NOUN", "NOUN", "NOUN", "VERB"] + doc = Doc(fa_vocab, words=words, heads=heads, deps=deps, pos=pos) + assert [c.text for c in doc.noun_chunks] == ["رئیس انجمن جراحان"] + + +def test_fa_noun_chunks_drops_leading_adp(fa_vocab): + words = ["در", "ایران", "بود"] + heads = [1, 2, 2] + deps = ["case", "obl", "ROOT"] + pos = ["ADP", "NOUN", "VERB"] + doc = Doc(fa_vocab, words=words, heads=heads, deps=deps, pos=pos) + assert [c.text for c in doc.noun_chunks] == ["ایران"] +``` + +## Other `spacy/lang/fa` gaps found while building this pipeline + +Ordered by how much they cost a real Persian pipeline: + +1. **Clitic splitting.** The `fa` tokenizer cannot split pronominal enclitics or the enclitic + copula (`پدرم` → `پدر` + `م`, `ساکتند` → `ساکت` + `ند`), which UD treebanks annotate as + multiword tokens. Measured on PerDT dev: gold-vs-tokenizer token F is 0.9823 when clitics + are kept split, and 1.49% of tokens are affected. Persian-specific `TOKENIZER_SUFFIXES` + entries for the enclitic set would remove the need for `--merge-subtokens` and eliminate + the composite XPOS tags it produces. +2. **`punctuation.py` defines only `TOKENIZER_SUFFIXES`** — no `TOKENIZER_PREFIXES`, no + `TOKENIZER_INFIXES`. ZWNJ (U+200C) is handled only implicitly through the 65 KB generated + verb-exception table. The missing infix rules bite on numerics: `spacy/lang/fa/examples.py` + ships the sentence `دیروز علی به من ۲۰۰۰.۱﷼ پول نقد داد.`, and the trained pipeline splits + `۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, with the stray `.` promoted to a sentence boundary — + one input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian + digits, but no tokenizer rule keeps a Persian decimal or a currency sign attached. +3. **No tokenizer tests at all** for `fa` — `spacy/tests/lang/fa/` contains only + `test_noun_chunks.py`, guarding none of the 65 KB exception table. +4. **`spacy-lookups-data` has no `fa_license.txt`**, although `fa_source.txt` records that the + lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" — + which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`; Persian should too. +5. **No `fa_lemma_lookup.json`** — only rule-mode lemmatizer assets exist, so + `mode="lookup"` is unavailable for Persian. + +Items 1–3 are self-contained code PRs. Item 4 is a licence-hygiene PR against +`spacy-lookups-data` and matters for anyone redistributing a Persian pipeline. diff --git a/project.yml b/project.yml new file mode 100644 index 0000000..b2f750d --- /dev/null +++ b/project.yml @@ -0,0 +1,197 @@ +title: "fa_core_news_sm" +description: > + A CPU-sized Persian (fa) core pipeline for spaCy 3.8: tagger (XPOS), morphologizer + (UPOS + FEATS), trainable lemmatizer, dependency parser and NER. + + UD components are trained on UD_Persian-PerDT (PerUDT v1.0, CC BY-SA 4.0). The NER + component is trained separately on ParsTwiNER (MIT) and merged in, because no + redistributably-licensed Persian NER corpus shares a genre with the treebank. See + docs/MODELS.md for the full source/licence analysis and docs/CONTRIBUTING-GUIDE.md + for how this gets published. + + Run everything with: `spacy project run all` + +vars: + lang: "fa" + package_name: "core_news_sm" + package_version: "3.8.0" + treebank: "fa_perdt" + # -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline. + gpu: -1 + n_sents: 10 + +directories: + - "assets" + - "corpus" + - "configs" + - "scripts" + - "training" + - "metrics" + - "packages" + +assets: + - dest: "assets/ud/fa_perdt-ud-train.conllu" + url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-train.conllu" + checksum: "f5a8ba901a776b4fd1941ecadcc6d506" + description: "UD_Persian-PerDT train split (CC BY-SA 4.0)" + - dest: "assets/ud/fa_perdt-ud-dev.conllu" + url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-dev.conllu" + checksum: "f103020da7c1e917aafb8a8321f4cb84" + description: "UD_Persian-PerDT dev split (CC BY-SA 4.0)" + - dest: "assets/ud/fa_perdt-ud-test.conllu" + url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-test.conllu" + checksum: "b62a66994cef2c50f7e524a1471102d8" + description: "UD_Persian-PerDT test split (CC BY-SA 4.0)" + - dest: "assets/ner/ParsTwiNER_corpus_v1.0.0.zip" + url: "https://github.com/overfit-ir/parstwiner/releases/download/v1.0.0/ParsTwiNER_corpus_v1.0.0.zip" + checksum: "54615340d76c4d51b8f54751b07a278d" + description: "ParsTwiNER Persian Twitter NER corpus, IOB2 (MIT)" + +workflows: + all: + - inspect + - convert-ud + - convert-ner + - debug-data + - train-core + - train-ner + - assemble + - evaluate + - finalize-meta + - package + - smoke + +commands: + - name: "inspect" + help: "Report annotation coverage of the downloaded treebank(s)" + script: + - "python scripts/inspect_treebanks.py assets/ud" + deps: + - "assets/ud/fa_perdt-ud-train.conllu" + - "scripts/inspect_treebanks.py" + + - name: "convert-ud" + help: > + CoNLL-U -> DocBin. --merge-subtokens fuses multiword-token clitics into single + tokens; docs/MODELS.md §5 has the measurement that justifies it. + script: + - "python -m spacy convert assets/ud/${vars.treebank}-ud-train.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens" + - "python -m spacy convert assets/ud/${vars.treebank}-ud-dev.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens" + - "python -m spacy convert assets/ud/${vars.treebank}-ud-test.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens" + # Reproduce the measurement in docs/MODELS.md §5: convert the dev split WITHOUT + # merging and compare both against the tokenizer we actually ship. + - "python -m spacy convert assets/ud/${vars.treebank}-ud-dev.conllu corpus/split --converter conllu --n-sents ${vars.n_sents}" + - "python scripts/tokenization_report.py corpus/merged/${vars.treebank}-ud-dev.spacy corpus/split/${vars.treebank}-ud-dev.spacy" + deps: + - "assets/ud/${vars.treebank}-ud-train.conllu" + - "assets/ud/${vars.treebank}-ud-dev.conllu" + - "assets/ud/${vars.treebank}-ud-test.conllu" + outputs: + - "corpus/merged/${vars.treebank}-ud-train.spacy" + - "corpus/merged/${vars.treebank}-ud-dev.spacy" + - "corpus/merged/${vars.treebank}-ud-test.spacy" + + - name: "convert-ner" + help: "Unpack ParsTwiNER and convert its IOB2 files to DocBin" + script: + - "python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner" + - "python -m spacy convert assets/ner/train.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}" + - "python -m spacy convert assets/ner/dev.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}" + - "python -m spacy convert assets/ner/test.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}" + deps: + - "assets/ner/ParsTwiNER_corpus_v1.0.0.zip" + - "scripts/extract_parstwiner.py" + outputs: + - "corpus/ner/train.spacy" + - "corpus/ner/dev.spacy" + - "corpus/ner/test.spacy" + + - name: "debug-data" + help: "Validate both corpora against their configs before burning CPU on training" + script: + - "python -m spacy debug data configs/fa_core_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy" + - "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy" + deps: + - "corpus/merged/${vars.treebank}-ud-train.spacy" + - "corpus/ner/train.spacy" + - "configs/fa_core_news_sm.cfg" + - "configs/fa_ner_sm.cfg" + + - name: "train-core" + help: "Train tok2vec + tagger + morphologizer + trainable_lemmatizer + parser on PerDT" + script: + - "python -m spacy train configs/fa_core_news_sm.cfg --output training/core --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}" + deps: + - "corpus/merged/${vars.treebank}-ud-train.spacy" + - "corpus/merged/${vars.treebank}-ud-dev.spacy" + - "configs/fa_core_news_sm.cfg" + outputs: + - "training/core/model-best" + + - name: "train-ner" + help: "Train the standalone NER component (own internal tok2vec) on ParsTwiNER" + script: + - "python -m spacy train configs/fa_ner_sm.cfg --output training/ner --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy --gpu-id ${vars.gpu}" + deps: + - "corpus/ner/train.spacy" + - "corpus/ner/dev.spacy" + - "configs/fa_ner_sm.cfg" + outputs: + - "training/ner/model-best" + + - name: "assemble" + help: "Source the trained ner into the core pipeline and write full meta.json" + script: + - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version}" + deps: + - "training/core/model-best" + - "training/ner/model-best" + - "scripts/assemble_core.py" + outputs: + - "training/fa_core_news_sm" + + - name: "evaluate" + help: "Score the assembled pipeline on both held-out test sets" + script: + - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}" + - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/ner/test.spacy --output metrics/ner-test.json --gpu-id ${vars.gpu}" + deps: + - "training/fa_core_news_sm" + - "corpus/merged/${vars.treebank}-ud-test.spacy" + - "corpus/ner/test.spacy" + outputs: + - "metrics/ud-test.json" + - "metrics/ner-test.json" + + - name: "finalize-meta" + help: > + Re-assemble, this time folding the test scores into meta.json["performance"]. + Separate from `assemble` because the scores can only exist after `evaluate`, and + `evaluate` needs an assembled pipeline to score. Cheap: it only copies models. + script: + - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version} --ud-metrics metrics/ud-test.json --ner-metrics metrics/ner-test.json" + deps: + - "metrics/ud-test.json" + - "metrics/ner-test.json" + - "scripts/assemble_core.py" + + - name: "package" + help: "Build the installable wheel + sdist" + script: + - "python -m spacy package training/fa_core_news_sm packages --name ${vars.package_name} --version ${vars.package_version} --build sdist,wheel --force" + deps: + - "training/fa_core_news_sm" + outputs: + - "packages/${vars.lang}_${vars.package_name}-${vars.package_version}" + + - name: "smoke" + help: "Load the packaged pipeline and run it over real Persian text" + script: + - "python scripts/smoke_test.py training/fa_core_news_sm" + deps: + - "training/fa_core_news_sm" + + - name: "clean" + help: "Drop corpora, training runs and metrics (keeps downloaded assets)" + script: + - "rm -rf corpus/merged corpus/split corpus/ner training metrics packages" diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..6f84b88 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,8 @@ +# Build environment for fa_core_news_sm. Python 3.12. +spacy>=3.8,<3.9 +# Persian rule-lemmatizer tables (fa_lemma_exc/index/rules). Not used by the trained +# pipeline (we train an edit-tree lemmatizer) but required if you want to compare against +# the rule-based lemmatizer, and pulled in by `spacy init config` validation paths. +spacy-lookups-data>=1.0.5 +# Publishing to the HF Hub (docs/CONTRIBUTING-GUIDE.md §2). Optional. +# spacy-huggingface-hub>=0.0.10 diff --git a/scripts/assemble_core.py b/scripts/assemble_core.py new file mode 100644 index 0000000..60b7a5b --- /dev/null +++ b/scripts/assemble_core.py @@ -0,0 +1,133 @@ +"""Merge the separately-trained NER component into the UD pipeline and write full metadata. + +The two components cannot be trained together: the UD components come from +UD_Persian-PerDT (CC BY-SA 4.0, edited prose) and `ner` comes from ParsTwiNER (MIT, +tweets). `ner` was therefore configured with its own embedded tok2vec (see +configs/fa_ner_sm.cfg) so it can be sourced into another pipeline without a dangling +Tok2VecListener — the same design as en_core_web_sm. + +Run it twice: once right after training (no metrics yet), and again after +`spacy benchmark accuracy` has produced metrics/*.json so `meta.json["performance"]` +reflects the assembled pipeline on the held-out test sets. + +Usage: + .venv/bin/python scripts/assemble_core.py training/core/model-best training/ner/model-best \\ + training/fa_core_news_sm --version 3.8.0 \\ + [--ud-metrics metrics/ud-test.json] [--ner-metrics metrics/ner-test.json] +""" + +import argparse +import json +from pathlib import Path + +import spacy + +DESCRIPTION = ( + "Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, " + "trainable_lemmatizer, parser, ner." +) + +SOURCES = [ + { + "name": "UD_Persian-PerDT (PerUDT v1.0)", + "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT", + "author": ( + "Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian" + ), + "license": "CC BY-SA 4.0", + }, + { + "name": "ParsTwiNER", + "url": "https://github.com/overfit-ir/parstwiner", + "author": "MohammadMahdi Aghajani, AliAkbar Badri, Hamid Beigy et al. (Overfit-IR)", + "license": "MIT", + }, + { + "name": "spaCy lang/fa language data (stop words originally from HAZM)", + "url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa", + "author": "Explosion and spaCy contributors", + "license": "MIT", + }, +] + +# CC BY-SA 4.0 on the treebank propagates to anything derived from it. +NOTES = ( + "The tagger, morphologizer, trainable_lemmatizer and parser are trained on " + "UD_Persian-PerDT, which is licensed CC BY-SA 4.0; this pipeline is therefore " + "distributed under CC BY-SA 4.0 with attribution to the treebank authors. " + "The ner component is trained on ParsTwiNER (MIT), a Twitter corpus, so entity " + "recognition is weaker on formal/edited prose than on social media text. " + "Multiword tokens in the treebank (pronominal clitics, enclitic copulas) were merged " + "with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite " + "(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space." +) + + +def load_metrics(path): + if not path: + return {} + p = Path(path) + if not p.exists(): + print(f" (no metrics at {p}, skipping)") + return {} + return json.loads(p.read_text(encoding="utf8")) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("core", help="trained UD pipeline (training/core/model-best)") + ap.add_argument("ner", help="trained NER pipeline (training/ner/model-best)") + ap.add_argument("output", help="destination directory for the assembled pipeline") + ap.add_argument("--version", default="3.8.0") + ap.add_argument("--ud-metrics", default=None) + ap.add_argument("--ner-metrics", default=None) + args = ap.parse_args() + + nlp = spacy.load(args.core) + ner_nlp = spacy.load(args.ner) + if "ner" in nlp.pipe_names: + nlp.remove_pipe("ner") + nlp.add_pipe("ner", source=ner_nlp) + print(f"pipeline: {nlp.pipe_names}") + print(f"ner labels: {sorted(nlp.get_pipe('ner').labels)}") + + ud = load_metrics(args.ud_metrics) + ner = load_metrics(args.ner_metrics) + performance = dict(nlp.meta.get("performance", {})) + for key in ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc", + "morph_acc", "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r", + "sents_f"): + if key in ud: + performance[key] = ud[key] + for key in ("ents_p", "ents_r", "ents_f"): + if key in ner: + performance[key] = ner[key] + if "ents_per_type" in ner: + performance["ents_per_type"] = ner["ents_per_type"] + if "dep_las_per_type" in ud: + performance["dep_las_per_type"] = ud["dep_las_per_type"] + + nlp.meta.update( + { + "lang": "fa", + "name": "core_news_sm", + "version": args.version, + "description": DESCRIPTION, + "author": "", + "email": "", + "url": "", + "license": "CC BY-SA 4.0", + "sources": SOURCES, + "notes": NOTES, + "performance": performance, + } + ) + + out = Path(args.output) + nlp.to_disk(out) + print(f"wrote {out}") + print(json.dumps(performance, indent=2, ensure_ascii=False)[:1200]) + + +if __name__ == "__main__": + main() diff --git a/scripts/check_noun_chunks.py b/scripts/check_noun_chunks.py new file mode 100644 index 0000000..bf4a39f --- /dev/null +++ b/scripts/check_noun_chunks.py @@ -0,0 +1,137 @@ +"""Evidence for the upstream `spacy/lang/fa/syntax_iterators.py` bug. + +The shipped Persian `noun_chunks` iterator matches these dependency labels: + + nsubj, dobj, nsubjpass, pcomp, pobj, dative, appos, attr, ROOT + +`dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They do not +exist in Universal Dependencies, and every Persian treebank is UD. So on a UD-trained `fa` +pipeline the iterator only ever fires on `nsubj`, `appos`, `ROOT` and `conj`. + +It also expands each chunk to `word.left_edge`, which is wrong for Persian: ezafe +constructions put modifiers to the RIGHT of the head noun (`رئیس انجمن جراحان قلب ایران`), +so left-only expansion truncates the phrase to the head. + +This script quantifies both problems against the dev corpus and prints a side-by-side +comparison with the proposed UD implementation (see docs/upstream/fa-noun-chunks.md). + +Usage: + .venv/bin/python scripts/check_noun_chunks.py training/core/model-best \\ + corpus/merged/fa_perdt-ud-dev.spacy +""" + +import sys +from collections import Counter + +import spacy +from spacy.lang.fa.syntax_iterators import noun_chunks as shipped_noun_chunks +from spacy.symbols import NOUN, PRON, PROPN +from spacy.tokens import DocBin + +# Labels a UD-trained parser can actually emit, as heads of a base noun phrase. +UD_LABELS = [ + "nsubj", + "nsubj:pass", + "obj", + "iobj", + "obl", + "obl:arg", + "nmod", + "appos", + "vocative", + "ROOT", +] +# Relations that continue a Persian noun phrase to the right of its head. +UD_POST_MODIFIERS = ["nmod", "nmod:poss", "amod", "flat", "flat:name", "flat:num", + "fixed", "compound", "det", "nummod"] + + +def proposed_noun_chunks(doclike): + """UD-label implementation with right-side ezafe expansion.""" + doc = doclike.doc + if not doc.has_annotation("DEP"): + raise ValueError("requires a dependency parse") + np_deps = {doc.vocab.strings.add(label) for label in UD_LABELS} + np_modifs = {doc.vocab.strings.add(label) for label in UD_POST_MODIFIERS} + np_label = doc.vocab.strings.add("NP") + conj = doc.vocab.strings.add("conj") + adp_pos = doc.vocab.strings.add("ADP") + cconj_pos = doc.vocab.strings.add("CCONJ") + prev_end = -1 + for word in doclike: + if word.pos not in (NOUN, PROPN, PRON): + continue + if word.left_edge.i <= prev_end: + continue + head_dep = word.dep + if head_dep == conj: + head = word.head + while head.dep == conj and head.head.i < head.i: + head = head.head + if head.dep not in np_deps: + continue + elif head_dep not in np_deps: + continue + # Expand right through ezafe / modifier chains. + right = word + for child in word.rights: + if child.dep in np_modifs: + right = child.right_edge + else: + break + start, end = word.left_edge.i, max(word.i, right.i) + 1 + # Mirror fr/es: a leading preposition or coordinator is not part of the NP. + while start < word.i and doc[start].pos in (adp_pos, cconj_pos): + start += 1 + if end <= prev_end: + continue + prev_end = end - 1 + yield start, end, np_label + + +def spans(doc, iterator): + return [doc[s:e].text for s, e, _ in iterator(doc)] + + +def main(): + model, corpus = sys.argv[1], sys.argv[2] + nlp = spacy.load(model) + docs = list(DocBin().from_disk(corpus).get_docs(nlp.vocab)) + + shipped_total = proposed_total = 0 + shipped_tokens = proposed_tokens = 0 + dep_hist = Counter() + for doc in docs: + for t in doc: + if t.pos in (NOUN, PROPN, PRON): + dep_hist[t.dep_] += 1 + for s, e, _ in shipped_noun_chunks(doc): + shipped_total += 1 + shipped_tokens += e - s + for s, e, _ in proposed_noun_chunks(doc): + proposed_total += 1 + proposed_tokens += e - s + + print(f"gold dev corpus: {len(docs)} docs") + print(f"shipped noun_chunks: {shipped_total:>6} chunks, " + f"{shipped_tokens / max(shipped_total, 1):.2f} tokens/chunk") + print(f"proposed noun_chunks: {proposed_total:>6} chunks, " + f"{proposed_tokens / max(proposed_total, 1):.2f} tokens/chunk") + print("\ndeprels on NOUN/PROPN/PRON tokens in dev (top 15):") + for dep, n in dep_hist.most_common(15): + reachable = "shipped" if dep in {"nsubj", "appos", "conj", "ROOT", "root"} else "-" + print(f" {dep:<14}{n:>7} {reachable}") + + print("\nside by side on live text:") + for text in [ + "رئیس انجمن جراحان قلب ایران تأکید کرد که امکانات پیشرفته در ایران وجود دارد.", + "دانشگاه تهران بزرگ‌ترین دانشگاه ایران است.", + ]: + doc = nlp(text) + print(f"\n {text}") + print(f" shipped : {spans(doc, shipped_noun_chunks)}") + print(f" proposed: {spans(doc, proposed_noun_chunks)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/extract_parstwiner.py b/scripts/extract_parstwiner.py new file mode 100644 index 0000000..9bf2994 --- /dev/null +++ b/scripts/extract_parstwiner.py @@ -0,0 +1,34 @@ +"""Unpack the ParsTwiNER release zip into flat IOB2 files. + +The archive carries macOS resource-fork junk (`__MACOSX/._*`) that would confuse +`spacy convert`, so only the three real splits are extracted. + +Usage: .venv/bin/python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner +""" + +import sys +import zipfile +from pathlib import Path + +SPLITS = ("train.txt", "dev.txt", "test.txt") + + +def main(): + archive = Path(sys.argv[1]) + dest = Path(sys.argv[2]) + dest.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(archive) as zf: + names = set(zf.namelist()) + missing = [s for s in SPLITS if s not in names] + if missing: + raise SystemExit(f"{archive}: missing expected members {missing}") + for split in SPLITS: + zf.extract(split, dest) + path = dest / split + sents = path.read_text(encoding="utf8").strip().split("\n\n") + tokens = sum(1 for line in path.open(encoding="utf8") if line.strip()) + print(f"{path}: {len(sents)} sentences, {tokens} tokens") + + +if __name__ == "__main__": + main() diff --git a/scripts/inspect_treebanks.py b/scripts/inspect_treebanks.py new file mode 100644 index 0000000..6f88724 --- /dev/null +++ b/scripts/inspect_treebanks.py @@ -0,0 +1,90 @@ +"""Report annotation coverage of the UD Persian treebanks in assets/ud/. + +Usage: .venv/bin/python scripts/inspect_treebanks.py [assets/ud] +""" + +import sys +from collections import Counter +from pathlib import Path + + +def read_conllu(path): + sent = [] + for line in path.open(encoding="utf8"): + line = line.rstrip("\n") + if not line: + if sent: + yield sent + sent = [] + continue + if line.startswith("#"): + continue + sent.append(line.split("\t")) + if sent: + yield sent + + +def stats(path): + s = { + "sents": 0, + "toks": 0, + "mwt": 0, + "empty": 0, + "lemma": 0, + "feats": 0, + "nonproj_root": 0, + } + upos, xpos, dep = Counter(), Counter(), Counter() + for sent in read_conllu(path): + s["sents"] += 1 + roots = 0 + for c in sent: + if "-" in c[0]: + s["mwt"] += 1 + continue + if "." in c[0]: + s["empty"] += 1 + continue + s["toks"] += 1 + upos[c[3]] += 1 + xpos[c[4]] += 1 + dep[c[7]] += 1 + if c[2] not in ("_", ""): + s["lemma"] += 1 + if c[5] != "_": + s["feats"] += 1 + if c[7] == "root": + roots += 1 + if roots != 1: + s["nonproj_root"] += 1 + return s, upos, xpos, dep + + +def main(): + root = Path(sys.argv[1] if len(sys.argv) > 1 else "assets/ud") + agg = {} + print( + f"{'file':<30}{'sents':>8}{'tokens':>10}{'MWT':>7}{'empty':>7}" + f"{'lemma%':>8}{'feats%':>8}{'UPOS':>6}{'XPOS':>6}{'DEP':>5}" + ) + for path in sorted(root.glob("*.conllu")): + s, upos, xpos, dep = stats(path) + agg[path.name] = (s, upos, xpos, dep) + print( + f"{path.name:<30}{s['sents']:>8}{s['toks']:>10}{s['mwt']:>7}{s['empty']:>7}" + f"{100 * s['lemma'] / s['toks']:>8.1f}{100 * s['feats'] / s['toks']:>8.1f}" + f"{len(upos):>6}{len(xpos):>6}{len(dep):>5}" + ) + + for name, (s, upos, xpos, dep) in agg.items(): + if not name.endswith("train.conllu"): + continue + print(f"\n=== {name} ===") + print(f"UPOS ({len(upos)}): {' '.join(sorted(upos))}") + print(f"XPOS ({len(xpos)}): {' '.join(sorted(xpos))}") + print(f"DEPREL ({len(dep)}): {' '.join(sorted(dep))}") + print(f"sentences with != 1 root: {s['nonproj_root']}") + + +if __name__ == "__main__": + main() diff --git a/scripts/smoke_test.py b/scripts/smoke_test.py new file mode 100644 index 0000000..ca496d0 --- /dev/null +++ b/scripts/smoke_test.py @@ -0,0 +1,45 @@ +"""Run the assembled pipeline over real Persian text and print every annotation layer. + +This is the end-to-end check that the artifact actually works: tokenizer -> tagger -> +morphologizer -> lemmatizer -> parser -> ner -> noun_chunks. + +Usage: .venv/bin/python scripts/smoke_test.py training/fa_core_news_sm +""" + +import sys + +import spacy +from spacy.lang.fa.examples import sentences as FA_EXAMPLES + +EXTRA = [ + # ZWNJ-heavy verb forms, an enclitic pronoun, and named entities. + "دانشگاه تهران در سال ۱۳۱۳ تأسیس شد و بزرگ‌ترین دانشگاه ایران است.", + "کتاب‌هایش را روی میز گذاشت و به سرعت از خانه بیرون رفت.", + "شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.", +] + + +def main(): + path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_core_news_sm" + nlp = spacy.load(path) + print(f"loaded {nlp.meta['lang']}_{nlp.meta['name']} {nlp.meta['version']}") + print(f"pipeline: {nlp.pipe_names}") + + for text in list(FA_EXAMPLES) + EXTRA: + doc = nlp(text) + print("\n" + "=" * 78) + print(text) + print(f"{'TEXT':<16}{'LEMMA':<16}{'UPOS':<7}{'TAG':<14}{'DEP':<14}HEAD") + for t in doc: + print( + f"{t.text:<16}{t.lemma_:<16}{t.pos_:<7}{t.tag_:<14}" + f"{t.dep_:<14}{t.head.text}" + ) + print(f"morph[0]: {doc[0].morph}") + print(f"sents: {[s.text for s in doc.sents]}") + print(f"ents: {[(e.text, e.label_) for e in doc.ents]}") + print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}") + + +if __name__ == "__main__": + main() diff --git a/scripts/tokenization_report.py b/scripts/tokenization_report.py new file mode 100644 index 0000000..534ee32 --- /dev/null +++ b/scripts/tokenization_report.py @@ -0,0 +1,63 @@ +"""Compare gold tokenisation against spaCy's rule-based `fa` tokenizer. + +Answers the only question that decides `spacy convert --merge-subtokens` for Persian: +how often can the tokenizer we ship at runtime reproduce the gold token boundaries? + +Usage: .venv/bin/python scripts/tokenization_report.py corpus/merged/fa_perdt-ud-dev.spacy [...] +""" + +import sys +from collections import Counter +from pathlib import Path + +import spacy +from spacy.tokens import DocBin + + +def offsets(doc): + return {(t.idx, t.idx + len(t.text)) for t in doc} + + +def report(path, nlp): + gold_docs = list(DocBin().from_disk(path).get_docs(nlp.vocab)) + tp = gold_n = pred_n = 0 + tags = Counter() + multi_lemma = 0 + tokens = 0 + for gold in gold_docs: + pred = nlp.make_doc(gold.text) + g, p = offsets(gold), offsets(pred) + tp += len(g & p) + gold_n += len(g) + pred_n += len(p) + for t in gold: + tokens += 1 + tags[t.tag_] += 1 + if " " in t.lemma_: + multi_lemma += 1 + precision = tp / pred_n + recall = tp / gold_n + f = 2 * precision * recall / (precision + recall) + composite = {t: n for t, n in tags.items() if "_" in t and t.count("_") > 1} + print(f"\n== {path}") + print(f"docs {len(gold_docs)} gold tokens {gold_n} predicted tokens {pred_n}") + print(f"token P {precision:.4f} R {recall:.4f} F {f:.4f}") + print(f"tag types {len(tags)}") + print(f"tags containing >1 underscore (merge artefacts): {len(composite)}" + f" covering {sum(composite.values())} tokens" + f" ({100 * sum(composite.values()) / tokens:.2f}%)") + if composite: + top = ", ".join(f"{t}={n}" for t, n in Counter(composite).most_common(8)) + print(f" most frequent: {top}") + print(f"lemmas containing a space (merge artefacts): {multi_lemma}" + f" ({100 * multi_lemma / tokens:.2f}%)") + + +def main(): + nlp = spacy.blank("fa") + for arg in sys.argv[1:]: + report(Path(arg), nlp) + + +if __name__ == "__main__": + main()