15 changed files with 1153 additions and 716 deletions
--- a/.gitignore
+++ b/.gitignore
@ -6,5 +6,12 @@ training/
 metrics/
 packages/
 .venv/
 # Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the
 # spaCy CLI in the training venv. See .omp/AGENTS.md.
 .venv-publish/
 __pycache__/
 *.pyc
 # Personal scratch list, not part of the project
 TODO.md
 # Agent/session notes, local only
 .omp/
--- a/.omp/AGENTS.md
+++ b/.omp/AGENTS.md
@ -1,79 +0,0 @@
 # Repo context for agents
 ## Remotes
 | Name | URL | Purpose |
 |---|---|---|
 | `gitea` | `https://gitea.cap.nlogn.ir/fazel/spacy-fa-pipeline.git` | Primary self-hosted Gitea |
 ## Pushing to Gitea
 Credentials come from `pass` — no token in git config or any tracked file.
 **Pass path:** `gitea/token_AllExceptAdmin`
 The credential helper is set locally in `.git/config` (never committed):
 ```
 credential.https://gitea.cap.nlogn.ir.helper=
  !f() { echo username=fazel; echo password=$(pass gitea/token_AllExceptAdmin); }; f
 ```
 **Push command** (bypass proxy — Gitea is on the local network):
 ```bash
 NO_PROXY="*" no_proxy="*" http_proxy="" https_proxy="" HTTP_PROXY="" HTTPS_PROXY="" \
  git push gitea main
 ```
 ### If `pass` returns an empty string
 The store is encrypted to `caci96@gmail.com`, whose secret key lives on an **OpenPGP
 smartcard** (`gpg --list-secret-keys --with-colons` shows serial `D276000124010304…`). When
 gpg-agent has no cached session and no pinentry can reach a terminal, `pass` blocks for ~60s
 and exits with empty stdout — it does *not* error. A credential helper wired to it then fails
 auth for no visible reason.
 Diagnose, do not guess:
 ```bash
 export GPG_TTY=$(tty)
 gpg --batch --pinentry-mode error --decrypt ~/.password-store/gitea/token_AllExceptAdmin.gpg
 ```
 Success means the agent is unlocked and `pass` will work. `Bad passphrase`/`No secret key`
 means insert the smartcard, or `export GPG_TTY=$(tty)` and re-run `pass` once interactively to
 satisfy pinentry.
 The same token is mirrored in `rbw` at `api/gitea_token_AllExceptAdmin` (verified
 byte-identical). It needs no smartcard, so it is the fallback when the card is unavailable:
 ```
 !f() { echo username=fazel; echo password=$(rbw get "api/gitea_token_AllExceptAdmin" | head -1); }; f
 ```
 To rotate: update in the Gitea UI, then `pass edit gitea/token_AllExceptAdmin` **and**
 `rbw edit "api/gitea_token_AllExceptAdmin"` so the two stay in sync. No git config change.
 ## What is NOT committed
 `.gitignore` excludes `assets/ corpus/ training/ metrics/ packages/ .venv/`. The repo holds
 only source: configs, scripts, `project.yml`, docs. Everything else is regenerated:
 ```bash
 python -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
 .venv/bin/python -m spacy project assets    # checksummed downloads
 .venv/bin/python -m spacy project run all   # ~2h on 4 CPU cores
 ```
 The trained wheel (13 MB) is a build artifact, not source. Publish it to the HF Hub instead —
 see `docs/CONTRIBUTING-GUIDE.md` §2.
 ## Environment
 The venv is Python 3.12.2, created from the conda env at `/home/fazel/anaconda3/envs/p12`
 (the anaconda base python is 3.7 and cannot run spaCy 3.8). The IPython kernel available to
 agents is bound to that 3.7 base and will fail on this project — shell out to `.venv/bin/python`.
 GPU (GTX 940MX, 2 GB) is unused: too small for a transformer, not worth the transfer overhead
 for an `sm` pipeline. All training is CPU, `--gpu-id -1`.
--- a/53
+++ b/53
@ -0,0 +1,53 @@
 This repository contains two kinds of material under two different licences.
 1. SOURCE CODE (scripts/, configs/, project.yml, docs/) — MIT, below.
 2. TRAINED PIPELINES (fa_dep_news_sm, fa_core_news_sm, fa_ent_news_sm, and any
   artifact under training/ or packages/) — Creative Commons Attribution-ShareAlike
   4.0 International (CC BY-SA 4.0).
   These are Adapted Material derived from UD_Persian-PerDT (PerUDT v1.0), which is
   licensed CC BY-SA 4.0, so the ShareAlike condition requires the same licence.
     Source:  https://github.com/UniversalDependencies/UD_Persian-PerDT
     Licence: https://creativecommons.org/licenses/by-sa/4.0/
     Authors: Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi,
              Alireza Nourian
   The entity annotations additionally derive from that treebank's
   not-to-release/Dadegan with NER tag/ layer, which its README states was produced
   with the Beheshti-NER tagger (Taher, Hoseini, Shamsfard 2020) plus manual
   corrections.
   Modifications made to the licensed material: CoNLL-U converted to spaCy DocBin
   with multiword tokens merged (`spacy convert --merge-subtokens`); entity spans
   realigned onto that tokenization by difflib, with spans that could not be aligned
   exactly dropped rather than guessed. See docs/MODELS.md.
   The licensed material is provided as-is, without warranties of any kind, and the
   licensor disclaims liability for damages arising from its use, to the extent
   permitted by CC BY-SA 4.0 sections 5(a) and 5(b).
 -----------------------------------------------------------------------------------
 MIT License
 Copyright (c) 2026 Kiyarash Fazeli
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
--- a/README.md
+++ b/README.md
@ -1,108 +1,143 @@
-# fa_core_news_sm — a Persian pipeline for spaCy
+# fa_core_news_sm and fa_dep_news_sm, Persian pipelines for spaCy
-There is no trained Persian pipeline for spaCy. `spacy.load("fa_core_news_sm")` has never
+spaCy has no trained Persian pipeline. `spacy.load("fa_core_news_sm")` has never worked, and
-worked; `spacy.blank("fa")` gives you a tokenizer and stop words and nothing else. This
+`spacy.blank("fa")` gives you a tokenizer and stop words. This project trains one from
-project builds the missing pipeline from openly-licensed data, using spaCy's own tooling, so
+openly-licensed data so the result can be redistributed.
 the result can actually be redistributed.
- **What the four English pipelines are, and what the four Persian equivalents should be:**
+- Pipeline inventory and source analysis: [`docs/MODELS.md`](docs/MODELS.md)
-  [`docs/MODELS.md`](docs/MODELS.md)
+- How spaCy models get published, and what upstream `fa` already has:
- **How models get contributed/published in the spaCy ecosystem, and what upstream `fa`
+  [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
-  already has:** [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
+- The build: [`project.yml`](project.yml)
 - **The build itself:** [`project.yml`](project.yml)
-## The short version
+## Two packages, one corpus
-| | |
+In spaCy's naming scheme `dep` = tagger + parser + lemmatizer, `core` = the same plus NER.
-| --- | --- |
+Both packages here are built entirely from UD_Persian-PerDT and differ only in whether NER is
-| Pipeline | `fa_core_news_sm` — tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner |
+included.
 | Syntax/morphology data | [UD_Persian-PerDT](https://github.com/UniversalDependencies/UD_Persian-PerDT) (PerUDT v1.0) — 29,107 sentences, **CC BY-SA 4.0** |
 | NER data | [ParsTwiNER](https://github.com/overfit-ir/parstwiner) — 7,667 tweets, **MIT** |
 | Language data | `spacy/lang/fa` upstream (its stop word list comes from hazm) |
 | Licence of the result | CC BY-SA 4.0 (inherited from the treebank) |
 | Hardware | 4-core CPU. No GPU needed. |
-### Results
+| Package | Components | Licence | Score | Wheel |
 | --- | --- | --- | --- | --- |
 | `fa_dep_news_sm` | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | CC BY-SA 4.0 | LAS 85.15, LEMMA 97.91 | 7.5 MB |
 | `fa_core_news_sm` | the above plus ner | CC BY-SA 4.0 | LAS 85.15, ENTS_F 71.87 | 13 MB |
-Trained and evaluated on this laptop (4-core i5-7200U, CPU only, 1h27m for the UD
+The NER is possible because the treebank ships its own entity layer in
-components, ~25 min for NER). Scores are on the **held-out test splits**, produced by
+`not-to-release/Dadegan with NER tag/`: 15,833 entities over the same 29,107 sentences, under
-`spacy benchmark accuracy` and stored in `metrics/`.
+the same CC BY-SA 4.0. That is what makes `core` honest here, since one corpus means one genre,
 one tokenization, one licence and one provenance chain. The alternative NER corpora are all
 worse on at least one of those axes: ARMAN, PEYMA and NSURL are research-use-only, and
 ParsTwiNER (MIT) is a Twitter corpus that costs about 23 F on prose.
-| Metric | `fa_core_news_sm` | reference |
+Two caveats to know before relying on the entities:
 - **The labels are silver.** The treebank README states they came from the BERT-based
  Beheshti-NER tagger with manual corrections for recall, so `ENTS_F 71.87` is measured against
  a silver test split and partly reflects agreement with that tagger.
 - **Three labels are thin.** `MON` (205 training examples), `TIM` (135) and `PCT` (121) score
  73.7, 66.7 and 57.1. `PER`, `LOC`, `ORG` and `DAT` have 1,300 or more each.
 Entity spans were transferred onto this pipeline's tokenization by difflib alignment at a 99.86%
 rate; spans that could not be aligned exactly were dropped rather than guessed
 (`scripts/transfer_perdt_ner.py`).
 Language data comes from `spacy/lang/fa` upstream, whose stop word list came from hazm.
 Everything trains on 4 CPU cores with no GPU.
 ## Results
 Held-out test splits, from `spacy benchmark accuracy`, stored in `metrics/`. Trained on a
 4-core i5-7200U: 1h27m for the UD components, 17 min for NER.
 Syntax and morphology, identical in both packages since they share the same trained components:
 | Metric | Score | Reference |
 | --- | --- | --- |
 | `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | |
-| `TAG_ACC` (XPOS) | **95.96** | |
+| `TAG_ACC` (XPOS) | 95.96 | |
-| `POS_ACC` (UPOS) | **96.24** | |
+| `POS_ACC` (UPOS) | 96.24 | |
 | `MORPH_ACC` | 96.29 | |
-| `LEMMA_ACC` | **97.91** | |
+| `LEMMA_ACC` | 97.91 | |
 | `SENTS_F` | 99.25 | |
-| `DEP_UAS` | **89.69** | hazm+ParsBERT: 92.46 |
+| `DEP_UAS` | 89.69 | hazm+ParsBERT: 92.46 |
-| `DEP_LAS` | **85.15** | hazm+ParsBERT: 89.34 |
+| `DEP_LAS` | 85.15 | hazm+ParsBERT: 89.34 |
-| `ENTS_P` / `ENTS_R` / `ENTS_F` | 74.77 / 61.06 / **67.22** | |
+| Speed | ~9,250 words/s | |
 | Speed | ~9,250 words/s (CPU) | |
 | Wheel size | 13 MB | `en_core_web_sm`: 12 MB |
-Per-entity F: `LOC` 73.9, `PER` 69.1, `NAT` 63.2, `ORG` 59.3, `POG` 41.2, `EVE` 30.0.
+Entities, `fa_core_news_sm` only, on the PerDT NER test split: `ENTS_P` 77.67, `ENTS_R` 66.87,
 `ENTS_F` 71.87. Per label:
-Read these honestly:
+| Label | F | Train examples |
 | --- | --- | --- |
 | `LOC` | 80.24 | 4,954 |
 | `DAT` | 74.45 | 1,323 |
 | `MON` | 73.68 | 205 |
 | `ORG` | 68.77 | 2,643 |
 | `TIM` | 66.67 | 135 |
 | `PER` | 65.29 | 4,847 |
 | `PCT` | 57.14 | 121 |
- **Parsing is 4.2 LAS behind hazm's parser**, which is the expected gap between a 13 MB
+Parsing is 4.2 LAS behind hazm's parser, which uses the same corpus and the same spaCy parser
-  CPU model with hash embeddings and a fine-tuned ParsBERT. It is the same corpus and the
+architecture with a fine-tuned ParsBERT instead of hash embeddings. That gap is the target for
-  same spaCy parser architecture, so the comparison is fair, and it sets the target for the
+a future `trf` tier.
  future `trf` tier.
 - **NER is the weak component.** 67 F reflects three compounding handicaps: an `sm` model
  with no static vectors, a 233k-token training corpus, and a genre mismatch (trained on
  tweets, most users will run it on prose). `EVE` and `POG` are near-useless. This is the
  price of using the only MIT-licensed Persian NER corpus that exists.
 - **Everything else is competitive with the English `sm` pipeline** (`en_core_web_sm`:
  TAG 97, LAS 90, ENTS_F 84 — on a much larger and cleaner corpus).
-Reproduce: `.venv/bin/python -m spacy project run all`.
+`PER` scoring below `LOC` and `ORG` despite having 4,847 examples is the silver labels showing
 through: PerDT includes titles and honorifics inside `PER` spans inconsistently (6.24% of spans
 start with one, against 1.41% in the human-annotated ParsTwiNER), so the boundaries the model
 has to learn are less regular than the label count suggests.
-### Install the built pipeline
+For comparison, `en_core_web_sm` scores TAG 97, LAS 90, ENTS_F 84 on a larger, cleaner corpus.
 Reproduce with `.venv/bin/python -m spacy project run all`, plus `run ent` for an NER-only
 package.
 ## Install
 ```bash
 .venv/bin/python -m pip install packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl
 # or, without NER:
 .venv/bin/python -m pip install packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl
 ```
 ```python
 import spacy
 nlp = spacy.load("fa_core_news_sm")
-doc = nlp("دانشگاه تهران در سال ۱۳۱۳ تأسیس شد.")
+doc = nlp("محمدرضا شجریان در مشهد به دنیا آمد.")
-print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
+print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc][:3])
-print(doc.ents)   # (دانشگاه تهران, ORG)
+# [('محمدرضا', 'PROPN', 'محمدرضا', 'nsubj'), ('شجریان', 'PROPN', 'شجریان', 'flat:name'), ...]
 print(doc.ents)   # (محمدرضا شجریان, مشهد)  -> PER, LOC
 doc = nlp("شرکت ایران خودرو تولید را ۲۰ درصد افزایش می‌دهد.")
 print([(e.text, e.label_) for e in doc.ents])   # ۲۰ درصد -> PCT
 ```
-### Why not hazm's own models
+## Why not hazm's own models
-hazm is the reference Persian NLP toolkit and it *does* publish spaCy-format pipelines on the
+hazm is the reference Persian NLP toolkit and publishes spaCy-format pipelines on the HF Hub,
-HF Hub, so it was the obvious starting point. It does not survive contact:
+so it was the obvious starting point. Four problems:
- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). There is no
+- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). The repo contains no
-  `config.cfg` and no `spacy train` anywhere in the repo — the `Spacy*` classes only download
+  `config.cfg` and no `spacy train`; the `Spacy*` classes only download pretrained pipelines.
-  pretrained pipelines.
+- Those pipelines are three single-task models (`transformer + tagger`, `transformer + parser`,
- Those pretrained pipelines are three *single-task* models (`transformer + tagger`,
+  `transformer + chunker`), each `version: 0.0.0` with an empty `license` field, pinned to
-  `transformer + parser`, `transformer + chunker`), each `version: 0.0.0` with an empty
+  spaCy 3.6. Using all three costs three ParsBERT forward passes and gives no shared `Doc`.
-  `license` field, pinned to spaCy 3.6. Using all three means three ParsBERT forward passes
+- Its tokenizer is incompatible with UD tokenization: the normaliser fuses ZWNJ affixes and
-  over the same text and no shared `Doc`.
+  `join_verb_parts()` glues multi-word verb chains into single tokens.
- Its tokenizer is deliberately incompatible with UD tokenization: the normaliser fuses ZWNJ
+- Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) sit behind `peykaregan.ir`
-  affixes and `join_verb_parts()` glues multi-word verb chains into single tokens.
+  or `dadegan.ir` under research-only terms.
 - Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) are gated behind
  `peykaregan.ir` / `dadegan.ir` under research-only terms.
-What hazm *does* give us: confirmation of the corpus choice — hazm's own spaCy parser was
+It did confirm the corpus choice. hazm's own spaCy parser was trained on
-trained on `modified_fa_perdt-ud-train.spacy`, i.e. the same treebank we use — plus the stop
+`modified_fa_perdt-ud-train.spacy`, the same treebank used here. Full analysis in
 word list already vendored into `spacy/lang/fa`. Full analysis in
 [`docs/MODELS.md`](docs/MODELS.md) §4.
-### Why licensing is the load-bearing constraint
+## Licensing drove most decisions here
-spaCy's maintainers state that Persian models trained back in 2018 were never published
+spaCy's maintainers say the Persian models trained in 2018 were never published because of
-*because of corpus licensing* (spaCy discussion #8233, after PR #2797 added `fa` tokenizer
+corpus licensing (spaCy discussion #8233, after PR #2797 added `fa` tokenizer support). ARMAN,
-support). The standard Persian NER corpora — ARMAN, PEYMA, NSURL — are all "research use
+PEYMA and NSURL are all research-use-only, and wrapping them in an Apache-2.0 toolkit does not
-only", and wrapping them in an Apache-2.0 toolkit does not launder that. ParsTwiNER (MIT) is
+change that.
-the only redistributable Persian NER corpus we could verify, which is why the NER component is
+
-trained on tweets. That trade-off is recorded in the model's `meta.json["notes"]`.
+The way out was finding that PerDT ships its own NER layer under the treebank's CC BY-SA 4.0,
 so the entire pipeline now derives from one corpus with one licence. The 2018 attempt also
 failed for a second reason worth knowing if you plan to publish: honnibal asked for scripts
 that could regenerate the model and got a notebook instead. `project.yml` is that script.
 ## Setup
@ -115,50 +150,71 @@ python -m venv .venv
 ## Build
-Everything is driven by [`project.yml`](project.yml):
+[`project.yml`](project.yml) has two workflows:
 ```bash
 .venv/bin/python -m spacy project assets      # download + checksum the corpora
-.venv/bin/python -m spacy project run all     # inspect -> convert -> train -> assemble -> evaluate -> package
+.venv/bin/python -m spacy project run all     # -> fa_dep_news_sm + fa_core_news_sm
 .venv/bin/python -m spacy project run ent     # -> fa_ent_news_sm, NER alone
 ```
 Individual steps:
 | Command | What it does |
 | --- | --- |
 | `inspect` | annotation coverage of the treebanks (`scripts/inspect_treebanks.py`) |
-| `convert-ud` | CoNLL-U → `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report |
+| `convert-ud` | CoNLL-U to `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report |
-| `convert-ner` | unpack ParsTwiNER, IOB2 → `DocBin` |
+| `transfer-ner` | align PerDT's NER layer onto that tokenization by difflib (`scripts/transfer_perdt_ner.py`) |
-| `debug-data` | `spacy debug data` on both corpora before spending CPU |
+| `convert-ner` | transferred IOB2 to `DocBin` |
-| `train-core` | tagger + morphologizer + trainable_lemmatizer + parser on PerDT |
+| `debug-data`, `debug-data-ner` | `spacy debug data` on both corpora before spending CPU |
-| `train-ner` | standalone `ner` with its own embedded tok2vec on ParsTwiNER |
+| `train-dep` | tagger + morphologizer + trainable_lemmatizer + parser |
-| `assemble` | source `ner` into the core pipeline, write full `meta.json` (`scripts/assemble_core.py`) |
+| `train-ner` | the `ner` component, with its own embedded tok2vec |
-| `evaluate` | `spacy benchmark accuracy` on both held-out test sets |
+| `finalize-dep` | write `fa_dep_news_sm` metadata: sources, licence, notes (`scripts/finalize_pipeline.py`) |
-| `package` | build the wheel + sdist |
+| `evaluate-dep` | `spacy benchmark accuracy` on the held-out UD test split |
-| `smoke` | run the pipeline over real Persian text and print every annotation layer |
+| `assemble-core` | source `ner` into the dep pipeline to produce `fa_core_news_sm` |
 | `evaluate-core` | score the assembled pipeline on both test splits |
 | `finalize-meta` | re-run finalize on both, folding test scores into `meta.json["performance"]` |
 | `package` | build wheels + sdists for both |
 | `smoke` | run both pipelines over Persian text and print every annotation layer |
-The two training runs are independent and can run concurrently — each is single-threaded.
+The two training runs are single-threaded and independent, so they can run concurrently.
-## Design decisions worth knowing before you touch anything
+`finalize` runs twice because of an ordering constraint: test scores only exist after
 evaluation, and evaluation needs a finalized pipeline to score. The second pass only copies
 models. `scripts/finalize_pipeline.py` enforces the shape of each variant, refusing to publish
 a `dep` pipeline that contains `ner` or a `core` one that does not, so the split cannot regress
 unnoticed.
-1. **`--merge-subtokens`.** spaCy has no multiword-token layer, and PerDT splits pronominal
+`--ud-metrics` and `--ner-metrics` are separate flags on purpose. Folding both reports over one
-   clitics (`پدرم` → `پدر` + `م`). Measured on dev: merging gives token F 0.9887 vs 0.9823 for
+key set silently corrupted `core`'s metadata during development: the NER corpus has no gold
-   the split version, at the cost of 34 composite XPOS tags on 1.5% of tokens. Merging wins
+tags, so its report carries `tag_acc: 0.0`, which overwrote the real 95.96.
-   because otherwise 1.5% of gold tokens are boundaries the shipped tokenizer can never
+
-   produce. Numbers: `scripts/tokenization_report.py`.
+## Design decisions
-2. **`ner` carries its own tok2vec.** A `Tok2VecListener` only resolves inside the pipeline it
+
-   was trained in, so a listener-based component cannot be sourced into another pipeline.
+1. `--merge-subtokens`. spaCy has no multiword-token layer, and PerDT splits pronominal clitics
-   `configs/fa_ner_sm.cfg` embeds the tok2vec instead — the same design as `en_core_web_sm`.
+   (`پدرم` into `پدر` + `م`). Measured on dev, merging gives token F 0.9887 against 0.9823 for
-3. **`morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer.**
+   the split version, costing 34 composite XPOS tags on 1.5% of tokens. Without it, 1.5% of gold
-   The English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD
+   tokens are boundaries the shipped tokenizer can never produce. See
-   gives us gold UPOS, FEATS and lemmas, so we train on them and get real `pos_acc`,
+   `scripts/tokenization_report.py`.
-   `morph_acc` and `lemma_acc` numbers instead of unmeasurable rule coverage.
+2. `ner` carries its own tok2vec. A `Tok2VecListener` only resolves inside the pipeline it was
-4. **PerDT, not Seraji.** 3.7× more tokens, and Seraji has no `PROPN` tag at all.
+   trained in, so a listener-based component cannot be sourced elsewhere.
   `configs/fa_ner_sm.cfg` embeds the tok2vec instead, as `en_core_web_sm` does.
 3. `morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer. The
   English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD gives
   gold UPOS, FEATS and lemmas, which yields real `pos_acc`, `morph_acc` and `lemma_acc` numbers
   instead of unmeasurable rule coverage.
 4. PerDT, not Seraji: 3.7x more tokens, and Seraji has no `PROPN` tag.
 ## Roadmap
-`md`/`lg` need floret vectors trained on Persian Wikipedia + OSCAR (see
+1. A human-annotated NER test set, ~500 sentences. PerDT's entity labels and its NER test split
-`spacy-vectors-builder`); floret rather than classic fastText because Persian's ZWNJ usage is
+   are both silver, so `ENTS_F 71.87` is not yet a fact. Tracked in
-inconsistent and explodes the surface vocabulary. `trf` needs a rented GPU and should use
+   [`../ner_dataset`](../ner_dataset/PLAN.md).
-`HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) rather than ParsBERT, whose model card
+2. A mixed-genre variant. Measured: this prose-trained NER scores 45.72 F on tweets, and mixing
-carries no licence. `senter` is one extra training run away. Details in `docs/MODELS.md` §2.
+   ParsTwiNER in recovers that to 66.49 for 0.69 F on prose. That belongs in a separate package
   rather than inside a `news` one.
 3. `md` and `lg` need floret vectors trained on Persian Wikipedia and OSCAR (see
   `spacy-vectors-builder`). Floret rather than classic fastText, because inconsistent ZWNJ
   usage explodes the surface vocabulary.
 4. `trf` needs a rented GPU and should use `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0)
   rather than ParsBERT, whose model card carries no licence.
 5. `senter` is one extra training run.
 6. Upstream PRs to `spacy/lang/fa`, see [`docs/upstream/fa-noun-chunks.md`](docs/upstream/fa-noun-chunks.md).
--- a/configs/fa_core_news_sm.cfg
+++ b/configs/fa_core_news_sm.cfg
@ -1,13 +1,14 @@
-# fa_core_news_sm — UD components (tagger, morphologizer, trainable_lemmatizer, parser).
+# fa_dep_news_sm — tagger, morphologizer, trainable_lemmatizer, parser.
 #
 # Generated with:
-#   spacy init config configs/fa_core_news_sm.cfg --lang fa \
+#   spacy init config configs/fa_dep_news_sm.cfg --lang fa \
 #     --pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency
 #
 # Deviations from the generated defaults:
 #   - eval_frequency 200 -> 400 (dev set is only 146 docs; 200 wastes CPU on a 4-core box)
-# The `ner` component is trained separately (configs/fa_ner_sm.cfg) from a different
+#
-# corpus and merged in by scripts/assemble_core.py.
+# No `ner` here by design. NER is a separate artifact built from a different corpus by the
 # `ner` workflow (configs/fa_ner_sm.cfg -> fa_ent_news_sm); see project.yml for why.
 [paths]
 train = null
--- a/configs/fa_ner_sm.cfg
+++ b/configs/fa_ner_sm.cfg
@ -1,15 +1,19 @@
-# fa NER component, CPU size (sm).
+# fa_ent_news_sm — Persian NER, CPU size (sm).
 #
-# Trained on ParsTwiNER (MIT) — a different corpus from the UD treebank that trains
+# A standalone artifact, not part of the shipping `dep` pipeline. It is trained on
-# the rest of fa_core_news_sm, so this is a standalone run whose `ner` component is
+# ParsTwiNER (MIT, tweets) while the rest of the project trains on UD_Persian-PerDT
-# later sourced into the core pipeline by scripts/assemble_core.py.
+# (CC BY-SA 4.0, edited prose), and it scores 67.22 F against 85-98 for the UD
 # components — different corpus, different genre, different quality tier, so it gets its
 # own package rather than being hidden inside a `core` one. Built by `spacy project run ner`.
 #
 # Deliberate deviation from `spacy init config --pipeline ner`: the tok2vec is
 # EMBEDDED inside components.ner.model instead of being a separate `tok2vec`
 # component with a Tok2VecListener. A listener can only resolve inside the pipeline
 # it was trained in; embedding makes the component self-contained and therefore
-# sourceable. This is the same design as en_core_web_sm, whose `ner` "has its own
+# sourceable into another pipeline via `nlp.add_pipe("ner", source=...)`. This is the same
-# independent internal tok2vec" (https://spacy.io/models#design).
+# design as en_core_web_sm, whose `ner` "has its own independent internal tok2vec"
 # (https://spacy.io/models#design). That is what will let a future fa_core_news_sm combine
 # this component (or its ../ner_dataset replacement) with the dep pipeline.
 [paths]
 train = null
--- a/docs/CONTRIBUTING-GUIDE.md
+++ b/docs/CONTRIBUTING-GUIDE.md
@ -1,89 +1,85 @@
-# Contribution guidelines: getting a Persian pipeline into the spaCy ecosystem
+# Getting a Persian pipeline into the spaCy ecosystem
-Everything below is sourced from spaCy's own primary docs/repos (URLs inline). Read this
+Sourced from spaCy's primary docs and repos, URLs inline. Where a contribution goes decides
-before writing code, because **where** a contribution goes decides how it must be shaped.
+how it has to be shaped, so read this before writing code.
-## 0. The one thing to internalise
+## 0. Two separate contribution surfaces
 spaCy splits Persian support into two *completely different* contribution surfaces:
 | Surface | What it is | Where it lives | How you contribute |
 | --- | --- | --- | --- |
-| **Language data** (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` |
+| Language data (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` |
-| **Trained pipeline** (`fa_core_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | **You cannot.** Publish it yourself (PyPI / HF Hub) and get it listed in spaCy Universe |
+| Trained pipeline (`fa_dep_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | You cannot. Publish it yourself (PyPI or HF Hub) and get it listed in spaCy Universe |
-`spacy/lang/fa` **already exists upstream**. What does not exist is any trained `fa` pipeline.
+`spacy/lang/fa` already exists upstream. No trained `fa` pipeline does. So this is a publishing
-So this project is a *publishing* project, not an upstream-PR project — with optional
+project, with optional upstream PRs for the language-data gaps found along the way.
 upstream PRs for the language-data gaps we find along the way.
 ## 1. Policy, verbatim
 From <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md>:
- **No CLA.** There is no contributor licence agreement section; contribution is an ordinary
+- No CLA. There is no contributor licence agreement section; contribution is an ordinary
  GitHub PR governed by the Contributor Covenant Code of Conduct v1.4.
- Inclusion philosophy — this is the sentence that decides everything:
+- Inclusion philosophy:
  > "Our philosophy is to prefer a smaller core library. […] If you're looking to implement a
  > new spaCy feature, starting with a custom component package is usually the best strategy.
  > […] And if it works well, we can always integrate it into the core library later."
- The only sanctioned route for non-trivial additions is the *Publishing spaCy extensions and
+- The sanctioned route for non-trivial additions is the "Publishing spaCy extensions and
-  plugins* section:
+  plugins" section:
  > "An extension or plugin should add substantial functionality, be well-documented and
  > open-source. It should be available for users to download and install as a Python package
  > – for example via PyPi."
  > "Once your extension is published, you can open a PR to suggest it for the Universe page."
- `CONTRIBUTING.md` contains **zero** mention of submitting trained models. Neither does
+- `CONTRIBUTING.md` never mentions submitting trained models. Neither does
-  <https://github.com/explosion/spacy-models> — its README only documents `compatibility.json`
+  <https://github.com/explosion/spacy-models>, whose README only documents `compatibility.json`
  as "the source of spaCy's internal compatibility check, performed when you run the download
  command". There is no PR template, issue label, or documented process for a third party to
-  land a pipeline in `spacy download`.
+  land a pipeline in `spacy download`. So the `spacy download` index is an Explosion-only
-  **Conclusion: the `spacy download` index is an Explosion-only release channel.**
+  release channel. (`[INFERENCE]` from the absence of any process across all three sources.)
-  (`[INFERENCE]` from the absence of any process across all three primary sources.)
+- `https://spacy.io/usage/adding-languages` no longer exists and redirects to
- `https://spacy.io/usage/adding-languages` no longer exists; it redirects to
+  <https://spacy.io/usage/linguistic-features#language-data>. The `BaseDefaults` and `Language`
-  <https://spacy.io/usage/linguistic-features#language-data>. The `BaseDefaults` /
+  contract is at <https://spacy.io/api/language>.
  `Language` contract is documented at <https://spacy.io/api/language>.
-### Code rules that apply to a `spacy/lang/fa` PR
+### Code rules for a `spacy/lang/fa` PR
 - `black` formatting, `flake8` clean, type hints where practical.
 - Tests go in `spacy/tests/lang/fa/`; regression tests use `@pytest.mark.issue(N)`.
 - Touching `.pyx` means the reviewer must rebuild: `python setup.py build_ext --inplace`.
- Language data must be *pure data + rules*: no model downloads, no network, no heavy deps.
+- Language data must be data and rules only: no model downloads, no network, no heavy deps.
 ## 2. Three publishing routes for the trained pipeline
-1. **Hugging Face Hub** — Explosion's own tool, the path of least resistance
+1. Hugging Face Hub, using Explosion's own tool
   (<https://github.com/explosion/spacy-huggingface-hub>):
   ```bash
   pip install spacy-huggingface-hub
   huggingface-cli login
-   python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build wheel
+   python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build wheel
-   python -m spacy huggingface-hub push packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl --org <org>
+   python -m spacy huggingface-hub push packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl --org <org>
   ```
-   Users then `pip install https://huggingface.co/<org>/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl`.
+   Users then `pip install https://huggingface.co/<org>/fa_dep_news_sm/resolve/main/fa_dep_news_sm-any-py3-none-any.whl`.
-2. **PyPI / self-hosted wheel** — `spacy package … --build sdist,wheel` then `twine upload`,
+2. PyPI or a self-hosted wheel: `spacy package … --build sdist,wheel` then `twine upload`, or
-   or attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
+   attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
-3. **spaCy Universe listing** — lists the package on spacy.io but hosts nothing. Per
+3. spaCy Universe, which lists the package on spacy.io but hosts nothing. Per
   <https://github.com/explosion/spaCy/blob/master/website/UNIVERSE.md>: fork `explosion/spaCy`,
-   append an entry to `website/meta/universe.json`, open a PR. Required fields:
+   append an entry to `website/meta/universe.json`, open a PR. Required fields: `id`, `title`,
-   `id`, `title`, `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`,
+   `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`, `url`, `thumb`,
-   `url`, `thumb`, `image`, `author`, `author_links`, `category`, `tags`.
+   `image`, `author`, `author_links`, `category`, `tags`. The package must be open-source with
-   Checklist: must be **open-source with a user-friendly license** and "at least somewhat documented".
+   a user-friendly license and "at least somewhat documented".
-**Route we take:** HF Hub for the artifact + a Universe PR once metrics are respectable.
+Route taken here: HF Hub for the artifact, plus a Universe PR once metrics are respectable.
-## 3. Naming and versioning conventions (non-negotiable if you want to look official)
+## 3. Naming and versioning
 From <https://spacy.io/models#conventions> and the `spacy-models` README:
 ```
-[lang]_[type]_[genre]_[size]        e.g. fa_core_news_sm
+[lang]_[type]_[genre]_[size]        e.g. fa_dep_news_sm
 ```
 | Slot | Allowed values | Meaning |
 | --- | --- | --- |
 | `type` | `core` | tagger + parser + lemmatizer + NER |
-| | `dep` | tagger + parser + lemmatizer, **no NER** |
+| | `dep` | tagger + parser + lemmatizer, no NER |
 | | `ent` | NER only |
 | | `sent` | sentence segmentation only |
 | `genre` | `news`, `web`, `wiki` | text domain of the training corpus |
@ -92,27 +88,27 @@ From <https://spacy.io/models#conventions> and the `spacy-models` README:
 | | `lg` | ~500k vectors (or 200k floret rows) |
 | | `trf` | transformer, no static vectors |
-**Version numbers are `a.b.c` = spaCy-major . spaCy-minor . model-revision.** Trained against
+Version numbers are `a.b.c` = spaCy-major, spaCy-minor, model-revision. Trained against spaCy
-spaCy 3.8 → the package version starts at `3.8.0`. Retraining on new data bumps `c`.
+3.8, so the package version starts at `3.8.0`; retraining on new data bumps `c`. Avoid
-Do **not** invent `0.1.0`-style versions; that is exactly how hazm's HF pipelines ended up with
+`0.1.0`-style versions, which is how hazm's HF pipelines ended up with `version: 0.0.0` and
-`version: 0.0.0` and `spacy_version: >=3.6.0,<3.7.0`, which is unusable metadata.
+`spacy_version: >=3.6.0,<3.7.0`.
 ### `meta.json` fields that matter (<https://spacy.io/api/data-formats#meta>)
 `lang`, `name`, `version`, `spacy_version` (range), `description`, `author`, `email`, `url`,
-`license`, **`sources`** (list of `{name, url, author, license}` — this is where treebank
+`license`, `sources` (list of `{name, url, author, license}`, which is where treebank
-provenance and its CC BY-SA obligation must be recorded), `requirements` (extra pip deps
+provenance and its CC BY-SA obligation is recorded), `requirements` (extra pip deps injected
-injected into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance`
+into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance` (auto-filled by
-(auto-filled by `spacy train`), `speed`, `spacy_git_version`.
+`spacy train`), `speed`, `spacy_git_version`.
-Note: in v3, `meta.json` "isn't used to construct the language class and pipeline anymore" —
+In v3, `meta.json` "isn't used to construct the language class and pipeline anymore".
-`config.cfg` is the single source of truth for loading. `meta.json` is metadata + packaging.
+`config.cfg` is the single source of truth for loading; `meta.json` is metadata and packaging.
-## 4. Canonical training workflow (what Explosion actually runs)
+## 4. Canonical training workflow
 Template: <https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud>
-(`project.yml` + `configs/default.cfg`). Directory layout `assets / corpus / configs /
+(`project.yml` + `configs/default.cfg`). The directory layout `assets / corpus / configs /
-training / metrics / packages` is the convention — this repo follows it.
+training / metrics / packages` is the convention, which this repo follows.
 ```bash
 # 1. assets: clone the UD treebank
@ -136,54 +132,55 @@ python -m spacy train configs/default.cfg --output training/ --gpu-id -1 \
 # 6. evaluate on held-out test
 python -m spacy benchmark accuracy training/model-best corpus/test.spacy --output metrics/test.json
-#   ('spacy evaluate' is now just an alias for 'benchmark accuracy')
+#   ('spacy evaluate' is now an alias for 'benchmark accuracy')
 # 7. package
-python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build sdist,wheel
+python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build sdist,wheel
 ```
-`spacy assemble` builds a pipeline from a config **without training** — useful for a
+`spacy assemble` builds a pipeline from a config without training, which suits a rule-only
-rule-only artifact (tokenizer + lookup lemmatizer + stop words).
+artifact (tokenizer + lookup lemmatizer + stop words).
-### Flags worth knowing on `spacy convert`
+### Flags that matter on `spacy convert`
- `--converter conllu` — explicit is better than `auto`.
+- `--converter conllu`, explicit rather than `auto`.
- `--n-sents N` — how many sentences per `Doc`; 10 is the Explosion default and gives the
+- `--n-sents N`, how many sentences per `Doc`. 10 is the Explosion default and gives the parser
-  parser/senter cross-sentence context.
+  and senter cross-sentence context.
- `--merge-subtokens` — spaCy has **no multi-word-token layer**. Persian UD treebanks use MWT
+- `--merge-subtokens`. spaCy has no multi-word-token layer, and Persian UD treebanks use MWT
-  ranges for clitics (`پدرم` → `پدر` + `م`), so this flag decides whether gold tokens are
+  ranges for clitics (`پدرم` = `پدر` + `م`), so this flag decides whether gold tokens are
-  clitic-split or fused. See `docs/MODELS.md` for the tokenisation decision.
+  clitic-split or fused. See `docs/MODELS.md` §5.
- `--morphology` — appends morph features to the tag (only if you *aren't* using a morphologizer).
+- `--morphology`, which appends morph features to the tag. Only for pipelines without a
  morphologizer.
-### How the four size tiers differ mechanically
+### How the size tiers differ mechanically
 | Tier | Embedding source | Config difference |
 | --- | --- | --- |
-| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, everything else a `spacy.Tok2VecListener` on the shared `tok2vec` |
+| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, every other component a `spacy.Tok2VecListener` on the shared `tok2vec` |
-| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = <dir built by init vectors>`; `lg` just has a bigger table |
+| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = <dir built by init vectors>`; `lg` has a bigger table |
-| `trf` | contextual | replace `tok2vec` with a `transformer` component; every other component listens via `TransformerListener` instead of `Tok2VecListener` |
+| `trf` | contextual | replace `tok2vec` with a `transformer` component; other components listen via `TransformerListener` instead of `Tok2VecListener` |
-`spacy pretrain` (Tok2Vec LM-style pretraining on raw text → `[initialize] init_tok2vec`) is
+`spacy pretrain` (Tok2Vec LM-style pretraining on raw text into `[initialize] init_tok2vec`) is
 orthogonal to the tier and optional.
-## 5. Upstream `spacy/lang/fa` — current state and gaps
+## 5. Upstream `spacy/lang/fa`: current state and gaps
 Contents of <https://github.com/explosion/spaCy/tree/master/spacy/lang/fa>:
-| File | Size | What it gives us |
+| File | Size | What it provides |
 | --- | --- | --- |
-| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a **`lemmatizer` factory defaulting to `mode="rule"`** |
+| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a `lemmatizer` factory defaulting to `mode="rule"` |
-| `tokenizer_exceptions.py` | 64.9 KB | Large generated compound-verb / enclitic exception table |
+| `tokenizer_exceptions.py` | 64.9 KB | Generated compound-verb and enclitic exception table |
 | `generate_verbs_exc.py` | 14.8 KB | Dev script that generates the above |
-| `stop_words.py` | 3.8 KB | ~500 stop words, comment says **"Stop words from HAZM package"** |
+| `stop_words.py` | 3.8 KB | ~500 stop words; the comment says "Stop words from HAZM package" |
-| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals + `ام`/`ین` suffixes) |
+| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals plus `ام` and `ین` suffixes) |
-| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only — **no prefixes, no infixes** |
+| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only, no prefixes and no infixes |
-| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, needs a trained parser to do anything |
+| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, which needs a trained parser |
-Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer, or
+Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer or
 stop-word tests.
-`spacy-lookups-data` (MIT) **already ships Persian rule-lemmatizer tables**:
+`spacy-lookups-data` (MIT) already ships Persian rule-lemmatizer tables:
 | File | Size |
 | --- | --- |
@ -192,48 +189,49 @@ stop-word tests.
 | `fa_lemma_rules.json` | 882 B |
 | `fa_source.txt` | "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" |
-Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`,
+Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`, and
 `fa_license.txt` (Catalan has one; Persian's Seraji-derived provenance is undocumented).
-**Gap list for a real `fa_core_news_*`:**
+Gaps blocking a full `fa_core_news_*`:
-1. No trained artefacts at all — no weights, no `config.cfg`, no `meta.json`, no entry in
+1. No trained artefacts: no weights, no `config.cfg`, no `meta.json`, no entry in
   `compatibility.json`. `spacy.load("fa_core_news_sm")` fails today.
-2. No word vectors for `fa` → `md`/`lg` need vectors built from scratch.
+2. No word vectors for `fa`, so `md` and `lg` need vectors built from scratch.
-3. No NER data or labels anywhere in `spacy/lang/fa` → `core` (which implies NER) needs an
+3. No NER data or labels anywhere in `spacy/lang/fa`, so `core` needs an external corpus.
-   external annotated corpus.
+4. The rule lemmatizer needs `token.pos` from a tagger or morphologizer, which is
-4. Rule lemmatizer needs `token.pos` from a tagger/morphologizer → chicken-and-egg until the
+   chicken-and-egg until the tagger exists. `trainable_lemmatizer` avoids this by learning edit
-   tagger exists (or use a `trainable_lemmatizer`, which learns edit trees and needs no tables).
+   trees and needing no tables.
-5. No Persian-specific prefix/infix rules — ZWNJ (U+200C) is only handled implicitly through
+5. No Persian-specific prefix or infix rules. ZWNJ (U+200C) is handled only implicitly through
   the verb-exception table.
-6. Zero tokenizer test coverage for a 65 KB exception table.
+6. No tokenizer test coverage for a 65 KB exception table.
-Items 5–7 are legitimate upstream PR material; items 1–4 are this project's job.
+Items 5 and 6 are upstream PR material, along with the `noun_chunks` bug in
 `docs/upstream/fa-noun-chunks.md`. Items 1 to 4 are this project's job.
-## 6. Prior art we must not duplicate badly
+## 6. Prior art
-hazm publishes **spaCy-format** Persian pipelines on the HF Hub (MIT-ish, but `license` field
+hazm publishes spaCy-format Persian pipelines on the HF Hub, each a single-task pipeline built
-left empty), each a single-task pipeline built on ParsBERT:
+on ParsBERT, with the `license` field left empty:
 | HF repo | Pipeline | Reported | Trained on (`[paths]` in its `config.cfg`) |
 | --- | --- | --- | --- |
 | `roshan-research/hazm-parsbert-postagger` | `transformer, tagger` | `tag_acc` 0.9862 | `data_train_98_rs10.spacy` (hazm's own EZ-augmented tagset: `NOUN,EZ`, `ADJ,EZ`, …) |
-| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy` — **UD_Persian-PerDT** |
+| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy`, i.e. UD_Persian-PerDT |
 | `roshan-research/hazm-parsbert-chunker` | `transformer, tagger` (IOB chunk tags) | `tag_acc` 0.9618 | hazm chunk data |
-Both use `spacy-transformers` `TransformerModel.v3` on
+All three use `spacy-transformers` `TransformerModel.v3` on
-`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 / stride 96,
+`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 and stride 96, with
 `spacy_version >= 3.6.0,<3.7.0`.
-What they get wrong, and what we fix:
+Problems this project avoids:
- Three separate pipelines instead of one `core` pipeline → users pay for three BERT forward
+- Three separate pipelines rather than one, so users pay for three BERT forward passes and
-  passes and cannot share a `Doc`.
+  cannot share a `Doc`.
- `version: 0.0.0`, empty `license`/`author`/`sources`, `name: "pipeline"` → violates every
+- `version: 0.0.0`, empty `license`, `author` and `sources`, and `name: "pipeline"`, which
-  convention in §3 and makes redistribution legally murky.
+  breaks the conventions in §3 and leaves redistribution rights unclear.
- Transformer-only, so no CPU-friendly tier at all.
+- Transformer-only, with no CPU-friendly tier.
- No lemmatizer, no morphologizer, no NER, no vectors.
+- No lemmatizer, morphologizer, NER or vectors.
- Pinned to spaCy 3.6; unusable on 3.8 without retraining.
+- Pinned to spaCy 3.6, so unusable on 3.8 without retraining.
-The valuable, reusable finding: **hazm's own parser recipe is "UD_Persian-PerDT + spaCy
+The reusable finding is hazm's parser recipe, UD_Persian-PerDT plus spaCy's
-`TransitionBasedParser`"**, which independently confirms the corpus choice in `docs/MODELS.md`.
+`TransitionBasedParser`, which independently confirms the corpus choice in `docs/MODELS.md`.
--- a/docs/MODELS.md
+++ b/docs/MODELS.md
@ -1,70 +1,87 @@
 # The Persian pipelines: what to build, from what, and why
-## 1. What "the 4 English pipelines" actually are
+## 1. What the four English pipelines are
-They are **one pipeline design at four embedding budgets**, not four different products.
+One pipeline design at four embedding budgets. Every one has the same components; the axis of
-Every one of them has the same components; the only real axis of variation is where token
+variation is where token representations come from.
 representations come from.
 | | `en_core_web_sm` | `en_core_web_md` | `en_core_web_lg` | `en_core_web_trf` |
 | --- | --- | --- | --- | --- |
 | Size on disk | 12 MB | 31 MB | 382 MB | 436 MB |
 | Embeddings | hash embeddings only | 685k keys / 20k vectors (300d) | 685k keys / 343k vectors (300d) | `roberta-base`, 768d contextual |
-| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | **transformer**, tagger, parser, attribute_ruler, lemmatizer, ner |
+| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | transformer, tagger, parser, attribute_ruler, lemmatizer, ner |
 | `TAG_ACC` | 0.97 | 0.97 | 0.97 | 0.98 |
 | `DEP_UAS` / `LAS` | 0.92 / 0.90 | 0.92 / 0.90 | 0.92 / 0.90 | 0.95 / 0.94 |
 | `ENTS_F` | 0.84 | 0.85 | 0.86 | 0.90 |
 | Training data | OntoNotes 5 (+ ClearNLP dep conversion, WordNet 3.0) | + Explosion vectors (OSCAR 2109 + Wikipedia + OpenSubtitles + WMT News Crawl) | same | OntoNotes 5 + roberta-base |
-Read the accuracy table honestly: **static vectors buy almost nothing for tagging and
+Static vectors buy nothing measurable for tagging and parsing (identical to two decimals) and
-parsing** (identical to two decimals) and ~1–2 F on NER. The transformer buys ~4 LAS and
+1 to 2 F on NER. The transformer buys about 4 LAS and 6 NER F, at 36x the size and a GPU
-~6 NER F, at 36× the size and a GPU requirement. That ordering dictates the roadmap below.
+requirement. That ordering sets the roadmap below.
-Sources: <https://spacy.io/models/en>.
+Source: <https://spacy.io/models/en>.
-## 2. Target: the four Persian pipelines
+## 2. Target: the Persian pipelines
-Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>).
+Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>). The
-`core` = tagger + parser + lemmatizer + NER. Genre is `news`, after the dominant genre of
+`type` slot carries real information: `dep` = tagger + parser + lemmatizer, `ent` = NER only,
-UD_Persian-PerDT (its README lists "news fiction nonfiction academic web blog", and spaCy
+`core` = both. Genre is `news`, after the dominant genre of UD_Persian-PerDT, whose README
-labels comparable treebank-trained pipelines such as `de_core_news_sm` as `news`).
+lists "news fiction nonfiction academic web blog". spaCy labels comparable treebank-trained
 pipelines such as `de_core_news_sm` as `news`.
-| Pipeline | Components | Embeddings | Trainable on this hardware? |
+| Pipeline | Components | Embeddings | Status |
 | --- | --- | --- | --- |
-| **`fa_core_news_sm`** | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner | hash embeddings | **yes — this is what we train now** |
+| `fa_dep_news_sm` | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | hash embeddings | built, shipping |
-| `fa_core_news_md` | same | floret vectors, 50k rows | yes, but vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) |
+| `fa_core_news_sm` | the above plus ner | hash embeddings | built, shipping |
-| `fa_core_news_lg` | same | floret vectors, 200k rows | same as md, bigger table |
+| `fa_ent_news_sm` | ner (own internal tok2vec) | hash embeddings | built, optional |
-| `fa_core_news_trf` | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | **no** — 2 GB VRAM (GTX 940MX) cannot fine-tune a 125M-param encoder; needs rented GPU |
+| `fa_core_web_sm` | same as core, mixed-genre training data | hash embeddings | not built; would add ParsTwiNER to cover social media |
 | `fa_core_news_md` | + static vectors | floret, 50k rows | vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) |
 | `fa_core_news_lg` | same | floret, 200k rows | same as md, bigger table |
 | `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | not on this hardware; 2 GB VRAM cannot fine-tune a 125M-param encoder |
-One deviation from the English design, deliberate: Persian gets a **`morphologizer`**
+### Why `core` is honest here
 (UPOS + morphological features) and a **`trainable_lemmatizer`** instead of English's
 `attribute_ruler` + rule `lemmatizer`. Reasons:
- The English pipelines use `attribute_ruler` because OntoNotes gives them PTB `tag`s and
+`core` promises that the NER belongs to the same pipeline, built to the same standard and
-  they *derive* UPOS from tags by rule. UD treebanks give UPOS and FEATS as gold data —
+versioned together. That holds because PerDT ships its own entity layer in
-  training a morphologizer on them is strictly more information, and Persian morphology
+`not-to-release/Dadegan with NER tag/`: 15,833 entities over the same 29,107 sentences, under
-  (Number, Person, Tense, Mood, Voice, Polarity, PronType) is worth predicting.
+the same CC BY-SA 4.0, in the same genre, aligned to the same tokenization.
 - Persian rule-lemmatizer tables *do* exist in `spacy-lookups-data` (`fa_lemma_exc.json`
  1.68 MB, `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but
  a rule lemmatizer's accuracy is unmeasurable against the corpus it was extracted from and it
  needs `token.pos` to work at all. `trainable_lemmatizer` learns edit trees from PerDT's gold
  lemmas and reports a real `lemma_acc`. PerDT yields **1,908 edit trees** with 100% lemma
  coverage — plenty.
-`senter` is intentionally omitted from v1: it is a separately trained component that ships
+An earlier revision of this project refused to ship `core`, and was right to given what it knew
-*disabled by default* in the English pipelines, and the parser already produces sentence
+then. The only redistributable Persian NER corpus found at that point was ParsTwiNER, a Twitter
-boundaries. Adding it later requires only one extra training run.
+corpus scoring 67.22 F against 85-98 for the UD components, and folding that into one package
 would have hidden a genre and quality gap behind a single name and version number. Finding the
 treebank's own layer removed the objection rather than answering it.
-## 3. Resource inventory (everything checked for license)
+`fa_dep_news_sm` still ships alongside `core`, for users who want a 7.5 MB syntax-only model or
 who would rather not depend on silver entity labels.
-### 3.1 Treebanks — the tagger / morphologizer / lemmatizer / parser data
+### Why `morphologizer` + `trainable_lemmatizer`
-| Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN? | Verdict |
+The English pipelines use `attribute_ruler` because OntoNotes gives them PTB tags and they
 derive UPOS by rule. UD treebanks give UPOS and FEATS as gold data, so a morphologizer trained
 on them has strictly more information, and Persian morphology (Number, Person, Tense, Mood,
 Voice, Polarity, PronType) is worth predicting.
 Persian rule-lemmatizer tables do exist in `spacy-lookups-data` (`fa_lemma_exc.json` 1.68 MB,
 `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but a rule
 lemmatizer's accuracy cannot be measured against the corpus it was extracted from, and it needs
 `token.pos` to run at all. `trainable_lemmatizer` learns edit trees from PerDT's gold lemmas and
 reports a real `lemma_acc`. PerDT yields 1,908 edit trees at 100% lemma coverage.
 `senter` is omitted from v1. It is a separately trained component that ships disabled by default
 in the English pipelines, and the parser already produces sentence boundaries. Adding it later
 takes one training run.
 ## 3. Resource inventory, with licences
 ### 3.1 Treebanks, for tagger / morphologizer / lemmatizer / parser
 | Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN | Verdict |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| **UD_Persian-PerDT** (PerUDT v1.0) | 29,107 | ~509k | **CC BY-SA 4.0** | converted + corrections | converted + corrections | manual native (34 types) | **yes** | **CHOSEN** |
+| UD_Persian-PerDT (PerUDT v1.0) | 29,107 | ~509k | CC BY-SA 4.0 | converted + corrections | converted + corrections | manual native (34 types) | yes | chosen |
-| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | **no** | secondary / cross-eval |
+| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | no | secondary, cross-eval |
-| UD_Persian-PUD | 1,000 | — | CC BY-SA 4.0 | — | — | none | — | test-only, parallel corpus |
+| UD_Persian-PUD | 1,000 | | CC BY-SA 4.0 | | | none | | test-only, parallel corpus |
-| UD_Persian-IPerUDT | tiny | — | CC BY-SA 4.0 | — | — | none | — | grammar examples, unusable |
+| UD_Persian-IPerUDT | tiny | | CC BY-SA 4.0 | | | none | | grammar examples, unusable |
 Measured locally with `scripts/inspect_treebanks.py` (train splits):
@ -74,146 +91,188 @@ fa_perdt-ud-train.conllu         26196    452496   6508      0   100.0    57.7
 fa_seraji-ud-train.conllu         4798    121067   1117      0   100.0    65.0    15    30   39
 ```
-Why PerDT over Seraji:
+PerDT over Seraji for three reasons. It has 3.7x more training tokens (452k against 121k), and
 at `sm` size data is the binding constraint. Seraji has no `PROPN`: proper nouns are tagged
 `NOUN`, which breaks the downstream tasks people use spaCy for. And hazm independently chose
 PerDT for its own spaCy dependency parser, whose `config.cfg` names
 `modified_fa_perdt-ud-train.spacy`, which makes the numbers comparable.
-1. **3.7× more training tokens** (452k vs 121k). At `sm` size, data is the binding constraint.
+Seraji's richer manual FEATS and fully manual lemmas make it the natural cross-evaluation set
-2. **Seraji has no `PROPN`** — proper nouns are tagged `NOUN`. A pipeline that cannot mark
+and a candidate for a future concatenated-corpus run. The two use different XPOS inventories,
-   proper nouns is crippled for exactly the downstream tasks people use spaCy for.
+so naive concatenation would corrupt the `tag` label space.
 3. hazm independently chose PerDT for its own spaCy dependency parser — its
  `config.cfg` names `modified_fa_perdt-ud-train.spacy` as the training set. Converging on
  the same corpus makes our numbers comparable to theirs.
-Seraji's advantages (richer manual FEATS, fully manual lemmas) are real; it is the natural
+### 3.2 NER
 cross-evaluation set and a candidate for a future concatenated-corpus run. The two use
 different XPOS inventories, so naive concatenation would corrupt the `tag` label space.
-### 3.2 NER — the one place with a licensing minefield
+| Dataset | Labels | Size | License | Usable |
 | Dataset | Labels | Size | License | Usable? |
 | --- | --- | --- | --- | --- |
-| **ParsTwiNER** | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens | **MIT** (verified via GitHub API on `overfit-ir/parstwiner`) | **CHOSEN** |
+| PerDT's own NER layer | PER, LOC, ORG, DAT, MON, TIM, PCT | 29,107 sentences / 484k tokens / 15,833 entities | CC BY-SA 4.0, same as the treebank | chosen |
 | ParsTwiNER | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens / 16,250 entities | MIT (verified via GitHub API on `overfit-ir/parstwiner`) | usable, wrong genre |
 | ARMAN (PersianNER) | 6 classes | 250k tokens | academic research only | no |
 | PEYMA | 7 classes | 302k tokens | "free for research purposes", no OSS licence | no |
 | NSURL-2019 Task 7 | PEYMA tagset | ~1M tokens | no explicit licence | no |
 | HooshvareLab merged ParsNER | 10 classes | ARMAN+PEYMA+WikiANN | inherits ARMAN/PEYMA restrictions; HF repo gated (401) | no |
-This is not pedantry. **spaCy's own maintainers state that Persian models trained back in
+spaCy's maintainers state that the Persian models trained in 2018 were never published because
-2018 were never published precisely because of corpus licensing** (spaCy discussion #8233,
+of corpus licensing (spaCy discussion #8233, following PR #2797, which added only
-following PR #2797 which added only `spacy.blank("fa")` tokenizer support). Repeating that
+`spacy.blank("fa")` tokenizer support).
 mistake would waste the whole exercise: a pipeline you cannot legally redistribute is not a
 pipeline, it is a local file.
-Consequence to state plainly in the model card: **the NER component is trained on Twitter
+The PerDT layer lives in `not-to-release/Dadegan with NER tag/{train,dev,test}_with_NER_tag.txt`
-text while the rest of the pipeline is trained on edited prose.** Expect NER to degrade on
+as two-column IOB2. In UD convention `not-to-release/` means "excluded from the official UD
-formal news text relative to what an ARMAN/PEYMA-trained model would score. That is the
+release build", normally working and source data, and the directory is public on GitHub under
-price of a redistributable artifact.
+the repo's `LICENSE.txt`. Confirm that reading with the PerDT authors before publishing anything
 derived from it, since redistribution rights are the whole point.
-### 3.3 Vectors (for md / lg)
+Three properties of that layer decide how it gets used:
 1. **The labels are silver.** The treebank README states they came from the BERT-based
   Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. The
   published `ents_f` is therefore measured against a silver test split and partly reflects
   agreement with that tagger. A human-annotated test set is the outstanding work.
 2. **Tokenization differs.** The NER files use the original Dadegan tokenization, which matches
   the released UD tokenization exactly in only 57 to 62% of sentences: the NER files drop some
   copulas and auxiliaries, and at least one honorific is corrupted (`ص` written as `،`).
   Entities sit on content words present in both, so `scripts/transfer_perdt_ner.py` aligns them
   with difflib, transferring 99.86% of train entities, 99.74% of dev and 99.51% of test. Spans
   whose tokens do not all map contiguously are dropped rather than guessed.
 3. **The label sets do not line up with ParsTwiNER.** PerDT has `DAT`, `MON`, `TIM` and `PCT`;
   ParsTwiNER has `NAT`, `EVE` and `POG`. The intersection is `PER`, `LOC`, `ORG`. Since spaCy
   assigns one label per token, concatenating the two raw would teach the model that dates are
   `O` in half the corpus. Any mixed-genre variant has to solve that first.
 Genre matters more than any of this. Measured with one config on the three shared labels, a
 PerDT-trained NER scores 72.80 F on prose and 45.72 on tweets, while a ParsTwiNER-trained one
 scores 68.59 on tweets and 55.59 on prose. Training on both gives 72.11 and 66.49, so mixing
 costs under 1 F on prose and 2 F on tweets while recovering roughly 20 F off-genre. That is the
 argument for a future `fa_core_web_sm`, and the reason the current `news` packages stay
 prose-only.
 ### 3.3 Vectors, for md and lg
 | Option | License | Note |
 | --- | --- | --- |
-| **floret vectors** trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | **recommended.** Subword + Bloom-hash embeddings: bounded table size, zero OOV |
+| floret vectors trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | recommended: subword + Bloom-hash embeddings, bounded table size, zero OOV |
-| fastText `cc.fa.300` | CC BY-SA 3.0 | quick fallback; classic word table, large and OOV-prone |
+| fastText `cc.fa.300` | CC BY-SA 3.0 | fallback; classic word table, large and OOV-prone |
-Floret is the right call for Persian specifically. Persian surface forms explode through
+Persian surface forms multiply through suffixation and through inconsistent ZWNJ (U+200C)
-suffixation *and* through inconsistent ZWNJ (U+200C) usage — the same word appears as
+usage: the same word appears as `می‌رود`, `میرود` and `می رود` in real text. A classic
-`می‌رود` / `میرود` / `می رود` in real text. A classic word-vector table has a miss for every
+word-vector table misses every variant it did not see, while floret's subword hashing covers
-variant; floret's subword hashing covers all of them. spaCy already ships floret vectors for
+them. spaCy ships floret vectors for Croatian, Finnish, Korean, Slovenian, Swedish and
-Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian for the same reason.
+Ukrainian for the same reason.
-### 3.4 Transformer encoders (for trf)
+### 3.4 Transformer encoders, for trf
 | Model | Arch | License | Note |
 | --- | --- | --- | --- |
-| **`HooshvareLab/roberta-fa-zwnj-base`** | RoBERTa-base | **Apache-2.0** | recommended: licensed, ZWNJ-aware, smallest of the credible options |
+| `HooshvareLab/roberta-fa-zwnj-base` | RoBERTa-base | Apache-2.0 | recommended: licensed, ZWNJ-aware, smallest credible option |
 | `FacebookAI/xlm-roberta-base` | XLM-R base, 278M | MIT | licensed but larger |
 | `PartAI/TookaBERT-Base` | BERT-base | Apache-2.0 | licensed |
 | `m3hrdadfi/albert-fa-base-v2` | ALBERT-base-v2 | Apache-2.0 | licensed, smallest |
-| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | **no licence on the card** | what hazm used; redistribution risk |
+| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | no licence on the card | what hazm used; redistribution risk |
-| `sbunlp/fabert` | BERT-base, 124M | **no licence on the card** | redistribution risk |
+| `sbunlp/fabert` | BERT-base, 124M | no licence on the card | redistribution risk |
-All of these are BERT/RoBERTa/XLM-R/ALBERT, so all are supported by both
+All are BERT, RoBERTa, XLM-R or ALBERT, so all work with `spacy-transformers` and with
-`spacy-transformers` and `spacy-curated-transformers` (the latter supports exactly
+`spacy-curated-transformers`, which supports exactly ALBERT, BERT, CamemBERT, RoBERTa and
-ALBERT / BERT / CamemBERT / RoBERTa / XLM-RoBERTa).
+XLM-RoBERTa.
-Note the trap: hazm's own pipelines use ParsBERT, which has **no license statement**. Copying
+hazm's pipelines use ParsBERT, which carries no licence statement. Copying that choice would
-that choice would reintroduce the redistribution problem that sank the 2018 attempt.
+reintroduce the redistribution problem that stopped the 2018 attempt.
-## 4. What already exists (and why it is not enough)
+## 4. What already exists, and why it is not enough
- **No trained spaCy Persian pipeline exists.** Zero `persian` / `farsi` / `fa_` hits in
+No trained spaCy Persian pipeline exists. There are zero `persian`, `farsi` or `fa_` hits in
-  spaCy's `website/meta/universe.json`. `spacy.load("fa_core_news_sm")` has never worked.
+spaCy's `website/meta/universe.json`, and `spacy.load("fa_core_news_sm")` has never worked.
 - **hazm** ships three *single-task* spaCy pipelines on the HF Hub:
  `hazm-parsbert-postagger` (`tag_acc` 0.9862, hazm's own EZ-augmented tagset),
  `hazm-bert-dependency-parser` (`dep_uas` 0.9246 / `dep_las` 0.8934, trained on PerDT),
  `hazm-parsbert-chunker` (`tag_acc` 0.9618). Each is `transformer + one component`,
  `version: 0.0.0`, empty `license`/`author`/`sources`, pinned to spaCy 3.6. Using all three
  means three separate BERT forward passes over the same text and no shared `Doc`.
 - **hazm's training code is not reusable.** Its trainable models are pycrfsuite CRFs
  (`hazm/sequence_tagger.py`); there is no `config.cfg` or `spacy train` anywhere in the repo.
  Its `Spacy*` classes only *download* the HF pipelines above.
 - **hazm's tokenizer is actively incompatible** with UD gold tokenization: `Normalizer`'s
  `AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes and `WordTokenizer.join_verb_parts()` glues
  multi-word verb chains into single underscore-joined tokens. Training against PerDT with
  hazm's tokenizer would misalign tokens systematically.
 - **DadmaTools** (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable
  spaCy pipeline package, and its NER wraps ARMAN/PEYMA — the restricted data again.
-So what hazm genuinely contributes to this project is **one validated design decision**
+hazm ships three single-task spaCy pipelines on the HF Hub: `hazm-parsbert-postagger`
-(PerDT is the corpus) and **the stop-word list already vendored into `spacy/lang/fa`**.
+(`tag_acc` 0.9862, hazm's own EZ-augmented tagset), `hazm-bert-dependency-parser` (`dep_uas`
-Everything else is built with spaCy-native tooling.
+0.9246, `dep_las` 0.8934, trained on PerDT) and `hazm-parsbert-chunker` (`tag_acc` 0.9618).
 Each is `transformer + one component`, `version: 0.0.0`, with empty `license`, `author` and
 `sources`, pinned to spaCy 3.6. Using all three costs three BERT forward passes over the same
 text and gives no shared `Doc`.
-## 5. Decision: train `fa_core_news_sm` first
+hazm's training code is not reusable: its trainable models are pycrfsuite CRFs
 (`hazm/sequence_tagger.py`), and the repo contains no `config.cfg` or `spacy train`. Its
 `Spacy*` classes only download the HF pipelines above.
-Not md/lg: those need floret vectors trained from scratch on Wikipedia+OSCAR (CPU-days) and
+hazm's tokenizer is incompatible with UD gold tokenization. `Normalizer`'s
-buy ~0.00 tag/dep accuracy in the English reference numbers.
+`AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes, and `WordTokenizer.join_verb_parts()` glues
-Not trf: 2 GB of VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait
+multi-word verb chains into single underscore-joined tokens, so training against PerDT with it
-until the CPU pipeline proves the data plumbing is right.
+would misalign tokens systematically.
 `sm` is also the tier every other tier is validated against: md/lg/trf reuse the exact same
 corpus conversion, config skeleton and evaluation harness.
-### Tokenization decision, settled with a measurement
+DadmaTools (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable spaCy
 pipeline package, and its NER wraps ARMAN and PEYMA.
 What hazm contributes here is one validated design decision, that PerDT is the corpus, and the
 stop-word list already vendored into `spacy/lang/fa`.
 ## 5. Decision: train the `sm` tier first
 Not md or lg: they need floret vectors trained from scratch on Wikipedia and OSCAR, costing
 CPU-days, and the English reference numbers show no tag or dep accuracy gain. Not trf: 2 GB of
 VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait until the CPU
 pipeline proves the data plumbing. `sm` is also the tier the others are validated against,
 since md, lg and trf reuse the same corpus conversion, config skeleton and evaluation harness.
 ### Tokenization, settled by measurement
 spaCy has no multi-word-token layer, so CoNLL-U MWT ranges (Persian pronominal clitics and
-copulas: `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on
+copulas, `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on
 the PerDT dev set with `scripts/tokenization_report.py`, comparing gold boundaries against
 `spacy.blank("fa")`'s tokenizer:
 | `spacy convert` mode | token P | token R | token F | XPOS types | merge artefacts |
 | --- | --- | --- | --- | --- | --- |
-| `--merge-subtokens` | 0.9860 | 0.9914 | **0.9887** | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space |
+| `--merge-subtokens` | 0.9860 | 0.9914 | 0.9887 | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space |
 | plain (clitics split) | 0.9875 | 0.9772 | 0.9823 | 35 | none |
-**Chosen: `--merge-subtokens`** (also what Explosion's `tagger_parser_ud` template does).
+Chosen: `--merge-subtokens`, which is also what Explosion's `tagger_parser_ud` template does.
-Rationale: without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never
+Without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never produce, so
-produce, so those tokens are permanently unlearnable and unpredictable at runtime. With it,
+those tokens are permanently unlearnable and unpredictable at runtime. With it, every gold
-every gold token is reachable; the cost is confined to rare composite XPOS tags such as
+token is reachable, and the cost is limited to rare composite XPOS tags such as
-`N_IANM_PR_JOPER` (noun + enclitic pronoun) — which are, at least, informative — and 1.5% of
+`N_IANM_PR_JOPER` (noun + enclitic pronoun) and 1.5% of lemmas that come out as two words.
 lemmas that come out as two words.
-The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which
+The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which is
-would be an upstream PR, not a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5.
+an upstream PR rather than a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5.
-### Final composition of `fa_core_news_sm`
+### Final composition
 Shared trained components, in both `fa_dep_news_sm` (7.5 MB) and `fa_core_news_sm` (13 MB),
 both CC BY-SA 4.0:
 | Component | Trained on | Metric | Test score |
 | --- | --- | --- | --- |
-| `tok2vec` | shared, PerDT | — | — |
+| `tok2vec` | shared, PerDT | | |
 | `tagger` (XPOS) | PerDT, 90 labels | `tag_acc` | 95.96 |
 | `morphologizer` (UPOS + FEATS) | PerDT, 298 labels | `pos_acc` / `morph_acc` | 96.24 / 96.29 |
 | `trainable_lemmatizer` | PerDT, 1,908 edit trees | `lemma_acc` | 97.91 |
 | `parser` | PerDT, 34 deprels | `dep_uas` / `dep_las` | 89.69 / 85.15 |
 | `ner` (own internal tok2vec) | ParsTwiNER, 6 labels | `ents_p/r/f` | 74.77 / 61.06 / 67.22 |
-Training cost on the target hardware (4-core i5-7200U, no GPU): **1h27m** for the UD
+`fa_core_news_sm` adds, and `fa_ent_news_sm` ships alone:
 components (early-stopped at step 10,800; best checkpoint step 9,200) and ~25 min for NER
 (best at step 4,000). Both runs are single-threaded, so they were run concurrently.
 Inference: ~9,250 words/s. Wheel: 13 MB.
-The `--merge-subtokens` artefacts predicted above are visible in the shipped model exactly as
+| Component | Trained on | Metric | Test score |
-expected — `کتاب‌هایش` ("his/her books") comes out as one token tagged `N_IANM_PR_JOPER` with
+| --- | --- | --- | --- |
-lemma `کتاب او`. Worth knowing before you consume `token.lemma_` downstream.
+| `ner` (own internal tok2vec) | PerDT NER layer, 7 labels | `ents_p/r/f` | 77.67 / 66.87 / 71.87 |
-Sources recorded in `meta.json`, both with their licenses, per
+Per label, F against training examples: `LOC` 80.24 (4,954), `DAT` 74.45 (1,323), `MON` 73.68
-<https://spacy.io/api/data-formats#meta>. CC BY-SA 4.0 on PerDT means the packaged pipeline
+(205), `ORG` 68.77 (2,643), `TIM` 66.67 (135), `PER` 65.29 (4,847), `PCT` 57.14 (121).
-must carry attribution and share-alike notice — handled in `scripts/assemble_core.py`.
+
 `PER` scoring below `LOC` and `ORG` on nearly the same amount of data is the silver labels
 showing through. PerDT includes titles and honorifics inside `PER` spans inconsistently: 6.24% of
 its `PER` spans start with one (`دکتر`, `مهندس`, `آقای`), against 1.41% in the human-annotated
 ParsTwiNER, so the boundaries are less regular than the count suggests. `MON`, `TIM` and `PCT`
 are thin enough that their scores rest on 4 to 11 test entities each and should be treated as
 indicative only. Persian money, times and percentages are regular enough that an `EntityRuler`
 may beat the statistical model for those three.
 Training cost on a 4-core i5-7200U with no GPU: 1h27m for the UD components (early-stopped at
 step 10,800, best checkpoint step 9,200) and 17 min for NER (best at step 6,000). Both runs are
 single-threaded and can run concurrently. Inference runs at about 9,250 words/s.
 The `--merge-subtokens` artefacts show up in the shipped model as predicted: `کتاب‌هایش`
 ("his/her books") is one token tagged `N_IANM_PR_JOPER` with lemma `کتاب او`. Check this before
 consuming `token.lemma_` downstream.
 Sources are recorded in each `meta.json` with their licences, per
 <https://spacy.io/api/data-formats#meta>, including the treebank's NER layer as its own entry
 crediting Beheshti-NER. CC BY-SA 4.0 on PerDT means every package carries attribution and a
 share-alike notice, handled in `scripts/finalize_pipeline.py`, which also enforces the shape of
 each variant: it refuses to publish a `dep` pipeline containing `ner`, or a `core` one without it.
--- a/docs/upstream/fa-noun-chunks.md
+++ b/docs/upstream/fa-noun-chunks.md
@ -9,20 +9,20 @@ labels = ["nsubj", "dobj", "nsubjpass", "pcomp", "pobj", "dative", "appos", "att
 ```
 `dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They are not
-Universal Dependencies relations, and **every Persian treebank is UD** (`UD_Persian-PerDT`,
+Universal Dependencies relations, and every Persian treebank is UD (`UD_Persian-PerDT`,
-`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline must
+`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline emits
-therefore emit UD labels, so five of the nine labels are dead code and `doc.noun_chunks`
+UD labels, so five of the nine labels are dead code and `doc.noun_chunks` returns bare head
-silently returns bare head nouns.
+nouns.
-Compare `spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py`, which
+`spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py` use UD labels
-use UD labels (`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their
+(`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their treebanks are UD.
-treebanks are UD too. `spacy/lang/de` legitimately uses TIGER labels (`sb`, `oa`, `nk`, …)
+`spacy/lang/de` uses TIGER labels (`sb`, `oa`, `nk`, …) because the German pipelines are
-because the German pipelines are trained on TIGER. Persian has no such excuse.
+trained on TIGER. Persian has no equivalent reason.
-Second, smaller problem: the iterator yields `word.left_edge.i` → `word.i + 1`, i.e. it only
+A second, smaller problem: the iterator yields `word.left_edge.i` to `word.i + 1`, so it only
-ever expands **left**. Persian noun phrases expand **right** through ezafe:
+expands left. Persian noun phrases expand right through ezafe. `رئیس انجمن جراحان قلب ایران`
-`رئیس انجمن جراحان قلب ایران` ("the head of the Iranian society of heart surgeons") is one NP
+("the head of the Iranian society of heart surgeons") is one NP whose head is the leftmost
-whose head is the leftmost token. Left-only expansion truncates it to `رئیس`.
+token, and left-only expansion truncates it to `رئیس`.
 ## Evidence
@ -48,9 +48,9 @@ deprels on NOUN/PROPN/PRON tokens in dev (top 15):
  nsubj:pass         38  -
 ```
-1.31 tokens per chunk is the tell: the shipped iterator is returning single head nouns.
+At 1.31 tokens per chunk the shipped iterator is returning single head nouns. The four most
-`nmod` (2894), `obl` (1401), `obl:arg` (1095) and `obj` (973) — the four most common
+common noun-bearing relations after `nsubj`, namely `nmod` (2894), `obl` (1401), `obl:arg`
-noun-bearing relations after `nsubj` — are all unreachable.
+(1095) and `obj` (973), are all unreachable.
 Live text:
@ -66,8 +66,8 @@ Live text:
 ## Proposed patch
-Swap in UD labels, expand right through modifier chains, and drop a leading `ADP`/`CCONJ`
+Swap in UD labels, expand right through modifier chains, and drop a leading `ADP` or `CCONJ`
-exactly as `fr`/`es` already do:
+as `fr` and `es` already do:
 ```diff
 --- a/spacy/lang/fa/syntax_iterators.py
@ -176,7 +176,7 @@ Add UD-label coverage:
 ```python
 def test_fa_noun_chunks_ezafe(fa_vocab):
-    # رئیس انجمن جراحان — "head of the surgeons' society"
+    # رئیس انجمن جراحان, "head of the surgeons' society"
    words = ["رئیس", "انجمن", "جراحان", "آمد"]
    heads = [3, 0, 1, 3]
    deps = ["nsubj", "nmod", "nmod", "ROOT"]
@ -196,28 +196,28 @@ def test_fa_noun_chunks_drops_leading_adp(fa_vocab):
 ## Other `spacy/lang/fa` gaps found while building this pipeline
-Ordered by how much they cost a real Persian pipeline:
+Ordered by how much they cost a Persian pipeline.
-1. **Clitic splitting.** The `fa` tokenizer cannot split pronominal enclitics or the enclitic
+1. Clitic splitting. The `fa` tokenizer cannot split pronominal enclitics or the enclitic
-   copula (`پدرم` → `پدر` + `م`, `ساکتند` → `ساکت` + `ند`), which UD treebanks annotate as
+   copula (`پدرم` = `پدر` + `م`, `ساکتند` = `ساکت` + `ند`), which UD treebanks annotate as
-   multiword tokens. Measured on PerDT dev: gold-vs-tokenizer token F is 0.9823 when clitics
+   multiword tokens. Measured on PerDT dev, gold-against-tokenizer token F is 0.9823 when
-   are kept split, and 1.49% of tokens are affected. Persian-specific `TOKENIZER_SUFFIXES`
+   clitics are kept split, affecting 1.49% of tokens. Persian-specific `TOKENIZER_SUFFIXES`
-   entries for the enclitic set would remove the need for `--merge-subtokens` and eliminate
+   entries for the enclitic set would remove the need for `--merge-subtokens` and the
-   the composite XPOS tags it produces.
+   composite XPOS tags it produces.
-2. **`punctuation.py` defines only `TOKENIZER_SUFFIXES`** — no `TOKENIZER_PREFIXES`, no
+2. `punctuation.py` defines only `TOKENIZER_SUFFIXES`, with no `TOKENIZER_PREFIXES` and no
   `TOKENIZER_INFIXES`. ZWNJ (U+200C) is handled only implicitly through the 65 KB generated
-   verb-exception table. The missing infix rules bite on numerics: `spacy/lang/fa/examples.py`
+   verb-exception table. The missing infix rules break numerics: `spacy/lang/fa/examples.py`
   ships the sentence `دیروز علی به من ۲۰۰۰.۱﷼ پول نقد داد.`, and the trained pipeline splits
-   `۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, with the stray `.` promoted to a sentence boundary —
+   `۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, promoting the stray `.` to a sentence boundary, so one
-   one input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian
+   input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian digits,
-   digits, but no tokenizer rule keeps a Persian decimal or a currency sign attached.
+   but no tokenizer rule keeps a Persian decimal or a currency sign attached.
-3. **No tokenizer tests at all** for `fa` — `spacy/tests/lang/fa/` contains only
+3. No tokenizer tests for `fa`. `spacy/tests/lang/fa/` contains only `test_noun_chunks.py`,
-   `test_noun_chunks.py`, guarding none of the 65 KB exception table.
+   which guards none of the 65 KB exception table.
-4. **`spacy-lookups-data` has no `fa_license.txt`**, although `fa_source.txt` records that the
+4. `spacy-lookups-data` has no `fa_license.txt`, although `fa_source.txt` records that the
-   lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" —
+   lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus",
-   which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`; Persian should too.
+   which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`.
-5. **No `fa_lemma_lookup.json`** — only rule-mode lemmatizer assets exist, so
+5. No `fa_lemma_lookup.json`. Only rule-mode lemmatizer assets exist, so `mode="lookup"` is
-   `mode="lookup"` is unavailable for Persian.
+   unavailable for Persian.
-Items 1–3 are self-contained code PRs. Item 4 is a licence-hygiene PR against
+Items 1 to 3 are self-contained code PRs. Item 4 is a licence-hygiene PR against
-`spacy-lookups-data` and matters for anyone redistributing a Persian pipeline.
+`spacy-lookups-data`, and affects anyone redistributing a Persian pipeline.
--- a/project.yml
+++ b/project.yml
@ -1,19 +1,29 @@
-title: "fa_core_news_sm"
+title: "fa_core_news_sm / fa_dep_news_sm"
 description: >
-  A CPU-sized Persian (fa) core pipeline for spaCy 3.8: tagger (XPOS), morphologizer
+  CPU-sized Persian (fa) pipelines for spaCy 3.8, built entirely from UD_Persian-PerDT
-  (UPOS + FEATS), trainable lemmatizer, dependency parser and NER.
+  (PerUDT v1.0, CC BY-SA 4.0): tagger (XPOS), morphologizer (UPOS + FEATS), trainable
  lemmatizer, dependency parser and NER.
-  UD components are trained on UD_Persian-PerDT (PerUDT v1.0, CC BY-SA 4.0). The NER
+  Two shipping packages, same corpus, differing only in whether NER is included:
-  component is trained separately on ParsTwiNER (MIT) and merged in, because no
+  `fa_dep_news_sm` (no NER) and `fa_core_news_sm` (with NER). The NER comes from the
-  redistributably-licensed Persian NER corpus shares a genre with the treebank. See
+  treebank's own `not-to-release/Dadegan with NER tag/` layer, so both packages share one
-  docs/MODELS.md for the full source/licence analysis and docs/CONTRIBUTING-GUIDE.md
+  corpus, one genre, one tokenization and one licence.
  for how this gets published.
-  Run everything with: `spacy project run all`
+  PerDT's NER labels are silver, produced by Beheshti-NER with manual corrections, and are
  transferred onto this pipeline's tokenization by difflib at a 99.86% rate. Per-label
  scores are published in meta.json; MON, TIM and PCT are thin.
  See docs/MODELS.md for the source and licence analysis and docs/CONTRIBUTING-GUIDE.md for
  how this gets published.
  Build both with: `spacy project run all`
  Standalone NER-only package: `spacy project run ent`
 vars:
  lang: "fa"
-  package_name: "core_news_sm"
+  dep_package_name: "dep_news_sm"
  core_package_name: "core_news_sm"
  ent_package_name: "ent_news_sm"
  package_version: "3.8.0"
  treebank: "fa_perdt"
  # -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline.
@ -42,24 +52,42 @@ assets:
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-test.conllu"
    checksum: "b62a66994cef2c50f7e524a1471102d8"
    description: "UD_Persian-PerDT test split (CC BY-SA 4.0)"
-  - dest: "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
+  - dest: "assets/ud-ner/train_with_NER_tag.txt"
-    url: "https://github.com/overfit-ir/parstwiner/releases/download/v1.0.0/ParsTwiNER_corpus_v1.0.0.zip"
+    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/train_with_NER_tag.txt"
-    checksum: "54615340d76c4d51b8f54751b07a278d"
+    checksum: "ecb96cf99b38bc485cac21d22914e413"
-    description: "ParsTwiNER Persian Twitter NER corpus, IOB2 (MIT)"
+    description: "PerDT NER layer, train split, IOB2 (CC BY-SA 4.0)"
  - dest: "assets/ud-ner/dev_with_NER_tag.txt"
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/dev_with_NER_tag.txt"
    checksum: "2a56ef7eb2e3732e221317af457d1c09"
    description: "PerDT NER layer, dev split, IOB2 (CC BY-SA 4.0)"
  - dest: "assets/ud-ner/test_with_NER_tag.txt"
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/test_with_NER_tag.txt"
    checksum: "6d80dd783527562c2ea5189f218a12b5"
    description: "PerDT NER layer, test split, IOB2 (CC BY-SA 4.0)"
 workflows:
  # Both shipping artifacts: fa_dep_news_sm and fa_core_news_sm.
  all:
    - inspect
    - convert-ud
    - transfer-ner
    - convert-ner
    - debug-data
-    - train-core
+    - debug-data-ner
    - train-dep
    - train-ner
-    - assemble
+    - finalize-dep
-    - evaluate
+    - evaluate-dep
    - assemble-core
    - evaluate-core
    - finalize-meta
    - package
    - smoke
  # Optional third artifact: the NER alone, for users who only want entities.
  ent:
    - finalize-ent
    - evaluate-ent
    - package-ent
 commands:
  - name: "inspect"
@ -91,107 +119,185 @@ commands:
      - "corpus/merged/${vars.treebank}-ud-dev.spacy"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
-  - name: "convert-ner"
+  - name: "transfer-ner"
-    help: "Unpack ParsTwiNER and convert its IOB2 files to DocBin"
+    help: >
      Align PerDT's NER layer onto the --merge-subtokens tokenization. The NER files use the
      original Dadegan tokenization, which matches the released UD tokenization in only 57 to
      62% of sentences, so spans are transferred by difflib. Measured rate 99.86%; spans that
      cannot be aligned exactly are dropped rather than guessed.
    script:
-      - "python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner"
+      - "python scripts/transfer_perdt_ner.py --conllu-dir assets/ud --ner-dir assets/ud-ner --out corpus/perdt-ner-iob"
      - "python -m spacy convert assets/ner/train.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
      - "python -m spacy convert assets/ner/dev.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
      - "python -m spacy convert assets/ner/test.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
    deps:
-      - "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
+      - "assets/ud/${vars.treebank}-ud-train.conllu"
-      - "scripts/extract_parstwiner.py"
+      - "assets/ud-ner/train_with_NER_tag.txt"
      - "scripts/transfer_perdt_ner.py"
    outputs:
-      - "corpus/ner/train.spacy"
+      - "corpus/perdt-ner-iob/train.txt"
-      - "corpus/ner/dev.spacy"
+      - "corpus/perdt-ner-iob/dev.txt"
-      - "corpus/ner/test.spacy"
+      - "corpus/perdt-ner-iob/test.txt"
  - name: "convert-ner"
    help: "Transferred IOB2 -> DocBin"
    script:
      - "python -m spacy convert corpus/perdt-ner-iob/train.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
      - "python -m spacy convert corpus/perdt-ner-iob/dev.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
      - "python -m spacy convert corpus/perdt-ner-iob/test.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
    deps:
      - "corpus/perdt-ner-iob/train.txt"
    outputs:
      - "corpus/perdt-ner/train.spacy"
      - "corpus/perdt-ner/dev.spacy"
      - "corpus/perdt-ner/test.spacy"
  - name: "debug-data"
-    help: "Validate both corpora against their configs before burning CPU on training"
+    help: "Validate the treebank against the config before burning CPU on training"
    script:
-      - "python -m spacy debug data configs/fa_core_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy"
+      - "python -m spacy debug data configs/fa_dep_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy"
      - "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy"
    deps:
      - "corpus/merged/${vars.treebank}-ud-train.spacy"
-      - "corpus/ner/train.spacy"
+      - "configs/fa_dep_news_sm.cfg"
-      - "configs/fa_core_news_sm.cfg"
+
  - name: "debug-data-ner"
    help: "Validate the transferred PerDT NER corpus against the NER config"
    script:
      - "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy"
    deps:
      - "corpus/perdt-ner/train.spacy"
      - "configs/fa_ner_sm.cfg"
-  - name: "train-core"
+  - name: "train-dep"
    help: "Train tok2vec + tagger + morphologizer + trainable_lemmatizer + parser on PerDT"
    script:
-      - "python -m spacy train configs/fa_core_news_sm.cfg --output training/core --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}"
+      - "python -m spacy train configs/fa_dep_news_sm.cfg --output training/dep --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}"
    deps:
      - "corpus/merged/${vars.treebank}-ud-train.spacy"
      - "corpus/merged/${vars.treebank}-ud-dev.spacy"
-      - "configs/fa_core_news_sm.cfg"
+      - "configs/fa_dep_news_sm.cfg"
    outputs:
-      - "training/core/model-best"
+      - "training/dep/model-best"
  - name: "train-ner"
-    help: "Train the standalone NER component (own internal tok2vec) on ParsTwiNER"
+    help: "Train the NER component (own embedded tok2vec) on the transferred PerDT layer"
    script:
-      - "python -m spacy train configs/fa_ner_sm.cfg --output training/ner --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy --gpu-id ${vars.gpu}"
+      - "python -m spacy train configs/fa_ner_sm.cfg --output training/perdt-ner --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy --gpu-id ${vars.gpu}"
    deps:
-      - "corpus/ner/train.spacy"
+      - "corpus/perdt-ner/train.spacy"
-      - "corpus/ner/dev.spacy"
+      - "corpus/perdt-ner/dev.spacy"
      - "configs/fa_ner_sm.cfg"
    outputs:
-      - "training/ner/model-best"
+      - "training/perdt-ner/model-best"
-  - name: "assemble"
+  - name: "finalize-dep"
-    help: "Source the trained ner into the core pipeline and write full meta.json"
+    help: "Write fa_dep_news_sm metadata (sources, licence, notes) onto the trained model"
    script:
-      - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version}"
+      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_dep_news_sm --variant dep --version ${vars.package_version}"
    deps:
-      - "training/core/model-best"
+      - "training/dep/model-best"
-      - "training/ner/model-best"
+      - "scripts/finalize_pipeline.py"
-      - "scripts/assemble_core.py"
+    outputs:
      - "training/fa_dep_news_sm"
  - name: "evaluate-dep"
    help: "Score fa_dep_news_sm on the held-out UD test split"
    script:
      - "python -m spacy benchmark accuracy training/fa_dep_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}"
    deps:
      - "training/fa_dep_news_sm"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
    outputs:
      - "metrics/ud-test.json"
  - name: "assemble-core"
    help: >
      Source the trained ner into the dep pipeline to produce fa_core_news_sm. Possible
      because configs/fa_ner_sm.cfg embeds its own tok2vec instead of a Tok2VecListener.
    script:
      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_core_news_sm --variant core --version ${vars.package_version} --add-ner training/perdt-ner/model-best"
    deps:
      - "training/dep/model-best"
      - "training/perdt-ner/model-best"
      - "scripts/finalize_pipeline.py"
    outputs:
      - "training/fa_core_news_sm"
-  - name: "evaluate"
+  - name: "evaluate-core"
-    help: "Score the assembled pipeline on both held-out test sets"
+    help: "Score fa_core_news_sm on both held-out test splits"
    script:
-      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}"
+      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/core-ud-test.json --gpu-id ${vars.gpu}"
-      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/ner/test.spacy --output metrics/ner-test.json --gpu-id ${vars.gpu}"
+      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/perdt-ner/test.spacy --output metrics/perdt-ner-test.json --gpu-id ${vars.gpu}"
    deps:
      - "training/fa_core_news_sm"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
-      - "corpus/ner/test.spacy"
+      - "corpus/perdt-ner/test.spacy"
    outputs:
-      - "metrics/ud-test.json"
+      - "metrics/core-ud-test.json"
-      - "metrics/ner-test.json"
+      - "metrics/perdt-ner-test.json"
  - name: "finalize-meta"
    help: >
-      Re-assemble, this time folding the test scores into meta.json["performance"].
+      Re-run finalize on both packages, folding test scores into meta.json["performance"].
-      Separate from `assemble` because the scores can only exist after `evaluate`, and
+      Separate because the scores only exist after evaluation, and evaluation needs a
-      `evaluate` needs an assembled pipeline to score. Cheap: it only copies models.
+      finalized pipeline to score. Cheap: it only copies models.
    script:
-      - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version} --ud-metrics metrics/ud-test.json --ner-metrics metrics/ner-test.json"
+      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_dep_news_sm --variant dep --version ${vars.package_version} --ud-metrics metrics/ud-test.json"
      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_core_news_sm --variant core --version ${vars.package_version} --add-ner training/perdt-ner/model-best --ud-metrics metrics/core-ud-test.json --ner-metrics metrics/perdt-ner-test.json"
    deps:
      - "metrics/ud-test.json"
-      - "metrics/ner-test.json"
+      - "metrics/perdt-ner-test.json"
-      - "scripts/assemble_core.py"
+      - "scripts/finalize_pipeline.py"
  - name: "package"
-    help: "Build the installable wheel + sdist"
+    help: "Build installable wheels + sdists for both shipping packages"
    script:
-      - "python -m spacy package training/fa_core_news_sm packages --name ${vars.package_name} --version ${vars.package_version} --build sdist,wheel --force"
+      - "python -m spacy package training/fa_dep_news_sm packages --name ${vars.dep_package_name} --version ${vars.package_version} --build sdist,wheel --force"
      - "python -m spacy package training/fa_core_news_sm packages --name ${vars.core_package_name} --version ${vars.package_version} --build sdist,wheel --force"
    deps:
      - "training/fa_dep_news_sm"
      - "training/fa_core_news_sm"
    outputs:
-      - "packages/${vars.lang}_${vars.package_name}-${vars.package_version}"
+      - "packages/${vars.lang}_${vars.dep_package_name}-${vars.package_version}"
      - "packages/${vars.lang}_${vars.core_package_name}-${vars.package_version}"
  - name: "smoke"
-    help: "Load the packaged pipeline and run it over real Persian text"
+    help: "Load both pipelines and run them over real Persian text"
    script:
      - "python scripts/smoke_test.py training/fa_dep_news_sm"
      - "python scripts/smoke_test.py training/fa_core_news_sm"
    deps:
      - "training/fa_dep_news_sm"
      - "training/fa_core_news_sm"
  - name: "finalize-ent"
    help: "Write fa_ent_news_sm metadata onto the trained NER model"
    script:
      - "python scripts/finalize_pipeline.py training/perdt-ner/model-best training/fa_ent_news_sm --variant ent --version ${vars.package_version}"
    deps:
      - "training/perdt-ner/model-best"
      - "scripts/finalize_pipeline.py"
    outputs:
      - "training/fa_ent_news_sm"
  - name: "evaluate-ent"
    help: "Score fa_ent_news_sm on the held-out PerDT NER test split"
    script:
      - "python -m spacy benchmark accuracy training/fa_ent_news_sm corpus/perdt-ner/test.spacy --output metrics/ent-test.json --gpu-id ${vars.gpu}"
      - "python scripts/finalize_pipeline.py training/perdt-ner/model-best training/fa_ent_news_sm --variant ent --version ${vars.package_version} --ner-metrics metrics/ent-test.json"
    deps:
      - "training/fa_ent_news_sm"
      - "corpus/perdt-ner/test.spacy"
    outputs:
      - "metrics/ent-test.json"
  - name: "package-ent"
    help: "Build the installable fa_ent_news_sm wheel + sdist"
    script:
      - "python -m spacy package training/fa_ent_news_sm packages --name ${vars.ent_package_name} --version ${vars.package_version} --build sdist,wheel --force"
    deps:
      - "training/fa_ent_news_sm"
    outputs:
      - "packages/${vars.lang}_${vars.ent_package_name}-${vars.package_version}"
  - name: "clean"
    help: "Drop corpora, training runs and metrics (keeps downloaded assets)"
    script:
-      - "rm -rf corpus/merged corpus/split corpus/ner training metrics packages"
+      - "rm -rf corpus training metrics packages"
--- a/scripts/assemble_core.py
+++ b/scripts/assemble_core.py
@ -1,133 +0,0 @@
 """Merge the separately-trained NER component into the UD pipeline and write full metadata.
 The two components cannot be trained together: the UD components come from
 UD_Persian-PerDT (CC BY-SA 4.0, edited prose) and `ner` comes from ParsTwiNER (MIT,
 tweets). `ner` was therefore configured with its own embedded tok2vec (see
 configs/fa_ner_sm.cfg) so it can be sourced into another pipeline without a dangling
 Tok2VecListener — the same design as en_core_web_sm.
 Run it twice: once right after training (no metrics yet), and again after
 `spacy benchmark accuracy` has produced metrics/*.json so `meta.json["performance"]`
 reflects the assembled pipeline on the held-out test sets.
 Usage:
  .venv/bin/python scripts/assemble_core.py training/core/model-best training/ner/model-best \\
      training/fa_core_news_sm --version 3.8.0 \\
      [--ud-metrics metrics/ud-test.json] [--ner-metrics metrics/ner-test.json]
 """
 import argparse
 import json
 from pathlib import Path
 import spacy
 DESCRIPTION = (
    "Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, "
    "trainable_lemmatizer, parser, ner."
 )
 SOURCES = [
    {
        "name": "UD_Persian-PerDT (PerUDT v1.0)",
        "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
        "author": (
            "Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian"
        ),
        "license": "CC BY-SA 4.0",
    },
    {
        "name": "ParsTwiNER",
        "url": "https://github.com/overfit-ir/parstwiner",
        "author": "MohammadMahdi Aghajani, AliAkbar Badri, Hamid Beigy et al. (Overfit-IR)",
        "license": "MIT",
    },
    {
        "name": "spaCy lang/fa language data (stop words originally from HAZM)",
        "url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa",
        "author": "Explosion and spaCy contributors",
        "license": "MIT",
    },
 ]
 # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
 NOTES = (
    "The tagger, morphologizer, trainable_lemmatizer and parser are trained on "
    "UD_Persian-PerDT, which is licensed CC BY-SA 4.0; this pipeline is therefore "
    "distributed under CC BY-SA 4.0 with attribution to the treebank authors. "
    "The ner component is trained on ParsTwiNER (MIT), a Twitter corpus, so entity "
    "recognition is weaker on formal/edited prose than on social media text. "
    "Multiword tokens in the treebank (pronominal clitics, enclitic copulas) were merged "
    "with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite "
    "(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space."
 )
 def load_metrics(path):
    if not path:
        return {}
    p = Path(path)
    if not p.exists():
        print(f"  (no metrics at {p}, skipping)")
        return {}
    return json.loads(p.read_text(encoding="utf8"))
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("core", help="trained UD pipeline (training/core/model-best)")
    ap.add_argument("ner", help="trained NER pipeline (training/ner/model-best)")
    ap.add_argument("output", help="destination directory for the assembled pipeline")
    ap.add_argument("--version", default="3.8.0")
    ap.add_argument("--ud-metrics", default=None)
    ap.add_argument("--ner-metrics", default=None)
    args = ap.parse_args()
    nlp = spacy.load(args.core)
    ner_nlp = spacy.load(args.ner)
    if "ner" in nlp.pipe_names:
        nlp.remove_pipe("ner")
    nlp.add_pipe("ner", source=ner_nlp)
    print(f"pipeline: {nlp.pipe_names}")
    print(f"ner labels: {sorted(nlp.get_pipe('ner').labels)}")
    ud = load_metrics(args.ud_metrics)
    ner = load_metrics(args.ner_metrics)
    performance = dict(nlp.meta.get("performance", {}))
    for key in ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc",
                "morph_acc", "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r",
                "sents_f"):
        if key in ud:
            performance[key] = ud[key]
    for key in ("ents_p", "ents_r", "ents_f"):
        if key in ner:
            performance[key] = ner[key]
    if "ents_per_type" in ner:
        performance["ents_per_type"] = ner["ents_per_type"]
    if "dep_las_per_type" in ud:
        performance["dep_las_per_type"] = ud["dep_las_per_type"]
    nlp.meta.update(
        {
            "lang": "fa",
            "name": "core_news_sm",
            "version": args.version,
            "description": DESCRIPTION,
            "author": "",
            "email": "",
            "url": "",
            "license": "CC BY-SA 4.0",
            "sources": SOURCES,
            "notes": NOTES,
            "performance": performance,
        }
    )
    out = Path(args.output)
    nlp.to_disk(out)
    print(f"wrote {out}")
    print(json.dumps(performance, indent=2, ensure_ascii=False)[:1200])
 if __name__ == "__main__":
    main()
--- a/scripts/extract_parstwiner.py
+++ b/scripts/extract_parstwiner.py
@ -1,34 +0,0 @@
 """Unpack the ParsTwiNER release zip into flat IOB2 files.
 The archive carries macOS resource-fork junk (`__MACOSX/._*`) that would confuse
 `spacy convert`, so only the three real splits are extracted.
 Usage: .venv/bin/python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner
 """
 import sys
 import zipfile
 from pathlib import Path
 SPLITS = ("train.txt", "dev.txt", "test.txt")
 def main():
    archive = Path(sys.argv[1])
    dest = Path(sys.argv[2])
    dest.mkdir(parents=True, exist_ok=True)
    with zipfile.ZipFile(archive) as zf:
        names = set(zf.namelist())
        missing = [s for s in SPLITS if s not in names]
        if missing:
            raise SystemExit(f"{archive}: missing expected members {missing}")
        for split in SPLITS:
            zf.extract(split, dest)
            path = dest / split
            sents = path.read_text(encoding="utf8").strip().split("\n\n")
            tokens = sum(1 for line in path.open(encoding="utf8") if line.strip())
            print(f"{path}: {len(sents)} sentences, {tokens} tokens")
 if __name__ == "__main__":
    main()
--- a/scripts/finalize_pipeline.py
+++ b/scripts/finalize_pipeline.py
@ -0,0 +1,226 @@
 """Write complete, convention-compliant metadata onto a trained pipeline.
 Three variants, following spaCy's `[lang]_[type]_[genre]_[size]` naming
 (https://spacy.io/models#conventions):
  dep  -> fa_dep_news_sm    tagger + morphologizer + trainable_lemmatizer + parser
  core -> fa_core_news_sm   the above plus ner
  ent  -> fa_ent_news_sm    ner only
 All three are built from UD_Persian-PerDT alone, including the NER, which comes from that
 treebank's own `not-to-release/Dadegan with NER tag/` layer. That is what makes `core`
 honest here: one corpus, one genre, one licence, one provenance chain. An earlier version of
 this script refused to emit `core` because the only redistributable Persian NER corpus we
 knew of was ParsTwiNER, a Twitter corpus scoring 67.22 F against 85-98 for the UD
 components; merging those into one package would have hidden a genre and quality gap behind
 a single name and version.
 PerDT's NER labels are silver, produced by Beheshti-NER (Taher et al., 2020) with manual
 corrections, so `notes` says so and the per-label scores are published as measured.
 Run twice: once after training (no metrics yet), then again after
 `spacy benchmark accuracy` so meta.json["performance"] carries held-out test scores.
 Usage:
  .venv/bin/python scripts/finalize_pipeline.py training/core/model-best training/fa_dep_news_sm \\
      --variant dep --version 3.8.0 [--ud-metrics metrics/ud-test.json]
  .venv/bin/python scripts/finalize_pipeline.py training/fa_core_news_sm training/fa_core_news_sm \\
      --variant core --version 3.8.0 --ud-metrics metrics/core-ud-test.json --ner-metrics metrics/perdt-ner-test.json
 """
 import argparse
 import json
 from pathlib import Path
 import spacy
 AUTHOR = "Kiyarash Fazeli"
 EMAIL = "kiyarash@nlogn.ir"
 PROJECT_URL = "https://github.com/Fazel94/spacy-persian"
 PERDT = {
    "name": "UD_Persian-PerDT (PerUDT v1.0)",
    "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
    "author": "Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian",
    "license": "CC BY-SA 4.0",
 }
 PERDT_NER = {
    "name": "UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)",
    "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
    "author": "PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020)",
    "license": "CC BY-SA 4.0",
 }
 LANG_DATA = {
    "name": "spaCy lang/fa language data (stop words originally from HAZM)",
    "url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa",
    "author": "Explosion and spaCy contributors",
    "license": "MIT",
 }
 NER_NOTE = (
    "The ner component is trained on the NER layer shipped in UD_Persian-PerDT's "
    "not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. "
    "Those labels are SILVER: the treebank README states they were produced by the BERT-based "
    "Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They "
    "were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% "
    "transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly "
    "were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more "
    "training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in "
    "`performance.ents_per_type` should be read before relying on them."
 )
 MWT_NOTE = (
    "Multiword tokens (pronominal clitics, enclitic copulas) were merged with "
    "`spacy convert --merge-subtokens`, so a small number of XPOS tags are composite "
    "(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space."
 )
 CHUNK_NOTE = (
    "doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches "
    "ClearNLP labels that do not exist in Universal Dependencies, see "
    "docs/upstream/fa-noun-chunks.md."
 )
 # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
 PERDT_LICENSE = "CC BY-SA 4.0"
 ATTRIBUTION = (
    "Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore "
    "distributed under CC BY-SA 4.0 with attribution to the treebank authors."
 )
 UD_KEYS = ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc", "morph_acc",
           "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r", "sents_f",
           "dep_las_per_type")
 NER_KEYS = ("ents_p", "ents_r", "ents_f", "ents_per_type")
 VARIANTS = {
    "dep": {
        "name": "dep_news_sm",
        "description": (
            "Persian dependency pipeline optimized for CPU. Components: tok2vec, tagger, "
            "morphologizer, trainable_lemmatizer, parser. No NER, see fa_core_news_sm."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, MWT_NOTE, CHUNK_NOTE]),
        "keys": UD_KEYS,
        "require": lambda pipes: "ner" not in pipes,
        "require_msg": "a 'dep' pipeline must not contain an ner component",
    },
    "core": {
        "name": "core_news_sm",
        "description": (
            "Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, "
            "trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, "
            "PCT."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT, PERDT_NER, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, NER_NOTE, MWT_NOTE, CHUNK_NOTE]),
        "keys": UD_KEYS + NER_KEYS,
        "require": lambda pipes: "ner" in pipes and "parser" in pipes,
        "require_msg": "a 'core' pipeline must contain both parser and ner",
    },
    "ent": {
        "name": "ent_news_sm",
        "description": (
            "Persian named entity recognizer optimized for CPU, with its own internal "
            "tok2vec. Labels: PER, LOC, ORG, DAT, MON, TIM, PCT."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT_NER, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, NER_NOTE]),
        "keys": ("token_acc",) + NER_KEYS,
        "require": lambda pipes: pipes == ["ner"],
        "require_msg": "an 'ent' pipeline must be exactly ['ner']",
    },
 }
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("model", help="trained pipeline, e.g. training/core/model-best")
    ap.add_argument("output", help="destination directory")
    ap.add_argument("--variant", choices=sorted(VARIANTS), required=True)
    ap.add_argument("--version", default="3.8.0")
    ap.add_argument("--ud-metrics", default=None,
                    help="benchmark accuracy JSON scored on the UD test split; supplies the "
                         "tagger/morph/lemma/parser keys only")
    ap.add_argument("--ner-metrics", default=None,
                    help="benchmark accuracy JSON scored on the NER test split; supplies the "
                         "ents_* keys only")
    ap.add_argument("--add-ner", default=None,
                    help="source the ner component from this trained pipeline first "
                         "(used to assemble 'core' from the dep model plus the ner model)")
    args = ap.parse_args()
    spec = VARIANTS[args.variant]
    nlp = spacy.load(args.model)
    if args.add_ner:
        ner_nlp = spacy.load(args.add_ner)
        if ner_nlp.pipe_names != ["ner"]:
            raise SystemExit(f"--add-ner expects a ['ner'] pipeline, got {ner_nlp.pipe_names}")
        if "ner" in nlp.pipe_names:
            nlp.remove_pipe("ner")
        # `ner` embeds its own tok2vec (configs/fa_ner_sm.cfg), so sourcing it here leaves no
        # dangling Tok2VecListener. See docs/MODELS.md.
        nlp.add_pipe("ner", source=ner_nlp)
        print(f"sourced ner from {args.add_ner}: {sorted(nlp.get_pipe('ner').labels)}")
    print(f"pipeline: {nlp.pipe_names}")
    if not spec["require"](list(nlp.pipe_names)):
        raise SystemExit(
            f"refusing to publish as '{args.variant}': {spec['require_msg']}"
            f" (got {list(nlp.pipe_names)})"
        )
    # Build from a strict whitelist rather than inheriting nlp.meta["performance"], which
    # after training also carries raw *_loss values and scorer keys that do not apply to
    # this pipeline (e.g. tag_micro_f: 0.0).
    #
    # Each metrics file supplies only the keys its corpus can actually evidence. Folding both
    # files over the same key set silently corrupted core's metadata: the NER corpus has no
    # gold tags, so its report carries tag_acc: 0.0, which overwrote the real 95.96, and its
    # --n-sents grouping produced a misleading sents_f.
    trained = nlp.meta.get("performance", {})
    performance = {k: trained[k] for k in spec["keys"] if trained.get(k) is not None}
    def fold(path, keys, label):
        if not path:
            return
        p = Path(path)
        if not p.exists():
            print(f"  (no {label} metrics at {p}, skipping; run the evaluate step first)")
            return
        scored = json.loads(p.read_text(encoding="utf8"))
        taken = [k for k in keys if k in spec["keys"] and scored.get(k) is not None]
        for key in taken:
            performance[key] = scored[key]
        if scored.get("speed") is not None:
            performance.setdefault("speed", scored["speed"])
        print(f"  folded {len(taken)} {label} keys from {p}")
    fold(args.ud_metrics, UD_KEYS + ("token_acc",), "UD")
    fold(args.ner_metrics, NER_KEYS, "NER")
    nlp.meta.update(
        {
            "lang": "fa",
            "name": spec["name"],
            "version": args.version,
            "description": spec["description"],
            "author": AUTHOR,
            "email": EMAIL,
            "url": PROJECT_URL,
            "license": spec["license"],
            "sources": spec["sources"],
            "notes": spec["notes"],
            "performance": performance,
        }
    )
    out = Path(args.output)
    nlp.to_disk(out)
    print(f"wrote {out} as fa_{spec['name']} {args.version} ({spec['license']})")
    scalars = {k: round(v * 100, 2) for k, v in performance.items() if isinstance(v, float)}
    print(json.dumps(scalars, indent=2))
 if __name__ == "__main__":
    main()
--- a/scripts/smoke_test.py
+++ b/scripts/smoke_test.py
@ -1,9 +1,10 @@
-"""Run the assembled pipeline over real Persian text and print every annotation layer.
+"""Run a trained pipeline over real Persian text and print every annotation layer.
 This is the end-to-end check that the artifact actually works: tokenizer -> tagger ->
-morphologizer -> lemmatizer -> parser -> ner -> noun_chunks.
+morphologizer -> lemmatizer -> parser -> noun_chunks, plus ner when the pipeline has one.
 Works on fa_dep_news_sm, fa_ent_news_sm, or the two combined.
-Usage: .venv/bin/python scripts/smoke_test.py training/fa_core_news_sm
+Usage: .venv/bin/python scripts/smoke_test.py training/fa_dep_news_sm
 """
 import sys
@ -20,7 +21,7 @@ EXTRA = [
 def main():
-    path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_core_news_sm"
+    path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_dep_news_sm"
    nlp = spacy.load(path)
    print(f"loaded {nlp.meta['lang']}_{nlp.meta['name']} {nlp.meta['version']}")
    print(f"pipeline: {nlp.pipe_names}")
@ -35,10 +36,13 @@ def main():
                f"{t.text:<16}{t.lemma_:<16}{t.pos_:<7}{t.tag_:<14}"
                f"{t.dep_:<14}{t.head.text}"
            )
-        print(f"morph[0]: {doc[0].morph}")
+        if doc.has_annotation("MORPH"):
-        print(f"sents: {[s.text for s in doc.sents]}")
+            print(f"morph[0]: {doc[0].morph}")
-        print(f"ents: {[(e.text, e.label_) for e in doc.ents]}")
+        if doc.has_annotation("DEP"):
-        print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}")
+            print(f"sents: {[s.text for s in doc.sents]}")
            print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}")
        if "ner" in nlp.pipe_names:
            print(f"ents: {[(e.text, e.label_) for e in doc.ents]}")
 if __name__ == "__main__":
--- a/scripts/transfer_perdt_ner.py
+++ b/scripts/transfer_perdt_ner.py
@ -0,0 +1,169 @@
 """Transfer PerDT's NER layer onto the tokenization `spacy convert --merge-subtokens` produces.
 The NER files use the original Dadegan tokenization, which matches the released UD tokenization
 in only 57 to 62% of sentences: the NER files drop some copulas and auxiliaries, and at least
 one honorific is corrupted (`ص` written as `،`). Entities sit on content words present in both,
 so difflib aligns them. A span transfers only if every one of its tokens maps and the result
 stays contiguous; anything else is dropped rather than guessed.
 Measured transfer rate: 99.86% (train), 99.74% (dev), 99.51% (test). See PLAN.md §1.
 Usage:
  python scripts/transfer_perdt_ner.py --conllu-dir assets/ud --ner-dir assets/ud-ner \\
      --out corpus/perdt-ner-iob
 With --out, writes IOB2 files on the merged tokenization to DIR/{split}.txt.
 Without it, only reports the transfer rate.
 """
 import argparse
 from collections import Counter
 from difflib import SequenceMatcher
 from pathlib import Path
 SPLITS = ("train", "dev", "test")
 def merged_tokens(conllu_path):
    """Tokens as --merge-subtokens yields them: multiword-token surface forms, not subtokens."""
    sents, cur, skip_to = [], [], 0
    for line in conllu_path.open(encoding="utf8"):
        line = line.rstrip("\n")
        if line.startswith("#"):
            continue
        if not line:
            if cur:
                sents.append(cur)
                cur, skip_to = [], 0
            continue
        c = line.split("\t")
        if "." in c[0]:
            continue
        if "-" in c[0]:
            skip_to = int(c[0].split("-")[1])
            cur.append(c[1])
            continue
        if skip_to and int(c[0]) <= skip_to:
            continue
        cur.append(c[1])
    if cur:
        sents.append(cur)
    return sents
 def iob_sents(path):
    sents, cur = [], []
    for line in path.open(encoding="utf8"):
        line = line.rstrip("\n")
        if not line.strip():
            if cur:
                sents.append(cur)
                cur = []
            continue
        p = line.split("\t")
        if len(p) < 2:
            p = line.split()
        if len(p) < 2:
            continue
        cur.append((p[0], p[-1]))
    if cur:
        sents.append(cur)
    return sents
 def spans_from_iob(tagged):
    """[(start, end, label)] over the source token indices."""
    spans, i = [], 0
    while i < len(tagged):
        tag = tagged[i][1]
        if tag.startswith("B-"):
            label, j = tag[2:], i + 1
            while j < len(tagged) and tagged[j][1] == f"I-{label}":
                j += 1
            spans.append((i, j, label))
            i = j
        else:
            i += 1
    return spans
 def index_map(src, dst):
    """src index -> dst index for tokens difflib considers equal."""
    out = {}
    matcher = SequenceMatcher(a=src, b=dst, autojunk=False)
    for a, b, size in matcher.get_matching_blocks():
        for k in range(size):
            out[a + k] = b + k
    return out
 def transfer_split(conllu, ner_file):
    """Return (tokens, tags) per sentence on the merged tokenization, plus counters."""
    gold = merged_tokens(conllu)
    ner = iob_sents(ner_file)
    moved, lost = Counter(), Counter()
    result = []
    for i in range(min(len(gold), len(ner))):
        dst = gold[i]
        src = [t for t, _ in ner[i]]
        m = index_map(src, dst)
        tags = ["O"] * len(dst)
        for start, end, label in spans_from_iob(ner[i]):
            idx = [m[k] for k in range(start, end) if k in m]
            contiguous = idx and idx == list(range(idx[0], idx[0] + len(idx)))
            if len(idx) == end - start and contiguous:
                tags[idx[0]] = f"B-{label}"
                for k in idx[1:]:
                    tags[k] = f"I-{label}"
                moved[label] += 1
            else:
                lost[label] += 1
        result.append((dst, tags))
    return result, moved, lost
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--conllu-dir", default="assets/ud",
                    help="directory holding fa_perdt-ud-{split}.conllu")
    ap.add_argument("--ner-dir", default="assets/ud-ner",
                    help="directory holding {split}_with_NER_tag.txt")
    ap.add_argument("--out", default=None, help="write IOB2 files here")
    args = ap.parse_args()
    conllu_dir, nerdir = Path(args.conllu_dir), Path(args.ner_dir)
    for d in (conllu_dir, nerdir):
        if not d.is_dir():
            raise SystemExit(f"not found: {d}")
    out = Path(args.out) if args.out else None
    if out:
        out.mkdir(parents=True, exist_ok=True)
    grand_moved, grand_lost = Counter(), Counter()
    for split in SPLITS:
        sents, moved, lost = transfer_split(
            conllu_dir / f"fa_perdt-ud-{split}.conllu",
            nerdir / f"{split}_with_NER_tag.txt",
        )
        m, dropped = sum(moved.values()), sum(lost.values())
        grand_moved += moved
        grand_lost += lost
        print(f"{split:<6} entities {m + dropped:>6}  transferred {m:>6}"
              f" ({100 * m / max(m + dropped, 1):.2f}%)  dropped {dropped}")
        if out:
            path = out / f"{split}.txt"
            with path.open("w", encoding="utf8") as fh:
                for toks, tags in sents:
                    for tok, tag in zip(toks, tags):
                        fh.write(f"{tok}\t{tag}\n")
                    fh.write("\n")
            print(f"       wrote {path}")
    print("\nper label:")
    for label in sorted(grand_moved | grand_lost, key=lambda k: -grand_moved[k]):
        m, dropped = grand_moved[label], grand_lost[label]
        print(f"  {label:<6} {m:>6}/{m + dropped:<6} ({100 * m / max(m + dropped, 1):5.1f}%)")
 if __name__ == "__main__":
    main()