15 changed files with 716 additions and 1153 deletions
--- a/.gitignore
+++ b/.gitignore
@ -6,12 +6,5 @@ training/
 metrics/
 packages/
 .venv/
 # Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the
 # spaCy CLI in the training venv. See .omp/AGENTS.md.
 .venv-publish/
 __pycache__/
 *.pyc
 # Personal scratch list, not part of the project
 TODO.md
 # Agent/session notes, local only
 .omp/
--- a/.omp/AGENTS.md
+++ b/.omp/AGENTS.md
@ -0,0 +1,79 @@
 # Repo context for agents
 ## Remotes
 | Name | URL | Purpose |
 |---|---|---|
 | `gitea` | `https://gitea.cap.nlogn.ir/fazel/spacy-fa-pipeline.git` | Primary self-hosted Gitea |
 ## Pushing to Gitea
 Credentials come from `pass` — no token in git config or any tracked file.
 **Pass path:** `gitea/token_AllExceptAdmin`
 The credential helper is set locally in `.git/config` (never committed):
 ```
 credential.https://gitea.cap.nlogn.ir.helper=
  !f() { echo username=fazel; echo password=$(pass gitea/token_AllExceptAdmin); }; f
 ```
 **Push command** (bypass proxy — Gitea is on the local network):
 ```bash
 NO_PROXY="*" no_proxy="*" http_proxy="" https_proxy="" HTTP_PROXY="" HTTPS_PROXY="" \
  git push gitea main
 ```
 ### If `pass` returns an empty string
 The store is encrypted to `caci96@gmail.com`, whose secret key lives on an **OpenPGP
 smartcard** (`gpg --list-secret-keys --with-colons` shows serial `D276000124010304…`). When
 gpg-agent has no cached session and no pinentry can reach a terminal, `pass` blocks for ~60s
 and exits with empty stdout — it does *not* error. A credential helper wired to it then fails
 auth for no visible reason.
 Diagnose, do not guess:
 ```bash
 export GPG_TTY=$(tty)
 gpg --batch --pinentry-mode error --decrypt ~/.password-store/gitea/token_AllExceptAdmin.gpg
 ```
 Success means the agent is unlocked and `pass` will work. `Bad passphrase`/`No secret key`
 means insert the smartcard, or `export GPG_TTY=$(tty)` and re-run `pass` once interactively to
 satisfy pinentry.
 The same token is mirrored in `rbw` at `api/gitea_token_AllExceptAdmin` (verified
 byte-identical). It needs no smartcard, so it is the fallback when the card is unavailable:
 ```
 !f() { echo username=fazel; echo password=$(rbw get "api/gitea_token_AllExceptAdmin" | head -1); }; f
 ```
 To rotate: update in the Gitea UI, then `pass edit gitea/token_AllExceptAdmin` **and**
 `rbw edit "api/gitea_token_AllExceptAdmin"` so the two stay in sync. No git config change.
 ## What is NOT committed
 `.gitignore` excludes `assets/ corpus/ training/ metrics/ packages/ .venv/`. The repo holds
 only source: configs, scripts, `project.yml`, docs. Everything else is regenerated:
 ```bash
 python -m venv .venv && .venv/bin/python -m pip install -r requirements.txt
 .venv/bin/python -m spacy project assets    # checksummed downloads
 .venv/bin/python -m spacy project run all   # ~2h on 4 CPU cores
 ```
 The trained wheel (13 MB) is a build artifact, not source. Publish it to the HF Hub instead —
 see `docs/CONTRIBUTING-GUIDE.md` §2.
 ## Environment
 The venv is Python 3.12.2, created from the conda env at `/home/fazel/anaconda3/envs/p12`
 (the anaconda base python is 3.7 and cannot run spaCy 3.8). The IPython kernel available to
 agents is bound to that 3.7 base and will fail on this project — shell out to `.venv/bin/python`.
 GPU (GTX 940MX, 2 GB) is unused: too small for a transformer, not worth the transfer overhead
 for an `sm` pipeline. All training is CPU, `--gpu-id -1`.
--- a/53
+++ b/53
@ -1,53 +0,0 @@
 This repository contains two kinds of material under two different licences.
 1. SOURCE CODE (scripts/, configs/, project.yml, docs/) — MIT, below.
 2. TRAINED PIPELINES (fa_dep_news_sm, fa_core_news_sm, fa_ent_news_sm, and any
   artifact under training/ or packages/) — Creative Commons Attribution-ShareAlike
   4.0 International (CC BY-SA 4.0).
   These are Adapted Material derived from UD_Persian-PerDT (PerUDT v1.0), which is
   licensed CC BY-SA 4.0, so the ShareAlike condition requires the same licence.
     Source:  https://github.com/UniversalDependencies/UD_Persian-PerDT
     Licence: https://creativecommons.org/licenses/by-sa/4.0/
     Authors: Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi,
              Alireza Nourian
   The entity annotations additionally derive from that treebank's
   not-to-release/Dadegan with NER tag/ layer, which its README states was produced
   with the Beheshti-NER tagger (Taher, Hoseini, Shamsfard 2020) plus manual
   corrections.
   Modifications made to the licensed material: CoNLL-U converted to spaCy DocBin
   with multiword tokens merged (`spacy convert --merge-subtokens`); entity spans
   realigned onto that tokenization by difflib, with spans that could not be aligned
   exactly dropped rather than guessed. See docs/MODELS.md.
   The licensed material is provided as-is, without warranties of any kind, and the
   licensor disclaims liability for damages arising from its use, to the extent
   permitted by CC BY-SA 4.0 sections 5(a) and 5(b).
 -----------------------------------------------------------------------------------
 MIT License
 Copyright (c) 2026 Kiyarash Fazeli
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
--- a/README.md
+++ b/README.md
@ -1,143 +1,108 @@
-# fa_core_news_sm and fa_dep_news_sm, Persian pipelines for spaCy
+# fa_core_news_sm — a Persian pipeline for spaCy
-spaCy has no trained Persian pipeline. `spacy.load("fa_core_news_sm")` has never worked, and
+There is no trained Persian pipeline for spaCy. `spacy.load("fa_core_news_sm")` has never
-`spacy.blank("fa")` gives you a tokenizer and stop words. This project trains one from
+worked; `spacy.blank("fa")` gives you a tokenizer and stop words and nothing else. This
-openly-licensed data so the result can be redistributed.
+project builds the missing pipeline from openly-licensed data, using spaCy's own tooling, so
 the result can actually be redistributed.
- Pipeline inventory and source analysis: [`docs/MODELS.md`](docs/MODELS.md)
+- **What the four English pipelines are, and what the four Persian equivalents should be:**
- How spaCy models get published, and what upstream `fa` already has:
+  [`docs/MODELS.md`](docs/MODELS.md)
-  [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
+- **How models get contributed/published in the spaCy ecosystem, and what upstream `fa`
- The build: [`project.yml`](project.yml)
+  already has:** [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
 - **The build itself:** [`project.yml`](project.yml)
-## Two packages, one corpus
+## The short version
-In spaCy's naming scheme `dep` = tagger + parser + lemmatizer, `core` = the same plus NER.
+| | |
-Both packages here are built entirely from UD_Persian-PerDT and differ only in whether NER is
+| --- | --- |
-included.
+| Pipeline | `fa_core_news_sm` — tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner |
 | Syntax/morphology data | [UD_Persian-PerDT](https://github.com/UniversalDependencies/UD_Persian-PerDT) (PerUDT v1.0) — 29,107 sentences, **CC BY-SA 4.0** |
 | NER data | [ParsTwiNER](https://github.com/overfit-ir/parstwiner) — 7,667 tweets, **MIT** |
 | Language data | `spacy/lang/fa` upstream (its stop word list comes from hazm) |
 | Licence of the result | CC BY-SA 4.0 (inherited from the treebank) |
 | Hardware | 4-core CPU. No GPU needed. |
-| Package | Components | Licence | Score | Wheel |
+### Results
 | --- | --- | --- | --- | --- |
 | `fa_dep_news_sm` | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | CC BY-SA 4.0 | LAS 85.15, LEMMA 97.91 | 7.5 MB |
 | `fa_core_news_sm` | the above plus ner | CC BY-SA 4.0 | LAS 85.15, ENTS_F 71.87 | 13 MB |
-The NER is possible because the treebank ships its own entity layer in
+Trained and evaluated on this laptop (4-core i5-7200U, CPU only, 1h27m for the UD
-`not-to-release/Dadegan with NER tag/`: 15,833 entities over the same 29,107 sentences, under
+components, ~25 min for NER). Scores are on the **held-out test splits**, produced by
-the same CC BY-SA 4.0. That is what makes `core` honest here, since one corpus means one genre,
+`spacy benchmark accuracy` and stored in `metrics/`.
 one tokenization, one licence and one provenance chain. The alternative NER corpora are all
 worse on at least one of those axes: ARMAN, PEYMA and NSURL are research-use-only, and
 ParsTwiNER (MIT) is a Twitter corpus that costs about 23 F on prose.
-Two caveats to know before relying on the entities:
+| Metric | `fa_core_news_sm` | reference |
 - **The labels are silver.** The treebank README states they came from the BERT-based
  Beheshti-NER tagger with manual corrections for recall, so `ENTS_F 71.87` is measured against
  a silver test split and partly reflects agreement with that tagger.
 - **Three labels are thin.** `MON` (205 training examples), `TIM` (135) and `PCT` (121) score
  73.7, 66.7 and 57.1. `PER`, `LOC`, `ORG` and `DAT` have 1,300 or more each.
 Entity spans were transferred onto this pipeline's tokenization by difflib alignment at a 99.86%
 rate; spans that could not be aligned exactly were dropped rather than guessed
 (`scripts/transfer_perdt_ner.py`).
 Language data comes from `spacy/lang/fa` upstream, whose stop word list came from hazm.
 Everything trains on 4 CPU cores with no GPU.
 ## Results
 Held-out test splits, from `spacy benchmark accuracy`, stored in `metrics/`. Trained on a
 4-core i5-7200U: 1h27m for the UD components, 17 min for NER.
 Syntax and morphology, identical in both packages since they share the same trained components:
 | Metric | Score | Reference |
 | --- | --- | --- |
 | `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | |
-| `TAG_ACC` (XPOS) | 95.96 | |
+| `TAG_ACC` (XPOS) | **95.96** | |
-| `POS_ACC` (UPOS) | 96.24 | |
+| `POS_ACC` (UPOS) | **96.24** | |
 | `MORPH_ACC` | 96.29 | |
-| `LEMMA_ACC` | 97.91 | |
+| `LEMMA_ACC` | **97.91** | |
 | `SENTS_F` | 99.25 | |
-| `DEP_UAS` | 89.69 | hazm+ParsBERT: 92.46 |
+| `DEP_UAS` | **89.69** | hazm+ParsBERT: 92.46 |
-| `DEP_LAS` | 85.15 | hazm+ParsBERT: 89.34 |
+| `DEP_LAS` | **85.15** | hazm+ParsBERT: 89.34 |
-| Speed | ~9,250 words/s | |
+| `ENTS_P` / `ENTS_R` / `ENTS_F` | 74.77 / 61.06 / **67.22** | |
 | Speed | ~9,250 words/s (CPU) | |
 | Wheel size | 13 MB | `en_core_web_sm`: 12 MB |
-Entities, `fa_core_news_sm` only, on the PerDT NER test split: `ENTS_P` 77.67, `ENTS_R` 66.87,
+Per-entity F: `LOC` 73.9, `PER` 69.1, `NAT` 63.2, `ORG` 59.3, `POG` 41.2, `EVE` 30.0.
 `ENTS_F` 71.87. Per label:
-| Label | F | Train examples |
+Read these honestly:
 | --- | --- | --- |
 | `LOC` | 80.24 | 4,954 |
 | `DAT` | 74.45 | 1,323 |
 | `MON` | 73.68 | 205 |
 | `ORG` | 68.77 | 2,643 |
 | `TIM` | 66.67 | 135 |
 | `PER` | 65.29 | 4,847 |
 | `PCT` | 57.14 | 121 |
-Parsing is 4.2 LAS behind hazm's parser, which uses the same corpus and the same spaCy parser
+- **Parsing is 4.2 LAS behind hazm's parser**, which is the expected gap between a 13 MB
-architecture with a fine-tuned ParsBERT instead of hash embeddings. That gap is the target for
+  CPU model with hash embeddings and a fine-tuned ParsBERT. It is the same corpus and the
-a future `trf` tier.
+  same spaCy parser architecture, so the comparison is fair, and it sets the target for the
  future `trf` tier.
 - **NER is the weak component.** 67 F reflects three compounding handicaps: an `sm` model
  with no static vectors, a 233k-token training corpus, and a genre mismatch (trained on
  tweets, most users will run it on prose). `EVE` and `POG` are near-useless. This is the
  price of using the only MIT-licensed Persian NER corpus that exists.
 - **Everything else is competitive with the English `sm` pipeline** (`en_core_web_sm`:
  TAG 97, LAS 90, ENTS_F 84 — on a much larger and cleaner corpus).
-`PER` scoring below `LOC` and `ORG` despite having 4,847 examples is the silver labels showing
+Reproduce: `.venv/bin/python -m spacy project run all`.
 through: PerDT includes titles and honorifics inside `PER` spans inconsistently (6.24% of spans
 start with one, against 1.41% in the human-annotated ParsTwiNER), so the boundaries the model
 has to learn are less regular than the label count suggests.
-For comparison, `en_core_web_sm` scores TAG 97, LAS 90, ENTS_F 84 on a larger, cleaner corpus.
+### Install the built pipeline
 Reproduce with `.venv/bin/python -m spacy project run all`, plus `run ent` for an NER-only
 package.
 ## Install
 ```bash
 .venv/bin/python -m pip install packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl
 # or, without NER:
 .venv/bin/python -m pip install packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl
 ```
 ```python
 import spacy
 nlp = spacy.load("fa_core_news_sm")
-doc = nlp("محمدرضا شجریان در مشهد به دنیا آمد.")
+doc = nlp("دانشگاه تهران در سال ۱۳۱۳ تأسیس شد.")
-print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc][:3])
+print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
-# [('محمدرضا', 'PROPN', 'محمدرضا', 'nsubj'), ('شجریان', 'PROPN', 'شجریان', 'flat:name'), ...]
+print(doc.ents)   # (دانشگاه تهران, ORG)
 print(doc.ents)   # (محمدرضا شجریان, مشهد)  -> PER, LOC
 doc = nlp("شرکت ایران خودرو تولید را ۲۰ درصد افزایش می‌دهد.")
 print([(e.text, e.label_) for e in doc.ents])   # ۲۰ درصد -> PCT
 ```
-## Why not hazm's own models
+### Why not hazm's own models
-hazm is the reference Persian NLP toolkit and publishes spaCy-format pipelines on the HF Hub,
+hazm is the reference Persian NLP toolkit and it *does* publish spaCy-format pipelines on the
-so it was the obvious starting point. Four problems:
+HF Hub, so it was the obvious starting point. It does not survive contact:
- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). The repo contains no
+- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). There is no
-  `config.cfg` and no `spacy train`; the `Spacy*` classes only download pretrained pipelines.
+  `config.cfg` and no `spacy train` anywhere in the repo — the `Spacy*` classes only download
- Those pipelines are three single-task models (`transformer + tagger`, `transformer + parser`,
+  pretrained pipelines.
-  `transformer + chunker`), each `version: 0.0.0` with an empty `license` field, pinned to
+- Those pretrained pipelines are three *single-task* models (`transformer + tagger`,
-  spaCy 3.6. Using all three costs three ParsBERT forward passes and gives no shared `Doc`.
+  `transformer + parser`, `transformer + chunker`), each `version: 0.0.0` with an empty
- Its tokenizer is incompatible with UD tokenization: the normaliser fuses ZWNJ affixes and
+  `license` field, pinned to spaCy 3.6. Using all three means three ParsBERT forward passes
-  `join_verb_parts()` glues multi-word verb chains into single tokens.
+  over the same text and no shared `Doc`.
- Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) sit behind `peykaregan.ir`
+- Its tokenizer is deliberately incompatible with UD tokenization: the normaliser fuses ZWNJ
-  or `dadegan.ir` under research-only terms.
+  affixes and `join_verb_parts()` glues multi-word verb chains into single tokens.
 - Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) are gated behind
  `peykaregan.ir` / `dadegan.ir` under research-only terms.
-It did confirm the corpus choice. hazm's own spaCy parser was trained on
+What hazm *does* give us: confirmation of the corpus choice — hazm's own spaCy parser was
-`modified_fa_perdt-ud-train.spacy`, the same treebank used here. Full analysis in
+trained on `modified_fa_perdt-ud-train.spacy`, i.e. the same treebank we use — plus the stop
 word list already vendored into `spacy/lang/fa`. Full analysis in
 [`docs/MODELS.md`](docs/MODELS.md) §4.
-## Licensing drove most decisions here
+### Why licensing is the load-bearing constraint
-spaCy's maintainers say the Persian models trained in 2018 were never published because of
+spaCy's maintainers state that Persian models trained back in 2018 were never published
-corpus licensing (spaCy discussion #8233, after PR #2797 added `fa` tokenizer support). ARMAN,
+*because of corpus licensing* (spaCy discussion #8233, after PR #2797 added `fa` tokenizer
-PEYMA and NSURL are all research-use-only, and wrapping them in an Apache-2.0 toolkit does not
+support). The standard Persian NER corpora — ARMAN, PEYMA, NSURL — are all "research use
-change that.
+only", and wrapping them in an Apache-2.0 toolkit does not launder that. ParsTwiNER (MIT) is
-
+the only redistributable Persian NER corpus we could verify, which is why the NER component is
-The way out was finding that PerDT ships its own NER layer under the treebank's CC BY-SA 4.0,
+trained on tweets. That trade-off is recorded in the model's `meta.json["notes"]`.
 so the entire pipeline now derives from one corpus with one licence. The 2018 attempt also
 failed for a second reason worth knowing if you plan to publish: honnibal asked for scripts
 that could regenerate the model and got a notebook instead. `project.yml` is that script.
 ## Setup
@ -150,71 +115,50 @@ python -m venv .venv
 ## Build
-[`project.yml`](project.yml) has two workflows:
+Everything is driven by [`project.yml`](project.yml):
 ```bash
 .venv/bin/python -m spacy project assets      # download + checksum the corpora
-.venv/bin/python -m spacy project run all     # -> fa_dep_news_sm + fa_core_news_sm
+.venv/bin/python -m spacy project run all     # inspect -> convert -> train -> assemble -> evaluate -> package
 .venv/bin/python -m spacy project run ent     # -> fa_ent_news_sm, NER alone
 ```
 Individual steps:
 | Command | What it does |
 | --- | --- |
 | `inspect` | annotation coverage of the treebanks (`scripts/inspect_treebanks.py`) |
-| `convert-ud` | CoNLL-U to `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report |
+| `convert-ud` | CoNLL-U → `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report |
-| `transfer-ner` | align PerDT's NER layer onto that tokenization by difflib (`scripts/transfer_perdt_ner.py`) |
+| `convert-ner` | unpack ParsTwiNER, IOB2 → `DocBin` |
-| `convert-ner` | transferred IOB2 to `DocBin` |
+| `debug-data` | `spacy debug data` on both corpora before spending CPU |
-| `debug-data`, `debug-data-ner` | `spacy debug data` on both corpora before spending CPU |
+| `train-core` | tagger + morphologizer + trainable_lemmatizer + parser on PerDT |
-| `train-dep` | tagger + morphologizer + trainable_lemmatizer + parser |
+| `train-ner` | standalone `ner` with its own embedded tok2vec on ParsTwiNER |
-| `train-ner` | the `ner` component, with its own embedded tok2vec |
+| `assemble` | source `ner` into the core pipeline, write full `meta.json` (`scripts/assemble_core.py`) |
-| `finalize-dep` | write `fa_dep_news_sm` metadata: sources, licence, notes (`scripts/finalize_pipeline.py`) |
+| `evaluate` | `spacy benchmark accuracy` on both held-out test sets |
-| `evaluate-dep` | `spacy benchmark accuracy` on the held-out UD test split |
+| `package` | build the wheel + sdist |
-| `assemble-core` | source `ner` into the dep pipeline to produce `fa_core_news_sm` |
+| `smoke` | run the pipeline over real Persian text and print every annotation layer |
 | `evaluate-core` | score the assembled pipeline on both test splits |
 | `finalize-meta` | re-run finalize on both, folding test scores into `meta.json["performance"]` |
 | `package` | build wheels + sdists for both |
 | `smoke` | run both pipelines over Persian text and print every annotation layer |
-The two training runs are single-threaded and independent, so they can run concurrently.
+The two training runs are independent and can run concurrently — each is single-threaded.
-`finalize` runs twice because of an ordering constraint: test scores only exist after
+## Design decisions worth knowing before you touch anything
 evaluation, and evaluation needs a finalized pipeline to score. The second pass only copies
 models. `scripts/finalize_pipeline.py` enforces the shape of each variant, refusing to publish
 a `dep` pipeline that contains `ner` or a `core` one that does not, so the split cannot regress
 unnoticed.
-`--ud-metrics` and `--ner-metrics` are separate flags on purpose. Folding both reports over one
+1. **`--merge-subtokens`.** spaCy has no multiword-token layer, and PerDT splits pronominal
-key set silently corrupted `core`'s metadata during development: the NER corpus has no gold
+   clitics (`پدرم` → `پدر` + `م`). Measured on dev: merging gives token F 0.9887 vs 0.9823 for
-tags, so its report carries `tag_acc: 0.0`, which overwrote the real 95.96.
+   the split version, at the cost of 34 composite XPOS tags on 1.5% of tokens. Merging wins
-
+   because otherwise 1.5% of gold tokens are boundaries the shipped tokenizer can never
-## Design decisions
+   produce. Numbers: `scripts/tokenization_report.py`.
-
+2. **`ner` carries its own tok2vec.** A `Tok2VecListener` only resolves inside the pipeline it
-1. `--merge-subtokens`. spaCy has no multiword-token layer, and PerDT splits pronominal clitics
+   was trained in, so a listener-based component cannot be sourced into another pipeline.
-   (`پدرم` into `پدر` + `م`). Measured on dev, merging gives token F 0.9887 against 0.9823 for
+   `configs/fa_ner_sm.cfg` embeds the tok2vec instead — the same design as `en_core_web_sm`.
-   the split version, costing 34 composite XPOS tags on 1.5% of tokens. Without it, 1.5% of gold
+3. **`morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer.**
-   tokens are boundaries the shipped tokenizer can never produce. See
+   The English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD
-   `scripts/tokenization_report.py`.
+   gives us gold UPOS, FEATS and lemmas, so we train on them and get real `pos_acc`,
-2. `ner` carries its own tok2vec. A `Tok2VecListener` only resolves inside the pipeline it was
+   `morph_acc` and `lemma_acc` numbers instead of unmeasurable rule coverage.
-   trained in, so a listener-based component cannot be sourced elsewhere.
+4. **PerDT, not Seraji.** 3.7× more tokens, and Seraji has no `PROPN` tag at all.
   `configs/fa_ner_sm.cfg` embeds the tok2vec instead, as `en_core_web_sm` does.
 3. `morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer. The
   English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD gives
   gold UPOS, FEATS and lemmas, which yields real `pos_acc`, `morph_acc` and `lemma_acc` numbers
   instead of unmeasurable rule coverage.
 4. PerDT, not Seraji: 3.7x more tokens, and Seraji has no `PROPN` tag.
 ## Roadmap
-1. A human-annotated NER test set, ~500 sentences. PerDT's entity labels and its NER test split
+`md`/`lg` need floret vectors trained on Persian Wikipedia + OSCAR (see
-   are both silver, so `ENTS_F 71.87` is not yet a fact. Tracked in
+`spacy-vectors-builder`); floret rather than classic fastText because Persian's ZWNJ usage is
-   [`../ner_dataset`](../ner_dataset/PLAN.md).
+inconsistent and explodes the surface vocabulary. `trf` needs a rented GPU and should use
-2. A mixed-genre variant. Measured: this prose-trained NER scores 45.72 F on tweets, and mixing
+`HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) rather than ParsBERT, whose model card
-   ParsTwiNER in recovers that to 66.49 for 0.69 F on prose. That belongs in a separate package
+carries no licence. `senter` is one extra training run away. Details in `docs/MODELS.md` §2.
   rather than inside a `news` one.
 3. `md` and `lg` need floret vectors trained on Persian Wikipedia and OSCAR (see
   `spacy-vectors-builder`). Floret rather than classic fastText, because inconsistent ZWNJ
   usage explodes the surface vocabulary.
 4. `trf` needs a rented GPU and should use `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0)
   rather than ParsBERT, whose model card carries no licence.
 5. `senter` is one extra training run.
 6. Upstream PRs to `spacy/lang/fa`, see [`docs/upstream/fa-noun-chunks.md`](docs/upstream/fa-noun-chunks.md).
--- a/configs/fa_core_news_sm.cfg
+++ b/configs/fa_core_news_sm.cfg
@ -1,14 +1,13 @@
-# fa_dep_news_sm — tagger, morphologizer, trainable_lemmatizer, parser.
+# fa_core_news_sm — UD components (tagger, morphologizer, trainable_lemmatizer, parser).
 #
 # Generated with:
-#   spacy init config configs/fa_dep_news_sm.cfg --lang fa \
+#   spacy init config configs/fa_core_news_sm.cfg --lang fa \
 #     --pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency
 #
 # Deviations from the generated defaults:
 #   - eval_frequency 200 -> 400 (dev set is only 146 docs; 200 wastes CPU on a 4-core box)
-#
+# The `ner` component is trained separately (configs/fa_ner_sm.cfg) from a different
-# No `ner` here by design. NER is a separate artifact built from a different corpus by the
+# corpus and merged in by scripts/assemble_core.py.
 # `ner` workflow (configs/fa_ner_sm.cfg -> fa_ent_news_sm); see project.yml for why.
 [paths]
 train = null
--- a/configs/fa_ner_sm.cfg
+++ b/configs/fa_ner_sm.cfg
@ -1,19 +1,15 @@
-# fa_ent_news_sm — Persian NER, CPU size (sm).
+# fa NER component, CPU size (sm).
 #
-# A standalone artifact, not part of the shipping `dep` pipeline. It is trained on
+# Trained on ParsTwiNER (MIT) — a different corpus from the UD treebank that trains
-# ParsTwiNER (MIT, tweets) while the rest of the project trains on UD_Persian-PerDT
+# the rest of fa_core_news_sm, so this is a standalone run whose `ner` component is
-# (CC BY-SA 4.0, edited prose), and it scores 67.22 F against 85-98 for the UD
+# later sourced into the core pipeline by scripts/assemble_core.py.
 # components — different corpus, different genre, different quality tier, so it gets its
 # own package rather than being hidden inside a `core` one. Built by `spacy project run ner`.
 #
 # Deliberate deviation from `spacy init config --pipeline ner`: the tok2vec is
 # EMBEDDED inside components.ner.model instead of being a separate `tok2vec`
 # component with a Tok2VecListener. A listener can only resolve inside the pipeline
 # it was trained in; embedding makes the component self-contained and therefore
-# sourceable into another pipeline via `nlp.add_pipe("ner", source=...)`. This is the same
+# sourceable. This is the same design as en_core_web_sm, whose `ner` "has its own
-# design as en_core_web_sm, whose `ner` "has its own independent internal tok2vec"
+# independent internal tok2vec" (https://spacy.io/models#design).
 # (https://spacy.io/models#design). That is what will let a future fa_core_news_sm combine
 # this component (or its ../ner_dataset replacement) with the dep pipeline.
 [paths]
 train = null
--- a/docs/CONTRIBUTING-GUIDE.md
+++ b/docs/CONTRIBUTING-GUIDE.md
@ -1,85 +1,89 @@
-# Getting a Persian pipeline into the spaCy ecosystem
+# Contribution guidelines: getting a Persian pipeline into the spaCy ecosystem
-Sourced from spaCy's primary docs and repos, URLs inline. Where a contribution goes decides
+Everything below is sourced from spaCy's own primary docs/repos (URLs inline). Read this
-how it has to be shaped, so read this before writing code.
+before writing code, because **where** a contribution goes decides how it must be shaped.
-## 0. Two separate contribution surfaces
+## 0. The one thing to internalise
 spaCy splits Persian support into two *completely different* contribution surfaces:
 | Surface | What it is | Where it lives | How you contribute |
 | --- | --- | --- | --- |
-| Language data (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` |
+| **Language data** (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` |
-| Trained pipeline (`fa_dep_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | You cannot. Publish it yourself (PyPI or HF Hub) and get it listed in spaCy Universe |
+| **Trained pipeline** (`fa_core_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | **You cannot.** Publish it yourself (PyPI / HF Hub) and get it listed in spaCy Universe |
-`spacy/lang/fa` already exists upstream. No trained `fa` pipeline does. So this is a publishing
+`spacy/lang/fa` **already exists upstream**. What does not exist is any trained `fa` pipeline.
-project, with optional upstream PRs for the language-data gaps found along the way.
+So this project is a *publishing* project, not an upstream-PR project — with optional
 upstream PRs for the language-data gaps we find along the way.
 ## 1. Policy, verbatim
 From <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md>:
- No CLA. There is no contributor licence agreement section; contribution is an ordinary
+- **No CLA.** There is no contributor licence agreement section; contribution is an ordinary
  GitHub PR governed by the Contributor Covenant Code of Conduct v1.4.
- Inclusion philosophy:
+- Inclusion philosophy — this is the sentence that decides everything:
  > "Our philosophy is to prefer a smaller core library. […] If you're looking to implement a
  > new spaCy feature, starting with a custom component package is usually the best strategy.
  > […] And if it works well, we can always integrate it into the core library later."
- The sanctioned route for non-trivial additions is the "Publishing spaCy extensions and
+- The only sanctioned route for non-trivial additions is the *Publishing spaCy extensions and
-  plugins" section:
+  plugins* section:
  > "An extension or plugin should add substantial functionality, be well-documented and
  > open-source. It should be available for users to download and install as a Python package
  > – for example via PyPi."
  > "Once your extension is published, you can open a PR to suggest it for the Universe page."
- `CONTRIBUTING.md` never mentions submitting trained models. Neither does
+- `CONTRIBUTING.md` contains **zero** mention of submitting trained models. Neither does
-  <https://github.com/explosion/spacy-models>, whose README only documents `compatibility.json`
+  <https://github.com/explosion/spacy-models> — its README only documents `compatibility.json`
  as "the source of spaCy's internal compatibility check, performed when you run the download
  command". There is no PR template, issue label, or documented process for a third party to
-  land a pipeline in `spacy download`. So the `spacy download` index is an Explosion-only
+  land a pipeline in `spacy download`.
-  release channel. (`[INFERENCE]` from the absence of any process across all three sources.)
+  **Conclusion: the `spacy download` index is an Explosion-only release channel.**
- `https://spacy.io/usage/adding-languages` no longer exists and redirects to
+  (`[INFERENCE]` from the absence of any process across all three primary sources.)
-  <https://spacy.io/usage/linguistic-features#language-data>. The `BaseDefaults` and `Language`
+- `https://spacy.io/usage/adding-languages` no longer exists; it redirects to
-  contract is at <https://spacy.io/api/language>.
+  <https://spacy.io/usage/linguistic-features#language-data>. The `BaseDefaults` /
  `Language` contract is documented at <https://spacy.io/api/language>.
-### Code rules for a `spacy/lang/fa` PR
+### Code rules that apply to a `spacy/lang/fa` PR
 - `black` formatting, `flake8` clean, type hints where practical.
 - Tests go in `spacy/tests/lang/fa/`; regression tests use `@pytest.mark.issue(N)`.
 - Touching `.pyx` means the reviewer must rebuild: `python setup.py build_ext --inplace`.
- Language data must be data and rules only: no model downloads, no network, no heavy deps.
+- Language data must be *pure data + rules*: no model downloads, no network, no heavy deps.
 ## 2. Three publishing routes for the trained pipeline
-1. Hugging Face Hub, using Explosion's own tool
+1. **Hugging Face Hub** — Explosion's own tool, the path of least resistance
   (<https://github.com/explosion/spacy-huggingface-hub>):
   ```bash
   pip install spacy-huggingface-hub
   huggingface-cli login
-   python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build wheel
+   python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build wheel
-   python -m spacy huggingface-hub push packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl --org <org>
+   python -m spacy huggingface-hub push packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl --org <org>
   ```
-   Users then `pip install https://huggingface.co/<org>/fa_dep_news_sm/resolve/main/fa_dep_news_sm-any-py3-none-any.whl`.
+   Users then `pip install https://huggingface.co/<org>/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl`.
-2. PyPI or a self-hosted wheel: `spacy package … --build sdist,wheel` then `twine upload`, or
+2. **PyPI / self-hosted wheel** — `spacy package … --build sdist,wheel` then `twine upload`,
-   attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
+   or attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
-3. spaCy Universe, which lists the package on spacy.io but hosts nothing. Per
+3. **spaCy Universe listing** — lists the package on spacy.io but hosts nothing. Per
   <https://github.com/explosion/spaCy/blob/master/website/UNIVERSE.md>: fork `explosion/spaCy`,
-   append an entry to `website/meta/universe.json`, open a PR. Required fields: `id`, `title`,
+   append an entry to `website/meta/universe.json`, open a PR. Required fields:
-   `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`, `url`, `thumb`,
+   `id`, `title`, `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`,
-   `image`, `author`, `author_links`, `category`, `tags`. The package must be open-source with
+   `url`, `thumb`, `image`, `author`, `author_links`, `category`, `tags`.
-   a user-friendly license and "at least somewhat documented".
+   Checklist: must be **open-source with a user-friendly license** and "at least somewhat documented".
-Route taken here: HF Hub for the artifact, plus a Universe PR once metrics are respectable.
+**Route we take:** HF Hub for the artifact + a Universe PR once metrics are respectable.
-## 3. Naming and versioning
+## 3. Naming and versioning conventions (non-negotiable if you want to look official)
 From <https://spacy.io/models#conventions> and the `spacy-models` README:
 ```
-[lang]_[type]_[genre]_[size]        e.g. fa_dep_news_sm
+[lang]_[type]_[genre]_[size]        e.g. fa_core_news_sm
 ```
 | Slot | Allowed values | Meaning |
 | --- | --- | --- |
 | `type` | `core` | tagger + parser + lemmatizer + NER |
-| | `dep` | tagger + parser + lemmatizer, no NER |
+| | `dep` | tagger + parser + lemmatizer, **no NER** |
 | | `ent` | NER only |
 | | `sent` | sentence segmentation only |
 | `genre` | `news`, `web`, `wiki` | text domain of the training corpus |
@ -88,27 +92,27 @@ From <https://spacy.io/models#conventions> and the `spacy-models` README:
 | | `lg` | ~500k vectors (or 200k floret rows) |
 | | `trf` | transformer, no static vectors |
-Version numbers are `a.b.c` = spaCy-major, spaCy-minor, model-revision. Trained against spaCy
+**Version numbers are `a.b.c` = spaCy-major . spaCy-minor . model-revision.** Trained against
-3.8, so the package version starts at `3.8.0`; retraining on new data bumps `c`. Avoid
+spaCy 3.8 → the package version starts at `3.8.0`. Retraining on new data bumps `c`.
-`0.1.0`-style versions, which is how hazm's HF pipelines ended up with `version: 0.0.0` and
+Do **not** invent `0.1.0`-style versions; that is exactly how hazm's HF pipelines ended up with
-`spacy_version: >=3.6.0,<3.7.0`.
+`version: 0.0.0` and `spacy_version: >=3.6.0,<3.7.0`, which is unusable metadata.
 ### `meta.json` fields that matter (<https://spacy.io/api/data-formats#meta>)
 `lang`, `name`, `version`, `spacy_version` (range), `description`, `author`, `email`, `url`,
-`license`, `sources` (list of `{name, url, author, license}`, which is where treebank
+`license`, **`sources`** (list of `{name, url, author, license}` — this is where treebank
-provenance and its CC BY-SA obligation is recorded), `requirements` (extra pip deps injected
+provenance and its CC BY-SA obligation must be recorded), `requirements` (extra pip deps
-into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance` (auto-filled by
+injected into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance`
-`spacy train`), `speed`, `spacy_git_version`.
+(auto-filled by `spacy train`), `speed`, `spacy_git_version`.
-In v3, `meta.json` "isn't used to construct the language class and pipeline anymore".
+Note: in v3, `meta.json` "isn't used to construct the language class and pipeline anymore" —
-`config.cfg` is the single source of truth for loading; `meta.json` is metadata and packaging.
+`config.cfg` is the single source of truth for loading. `meta.json` is metadata + packaging.
-## 4. Canonical training workflow
+## 4. Canonical training workflow (what Explosion actually runs)
 Template: <https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud>
-(`project.yml` + `configs/default.cfg`). The directory layout `assets / corpus / configs /
+(`project.yml` + `configs/default.cfg`). Directory layout `assets / corpus / configs /
-training / metrics / packages` is the convention, which this repo follows.
+training / metrics / packages` is the convention — this repo follows it.
 ```bash
 # 1. assets: clone the UD treebank
@ -132,55 +136,54 @@ python -m spacy train configs/default.cfg --output training/ --gpu-id -1 \
 # 6. evaluate on held-out test
 python -m spacy benchmark accuracy training/model-best corpus/test.spacy --output metrics/test.json
-#   ('spacy evaluate' is now an alias for 'benchmark accuracy')
+#   ('spacy evaluate' is now just an alias for 'benchmark accuracy')
 # 7. package
-python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build sdist,wheel
+python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build sdist,wheel
 ```
-`spacy assemble` builds a pipeline from a config without training, which suits a rule-only
+`spacy assemble` builds a pipeline from a config **without training** — useful for a
-artifact (tokenizer + lookup lemmatizer + stop words).
+rule-only artifact (tokenizer + lookup lemmatizer + stop words).
-### Flags that matter on `spacy convert`
+### Flags worth knowing on `spacy convert`
- `--converter conllu`, explicit rather than `auto`.
+- `--converter conllu` — explicit is better than `auto`.
- `--n-sents N`, how many sentences per `Doc`. 10 is the Explosion default and gives the parser
+- `--n-sents N` — how many sentences per `Doc`; 10 is the Explosion default and gives the
-  and senter cross-sentence context.
+  parser/senter cross-sentence context.
- `--merge-subtokens`. spaCy has no multi-word-token layer, and Persian UD treebanks use MWT
+- `--merge-subtokens` — spaCy has **no multi-word-token layer**. Persian UD treebanks use MWT
-  ranges for clitics (`پدرم` = `پدر` + `م`), so this flag decides whether gold tokens are
+  ranges for clitics (`پدرم` → `پدر` + `م`), so this flag decides whether gold tokens are
-  clitic-split or fused. See `docs/MODELS.md` §5.
+  clitic-split or fused. See `docs/MODELS.md` for the tokenisation decision.
- `--morphology`, which appends morph features to the tag. Only for pipelines without a
+- `--morphology` — appends morph features to the tag (only if you *aren't* using a morphologizer).
  morphologizer.
-### How the size tiers differ mechanically
+### How the four size tiers differ mechanically
 | Tier | Embedding source | Config difference |
 | --- | --- | --- |
-| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, every other component a `spacy.Tok2VecListener` on the shared `tok2vec` |
+| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, everything else a `spacy.Tok2VecListener` on the shared `tok2vec` |
-| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = <dir built by init vectors>`; `lg` has a bigger table |
+| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = <dir built by init vectors>`; `lg` just has a bigger table |
-| `trf` | contextual | replace `tok2vec` with a `transformer` component; other components listen via `TransformerListener` instead of `Tok2VecListener` |
+| `trf` | contextual | replace `tok2vec` with a `transformer` component; every other component listens via `TransformerListener` instead of `Tok2VecListener` |
-`spacy pretrain` (Tok2Vec LM-style pretraining on raw text into `[initialize] init_tok2vec`) is
+`spacy pretrain` (Tok2Vec LM-style pretraining on raw text → `[initialize] init_tok2vec`) is
 orthogonal to the tier and optional.
-## 5. Upstream `spacy/lang/fa`: current state and gaps
+## 5. Upstream `spacy/lang/fa` — current state and gaps
 Contents of <https://github.com/explosion/spaCy/tree/master/spacy/lang/fa>:
-| File | Size | What it provides |
+| File | Size | What it gives us |
 | --- | --- | --- |
-| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a `lemmatizer` factory defaulting to `mode="rule"` |
+| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a **`lemmatizer` factory defaulting to `mode="rule"`** |
-| `tokenizer_exceptions.py` | 64.9 KB | Generated compound-verb and enclitic exception table |
+| `tokenizer_exceptions.py` | 64.9 KB | Large generated compound-verb / enclitic exception table |
 | `generate_verbs_exc.py` | 14.8 KB | Dev script that generates the above |
-| `stop_words.py` | 3.8 KB | ~500 stop words; the comment says "Stop words from HAZM package" |
+| `stop_words.py` | 3.8 KB | ~500 stop words, comment says **"Stop words from HAZM package"** |
-| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals plus `ام` and `ین` suffixes) |
+| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals + `ام`/`ین` suffixes) |
-| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only, no prefixes and no infixes |
+| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only — **no prefixes, no infixes** |
-| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, which needs a trained parser |
+| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, needs a trained parser to do anything |
-Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer or
+Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer, or
 stop-word tests.
-`spacy-lookups-data` (MIT) already ships Persian rule-lemmatizer tables:
+`spacy-lookups-data` (MIT) **already ships Persian rule-lemmatizer tables**:
 | File | Size |
 | --- | --- |
@ -189,49 +192,48 @@ stop-word tests.
 | `fa_lemma_rules.json` | 882 B |
 | `fa_source.txt` | "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" |
-Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`, and
+Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`,
 `fa_license.txt` (Catalan has one; Persian's Seraji-derived provenance is undocumented).
-Gaps blocking a full `fa_core_news_*`:
+**Gap list for a real `fa_core_news_*`:**
-1. No trained artefacts: no weights, no `config.cfg`, no `meta.json`, no entry in
+1. No trained artefacts at all — no weights, no `config.cfg`, no `meta.json`, no entry in
   `compatibility.json`. `spacy.load("fa_core_news_sm")` fails today.
-2. No word vectors for `fa`, so `md` and `lg` need vectors built from scratch.
+2. No word vectors for `fa` → `md`/`lg` need vectors built from scratch.
-3. No NER data or labels anywhere in `spacy/lang/fa`, so `core` needs an external corpus.
+3. No NER data or labels anywhere in `spacy/lang/fa` → `core` (which implies NER) needs an
-4. The rule lemmatizer needs `token.pos` from a tagger or morphologizer, which is
+   external annotated corpus.
-   chicken-and-egg until the tagger exists. `trainable_lemmatizer` avoids this by learning edit
+4. Rule lemmatizer needs `token.pos` from a tagger/morphologizer → chicken-and-egg until the
-   trees and needing no tables.
+   tagger exists (or use a `trainable_lemmatizer`, which learns edit trees and needs no tables).
-5. No Persian-specific prefix or infix rules. ZWNJ (U+200C) is handled only implicitly through
+5. No Persian-specific prefix/infix rules — ZWNJ (U+200C) is only handled implicitly through
   the verb-exception table.
-6. No tokenizer test coverage for a 65 KB exception table.
+6. Zero tokenizer test coverage for a 65 KB exception table.
-Items 5 and 6 are upstream PR material, along with the `noun_chunks` bug in
+Items 5–7 are legitimate upstream PR material; items 1–4 are this project's job.
 `docs/upstream/fa-noun-chunks.md`. Items 1 to 4 are this project's job.
-## 6. Prior art
+## 6. Prior art we must not duplicate badly
-hazm publishes spaCy-format Persian pipelines on the HF Hub, each a single-task pipeline built
+hazm publishes **spaCy-format** Persian pipelines on the HF Hub (MIT-ish, but `license` field
-on ParsBERT, with the `license` field left empty:
+left empty), each a single-task pipeline built on ParsBERT:
 | HF repo | Pipeline | Reported | Trained on (`[paths]` in its `config.cfg`) |
 | --- | --- | --- | --- |
 | `roshan-research/hazm-parsbert-postagger` | `transformer, tagger` | `tag_acc` 0.9862 | `data_train_98_rs10.spacy` (hazm's own EZ-augmented tagset: `NOUN,EZ`, `ADJ,EZ`, …) |
-| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy`, i.e. UD_Persian-PerDT |
+| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy` — **UD_Persian-PerDT** |
 | `roshan-research/hazm-parsbert-chunker` | `transformer, tagger` (IOB chunk tags) | `tag_acc` 0.9618 | hazm chunk data |
-All three use `spacy-transformers` `TransformerModel.v3` on
+Both use `spacy-transformers` `TransformerModel.v3` on
-`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 and stride 96, with
+`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 / stride 96,
 `spacy_version >= 3.6.0,<3.7.0`.
-Problems this project avoids:
+What they get wrong, and what we fix:
- Three separate pipelines rather than one, so users pay for three BERT forward passes and
+- Three separate pipelines instead of one `core` pipeline → users pay for three BERT forward
-  cannot share a `Doc`.
+  passes and cannot share a `Doc`.
- `version: 0.0.0`, empty `license`, `author` and `sources`, and `name: "pipeline"`, which
+- `version: 0.0.0`, empty `license`/`author`/`sources`, `name: "pipeline"` → violates every
-  breaks the conventions in §3 and leaves redistribution rights unclear.
+  convention in §3 and makes redistribution legally murky.
- Transformer-only, with no CPU-friendly tier.
+- Transformer-only, so no CPU-friendly tier at all.
- No lemmatizer, morphologizer, NER or vectors.
+- No lemmatizer, no morphologizer, no NER, no vectors.
- Pinned to spaCy 3.6, so unusable on 3.8 without retraining.
+- Pinned to spaCy 3.6; unusable on 3.8 without retraining.
-The reusable finding is hazm's parser recipe, UD_Persian-PerDT plus spaCy's
+The valuable, reusable finding: **hazm's own parser recipe is "UD_Persian-PerDT + spaCy
-`TransitionBasedParser`, which independently confirms the corpus choice in `docs/MODELS.md`.
+`TransitionBasedParser`"**, which independently confirms the corpus choice in `docs/MODELS.md`.
--- a/docs/MODELS.md
+++ b/docs/MODELS.md
@ -1,87 +1,70 @@
 # The Persian pipelines: what to build, from what, and why
-## 1. What the four English pipelines are
+## 1. What "the 4 English pipelines" actually are
-One pipeline design at four embedding budgets. Every one has the same components; the axis of
+They are **one pipeline design at four embedding budgets**, not four different products.
-variation is where token representations come from.
+Every one of them has the same components; the only real axis of variation is where token
 representations come from.
 | | `en_core_web_sm` | `en_core_web_md` | `en_core_web_lg` | `en_core_web_trf` |
 | --- | --- | --- | --- | --- |
 | Size on disk | 12 MB | 31 MB | 382 MB | 436 MB |
 | Embeddings | hash embeddings only | 685k keys / 20k vectors (300d) | 685k keys / 343k vectors (300d) | `roberta-base`, 768d contextual |
-| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | transformer, tagger, parser, attribute_ruler, lemmatizer, ner |
+| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | **transformer**, tagger, parser, attribute_ruler, lemmatizer, ner |
 | `TAG_ACC` | 0.97 | 0.97 | 0.97 | 0.98 |
 | `DEP_UAS` / `LAS` | 0.92 / 0.90 | 0.92 / 0.90 | 0.92 / 0.90 | 0.95 / 0.94 |
 | `ENTS_F` | 0.84 | 0.85 | 0.86 | 0.90 |
 | Training data | OntoNotes 5 (+ ClearNLP dep conversion, WordNet 3.0) | + Explosion vectors (OSCAR 2109 + Wikipedia + OpenSubtitles + WMT News Crawl) | same | OntoNotes 5 + roberta-base |
-Static vectors buy nothing measurable for tagging and parsing (identical to two decimals) and
+Read the accuracy table honestly: **static vectors buy almost nothing for tagging and
-1 to 2 F on NER. The transformer buys about 4 LAS and 6 NER F, at 36x the size and a GPU
+parsing** (identical to two decimals) and ~1–2 F on NER. The transformer buys ~4 LAS and
-requirement. That ordering sets the roadmap below.
+~6 NER F, at 36× the size and a GPU requirement. That ordering dictates the roadmap below.
-Source: <https://spacy.io/models/en>.
+Sources: <https://spacy.io/models/en>.
-## 2. Target: the Persian pipelines
+## 2. Target: the four Persian pipelines
-Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>). The
+Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>).
-`type` slot carries real information: `dep` = tagger + parser + lemmatizer, `ent` = NER only,
+`core` = tagger + parser + lemmatizer + NER. Genre is `news`, after the dominant genre of
-`core` = both. Genre is `news`, after the dominant genre of UD_Persian-PerDT, whose README
+UD_Persian-PerDT (its README lists "news fiction nonfiction academic web blog", and spaCy
-lists "news fiction nonfiction academic web blog". spaCy labels comparable treebank-trained
+labels comparable treebank-trained pipelines such as `de_core_news_sm` as `news`).
 pipelines such as `de_core_news_sm` as `news`.
-| Pipeline | Components | Embeddings | Status |
+| Pipeline | Components | Embeddings | Trainable on this hardware? |
 | --- | --- | --- | --- |
-| `fa_dep_news_sm` | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | hash embeddings | built, shipping |
+| **`fa_core_news_sm`** | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner | hash embeddings | **yes — this is what we train now** |
-| `fa_core_news_sm` | the above plus ner | hash embeddings | built, shipping |
+| `fa_core_news_md` | same | floret vectors, 50k rows | yes, but vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) |
-| `fa_ent_news_sm` | ner (own internal tok2vec) | hash embeddings | built, optional |
+| `fa_core_news_lg` | same | floret vectors, 200k rows | same as md, bigger table |
-| `fa_core_web_sm` | same as core, mixed-genre training data | hash embeddings | not built; would add ParsTwiNER to cover social media |
+| `fa_core_news_trf` | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | **no** — 2 GB VRAM (GTX 940MX) cannot fine-tune a 125M-param encoder; needs rented GPU |
 | `fa_core_news_md` | + static vectors | floret, 50k rows | vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) |
 | `fa_core_news_lg` | same | floret, 200k rows | same as md, bigger table |
 | `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | not on this hardware; 2 GB VRAM cannot fine-tune a 125M-param encoder |
-### Why `core` is honest here
+One deviation from the English design, deliberate: Persian gets a **`morphologizer`**
 (UPOS + morphological features) and a **`trainable_lemmatizer`** instead of English's
 `attribute_ruler` + rule `lemmatizer`. Reasons:
-`core` promises that the NER belongs to the same pipeline, built to the same standard and
+- The English pipelines use `attribute_ruler` because OntoNotes gives them PTB `tag`s and
-versioned together. That holds because PerDT ships its own entity layer in
+  they *derive* UPOS from tags by rule. UD treebanks give UPOS and FEATS as gold data —
-`not-to-release/Dadegan with NER tag/`: 15,833 entities over the same 29,107 sentences, under
+  training a morphologizer on them is strictly more information, and Persian morphology
-the same CC BY-SA 4.0, in the same genre, aligned to the same tokenization.
+  (Number, Person, Tense, Mood, Voice, Polarity, PronType) is worth predicting.
 - Persian rule-lemmatizer tables *do* exist in `spacy-lookups-data` (`fa_lemma_exc.json`
  1.68 MB, `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but
  a rule lemmatizer's accuracy is unmeasurable against the corpus it was extracted from and it
  needs `token.pos` to work at all. `trainable_lemmatizer` learns edit trees from PerDT's gold
  lemmas and reports a real `lemma_acc`. PerDT yields **1,908 edit trees** with 100% lemma
  coverage — plenty.
-An earlier revision of this project refused to ship `core`, and was right to given what it knew
+`senter` is intentionally omitted from v1: it is a separately trained component that ships
-then. The only redistributable Persian NER corpus found at that point was ParsTwiNER, a Twitter
+*disabled by default* in the English pipelines, and the parser already produces sentence
-corpus scoring 67.22 F against 85-98 for the UD components, and folding that into one package
+boundaries. Adding it later requires only one extra training run.
 would have hidden a genre and quality gap behind a single name and version number. Finding the
 treebank's own layer removed the objection rather than answering it.
-`fa_dep_news_sm` still ships alongside `core`, for users who want a 7.5 MB syntax-only model or
+## 3. Resource inventory (everything checked for license)
 who would rather not depend on silver entity labels.
-### Why `morphologizer` + `trainable_lemmatizer`
+### 3.1 Treebanks — the tagger / morphologizer / lemmatizer / parser data
-The English pipelines use `attribute_ruler` because OntoNotes gives them PTB tags and they
+| Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN? | Verdict |
 derive UPOS by rule. UD treebanks give UPOS and FEATS as gold data, so a morphologizer trained
 on them has strictly more information, and Persian morphology (Number, Person, Tense, Mood,
 Voice, Polarity, PronType) is worth predicting.
 Persian rule-lemmatizer tables do exist in `spacy-lookups-data` (`fa_lemma_exc.json` 1.68 MB,
 `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but a rule
 lemmatizer's accuracy cannot be measured against the corpus it was extracted from, and it needs
 `token.pos` to run at all. `trainable_lemmatizer` learns edit trees from PerDT's gold lemmas and
 reports a real `lemma_acc`. PerDT yields 1,908 edit trees at 100% lemma coverage.
 `senter` is omitted from v1. It is a separately trained component that ships disabled by default
 in the English pipelines, and the parser already produces sentence boundaries. Adding it later
 takes one training run.
 ## 3. Resource inventory, with licences
 ### 3.1 Treebanks, for tagger / morphologizer / lemmatizer / parser
 | Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN | Verdict |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| UD_Persian-PerDT (PerUDT v1.0) | 29,107 | ~509k | CC BY-SA 4.0 | converted + corrections | converted + corrections | manual native (34 types) | yes | chosen |
+| **UD_Persian-PerDT** (PerUDT v1.0) | 29,107 | ~509k | **CC BY-SA 4.0** | converted + corrections | converted + corrections | manual native (34 types) | **yes** | **CHOSEN** |
-| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | no | secondary, cross-eval |
+| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | **no** | secondary / cross-eval |
-| UD_Persian-PUD | 1,000 | | CC BY-SA 4.0 | | | none | | test-only, parallel corpus |
+| UD_Persian-PUD | 1,000 | — | CC BY-SA 4.0 | — | — | none | — | test-only, parallel corpus |
-| UD_Persian-IPerUDT | tiny | | CC BY-SA 4.0 | | | none | | grammar examples, unusable |
+| UD_Persian-IPerUDT | tiny | — | CC BY-SA 4.0 | — | — | none | — | grammar examples, unusable |
 Measured locally with `scripts/inspect_treebanks.py` (train splits):
@ -91,188 +74,146 @@ fa_perdt-ud-train.conllu         26196    452496   6508      0   100.0    57.7
 fa_seraji-ud-train.conllu         4798    121067   1117      0   100.0    65.0    15    30   39
 ```
-PerDT over Seraji for three reasons. It has 3.7x more training tokens (452k against 121k), and
+Why PerDT over Seraji:
 at `sm` size data is the binding constraint. Seraji has no `PROPN`: proper nouns are tagged
 `NOUN`, which breaks the downstream tasks people use spaCy for. And hazm independently chose
 PerDT for its own spaCy dependency parser, whose `config.cfg` names
 `modified_fa_perdt-ud-train.spacy`, which makes the numbers comparable.
-Seraji's richer manual FEATS and fully manual lemmas make it the natural cross-evaluation set
+1. **3.7× more training tokens** (452k vs 121k). At `sm` size, data is the binding constraint.
-and a candidate for a future concatenated-corpus run. The two use different XPOS inventories,
+2. **Seraji has no `PROPN`** — proper nouns are tagged `NOUN`. A pipeline that cannot mark
-so naive concatenation would corrupt the `tag` label space.
+   proper nouns is crippled for exactly the downstream tasks people use spaCy for.
 3. hazm independently chose PerDT for its own spaCy dependency parser — its
  `config.cfg` names `modified_fa_perdt-ud-train.spacy` as the training set. Converging on
  the same corpus makes our numbers comparable to theirs.
-### 3.2 NER
+Seraji's advantages (richer manual FEATS, fully manual lemmas) are real; it is the natural
 cross-evaluation set and a candidate for a future concatenated-corpus run. The two use
 different XPOS inventories, so naive concatenation would corrupt the `tag` label space.
-| Dataset | Labels | Size | License | Usable |
+### 3.2 NER — the one place with a licensing minefield
 | Dataset | Labels | Size | License | Usable? |
 | --- | --- | --- | --- | --- |
-| PerDT's own NER layer | PER, LOC, ORG, DAT, MON, TIM, PCT | 29,107 sentences / 484k tokens / 15,833 entities | CC BY-SA 4.0, same as the treebank | chosen |
+| **ParsTwiNER** | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens | **MIT** (verified via GitHub API on `overfit-ir/parstwiner`) | **CHOSEN** |
 | ParsTwiNER | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens / 16,250 entities | MIT (verified via GitHub API on `overfit-ir/parstwiner`) | usable, wrong genre |
 | ARMAN (PersianNER) | 6 classes | 250k tokens | academic research only | no |
 | PEYMA | 7 classes | 302k tokens | "free for research purposes", no OSS licence | no |
 | NSURL-2019 Task 7 | PEYMA tagset | ~1M tokens | no explicit licence | no |
 | HooshvareLab merged ParsNER | 10 classes | ARMAN+PEYMA+WikiANN | inherits ARMAN/PEYMA restrictions; HF repo gated (401) | no |
-spaCy's maintainers state that the Persian models trained in 2018 were never published because
+This is not pedantry. **spaCy's own maintainers state that Persian models trained back in
-of corpus licensing (spaCy discussion #8233, following PR #2797, which added only
+2018 were never published precisely because of corpus licensing** (spaCy discussion #8233,
-`spacy.blank("fa")` tokenizer support).
+following PR #2797 which added only `spacy.blank("fa")` tokenizer support). Repeating that
 mistake would waste the whole exercise: a pipeline you cannot legally redistribute is not a
 pipeline, it is a local file.
-The PerDT layer lives in `not-to-release/Dadegan with NER tag/{train,dev,test}_with_NER_tag.txt`
+Consequence to state plainly in the model card: **the NER component is trained on Twitter
-as two-column IOB2. In UD convention `not-to-release/` means "excluded from the official UD
+text while the rest of the pipeline is trained on edited prose.** Expect NER to degrade on
-release build", normally working and source data, and the directory is public on GitHub under
+formal news text relative to what an ARMAN/PEYMA-trained model would score. That is the
-the repo's `LICENSE.txt`. Confirm that reading with the PerDT authors before publishing anything
+price of a redistributable artifact.
 derived from it, since redistribution rights are the whole point.
-Three properties of that layer decide how it gets used:
+### 3.3 Vectors (for md / lg)
 1. **The labels are silver.** The treebank README states they came from the BERT-based
   Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. The
   published `ents_f` is therefore measured against a silver test split and partly reflects
   agreement with that tagger. A human-annotated test set is the outstanding work.
 2. **Tokenization differs.** The NER files use the original Dadegan tokenization, which matches
   the released UD tokenization exactly in only 57 to 62% of sentences: the NER files drop some
   copulas and auxiliaries, and at least one honorific is corrupted (`ص` written as `،`).
   Entities sit on content words present in both, so `scripts/transfer_perdt_ner.py` aligns them
   with difflib, transferring 99.86% of train entities, 99.74% of dev and 99.51% of test. Spans
   whose tokens do not all map contiguously are dropped rather than guessed.
 3. **The label sets do not line up with ParsTwiNER.** PerDT has `DAT`, `MON`, `TIM` and `PCT`;
   ParsTwiNER has `NAT`, `EVE` and `POG`. The intersection is `PER`, `LOC`, `ORG`. Since spaCy
   assigns one label per token, concatenating the two raw would teach the model that dates are
   `O` in half the corpus. Any mixed-genre variant has to solve that first.
 Genre matters more than any of this. Measured with one config on the three shared labels, a
 PerDT-trained NER scores 72.80 F on prose and 45.72 on tweets, while a ParsTwiNER-trained one
 scores 68.59 on tweets and 55.59 on prose. Training on both gives 72.11 and 66.49, so mixing
 costs under 1 F on prose and 2 F on tweets while recovering roughly 20 F off-genre. That is the
 argument for a future `fa_core_web_sm`, and the reason the current `news` packages stay
 prose-only.
 ### 3.3 Vectors, for md and lg
 | Option | License | Note |
 | --- | --- | --- |
-| floret vectors trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | recommended: subword + Bloom-hash embeddings, bounded table size, zero OOV |
+| **floret vectors** trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | **recommended.** Subword + Bloom-hash embeddings: bounded table size, zero OOV |
-| fastText `cc.fa.300` | CC BY-SA 3.0 | fallback; classic word table, large and OOV-prone |
+| fastText `cc.fa.300` | CC BY-SA 3.0 | quick fallback; classic word table, large and OOV-prone |
-Persian surface forms multiply through suffixation and through inconsistent ZWNJ (U+200C)
+Floret is the right call for Persian specifically. Persian surface forms explode through
-usage: the same word appears as `می‌رود`, `میرود` and `می رود` in real text. A classic
+suffixation *and* through inconsistent ZWNJ (U+200C) usage — the same word appears as
-word-vector table misses every variant it did not see, while floret's subword hashing covers
+`می‌رود` / `میرود` / `می رود` in real text. A classic word-vector table has a miss for every
-them. spaCy ships floret vectors for Croatian, Finnish, Korean, Slovenian, Swedish and
+variant; floret's subword hashing covers all of them. spaCy already ships floret vectors for
-Ukrainian for the same reason.
+Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian for the same reason.
-### 3.4 Transformer encoders, for trf
+### 3.4 Transformer encoders (for trf)
 | Model | Arch | License | Note |
 | --- | --- | --- | --- |
-| `HooshvareLab/roberta-fa-zwnj-base` | RoBERTa-base | Apache-2.0 | recommended: licensed, ZWNJ-aware, smallest credible option |
+| **`HooshvareLab/roberta-fa-zwnj-base`** | RoBERTa-base | **Apache-2.0** | recommended: licensed, ZWNJ-aware, smallest of the credible options |
 | `FacebookAI/xlm-roberta-base` | XLM-R base, 278M | MIT | licensed but larger |
 | `PartAI/TookaBERT-Base` | BERT-base | Apache-2.0 | licensed |
 | `m3hrdadfi/albert-fa-base-v2` | ALBERT-base-v2 | Apache-2.0 | licensed, smallest |
-| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | no licence on the card | what hazm used; redistribution risk |
+| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | **no licence on the card** | what hazm used; redistribution risk |
-| `sbunlp/fabert` | BERT-base, 124M | no licence on the card | redistribution risk |
+| `sbunlp/fabert` | BERT-base, 124M | **no licence on the card** | redistribution risk |
-All are BERT, RoBERTa, XLM-R or ALBERT, so all work with `spacy-transformers` and with
+All of these are BERT/RoBERTa/XLM-R/ALBERT, so all are supported by both
-`spacy-curated-transformers`, which supports exactly ALBERT, BERT, CamemBERT, RoBERTa and
+`spacy-transformers` and `spacy-curated-transformers` (the latter supports exactly
-XLM-RoBERTa.
+ALBERT / BERT / CamemBERT / RoBERTa / XLM-RoBERTa).
-hazm's pipelines use ParsBERT, which carries no licence statement. Copying that choice would
+Note the trap: hazm's own pipelines use ParsBERT, which has **no license statement**. Copying
-reintroduce the redistribution problem that stopped the 2018 attempt.
+that choice would reintroduce the redistribution problem that sank the 2018 attempt.
-## 4. What already exists, and why it is not enough
+## 4. What already exists (and why it is not enough)
-No trained spaCy Persian pipeline exists. There are zero `persian`, `farsi` or `fa_` hits in
+- **No trained spaCy Persian pipeline exists.** Zero `persian` / `farsi` / `fa_` hits in
-spaCy's `website/meta/universe.json`, and `spacy.load("fa_core_news_sm")` has never worked.
+  spaCy's `website/meta/universe.json`. `spacy.load("fa_core_news_sm")` has never worked.
 - **hazm** ships three *single-task* spaCy pipelines on the HF Hub:
  `hazm-parsbert-postagger` (`tag_acc` 0.9862, hazm's own EZ-augmented tagset),
  `hazm-bert-dependency-parser` (`dep_uas` 0.9246 / `dep_las` 0.8934, trained on PerDT),
  `hazm-parsbert-chunker` (`tag_acc` 0.9618). Each is `transformer + one component`,
  `version: 0.0.0`, empty `license`/`author`/`sources`, pinned to spaCy 3.6. Using all three
  means three separate BERT forward passes over the same text and no shared `Doc`.
 - **hazm's training code is not reusable.** Its trainable models are pycrfsuite CRFs
  (`hazm/sequence_tagger.py`); there is no `config.cfg` or `spacy train` anywhere in the repo.
  Its `Spacy*` classes only *download* the HF pipelines above.
 - **hazm's tokenizer is actively incompatible** with UD gold tokenization: `Normalizer`'s
  `AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes and `WordTokenizer.join_verb_parts()` glues
  multi-word verb chains into single underscore-joined tokens. Training against PerDT with
  hazm's tokenizer would misalign tokens systematically.
 - **DadmaTools** (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable
  spaCy pipeline package, and its NER wraps ARMAN/PEYMA — the restricted data again.
-hazm ships three single-task spaCy pipelines on the HF Hub: `hazm-parsbert-postagger`
+So what hazm genuinely contributes to this project is **one validated design decision**
-(`tag_acc` 0.9862, hazm's own EZ-augmented tagset), `hazm-bert-dependency-parser` (`dep_uas`
+(PerDT is the corpus) and **the stop-word list already vendored into `spacy/lang/fa`**.
-0.9246, `dep_las` 0.8934, trained on PerDT) and `hazm-parsbert-chunker` (`tag_acc` 0.9618).
+Everything else is built with spaCy-native tooling.
 Each is `transformer + one component`, `version: 0.0.0`, with empty `license`, `author` and
 `sources`, pinned to spaCy 3.6. Using all three costs three BERT forward passes over the same
 text and gives no shared `Doc`.
-hazm's training code is not reusable: its trainable models are pycrfsuite CRFs
+## 5. Decision: train `fa_core_news_sm` first
 (`hazm/sequence_tagger.py`), and the repo contains no `config.cfg` or `spacy train`. Its
 `Spacy*` classes only download the HF pipelines above.
-hazm's tokenizer is incompatible with UD gold tokenization. `Normalizer`'s
+Not md/lg: those need floret vectors trained from scratch on Wikipedia+OSCAR (CPU-days) and
-`AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes, and `WordTokenizer.join_verb_parts()` glues
+buy ~0.00 tag/dep accuracy in the English reference numbers.
-multi-word verb chains into single underscore-joined tokens, so training against PerDT with it
+Not trf: 2 GB of VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait
-would misalign tokens systematically.
+until the CPU pipeline proves the data plumbing is right.
 `sm` is also the tier every other tier is validated against: md/lg/trf reuse the exact same
 corpus conversion, config skeleton and evaluation harness.
-DadmaTools (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable spaCy
+### Tokenization decision, settled with a measurement
 pipeline package, and its NER wraps ARMAN and PEYMA.
 What hazm contributes here is one validated design decision, that PerDT is the corpus, and the
 stop-word list already vendored into `spacy/lang/fa`.
 ## 5. Decision: train the `sm` tier first
 Not md or lg: they need floret vectors trained from scratch on Wikipedia and OSCAR, costing
 CPU-days, and the English reference numbers show no tag or dep accuracy gain. Not trf: 2 GB of
 VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait until the CPU
 pipeline proves the data plumbing. `sm` is also the tier the others are validated against,
 since md, lg and trf reuse the same corpus conversion, config skeleton and evaluation harness.
 ### Tokenization, settled by measurement
 spaCy has no multi-word-token layer, so CoNLL-U MWT ranges (Persian pronominal clitics and
-copulas, `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on
+copulas: `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on
 the PerDT dev set with `scripts/tokenization_report.py`, comparing gold boundaries against
 `spacy.blank("fa")`'s tokenizer:
 | `spacy convert` mode | token P | token R | token F | XPOS types | merge artefacts |
 | --- | --- | --- | --- | --- | --- |
-| `--merge-subtokens` | 0.9860 | 0.9914 | 0.9887 | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space |
+| `--merge-subtokens` | 0.9860 | 0.9914 | **0.9887** | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space |
 | plain (clitics split) | 0.9875 | 0.9772 | 0.9823 | 35 | none |
-Chosen: `--merge-subtokens`, which is also what Explosion's `tagger_parser_ud` template does.
+**Chosen: `--merge-subtokens`** (also what Explosion's `tagger_parser_ud` template does).
-Without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never produce, so
+Rationale: without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never
-those tokens are permanently unlearnable and unpredictable at runtime. With it, every gold
+produce, so those tokens are permanently unlearnable and unpredictable at runtime. With it,
-token is reachable, and the cost is limited to rare composite XPOS tags such as
+every gold token is reachable; the cost is confined to rare composite XPOS tags such as
-`N_IANM_PR_JOPER` (noun + enclitic pronoun) and 1.5% of lemmas that come out as two words.
+`N_IANM_PR_JOPER` (noun + enclitic pronoun) — which are, at least, informative — and 1.5% of
 lemmas that come out as two words.
-The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which is
+The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which
-an upstream PR rather than a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5.
+would be an upstream PR, not a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5.
-### Final composition
+### Final composition of `fa_core_news_sm`
 Shared trained components, in both `fa_dep_news_sm` (7.5 MB) and `fa_core_news_sm` (13 MB),
 both CC BY-SA 4.0:
 | Component | Trained on | Metric | Test score |
 | --- | --- | --- | --- |
-| `tok2vec` | shared, PerDT | | |
+| `tok2vec` | shared, PerDT | — | — |
 | `tagger` (XPOS) | PerDT, 90 labels | `tag_acc` | 95.96 |
 | `morphologizer` (UPOS + FEATS) | PerDT, 298 labels | `pos_acc` / `morph_acc` | 96.24 / 96.29 |
 | `trainable_lemmatizer` | PerDT, 1,908 edit trees | `lemma_acc` | 97.91 |
 | `parser` | PerDT, 34 deprels | `dep_uas` / `dep_las` | 89.69 / 85.15 |
 | `ner` (own internal tok2vec) | ParsTwiNER, 6 labels | `ents_p/r/f` | 74.77 / 61.06 / 67.22 |
-`fa_core_news_sm` adds, and `fa_ent_news_sm` ships alone:
+Training cost on the target hardware (4-core i5-7200U, no GPU): **1h27m** for the UD
 components (early-stopped at step 10,800; best checkpoint step 9,200) and ~25 min for NER
 (best at step 4,000). Both runs are single-threaded, so they were run concurrently.
 Inference: ~9,250 words/s. Wheel: 13 MB.
-| Component | Trained on | Metric | Test score |
+The `--merge-subtokens` artefacts predicted above are visible in the shipped model exactly as
-| --- | --- | --- | --- |
+expected — `کتاب‌هایش` ("his/her books") comes out as one token tagged `N_IANM_PR_JOPER` with
-| `ner` (own internal tok2vec) | PerDT NER layer, 7 labels | `ents_p/r/f` | 77.67 / 66.87 / 71.87 |
+lemma `کتاب او`. Worth knowing before you consume `token.lemma_` downstream.
-Per label, F against training examples: `LOC` 80.24 (4,954), `DAT` 74.45 (1,323), `MON` 73.68
+Sources recorded in `meta.json`, both with their licenses, per
-(205), `ORG` 68.77 (2,643), `TIM` 66.67 (135), `PER` 65.29 (4,847), `PCT` 57.14 (121).
+<https://spacy.io/api/data-formats#meta>. CC BY-SA 4.0 on PerDT means the packaged pipeline
-
+must carry attribution and share-alike notice — handled in `scripts/assemble_core.py`.
 `PER` scoring below `LOC` and `ORG` on nearly the same amount of data is the silver labels
 showing through. PerDT includes titles and honorifics inside `PER` spans inconsistently: 6.24% of
 its `PER` spans start with one (`دکتر`, `مهندس`, `آقای`), against 1.41% in the human-annotated
 ParsTwiNER, so the boundaries are less regular than the count suggests. `MON`, `TIM` and `PCT`
 are thin enough that their scores rest on 4 to 11 test entities each and should be treated as
 indicative only. Persian money, times and percentages are regular enough that an `EntityRuler`
 may beat the statistical model for those three.
 Training cost on a 4-core i5-7200U with no GPU: 1h27m for the UD components (early-stopped at
 step 10,800, best checkpoint step 9,200) and 17 min for NER (best at step 6,000). Both runs are
 single-threaded and can run concurrently. Inference runs at about 9,250 words/s.
 The `--merge-subtokens` artefacts show up in the shipped model as predicted: `کتاب‌هایش`
 ("his/her books") is one token tagged `N_IANM_PR_JOPER` with lemma `کتاب او`. Check this before
 consuming `token.lemma_` downstream.
 Sources are recorded in each `meta.json` with their licences, per
 <https://spacy.io/api/data-formats#meta>, including the treebank's NER layer as its own entry
 crediting Beheshti-NER. CC BY-SA 4.0 on PerDT means every package carries attribution and a
 share-alike notice, handled in `scripts/finalize_pipeline.py`, which also enforces the shape of
 each variant: it refuses to publish a `dep` pipeline containing `ner`, or a `core` one without it.
--- a/docs/upstream/fa-noun-chunks.md
+++ b/docs/upstream/fa-noun-chunks.md
@ -9,20 +9,20 @@ labels = ["nsubj", "dobj", "nsubjpass", "pcomp", "pobj", "dative", "appos", "att
 ```
 `dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They are not
-Universal Dependencies relations, and every Persian treebank is UD (`UD_Persian-PerDT`,
+Universal Dependencies relations, and **every Persian treebank is UD** (`UD_Persian-PerDT`,
-`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline emits
+`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline must
-UD labels, so five of the nine labels are dead code and `doc.noun_chunks` returns bare head
+therefore emit UD labels, so five of the nine labels are dead code and `doc.noun_chunks`
-nouns.
+silently returns bare head nouns.
-`spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py` use UD labels
+Compare `spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py`, which
-(`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their treebanks are UD.
+use UD labels (`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their
-`spacy/lang/de` uses TIGER labels (`sb`, `oa`, `nk`, …) because the German pipelines are
+treebanks are UD too. `spacy/lang/de` legitimately uses TIGER labels (`sb`, `oa`, `nk`, …)
-trained on TIGER. Persian has no equivalent reason.
+because the German pipelines are trained on TIGER. Persian has no such excuse.
-A second, smaller problem: the iterator yields `word.left_edge.i` to `word.i + 1`, so it only
+Second, smaller problem: the iterator yields `word.left_edge.i` → `word.i + 1`, i.e. it only
-expands left. Persian noun phrases expand right through ezafe. `رئیس انجمن جراحان قلب ایران`
+ever expands **left**. Persian noun phrases expand **right** through ezafe:
-("the head of the Iranian society of heart surgeons") is one NP whose head is the leftmost
+`رئیس انجمن جراحان قلب ایران` ("the head of the Iranian society of heart surgeons") is one NP
-token, and left-only expansion truncates it to `رئیس`.
+whose head is the leftmost token. Left-only expansion truncates it to `رئیس`.
 ## Evidence
@ -48,9 +48,9 @@ deprels on NOUN/PROPN/PRON tokens in dev (top 15):
  nsubj:pass         38  -
 ```
-At 1.31 tokens per chunk the shipped iterator is returning single head nouns. The four most
+1.31 tokens per chunk is the tell: the shipped iterator is returning single head nouns.
-common noun-bearing relations after `nsubj`, namely `nmod` (2894), `obl` (1401), `obl:arg`
+`nmod` (2894), `obl` (1401), `obl:arg` (1095) and `obj` (973) — the four most common
-(1095) and `obj` (973), are all unreachable.
+noun-bearing relations after `nsubj` — are all unreachable.
 Live text:
@ -66,8 +66,8 @@ Live text:
 ## Proposed patch
-Swap in UD labels, expand right through modifier chains, and drop a leading `ADP` or `CCONJ`
+Swap in UD labels, expand right through modifier chains, and drop a leading `ADP`/`CCONJ`
-as `fr` and `es` already do:
+exactly as `fr`/`es` already do:
 ```diff
 --- a/spacy/lang/fa/syntax_iterators.py
@ -176,7 +176,7 @@ Add UD-label coverage:
 ```python
 def test_fa_noun_chunks_ezafe(fa_vocab):
-    # رئیس انجمن جراحان, "head of the surgeons' society"
+    # رئیس انجمن جراحان — "head of the surgeons' society"
    words = ["رئیس", "انجمن", "جراحان", "آمد"]
    heads = [3, 0, 1, 3]
    deps = ["nsubj", "nmod", "nmod", "ROOT"]
@ -196,28 +196,28 @@ def test_fa_noun_chunks_drops_leading_adp(fa_vocab):
 ## Other `spacy/lang/fa` gaps found while building this pipeline
-Ordered by how much they cost a Persian pipeline.
+Ordered by how much they cost a real Persian pipeline:
-1. Clitic splitting. The `fa` tokenizer cannot split pronominal enclitics or the enclitic
+1. **Clitic splitting.** The `fa` tokenizer cannot split pronominal enclitics or the enclitic
-   copula (`پدرم` = `پدر` + `م`, `ساکتند` = `ساکت` + `ند`), which UD treebanks annotate as
+   copula (`پدرم` → `پدر` + `م`, `ساکتند` → `ساکت` + `ند`), which UD treebanks annotate as
-   multiword tokens. Measured on PerDT dev, gold-against-tokenizer token F is 0.9823 when
+   multiword tokens. Measured on PerDT dev: gold-vs-tokenizer token F is 0.9823 when clitics
-   clitics are kept split, affecting 1.49% of tokens. Persian-specific `TOKENIZER_SUFFIXES`
+   are kept split, and 1.49% of tokens are affected. Persian-specific `TOKENIZER_SUFFIXES`
-   entries for the enclitic set would remove the need for `--merge-subtokens` and the
+   entries for the enclitic set would remove the need for `--merge-subtokens` and eliminate
-   composite XPOS tags it produces.
+   the composite XPOS tags it produces.
-2. `punctuation.py` defines only `TOKENIZER_SUFFIXES`, with no `TOKENIZER_PREFIXES` and no
+2. **`punctuation.py` defines only `TOKENIZER_SUFFIXES`** — no `TOKENIZER_PREFIXES`, no
   `TOKENIZER_INFIXES`. ZWNJ (U+200C) is handled only implicitly through the 65 KB generated
-   verb-exception table. The missing infix rules break numerics: `spacy/lang/fa/examples.py`
+   verb-exception table. The missing infix rules bite on numerics: `spacy/lang/fa/examples.py`
   ships the sentence `دیروز علی به من ۲۰۰۰.۱﷼ پول نقد داد.`, and the trained pipeline splits
-   `۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, promoting the stray `.` to a sentence boundary, so one
+   `۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, with the stray `.` promoted to a sentence boundary —
-   input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian digits,
+   one input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian
-   but no tokenizer rule keeps a Persian decimal or a currency sign attached.
+   digits, but no tokenizer rule keeps a Persian decimal or a currency sign attached.
-3. No tokenizer tests for `fa`. `spacy/tests/lang/fa/` contains only `test_noun_chunks.py`,
+3. **No tokenizer tests at all** for `fa` — `spacy/tests/lang/fa/` contains only
-   which guards none of the 65 KB exception table.
+   `test_noun_chunks.py`, guarding none of the 65 KB exception table.
-4. `spacy-lookups-data` has no `fa_license.txt`, although `fa_source.txt` records that the
+4. **`spacy-lookups-data` has no `fa_license.txt`**, although `fa_source.txt` records that the
-   lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus",
+   lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" —
-   which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`.
+   which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`; Persian should too.
-5. No `fa_lemma_lookup.json`. Only rule-mode lemmatizer assets exist, so `mode="lookup"` is
+5. **No `fa_lemma_lookup.json`** — only rule-mode lemmatizer assets exist, so
-   unavailable for Persian.
+   `mode="lookup"` is unavailable for Persian.
-Items 1 to 3 are self-contained code PRs. Item 4 is a licence-hygiene PR against
+Items 1–3 are self-contained code PRs. Item 4 is a licence-hygiene PR against
-`spacy-lookups-data`, and affects anyone redistributing a Persian pipeline.
+`spacy-lookups-data` and matters for anyone redistributing a Persian pipeline.
--- a/project.yml
+++ b/project.yml
@ -1,29 +1,19 @@
-title: "fa_core_news_sm / fa_dep_news_sm"
+title: "fa_core_news_sm"
 description: >
-  CPU-sized Persian (fa) pipelines for spaCy 3.8, built entirely from UD_Persian-PerDT
+  A CPU-sized Persian (fa) core pipeline for spaCy 3.8: tagger (XPOS), morphologizer
-  (PerUDT v1.0, CC BY-SA 4.0): tagger (XPOS), morphologizer (UPOS + FEATS), trainable
+  (UPOS + FEATS), trainable lemmatizer, dependency parser and NER.
  lemmatizer, dependency parser and NER.
-  Two shipping packages, same corpus, differing only in whether NER is included:
+  UD components are trained on UD_Persian-PerDT (PerUDT v1.0, CC BY-SA 4.0). The NER
-  `fa_dep_news_sm` (no NER) and `fa_core_news_sm` (with NER). The NER comes from the
+  component is trained separately on ParsTwiNER (MIT) and merged in, because no
-  treebank's own `not-to-release/Dadegan with NER tag/` layer, so both packages share one
+  redistributably-licensed Persian NER corpus shares a genre with the treebank. See
-  corpus, one genre, one tokenization and one licence.
+  docs/MODELS.md for the full source/licence analysis and docs/CONTRIBUTING-GUIDE.md
  for how this gets published.
-  PerDT's NER labels are silver, produced by Beheshti-NER with manual corrections, and are
+  Run everything with: `spacy project run all`
  transferred onto this pipeline's tokenization by difflib at a 99.86% rate. Per-label
  scores are published in meta.json; MON, TIM and PCT are thin.
  See docs/MODELS.md for the source and licence analysis and docs/CONTRIBUTING-GUIDE.md for
  how this gets published.
  Build both with: `spacy project run all`
  Standalone NER-only package: `spacy project run ent`
 vars:
  lang: "fa"
-  dep_package_name: "dep_news_sm"
+  package_name: "core_news_sm"
  core_package_name: "core_news_sm"
  ent_package_name: "ent_news_sm"
  package_version: "3.8.0"
  treebank: "fa_perdt"
  # -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline.
@ -52,42 +42,24 @@ assets:
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-test.conllu"
    checksum: "b62a66994cef2c50f7e524a1471102d8"
    description: "UD_Persian-PerDT test split (CC BY-SA 4.0)"
-  - dest: "assets/ud-ner/train_with_NER_tag.txt"
+  - dest: "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
-    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/train_with_NER_tag.txt"
+    url: "https://github.com/overfit-ir/parstwiner/releases/download/v1.0.0/ParsTwiNER_corpus_v1.0.0.zip"
-    checksum: "ecb96cf99b38bc485cac21d22914e413"
+    checksum: "54615340d76c4d51b8f54751b07a278d"
-    description: "PerDT NER layer, train split, IOB2 (CC BY-SA 4.0)"
+    description: "ParsTwiNER Persian Twitter NER corpus, IOB2 (MIT)"
  - dest: "assets/ud-ner/dev_with_NER_tag.txt"
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/dev_with_NER_tag.txt"
    checksum: "2a56ef7eb2e3732e221317af457d1c09"
    description: "PerDT NER layer, dev split, IOB2 (CC BY-SA 4.0)"
  - dest: "assets/ud-ner/test_with_NER_tag.txt"
    url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/test_with_NER_tag.txt"
    checksum: "6d80dd783527562c2ea5189f218a12b5"
    description: "PerDT NER layer, test split, IOB2 (CC BY-SA 4.0)"
 workflows:
  # Both shipping artifacts: fa_dep_news_sm and fa_core_news_sm.
  all:
    - inspect
    - convert-ud
    - transfer-ner
    - convert-ner
    - debug-data
-    - debug-data-ner
+    - train-core
    - train-dep
    - train-ner
-    - finalize-dep
+    - assemble
-    - evaluate-dep
+    - evaluate
    - assemble-core
    - evaluate-core
    - finalize-meta
    - package
    - smoke
  # Optional third artifact: the NER alone, for users who only want entities.
  ent:
    - finalize-ent
    - evaluate-ent
    - package-ent
 commands:
  - name: "inspect"
@ -119,185 +91,107 @@ commands:
      - "corpus/merged/${vars.treebank}-ud-dev.spacy"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
  - name: "transfer-ner"
    help: >
      Align PerDT's NER layer onto the --merge-subtokens tokenization. The NER files use the
      original Dadegan tokenization, which matches the released UD tokenization in only 57 to
      62% of sentences, so spans are transferred by difflib. Measured rate 99.86%; spans that
      cannot be aligned exactly are dropped rather than guessed.
    script:
      - "python scripts/transfer_perdt_ner.py --conllu-dir assets/ud --ner-dir assets/ud-ner --out corpus/perdt-ner-iob"
    deps:
      - "assets/ud/${vars.treebank}-ud-train.conllu"
      - "assets/ud-ner/train_with_NER_tag.txt"
      - "scripts/transfer_perdt_ner.py"
    outputs:
      - "corpus/perdt-ner-iob/train.txt"
      - "corpus/perdt-ner-iob/dev.txt"
      - "corpus/perdt-ner-iob/test.txt"
  - name: "convert-ner"
-    help: "Transferred IOB2 -> DocBin"
+    help: "Unpack ParsTwiNER and convert its IOB2 files to DocBin"
    script:
-      - "python -m spacy convert corpus/perdt-ner-iob/train.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
+      - "python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner"
-      - "python -m spacy convert corpus/perdt-ner-iob/dev.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
+      - "python -m spacy convert assets/ner/train.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
-      - "python -m spacy convert corpus/perdt-ner-iob/test.txt corpus/perdt-ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
+      - "python -m spacy convert assets/ner/dev.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
      - "python -m spacy convert assets/ner/test.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
    deps:
-      - "corpus/perdt-ner-iob/train.txt"
+      - "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
      - "scripts/extract_parstwiner.py"
    outputs:
-      - "corpus/perdt-ner/train.spacy"
+      - "corpus/ner/train.spacy"
-      - "corpus/perdt-ner/dev.spacy"
+      - "corpus/ner/dev.spacy"
-      - "corpus/perdt-ner/test.spacy"
+      - "corpus/ner/test.spacy"
  - name: "debug-data"
-    help: "Validate the treebank against the config before burning CPU on training"
+    help: "Validate both corpora against their configs before burning CPU on training"
    script:
-      - "python -m spacy debug data configs/fa_dep_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy"
+      - "python -m spacy debug data configs/fa_core_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy"
      - "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy"
    deps:
      - "corpus/merged/${vars.treebank}-ud-train.spacy"
-      - "configs/fa_dep_news_sm.cfg"
+      - "corpus/ner/train.spacy"
-
+      - "configs/fa_core_news_sm.cfg"
  - name: "debug-data-ner"
    help: "Validate the transferred PerDT NER corpus against the NER config"
    script:
      - "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy"
    deps:
      - "corpus/perdt-ner/train.spacy"
      - "configs/fa_ner_sm.cfg"
-  - name: "train-dep"
+  - name: "train-core"
    help: "Train tok2vec + tagger + morphologizer + trainable_lemmatizer + parser on PerDT"
    script:
-      - "python -m spacy train configs/fa_dep_news_sm.cfg --output training/dep --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}"
+      - "python -m spacy train configs/fa_core_news_sm.cfg --output training/core --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}"
    deps:
      - "corpus/merged/${vars.treebank}-ud-train.spacy"
      - "corpus/merged/${vars.treebank}-ud-dev.spacy"
-      - "configs/fa_dep_news_sm.cfg"
+      - "configs/fa_core_news_sm.cfg"
    outputs:
-      - "training/dep/model-best"
+      - "training/core/model-best"
  - name: "train-ner"
-    help: "Train the NER component (own embedded tok2vec) on the transferred PerDT layer"
+    help: "Train the standalone NER component (own internal tok2vec) on ParsTwiNER"
    script:
-      - "python -m spacy train configs/fa_ner_sm.cfg --output training/perdt-ner --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy --gpu-id ${vars.gpu}"
+      - "python -m spacy train configs/fa_ner_sm.cfg --output training/ner --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy --gpu-id ${vars.gpu}"
    deps:
-      - "corpus/perdt-ner/train.spacy"
+      - "corpus/ner/train.spacy"
-      - "corpus/perdt-ner/dev.spacy"
+      - "corpus/ner/dev.spacy"
      - "configs/fa_ner_sm.cfg"
    outputs:
-      - "training/perdt-ner/model-best"
+      - "training/ner/model-best"
-  - name: "finalize-dep"
+  - name: "assemble"
-    help: "Write fa_dep_news_sm metadata (sources, licence, notes) onto the trained model"
+    help: "Source the trained ner into the core pipeline and write full meta.json"
    script:
-      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_dep_news_sm --variant dep --version ${vars.package_version}"
+      - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version}"
    deps:
-      - "training/dep/model-best"
+      - "training/core/model-best"
-      - "scripts/finalize_pipeline.py"
+      - "training/ner/model-best"
      - "scripts/assemble_core.py"
    outputs:
-      - "training/fa_dep_news_sm"
+      - "training/fa_core_news_sm"
-  - name: "evaluate-dep"
+  - name: "evaluate"
-    help: "Score fa_dep_news_sm on the held-out UD test split"
+    help: "Score the assembled pipeline on both held-out test sets"
    script:
-      - "python -m spacy benchmark accuracy training/fa_dep_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}"
+      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}"
      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/ner/test.spacy --output metrics/ner-test.json --gpu-id ${vars.gpu}"
    deps:
-      - "training/fa_dep_news_sm"
+      - "training/fa_core_news_sm"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
      - "corpus/ner/test.spacy"
    outputs:
      - "metrics/ud-test.json"
-
+      - "metrics/ner-test.json"
  - name: "assemble-core"
    help: >
      Source the trained ner into the dep pipeline to produce fa_core_news_sm. Possible
      because configs/fa_ner_sm.cfg embeds its own tok2vec instead of a Tok2VecListener.
    script:
      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_core_news_sm --variant core --version ${vars.package_version} --add-ner training/perdt-ner/model-best"
    deps:
      - "training/dep/model-best"
      - "training/perdt-ner/model-best"
      - "scripts/finalize_pipeline.py"
    outputs:
      - "training/fa_core_news_sm"
  - name: "evaluate-core"
    help: "Score fa_core_news_sm on both held-out test splits"
    script:
      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/core-ud-test.json --gpu-id ${vars.gpu}"
      - "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/perdt-ner/test.spacy --output metrics/perdt-ner-test.json --gpu-id ${vars.gpu}"
    deps:
      - "training/fa_core_news_sm"
      - "corpus/merged/${vars.treebank}-ud-test.spacy"
      - "corpus/perdt-ner/test.spacy"
    outputs:
      - "metrics/core-ud-test.json"
      - "metrics/perdt-ner-test.json"
  - name: "finalize-meta"
    help: >
-      Re-run finalize on both packages, folding test scores into meta.json["performance"].
+      Re-assemble, this time folding the test scores into meta.json["performance"].
-      Separate because the scores only exist after evaluation, and evaluation needs a
+      Separate from `assemble` because the scores can only exist after `evaluate`, and
-      finalized pipeline to score. Cheap: it only copies models.
+      `evaluate` needs an assembled pipeline to score. Cheap: it only copies models.
    script:
-      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_dep_news_sm --variant dep --version ${vars.package_version} --ud-metrics metrics/ud-test.json"
+      - "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version} --ud-metrics metrics/ud-test.json --ner-metrics metrics/ner-test.json"
      - "python scripts/finalize_pipeline.py training/dep/model-best training/fa_core_news_sm --variant core --version ${vars.package_version} --add-ner training/perdt-ner/model-best --ud-metrics metrics/core-ud-test.json --ner-metrics metrics/perdt-ner-test.json"
    deps:
      - "metrics/ud-test.json"
-      - "metrics/perdt-ner-test.json"
+      - "metrics/ner-test.json"
-      - "scripts/finalize_pipeline.py"
+      - "scripts/assemble_core.py"
  - name: "package"
-    help: "Build installable wheels + sdists for both shipping packages"
+    help: "Build the installable wheel + sdist"
    script:
-      - "python -m spacy package training/fa_dep_news_sm packages --name ${vars.dep_package_name} --version ${vars.package_version} --build sdist,wheel --force"
+      - "python -m spacy package training/fa_core_news_sm packages --name ${vars.package_name} --version ${vars.package_version} --build sdist,wheel --force"
      - "python -m spacy package training/fa_core_news_sm packages --name ${vars.core_package_name} --version ${vars.package_version} --build sdist,wheel --force"
    deps:
      - "training/fa_dep_news_sm"
      - "training/fa_core_news_sm"
    outputs:
-      - "packages/${vars.lang}_${vars.dep_package_name}-${vars.package_version}"
+      - "packages/${vars.lang}_${vars.package_name}-${vars.package_version}"
      - "packages/${vars.lang}_${vars.core_package_name}-${vars.package_version}"
  - name: "smoke"
-    help: "Load both pipelines and run them over real Persian text"
+    help: "Load the packaged pipeline and run it over real Persian text"
    script:
      - "python scripts/smoke_test.py training/fa_dep_news_sm"
      - "python scripts/smoke_test.py training/fa_core_news_sm"
    deps:
      - "training/fa_dep_news_sm"
      - "training/fa_core_news_sm"
  - name: "finalize-ent"
    help: "Write fa_ent_news_sm metadata onto the trained NER model"
    script:
      - "python scripts/finalize_pipeline.py training/perdt-ner/model-best training/fa_ent_news_sm --variant ent --version ${vars.package_version}"
    deps:
      - "training/perdt-ner/model-best"
      - "scripts/finalize_pipeline.py"
    outputs:
      - "training/fa_ent_news_sm"
  - name: "evaluate-ent"
    help: "Score fa_ent_news_sm on the held-out PerDT NER test split"
    script:
      - "python -m spacy benchmark accuracy training/fa_ent_news_sm corpus/perdt-ner/test.spacy --output metrics/ent-test.json --gpu-id ${vars.gpu}"
      - "python scripts/finalize_pipeline.py training/perdt-ner/model-best training/fa_ent_news_sm --variant ent --version ${vars.package_version} --ner-metrics metrics/ent-test.json"
    deps:
      - "training/fa_ent_news_sm"
      - "corpus/perdt-ner/test.spacy"
    outputs:
      - "metrics/ent-test.json"
  - name: "package-ent"
    help: "Build the installable fa_ent_news_sm wheel + sdist"
    script:
      - "python -m spacy package training/fa_ent_news_sm packages --name ${vars.ent_package_name} --version ${vars.package_version} --build sdist,wheel --force"
    deps:
      - "training/fa_ent_news_sm"
    outputs:
      - "packages/${vars.lang}_${vars.ent_package_name}-${vars.package_version}"
  - name: "clean"
    help: "Drop corpora, training runs and metrics (keeps downloaded assets)"
    script:
-      - "rm -rf corpus training metrics packages"
+      - "rm -rf corpus/merged corpus/split corpus/ner training metrics packages"
--- a/scripts/assemble_core.py
+++ b/scripts/assemble_core.py
@ -0,0 +1,133 @@
 """Merge the separately-trained NER component into the UD pipeline and write full metadata.
 The two components cannot be trained together: the UD components come from
 UD_Persian-PerDT (CC BY-SA 4.0, edited prose) and `ner` comes from ParsTwiNER (MIT,
 tweets). `ner` was therefore configured with its own embedded tok2vec (see
 configs/fa_ner_sm.cfg) so it can be sourced into another pipeline without a dangling
 Tok2VecListener — the same design as en_core_web_sm.
 Run it twice: once right after training (no metrics yet), and again after
 `spacy benchmark accuracy` has produced metrics/*.json so `meta.json["performance"]`
 reflects the assembled pipeline on the held-out test sets.
 Usage:
  .venv/bin/python scripts/assemble_core.py training/core/model-best training/ner/model-best \\
      training/fa_core_news_sm --version 3.8.0 \\
      [--ud-metrics metrics/ud-test.json] [--ner-metrics metrics/ner-test.json]
 """
 import argparse
 import json
 from pathlib import Path
 import spacy
 DESCRIPTION = (
    "Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, "
    "trainable_lemmatizer, parser, ner."
 )
 SOURCES = [
    {
        "name": "UD_Persian-PerDT (PerUDT v1.0)",
        "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
        "author": (
            "Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian"
        ),
        "license": "CC BY-SA 4.0",
    },
    {
        "name": "ParsTwiNER",
        "url": "https://github.com/overfit-ir/parstwiner",
        "author": "MohammadMahdi Aghajani, AliAkbar Badri, Hamid Beigy et al. (Overfit-IR)",
        "license": "MIT",
    },
    {
        "name": "spaCy lang/fa language data (stop words originally from HAZM)",
        "url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa",
        "author": "Explosion and spaCy contributors",
        "license": "MIT",
    },
 ]
 # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
 NOTES = (
    "The tagger, morphologizer, trainable_lemmatizer and parser are trained on "
    "UD_Persian-PerDT, which is licensed CC BY-SA 4.0; this pipeline is therefore "
    "distributed under CC BY-SA 4.0 with attribution to the treebank authors. "
    "The ner component is trained on ParsTwiNER (MIT), a Twitter corpus, so entity "
    "recognition is weaker on formal/edited prose than on social media text. "
    "Multiword tokens in the treebank (pronominal clitics, enclitic copulas) were merged "
    "with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite "
    "(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space."
 )
 def load_metrics(path):
    if not path:
        return {}
    p = Path(path)
    if not p.exists():
        print(f"  (no metrics at {p}, skipping)")
        return {}
    return json.loads(p.read_text(encoding="utf8"))
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("core", help="trained UD pipeline (training/core/model-best)")
    ap.add_argument("ner", help="trained NER pipeline (training/ner/model-best)")
    ap.add_argument("output", help="destination directory for the assembled pipeline")
    ap.add_argument("--version", default="3.8.0")
    ap.add_argument("--ud-metrics", default=None)
    ap.add_argument("--ner-metrics", default=None)
    args = ap.parse_args()
    nlp = spacy.load(args.core)
    ner_nlp = spacy.load(args.ner)
    if "ner" in nlp.pipe_names:
        nlp.remove_pipe("ner")
    nlp.add_pipe("ner", source=ner_nlp)
    print(f"pipeline: {nlp.pipe_names}")
    print(f"ner labels: {sorted(nlp.get_pipe('ner').labels)}")
    ud = load_metrics(args.ud_metrics)
    ner = load_metrics(args.ner_metrics)
    performance = dict(nlp.meta.get("performance", {}))
    for key in ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc",
                "morph_acc", "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r",
                "sents_f"):
        if key in ud:
            performance[key] = ud[key]
    for key in ("ents_p", "ents_r", "ents_f"):
        if key in ner:
            performance[key] = ner[key]
    if "ents_per_type" in ner:
        performance["ents_per_type"] = ner["ents_per_type"]
    if "dep_las_per_type" in ud:
        performance["dep_las_per_type"] = ud["dep_las_per_type"]
    nlp.meta.update(
        {
            "lang": "fa",
            "name": "core_news_sm",
            "version": args.version,
            "description": DESCRIPTION,
            "author": "",
            "email": "",
            "url": "",
            "license": "CC BY-SA 4.0",
            "sources": SOURCES,
            "notes": NOTES,
            "performance": performance,
        }
    )
    out = Path(args.output)
    nlp.to_disk(out)
    print(f"wrote {out}")
    print(json.dumps(performance, indent=2, ensure_ascii=False)[:1200])
 if __name__ == "__main__":
    main()
--- a/scripts/extract_parstwiner.py
+++ b/scripts/extract_parstwiner.py
@ -0,0 +1,34 @@
 """Unpack the ParsTwiNER release zip into flat IOB2 files.
 The archive carries macOS resource-fork junk (`__MACOSX/._*`) that would confuse
 `spacy convert`, so only the three real splits are extracted.
 Usage: .venv/bin/python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner
 """
 import sys
 import zipfile
 from pathlib import Path
 SPLITS = ("train.txt", "dev.txt", "test.txt")
 def main():
    archive = Path(sys.argv[1])
    dest = Path(sys.argv[2])
    dest.mkdir(parents=True, exist_ok=True)
    with zipfile.ZipFile(archive) as zf:
        names = set(zf.namelist())
        missing = [s for s in SPLITS if s not in names]
        if missing:
            raise SystemExit(f"{archive}: missing expected members {missing}")
        for split in SPLITS:
            zf.extract(split, dest)
            path = dest / split
            sents = path.read_text(encoding="utf8").strip().split("\n\n")
            tokens = sum(1 for line in path.open(encoding="utf8") if line.strip())
            print(f"{path}: {len(sents)} sentences, {tokens} tokens")
 if __name__ == "__main__":
    main()
--- a/scripts/finalize_pipeline.py
+++ b/scripts/finalize_pipeline.py
@ -1,226 +0,0 @@
 """Write complete, convention-compliant metadata onto a trained pipeline.
 Three variants, following spaCy's `[lang]_[type]_[genre]_[size]` naming
 (https://spacy.io/models#conventions):
  dep  -> fa_dep_news_sm    tagger + morphologizer + trainable_lemmatizer + parser
  core -> fa_core_news_sm   the above plus ner
  ent  -> fa_ent_news_sm    ner only
 All three are built from UD_Persian-PerDT alone, including the NER, which comes from that
 treebank's own `not-to-release/Dadegan with NER tag/` layer. That is what makes `core`
 honest here: one corpus, one genre, one licence, one provenance chain. An earlier version of
 this script refused to emit `core` because the only redistributable Persian NER corpus we
 knew of was ParsTwiNER, a Twitter corpus scoring 67.22 F against 85-98 for the UD
 components; merging those into one package would have hidden a genre and quality gap behind
 a single name and version.
 PerDT's NER labels are silver, produced by Beheshti-NER (Taher et al., 2020) with manual
 corrections, so `notes` says so and the per-label scores are published as measured.
 Run twice: once after training (no metrics yet), then again after
 `spacy benchmark accuracy` so meta.json["performance"] carries held-out test scores.
 Usage:
  .venv/bin/python scripts/finalize_pipeline.py training/core/model-best training/fa_dep_news_sm \\
      --variant dep --version 3.8.0 [--ud-metrics metrics/ud-test.json]
  .venv/bin/python scripts/finalize_pipeline.py training/fa_core_news_sm training/fa_core_news_sm \\
      --variant core --version 3.8.0 --ud-metrics metrics/core-ud-test.json --ner-metrics metrics/perdt-ner-test.json
 """
 import argparse
 import json
 from pathlib import Path
 import spacy
 AUTHOR = "Kiyarash Fazeli"
 EMAIL = "kiyarash@nlogn.ir"
 PROJECT_URL = "https://github.com/Fazel94/spacy-persian"
 PERDT = {
    "name": "UD_Persian-PerDT (PerUDT v1.0)",
    "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
    "author": "Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian",
    "license": "CC BY-SA 4.0",
 }
 PERDT_NER = {
    "name": "UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)",
    "url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
    "author": "PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020)",
    "license": "CC BY-SA 4.0",
 }
 LANG_DATA = {
    "name": "spaCy lang/fa language data (stop words originally from HAZM)",
    "url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa",
    "author": "Explosion and spaCy contributors",
    "license": "MIT",
 }
 NER_NOTE = (
    "The ner component is trained on the NER layer shipped in UD_Persian-PerDT's "
    "not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. "
    "Those labels are SILVER: the treebank README states they were produced by the BERT-based "
    "Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They "
    "were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% "
    "transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly "
    "were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more "
    "training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in "
    "`performance.ents_per_type` should be read before relying on them."
 )
 MWT_NOTE = (
    "Multiword tokens (pronominal clitics, enclitic copulas) were merged with "
    "`spacy convert --merge-subtokens`, so a small number of XPOS tags are composite "
    "(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space."
 )
 CHUNK_NOTE = (
    "doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches "
    "ClearNLP labels that do not exist in Universal Dependencies, see "
    "docs/upstream/fa-noun-chunks.md."
 )
 # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
 PERDT_LICENSE = "CC BY-SA 4.0"
 ATTRIBUTION = (
    "Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore "
    "distributed under CC BY-SA 4.0 with attribution to the treebank authors."
 )
 UD_KEYS = ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc", "morph_acc",
           "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r", "sents_f",
           "dep_las_per_type")
 NER_KEYS = ("ents_p", "ents_r", "ents_f", "ents_per_type")
 VARIANTS = {
    "dep": {
        "name": "dep_news_sm",
        "description": (
            "Persian dependency pipeline optimized for CPU. Components: tok2vec, tagger, "
            "morphologizer, trainable_lemmatizer, parser. No NER, see fa_core_news_sm."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, MWT_NOTE, CHUNK_NOTE]),
        "keys": UD_KEYS,
        "require": lambda pipes: "ner" not in pipes,
        "require_msg": "a 'dep' pipeline must not contain an ner component",
    },
    "core": {
        "name": "core_news_sm",
        "description": (
            "Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, "
            "trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, "
            "PCT."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT, PERDT_NER, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, NER_NOTE, MWT_NOTE, CHUNK_NOTE]),
        "keys": UD_KEYS + NER_KEYS,
        "require": lambda pipes: "ner" in pipes and "parser" in pipes,
        "require_msg": "a 'core' pipeline must contain both parser and ner",
    },
    "ent": {
        "name": "ent_news_sm",
        "description": (
            "Persian named entity recognizer optimized for CPU, with its own internal "
            "tok2vec. Labels: PER, LOC, ORG, DAT, MON, TIM, PCT."
        ),
        "license": PERDT_LICENSE,
        "sources": [PERDT_NER, LANG_DATA],
        "notes": " ".join([ATTRIBUTION, NER_NOTE]),
        "keys": ("token_acc",) + NER_KEYS,
        "require": lambda pipes: pipes == ["ner"],
        "require_msg": "an 'ent' pipeline must be exactly ['ner']",
    },
 }
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("model", help="trained pipeline, e.g. training/core/model-best")
    ap.add_argument("output", help="destination directory")
    ap.add_argument("--variant", choices=sorted(VARIANTS), required=True)
    ap.add_argument("--version", default="3.8.0")
    ap.add_argument("--ud-metrics", default=None,
                    help="benchmark accuracy JSON scored on the UD test split; supplies the "
                         "tagger/morph/lemma/parser keys only")
    ap.add_argument("--ner-metrics", default=None,
                    help="benchmark accuracy JSON scored on the NER test split; supplies the "
                         "ents_* keys only")
    ap.add_argument("--add-ner", default=None,
                    help="source the ner component from this trained pipeline first "
                         "(used to assemble 'core' from the dep model plus the ner model)")
    args = ap.parse_args()
    spec = VARIANTS[args.variant]
    nlp = spacy.load(args.model)
    if args.add_ner:
        ner_nlp = spacy.load(args.add_ner)
        if ner_nlp.pipe_names != ["ner"]:
            raise SystemExit(f"--add-ner expects a ['ner'] pipeline, got {ner_nlp.pipe_names}")
        if "ner" in nlp.pipe_names:
            nlp.remove_pipe("ner")
        # `ner` embeds its own tok2vec (configs/fa_ner_sm.cfg), so sourcing it here leaves no
        # dangling Tok2VecListener. See docs/MODELS.md.
        nlp.add_pipe("ner", source=ner_nlp)
        print(f"sourced ner from {args.add_ner}: {sorted(nlp.get_pipe('ner').labels)}")
    print(f"pipeline: {nlp.pipe_names}")
    if not spec["require"](list(nlp.pipe_names)):
        raise SystemExit(
            f"refusing to publish as '{args.variant}': {spec['require_msg']}"
            f" (got {list(nlp.pipe_names)})"
        )
    # Build from a strict whitelist rather than inheriting nlp.meta["performance"], which
    # after training also carries raw *_loss values and scorer keys that do not apply to
    # this pipeline (e.g. tag_micro_f: 0.0).
    #
    # Each metrics file supplies only the keys its corpus can actually evidence. Folding both
    # files over the same key set silently corrupted core's metadata: the NER corpus has no
    # gold tags, so its report carries tag_acc: 0.0, which overwrote the real 95.96, and its
    # --n-sents grouping produced a misleading sents_f.
    trained = nlp.meta.get("performance", {})
    performance = {k: trained[k] for k in spec["keys"] if trained.get(k) is not None}
    def fold(path, keys, label):
        if not path:
            return
        p = Path(path)
        if not p.exists():
            print(f"  (no {label} metrics at {p}, skipping; run the evaluate step first)")
            return
        scored = json.loads(p.read_text(encoding="utf8"))
        taken = [k for k in keys if k in spec["keys"] and scored.get(k) is not None]
        for key in taken:
            performance[key] = scored[key]
        if scored.get("speed") is not None:
            performance.setdefault("speed", scored["speed"])
        print(f"  folded {len(taken)} {label} keys from {p}")
    fold(args.ud_metrics, UD_KEYS + ("token_acc",), "UD")
    fold(args.ner_metrics, NER_KEYS, "NER")
    nlp.meta.update(
        {
            "lang": "fa",
            "name": spec["name"],
            "version": args.version,
            "description": spec["description"],
            "author": AUTHOR,
            "email": EMAIL,
            "url": PROJECT_URL,
            "license": spec["license"],
            "sources": spec["sources"],
            "notes": spec["notes"],
            "performance": performance,
        }
    )
    out = Path(args.output)
    nlp.to_disk(out)
    print(f"wrote {out} as fa_{spec['name']} {args.version} ({spec['license']})")
    scalars = {k: round(v * 100, 2) for k, v in performance.items() if isinstance(v, float)}
    print(json.dumps(scalars, indent=2))
 if __name__ == "__main__":
    main()
--- a/scripts/smoke_test.py
+++ b/scripts/smoke_test.py
@ -1,10 +1,9 @@
-"""Run a trained pipeline over real Persian text and print every annotation layer.
+"""Run the assembled pipeline over real Persian text and print every annotation layer.
 This is the end-to-end check that the artifact actually works: tokenizer -> tagger ->
-morphologizer -> lemmatizer -> parser -> noun_chunks, plus ner when the pipeline has one.
+morphologizer -> lemmatizer -> parser -> ner -> noun_chunks.
 Works on fa_dep_news_sm, fa_ent_news_sm, or the two combined.
-Usage: .venv/bin/python scripts/smoke_test.py training/fa_dep_news_sm
+Usage: .venv/bin/python scripts/smoke_test.py training/fa_core_news_sm
 """
 import sys
@ -21,7 +20,7 @@ EXTRA = [
 def main():
-    path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_dep_news_sm"
+    path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_core_news_sm"
    nlp = spacy.load(path)
    print(f"loaded {nlp.meta['lang']}_{nlp.meta['name']} {nlp.meta['version']}")
    print(f"pipeline: {nlp.pipe_names}")
@ -36,13 +35,10 @@ def main():
                f"{t.text:<16}{t.lemma_:<16}{t.pos_:<7}{t.tag_:<14}"
                f"{t.dep_:<14}{t.head.text}"
            )
-        if doc.has_annotation("MORPH"):
+        print(f"morph[0]: {doc[0].morph}")
-            print(f"morph[0]: {doc[0].morph}")
+        print(f"sents: {[s.text for s in doc.sents]}")
-        if doc.has_annotation("DEP"):
+        print(f"ents: {[(e.text, e.label_) for e in doc.ents]}")
-            print(f"sents: {[s.text for s in doc.sents]}")
+        print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}")
            print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}")
        if "ner" in nlp.pipe_names:
            print(f"ents: {[(e.text, e.label_) for e in doc.ents]}")
 if __name__ == "__main__":
--- a/scripts/transfer_perdt_ner.py
+++ b/scripts/transfer_perdt_ner.py
@ -1,169 +0,0 @@
 """Transfer PerDT's NER layer onto the tokenization `spacy convert --merge-subtokens` produces.
 The NER files use the original Dadegan tokenization, which matches the released UD tokenization
 in only 57 to 62% of sentences: the NER files drop some copulas and auxiliaries, and at least
 one honorific is corrupted (`ص` written as `،`). Entities sit on content words present in both,
 so difflib aligns them. A span transfers only if every one of its tokens maps and the result
 stays contiguous; anything else is dropped rather than guessed.
 Measured transfer rate: 99.86% (train), 99.74% (dev), 99.51% (test). See PLAN.md §1.
 Usage:
  python scripts/transfer_perdt_ner.py --conllu-dir assets/ud --ner-dir assets/ud-ner \\
      --out corpus/perdt-ner-iob
 With --out, writes IOB2 files on the merged tokenization to DIR/{split}.txt.
 Without it, only reports the transfer rate.
 """
 import argparse
 from collections import Counter
 from difflib import SequenceMatcher
 from pathlib import Path
 SPLITS = ("train", "dev", "test")
 def merged_tokens(conllu_path):
    """Tokens as --merge-subtokens yields them: multiword-token surface forms, not subtokens."""
    sents, cur, skip_to = [], [], 0
    for line in conllu_path.open(encoding="utf8"):
        line = line.rstrip("\n")
        if line.startswith("#"):
            continue
        if not line:
            if cur:
                sents.append(cur)
                cur, skip_to = [], 0
            continue
        c = line.split("\t")
        if "." in c[0]:
            continue
        if "-" in c[0]:
            skip_to = int(c[0].split("-")[1])
            cur.append(c[1])
            continue
        if skip_to and int(c[0]) <= skip_to:
            continue
        cur.append(c[1])
    if cur:
        sents.append(cur)
    return sents
 def iob_sents(path):
    sents, cur = [], []
    for line in path.open(encoding="utf8"):
        line = line.rstrip("\n")
        if not line.strip():
            if cur:
                sents.append(cur)
                cur = []
            continue
        p = line.split("\t")
        if len(p) < 2:
            p = line.split()
        if len(p) < 2:
            continue
        cur.append((p[0], p[-1]))
    if cur:
        sents.append(cur)
    return sents
 def spans_from_iob(tagged):
    """[(start, end, label)] over the source token indices."""
    spans, i = [], 0
    while i < len(tagged):
        tag = tagged[i][1]
        if tag.startswith("B-"):
            label, j = tag[2:], i + 1
            while j < len(tagged) and tagged[j][1] == f"I-{label}":
                j += 1
            spans.append((i, j, label))
            i = j
        else:
            i += 1
    return spans
 def index_map(src, dst):
    """src index -> dst index for tokens difflib considers equal."""
    out = {}
    matcher = SequenceMatcher(a=src, b=dst, autojunk=False)
    for a, b, size in matcher.get_matching_blocks():
        for k in range(size):
            out[a + k] = b + k
    return out
 def transfer_split(conllu, ner_file):
    """Return (tokens, tags) per sentence on the merged tokenization, plus counters."""
    gold = merged_tokens(conllu)
    ner = iob_sents(ner_file)
    moved, lost = Counter(), Counter()
    result = []
    for i in range(min(len(gold), len(ner))):
        dst = gold[i]
        src = [t for t, _ in ner[i]]
        m = index_map(src, dst)
        tags = ["O"] * len(dst)
        for start, end, label in spans_from_iob(ner[i]):
            idx = [m[k] for k in range(start, end) if k in m]
            contiguous = idx and idx == list(range(idx[0], idx[0] + len(idx)))
            if len(idx) == end - start and contiguous:
                tags[idx[0]] = f"B-{label}"
                for k in idx[1:]:
                    tags[k] = f"I-{label}"
                moved[label] += 1
            else:
                lost[label] += 1
        result.append((dst, tags))
    return result, moved, lost
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--conllu-dir", default="assets/ud",
                    help="directory holding fa_perdt-ud-{split}.conllu")
    ap.add_argument("--ner-dir", default="assets/ud-ner",
                    help="directory holding {split}_with_NER_tag.txt")
    ap.add_argument("--out", default=None, help="write IOB2 files here")
    args = ap.parse_args()
    conllu_dir, nerdir = Path(args.conllu_dir), Path(args.ner_dir)
    for d in (conllu_dir, nerdir):
        if not d.is_dir():
            raise SystemExit(f"not found: {d}")
    out = Path(args.out) if args.out else None
    if out:
        out.mkdir(parents=True, exist_ok=True)
    grand_moved, grand_lost = Counter(), Counter()
    for split in SPLITS:
        sents, moved, lost = transfer_split(
            conllu_dir / f"fa_perdt-ud-{split}.conllu",
            nerdir / f"{split}_with_NER_tag.txt",
        )
        m, dropped = sum(moved.values()), sum(lost.values())
        grand_moved += moved
        grand_lost += lost
        print(f"{split:<6} entities {m + dropped:>6}  transferred {m:>6}"
              f" ({100 * m / max(m + dropped, 1):.2f}%)  dropped {dropped}")
        if out:
            path = out / f"{split}.txt"
            with path.open("w", encoding="utf8") as fh:
                for toks, tags in sents:
                    for tok, tag in zip(toks, tags):
                        fh.write(f"{tok}\t{tag}\n")
                    fh.write("\n")
            print(f"       wrote {path}")
    print("\nper label:")
    for label in sorted(grand_moved | grand_lost, key=lambda k: -grand_moved[k]):
        m, dropped = grand_moved[label], grand_lost[label]
        print(f"  {label:<6} {m:>6}/{m + dropped:<6} ({100 * m / max(m + dropped, 1):5.1f}%)")
 if __name__ == "__main__":
    main()