fazel
/
spacy-fa-pipeline
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: fazel:colab-lg-trf-training
Branches Tags
fazel:main
fazel:colab-lg-trf-training
..
compare: fazel:main
Branches Tags
fazel:main
fazel:colab-lg-trf-training

18 Commits
colab-lg-t ... main

Author SHA1 Message Date
Mohamad Fazeli 0626a12352
README edits: subheadings under Results, fix Xeon column missing from throughput table, fix markdown bullet break in training paragraph, flatten note italics, de-AI-tell wording 2026-08-15 17:03:06 +03:30
Mohamad Fazeli 37c85d0186
fix layout of README.md 2026-08-14 11:57:35 +03:30
Mohamad Fazeli e37d052435
Correct the 940MX trf figure to a 9-run median 
1,106 words/s, not the 1,158 from the first 3-run sample. Two independent 3-run
samples disagreed by 3.4%, so the larger sample settles it.
 2026-08-13 17:52:15 +03:30
Mohamad Fazeli c80bcb27f3
List the three newly published packages in the Persian README 2026-08-13 17:35:14 +03:30
Mohamad Fazeli b26b194d9f
Publish the remaining three packages and link them 
fa_ent_news_sm, fa_dep_news_md and fa_core_news_md are now on the Hub, each with
a generated card carrying accuracy and measured throughput.

The md wheels in packages/ were stale 0.1.0 artifacts from an earlier manual run,
even though project.yml passes --version 3.8.0. Rebuilt at 3.8.0 so they match
their published siblings before upload.
 2026-08-13 17:34:41 +03:30
Mohamad Fazeli f1a95d92c3
Link each package to its Hugging Face page 
Add the lg packages to the table, which had never been listed, with measured
wheel sizes rather than the rounded ones. fa_ent_news_sm, fa_dep_news_md and
fa_core_news_md are built by project.yml but not on the Hub, so they stay
unlinked and are called out as unpublished. All nine links verified to return
200.

Move the cu126 torch requirement out of both READMEs into MODELS.md 9.
 2026-08-13 17:17:57 +03:30
Mohamad Fazeli e7871a29a1
Cut the throughput notes to measurements 
State the numbers and the cu126 requirement; drop the narration around them.
 2026-08-13 17:12:50 +03:30
Mohamad Fazeli 4135142e72
Fill in trf throughput on the 940MX 
The 940MX does run trf, at 1,158 words/s with batch 32 inside 2 GB, 6.2x its host
CPU. The earlier claim that current PyTorch wheels cannot target sm_50 was only
half right: Maxwell kernels were dropped from the cu128 and cu129 builds starting
torch 2.8, which is what `pip install torch` now resolves to, but the cu126 build
of 2.7.1 still ships sm_50 and works.

.venv-trf-gpu pins torch==2.7.1+cu126 for this and stays separate from .venv,
whose cupy runs on nvidia-* 12.9 wheels that torch would downgrade to 12.6.

Drop the editorial sentence from the generated model card; the table states it.
 2026-08-13 17:11:25 +03:30
Mohamad Fazeli 6fa8aa70b8
Lead the README tables with trf instead of sm 
The Hazm and en_core_web_sm comparison now cites fa_core_news_trf, which is the
strongest tier and the only one to pass the hazm+ParsBERT DEP_LAS reference.
Drop the package-size row from that table: it compared a 608 MB transformer
against a 7 MB rule-based toolkit, which says nothing about accuracy.

Add trf to the package list and to the per-label NER table, alongside lg, which
had never been added. Its largest per-label gains are PER +9.25 and DAT +11.69
over lg. Drop the gold-count column there and trim the surrounding notes.
 2026-08-13 16:52:37 +03:30
Mohamad Fazeli 5da9dd1524
Add fa_core_news_trf tier on a shared fine-tuned ParsBERT 
Trains tagger, morphologizer, trainable_lemmatizer, parser and ner against one
fine-tuned HooshvareLab/bert-base-parsbert-uncased through TransformerListener,
rather than the sm/md/lg split where ner is trained separately and sourced in.
Fine-tuning a 162M-parameter encoder twice would double GPU cost, ship two
encoders in one wheel, and collide on the `transformer` component name.

A shared encoder needs one corpus carrying both annotation layers, so
merge_joint_corpus.py fuses them. corpus/perdt-ner/ came from the same
--merge-subtokens CoNLL-U as corpus/merged/ with the same --n-sents, so the
DocBins are token-for-token identical; the script asserts that per document and
copies doc.ents by token index. Char offsets do not work here because the two
converters differ in trailing whitespace, which pushes char_span off the token
grid and returns None.

Trained on a Colab T4 in 1h58m, 3000 steps, no early stop. Test scores against
lg: DEP_LAS 90.79 (+4.19), ENTS_F 82.89 (+6.95, almost all recall), TAG_ACC
97.62 (+1.07). DEP_LAS passes the hazm+ParsBERT reference of 89.34, which no CPU
tier reached. LEMMA_ACC 97.31 and SENTS_F 97.35 regress against lg; the likely
cause of the latter is strided_spans leaving only 32 tokens of overlap.

max_steps and learn_rate.total_steps are held equal on purpose. They are
independent knobs, and a patience stop under a longer total_steps ends training
at a high learning rate, discarding the annealing tail.

finalize_pipeline.py now reads the encoder name out of the trained config
instead of hardcoding it, tracks per-encoder licences, and writes a
redistribution warning into meta.json when the encoder states none. ParsBERT
states none, so that wheel is not redistributable; roberta-fa-zwnj-base
(Apache-2.0) is a one-line change to `name`.

Add benchmark_throughput.py, which times nlp.pipe alone. The words/s from
`spacy benchmark accuracy` includes the Scorer's per-token alignment, which is
why MODELS.md 7 reported ent_lg as faster than sm despite an identical ner
architecture. Remeasured every tier on CPU and the 940MX; a background rsync
halved every figure, so the final numbers are 9-run medians on an idle machine.

Add make_model_card.py, which composes the Hub card from meta.json and the
throughput records. The card spacy package writes has no frontmatter, so the Hub
cannot index the model by language, and no install line or usage.
 2026-08-13 16:34:55 +03:30
Mohamad Fazeli f45d0db643
Remove em dashes from lg tier docs and config headers 2026-08-12 22:07:49 +03:30
Mohamad Fazeli 9e8ed06361
Complete lg tier: fa_dep_news_lg / fa_core_news_lg 
Same 200k-row floret table as fa_ent_news_lg, now with dep/core too.

- configs/fa_dep_news_lg.cfg (fa_dep_news_md.cfg unchanged except vectors)
- project.yml: lg workflow expanded to train-dep-lg, finalize-dep-lg,
  assemble-core-lg, evaluate-lg, finalize-meta-lg, package-lg,
  smoke-lg (mirrors the md tier's dep/core commands)
- scripts/compare_tiers.py: dep/core UD test groups now 3-way (sm/md/lg)
- scripts/finalize_pipeline.py: FLORET_LG url points at the published
  vectors, https://huggingface.co/Phazel/fa-floret-wiki-vectors

UD test: DEP_UAS 90.96 (sm 89.69, md 90.52), DEP_LAS 86.60 (sm 85.15,
md 86.34). NER unchanged from the earlier fa_ent_news_lg run, ENTS_F
75.94. docs/MODELS.md §7 rewritten from ent-only to the full tier.
 2026-08-12 22:03:50 +03:30
Mohamad Fazeli b89b01ceb6
Add lg tier: fa_ent_news_lg on 200k floret vectors 
New table: 200k rows x 300d, full Persian Wikipedia dump, 5 epochs,
vs md's 50k rows / 400k documents.

- configs/fa_ner_lg.cfg, project.yml ent-lg workflow (vectors-lg
  through smoke-ent-lg)
- scripts/compare_tiers.py: generalized sm/md pair to N tiers; ent
  NER test now includes lg; fixed sm baseline to the file that's
  actually scored (perdt-ner-test.json, not the missing ent-test.json)
- scripts/finalize_pipeline.py: FLORET_LG source and vectors_note_lg
  corrected to full Wikipedia, 5 epochs (were a generic Wikipedia +
  OSCAR placeholder)
- docs/MODELS.md §7: PerDT NER test ENTS_F 75.94 (sm 71.87, md
  74.71), full per-label table, cost (217 MB wheel)

Not built: fa_dep_news_lg / fa_core_news_lg.
 2026-08-12 19:01:32 +03:30
Mohamad Fazeli c3cb02d9c3
README: dedicated NER section, drop AI-tell formatting 
- New ## Named entity recognition section: labels, silver-provenance,
  per-label breakdown table, package options. Moved out of ## Results
  (which now just points to it) and out of ## Caveats (silver-label and
  thin-label bullets folded into the new section's prose instead of
  repeating them).
- Added fa_ent_news_sm to the package table (built, was undocumented).
- Design decision 4 no longer repeats the NER transfer methodology now
  that it has a dedicated home.
- Dropped the warning-emoji decoration on the comparability note.
 2026-08-12 13:49:05 +03:30
Mohamad Fazeli d39c09daca
Merge github/main (README punctuation fixes) into main 
# Conflicts:
#	README.md
 2026-08-12 13:21:51 +03:30
Mohamad Fazeli 8e42c38ed6
Ship the md tier: floret vectors, docs, and HF publish fixes 
- project.yml: new md workflow (vectors-md, train-dep-md, train-ner-md,
  finalize-dep-md, assemble-core-md, evaluate-md, finalize-meta-md,
  compare-md, package-md, smoke-md), same corpus/architecture as sm plus
  the fa_floret static vector table (50k rows x 300d, 400k Persian
  documents).
- configs/fa_dep_news_md.cfg, configs/fa_ner_md.cfg: byte-identical to the
  sm configs except include_static_vectors, isolating what the vectors buy.
- scripts/unpack_vectors.py: extracts a floret wheel's vectors-only
  pipeline into a directory --paths.vectors can point at.
- scripts/finalize_pipeline.py: --size now accepts md (floret source +
  vectors note), plus lg/trf (used by the pending Colab notebook on
  colab-lg-trf-training; lg's vectors note is generated from the trained
  model's actual vector table shape since that tier is still being
  iterated on).
- scripts/compare_tiers.py: sm vs md metrics diff.
- docs/MODELS.md, README.md, README.fa.md: md tier results, fa_ent_news_md
  package row, and the analysis of why floret helps NER recall.
- docs/CONTRIBUTING-GUIDE.md, README.md, README.fa.md: fixed every
  documented pip install URL. spacy huggingface-hub push names the
  uploaded wheel '<name>-any-py3-none-any.whl'; 'any' is not a valid PEP
  440 version, so current pip rejects it. Re-uploaded a correctly
  versioned copy of every published wheel (fa_core_news_sm, fa_dep_news_sm,
  fa_ent_news_md) to the Hub and repointed the docs at that filename.
 2026-08-12 13:15:55 +03:30
Kiyarash Fazeli fe6c81272f
Refine README.md for clarity and punctuation 
Corrected punctuation and improved clarity in performance and reproducibility sections.
 2026-08-11 22:18:59 +03:30
Kiyarash Fazeli 60516f6d35
Fix formatting in README.md 
Removed unnecessary line break in README.
 2026-08-11 14:34:00 +03:30
15 changed files with 2081 additions and 620 deletions
Show Stats Expand all files Collapse all files

4
.gitignore vendored
View File

@ -9,6 +9,10 @@ packages/
# Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the # Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the
# spaCy CLI in the training venv. See .omp/AGENTS.md. # spaCy CLI in the training venv. See .omp/AGENTS.md.
.venv-publish/ .venv-publish/
# Local envs for verifying and benchmarking the trf wheel: CPU-only torch, and a cu126 build
# for the 940MX. Kept out of .venv so a CUDA-lib downgrade cannot reach the training env.
.venv-trf/
.venv-trf-gpu/
__pycache__/ __pycache__/
*.pyc *.pyc
# Personal scratch list, not part of the project # Personal scratch list, not part of the project

74
README.fa.md
View File

@ -12,7 +12,7 @@
**بازشناسی موجودیت‌های نام‌دار** را دارد. هر دو تحت لیسانس CC BY-SA ۴٫۰ منتشر شده‌اند. **بازشناسی موجودیت‌های نام‌دار** را دارد. هر دو تحت لیسانس CC BY-SA ۴٫۰ منتشر شده‌اند.
```bash ```bash
pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-3.8.0-py3-none-any.whl
``` ```
```python ```python
@ -22,22 +22,51 @@ doc = nlp("محمدرضا شجریان در مشهد به دنیا آمد.")
print(doc.ents) # (محمدرضا شجریان, مشهد) print(doc.ents) # (محمدرضا شجریان, مشهد)
``` ```
بسته‌های منتشرشده روی Hugging Face:
[`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) ·
[`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) ·
[`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) ·
[`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) ·
[`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) ·
[`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) ·
[`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) ·
[`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) ·
[`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) ·
[`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf).
جدول‌های بردار floret جداگانه (فقط بردار، بدون هیچ مؤلفه‌ای):
```bash
# ۵۰ هزار سطر × ۳۰۰ بعد، ۴۰۰ هزار سند فارسی (جدول ردهٔ md)
pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
# ۵۰ هزار سطر × ۳۰۰ بعد، کل دامپ ویکی‌پدیای فارسی
pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
# ۲۰۰ هزار سطر × ۳۰۰ بعد، کل دامپ ویکی‌پدیای فارسی، ۵ دوره (جدول ردهٔ lg)
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
```
## کارایی ## کارایی
ارزیابی با `spacy benchmark accuracy` روی بخش آزمون همان پیکره انجام شده است: ارزیابی با `spacy benchmark accuracy` روی بخش آزمون همان پیکره انجام شده است:
| سنجه | امتیاز | مرجع | | سنجه | `sm` | `md` | `lg` | `trf` | مرجع |
| --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `TOKEN_ACC` / `TOKEN_F` | ۹۹٫۹۶ / ۹۹٫۱۱ | | | `TOKEN_ACC` / `TOKEN_F` | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | |
| `TAG_ACC` (XPOS) | ۹۵٫۹۶ | | | `TAG_ACC` (XPOS) | ۹۵٫۹۶ | ۹۶٫۲۵ | ۹۶٫۵۵ | **۹۷٫۶۲** | |
| `POS_ACC` (UPOS) | ۹۶٫۲۴ | | | `POS_ACC` (UPOS) | ۹۶٫۲۴ | ۹۶٫۶۴ | ۹۶٫۶۸ | **۹۷٫۶۳** | |
| `MORPH_ACC` | ۹۶٫۲۹ | | | `MORPH_ACC` | ۹۶٫۲۹ | ۹۶٫۶۴ | ۹۶٫۷۰ | **۹۷٫۸۲** | |
| `LEMMA_ACC` | ۹۷٫۹۱ | | | `LEMMA_ACC` | ۹۷٫۹۱ | ۹۷٫۹۶ | **۹۸٫۰۸** | ۹۷٫۳۱ | |
| `SENTS_F` | ۹۹٫۲۵ | | | `SENTS_F` | ۹۹٫۲۵ | **۹۹٫۲۸** | ۹۹٫۱۸ | ۹۷٫۳۵ | |
| `DEP_UAS` | ۸۹٫۶۹ | hazm+ParsBERT: ۹۲٫۴۶ | | `DEP_UAS` | ۸۹٫۶۹ | ۹۰٫۵۲ | ۹۰٫۹۶ | **۹۳٫۸۷** | hazm+ParsBERT: ۹۲٫۴۶ |
| `DEP_LAS` | ۸۵٫۱۵ | hazm+ParsBERT: ۸۹٫۳۴ | | `DEP_LAS` | ۸۵٫۱۵ | ۸۶٫۳۴ | ۸۶٫۶۰ | **۹۰٫۷۹** | hazm+ParsBERT: ۸۹٫۳۴ |
| `ENTS_F` | ۷۱٫۸۷ | تنها در `fa_core_news_sm` | | `ENTS_P` | ۷۷٫۶۷ | ۷۶٫۵۶ | ۸۱٫۵۱ | **۸۴٫۰۶** | |
| سرعت | حدود ۹٬۲۵۰ واژه بر ثانیه | | | `ENTS_R` | ۶۶٫۸۷ | ۷۲٫۹۵ | ۷۱٫۰۹ | **۸۱٫۷۶** | |
| `ENTS_F` | ۷۱٫۸۷ | ۷۴٫۷۱ | ۷۵٫۹۴ | **۸۲٫۸۹** | |
| سرعت (940MX، دستهٔ ۳۲) | ۱۰٬۲۳۵ | ۹٬۰۵۸ | ۹٬۲۱۵ | بخش توان عملیاتی | |
| حجم بستهٔ نصب | ۱۳٫۵ مگابایت | ۶۸٫۵ مگابایت | ۲۳۵ مگابایت | ۶۰۸ مگابایت | |
ردهٔ `trf` در همه‌جا جلو است مگر در واژه‌یابی و مرزبندی جمله، و تنها ردهٔ‌ای است که از مرجع
`DEP_LAS` برابر ۸۹٫۳۴ عبور می‌کند. به کارت گرافیک نیاز دارد و مدل پایهٔ آن پروانهٔ مشخصی ندارد،
پس قابل بازانتشار نیست (`docs/MODELS.md` بخش ۸).
برچسب‌های موجودیت «نقره‌ای» هستند: از لایه‌ای در خود پیکره می‌آیند که با برچسب‌زن Beheshti-NER برچسب‌های موجودیت «نقره‌ای» هستند: از لایه‌ای در خود پیکره می‌آیند که با برچسب‌زن Beheshti-NER
تولید و سپس دستی اصلاح شده است. بنابراین `ENTS_F` تا اندازه‌ای هم‌خوانی با آن برچسب‌زن را تولید و سپس دستی اصلاح شده است. بنابراین `ENTS_F` تا اندازه‌ای هم‌خوانی با آن برچسب‌زن را
@ -46,6 +75,25 @@ print(doc.ents) # (محمدرضا شجریان, مشهد)
آموزش روی یک پردازندهٔ چهارهسته‌ای i5-7200U و بدون کارت گرافیک انجام شده است: ۱ ساعت و ۲۷ دقیقه آموزش روی یک پردازندهٔ چهارهسته‌ای i5-7200U و بدون کارت گرافیک انجام شده است: ۱ ساعت و ۲۷ دقیقه
برای اجزای نحوی و ۱۷ دقیقه برای NER. این دو اجرا مستقل‌اند و می‌توانند هم‌زمان انجام شوند. برای اجزای نحوی و ۱۷ دقیقه برای NER. این دو اجرا مستقل‌اند و می‌توانند هم‌زمان انجام شوند.
## توان عملیاتی
میانهٔ چند اجرای پیاپی `nlp.pipe` روی ۱۴۶ سند بخش آزمون PerDT (۲۳٬۸۲۵ توکن). تنها زمان خودِ
`pipe` اندازه‌گیری شده و اجرای گرم‌کردن کنار گذاشته می‌شود. برای بازتولید:
`python scripts/benchmark_throughput.py <model> --gpu-id <n>`؛ دادهٔ خام در
`metrics/throughput-*.json` است.
| رده | پردازنده i5-7200U | کارت 940MX | کارت Tesla T4 |
| --- | ---: | ---: | ---: |
| `sm` | ۵٬۴۸۴ | ۱۰٬۲۳۵ | |
| `md` | ۵٬۴۰۸ | ۹٬۰۵۸ | |
| `lg` | ۴٬۷۱۵ | ۹٬۲۱۵ | |
| `trf` | ۱۸۷ | ۱٬۱۰۶ | ۸٬۳۲۰ |
ردهٔ `trf` روی یک پردازنده ۲۹ برابر کندتر از `sm` است. عددهای T4 و Xeon از یک ماشین Colab
می‌آیند، یعنی شتاب ۲۵ برابری. فاصلهٔ رده‌های پردازنده‌ای کمتر از ۱۵ درصد است، پس گلوگاه
تجزیه‌گر و واژه‌یاب است نه جست‌وجوی tok2vec. پراکندگی اجراها روی لپ‌تاپ حدود ۱۰± درصد است.
اجرای `trf` روی 940MX به نسخهٔ مشخصی از torch نیاز دارد؛ بخش ۹ از `docs/MODELS.md` را ببینید.
گام‌های تبدیل پیکره، آموزش، ارزیابی و بسته‌بندی در [`project.yml`](project.yml) تعریف شده‌اند. گام‌های تبدیل پیکره، آموزش، ارزیابی و بسته‌بندی در [`project.yml`](project.yml) تعریف شده‌اند.
توضیح بیشتر دربارهٔ گزینش پیکره و پروانه‌ها در [`docs/MODELS.md`](docs/MODELS.md) و شرح انگلیسی توضیح بیشتر دربارهٔ گزینش پیکره و پروانه‌ها در [`docs/MODELS.md`](docs/MODELS.md) و شرح انگلیسی
پروژه در [`README.md`](README.md) آمده است. پروژه در [`README.md`](README.md) آمده است.

188
README.md
View File

@ -1,9 +1,9 @@
# Persian (Farsi) pipelines for spaCy # Persian (Farsi) pipelines for spaCy
Trained spaCy pipelines for Persian, installable now. spaCy has never shipped an official one, and `spacy.blank("fa")` only gives you a tokenizer and stop words. choose between `fa_core_news_sm` (full syntax + NER) or `fa_dep_news_sm` (syntax only). Trained spaCy pipelines for Persian, installable with pip. spaCy has never shipped an official one, and `spacy.blank("fa")` only gives you a tokenizer and stop words. Choose between `fa_core_news_sm` (full syntax + NER) or `fa_dep_news_sm` (syntax only).
```bash ```bash
pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-3.8.0-py3-none-any.whl
``` ```
```python ```python
@ -21,84 +21,153 @@ pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_n
[('ایران خودرو', 'ORG'), ('۲۰ درصد', 'PCT')] [('ایران خودرو', 'ORG'), ('۲۰ درصد', 'PCT')]
``` ```
## Why spacy-persian?
- **⚡ Performance** – **96.24%** POS · **97.91%** Lemma · **85.15%** LAS – competitive with English `en_core_web_sm` on syntax.
- **🚀 Speed** – ~9,250 words/sec on a standard CPU. No GPU required.
- **📦 Flexibility** – Choose `fa_core_news_sm` (13MB, syntax + NER) or `fa_dep_news_sm` (7.5MB, syntax-only).
- **🔁 Reproducibility** – Checksummed, versioned builds from UD_Persian-PerDT – no black boxes.
- **🔌 Native spaCy** – Drop-in replacement. `spacy.load()` works instantly with standard `Doc` objects.
-
## Results ## Results
`spacy-persian` delivers production‑ready Persian NLP that stands alongside Hazm—the most popular Persian toolkit—while bringing the full power of the spaCy ecosystem. Compared against Hazm (the most-used Persian toolkit) and `en_core_web_sm` (English reference).
| Metric | **`spacy-persian`**<br>`fa_core_news_sm` | **Hazm**<br>(Persian toolkit) | `en_core_web_sm`<br>(English reference) | | Metric | **`spacy-persian`**<br>`fa_core_news_trf` | **Hazm**<br>(Persian toolkit) | `en_core_web_sm`<br>(English reference) |
|--------|:---:|:---:|:---:| |--------|:---:|:---:|:---:|
| **POS Accuracy (UPOS)** | **96.24%** | ~95.69%¹ | 97.21%² | | **POS Accuracy (UPOS)** | **97.63%** | ~95.69%¹ | 97.21%² |
| **Lemma Accuracy** | **97.91%** | 89.9%¹ | — | | **Lemma Accuracy** | **97.31%** | 89.9%¹ | — |
| **Dependency LAS** | 85.15% | 85.6%¹ | 91.85%² | | **Dependency LAS** | **90.79%** | 85.6%¹ | 91.85%² |
| **NER F-score** | 71.87% | — | 83.80%² | | **NER F-score** | **82.89%** | — | 83.80%² |
| **Package Size** | **13 MB** (syntax+NER)<br>**7.5 MB** (syntax-only) | ~7 MB | 12 MB |
> **¹** Hazm scores from its official README > **¹** Hazm scores from its official README
> **²** `en_core_web_sm` scores from spaCy's official model card > **²** `en_core_web_sm` scores from spaCy's official model card
> ⚠️ **Note on comparability:** These benchmarks come from *different evaluation sets, treebanks, and test splits*. > **Note on comparability:** these benchmarks come from different evaluation sets, treebanks, and test splits.
### Packages
| Package | Components | Licence | Score | Wheel |
| --- | --- | --- | --- | --- |
| [`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | CC BY-SA 4.0 | LEMMA 97.91 | 7.9 MB |
| [`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) | the above plus ner | CC BY-SA 4.0 | ENTS_F 71.87 | 13.5 MB |
| [`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) | `ner` alone (own embedded tok2vec) | CC BY-SA 4.0 | ENTS_F 71.87 | 5.9 MB |
| [`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) | same as `fa_dep_news_sm`, plus floret vectors | CC BY-SA 4.0 | LEMMA 97.96 | 62.6 MB |
| [`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) | same as `fa_core_news_sm`, plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 68.5 MB |
| [`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) | `ner` alone (own embedded tok2vec), plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 60.6 MB |
| [`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) | same as `fa_dep_news_sm`, plus full-wiki floret vectors | CC BY-SA 4.0 | LEMMA 98.08 | 229.3 MB |
| [`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) | same as `fa_core_news_sm`, plus full-wiki floret vectors | CC BY-SA 4.0 | ENTS_F 75.94 | 235.2 MB |
| [`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) | `ner` alone (own embedded tok2vec), plus full-wiki floret vectors | CC BY-SA 4.0 | ENTS_F 75.94 | 227.3 MB |
| [`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf) | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | see §8, encoder unlicensed | ENTS_F 82.89, LAS 90.79 | 608.2 MB |
| Metric | Score | Reference | These scores are from `spacy benchmark accuracy`, stored in `metrics/`.
| --- | --- | --- |
| `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | |
| `TAG_ACC` (XPOS) | 95.96 | |
| `POS_ACC` (UPOS) | 96.24 | |
| `MORPH_ACC` | 96.29 | |
| `LEMMA_ACC` | 97.91 | |
| `SENTS_F` | 99.25 | |
| `DEP_UAS` | 89.69 | hazm+ParsBERT: 92.46 |
| `DEP_LAS` | 85.15 | hazm+ParsBERT: 89.34 |
| Speed | ~9,250 words/s | |
Entities, `fa_core_news_sm` only, on the PerDT NER test split: `ENTS_P` 77.67, `ENTS_R` 66.87, Raw `fa.floret` and `fa.vec` exports of the `lg` tier's 200k-row table are in
`ENTS_F` 71.87. [`fa-floret-wiki-vectors`](https://huggingface.co/Phazel/fa-floret-wiki-vectors).
| Label | F | Train examples | ### Tier comparison
| --- | --- | --- |
| `LOC` | 80.24 | 4,954 |
| `DAT` | 74.45 | 1,323 |
| `MON` | 73.68 | 205 |
| `ORG` | 68.77 | 2,643 |
| `TIM` | 66.67 | 135 |
| `PER` | 65.29 | 4,847 |
| `PCT` | 57.14 | 121 |
The `md` tier adds a 50k x 300d floret vector table trained on 400k Persian documents. Its
config differs from `sm` by exactly one line (`include_static_vectors`), so the columns below
isolate what the vectors buy. Full breakdown in `docs/MODELS.md` §6.
For comparison, `en_core_web_sm` scores TAG 97, LAS 90, ENTS_F 84 on a larger, cleaner corpus. | Metric | `sm` | `md` | `lg` | `trf` | Reference |
Trained on a 4-core i5-7200U with no GPU: 1h27m for the syntax components, 17 min for NER. | --- | --- | --- | --- | --- | --- |
| `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | 99.96 / 99.11 | 99.96 / 99.11 | 99.96 / 99.11 | |
| `TAG_ACC` (XPOS) | 95.96 | 96.25 | 96.55 | **97.62** | |
| `POS_ACC` (UPOS) | 96.24 | 96.64 | 96.68 | **97.63** | |
| `MORPH_ACC` | 96.29 | 96.64 | 96.70 | **97.82** | |
| `LEMMA_ACC` | 97.91 | 97.96 | **98.08** | 97.31 | |
| `SENTS_F` | 99.25 | **99.28** | 99.18 | 97.35 | |
| `DEP_UAS` | 89.69 | 90.52 | 90.96 | **93.87** | Hazm+ParsBERT: 92.46 |
| `DEP_LAS` | 85.15 | 86.34 | 86.60 | **90.79** | Hazm+ParsBERT: 89.34 |
| `ENTS_P` | 77.67 | 76.56 | 81.51 | **84.06** | |
| `ENTS_R` | 66.87 | 72.95 | 71.09 | **81.76** | |
| `ENTS_F` | 71.87 | 74.71 | 75.94 | **82.89** | |
| Speed (940MX, batch 32) | 10,235 words/s | 9,058 words/s | 9,215 words/s | 1,106 words/s | |
| Wheel size | 13.5 MB | 68.5 MB | 235 MB | 608 MB | |
`trf` leads on every metric except lemmatization and sentence segmentation. It is also the only
tier to clear the Hazm+ParsBERT `DEP_LAS` reference of 89.34. It needs a GPU, and its encoder
states no licence, so it is not redistributable (`docs/MODELS.md` §8).
Entity scores are `fa_core_news_*` on the PerDT NER test split; per-label breakdown and
caveats are in [Named entity recognition](#named-entity-recognition).
Trained on a 4-core i5-7200U with no GPU: `sm` took 1h27m for syntax plus 17 min for NER,
`md` 1h54m plus 25 min (the two `md` runs overlapped, so wall clock overstates each).
### Vector packages
Standalone floret vector packages (vectors only, `pipeline: []`), usable as
`--paths.vectors` for your own training or as a plain embedding table:
```bash
# 50k rows x 300d, 400k Persian documents (the md tier's table)
pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
# 50k rows x 300d, full Persian Wikipedia dump
pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
# 200k rows x 300d, full Persian Wikipedia dump, 5 epochs (the lg tier's table)
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
```
## Throughput
Median of repeated `nlp.pipe` passes over the 146-document PerDT test split (23,825 tokens),
timing the pipe only, warmup discarded. Reproduce with
`python scripts/benchmark_throughput.py <model> --gpu-id <n>`; raw records are in
`metrics/throughput-*.json`.
| Tier | CPU, i5-7200U | GPU, GeForce 940MX | CPU, Xeon @ 2.00GHz | GPU, Tesla T4 |
| --- | ---: | ---: | ---: | ---: |
| `sm` | 5,484 | 10,235 | | |
| `md` | 5,408 | 9,058 | | |
| `lg` | 4,715 | 9,215 | | |
| `trf` | 187 | 1,106 | 336 | 8,320 |
`trf` is 29x slower than `sm` on the same CPU. The Xeon and T4 columns come from one Colab VM,
a 25x GPU speedup. The CPU tiers sit within 15% of each other, so the bottleneck is the parser
and lemmatizer, not the tok2vec lookup. Laptop spread is about 10% with thermal state. Running
`trf` on the 940MX needs a `cu126` torch build, see `docs/MODELS.md` §9.
## Named entity recognition
Seven labels: `LOC`, `PER`, `ORG`, `DAT`, `MON`, `TIM`, `PCT`. They come from PerDT's own
`not-to-release/Dadegan with NER tag/` layer, transferred onto this pipeline's tokenization
by difflib at a 99.86% alignment rate (`scripts/transfer_perdt_ner.py`). Spans that could not
be aligned exactly were dropped rather than guessed. That layer is silver: PerDT's README
states it was produced by the BERT-based Beheshti-NER tagger with manual corrections for
recall, so the `ENTS_F` numbers below partly reflect agreement with that tagger, not with
human annotation.
`ner` runs standalone with its own embedded tok2vec (`fa_ent_news_sm`, `fa_ent_news_md`), or
bundled into `fa_core_news_sm`/`fa_core_news_md` alongside the syntax pipeline. In `trf` it is
trained jointly against the shared transformer instead, so there is no standalone trf variant.
| Label | `sm` F | `md` F | `lg` F | `trf` F | Train examples |
| --- | --- | --- | --- | --- | --- |
| `LOC` | 80.24 | 84.05 | 83.66 | **87.78** | 4,954 |
| `PER` | 65.29 | 68.18 | 72.63 | **81.88** | 4,847 |
| `ORG` | 68.77 | 70.25 | 71.01 | **78.50** | 2,643 |
| `DAT` | 74.45 | 76.19 | 70.83 | **82.52** | 1,323 |
| `MON` | 73.68 | 84.21 | 88.89 | 88.89 | 205 |
| `TIM` | 66.67 | 66.67 | 61.54 | 50.00 | 135 |
| `PCT` | 57.14 | 33.33 | 57.14 | 33.33 | 121 |
`MON`, `TIM` and `PCT` have single-digit support in the test split, so their deltas are one or
two entities changing hands, not signal. `PER`, `LOC` and `ORG` carry the split. The `md` gain
over `sm` (`ENTS_F` 71.87 to 74.71) is almost entirely recall (+6.08), the lexical prior static
vectors give rare proper nouns that hash embeddings never had. `trf` adds another +6.95 F over
`lg`, again mostly recall (71.09 to 81.76), and its largest per-label gains are `PER` (+9.25)
and `DAT` (+11.69).
## Install ## Install
```bash ```bash
pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_news_sm-3.8.0-py3-none-any.whl
# or, without NER: # or, without NER:
pip install https://huggingface.co/Phazel/fa_dep_news_sm/resolve/main/fa_dep_news_sm-any-py3-none-any.whl pip install https://huggingface.co/Phazel/fa_dep_news_sm/resolve/main/fa_dep_news_sm-3.8.0-py3-none-any.whl
``` ```
## Caveats ## Caveats
- **The entity labels are silver.** They come from the treebank's own
`not-to-release/Dadegan with NER tag/` layer, which its README states was produced by the
BERT-based Beheshti-NER tagger with manual corrections for recall. `ENTS_F 71.87` is measured
against a silver test split and partly reflects agreement with that tagger.
- **Three entity labels are thin.** `MON` (205 training examples), `TIM` (135) and `PCT` (121)
rest on 4 to 11 test entities each. `PER`, `LOC`, `ORG` and `DAT` have 1,300 or more.
- **Some lemmas contain a space.** Multiword tokens were merged, so `کتاب‌هایش` is one token - **Some lemmas contain a space.** Multiword tokens were merged, so `کتاب‌هایش` is one token
tagged `N_IANM_PR_JOPER` with lemma `کتاب او`. This affects about 1.5% of tokens. tagged `N_IANM_PR_JOPER` with lemma `کتاب او`. This affects about 1.5% of tokens.
- **`doc.noun_chunks` under-fires.** `spacy/lang/fa/syntax_iterators.py` upstream matches - **`doc.noun_chunks` under-fires.** `spacy/lang/fa/syntax_iterators.py` upstream matches
ClearNLP labels that do not exist in Universal Dependencies. Patch in ClearNLP labels that do not exist in Universal Dependencies. Bug analysis and proposed
[`docs/upstream/fa-noun-chunks.md`](docs/upstream/fa-noun-chunks.md). upstream patch in [`docs/upstream/fa-noun-chunks.md`](docs/upstream/fa-noun-chunks.md).
## Build ## Build
@ -147,14 +216,11 @@ The two training runs are single-threaded and independent, so they can run concu
English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD gives English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD gives
gold UPOS, FEATS and lemmas, which yields real `pos_acc`, `morph_acc` and `lemma_acc` numbers gold UPOS, FEATS and lemmas, which yields real `pos_acc`, `morph_acc` and `lemma_acc` numbers
instead of unmeasurable rule coverage. instead of unmeasurable rule coverage.
4. PerDT, not Seraji: 3.7x more tokens, and Seraji has no `PROPN` tag. Entity spans were 4. PerDT, not Seraji: 3.7x more tokens, and Seraji has no `PROPN` tag.
transferred onto this pipeline's tokenization by difflib at a 99.86% rate, and spans that
could not be aligned exactly were dropped rather than guessed
(`scripts/transfer_perdt_ner.py`).
## Why not hazm's own models ## Why not Hazm's own models
hazm is the reference Persian NLP toolkit and publishes spaCy-format pipelines on the HF Hub, Hazm is the reference Persian NLP toolkit and publishes spaCy-format pipelines on the HF Hub,
so it was the obvious starting point. Four problems: so it was the obvious starting point. Four problems:
- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). The repo contains no - Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). The repo contains no
@ -167,7 +233,7 @@ so it was the obvious starting point. Four problems:
- Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) sit behind `peykaregan.ir` - Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) sit behind `peykaregan.ir`
or `dadegan.ir` under research-only terms. or `dadegan.ir` under research-only terms.
It did confirm the corpus choice. hazm's own spaCy parser was trained on It did confirm the corpus choice. Hazm's own spaCy parser was trained on
`modified_fa_perdt-ud-train.spacy`, the same treebank used here. `modified_fa_perdt-ud-train.spacy`, the same treebank used here.
## More ## More
@ -176,5 +242,5 @@ It did confirm the corpus choice. hazm's own spaCy parser was trained on
- How spaCy models get published, and what upstream `fa` already has: - How spaCy models get published, and what upstream `fa` already has:
[`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md) [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
- The build: [`project.yml`](project.yml) - The build: [`project.yml`](project.yml)
- Language data comes from `spacy/lang/fa` upstream, whose stop word list came from Hazm.
- خلاصهٔ فارسی: [`README.fa.md`](README.fa.md) - خلاصهٔ فارسی: [`README.fa.md`](README.fa.md)
- Language data comes from `spacy/lang/fa` upstream, whose stop word list came from hazm.

291
configs/fa_core_news_trf.cfg Normal file
View File

@ -0,0 +1,291 @@
# fa_core_news_trf: the whole pipeline on one fine-tuned ParsBERT encoder.
#
# Differences from the sm/md/lg tiers, all forced by the transformer:
#
# * One corpus, not two. sm/md/lg train `ner` separately (own embedded tok2vec) and source
# it into the dep model. Fine-tuning a 162M-parameter encoder twice would double GPU cost
# and ship two encoders in one wheel, and the second would collide on the `transformer`
# component name. So every component listens to a single shared transformer and trains
# against corpus/joint/, built by scripts/merge_joint_corpus.py (UD layer + the
# difflib-transferred NER layer on identical tokenization).
# * `use_upper = false` on both transition-based parsers: with a transformer upstream the
# extra maxout layer is redundant, and this matches the upstream *_trf configs.
# * Adam + warmup_linear and accumulate_gradient=3, not the flat 0.001 the CPU tiers use.
# Fine-tuning a pretrained encoder at 1e-3 diverges.
# * gpu_allocator = "pytorch" so thinc and torch share one CUDA memory pool.
#
# Encoder: HooshvareLab/bert-base-parsbert-uncased. NOTE the licence caveat in
# docs/MODELS.md §3.4 - ParsBERT's model card carries no licence statement, so this wheel is
# NOT redistributable on those grounds; HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) is the
# publishable alternative and drops in by changing `name` below.
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
[system]
gpu_allocator = "pytorch"
seed = 0
[nlp]
lang = "fa"
pipeline = ["transformer","tagger","morphologizer","trainable_lemmatizer","parser","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1"
[nlp.vectors]
@vectors = "spacy.Vectors.v1"
[components]
[components.transformer]
factory = "transformer"
max_batch_items = 4096
[components.transformer.set_extra_annotations]
@annotation_setters = "spacy-transformers.null_annotation_setter.v1"
[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "HooshvareLab/bert-base-parsbert-uncased"
mixed_precision = false
[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96
[components.transformer.model.tokenizer_config]
use_fast = true
[components.transformer.model.transformer_config]
[components.transformer.model.grad_scaler_config]
[components.tagger]
factory = "tagger"
label_smoothing = 0.05
overwrite = false
neg_prefix = "!"
[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.tagger.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.tagger.scorer]
@scorers = "spacy.tagger_scorer.v1"
[components.morphologizer]
factory = "morphologizer"
label_smoothing = 0.05
overwrite = true
extend = false
[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.morphologizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.morphologizer.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.morphologizer.scorer]
@scorers = "spacy.morphologizer_scorer.v1"
[components.trainable_lemmatizer]
factory = "trainable_lemmatizer"
backoff = "orth"
min_tree_freq = 3
overwrite = false
top_k = 1
[components.trainable_lemmatizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.trainable_lemmatizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.trainable_lemmatizer.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.trainable_lemmatizer.scorer]
@scorers = "spacy.lemmatizer_scorer.v1"
[components.parser]
factory = "parser"
moves = null
update_with_oracle_cut_size = 100
learn_tokens = false
min_action_freq = 30
[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null
[components.parser.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.parser.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.parser.scorer]
@scorers = "spacy.parser_scorer.v1"
[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100
incorrect_spans_key = null
[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null
[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.ner.scorer]
@scorers = "spacy.ner_scorer.v1"
[corpora]
[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 3
# 3000 steps is ~40 epochs over this 445k-token corpus, measured at ~29 steps/min on a T4
# (~1.8h). The CPU tiers' 20000/1600 would be ~270 epochs and ~12h here, and worse than
# wasteful: warmup_linear anneals against `total_steps`, so a run stopped early by patience
# never leaves the peak learning rate. Budget and schedule are kept equal on purpose:
# training.optimizer.learn_rate.total_steps must track any change to max_steps.
patience = 600
max_epochs = 0
max_steps = 3000
eval_frequency = 100
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null
[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 1e-08
[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 3000
initial_rate = 5e-5
[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null
[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false
[training.score_weights]
tag_acc = 0.16
pos_acc = 0.08
tag_micro_p = null
tag_micro_r = null
tag_micro_f = null
morph_acc = 0.08
morph_per_feat = null
lemma_acc = 0.16
dep_uas = 0.08
dep_las = 0.16
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = 0.0
ents_f = 0.28
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null
[initialize.tokenizer]
[initialize.components]
[pretraining]

231
configs/fa_dep_news_lg.cfg Normal file
View File

@ -0,0 +1,231 @@
# fa_dep_news_lg: tagger, morphologizer, trainable_lemmatizer, parser, WITH the lg-tier
# static vectors.
#
# Byte-identical to configs/fa_dep_news_md.cfg. Only the vector table supplied at train time
# via --paths.vectors differs: fa_floret, 200k rows x 300d, floret mode, trained on the full
# Persian Wikipedia dump for 5 epochs (assets/vectors/fa_floret_lg), vs md's 50k rows x 300d
# trained on 400k Persian documents. Seed, widths, rows, batcher, patience, eval_frequency all
# held constant so the delta measures the vector table and nothing else.
#
# No `ner` here by design; see configs/fa_ner_lg.cfg and project.yml.
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
[system]
gpu_allocator = null
seed = 0
[nlp]
lang = "fa"
pipeline = ["tok2vec", "tagger", "morphologizer", "trainable_lemmatizer", "parser"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
[corpora]
[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 400
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null
[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null
[components]
[pretraining]
[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1"
[nlp.vectors]
@vectors = "spacy.Vectors.v1"
[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 1e-08
learn_rate = 0.001
[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null
[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false
[training.score_weights]
tag_acc = 0.25
pos_acc = 0.12
tag_micro_p = null
tag_micro_r = null
tag_micro_f = null
morph_acc = 0.12
morph_per_feat = null
lemma_acc = 0.25
dep_uas = 0.12
dep_las = 0.12
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = 0.0
[initialize.tokenizer]
[initialize.components]
[components.tok2vec]
factory = "tok2vec"
[components.tagger]
factory = "tagger"
label_smoothing = 0.05
overwrite = false
neg_prefix = "!"
[components.morphologizer]
factory = "morphologizer"
label_smoothing = 0.05
overwrite = true
extend = false
[components.trainable_lemmatizer]
factory = "trainable_lemmatizer"
backoff = "orth"
min_tree_freq = 3
overwrite = false
top_k = 1
[components.parser]
factory = "parser"
moves = null
update_with_oracle_cut_size = 100
learn_tokens = false
min_action_freq = 30
[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0
[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"
[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.tagger.scorer]
@scorers = "spacy.tagger_scorer.v1"
[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.morphologizer.scorer]
@scorers = "spacy.morphologizer_scorer.v1"
[components.trainable_lemmatizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.trainable_lemmatizer.scorer]
@scorers = "spacy.lemmatizer_scorer.v1"
[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null
[components.parser.scorer]
@scorers = "spacy.parser_scorer.v1"
[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = true
[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3
[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"
[components.morphologizer.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"
[components.trainable_lemmatizer.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"
[components.parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

154
configs/fa_ner_lg.cfg Normal file
View File

@ -0,0 +1,154 @@
# fa_ent_news_lg: Persian NER with the lg-tier static floret vectors.
#
# Identical to configs/fa_ner_md.cfg (which is identical to fa_ner_sm.cfg except
# include_static_vectors: true). Only the vector table supplied at train time via
# --paths.vectors differs: fa_floret, 200k rows x 300d, floret mode, trained on the full
# Persian Wikipedia dump for 5 epochs (assets/vectors/fa_floret_lg), vs md's 50k rows x 300d
# trained on 400k Persian documents.
#
# Same embedded-tok2vec design as sm/md (no Tok2VecListener), so the trained component stays
# sourceable into a future fa_core_news_lg via `nlp.add_pipe("ner", source=...)`.
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
[system]
gpu_allocator = null
seed = 0
[nlp]
lang = "fa"
pipeline = ["ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1"
[nlp.vectors]
@vectors = "spacy.Vectors.v1"
[components]
[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100
incorrect_spans_key = null
[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null
[components.ner.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"
[components.ner.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.ner.model.tok2vec.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = true
[components.ner.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3
[components.ner.scorer]
@scorers = "spacy.ner_scorer.v1"
[corpora]
[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 400
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null
[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 1e-08
learn_rate = 0.001
[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null
[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0
[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false
[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null
[initialize.tokenizer]
[initialize.components]
[pretraining]

7
docs/CONTRIBUTING-GUIDE.md
View File

@ -56,7 +56,12 @@ From <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md>:
python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build wheel python -m spacy package training/fa_dep_news_sm packages --name dep_news_sm --version 3.8.0 --build wheel
python -m spacy huggingface-hub push packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl --org <org> python -m spacy huggingface-hub push packages/fa_dep_news_sm-3.8.0/dist/fa_dep_news_sm-3.8.0-py3-none-any.whl --org <org>
``` ```
Users then `pip install https://huggingface.co/<org>/fa_dep_news_sm/resolve/main/fa_dep_news_sm-any-py3-none-any.whl`. Users then `pip install https://huggingface.co/<org>/fa_dep_news_sm/resolve/main/fa_dep_news_sm-3.8.0-py3-none-any.whl`.
Note the filename: `spacy huggingface-hub push` uploads the wheel as `<name>-any-py3-none-any.whl`,
but `"any"` is not a valid PEP 440 version and current pip rejects it
(`Invalid wheel filename (invalid version)`). Upload a second copy under its real versioned
filename too (`api.upload_file(path_in_repo=f"{name}-{version}-py3-none-any.whl", ...)`) and
link to that one instead.
2. PyPI or a self-hosted wheel: `spacy package … --build sdist,wheel` then `twine upload`, or 2. PyPI or a self-hosted wheel: `spacy package … --build sdist,wheel` then `twine upload`, or
attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>. attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
3. spaCy Universe, which lists the package on spacy.io but hosts nothing. Per 3. spaCy Universe, which lists the package on spacy.io but hosts nothing. Per

252
docs/MODELS.md
View File

@ -21,6 +21,13 @@ requirement. That ordering sets the roadmap below.
Source: <https://spacy.io/models/en>. Source: <https://spacy.io/models/en>.
That last point does **not** transfer to Persian. The `sm` -> `md` step measured on this
project buys +1.19 LAS and +2.85 NER F (§6), where English gets ~0.00 LAS. Two reasons: PerDT
is roughly a tenth the size of OntoNotes, so hash embeddings have far less signal to learn a
lexicon from, and floret's subword hashing gives 0% OOV on a language whose ZWNJ variation
(می‌رود / میرود / می رود) fragments any fixed word-key table. English `md` uses 20k classic
word vectors and hits OOV constantly. Do not use the English row as the Persian prior.
## 2. Target: the Persian pipelines ## 2. Target: the Persian pipelines
Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>). The Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>). The
@ -35,9 +42,12 @@ pipelines such as `de_core_news_sm` as `news`.
| `fa_core_news_sm` | the above plus ner | hash embeddings | built, shipping | | `fa_core_news_sm` | the above plus ner | hash embeddings | built, shipping |
| `fa_ent_news_sm` | ner (own internal tok2vec) | hash embeddings | built, optional | | `fa_ent_news_sm` | ner (own internal tok2vec) | hash embeddings | built, optional |
| `fa_core_web_sm` | same as core, mixed-genre training data | hash embeddings | not built; would add ParsTwiNER to cover social media | | `fa_core_web_sm` | same as core, mixed-genre training data | hash embeddings | not built; would add ParsTwiNER to cover social media |
| `fa_core_news_md` | + static vectors | floret, 50k rows | vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) | | `fa_dep_news_md` | same as `fa_dep_news_sm` | floret, 50k rows / 300d | built, shipping |
| `fa_core_news_lg` | same | floret, 200k rows | same as md, bigger table | | `fa_core_news_md` | same as `fa_core_news_sm` | floret, 50k rows / 300d | built, shipping |
| `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | not on this hardware; 2 GB VRAM cannot fine-tune a 125M-param encoder | | `fa_dep_news_lg` | same as `fa_dep_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping |
| `fa_core_news_lg` | same as `fa_core_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping |
| `fa_ent_news_lg` | ner (own internal tok2vec) | floret, 200k rows / 300d, full-wiki 5 epochs | built, optional |
| `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/bert-base-parsbert-uncased`, fine-tuned | built on a rented Colab T4 (not on this hardware: 2 GB VRAM cannot fine-tune a 125M-param encoder), shipping with a redistribution caveat because that encoder's card states no licence; §3.4 and §8 |
### Why `core` is honest here ### Why `core` is honest here
@ -276,3 +286,239 @@ Sources are recorded in each `meta.json` with their licences, per
crediting Beheshti-NER. CC BY-SA 4.0 on PerDT means every package carries attribution and a crediting Beheshti-NER. CC BY-SA 4.0 on PerDT means every package carries attribution and a
share-alike notice, handled in `scripts/finalize_pipeline.py`, which also enforces the shape of share-alike notice, handled in `scripts/finalize_pipeline.py`, which also enforces the shape of
each variant: it refuses to publish a `dep` pipeline containing `ner`, or a `core` one without it. each variant: it refuses to publish a `dep` pipeline containing `ner`, or a `core` one without it.
## 6. The `md` tier: floret static vectors
Built after the `sm` tier, from `fa_floret`: 50,000 rows x 300d, floret mode, `minn=maxn=5`,
`hash_count=2`, trained on 400,000 Persian documents. The wheel is a vectors-only pipeline;
`scripts/unpack_vectors.py` unwraps it into a directory `--paths.vectors` can read, so nothing
needs pip-installing to train against it.
`configs/fa_dep_news_md.cfg` and `configs/fa_ner_md.cfg` are their `sm` counterparts with one
line changed, `include_static_vectors = false -> true`. Same seed, same corpus, same widths,
same batcher, same patience. The deltas below are therefore attributable to the vector table
and nothing else. Reproduce with `spacy project run md`, or the table alone with
`python scripts/compare_tiers.py`.
### UD_Persian-PerDT test split
| Metric | `sm` | `md` | Delta |
| --- | --- | --- | --- |
| `TAG_ACC` | 95.96 | 96.25 | +0.29 |
| `POS_ACC` | 96.24 | 96.64 | +0.40 |
| `MORPH_ACC` | 96.29 | 96.64 | +0.35 |
| `LEMMA_ACC` | 97.91 | 97.96 | +0.05 |
| `SENTS_F` | 99.25 | 99.28 | +0.03 |
| `DEP_UAS` | 89.69 | 90.52 | +0.83 |
| `DEP_LAS` | 85.15 | 86.34 | +1.19 |
| Speed (dep) | 12,505 w/s | 10,493 w/s | -16.1% |
### PerDT NER test split, `fa_core_news_md`
| Metric | `sm` | `md` | Delta |
| --- | --- | --- | --- |
| `ENTS_P` | 77.67 | 76.56 | -1.10 |
| `ENTS_R` | 66.87 | 72.95 | +6.08 |
| `ENTS_F` | 71.87 | 74.71 | +2.85 |
Almost all of the NER gain is recall. That is the expected shape of a fix for a coverage
problem: hash embeddings had no lexical prior for rare proper nouns, so the `sm` model
declined to tag them. Precision slips ~1 point because the model now guesses more.
| Label | Gold in test | `sm` F | `md` F | Delta |
| --- | --- | --- | --- | --- |
| `PER` | 297 | 65.29 | 68.18 | +2.89 |
| `LOC` | 273 | 80.24 | 84.05 | +3.81 |
| `ORG` | 144 | 68.77 | 70.25 | +1.48 |
| `DAT` | 69 | 74.45 | 76.19 | +1.74 |
| `MON` | 10 | 73.68 | 84.21 | +10.53 |
| `TIM` | 9 | 66.67 | 66.67 | +0.00 |
| `PCT` | 4 | 57.14 | 33.33 | -23.81 |
Read the bottom three rows as noise, not signal. `PCT` has four gold entities in the whole
test split, so its -23.81 F is one entity changing hands; `MON`'s +10.53 is likewise one of
ten. The three labels with real support (`PER`, `LOC`, `ORG`, 714 entities between them) all
improve, which is the finding.
### Cost
The vectors dominate the artifact: `fa_dep_news_md` is a 62 MB wheel against 7.5 MB for `sm`,
`fa_core_news_md` 68 MB against 13 MB. Inference is ~16% slower across all three pipelines,
a uniform hit consistent with the extra 300d concatenation per token rather than anything
component-specific. Training cost was comparable to `sm` (early stop at step 12,400 of 20,000,
best checkpoint near 10,800).
Whether that trade is worth it depends on deployment. For a 1.19 LAS and 2.85 NER F gain, a
9x larger download and 16% slower parse is a good deal on a server and a bad one in a browser
or a Lambda cold start. Both tiers ship; pick per target.
## 7. The `lg` tier: bigger floret table, full pipeline
Built after `md`, from a new `fa_floret` table: 200,000 rows x 300d, floret mode,
`minn=maxn=5`, `hash_count=2`, trained on the full Persian Wikipedia dump for 5 epochs (4x
the rows of `md`'s 50k-row table trained on 400k documents). Raw `.floret`/`.vec` and the
packaged spaCy wheel are at <https://huggingface.co/Phazel/fa-floret-wiki-vectors>. Unpacked
the same way as `md` via `scripts/unpack_vectors.py`, into `assets/vectors/fa_floret_lg`.
`configs/fa_ner_lg.cfg` and `configs/fa_dep_news_lg.cfg` are `fa_ner_md.cfg`/
`fa_dep_news_md.cfg` unchanged except `--paths.vectors`. Same seed, same corpus, same
architecture as `sm`/`md` throughout, so the deltas below are attributable to the vector
table alone. Reproduce with `spacy project run lg`, or the tables alone with
`python scripts/compare_tiers.py`.
### UD test split, `fa_dep_news_lg` / `fa_core_news_lg`
| Metric | `sm` | `md` | `lg` | Delta (lg vs sm) | Delta (lg vs md) |
| --- | --- | --- | --- | --- | --- |
| `TAG_ACC` | 95.96 | 96.25 | 96.55 | +0.59 | +0.30 |
| `POS_ACC` | 96.24 | 96.64 | 96.68 | +0.44 | +0.04 |
| `MORPH_ACC` | 96.29 | 96.64 | 96.70 | +0.41 | +0.06 |
| `LEMMA_ACC` | 97.91 | 97.96 | 98.08 | +0.17 | +0.12 |
| `DEP_UAS` | 89.69 | 90.52 | 90.96 | +1.27 | +0.44 |
| `DEP_LAS` | 85.15 | 86.34 | 86.60 | +1.45 | +0.26 |
`lg` beats `md` on every UD metric, the same monotonic pattern as `md` beating `sm` in §6.
The bigger, less collision-prone floret table keeps paying off, though the `md`-to-`lg`
gains (4x the vector rows) are smaller than the `sm`-to-`md` gains (going from none to 50k
rows): diminishing returns, as expected.
### PerDT NER test split, `fa_ent_news_lg` (identical `ner` component embedded in `fa_core_news_lg`)
| Metric | `sm` | `md` | `lg` | Delta (lg vs sm) | Delta (lg vs md) |
| --- | --- | --- | --- | --- | --- |
| `ENTS_P` | 77.67 | 76.56 | 81.51 | +3.84 | +4.95 |
| `ENTS_R` | 66.87 | 72.95 | 71.09 | +4.22 | -1.86 |
| `ENTS_F` | 71.87 | 74.71 | 75.94 | +4.08 | +1.23 |
`lg` beats both `sm` and `md` on `ENTS_F`, and unlike `md`'s recall-only gain over `sm`, `lg`
improves precision too (+3.84 over `sm`, whereas `md` cost -1.10). Consistent with a bigger,
less collision-prone floret table giving both better recall on rare proper nouns and fewer
false positives from hash collisions.
| Label | Gold in test | `sm` F | `md` F | `lg` F | Delta (lg vs sm) |
| --- | --- | --- | --- | --- | --- |
| `PER` | 297 | 65.29 | 68.18 | 72.63 | +7.33 |
| `LOC` | 273 | 80.24 | 84.05 | 83.66 | +3.42 |
| `ORG` | 144 | 68.77 | 70.25 | 71.01 | +2.24 |
| `DAT` | 69 | 74.45 | 76.19 | 70.83 | -3.62 |
| `MON` | 10 | 73.68 | 84.21 | 88.89 | +15.20 |
| `TIM` | 9 | 66.67 | 66.67 | 61.54 | -5.13 |
| `PCT` | 4 | 57.14 | 33.33 | 57.14 | +0.00 |
`PER`, `LOC` and `ORG` (714 entities, the labels with real support) all improve over both
smaller tiers. `DAT` and `TIM` regress a few points against `md`; `MON`/`TIM`/`PCT` swings are
one-or-two-entity noise, same caveat as §6.
### Cost
The bigger table dominates the artifact even more than `md`'s did: the 200k x 300d float32
vector table is ~240 MB uncompressed, so `fa_dep_news_lg` is a 219 MB wheel (vs 7.5 MB `sm`,
60 MB `md`), `fa_core_news_lg` 225 MB (vs 13 MB `sm`, 66 MB `md`), and `fa_ent_news_lg` alone
217 MB (vs 5.6 MB `sm`, 58 MB `md`). Training cost roughly doubled `md`'s: `dep_lg` ran to
early stop at step 12,000 of 20,000 over ~2h08m CPU wall time (vs `dep_md`'s single-digit
minutes territory implied by its architecture-identical config; `lg`'s extra time is
entirely the larger embedding table's per-step cost, not more steps). `ner_lg` early-stopped
at step 7,200, ~13 min, in line with `sm`/`md`.
`words/s` from `spacy benchmark accuracy` were noisier at this tier than `sm`-vs-`md`: dep/core
throughput dropped as expected (9,387 / 6,655 words/s vs `sm`'s 12,505 / 8,834, `md`'s
10,493 / 7,269 words/s; the larger table costs real lookup time), but the standalone `ent_lg` run
showed 15,614 words/s, higher than `sm`/`md`'s ent runs despite an identical `ner`
architecture and the same larger table. That figure was single-run CPU contention noise on
shared hardware, not a real speedup. Those numbers are superseded by §9, which times
`nlp.pipe` alone instead of reading a scoring-contaminated figure off the benchmark command.
For a 4x download over `md` (and up to 39x over `sm`) buying +1.45 DEP_LAS / +1.23 ENTS_F
over `md` (+1.45 DEP_LAS / +4.08 ENTS_F over `sm`), `lg` is a server/offline-batch pipeline,
not something to ship to a browser or a cold-start function. All three variants (`dep`,
`ent`, `core`) are built and evaluated at this tier, same as `md`.
## 8. The `trf` tier: one fine-tuned ParsBERT
`configs/fa_core_news_trf.cfg` replaces the static-vector tok2vec with
`HooshvareLab/bert-base-parsbert-uncased`, fine-tuned during training. Trained on a rented
Colab T4 in 1h58m: 3000 steps, no early stop, the full learning-rate anneal.
### One corpus, because a transformer cannot be trained twice
The `sm`/`md`/`lg` tiers train `ner` as its own pipeline with its own embedded tok2vec and
then source it into the dep model. That is affordable because a hash-embed tok2vec is cheap.
A 162M-parameter encoder is not: fine-tuning it once per component would double GPU cost and
put two encoders in one wheel, and sourcing the second would collide on the `transformer`
component name.
So every component listens to a single shared transformer through a `TransformerListener`,
which requires one corpus carrying both the UD and NER annotation layers on the same `Doc`.
`scripts/merge_joint_corpus.py` builds it. The fusion is exact rather than approximate:
`corpus/perdt-ner/` was converted from the same `--merge-subtokens` CoNLL-U as
`corpus/merged/` with the same `--n-sents`, so the two DocBins are token-for-token identical.
The script asserts that per document and copies only `doc.ents` across. Char offsets are not
usable for the copy, because the two converters differ in trailing whitespace, which shifts
`char_span` off the token grid and returns None; the transfer goes by token index.
### Results against `lg`
| Metric | `lg` | `trf` | Delta |
| --- | ---: | ---: | ---: |
| `TAG_ACC` | 96.55 | 97.62 | +1.07 |
| `POS_ACC` | 96.68 | 97.63 | +0.95 |
| `MORPH_ACC` | 96.70 | 97.82 | +1.12 |
| `LEMMA_ACC` | 98.08 | 97.31 | -0.77 |
| `DEP_UAS` | 90.96 | 93.87 | +2.91 |
| `DEP_LAS` | 86.60 | 90.79 | +4.19 |
| `SENTS_F` | 99.18 | 97.35 | -1.83 |
| `ENTS_F` | 75.94 | 82.89 | +6.95 |
The parser gain is the headline: `DEP_LAS` 90.79 passes the hazm+ParsBERT reference of 89.34,
which no CPU tier reached. NER gains 6.95 F, almost all of it recall (71.09 to 81.76) at
higher precision, which is what a pretrained encoder buys on the difflib-transferred layer.
Two metrics regress. `SENTS_F` drops 1.83, most likely because `strided_spans` at
`window = 128, stride = 96` leaves 32 tokens of overlap, so tokens near a span edge see
truncated right context where the CPU tiers' tok2vec sees the whole doc. `LEMMA_ACC` drops
0.77 and is the one metric where a static-vector tier wins: `trainable_lemmatizer` reads a
single `reduce_mean`-pooled vector per token, while `lg` runs an edit-tree lemmatizer over
floret subwords that model Persian orthography directly. Neither is a training-length
problem; see TODO.md for the evidence that more steps do not help.
### Cost, and the licence problem
608 MB wheel, 2.6x `lg` and 45x `sm`. 187 words/s on the laptop CPU against `sm`'s 5,484
(§9), so this tier needs a GPU in production rather than merely benefiting from one.
ParsBERT's model card states no licence. §3.4 picked `HooshvareLab/roberta-fa-zwnj-base`
(Apache-2.0) for exactly this reason, and the published wheel therefore embeds weights whose
redistribution terms are unknown. `scripts/finalize_pipeline.py` reads the encoder name out
of the trained config and writes a redistribution warning into `meta.json` when the encoder
has no licence, so the artifact carries the caveat. Retraining on the Apache-2.0 encoder is a
one-line change to `name` in the config.
## 9. Throughput
Measured with `scripts/benchmark_throughput.py`, which times `nlp.pipe` and nothing else.
The `words/s` printed by `spacy benchmark accuracy` runs the Scorer's per-token alignment
inside the timed region, which is why the §7 numbers disagree with these and why one of them
was impossible.
Median of repeated passes over the 146-document PerDT test split (23,825 tokens), batch 32,
warmup discarded. Raw records in `metrics/throughput-*.json`.
| Tier | CPU, i5-7200U | GPU, GeForce 940MX | CPU, Xeon @ 2.00GHz | GPU, Tesla T4 |
| --- | ---: | ---: | ---: | ---: |
| `sm` | 5,484 | 10,235 | | |
| `md` | 5,408 | 9,058 | | |
| `lg` | 4,715 | 9,215 | | |
| `trf` | 187 | 1,106 | 336 | 8,320 |
The CPU tiers sit within about 15% of each other, less than their vector-table sizes suggest,
so the tok2vec lookup is not the bottleneck; the parser and lemmatizer are. Run-to-run spread
on the laptop is roughly 10% either way with thermal state, and a background rsync halved
every number, so treat small differences as noise.
`trf` is 29x slower than `sm` on the same CPU. The T4 and Xeon columns come from the same Colab
VM, giving a clean 25x GPU speedup for the transformer.
`trf` on the 940MX needs a `cu126` torch build. sm_50 kernels were dropped from the `cu128` and
`cu129` wheels at torch 2.8, which is what `pip install torch` resolves to. `.venv-trf-gpu` pins
`torch==2.7.1+cu126`, separate from `.venv` because torch's pinned `nvidia-*` wheels downgrade
the CUDA libraries cupy uses there from 12.9 to 12.6. Batch 32 fits in 2 GB.

523
notebooks/fa_lg_trf_training.ipynb
View File

@ -1,523 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# fa_dep_news_lg / fa_core_news_lg / fa_dep_news_trf / fa_core_news_trf — Colab training\n",
"\n",
"Trains the `lg` (bigger floret vectors) and `trf` (fine-tuned transformer) tiers of the\n",
"Persian `spacy-fa-pipeline` project on a Colab GPU. `sm`/`md` are already built on CPU\n",
"locally — this notebook only adds the two tiers that need real GPU memory.\n",
"\n",
"**`trf` uses `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0), not ParsBERT** — ParsBERT's\n",
"model card carries no explicit licence, which is disqualifying for a package meant to be\n",
"redistributed. See `TODO.md` in the repo.\n",
"\n",
"Floret vector *training* itself (the actual `lg`-tier 200k-row Wikipedia+OSCAR table) is\n",
"not part of this notebook — that happens elsewhere (CPU-days, `spacy-vectors-builder`).\n",
"This notebook only trains spaCy pipelines against whatever floret wheel you upload in\n",
"step 6.\n",
"\n",
"## Before you run this\n",
"\n",
"1. **Runtime -> Change runtime type -> GPU** (a 16 GB T4/A10 is plenty for a base-size\n",
" transformer; no need for A100).\n",
"2. Have ready, to upload when asked:\n",
" - A zip of the repo's **source only** (`git archive -o repo.zip HEAD` from the repo\n",
" root -- this naturally excludes everything `.gitignore` excludes: `assets/ corpus/\n",
" training/ metrics/ packages/ .venv/`). The self-hosted Gitea remote is LAN-only and\n",
" unreachable from Colab, so this notebook cannot `git clone` it directly.\n",
" - An `lg`-tier floret wheel (`fa_floret-0.1.0-py3-none-any-*.whl`) once it's built\n",
" elsewhere. If you don't have one yet, upload whatever `md`-tier wheel you have as a\n",
" stand-in -- the run will still be valid, just not the final `lg` numbers.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Confirm the GPU"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!nvidia-smi\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Upload the repo source\n",
"\n",
"Upload the `repo.zip` produced by `git archive -o repo.zip HEAD` (run locally, in the repo\n",
"root, before starting this notebook).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from google.colab import files\n",
"import zipfile, pathlib\n",
"\n",
"REPO = pathlib.Path(\"/content/repo\")\n",
"REPO.mkdir(parents=True, exist_ok=True)\n",
"\n",
"uploaded = files.upload()\n",
"(zip_name,) = uploaded.keys()\n",
"with zipfile.ZipFile(zip_name) as z:\n",
" z.extractall(REPO)\n",
"\n",
"%cd {REPO}\n",
"!ls\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Install dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Colab ships a CUDA-enabled torch already; spacy[transformers] pulls in spacy-transformers +\n",
"# a matching transformers/tokenizers. Installing spacy[cuda-autodetect] too is cheap insurance\n",
"# for the GPU allocator path (unlike the local 940MX box, this doesn't need a manual cupy[ctk]\n",
"# CUDA-toolkit install -- Colab's base image already has the CUDA libs on the system path).\n",
"!pip install -q -U pip\n",
"!pip install -q \"spacy[transformers,cuda-autodetect]\" spacy-transformers spacy-lookups-data\n",
"\n",
"import spacy, torch, spacy_transformers\n",
"print(\"spacy\", spacy.__version__)\n",
"print(\"spacy-transformers\", spacy_transformers.__version__)\n",
"print(\"torch\", torch.__version__, \"cuda available:\", torch.cuda.is_available())\n",
"\n",
"from thinc.api import prefer_gpu\n",
"print(\"thinc prefer_gpu:\", prefer_gpu())\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Download the UD_Persian-PerDT assets\n",
"\n",
"Same public GitHub URLs and checksums as `project.yml` -- no private infrastructure needed.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import hashlib, urllib.request, pathlib\n",
"\n",
"ASSETS = [\n",
" (\"assets/ud/fa_perdt-ud-train.conllu\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-train.conllu\",\n",
" \"f5a8ba901a776b4fd1941ecadcc6d506\"),\n",
" (\"assets/ud/fa_perdt-ud-dev.conllu\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-dev.conllu\",\n",
" \"f103020da7c1e917aafb8a8321f4cb84\"),\n",
" (\"assets/ud/fa_perdt-ud-test.conllu\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-test.conllu\",\n",
" \"b62a66994cef2c50f7e524a1471102d8\"),\n",
" (\"assets/ud-ner/train_with_NER_tag.txt\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/train_with_NER_tag.txt\",\n",
" \"ecb96cf99b38bc485cac21d22914e413\"),\n",
" (\"assets/ud-ner/dev_with_NER_tag.txt\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/dev_with_NER_tag.txt\",\n",
" \"2a56ef7eb2e3732e221317af457d1c09\"),\n",
" (\"assets/ud-ner/test_with_NER_tag.txt\",\n",
" \"https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/not-to-release/Dadegan%20with%20NER%20tag/test_with_NER_tag.txt\",\n",
" \"6d80dd783527562c2ea5189f218a12b5\"),\n",
"]\n",
"\n",
"for dest, url, checksum in ASSETS:\n",
" dest = pathlib.Path(dest)\n",
" dest.parent.mkdir(parents=True, exist_ok=True)\n",
" urllib.request.urlretrieve(url, dest)\n",
" got = hashlib.md5(dest.read_bytes()).hexdigest()\n",
" status = \"OK\" if got == checksum else f\"MISMATCH (got {got})\"\n",
" print(f\"{dest}: {status}\")\n",
" assert got == checksum, f\"checksum mismatch on {dest}\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Build the UD + NER corpora (mirrors `project.yml`'s `convert-ud`/`transfer-ner`/`convert-ner`)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy convert assets/ud/fa_perdt-ud-train.conllu corpus/merged --converter conllu --n-sents 10 --merge-subtokens\n",
"!python -m spacy convert assets/ud/fa_perdt-ud-dev.conllu corpus/merged --converter conllu --n-sents 10 --merge-subtokens\n",
"!python -m spacy convert assets/ud/fa_perdt-ud-test.conllu corpus/merged --converter conllu --n-sents 10 --merge-subtokens\n",
"\n",
"!python scripts/transfer_perdt_ner.py --conllu-dir assets/ud --ner-dir assets/ud-ner --out corpus/perdt-ner-iob\n",
"\n",
"!python -m spacy convert corpus/perdt-ner-iob/train.txt corpus/perdt-ner --converter ner --n-sents 10 --lang fa\n",
"!python -m spacy convert corpus/perdt-ner-iob/dev.txt corpus/perdt-ner --converter ner --n-sents 10 --lang fa\n",
"!python -m spacy convert corpus/perdt-ner-iob/test.txt corpus/perdt-ner --converter ner --n-sents 10 --lang fa\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. `lg`-tier floret vectors\n",
"\n",
"Upload a floret wheel (`fa_floret-0.1.0-py3-none-any-*.whl`), built elsewhere. Use the real\n",
"200k-row Wikipedia+OSCAR table if you have one; otherwise upload whatever `md`-tier wheel\n",
"you have as a stand-in -- the run will still be valid, just not the final `lg` numbers.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from google.colab import files\n",
"\n",
"uploaded = files.upload()\n",
"(floret_wheel,) = uploaded.keys()\n",
"\n",
"!python scripts/unpack_vectors.py {floret_wheel} assets/vectors/fa_floret_lg\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Train the `lg` tier\n",
"\n",
"Byte-identical to the `md` recipe (`configs/fa_dep_news_md.cfg` / `configs/fa_ner_md.cfg`) --\n",
"only `--paths.vectors` changes, isolating the effect of the bigger table exactly the way\n",
"`md` isolated the effect of adding vectors over `sm`. No new config file needed.\n",
"\n",
"`--gpu-id 0` for both here: the earlier CPU-vs-GPU timing experiment ran on a 2 GB GTX 940MX,\n",
"where the small NER architecture's transfer/launch overhead beat its GPU compute win. A 16 GB\n",
"Colab GPU has far more bandwidth/compute headroom, so that conclusion may not hold here --\n",
"worth timing both `--gpu-id 0` and `--gpu-id -1` yourself if you want to confirm.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy train configs/fa_dep_news_md.cfg --output training/dep-lg \\\n",
" --paths.train corpus/merged/fa_perdt-ud-train.spacy \\\n",
" --paths.dev corpus/merged/fa_perdt-ud-dev.spacy \\\n",
" --paths.vectors assets/vectors/fa_floret_lg \\\n",
" --gpu-id 0\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy train configs/fa_ner_md.cfg --output training/perdt-ner-lg \\\n",
" --paths.train corpus/perdt-ner/train.spacy \\\n",
" --paths.dev corpus/perdt-ner/dev.spacy \\\n",
" --paths.vectors assets/vectors/fa_floret_lg \\\n",
" --gpu-id 0\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Assemble + evaluate `lg`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy benchmark accuracy training/dep-lg/model-best corpus/merged/fa_perdt-ud-test.spacy \\\n",
" --output metrics/lg-ud-test.json --gpu-id 0\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_dep_news_lg \\\n",
" --variant dep --size lg --version 3.8.0 --ud-metrics metrics/lg-ud-test.json\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_core_news_lg \\\n",
" --variant core --size lg --version 3.8.0 --add-ner training/perdt-ner-lg/model-best\n",
"\n",
"!python -m spacy benchmark accuracy training/fa_core_news_lg corpus/merged/fa_perdt-ud-test.spacy \\\n",
" --output metrics/lg-core-ud-test.json --gpu-id 0\n",
"!python -m spacy benchmark accuracy training/fa_core_news_lg corpus/perdt-ner/test.spacy \\\n",
" --output metrics/lg-perdt-ner-test.json --gpu-id 0\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_core_news_lg \\\n",
" --variant core --size lg --version 3.8.0 --add-ner training/perdt-ner-lg/model-best \\\n",
" --ud-metrics metrics/lg-core-ud-test.json --ner-metrics metrics/lg-perdt-ner-test.json\n",
"\n",
"!python scripts/smoke_test.py training/fa_core_news_lg\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Generate the `trf` configs\n",
"\n",
"`spacy init config --optimize accuracy -G` fills in a valid `spacy-transformers`\n",
"architecture automatically (letting spaCy own the schema instead of hand-writing one).\n",
"The only edit afterward is swapping the default transformer name for\n",
"`HooshvareLab/roberta-fa-zwnj-base` and turning on mixed precision, since 16 GB has room\n",
"for it.\n",
"\n",
"Same split as `sm`/`md`/`lg`: `dep` (tagger/morphologizer/lemmatizer/parser) and `ner`\n",
"trained as separate pipelines, each with its own transformer, so `ner` can be re-sourced\n",
"into `core` afterward exactly like the CPU tiers.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy init config configs/fa_dep_news_trf.cfg --lang fa \\\n",
" --pipeline tagger,morphologizer,trainable_lemmatizer,parser \\\n",
" --optimize accuracy -G --force\n",
"\n",
"!python -m spacy init config configs/fa_ner_trf.cfg --lang fa \\\n",
" --pipeline ner \\\n",
" --optimize accuracy -G --force\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"import re\n",
"\n",
"TRANSFORMER_NAME = \"HooshvareLab/roberta-fa-zwnj-base\"\n",
"\n",
"for path in [\"configs/fa_dep_news_trf.cfg\", \"configs/fa_ner_trf.cfg\"]:\n",
" text = open(path, encoding=\"utf8\").read()\n",
" # Swap whatever default transformer `init config` picked for roberta-fa-zwnj-base.\n",
" text = re.sub(\n",
" r'(\\[components\\.transformer\\.model\\]\\nname = )\"[^\"]+\"',\n",
" lambda m: m.group(1) + '\"' + TRANSFORMER_NAME + '\"',\n",
" text,\n",
" )\n",
" # 16 GB has room for mixed precision; halves activation memory, meaningfully faster.\n",
" if \"mixed_precision\" in text:\n",
" text = text.replace(\"mixed_precision = false\", \"mixed_precision = true\")\n",
" else:\n",
" text = text.replace(\"[training]\\n\", \"[training]\\nmixed_precision = true\\n\", 1)\n",
" open(path, \"w\", encoding=\"utf8\").write(text)\n",
" print(\"patched\", path, \"transformer =\", TRANSFORMER_NAME)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy debug config configs/fa_dep_news_trf.cfg \\\n",
" --paths.train corpus/merged/fa_perdt-ud-train.spacy \\\n",
" --paths.dev corpus/merged/fa_perdt-ud-dev.spacy\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Train the `trf` tier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy train configs/fa_dep_news_trf.cfg --output training/dep-trf \\\n",
" --paths.train corpus/merged/fa_perdt-ud-train.spacy \\\n",
" --paths.dev corpus/merged/fa_perdt-ud-dev.spacy \\\n",
" --gpu-id 0\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy train configs/fa_ner_trf.cfg --output training/perdt-ner-trf \\\n",
" --paths.train corpus/perdt-ner/train.spacy \\\n",
" --paths.dev corpus/perdt-ner/dev.spacy \\\n",
" --gpu-id 0\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Assemble + evaluate `trf`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy benchmark accuracy training/dep-trf/model-best corpus/merged/fa_perdt-ud-test.spacy \\\n",
" --output metrics/trf-ud-test.json --gpu-id 0\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-trf/model-best training/fa_dep_news_trf \\\n",
" --variant dep --size trf --version 3.8.0 --ud-metrics metrics/trf-ud-test.json\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-trf/model-best training/fa_core_news_trf \\\n",
" --variant core --size trf --version 3.8.0 --add-ner training/perdt-ner-trf/model-best\n",
"\n",
"!python -m spacy benchmark accuracy training/fa_core_news_trf corpus/merged/fa_perdt-ud-test.spacy \\\n",
" --output metrics/trf-core-ud-test.json --gpu-id 0\n",
"!python -m spacy benchmark accuracy training/fa_core_news_trf corpus/perdt-ner/test.spacy \\\n",
" --output metrics/trf-perdt-ner-test.json --gpu-id 0\n",
"\n",
"!python scripts/finalize_pipeline.py training/dep-trf/model-best training/fa_core_news_trf \\\n",
" --variant core --size trf --version 3.8.0 --add-ner training/perdt-ner-trf/model-best \\\n",
" --ud-metrics metrics/trf-core-ud-test.json --ner-metrics metrics/trf-perdt-ner-test.json\n",
"\n",
"!python scripts/smoke_test.py training/fa_core_news_trf\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 12. Compare every tier\n",
"\n",
"Reads whichever `metrics/*-ud-test.json` / `metrics/*-perdt-ner-test.json` files exist in\n",
"this Colab session (only `lg` and `trf`, produced above). To compare against the local\n",
"`sm`/`md` numbers, upload `metrics/core-ud-test.json`, `metrics/perdt-ner-test.json`,\n",
"`metrics/md-core-ud-test.json`, `metrics/md-perdt-ner-test.json` from the repo first.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json, pathlib\n",
"\n",
"ROWS = [\n",
" (\"sm\", \"metrics/core-ud-test.json\", \"metrics/perdt-ner-test.json\"),\n",
" (\"md\", \"metrics/md-core-ud-test.json\", \"metrics/md-perdt-ner-test.json\"),\n",
" (\"lg\", \"metrics/lg-core-ud-test.json\", \"metrics/lg-perdt-ner-test.json\"),\n",
" (\"trf\", \"metrics/trf-core-ud-test.json\", \"metrics/trf-perdt-ner-test.json\"),\n",
"]\n",
"\n",
"def load(path):\n",
" p = pathlib.Path(path)\n",
" return json.loads(p.read_text()) if p.exists() else None\n",
"\n",
"print(f\"{'tier':<5}{'tag_acc':>9}{'dep_las':>9}{'lemma_acc':>11}{'ents_f':>9}\")\n",
"for tier, ud_path, ner_path in ROWS:\n",
" ud, ner = load(ud_path), load(ner_path)\n",
" tag = f\"{ud['tag_acc']*100:.2f}\" if ud and ud.get('tag_acc') is not None else \"-\"\n",
" las = f\"{ud['dep_las']*100:.2f}\" if ud and ud.get('dep_las') is not None else \"-\"\n",
" lem = f\"{ud['lemma_acc']*100:.2f}\" if ud and ud.get('lemma_acc') is not None else \"-\"\n",
" entf = f\"{ner['ents_f']*100:.2f}\" if ner and ner.get('ents_f') is not None else \"-\"\n",
" print(f\"{tier:<5}{tag:>9}{las:>9}{lem:>11}{entf:>9}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 13. Download the results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from google.colab import files\n",
"\n",
"!zip -r /content/lg_trf_results.zip training/fa_dep_news_lg training/fa_core_news_lg \\\n",
" training/fa_dep_news_trf training/fa_core_news_trf metrics assets/vectors/fa_floret_lg \\\n",
" configs/fa_dep_news_trf.cfg configs/fa_ner_trf.cfg\n",
"\n",
"files.download(\"/content/lg_trf_results.zip\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 14. (Optional) package as installable wheels"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!python -m spacy package training/fa_dep_news_lg packages --name dep_news_lg --version 3.8.0 --build sdist,wheel --force\n",
"!python -m spacy package training/fa_core_news_lg packages --name core_news_lg --version 3.8.0 --build sdist,wheel --force\n",
"!python -m spacy package training/fa_dep_news_trf packages --name dep_news_trf --version 3.8.0 --build sdist,wheel --force\n",
"!python -m spacy package training/fa_core_news_trf packages --name core_news_trf --version 3.8.0 --build sdist,wheel --force\n",
"\n",
"!zip -r /content/packages.zip packages\n",
"files.download(\"/content/packages.zip\")\n"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"name": "fa_lg_trf_training.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

409
project.yml
View File

@ -29,6 +29,22 @@ vars:
# -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline. # -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline.
gpu: -1 gpu: -1
n_sents: 10 n_sents: 10
# md tier. Same architecture as sm plus the fa_floret static vector table.
dep_md_package_name: "dep_news_md"
core_md_package_name: "core_news_md"
floret_wheel: "fa_floret-0.1.0-py3-none-any-400k-documents.whl"
vectors_dir: "assets/vectors/fa_floret_400k"
# lg tier: same architecture as sm/md, larger floret table (200k rows x 300d, trained on
# the full Persian Wikipedia dump for 5 epochs, vs md's 50k rows / 400k documents).
ent_lg_package_name: "ent_news_lg"
dep_lg_package_name: "dep_news_lg"
core_lg_package_name: "core_news_lg"
floret_lg_wheel: "fa_floret-0.1.0-py3-none-any-full-wiki-200k-5epoch.whl"
vectors_lg_dir: "assets/vectors/fa_floret_lg"
# trf tier: one fine-tuned ParsBERT shared by every component. Needs a real GPU; the
# 940MX cannot fine-tune a 162M-parameter encoder, so `gpu_trf` is set for a rented card.
core_trf_package_name: "core_news_trf"
gpu_trf: 0
directories: directories:
- "assets" - "assets"
@ -88,6 +104,42 @@ workflows:
- finalize-ent - finalize-ent
- evaluate-ent - evaluate-ent
- package-ent - package-ent
# The lg tier: same corpus and architecture as sm/md, with a bigger floret table (200k
# rows, full Persian Wikipedia, 5 epochs) than md's (50k rows, 400k documents).
lg:
- vectors-lg
- train-dep-lg
- train-ner-lg
- finalize-dep-lg
- assemble-core-lg
- evaluate-lg
- finalize-meta-lg
- compare-lg
- package-lg
- smoke-lg
# The md tier: same corpus and architecture, plus the fa_floret static vectors.
md:
- vectors-md
- train-dep-md
- train-ner-md
- finalize-dep-md
- assemble-core-md
- evaluate-md
- finalize-meta-md
- compare-md
- package-md
- smoke-md
# The trf tier: one fine-tuned ParsBERT shared by every component, including ner, so it
# trains against a single joint corpus instead of the sm/md/lg dep+ner split. GPU only.
trf:
- merge-joint
- debug-data-trf
- train-trf
- finalize-trf
- evaluate-trf
- finalize-meta-trf
- package-trf
- smoke-trf
commands: commands:
- name: "inspect" - name: "inspect"
@ -297,6 +349,363 @@ commands:
outputs: outputs:
- "packages/${vars.lang}_${vars.ent_package_name}-${vars.package_version}" - "packages/${vars.lang}_${vars.ent_package_name}-${vars.package_version}"
# ---------------------------------------------------------------- lg tier (ner only)
- name: "vectors-lg"
help: >
Unpack the lg-tier fa_floret wheel into a plain spaCy model directory. 200k rows x
300d in floret mode, trained on the full Persian Wikipedia dump for 5 epochs, vs
vectors-md's 50k rows / 400k documents.
script:
- "python scripts/unpack_vectors.py ${vars.floret_lg_wheel} ${vars.vectors_lg_dir}"
deps:
- "${vars.floret_lg_wheel}"
- "scripts/unpack_vectors.py"
outputs:
- "${vars.vectors_lg_dir}"
- name: "train-dep-lg"
help: "Train the dep pipeline with the lg-tier static floret vectors"
script:
- "python -m spacy train configs/fa_dep_news_lg.cfg --output training/dep-lg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --paths.vectors ${vars.vectors_lg_dir} --gpu-id ${vars.gpu}"
deps:
- "corpus/merged/${vars.treebank}-ud-train.spacy"
- "corpus/merged/${vars.treebank}-ud-dev.spacy"
- "configs/fa_dep_news_lg.cfg"
- "${vars.vectors_lg_dir}"
outputs:
- "training/dep-lg/model-best"
- name: "train-ner-lg"
help: "Train the NER component with the lg-tier static floret vectors"
script:
- "python -m spacy train configs/fa_ner_lg.cfg --output training/perdt-ner-lg --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy --paths.vectors ${vars.vectors_lg_dir} --gpu-id ${vars.gpu}"
deps:
- "corpus/perdt-ner/train.spacy"
- "corpus/perdt-ner/dev.spacy"
- "configs/fa_ner_lg.cfg"
- "${vars.vectors_lg_dir}"
outputs:
- "training/perdt-ner-lg/model-best"
- name: "finalize-ent-lg"
help: "Write fa_ent_news_lg metadata onto the trained lg model"
script:
- "python scripts/finalize_pipeline.py training/perdt-ner-lg/model-best training/fa_ent_news_lg --variant ent --size lg --version ${vars.package_version}"
deps:
- "training/perdt-ner-lg/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_ent_news_lg"
- name: "finalize-dep-lg"
help: "Write fa_dep_news_lg metadata onto the trained lg model"
script:
- "python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_dep_news_lg --variant dep --size lg --version ${vars.package_version}"
deps:
- "training/dep-lg/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_dep_news_lg"
- name: "assemble-core-lg"
help: "Source the lg ner into the lg dep pipeline to produce fa_core_news_lg"
script:
- "python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_core_news_lg --variant core --size lg --version ${vars.package_version} --add-ner training/perdt-ner-lg/model-best"
deps:
- "training/dep-lg/model-best"
- "training/perdt-ner-lg/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_core_news_lg"
- name: "evaluate-ent-lg"
help: "Score fa_ent_news_lg on the held-out PerDT NER test split"
script:
- "python -m spacy benchmark accuracy training/fa_ent_news_lg corpus/perdt-ner/test.spacy --output metrics/lg-perdt-ner-test.json --gpu-id ${vars.gpu}"
- "python scripts/finalize_pipeline.py training/perdt-ner-lg/model-best training/fa_ent_news_lg --variant ent --size lg --version ${vars.package_version} --ner-metrics metrics/lg-perdt-ner-test.json"
deps:
- "training/fa_ent_news_lg"
- "corpus/perdt-ner/test.spacy"
outputs:
- "metrics/lg-perdt-ner-test.json"
- name: "evaluate-lg"
help: "Score both lg packages (dep, core) on the held-out test splits"
script:
- "python -m spacy benchmark accuracy training/fa_dep_news_lg corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/lg-ud-test.json --gpu-id ${vars.gpu}"
- "python -m spacy benchmark accuracy training/fa_core_news_lg corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/lg-core-ud-test.json --gpu-id ${vars.gpu}"
- "python -m spacy benchmark accuracy training/fa_core_news_lg corpus/perdt-ner/test.spacy --output metrics/lg-core-perdt-ner-test.json --gpu-id ${vars.gpu}"
deps:
- "training/fa_dep_news_lg"
- "training/fa_core_news_lg"
outputs:
- "metrics/lg-ud-test.json"
- "metrics/lg-core-ud-test.json"
- "metrics/lg-core-perdt-ner-test.json"
- name: "finalize-meta-lg"
help: "Fold the lg test scores into both lg meta.json files"
script:
- "python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_dep_news_lg --variant dep --size lg --version ${vars.package_version} --ud-metrics metrics/lg-ud-test.json"
- "python scripts/finalize_pipeline.py training/dep-lg/model-best training/fa_core_news_lg --variant core --size lg --version ${vars.package_version} --add-ner training/perdt-ner-lg/model-best --ud-metrics metrics/lg-core-ud-test.json --ner-metrics metrics/lg-core-perdt-ner-test.json"
deps:
- "metrics/lg-ud-test.json"
- "metrics/lg-core-perdt-ner-test.json"
- "scripts/finalize_pipeline.py"
- name: "compare-lg"
help: "Table the sm vs md vs lg deltas from the metrics/ JSON reports"
script:
- "python scripts/compare_tiers.py"
deps:
- "metrics/ud-test.json"
- "metrics/md-ud-test.json"
- "metrics/lg-ud-test.json"
- "metrics/perdt-ner-test.json"
- "metrics/md-perdt-ner-test.json"
- "metrics/lg-perdt-ner-test.json"
- "scripts/compare_tiers.py"
- name: "package-ent-lg"
help: "Build the installable fa_ent_news_lg wheel + sdist"
script:
- "python -m spacy package training/fa_ent_news_lg packages --name ${vars.ent_lg_package_name} --version ${vars.package_version} --build sdist,wheel --force"
deps:
- "training/fa_ent_news_lg"
outputs:
- "packages/${vars.lang}_${vars.ent_lg_package_name}-${vars.package_version}"
- name: "package-lg"
help: "Build installable wheels + sdists for both lg packages"
script:
- "python -m spacy package training/fa_dep_news_lg packages --name ${vars.dep_lg_package_name} --version ${vars.package_version} --build sdist,wheel --force"
- "python -m spacy package training/fa_core_news_lg packages --name ${vars.core_lg_package_name} --version ${vars.package_version} --build sdist,wheel --force"
deps:
- "training/fa_dep_news_lg"
- "training/fa_core_news_lg"
outputs:
- "packages/${vars.lang}_${vars.dep_lg_package_name}-${vars.package_version}"
- "packages/${vars.lang}_${vars.core_lg_package_name}-${vars.package_version}"
- name: "smoke-ent-lg"
help: "Load fa_ent_news_lg and run it over real Persian text"
script:
- "python scripts/smoke_test.py training/fa_ent_news_lg"
deps:
- "training/fa_ent_news_lg"
- name: "smoke-lg"
help: "Load both lg pipelines and run them over real Persian text"
script:
- "python scripts/smoke_test.py training/fa_dep_news_lg"
- "python scripts/smoke_test.py training/fa_core_news_lg"
deps:
- "training/fa_dep_news_lg"
- "training/fa_core_news_lg"
# ---------------------------------------------------------------- md tier
- name: "vectors-md"
help: >
Unpack the fa_floret wheel into a plain spaCy model directory that
`--paths.vectors` can point at. The wheel is a vectors-only pipeline
(empty `pipeline: []`), 50k rows x 300d in floret mode, trained on 400k
Persian documents, so no `spacy init vectors` step is needed.
script:
- "python scripts/unpack_vectors.py ${vars.floret_wheel} ${vars.vectors_dir}"
deps:
- "${vars.floret_wheel}"
- "scripts/unpack_vectors.py"
outputs:
- "${vars.vectors_dir}"
- name: "train-dep-md"
help: "Train the dep pipeline with static floret vectors"
script:
- "python -m spacy train configs/fa_dep_news_md.cfg --output training/dep-md --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --paths.vectors ${vars.vectors_dir} --gpu-id ${vars.gpu}"
deps:
- "corpus/merged/${vars.treebank}-ud-train.spacy"
- "corpus/merged/${vars.treebank}-ud-dev.spacy"
- "configs/fa_dep_news_md.cfg"
- "${vars.vectors_dir}"
outputs:
- "training/dep-md/model-best"
- name: "train-ner-md"
help: "Train the NER component with static floret vectors"
script:
- "python -m spacy train configs/fa_ner_md.cfg --output training/perdt-ner-md --paths.train corpus/perdt-ner/train.spacy --paths.dev corpus/perdt-ner/dev.spacy --paths.vectors ${vars.vectors_dir} --gpu-id ${vars.gpu}"
deps:
- "corpus/perdt-ner/train.spacy"
- "corpus/perdt-ner/dev.spacy"
- "configs/fa_ner_md.cfg"
- "${vars.vectors_dir}"
outputs:
- "training/perdt-ner-md/model-best"
- name: "finalize-dep-md"
help: "Write fa_dep_news_md metadata onto the trained md model"
script:
- "python scripts/finalize_pipeline.py training/dep-md/model-best training/fa_dep_news_md --variant dep --size md --version ${vars.package_version}"
deps:
- "training/dep-md/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_dep_news_md"
- name: "assemble-core-md"
help: "Source the md ner into the md dep pipeline to produce fa_core_news_md"
script:
- "python scripts/finalize_pipeline.py training/dep-md/model-best training/fa_core_news_md --variant core --size md --version ${vars.package_version} --add-ner training/perdt-ner-md/model-best"
deps:
- "training/dep-md/model-best"
- "training/perdt-ner-md/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_core_news_md"
- name: "evaluate-md"
help: "Score both md packages on the held-out test splits"
script:
- "python -m spacy benchmark accuracy training/fa_dep_news_md corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/md-ud-test.json --gpu-id ${vars.gpu}"
- "python -m spacy benchmark accuracy training/fa_core_news_md corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/md-core-ud-test.json --gpu-id ${vars.gpu}"
- "python -m spacy benchmark accuracy training/fa_core_news_md corpus/perdt-ner/test.spacy --output metrics/md-perdt-ner-test.json --gpu-id ${vars.gpu}"
deps:
- "training/fa_dep_news_md"
- "training/fa_core_news_md"
outputs:
- "metrics/md-ud-test.json"
- "metrics/md-core-ud-test.json"
- "metrics/md-perdt-ner-test.json"
- name: "finalize-meta-md"
help: "Fold the md test scores into both md meta.json files"
script:
- "python scripts/finalize_pipeline.py training/dep-md/model-best training/fa_dep_news_md --variant dep --size md --version ${vars.package_version} --ud-metrics metrics/md-ud-test.json"
- "python scripts/finalize_pipeline.py training/dep-md/model-best training/fa_core_news_md --variant core --size md --version ${vars.package_version} --add-ner training/perdt-ner-md/model-best --ud-metrics metrics/md-core-ud-test.json --ner-metrics metrics/md-perdt-ner-test.json"
deps:
- "metrics/md-ud-test.json"
- "metrics/md-perdt-ner-test.json"
- "scripts/finalize_pipeline.py"
- name: "compare-md"
help: "Table the sm vs md deltas from the metrics/ JSON reports"
script:
- "python scripts/compare_tiers.py"
deps:
- "metrics/md-ud-test.json"
- "metrics/md-perdt-ner-test.json"
- "scripts/compare_tiers.py"
- name: "package-md"
help: "Build installable wheels + sdists for both md packages"
script:
- "python -m spacy package training/fa_dep_news_md packages --name ${vars.dep_md_package_name} --version ${vars.package_version} --build sdist,wheel --force"
- "python -m spacy package training/fa_core_news_md packages --name ${vars.core_md_package_name} --version ${vars.package_version} --build sdist,wheel --force"
deps:
- "training/fa_dep_news_md"
- "training/fa_core_news_md"
outputs:
- "packages/${vars.lang}_${vars.dep_md_package_name}-${vars.package_version}"
- "packages/${vars.lang}_${vars.core_md_package_name}-${vars.package_version}"
- name: "smoke-md"
help: "Load both md pipelines and run them over real Persian text"
script:
- "python scripts/smoke_test.py training/fa_dep_news_md"
- "python scripts/smoke_test.py training/fa_core_news_md"
deps:
- "training/fa_dep_news_md"
- "training/fa_core_news_md"
# ---------------------------------------------------------------- trf tier
- name: "merge-joint"
help: >
Fuse the UD layer and the transferred NER layer onto one set of Docs. The trf tier
shares a single transformer across every component, so it needs one corpus carrying
both annotation layers; the two DocBins are token-for-token identical by construction
and the script asserts it.
script:
- "python scripts/merge_joint_corpus.py --ud-dir corpus/merged --ner-dir corpus/perdt-ner --out corpus/joint"
deps:
- "corpus/merged/${vars.treebank}-ud-train.spacy"
- "corpus/perdt-ner/train.spacy"
- "scripts/merge_joint_corpus.py"
outputs:
- "corpus/joint/train.spacy"
- "corpus/joint/dev.spacy"
- "corpus/joint/test.spacy"
- name: "debug-data-trf"
help: "Validate the joint corpus against the trf config before renting GPU time"
script:
- "python -m spacy debug data configs/fa_core_news_trf.cfg --paths.train corpus/joint/train.spacy --paths.dev corpus/joint/dev.spacy"
deps:
- "corpus/joint/train.spacy"
- "configs/fa_core_news_trf.cfg"
- name: "train-trf"
help: "Fine-tune ParsBERT with tagger + morphologizer + lemmatizer + parser + ner listening"
script:
- "python -m spacy train configs/fa_core_news_trf.cfg --output training/core-trf --paths.train corpus/joint/train.spacy --paths.dev corpus/joint/dev.spacy --gpu-id ${vars.gpu_trf}"
deps:
- "corpus/joint/train.spacy"
- "corpus/joint/dev.spacy"
- "configs/fa_core_news_trf.cfg"
outputs:
- "training/core-trf/model-best"
- name: "finalize-trf"
help: "Write fa_core_news_trf metadata onto the trained model"
script:
- "python scripts/finalize_pipeline.py training/core-trf/model-best training/fa_core_news_trf --variant core --size trf --version ${vars.package_version}"
deps:
- "training/core-trf/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_core_news_trf"
- name: "evaluate-trf"
help: "Score fa_core_news_trf on the held-out UD and NER test splits"
script:
- "python -m spacy benchmark accuracy training/fa_core_news_trf corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/trf-core-ud-test.json --gpu-id ${vars.gpu_trf}"
- "python -m spacy benchmark accuracy training/fa_core_news_trf corpus/perdt-ner/test.spacy --output metrics/trf-perdt-ner-test.json --gpu-id ${vars.gpu_trf}"
deps:
- "training/fa_core_news_trf"
- "corpus/merged/${vars.treebank}-ud-test.spacy"
- "corpus/perdt-ner/test.spacy"
outputs:
- "metrics/trf-core-ud-test.json"
- "metrics/trf-perdt-ner-test.json"
- name: "finalize-meta-trf"
help: "Fold the trf test scores into meta.json"
script:
- "python scripts/finalize_pipeline.py training/core-trf/model-best training/fa_core_news_trf --variant core --size trf --version ${vars.package_version} --ud-metrics metrics/trf-core-ud-test.json --ner-metrics metrics/trf-perdt-ner-test.json"
deps:
- "metrics/trf-core-ud-test.json"
- "metrics/trf-perdt-ner-test.json"
- "scripts/finalize_pipeline.py"
- name: "package-trf"
help: "Build the installable fa_core_news_trf wheel + sdist"
script:
- "python -m spacy package training/fa_core_news_trf packages --name ${vars.core_trf_package_name} --version ${vars.package_version} --build sdist,wheel --force"
deps:
- "training/fa_core_news_trf"
outputs:
- "packages/${vars.lang}_${vars.core_trf_package_name}-${vars.package_version}"
- name: "smoke-trf"
help: "Load fa_core_news_trf and run it over real Persian text"
script:
- "python scripts/smoke_test.py training/fa_core_news_trf"
deps:
- "training/fa_core_news_trf"
- name: "clean" - name: "clean"
help: "Drop corpora, training runs and metrics (keeps downloaded assets)" help: "Drop corpora, training runs and metrics (keeps downloaded assets)"
script: script:

112
scripts/benchmark_throughput.py Normal file
View File

@ -0,0 +1,112 @@
"""Measure inference throughput (words/second) for a pipeline, on CPU or GPU.
`spacy benchmark accuracy` prints a speed number, but it is scoring-contaminated: the
Scorer's per-token alignment and per-type bookkeeping run inside the timed region, which
matters a lot for the cheap CPU tiers and understates them. This times `nlp.pipe` only.
Reported figure is the median of `--runs` passes over the same texts, after a discarded
warmup pass. Median rather than mean because the first CUDA kernel launches, cuBLAS
autotuning and any page-cache miss produce outliers that a mean would smear into the result.
Batch size matters far more for the trf tier than the CPU tiers (a transformer amortizes a
GEMM over the batch; a hash-embed tok2vec barely cares), so it is a parameter and gets
recorded in the output rather than being left implicit.
"""
import argparse
import json
import platform
import statistics
import subprocess
import time
from pathlib import Path
import spacy
from spacy.tokens import DocBin
def cpu_model():
try:
for line in Path("/proc/cpuinfo").read_text().splitlines():
if line.startswith("model name"):
return line.split(":", 1)[1].strip()
except OSError:
pass
return platform.processor() or "unknown"
def gpu_model():
try:
out = subprocess.run(
["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader"],
capture_output=True, text=True, timeout=30,
)
if out.returncode == 0:
return out.stdout.strip().splitlines()[0].strip()
except (OSError, subprocess.SubprocessError):
pass
return "unknown"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("model", help="installed package name or path to a pipeline")
ap.add_argument("--corpus", default="corpus/merged/fa_perdt-ud-test.spacy",
help="DocBin whose raw texts are used as input")
ap.add_argument("--gpu-id", type=int, default=-1, help="-1 for CPU")
ap.add_argument("--batch-size", type=int, default=32)
ap.add_argument("--runs", type=int, default=3)
ap.add_argument("--limit", type=int, default=0, help="cap number of docs (0 = all)")
ap.add_argument("--output", default=None, help="write a JSON record here")
args = ap.parse_args()
if args.gpu_id >= 0:
# require_gpu, not prefer_gpu: a silent fall back to CPU would be reported as a GPU
# number, which is exactly the measurement error this script exists to avoid.
spacy.require_gpu(args.gpu_id)
device = f"gpu:{args.gpu_id} ({gpu_model()})"
else:
device = f"cpu ({cpu_model()})"
nlp = spacy.load(args.model)
vocab_docs = list(DocBin().from_disk(args.corpus).get_docs(spacy.blank("fa").vocab))
if args.limit:
vocab_docs = vocab_docs[:args.limit]
texts = [d.text for d in vocab_docs]
n_words = sum(len(d) for d in vocab_docs)
# Warmup: first pass pays for lazy CUDA context creation, cuBLAS handles and any
# transformer weight transfer. Timing it would misattribute setup cost to throughput.
for _ in nlp.pipe(texts[:args.batch_size], batch_size=args.batch_size):
pass
wps = []
for _ in range(args.runs):
t0 = time.perf_counter()
for _ in nlp.pipe(texts, batch_size=args.batch_size):
pass
elapsed = time.perf_counter() - t0
wps.append(n_words / elapsed)
median = statistics.median(wps)
record = {
"model": args.model,
"pipeline": list(nlp.pipe_names),
"device": device,
"batch_size": args.batch_size,
"docs": len(texts),
"words": n_words,
"runs": [round(w, 1) for w in wps],
"wps_median": round(median, 1),
"spacy_version": spacy.__version__,
}
print(json.dumps(record, indent=2, ensure_ascii=False))
if args.output:
p = Path(args.output)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(record, indent=2, ensure_ascii=False) + "\n")
print(f"wrote {p}")
if __name__ == "__main__":
main()

166
scripts/compare_tiers.py Normal file
View File

@ -0,0 +1,166 @@
"""Table the sm vs md vs lg test-set deltas.
All tiers are trained from the same corpus, the same seed and the same architecture; the
only difference is the static vector table (none for sm, fa_floret 50k rows for md, fa_floret
200k rows for lg) via `include_static_vectors`. So the delta printed here is attributable to
the vector table and nothing else.
Reads the `spacy benchmark accuracy` reports written by the `evaluate-*` targets. Missing
files are reported rather than fatal, so this is runnable mid-build.
Usage:
python scripts/compare_tiers.py [--metrics-dir metrics]
"""
import argparse
import json
from pathlib import Path
# (label, {tier_label: report_filename})
GROUPS = [
(
"dep pipeline, UD test",
{"sm": "ud-test.json", "md": "md-ud-test.json", "lg": "lg-ud-test.json"},
),
(
"core pipeline, UD test",
{
"sm": "core-ud-test.json",
"md": "md-core-ud-test.json",
"lg": "lg-core-ud-test.json",
},
),
(
"ent NER test",
{
"sm": "perdt-ner-test.json",
"md": "md-perdt-ner-test.json",
"lg": "lg-perdt-ner-test.json",
},
),
]
SCALARS = [
("tag_acc", "TAG_ACC"),
("pos_acc", "POS_ACC"),
("morph_acc", "MORPH_ACC"),
("lemma_acc", "LEMMA_ACC"),
("dep_uas", "DEP_UAS"),
("dep_las", "DEP_LAS"),
("sents_f", "SENTS_F"),
("ents_p", "ENTS_P"),
("ents_r", "ENTS_R"),
("ents_f", "ENTS_F"),
]
def load(path):
return json.loads(path.read_text(encoding="utf8")) if path.exists() else None
def table(title, tiers, rows):
"""tiers: list of (label, data-dict-or-None), first tier is the baseline for deltas."""
labels = [label for label, _ in tiers]
base_label, base = tiers[0]
print(f"\n## {title}\n")
header = " | ".join(f"{label:>7}" for label in labels)
delta_header = " | ".join(f"{'d(' + label + ')':>9}" for label, _ in tiers[1:])
print(f"| {'metric':<12} | {header} | {delta_header} |")
sep = " | ".join("-" * 7 for _ in labels)
delta_sep = " | ".join("-" * 9 for _ in tiers[1:])
print(f"| {'-' * 12} | {sep} | {delta_sep} |")
for key, label in rows:
values = [d.get(key) if d is not None else None for _, d in tiers]
if all(v is None for v in values):
continue
# The NER report scores tag_acc 0.0 because its corpus has no gold tags.
if all(v == 0.0 for v in values):
continue
cells = [f"{v * 100:.2f}" if isinstance(v, float) else "-" for v in values]
deltas = []
for v in values[1:]:
a, b = values[0], v
deltas.append(
f"{(b - a) * 100:+.2f}" if isinstance(a, float) and isinstance(b, float) else "-"
)
row = " | ".join(f"{c:>7}" for c in cells)
drow = " | ".join(f"{d:>9}" for d in deltas)
print(f"| {label:<12} | {row} | {drow} |")
speeds = [d.get("speed") if d is not None else None for _, d in tiers]
if isinstance(speeds[0], float):
cells = [f"{s:.0f}" if isinstance(s, float) else "-" for s in speeds]
deltas = [
f"{s / speeds[0] - 1:+.1%}" if isinstance(s, float) else "-" for s in speeds[1:]
]
row = " | ".join(f"{c:>7}" for c in cells)
drow = " | ".join(f"{d:>9}" for d in deltas)
print(f"| {'words/s':<12} | {row} | {drow} |")
def per_type(title, tiers):
per_types = [(label, (d or {}).get("ents_per_type")) for label, d in tiers]
if not any(pt for _, pt in per_types):
return
labels = [label for label, _ in tiers]
def pct(v):
return f"{v * 100:.2f}" if v is not None else "-"
all_labels = set()
for _, pt in per_types:
if pt:
all_labels |= set(pt)
print(f"\n### {title}, per label\n")
header = " | ".join(f"{label + ' F':>7}" for label in labels)
delta_header = " | ".join(f"{'d(' + label + ')':>9}" for label in labels[1:])
print(f"| {'label':<6} | {header} | {delta_header} |")
sep = " | ".join("-" * 7 for _ in labels)
delta_sep = " | ".join("-" * 9 for _ in labels[1:])
print(f"| {'-' * 6} | {sep} | {delta_sep} |")
def sort_key(entity_label):
last_pt = per_types[-1][1] or {}
return -last_pt.get(entity_label, {}).get("f", 0)
for entity_label in sorted(all_labels, key=sort_key):
fs = [(pt or {}).get(entity_label, {}).get("f") for _, pt in per_types]
cells = [pct(f) for f in fs]
deltas = []
for f in fs[1:]:
a = fs[0]
deltas.append(f"{(f - a) * 100:+.2f}" if a is not None and f is not None else "-")
row = " | ".join(f"{c:>7}" for c in cells)
drow = " | ".join(f"{d:>9}" for d in deltas)
print(f"| {entity_label:<6} | {row} | {drow} |")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--metrics-dir", type=Path, default=Path("metrics"))
args = ap.parse_args()
print("# sm vs md vs lg (fa_floret static vectors)")
print("\nSame corpus, same seed, same architecture per group. Only difference:")
print("`include_static_vectors = false -> true`, and which floret table (md: 50k rows,")
print("400k documents; lg: 200k rows, full Persian Wikipedia, 5 epochs).")
for title, reports in GROUPS:
tiers = []
missing = []
for label, fname in reports.items():
data = load(args.metrics_dir / fname)
if data is None:
missing.append(fname)
tiers.append((label, data))
if tiers[0][1] is None:
print(f"\n## {title}\n\n (skipped, missing baseline {reports[list(reports)[0]]})")
continue
table(title, tiers, SCALARS)
per_type(title, tiers)
if missing:
print(f"\n (missing: {', '.join(missing)})")
if __name__ == "__main__":
main()

86
scripts/finalize_pipeline.py
View File

@ -67,17 +67,21 @@ FLORET = {
"license": "CC BY-SA 4.0", "license": "CC BY-SA 4.0",
} }
FLORET_LG = { FLORET_LG = {
"name": "fa_floret static vectors (lg tier: larger floret table trained on fa Wikipedia + " "name": "fa_floret static vectors (lg tier: 200k rows x 300d floret table trained on "
"OSCAR via spacy-vectors-builder)", "the full Persian Wikipedia dump, 5 epochs, via spacy-vectors-builder)",
"url": PROJECT_URL, "url": "https://huggingface.co/Phazel/fa-floret-wiki-vectors",
"author": "Kiyarash Fazeli", "author": "Kiyarash Fazeli",
"license": "CC BY-SA 4.0", "license": "CC BY-SA 4.0",
} }
TRANSFORMER = { # Whatever encoder the config actually names wins; hardcoding one would silently mislabel a
"name": "HooshvareLab/roberta-fa-zwnj-base", # wheel the moment configs/fa_core_news_trf.cfg's `name` changes. Licences are recorded per
"url": "https://huggingface.co/HooshvareLab/roberta-fa-zwnj-base", # encoder because they differ sharply, and two of the Persian ones have none at all.
"author": "Hooshvare Team", ENCODER_LICENSES = {
"license": "Apache-2.0", "HooshvareLab/roberta-fa-zwnj-base": ("Hooshvare Team", "Apache-2.0"),
"HooshvareLab/bert-fa-zwnj-base": ("Hooshvare Team", "Apache-2.0"),
"m3hrdadfi/albert-fa-base-v2": ("Mehrdad Farahani", "Apache-2.0"),
"HooshvareLab/bert-base-parsbert-uncased": ("Hooshvare Team", "no licence stated on the model card"),
"sbunlp/fabert": ("SBU NLP Lab", "no licence stated on the model card"),
} }
NER_NOTE = ( NER_NOTE = (
@ -119,18 +123,55 @@ def vectors_note_lg(nlp):
return ( return (
f"This is the `lg` tier: identical architecture to `sm`/`md` but a larger static " f"This is the `lg` tier: identical architecture to `sm`/`md` but a larger static "
f"floret vector table ({rows:,} rows x {dim} dimensions, minn=maxn=5, hash_count=2) " f"floret vector table ({rows:,} rows x {dim} dimensions, minn=maxn=5, hash_count=2) "
f"trained on Persian Wikipedia + OSCAR via spacy-vectors-builder. Same zero-OOV " f"trained on the full Persian Wikipedia dump for 5 epochs via spacy-vectors-builder. "
f"rationale as `md` (see docs/MODELS.md): floret hashes subwords into a fixed table, " f"Same zero-OOV rationale as `md` (see docs/MODELS.md): floret hashes subwords into "
f"so `token.has_vector` is always True despite Persian's ZWNJ (U+200C) inconsistency." f"a fixed table, so `token.has_vector` is always True despite Persian's ZWNJ "
f"(U+200C) inconsistency."
) )
TRANSFORMER_NOTE = ( def encoder_name(nlp):
"This is the `trf` tier: no static vectors; contextual embeddings instead come from a " """Read the encoder out of the trained pipeline's own config."""
"fine-tuned HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) transformer via " try:
"spacy-transformers. Not ParsBERT: its model card carries no licence. GPU is recommended " return nlp.config["components"]["transformer"]["model"]["name"]
"for both training and inference." except KeyError:
) raise SystemExit(
"--size trf expects a pipeline with a `transformer` component whose model names "
f"an encoder; got pipeline {list(nlp.pipe_names)}"
)
def transformer_source(nlp):
name = encoder_name(nlp)
author, license_ = ENCODER_LICENSES.get(name, ("unknown", "unknown, check the model card"))
return {
"name": name,
"url": f"https://huggingface.co/{name}",
"author": author,
"license": license_,
}
def transformer_note(nlp):
name = encoder_name(nlp)
_, license_ = ENCODER_LICENSES.get(name, ("unknown", "unknown, check the model card"))
note = (
f"This is the `trf` tier: no static vectors. Contextual embeddings come from a "
f"fine-tuned {name} ({license_}) via spacy-transformers, shared by every component "
f"through a TransformerListener, so one encoder forward pass serves the tagger, "
f"morphologizer, lemmatizer, parser and ner. Unlike the sm/md/lg tiers the ner is "
f"trained jointly rather than sourced, because a shared encoder cannot be fine-tuned "
f"twice and then merged. GPU is strongly recommended for both training and inference."
)
if "no licence" in license_ or license_.startswith("unknown"):
note += (
f" REDISTRIBUTION WARNING: {name} states no licence, so this wheel embeds weights "
f"whose terms are unknown and must not be republished. Retrain against "
f"HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) for a publishable artifact."
)
return note
# CC BY-SA 4.0 on the treebank propagates to anything derived from it. # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
PERDT_LICENSE = "CC BY-SA 4.0" PERDT_LICENSE = "CC BY-SA 4.0"
ATTRIBUTION = ( ATTRIBUTION = (
@ -221,8 +262,15 @@ def main():
sources.append(FLORET_LG) sources.append(FLORET_LG)
notes = " ".join([notes, vectors_note_lg(nlp)]) notes = " ".join([notes, vectors_note_lg(nlp)])
elif args.size == "trf": elif args.size == "trf":
sources.append(TRANSFORMER) sources.append(transformer_source(nlp))
notes = " ".join([notes, TRANSFORMER_NOTE]) notes = " ".join([notes, transformer_note(nlp)])
# The stock description advertises a CPU tok2vec pipeline, which is wrong here.
description = (
"Persian pipeline built on a fine-tuned "
f"{encoder_name(nlp)} transformer. Components: transformer, tagger, "
"morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, "
"DAT, MON, TIM, PCT. GPU recommended."
)
if args.add_ner: if args.add_ner:
ner_nlp = spacy.load(args.add_ner) ner_nlp = spacy.load(args.add_ner)
if ner_nlp.pipe_names != ["ner"]: if ner_nlp.pipe_names != ["ner"]:

136
scripts/make_model_card.py Normal file
View File

@ -0,0 +1,136 @@
"""Build the Hugging Face model card for a packaged pipeline.
`spacy package` already writes a README into the wheel, and `spacy huggingface-hub push`
uploads it as the card. That card is a metadata dump: no install line, no usage, no
throughput, and no YAML frontmatter, so the Hub cannot index the model by language or task.
This composes a card from the same sources of truth (`meta.json` and the JSON written by
scripts/benchmark_throughput.py) rather than from hand-copied numbers, so the card cannot
drift from the artifact it describes.
"""
import argparse
import json
from pathlib import Path
# meta.json key -> (row label, reference note). Only keys the pipeline actually evidences
# are emitted; a missing key means the corpus could not score it.
METRICS = [
("token_acc", "Tokenization accuracy", ""),
("tag_acc", "XPOS tag accuracy", ""),
("pos_acc", "UPOS tag accuracy", ""),
("morph_acc", "Morphological features", ""),
("lemma_acc", "Lemma accuracy", ""),
("dep_uas", "Unlabelled attachment (UAS)", ""),
("dep_las", "Labelled attachment (LAS)", ""),
("sents_f", "Sentence segmentation F", ""),
("ents_p", "NER precision", ""),
("ents_r", "NER recall", ""),
("ents_f", "NER F-score", ""),
]
def load(path):
return json.loads(Path(path).read_text())
def throughput_rows(paths):
rows = []
for p in paths:
if not Path(p).exists():
continue
d = load(p)
rows.append((d["device"], d["batch_size"], d["wps_median"]))
return rows
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--meta", required=True, help="meta.json of the finalized pipeline")
ap.add_argument("--throughput", nargs="*", default=[], help="benchmark_throughput JSONs")
ap.add_argument("--repo-id", required=True, help="e.g. Phazel/fa_core_news_trf")
ap.add_argument("--wheel-name", required=True)
ap.add_argument("--out", required=True)
args = ap.parse_args()
meta = load(args.meta)
name = f"{meta['lang']}_{meta['name']}"
perf = meta.get("performance", {})
lines = []
# Frontmatter: without this the Hub cannot filter the model by language or library.
lines += [
"---",
"language:",
"- fa",
f"license: {meta.get('license', 'cc-by-sa-4.0').lower().replace(' ', '-')}",
"library_name: spacy",
"pipeline_tag: token-classification",
"tags:",
"- spacy",
"- token-classification",
"- persian",
"- farsi",
"---",
"",
f"# {name}",
"",
meta.get("description", "").strip(),
"",
]
lines += [
"## Install",
"",
"```bash",
f"pip install https://huggingface.co/{args.repo_id}/resolve/main/{args.wheel_name}",
"```",
"",
"```python",
"import spacy",
f'nlp = spacy.load("{name}")',
'doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می\u200cدهد.")',
"print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])",
"print([(e.text, e.label_) for e in doc.ents])",
"```",
"",
]
lines += ["## Accuracy", "",
"Scored with `spacy benchmark accuracy` on the held-out PerDT test split.",
"", "| Metric | Score |", "| --- | ---: |"]
for key, label, _ in METRICS:
v = perf.get(key)
if isinstance(v, (int, float)):
lines.append(f"| {label} | {v * 100:.2f} |")
lines.append("")
rows = throughput_rows(args.throughput)
if rows:
lines += ["## Throughput", "",
"Median of repeated `nlp.pipe` passes over the 146-document PerDT test",
"split (23,825 tokens), timing the pipe only. Warmup pass discarded.",
"", "| Device | Batch | Words/s |", "| --- | ---: | ---: |"]
for device, batch, wps in rows:
lines.append(f"| {device} | {batch} | {wps:,.0f} |")
lines.append("")
lines += ["## Sources", "", "| Source | Author | Licence |", "| --- | --- | --- |"]
for s in meta.get("sources", []):
url, nm = s.get("url"), s.get("name", "")
label = f"[{nm}]({url})" if url else nm
lines.append(f"| {label} | {s.get('author', '')} | {s.get('license', '')} |")
lines.append("")
notes = (meta.get("notes") or "").strip()
if notes:
lines += ["## Notes", "", notes, ""]
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(lines), encoding="utf-8")
print(f"wrote {out} ({out.stat().st_size} bytes)")
if __name__ == "__main__":
main()

68
scripts/merge_joint_corpus.py Normal file
View File

@ -0,0 +1,68 @@
"""Fuse the UD annotation layer and the transferred NER layer into one DocBin.
The sm/md/lg tiers train `ner` as a separate pipeline with its own embedded tok2vec, then
source it into the dep model (project.yml `assemble-core`). That works because a hash-embed
tok2vec is cheap enough to train twice.
A transformer is not. Fine-tuning ParsBERT once per component would double GPU cost and
produce a package carrying two independent 162M-parameter encoders, and sourcing the second
one would collide on the `transformer` component name. So the trf tier trains every component
against a single shared transformer via TransformerListener, which requires a single corpus
carrying both annotation layers on the same Doc.
That fusion is exact, not approximate: `corpus/perdt-ner/` was produced by
scripts/transfer_perdt_ner.py from the same `--merge-subtokens` CoNLL-U as `corpus/merged/`,
then converted with the same `--n-sents`, so the two DocBins are token-for-token identical
(verified below and asserted at runtime). Only `doc.ents` is copied across; every other
annotation stays on the UD doc.
"""
import argparse
from pathlib import Path
import spacy
from spacy.tokens import DocBin, Span
SPLITS = (("train", "fa_perdt-ud-train"), ("dev", "fa_perdt-ud-dev"), ("test", "fa_perdt-ud-test"))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--ud-dir", default="corpus/merged")
ap.add_argument("--ner-dir", default="corpus/perdt-ner")
ap.add_argument("--out", default="corpus/joint")
ap.add_argument("--lang", default="fa")
args = ap.parse_args()
nlp = spacy.blank(args.lang)
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
for split, ud_stem in SPLITS:
ud_docs = list(DocBin().from_disk(Path(args.ud_dir) / f"{ud_stem}.spacy").get_docs(nlp.vocab))
ner_docs = list(DocBin().from_disk(Path(args.ner_dir) / f"{split}.spacy").get_docs(nlp.vocab))
if len(ud_docs) != len(ner_docs):
raise SystemExit(
f"{split}: {len(ud_docs)} UD docs vs {len(ner_docs)} NER docs; the two corpora "
"were not converted from the same source with the same --n-sents"
)
db = DocBin(store_user_data=True)
n_ents = 0
for i, (ud, ner) in enumerate(zip(ud_docs, ner_docs)):
if [t.text for t in ud] != [t.text for t in ner]:
raise SystemExit(f"{split} doc {i}: tokenization differs between UD and NER layers")
# Tokens are index-aligned, so rebuild by token index. Char offsets are NOT
# safe here: the two converters can differ in trailing whitespace, which shifts
# `char_span` off the token grid and silently yields None.
ud.ents = [Span(ud, e.start, e.end, label=e.label_) for e in ner.ents]
n_ents += len(ud.ents)
db.add(ud)
dest = out / f"{split}.spacy"
db.to_disk(dest)
print(f"{dest}: {len(ud_docs)} docs, {n_ents} entities")
if __name__ == "__main__":
main()