fazel
/
spacy-fa-pipeline
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: fazel:f45d0db643ce58abbcf79a8cf1b7c97149602b1d
Branches Tags
fazel:main
fazel:colab-lg-trf-training
...
compare: fazel:37c85d018647f4ec1afb27ac28fc4a4fa51ddd92
Branches Tags
fazel:main
fazel:colab-lg-trf-training

9 Commits
f45d0db643 ... 37c85d0186

Author SHA1 Message Date
Mohamad Fazeli 37c85d0186
fix layout of README.md 2026-08-14 11:57:35 +03:30
Mohamad Fazeli e37d052435
Correct the 940MX trf figure to a 9-run median 
1,106 words/s, not the 1,158 from the first 3-run sample. Two independent 3-run
samples disagreed by 3.4%, so the larger sample settles it.
 2026-08-13 17:52:15 +03:30
Mohamad Fazeli c80bcb27f3
List the three newly published packages in the Persian README 2026-08-13 17:35:14 +03:30
Mohamad Fazeli b26b194d9f
Publish the remaining three packages and link them 
fa_ent_news_sm, fa_dep_news_md and fa_core_news_md are now on the Hub, each with
a generated card carrying accuracy and measured throughput.

The md wheels in packages/ were stale 0.1.0 artifacts from an earlier manual run,
even though project.yml passes --version 3.8.0. Rebuilt at 3.8.0 so they match
their published siblings before upload.
 2026-08-13 17:34:41 +03:30
Mohamad Fazeli f1a95d92c3
Link each package to its Hugging Face page 
Add the lg packages to the table, which had never been listed, with measured
wheel sizes rather than the rounded ones. fa_ent_news_sm, fa_dep_news_md and
fa_core_news_md are built by project.yml but not on the Hub, so they stay
unlinked and are called out as unpublished. All nine links verified to return
200.

Move the cu126 torch requirement out of both READMEs into MODELS.md 9.
 2026-08-13 17:17:57 +03:30
Mohamad Fazeli e7871a29a1
Cut the throughput notes to measurements 
State the numbers and the cu126 requirement; drop the narration around them.
 2026-08-13 17:12:50 +03:30
Mohamad Fazeli 4135142e72
Fill in trf throughput on the 940MX 
The 940MX does run trf, at 1,158 words/s with batch 32 inside 2 GB, 6.2x its host
CPU. The earlier claim that current PyTorch wheels cannot target sm_50 was only
half right: Maxwell kernels were dropped from the cu128 and cu129 builds starting
torch 2.8, which is what `pip install torch` now resolves to, but the cu126 build
of 2.7.1 still ships sm_50 and works.

.venv-trf-gpu pins torch==2.7.1+cu126 for this and stays separate from .venv,
whose cupy runs on nvidia-* 12.9 wheels that torch would downgrade to 12.6.

Drop the editorial sentence from the generated model card; the table states it.
 2026-08-13 17:11:25 +03:30
Mohamad Fazeli 6fa8aa70b8
Lead the README tables with trf instead of sm 
The Hazm and en_core_web_sm comparison now cites fa_core_news_trf, which is the
strongest tier and the only one to pass the hazm+ParsBERT DEP_LAS reference.
Drop the package-size row from that table: it compared a 608 MB transformer
against a 7 MB rule-based toolkit, which says nothing about accuracy.

Add trf to the package list and to the per-label NER table, alongside lg, which
had never been added. Its largest per-label gains are PER +9.25 and DAT +11.69
over lg. Drop the gold-count column there and trim the surrounding notes.
 2026-08-13 16:52:37 +03:30
Mohamad Fazeli 5da9dd1524
Add fa_core_news_trf tier on a shared fine-tuned ParsBERT 
Trains tagger, morphologizer, trainable_lemmatizer, parser and ner against one
fine-tuned HooshvareLab/bert-base-parsbert-uncased through TransformerListener,
rather than the sm/md/lg split where ner is trained separately and sourced in.
Fine-tuning a 162M-parameter encoder twice would double GPU cost, ship two
encoders in one wheel, and collide on the `transformer` component name.

A shared encoder needs one corpus carrying both annotation layers, so
merge_joint_corpus.py fuses them. corpus/perdt-ner/ came from the same
--merge-subtokens CoNLL-U as corpus/merged/ with the same --n-sents, so the
DocBins are token-for-token identical; the script asserts that per document and
copies doc.ents by token index. Char offsets do not work here because the two
converters differ in trailing whitespace, which pushes char_span off the token
grid and returns None.

Trained on a Colab T4 in 1h58m, 3000 steps, no early stop. Test scores against
lg: DEP_LAS 90.79 (+4.19), ENTS_F 82.89 (+6.95, almost all recall), TAG_ACC
97.62 (+1.07). DEP_LAS passes the hazm+ParsBERT reference of 89.34, which no CPU
tier reached. LEMMA_ACC 97.31 and SENTS_F 97.35 regress against lg; the likely
cause of the latter is strided_spans leaving only 32 tokens of overlap.

max_steps and learn_rate.total_steps are held equal on purpose. They are
independent knobs, and a patience stop under a longer total_steps ends training
at a high learning rate, discarding the annealing tail.

finalize_pipeline.py now reads the encoder name out of the trained config
instead of hardcoding it, tracks per-encoder licences, and writes a
redistribution warning into meta.json when the encoder states none. ParsBERT
states none, so that wheel is not redistributable; roberta-fa-zwnj-base
(Apache-2.0) is a one-line change to `name`.

Add benchmark_throughput.py, which times nlp.pipe alone. The words/s from
`spacy benchmark accuracy` includes the Scorer's per-token alignment, which is
why MODELS.md 7 reported ent_lg as faster than sm despite an identical ner
architecture. Remeasured every tier on CPU and the 940MX; a background rsync
halved every figure, so the final numbers are 9-run medians on an idle machine.

Add make_model_card.py, which composes the Hub card from meta.json and the
throughput records. The card spacy package writes has no frontmatter, so the Hub
cannot index the model by language, and no install line or usage.
 2026-08-13 16:34:55 +03:30
10 changed files with 1013 additions and 69 deletions
Show Stats Expand all files Collapse all files

4
.gitignore vendored
View File

@ -9,6 +9,10 @@ packages/
# Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the # Separate env for `spacy huggingface-hub push`: it caps typer<0.8, which breaks the
# spaCy CLI in the training venv. See .omp/AGENTS.md. # spaCy CLI in the training venv. See .omp/AGENTS.md.
.venv-publish/ .venv-publish/
# Local envs for verifying and benchmarking the trf wheel: CPU-only torch, and a cu126 build
# for the 940MX. Kept out of .venv so a CUDA-lib downgrade cannot reach the training env.
.venv-trf/
.venv-trf-gpu/
__pycache__/ __pycache__/
*.pyc *.pyc
# Personal scratch list, not part of the project # Personal scratch list, not part of the project

72
README.fa.md
View File

@ -22,22 +22,51 @@ doc = nlp("محمدرضا شجریان در مشهد به دنیا آمد.")
print(doc.ents) # (محمدرضا شجریان, مشهد) print(doc.ents) # (محمدرضا شجریان, مشهد)
``` ```
بسته‌های منتشرشده روی Hugging Face:
[`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) ·
[`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) ·
[`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) ·
[`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) ·
[`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) ·
[`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) ·
[`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) ·
[`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) ·
[`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) ·
[`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf).
جدول‌های بردار floret جداگانه (فقط بردار، بدون هیچ مؤلفه‌ای):
```bash
# ۵۰ هزار سطر × ۳۰۰ بعد، ۴۰۰ هزار سند فارسی (جدول ردهٔ md)
pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
# ۵۰ هزار سطر × ۳۰۰ بعد، کل دامپ ویکی‌پدیای فارسی
pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
# ۲۰۰ هزار سطر × ۳۰۰ بعد، کل دامپ ویکی‌پدیای فارسی، ۵ دوره (جدول ردهٔ lg)
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
```
## کارایی ## کارایی
ارزیابی با `spacy benchmark accuracy` روی بخش آزمون همان پیکره انجام شده است: ارزیابی با `spacy benchmark accuracy` روی بخش آزمون همان پیکره انجام شده است:
| سنجه | امتیاز | مرجع | | سنجه | `sm` | `md` | `lg` | `trf` | مرجع |
| --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `TOKEN_ACC` / `TOKEN_F` | ۹۹٫۹۶ / ۹۹٫۱۱ | | | `TOKEN_ACC` / `TOKEN_F` | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | ۹۹٫۹۶ / ۹۹٫۱۱ | |
| `TAG_ACC` (XPOS) | ۹۵٫۹۶ | | | `TAG_ACC` (XPOS) | ۹۵٫۹۶ | ۹۶٫۲۵ | ۹۶٫۵۵ | **۹۷٫۶۲** | |
| `POS_ACC` (UPOS) | ۹۶٫۲۴ | | | `POS_ACC` (UPOS) | ۹۶٫۲۴ | ۹۶٫۶۴ | ۹۶٫۶۸ | **۹۷٫۶۳** | |
| `MORPH_ACC` | ۹۶٫۲۹ | | | `MORPH_ACC` | ۹۶٫۲۹ | ۹۶٫۶۴ | ۹۶٫۷۰ | **۹۷٫۸۲** | |
| `LEMMA_ACC` | ۹۷٫۹۱ | | | `LEMMA_ACC` | ۹۷٫۹۱ | ۹۷٫۹۶ | **۹۸٫۰۸** | ۹۷٫۳۱ | |
| `SENTS_F` | ۹۹٫۲۵ | | | `SENTS_F` | ۹۹٫۲۵ | **۹۹٫۲۸** | ۹۹٫۱۸ | ۹۷٫۳۵ | |
| `DEP_UAS` | ۸۹٫۶۹ | hazm+ParsBERT: ۹۲٫۴۶ | | `DEP_UAS` | ۸۹٫۶۹ | ۹۰٫۵۲ | ۹۰٫۹۶ | **۹۳٫۸۷** | hazm+ParsBERT: ۹۲٫۴۶ |
| `DEP_LAS` | ۸۵٫۱۵ | hazm+ParsBERT: ۸۹٫۳۴ | | `DEP_LAS` | ۸۵٫۱۵ | ۸۶٫۳۴ | ۸۶٫۶۰ | **۹۰٫۷۹** | hazm+ParsBERT: ۸۹٫۳۴ |
| `ENTS_F` | ۷۱٫۸۷ | تنها در `fa_core_news_sm` | | `ENTS_P` | ۷۷٫۶۷ | ۷۶٫۵۶ | ۸۱٫۵۱ | **۸۴٫۰۶** | |
| سرعت | حدود ۹٬۲۵۰ واژه بر ثانیه | | | `ENTS_R` | ۶۶٫۸۷ | ۷۲٫۹۵ | ۷۱٫۰۹ | **۸۱٫۷۶** | |
| `ENTS_F` | ۷۱٫۸۷ | ۷۴٫۷۱ | ۷۵٫۹۴ | **۸۲٫۸۹** | |
| سرعت (940MX، دستهٔ ۳۲) | ۱۰٬۲۳۵ | ۹٬۰۵۸ | ۹٬۲۱۵ | بخش توان عملیاتی | |
| حجم بستهٔ نصب | ۱۳٫۵ مگابایت | ۶۸٫۵ مگابایت | ۲۳۵ مگابایت | ۶۰۸ مگابایت | |
ردهٔ `trf` در همه‌جا جلو است مگر در واژه‌یابی و مرزبندی جمله، و تنها ردهٔ‌ای است که از مرجع
`DEP_LAS` برابر ۸۹٫۳۴ عبور می‌کند. به کارت گرافیک نیاز دارد و مدل پایهٔ آن پروانهٔ مشخصی ندارد،
پس قابل بازانتشار نیست (`docs/MODELS.md` بخش ۸).
برچسب‌های موجودیت «نقره‌ای» هستند: از لایه‌ای در خود پیکره می‌آیند که با برچسب‌زن Beheshti-NER برچسب‌های موجودیت «نقره‌ای» هستند: از لایه‌ای در خود پیکره می‌آیند که با برچسب‌زن Beheshti-NER
تولید و سپس دستی اصلاح شده است. بنابراین `ENTS_F` تا اندازه‌ای هم‌خوانی با آن برچسب‌زن را تولید و سپس دستی اصلاح شده است. بنابراین `ENTS_F` تا اندازه‌ای هم‌خوانی با آن برچسب‌زن را
@ -46,6 +75,25 @@ print(doc.ents) # (محمدرضا شجریان, مشهد)
آموزش روی یک پردازندهٔ چهارهسته‌ای i5-7200U و بدون کارت گرافیک انجام شده است: ۱ ساعت و ۲۷ دقیقه آموزش روی یک پردازندهٔ چهارهسته‌ای i5-7200U و بدون کارت گرافیک انجام شده است: ۱ ساعت و ۲۷ دقیقه
برای اجزای نحوی و ۱۷ دقیقه برای NER. این دو اجرا مستقل‌اند و می‌توانند هم‌زمان انجام شوند. برای اجزای نحوی و ۱۷ دقیقه برای NER. این دو اجرا مستقل‌اند و می‌توانند هم‌زمان انجام شوند.
## توان عملیاتی
میانهٔ چند اجرای پیاپی `nlp.pipe` روی ۱۴۶ سند بخش آزمون PerDT (۲۳٬۸۲۵ توکن). تنها زمان خودِ
`pipe` اندازه‌گیری شده و اجرای گرم‌کردن کنار گذاشته می‌شود. برای بازتولید:
`python scripts/benchmark_throughput.py <model> --gpu-id <n>`؛ دادهٔ خام در
`metrics/throughput-*.json` است.
| رده | پردازنده i5-7200U | کارت 940MX | کارت Tesla T4 |
| --- | ---: | ---: | ---: |
| `sm` | ۵٬۴۸۴ | ۱۰٬۲۳۵ | |
| `md` | ۵٬۴۰۸ | ۹٬۰۵۸ | |
| `lg` | ۴٬۷۱۵ | ۹٬۲۱۵ | |
| `trf` | ۱۸۷ | ۱٬۱۰۶ | ۸٬۳۲۰ |
ردهٔ `trf` روی یک پردازنده ۲۹ برابر کندتر از `sm` است. عددهای T4 و Xeon از یک ماشین Colab
می‌آیند، یعنی شتاب ۲۵ برابری. فاصلهٔ رده‌های پردازنده‌ای کمتر از ۱۵ درصد است، پس گلوگاه
تجزیه‌گر و واژه‌یاب است نه جست‌وجوی tok2vec. پراکندگی اجراها روی لپ‌تاپ حدود ۱۰± درصد است.
اجرای `trf` روی 940MX به نسخهٔ مشخصی از torch نیاز دارد؛ بخش ۹ از `docs/MODELS.md` را ببینید.
گام‌های تبدیل پیکره، آموزش، ارزیابی و بسته‌بندی در [`project.yml`](project.yml) تعریف شده‌اند. گام‌های تبدیل پیکره، آموزش، ارزیابی و بسته‌بندی در [`project.yml`](project.yml) تعریف شده‌اند.
توضیح بیشتر دربارهٔ گزینش پیکره و پروانه‌ها در [`docs/MODELS.md`](docs/MODELS.md) و شرح انگلیسی توضیح بیشتر دربارهٔ گزینش پیکره و پروانه‌ها در [`docs/MODELS.md`](docs/MODELS.md) و شرح انگلیسی
پروژه در [`README.md`](README.md) آمده است. پروژه در [`README.md`](README.md) آمده است.

128
README.md
View File

@ -25,13 +25,12 @@ pip install https://huggingface.co/Phazel/fa_core_news_sm/resolve/main/fa_core_n
Compared against Hazm (the most-used Persian toolkit) and `en_core_web_sm` (English reference). Compared against Hazm (the most-used Persian toolkit) and `en_core_web_sm` (English reference).
| Metric | **`spacy-persian`**<br>`fa_core_news_sm` | **Hazm**<br>(Persian toolkit) | `en_core_web_sm`<br>(English reference) | | Metric | **`spacy-persian`**<br>`fa_core_news_trf` | **Hazm**<br>(Persian toolkit) | `en_core_web_sm`<br>(English reference) |
|--------|:---:|:---:|:---:| |--------|:---:|:---:|:---:|
| **POS Accuracy (UPOS)** | **96.24%** | ~95.69%¹ | 97.21%² | | **POS Accuracy (UPOS)** | **97.63%** | ~95.69%¹ | 97.21%² |
| **Lemma Accuracy** | **97.91%** | 89.9%¹ | — | | **Lemma Accuracy** | **97.31%** | 89.9%¹ | — |
| **Dependency LAS** | 85.15% | 85.6%¹ | 91.85%² | | **Dependency LAS** | **90.79%** | 85.6%¹ | 91.85%² |
| **NER F-score** | 71.87% | — | 83.80%² | | **NER F-score** | **82.89%** | — | 83.80%² |
| **Package Size** | **13 MB** (syntax+NER)<br>**7.5 MB** (syntax-only) | ~7 MB | 12 MB |
> **¹** Hazm scores from its official README > **¹** Hazm scores from its official README
> **²** `en_core_web_sm` scores from spaCy's official model card > **²** `en_core_web_sm` scores from spaCy's official model card
@ -42,31 +41,43 @@ Compared against Hazm (the most-used Persian toolkit) and `en_core_web_sm` (Engl
From `spacy benchmark accuracy`, stored in `metrics/`. From `spacy benchmark accuracy`, stored in `metrics/`.
| Package | Components | Licence | Score | Wheel | | Package | Components | Licence | Score | Wheel |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- |
| `fa_dep_news_sm` | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | CC BY-SA 4.0 | LEMMA 97.91 | 7.5 MB | | [`fa_dep_news_sm`](https://huggingface.co/Phazel/fa_dep_news_sm) | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser | CC BY-SA 4.0 | LEMMA 97.91 | 7.9 MB |
| `fa_core_news_sm` | the above plus ner | CC BY-SA 4.0 | ENTS_F 71.87 | 13 MB | | [`fa_core_news_sm`](https://huggingface.co/Phazel/fa_core_news_sm) | the above plus ner | CC BY-SA 4.0 | ENTS_F 71.87 | 13.5 MB |
| `fa_ent_news_sm` | `ner` alone (own embedded tok2vec) | CC BY-SA 4.0 | ENTS_F 71.87 | 5.6 MB | | [`fa_ent_news_sm`](https://huggingface.co/Phazel/fa_ent_news_sm) | `ner` alone (own embedded tok2vec) | CC BY-SA 4.0 | ENTS_F 71.87 | 5.9 MB |
| `fa_dep_news_md` | same as `fa_dep_news_sm`, plus floret vectors | CC BY-SA 4.0 | LEMMA 97.96 | 62 MB | | [`fa_dep_news_md`](https://huggingface.co/Phazel/fa_dep_news_md) | same as `fa_dep_news_sm`, plus floret vectors | CC BY-SA 4.0 | LEMMA 97.96 | 62.6 MB |
| `fa_core_news_md` | same as `fa_core_news_sm`, plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 68 MB | | [`fa_core_news_md`](https://huggingface.co/Phazel/fa_core_news_md) | same as `fa_core_news_sm`, plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 68.5 MB |
| `fa_ent_news_md` | `ner` alone (own embedded tok2vec), plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 58 MB | | [`fa_ent_news_md`](https://huggingface.co/Phazel/fa_ent_news_md) | `ner` alone (own embedded tok2vec), plus floret vectors | CC BY-SA 4.0 | ENTS_F 74.71 | 60.6 MB |
| [`fa_dep_news_lg`](https://huggingface.co/Phazel/fa_dep_news_lg) | same as `fa_dep_news_sm`, plus full-wiki floret vectors | CC BY-SA 4.0 | LEMMA 98.08 | 229.3 MB |
| [`fa_core_news_lg`](https://huggingface.co/Phazel/fa_core_news_lg) | same as `fa_core_news_sm`, plus full-wiki floret vectors | CC BY-SA 4.0 | ENTS_F 75.94 | 235.2 MB |
| [`fa_ent_news_lg`](https://huggingface.co/Phazel/fa_ent_news_lg) | `ner` alone (own embedded tok2vec), plus full-wiki floret vectors | CC BY-SA 4.0 | ENTS_F 75.94 | 227.3 MB |
| [`fa_core_news_trf`](https://huggingface.co/Phazel/fa_core_news_trf) | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | see §8, encoder unlicensed | ENTS_F 82.89, LAS 90.79 | 608.2 MB |
Raw `fa.floret` and `fa.vec` exports of the 200k table are in
[`fa-floret-wiki-vectors`](https://huggingface.co/Phazel/fa-floret-wiki-vectors).
The `md` tier adds a 50k x 300d floret vector table trained on 400k Persian documents. Its The `md` tier adds a 50k x 300d floret vector table trained on 400k Persian documents. Its
config differs from `sm` by exactly one line (`include_static_vectors`), so the columns below config differs from `sm` by exactly one line (`include_static_vectors`), so the columns below
isolate what the vectors buy. Full breakdown in `docs/MODELS.md` §6. isolate what the vectors buy. Full breakdown in `docs/MODELS.md` §6.
| Metric | `sm` | `md` | Reference | | Metric | `sm` | `md` | `lg` | `trf` | Reference |
| --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | 99.96 / 99.11 | | | `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | 99.96 / 99.11 | 99.96 / 99.11 | 99.96 / 99.11 | |
| `TAG_ACC` (XPOS) | 95.96 | 96.25 | | | `TAG_ACC` (XPOS) | 95.96 | 96.25 | 96.55 | **97.62** | |
| `POS_ACC` (UPOS) | 96.24 | 96.64 | | | `POS_ACC` (UPOS) | 96.24 | 96.64 | 96.68 | **97.63** | |
| `MORPH_ACC` | 96.29 | 96.64 | | | `MORPH_ACC` | 96.29 | 96.64 | 96.70 | **97.82** | |
| `LEMMA_ACC` | 97.91 | 97.96 | | | `LEMMA_ACC` | 97.91 | 97.96 | **98.08** | 97.31 | |
| `SENTS_F` | 99.25 | 99.28 | | | `SENTS_F` | 99.25 | **99.28** | 99.18 | 97.35 | |
| `DEP_UAS` | 89.69 | 90.52 | hazm+ParsBERT: 92.46 | | `DEP_UAS` | 89.69 | 90.52 | 90.96 | **93.87** | hazm+ParsBERT: 92.46 |
| `DEP_LAS` | 85.15 | 86.34 | hazm+ParsBERT: 89.34 | | `DEP_LAS` | 85.15 | 86.34 | 86.60 | **90.79** | hazm+ParsBERT: 89.34 |
| `ENTS_P` | 77.67 | 76.56 | | | `ENTS_P` | 77.67 | 76.56 | 81.51 | **84.06** | |
| `ENTS_R` | 66.87 | 72.95 | | | `ENTS_R` | 66.87 | 72.95 | 71.09 | **81.76** | |
| `ENTS_F` | 71.87 | 74.71 | | | `ENTS_F` | 71.87 | 74.71 | 75.94 | **82.89** | |
| Speed | ~9,250 words/s | ~7,700 words/s | | | Speed (940MX, batch 32) | 10,235 words/s | 9,058 words/s | 9,215 words/s | 1,106 words/s | |
| Wheel size | 13.5 MB | 68.5 MB | 235 MB | 608 MB | |
`trf` leads everywhere except lemmatization and sentence segmentation, and is the only tier to
pass the hazm+ParsBERT `DEP_LAS` reference of 89.34. It needs a GPU, and its encoder states no
licence so it is not redistributable (`docs/MODELS.md` §8).
Entity scores are `fa_core_news_*` on the PerDT NER test split; per-label breakdown and Entity scores are `fa_core_news_*` on the PerDT NER test split; per-label breakdown and
caveats are in [Named entity recognition](#named-entity-recognition). caveats are in [Named entity recognition](#named-entity-recognition).
@ -76,6 +87,39 @@ For comparison, `en_core_web_sm` scores TAG 97, LAS 90, ENTS_F 84 on a larger, c
Trained on a 4-core i5-7200U with no GPU: `sm` 1h27m syntax + 17 min NER, `md` 1h54m syntax Trained on a 4-core i5-7200U with no GPU: `sm` 1h27m syntax + 17 min NER, `md` 1h54m syntax
+ 25 min NER (the two `md` runs overlapped, so wall clock overstates each). + 25 min NER (the two `md` runs overlapped, so wall clock overstates each).
Standalone floret vector packages (vectors only, `pipeline: []`), usable as
`--paths.vectors` for your own training or as a plain embedding table:
```bash
# 50k rows x 300d, 400k Persian documents (the md tier's table)
pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl
# 50k rows x 300d, full Persian Wikipedia dump
pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
# 200k rows x 300d, full Persian Wikipedia dump, 5 epochs (the lg tier's table)
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
```
## Throughput
Median of repeated `nlp.pipe` passes over the 146-document PerDT test split (23,825 tokens),
timing the pipe only, warmup discarded. Reproduce with
`python scripts/benchmark_throughput.py <model> --gpu-id <n>`; raw records are in
`metrics/throughput-*.json`.
| Tier | CPU, i5-7200U | GPU, GeForce 940MX | GPU, Tesla T4 |
| --- | ---: | ---: | ---: |
| `sm` | 5,484 | 10,235 | |
| `md` | 5,408 | 9,058 | |
| `lg` | 4,715 | 9,215 | |
| `trf` | 187 | 1,106 | 8,320 |
`trf` is 29x slower than `sm` on the same CPU. The T4 and Xeon figures come from one Colab VM,
a 25x GPU speedup. The CPU tiers sit within 15% of each other, so the bottleneck is the parser
and lemmatizer, not the tok2vec lookup. Laptop spread is about 10% with thermal state. Running
`trf` on the 940MX needs a specific torch build, see `docs/MODELS.md` §9.
## Named entity recognition ## Named entity recognition
Seven labels: `LOC`, `PER`, `ORG`, `DAT`, `MON`, `TIM`, `PCT`. They come from PerDT's own Seven labels: `LOC`, `PER`, `ORG`, `DAT`, `MON`, `TIM`, `PCT`. They come from PerDT's own
@ -87,23 +131,25 @@ recall, so the `ENTS_F` numbers below partly reflect agreement with that tagger,
human annotation. human annotation.
`ner` runs standalone with its own embedded tok2vec (`fa_ent_news_sm`, `fa_ent_news_md`), or `ner` runs standalone with its own embedded tok2vec (`fa_ent_news_sm`, `fa_ent_news_md`), or
bundled into `fa_core_news_sm`/`fa_core_news_md` alongside the syntax pipeline. bundled into `fa_core_news_sm`/`fa_core_news_md` alongside the syntax pipeline. In `trf` it is
trained jointly against the shared transformer instead, so there is no standalone trf variant.
| Label | Gold in test | `sm` F | `md` F | Train examples | | Label | `sm` F | `md` F | `lg` F | `trf` F | Train examples |
| --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
| `LOC` | 273 | 80.24 | 84.05 | 4,954 | | `LOC` | 80.24 | 84.05 | 83.66 | **87.78** | 4,954 |
| `PER` | 297 | 65.29 | 68.18 | 4,847 | | `PER` | 65.29 | 68.18 | 72.63 | **81.88** | 4,847 |
| `ORG` | 144 | 68.77 | 70.25 | 2,643 | | `ORG` | 68.77 | 70.25 | 71.01 | **78.50** | 2,643 |
| `DAT` | 69 | 74.45 | 76.19 | 1,323 | | `DAT` | 74.45 | 76.19 | 70.83 | **82.52** | 1,323 |
| `MON` | 10 | 73.68 | 84.21 | 205 | | `MON` | 73.68 | 84.21 | 88.89 | 88.89 | 205 |
| `TIM` | 9 | 66.67 | 66.67 | 135 | | `TIM` | 66.67 | 66.67 | 61.54 | 50.00 | 135 |
| `PCT` | 4 | 57.14 | 33.33 | 121 | | `PCT` | 57.14 | 33.33 | 57.14 | 33.33 | 121 |
`MON`, `TIM` and `PCT` have single-digit support in the test split, so their deltas are one `MON`, `TIM` and `PCT` have single-digit support in the test split, so their deltas are one or
or two entities changing hands, not signal. `PER`, `LOC` and `ORG` carry the split and all two entities changing hands, not signal. `PER`, `LOC` and `ORG` carry the split. The `md` gain
improve with floret vectors; the `md` gain over `sm` (`ENTS_F` 71.87 to 74.71) is almost over `sm` (`ENTS_F` 71.87 to 74.71) is almost entirely recall (+6.08), the lexical prior static
entirely recall (+6.08), the lexical prior static vectors give rare proper nouns that hash vectors give rare proper nouns that hash embeddings never had. `trf` adds another +6.95 F over
embeddings never had. `lg`, again mostly recall (71.09 to 81.76), and its largest per-label gains are `PER` (+9.25)
and `DAT` (+11.69).
## Install ## Install

291
configs/fa_core_news_trf.cfg Normal file
View File

@ -0,0 +1,291 @@
# fa_core_news_trf: the whole pipeline on one fine-tuned ParsBERT encoder.
#
# Differences from the sm/md/lg tiers, all forced by the transformer:
#
# * One corpus, not two. sm/md/lg train `ner` separately (own embedded tok2vec) and source
# it into the dep model. Fine-tuning a 162M-parameter encoder twice would double GPU cost
# and ship two encoders in one wheel, and the second would collide on the `transformer`
# component name. So every component listens to a single shared transformer and trains
# against corpus/joint/, built by scripts/merge_joint_corpus.py (UD layer + the
# difflib-transferred NER layer on identical tokenization).
# * `use_upper = false` on both transition-based parsers: with a transformer upstream the
# extra maxout layer is redundant, and this matches the upstream *_trf configs.
# * Adam + warmup_linear and accumulate_gradient=3, not the flat 0.001 the CPU tiers use.
# Fine-tuning a pretrained encoder at 1e-3 diverges.
# * gpu_allocator = "pytorch" so thinc and torch share one CUDA memory pool.
#
# Encoder: HooshvareLab/bert-base-parsbert-uncased. NOTE the licence caveat in
# docs/MODELS.md §3.4 - ParsBERT's model card carries no licence statement, so this wheel is
# NOT redistributable on those grounds; HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) is the
# publishable alternative and drops in by changing `name` below.
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null
[system]
gpu_allocator = "pytorch"
seed = 0
[nlp]
lang = "fa"
pipeline = ["transformer","tagger","morphologizer","trainable_lemmatizer","parser","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
[nlp.tokenizer]
@tokenizers = "spacy.Tokenizer.v1"
[nlp.vectors]
@vectors = "spacy.Vectors.v1"
[components]
[components.transformer]
factory = "transformer"
max_batch_items = 4096
[components.transformer.set_extra_annotations]
@annotation_setters = "spacy-transformers.null_annotation_setter.v1"
[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "HooshvareLab/bert-base-parsbert-uncased"
mixed_precision = false
[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96
[components.transformer.model.tokenizer_config]
use_fast = true
[components.transformer.model.transformer_config]
[components.transformer.model.grad_scaler_config]
[components.tagger]
factory = "tagger"
label_smoothing = 0.05
overwrite = false
neg_prefix = "!"
[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.tagger.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.tagger.scorer]
@scorers = "spacy.tagger_scorer.v1"
[components.morphologizer]
factory = "morphologizer"
label_smoothing = 0.05
overwrite = true
extend = false
[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.morphologizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.morphologizer.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.morphologizer.scorer]
@scorers = "spacy.morphologizer_scorer.v1"
[components.trainable_lemmatizer]
factory = "trainable_lemmatizer"
backoff = "orth"
min_tree_freq = 3
overwrite = false
top_k = 1
[components.trainable_lemmatizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false
[components.trainable_lemmatizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.trainable_lemmatizer.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.trainable_lemmatizer.scorer]
@scorers = "spacy.lemmatizer_scorer.v1"
[components.parser]
factory = "parser"
moves = null
update_with_oracle_cut_size = 100
learn_tokens = false
min_action_freq = 30
[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null
[components.parser.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.parser.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.parser.scorer]
@scorers = "spacy.parser_scorer.v1"
[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100
incorrect_spans_key = null
[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null
[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"
[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"
[components.ner.scorer]
@scorers = "spacy.ner_scorer.v1"
[corpora]
[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 3
# 3000 steps is ~40 epochs over this 445k-token corpus, measured at ~29 steps/min on a T4
# (~1.8h). The CPU tiers' 20000/1600 would be ~270 epochs and ~12h here, and worse than
# wasteful: warmup_linear anneals against `total_steps`, so a run stopped early by patience
# never leaves the peak learning rate. Budget and schedule are kept equal on purpose:
# training.optimizer.learn_rate.total_steps must track any change to max_steps.
patience = 600
max_epochs = 0
max_steps = 3000
eval_frequency = 100
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null
[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 1e-08
[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 3000
initial_rate = 5e-5
[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null
[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false
[training.score_weights]
tag_acc = 0.16
pos_acc = 0.08
tag_micro_p = null
tag_micro_r = null
tag_micro_f = null
morph_acc = 0.08
morph_per_feat = null
lemma_acc = 0.16
dep_uas = 0.08
dep_las = 0.16
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = 0.0
ents_f = 0.28
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null
[initialize.tokenizer]
[initialize.components]
[pretraining]

97
docs/MODELS.md
View File

@ -47,7 +47,7 @@ pipelines such as `de_core_news_sm` as `news`.
| `fa_dep_news_lg` | same as `fa_dep_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping | | `fa_dep_news_lg` | same as `fa_dep_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping |
| `fa_core_news_lg` | same as `fa_core_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping | | `fa_core_news_lg` | same as `fa_core_news_sm` | floret, 200k rows / 300d, full-wiki 5 epochs | built, shipping |
| `fa_ent_news_lg` | ner (own internal tok2vec) | floret, 200k rows / 300d, full-wiki 5 epochs | built, optional | | `fa_ent_news_lg` | ner (own internal tok2vec) | floret, 200k rows / 300d, full-wiki 5 epochs | built, optional |
| `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | not on this hardware; 2 GB VRAM cannot fine-tune a 125M-param encoder | | `fa_core_news_trf` | transformer instead of tok2vec | `HooshvareLab/bert-base-parsbert-uncased`, fine-tuned | built on a rented Colab T4 (not on this hardware: 2 GB VRAM cannot fine-tune a 125M-param encoder), shipping with a redistribution caveat because that encoder's card states no licence; §3.4 and §8 |
### Why `core` is honest here ### Why `core` is honest here
@ -424,10 +424,101 @@ at step 7,200, ~13 min, in line with `sm`/`md`.
throughput dropped as expected (9,387 / 6,655 words/s vs `sm`'s 12,505 / 8,834, `md`'s throughput dropped as expected (9,387 / 6,655 words/s vs `sm`'s 12,505 / 8,834, `md`'s
10,493 / 7,269 words/s; the larger table costs real lookup time), but the standalone `ent_lg` run 10,493 / 7,269 words/s; the larger table costs real lookup time), but the standalone `ent_lg` run
showed 15,614 words/s, higher than `sm`/`md`'s ent runs despite an identical `ner` showed 15,614 words/s, higher than `sm`/`md`'s ent runs despite an identical `ner`
architecture and the same larger table. Treat that one figure as single-run CPU contention architecture and the same larger table. That figure was single-run CPU contention noise on
noise on shared hardware, not a real speedup, and re-benchmark before citing it. shared hardware, not a real speedup. Those numbers are superseded by §9, which times
`nlp.pipe` alone instead of reading a scoring-contaminated figure off the benchmark command.
For a 4x download over `md` (and up to 39x over `sm`) buying +1.45 DEP_LAS / +1.23 ENTS_F For a 4x download over `md` (and up to 39x over `sm`) buying +1.45 DEP_LAS / +1.23 ENTS_F
over `md` (+1.45 DEP_LAS / +4.08 ENTS_F over `sm`), `lg` is a server/offline-batch pipeline, over `md` (+1.45 DEP_LAS / +4.08 ENTS_F over `sm`), `lg` is a server/offline-batch pipeline,
not something to ship to a browser or a cold-start function. All three variants (`dep`, not something to ship to a browser or a cold-start function. All three variants (`dep`,
`ent`, `core`) are built and evaluated at this tier, same as `md`. `ent`, `core`) are built and evaluated at this tier, same as `md`.
## 8. The `trf` tier: one fine-tuned ParsBERT
`configs/fa_core_news_trf.cfg` replaces the static-vector tok2vec with
`HooshvareLab/bert-base-parsbert-uncased`, fine-tuned during training. Trained on a rented
Colab T4 in 1h58m: 3000 steps, no early stop, the full learning-rate anneal.
### One corpus, because a transformer cannot be trained twice
The `sm`/`md`/`lg` tiers train `ner` as its own pipeline with its own embedded tok2vec and
then source it into the dep model. That is affordable because a hash-embed tok2vec is cheap.
A 162M-parameter encoder is not: fine-tuning it once per component would double GPU cost and
put two encoders in one wheel, and sourcing the second would collide on the `transformer`
component name.
So every component listens to a single shared transformer through a `TransformerListener`,
which requires one corpus carrying both the UD and NER annotation layers on the same `Doc`.
`scripts/merge_joint_corpus.py` builds it. The fusion is exact rather than approximate:
`corpus/perdt-ner/` was converted from the same `--merge-subtokens` CoNLL-U as
`corpus/merged/` with the same `--n-sents`, so the two DocBins are token-for-token identical.
The script asserts that per document and copies only `doc.ents` across. Char offsets are not
usable for the copy, because the two converters differ in trailing whitespace, which shifts
`char_span` off the token grid and returns None; the transfer goes by token index.
### Results against `lg`
| Metric | `lg` | `trf` | Delta |
| --- | ---: | ---: | ---: |
| `TAG_ACC` | 96.55 | 97.62 | +1.07 |
| `POS_ACC` | 96.68 | 97.63 | +0.95 |
| `MORPH_ACC` | 96.70 | 97.82 | +1.12 |
| `LEMMA_ACC` | 98.08 | 97.31 | -0.77 |
| `DEP_UAS` | 90.96 | 93.87 | +2.91 |
| `DEP_LAS` | 86.60 | 90.79 | +4.19 |
| `SENTS_F` | 99.18 | 97.35 | -1.83 |
| `ENTS_F` | 75.94 | 82.89 | +6.95 |
The parser gain is the headline: `DEP_LAS` 90.79 passes the hazm+ParsBERT reference of 89.34,
which no CPU tier reached. NER gains 6.95 F, almost all of it recall (71.09 to 81.76) at
higher precision, which is what a pretrained encoder buys on the difflib-transferred layer.
Two metrics regress. `SENTS_F` drops 1.83, most likely because `strided_spans` at
`window = 128, stride = 96` leaves 32 tokens of overlap, so tokens near a span edge see
truncated right context where the CPU tiers' tok2vec sees the whole doc. `LEMMA_ACC` drops
0.77 and is the one metric where a static-vector tier wins: `trainable_lemmatizer` reads a
single `reduce_mean`-pooled vector per token, while `lg` runs an edit-tree lemmatizer over
floret subwords that model Persian orthography directly. Neither is a training-length
problem; see TODO.md for the evidence that more steps do not help.
### Cost, and the licence problem
608 MB wheel, 2.6x `lg` and 45x `sm`. 187 words/s on the laptop CPU against `sm`'s 5,484
(§9), so this tier needs a GPU in production rather than merely benefiting from one.
ParsBERT's model card states no licence. §3.4 picked `HooshvareLab/roberta-fa-zwnj-base`
(Apache-2.0) for exactly this reason, and the published wheel therefore embeds weights whose
redistribution terms are unknown. `scripts/finalize_pipeline.py` reads the encoder name out
of the trained config and writes a redistribution warning into `meta.json` when the encoder
has no licence, so the artifact carries the caveat. Retraining on the Apache-2.0 encoder is a
one-line change to `name` in the config.
## 9. Throughput
Measured with `scripts/benchmark_throughput.py`, which times `nlp.pipe` and nothing else.
The `words/s` printed by `spacy benchmark accuracy` runs the Scorer's per-token alignment
inside the timed region, which is why the §7 numbers disagree with these and why one of them
was impossible.
Median of repeated passes over the 146-document PerDT test split (23,825 tokens), batch 32,
warmup discarded. Raw records in `metrics/throughput-*.json`.
| Tier | CPU, i5-7200U | GPU, GeForce 940MX | CPU, Xeon @ 2.00GHz | GPU, Tesla T4 |
| --- | ---: | ---: | ---: | ---: |
| `sm` | 5,484 | 10,235 | | |
| `md` | 5,408 | 9,058 | | |
| `lg` | 4,715 | 9,215 | | |
| `trf` | 187 | 1,106 | 336 | 8,320 |
The CPU tiers sit within about 15% of each other, less than their vector-table sizes suggest,
so the tok2vec lookup is not the bottleneck; the parser and lemmatizer are. Run-to-run spread
on the laptop is roughly 10% either way with thermal state, and a background rsync halved
every number, so treat small differences as noise.
`trf` is 29x slower than `sm` on the same CPU. The T4 and Xeon columns come from the same Colab
VM, giving a clean 25x GPU speedup for the transformer.
`trf` on the 940MX needs a `cu126` torch build. sm_50 kernels were dropped from the `cu128` and
`cu129` wheels at torch 2.8, which is what `pip install torch` resolves to. `.venv-trf-gpu` pins
`torch==2.7.1+cu126`, separate from `.venv` because torch's pinned `nvidia-*` wheels downgrade
the CUDA libraries cupy uses there from 12.9 to 12.6. Batch 32 fits in 2 GB.

101
project.yml
View File

@ -41,6 +41,10 @@ vars:
core_lg_package_name: "core_news_lg" core_lg_package_name: "core_news_lg"
floret_lg_wheel: "fa_floret-0.1.0-py3-none-any-full-wiki-200k-5epoch.whl" floret_lg_wheel: "fa_floret-0.1.0-py3-none-any-full-wiki-200k-5epoch.whl"
vectors_lg_dir: "assets/vectors/fa_floret_lg" vectors_lg_dir: "assets/vectors/fa_floret_lg"
# trf tier: one fine-tuned ParsBERT shared by every component. Needs a real GPU; the
# 940MX cannot fine-tune a 162M-parameter encoder, so `gpu_trf` is set for a rented card.
core_trf_package_name: "core_news_trf"
gpu_trf: 0
directories: directories:
- "assets" - "assets"
@ -125,6 +129,17 @@ workflows:
- compare-md - compare-md
- package-md - package-md
- smoke-md - smoke-md
# The trf tier: one fine-tuned ParsBERT shared by every component, including ner, so it
# trains against a single joint corpus instead of the sm/md/lg dep+ner split. GPU only.
trf:
- merge-joint
- debug-data-trf
- train-trf
- finalize-trf
- evaluate-trf
- finalize-meta-trf
- package-trf
- smoke-trf
commands: commands:
- name: "inspect" - name: "inspect"
@ -604,6 +619,92 @@ commands:
- "training/fa_dep_news_md" - "training/fa_dep_news_md"
- "training/fa_core_news_md" - "training/fa_core_news_md"
# ---------------------------------------------------------------- trf tier
- name: "merge-joint"
help: >
Fuse the UD layer and the transferred NER layer onto one set of Docs. The trf tier
shares a single transformer across every component, so it needs one corpus carrying
both annotation layers; the two DocBins are token-for-token identical by construction
and the script asserts it.
script:
- "python scripts/merge_joint_corpus.py --ud-dir corpus/merged --ner-dir corpus/perdt-ner --out corpus/joint"
deps:
- "corpus/merged/${vars.treebank}-ud-train.spacy"
- "corpus/perdt-ner/train.spacy"
- "scripts/merge_joint_corpus.py"
outputs:
- "corpus/joint/train.spacy"
- "corpus/joint/dev.spacy"
- "corpus/joint/test.spacy"
- name: "debug-data-trf"
help: "Validate the joint corpus against the trf config before renting GPU time"
script:
- "python -m spacy debug data configs/fa_core_news_trf.cfg --paths.train corpus/joint/train.spacy --paths.dev corpus/joint/dev.spacy"
deps:
- "corpus/joint/train.spacy"
- "configs/fa_core_news_trf.cfg"
- name: "train-trf"
help: "Fine-tune ParsBERT with tagger + morphologizer + lemmatizer + parser + ner listening"
script:
- "python -m spacy train configs/fa_core_news_trf.cfg --output training/core-trf --paths.train corpus/joint/train.spacy --paths.dev corpus/joint/dev.spacy --gpu-id ${vars.gpu_trf}"
deps:
- "corpus/joint/train.spacy"
- "corpus/joint/dev.spacy"
- "configs/fa_core_news_trf.cfg"
outputs:
- "training/core-trf/model-best"
- name: "finalize-trf"
help: "Write fa_core_news_trf metadata onto the trained model"
script:
- "python scripts/finalize_pipeline.py training/core-trf/model-best training/fa_core_news_trf --variant core --size trf --version ${vars.package_version}"
deps:
- "training/core-trf/model-best"
- "scripts/finalize_pipeline.py"
outputs:
- "training/fa_core_news_trf"
- name: "evaluate-trf"
help: "Score fa_core_news_trf on the held-out UD and NER test splits"
script:
- "python -m spacy benchmark accuracy training/fa_core_news_trf corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/trf-core-ud-test.json --gpu-id ${vars.gpu_trf}"
- "python -m spacy benchmark accuracy training/fa_core_news_trf corpus/perdt-ner/test.spacy --output metrics/trf-perdt-ner-test.json --gpu-id ${vars.gpu_trf}"
deps:
- "training/fa_core_news_trf"
- "corpus/merged/${vars.treebank}-ud-test.spacy"
- "corpus/perdt-ner/test.spacy"
outputs:
- "metrics/trf-core-ud-test.json"
- "metrics/trf-perdt-ner-test.json"
- name: "finalize-meta-trf"
help: "Fold the trf test scores into meta.json"
script:
- "python scripts/finalize_pipeline.py training/core-trf/model-best training/fa_core_news_trf --variant core --size trf --version ${vars.package_version} --ud-metrics metrics/trf-core-ud-test.json --ner-metrics metrics/trf-perdt-ner-test.json"
deps:
- "metrics/trf-core-ud-test.json"
- "metrics/trf-perdt-ner-test.json"
- "scripts/finalize_pipeline.py"
- name: "package-trf"
help: "Build the installable fa_core_news_trf wheel + sdist"
script:
- "python -m spacy package training/fa_core_news_trf packages --name ${vars.core_trf_package_name} --version ${vars.package_version} --build sdist,wheel --force"
deps:
- "training/fa_core_news_trf"
outputs:
- "packages/${vars.lang}_${vars.core_trf_package_name}-${vars.package_version}"
- name: "smoke-trf"
help: "Load fa_core_news_trf and run it over real Persian text"
script:
- "python scripts/smoke_test.py training/fa_core_news_trf"
deps:
- "training/fa_core_news_trf"
- name: "clean" - name: "clean"
help: "Drop corpora, training runs and metrics (keeps downloaded assets)" help: "Drop corpora, training runs and metrics (keeps downloaded assets)"

112
scripts/benchmark_throughput.py Normal file
View File

@ -0,0 +1,112 @@
"""Measure inference throughput (words/second) for a pipeline, on CPU or GPU.
`spacy benchmark accuracy` prints a speed number, but it is scoring-contaminated: the
Scorer's per-token alignment and per-type bookkeeping run inside the timed region, which
matters a lot for the cheap CPU tiers and understates them. This times `nlp.pipe` only.
Reported figure is the median of `--runs` passes over the same texts, after a discarded
warmup pass. Median rather than mean because the first CUDA kernel launches, cuBLAS
autotuning and any page-cache miss produce outliers that a mean would smear into the result.
Batch size matters far more for the trf tier than the CPU tiers (a transformer amortizes a
GEMM over the batch; a hash-embed tok2vec barely cares), so it is a parameter and gets
recorded in the output rather than being left implicit.
"""
import argparse
import json
import platform
import statistics
import subprocess
import time
from pathlib import Path
import spacy
from spacy.tokens import DocBin
def cpu_model():
try:
for line in Path("/proc/cpuinfo").read_text().splitlines():
if line.startswith("model name"):
return line.split(":", 1)[1].strip()
except OSError:
pass
return platform.processor() or "unknown"
def gpu_model():
try:
out = subprocess.run(
["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader"],
capture_output=True, text=True, timeout=30,
)
if out.returncode == 0:
return out.stdout.strip().splitlines()[0].strip()
except (OSError, subprocess.SubprocessError):
pass
return "unknown"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("model", help="installed package name or path to a pipeline")
ap.add_argument("--corpus", default="corpus/merged/fa_perdt-ud-test.spacy",
help="DocBin whose raw texts are used as input")
ap.add_argument("--gpu-id", type=int, default=-1, help="-1 for CPU")
ap.add_argument("--batch-size", type=int, default=32)
ap.add_argument("--runs", type=int, default=3)
ap.add_argument("--limit", type=int, default=0, help="cap number of docs (0 = all)")
ap.add_argument("--output", default=None, help="write a JSON record here")
args = ap.parse_args()
if args.gpu_id >= 0:
# require_gpu, not prefer_gpu: a silent fall back to CPU would be reported as a GPU
# number, which is exactly the measurement error this script exists to avoid.
spacy.require_gpu(args.gpu_id)
device = f"gpu:{args.gpu_id} ({gpu_model()})"
else:
device = f"cpu ({cpu_model()})"
nlp = spacy.load(args.model)
vocab_docs = list(DocBin().from_disk(args.corpus).get_docs(spacy.blank("fa").vocab))
if args.limit:
vocab_docs = vocab_docs[:args.limit]
texts = [d.text for d in vocab_docs]
n_words = sum(len(d) for d in vocab_docs)
# Warmup: first pass pays for lazy CUDA context creation, cuBLAS handles and any
# transformer weight transfer. Timing it would misattribute setup cost to throughput.
for _ in nlp.pipe(texts[:args.batch_size], batch_size=args.batch_size):
pass
wps = []
for _ in range(args.runs):
t0 = time.perf_counter()
for _ in nlp.pipe(texts, batch_size=args.batch_size):
pass
elapsed = time.perf_counter() - t0
wps.append(n_words / elapsed)
median = statistics.median(wps)
record = {
"model": args.model,
"pipeline": list(nlp.pipe_names),
"device": device,
"batch_size": args.batch_size,
"docs": len(texts),
"words": n_words,
"runs": [round(w, 1) for w in wps],
"wps_median": round(median, 1),
"spacy_version": spacy.__version__,
}
print(json.dumps(record, indent=2, ensure_ascii=False))
if args.output:
p = Path(args.output)
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(record, indent=2, ensure_ascii=False) + "\n")
print(f"wrote {p}")
if __name__ == "__main__":
main()

71
scripts/finalize_pipeline.py
View File

@ -73,11 +73,15 @@ FLORET_LG = {
"author": "Kiyarash Fazeli", "author": "Kiyarash Fazeli",
"license": "CC BY-SA 4.0", "license": "CC BY-SA 4.0",
} }
TRANSFORMER = { # Whatever encoder the config actually names wins; hardcoding one would silently mislabel a
"name": "HooshvareLab/roberta-fa-zwnj-base", # wheel the moment configs/fa_core_news_trf.cfg's `name` changes. Licences are recorded per
"url": "https://huggingface.co/HooshvareLab/roberta-fa-zwnj-base", # encoder because they differ sharply, and two of the Persian ones have none at all.
"author": "Hooshvare Team", ENCODER_LICENSES = {
"license": "Apache-2.0", "HooshvareLab/roberta-fa-zwnj-base": ("Hooshvare Team", "Apache-2.0"),
"HooshvareLab/bert-fa-zwnj-base": ("Hooshvare Team", "Apache-2.0"),
"m3hrdadfi/albert-fa-base-v2": ("Mehrdad Farahani", "Apache-2.0"),
"HooshvareLab/bert-base-parsbert-uncased": ("Hooshvare Team", "no licence stated on the model card"),
"sbunlp/fabert": ("SBU NLP Lab", "no licence stated on the model card"),
} }
NER_NOTE = ( NER_NOTE = (
@ -126,12 +130,48 @@ def vectors_note_lg(nlp):
) )
TRANSFORMER_NOTE = ( def encoder_name(nlp):
"This is the `trf` tier: no static vectors; contextual embeddings instead come from a " """Read the encoder out of the trained pipeline's own config."""
"fine-tuned HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) transformer via " try:
"spacy-transformers. Not ParsBERT: its model card carries no licence. GPU is recommended " return nlp.config["components"]["transformer"]["model"]["name"]
"for both training and inference." except KeyError:
raise SystemExit(
"--size trf expects a pipeline with a `transformer` component whose model names "
f"an encoder; got pipeline {list(nlp.pipe_names)}"
) )
def transformer_source(nlp):
name = encoder_name(nlp)
author, license_ = ENCODER_LICENSES.get(name, ("unknown", "unknown, check the model card"))
return {
"name": name,
"url": f"https://huggingface.co/{name}",
"author": author,
"license": license_,
}
def transformer_note(nlp):
name = encoder_name(nlp)
_, license_ = ENCODER_LICENSES.get(name, ("unknown", "unknown, check the model card"))
note = (
f"This is the `trf` tier: no static vectors. Contextual embeddings come from a "
f"fine-tuned {name} ({license_}) via spacy-transformers, shared by every component "
f"through a TransformerListener, so one encoder forward pass serves the tagger, "
f"morphologizer, lemmatizer, parser and ner. Unlike the sm/md/lg tiers the ner is "
f"trained jointly rather than sourced, because a shared encoder cannot be fine-tuned "
f"twice and then merged. GPU is strongly recommended for both training and inference."
)
if "no licence" in license_ or license_.startswith("unknown"):
note += (
f" REDISTRIBUTION WARNING: {name} states no licence, so this wheel embeds weights "
f"whose terms are unknown and must not be republished. Retrain against "
f"HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) for a publishable artifact."
)
return note
# CC BY-SA 4.0 on the treebank propagates to anything derived from it. # CC BY-SA 4.0 on the treebank propagates to anything derived from it.
PERDT_LICENSE = "CC BY-SA 4.0" PERDT_LICENSE = "CC BY-SA 4.0"
ATTRIBUTION = ( ATTRIBUTION = (
@ -222,8 +262,15 @@ def main():
sources.append(FLORET_LG) sources.append(FLORET_LG)
notes = " ".join([notes, vectors_note_lg(nlp)]) notes = " ".join([notes, vectors_note_lg(nlp)])
elif args.size == "trf": elif args.size == "trf":
sources.append(TRANSFORMER) sources.append(transformer_source(nlp))
notes = " ".join([notes, TRANSFORMER_NOTE]) notes = " ".join([notes, transformer_note(nlp)])
# The stock description advertises a CPU tok2vec pipeline, which is wrong here.
description = (
"Persian pipeline built on a fine-tuned "
f"{encoder_name(nlp)} transformer. Components: transformer, tagger, "
"morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, "
"DAT, MON, TIM, PCT. GPU recommended."
)
if args.add_ner: if args.add_ner:
ner_nlp = spacy.load(args.add_ner) ner_nlp = spacy.load(args.add_ner)
if ner_nlp.pipe_names != ["ner"]: if ner_nlp.pipe_names != ["ner"]:

136
scripts/make_model_card.py Normal file
View File

@ -0,0 +1,136 @@
"""Build the Hugging Face model card for a packaged pipeline.
`spacy package` already writes a README into the wheel, and `spacy huggingface-hub push`
uploads it as the card. That card is a metadata dump: no install line, no usage, no
throughput, and no YAML frontmatter, so the Hub cannot index the model by language or task.
This composes a card from the same sources of truth (`meta.json` and the JSON written by
scripts/benchmark_throughput.py) rather than from hand-copied numbers, so the card cannot
drift from the artifact it describes.
"""
import argparse
import json
from pathlib import Path
# meta.json key -> (row label, reference note). Only keys the pipeline actually evidences
# are emitted; a missing key means the corpus could not score it.
METRICS = [
("token_acc", "Tokenization accuracy", ""),
("tag_acc", "XPOS tag accuracy", ""),
("pos_acc", "UPOS tag accuracy", ""),
("morph_acc", "Morphological features", ""),
("lemma_acc", "Lemma accuracy", ""),
("dep_uas", "Unlabelled attachment (UAS)", ""),
("dep_las", "Labelled attachment (LAS)", ""),
("sents_f", "Sentence segmentation F", ""),
("ents_p", "NER precision", ""),
("ents_r", "NER recall", ""),
("ents_f", "NER F-score", ""),
]
def load(path):
return json.loads(Path(path).read_text())
def throughput_rows(paths):
rows = []
for p in paths:
if not Path(p).exists():
continue
d = load(p)
rows.append((d["device"], d["batch_size"], d["wps_median"]))
return rows
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--meta", required=True, help="meta.json of the finalized pipeline")
ap.add_argument("--throughput", nargs="*", default=[], help="benchmark_throughput JSONs")
ap.add_argument("--repo-id", required=True, help="e.g. Phazel/fa_core_news_trf")
ap.add_argument("--wheel-name", required=True)
ap.add_argument("--out", required=True)
args = ap.parse_args()
meta = load(args.meta)
name = f"{meta['lang']}_{meta['name']}"
perf = meta.get("performance", {})
lines = []
# Frontmatter: without this the Hub cannot filter the model by language or library.
lines += [
"---",
"language:",
"- fa",
f"license: {meta.get('license', 'cc-by-sa-4.0').lower().replace(' ', '-')}",
"library_name: spacy",
"pipeline_tag: token-classification",
"tags:",
"- spacy",
"- token-classification",
"- persian",
"- farsi",
"---",
"",
f"# {name}",
"",
meta.get("description", "").strip(),
"",
]
lines += [
"## Install",
"",
"```bash",
f"pip install https://huggingface.co/{args.repo_id}/resolve/main/{args.wheel_name}",
"```",
"",
"```python",
"import spacy",
f'nlp = spacy.load("{name}")',
'doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می\u200cدهد.")',
"print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])",
"print([(e.text, e.label_) for e in doc.ents])",
"```",
"",
]
lines += ["## Accuracy", "",
"Scored with `spacy benchmark accuracy` on the held-out PerDT test split.",
"", "| Metric | Score |", "| --- | ---: |"]
for key, label, _ in METRICS:
v = perf.get(key)
if isinstance(v, (int, float)):
lines.append(f"| {label} | {v * 100:.2f} |")
lines.append("")
rows = throughput_rows(args.throughput)
if rows:
lines += ["## Throughput", "",
"Median of repeated `nlp.pipe` passes over the 146-document PerDT test",
"split (23,825 tokens), timing the pipe only. Warmup pass discarded.",
"", "| Device | Batch | Words/s |", "| --- | ---: | ---: |"]
for device, batch, wps in rows:
lines.append(f"| {device} | {batch} | {wps:,.0f} |")
lines.append("")
lines += ["## Sources", "", "| Source | Author | Licence |", "| --- | --- | --- |"]
for s in meta.get("sources", []):
url, nm = s.get("url"), s.get("name", "")
label = f"[{nm}]({url})" if url else nm
lines.append(f"| {label} | {s.get('author', '')} | {s.get('license', '')} |")
lines.append("")
notes = (meta.get("notes") or "").strip()
if notes:
lines += ["## Notes", "", notes, ""]
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(lines), encoding="utf-8")
print(f"wrote {out} ({out.stat().st_size} bytes)")
if __name__ == "__main__":
main()

68
scripts/merge_joint_corpus.py Normal file
View File

@ -0,0 +1,68 @@
"""Fuse the UD annotation layer and the transferred NER layer into one DocBin.
The sm/md/lg tiers train `ner` as a separate pipeline with its own embedded tok2vec, then
source it into the dep model (project.yml `assemble-core`). That works because a hash-embed
tok2vec is cheap enough to train twice.
A transformer is not. Fine-tuning ParsBERT once per component would double GPU cost and
produce a package carrying two independent 162M-parameter encoders, and sourcing the second
one would collide on the `transformer` component name. So the trf tier trains every component
against a single shared transformer via TransformerListener, which requires a single corpus
carrying both annotation layers on the same Doc.
That fusion is exact, not approximate: `corpus/perdt-ner/` was produced by
scripts/transfer_perdt_ner.py from the same `--merge-subtokens` CoNLL-U as `corpus/merged/`,
then converted with the same `--n-sents`, so the two DocBins are token-for-token identical
(verified below and asserted at runtime). Only `doc.ents` is copied across; every other
annotation stays on the UD doc.
"""
import argparse
from pathlib import Path
import spacy
from spacy.tokens import DocBin, Span
SPLITS = (("train", "fa_perdt-ud-train"), ("dev", "fa_perdt-ud-dev"), ("test", "fa_perdt-ud-test"))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--ud-dir", default="corpus/merged")
ap.add_argument("--ner-dir", default="corpus/perdt-ner")
ap.add_argument("--out", default="corpus/joint")
ap.add_argument("--lang", default="fa")
args = ap.parse_args()
nlp = spacy.blank(args.lang)
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
for split, ud_stem in SPLITS:
ud_docs = list(DocBin().from_disk(Path(args.ud_dir) / f"{ud_stem}.spacy").get_docs(nlp.vocab))
ner_docs = list(DocBin().from_disk(Path(args.ner_dir) / f"{split}.spacy").get_docs(nlp.vocab))
if len(ud_docs) != len(ner_docs):
raise SystemExit(
f"{split}: {len(ud_docs)} UD docs vs {len(ner_docs)} NER docs; the two corpora "
"were not converted from the same source with the same --n-sents"
)
db = DocBin(store_user_data=True)
n_ents = 0
for i, (ud, ner) in enumerate(zip(ud_docs, ner_docs)):
if [t.text for t in ud] != [t.text for t in ner]:
raise SystemExit(f"{split} doc {i}: tokenization differs between UD and NER layers")
# Tokens are index-aligned, so rebuild by token index. Char offsets are NOT
# safe here: the two converters can differ in trailing whitespace, which shifts
# `char_span` off the token grid and silently yields None.
ud.ents = [Span(ud, e.start, e.end, label=e.label_) for e in ner.ents]
n_ents += len(ud.ents)
db.add(ud)
dest = out / f"{split}.spacy"
db.to_disk(dest)
print(f"{dest}: {len(ud_docs)} docs, {n_ents} entities")
if __name__ == "__main__":
main()