The 940MX does run trf, at 1,158 words/s with batch 32 inside 2 GB, 6.2x its host
CPU. The earlier claim that current PyTorch wheels cannot target sm_50 was only
half right: Maxwell kernels were dropped from the cu128 and cu129 builds starting
torch 2.8, which is what `pip install torch` now resolves to, but the cu126 build
of 2.7.1 still ships sm_50 and works.
.venv-trf-gpu pins torch==2.7.1+cu126 for this and stays separate from .venv,
whose cupy runs on nvidia-* 12.9 wheels that torch would downgrade to 12.6.
Drop the editorial sentence from the generated model card; the table states it.
The Hazm and en_core_web_sm comparison now cites fa_core_news_trf, which is the
strongest tier and the only one to pass the hazm+ParsBERT DEP_LAS reference.
Drop the package-size row from that table: it compared a 608 MB transformer
against a 7 MB rule-based toolkit, which says nothing about accuracy.
Add trf to the package list and to the per-label NER table, alongside lg, which
had never been added. Its largest per-label gains are PER +9.25 and DAT +11.69
over lg. Drop the gold-count column there and trim the surrounding notes.
Trains tagger, morphologizer, trainable_lemmatizer, parser and ner against one
fine-tuned HooshvareLab/bert-base-parsbert-uncased through TransformerListener,
rather than the sm/md/lg split where ner is trained separately and sourced in.
Fine-tuning a 162M-parameter encoder twice would double GPU cost, ship two
encoders in one wheel, and collide on the `transformer` component name.
A shared encoder needs one corpus carrying both annotation layers, so
merge_joint_corpus.py fuses them. corpus/perdt-ner/ came from the same
--merge-subtokens CoNLL-U as corpus/merged/ with the same --n-sents, so the
DocBins are token-for-token identical; the script asserts that per document and
copies doc.ents by token index. Char offsets do not work here because the two
converters differ in trailing whitespace, which pushes char_span off the token
grid and returns None.
Trained on a Colab T4 in 1h58m, 3000 steps, no early stop. Test scores against
lg: DEP_LAS 90.79 (+4.19), ENTS_F 82.89 (+6.95, almost all recall), TAG_ACC
97.62 (+1.07). DEP_LAS passes the hazm+ParsBERT reference of 89.34, which no CPU
tier reached. LEMMA_ACC 97.31 and SENTS_F 97.35 regress against lg; the likely
cause of the latter is strided_spans leaving only 32 tokens of overlap.
max_steps and learn_rate.total_steps are held equal on purpose. They are
independent knobs, and a patience stop under a longer total_steps ends training
at a high learning rate, discarding the annealing tail.
finalize_pipeline.py now reads the encoder name out of the trained config
instead of hardcoding it, tracks per-encoder licences, and writes a
redistribution warning into meta.json when the encoder states none. ParsBERT
states none, so that wheel is not redistributable; roberta-fa-zwnj-base
(Apache-2.0) is a one-line change to `name`.
Add benchmark_throughput.py, which times nlp.pipe alone. The words/s from
`spacy benchmark accuracy` includes the Scorer's per-token alignment, which is
why MODELS.md 7 reported ent_lg as faster than sm despite an identical ner
architecture. Remeasured every tier on CPU and the 940MX; a background rsync
halved every figure, so the final numbers are 9-run medians on an idle machine.
Add make_model_card.py, which composes the Hub card from meta.json and the
throughput records. The card spacy package writes has no frontmatter, so the Hub
cannot index the model by language, and no install line or usage.
- project.yml: new md workflow (vectors-md, train-dep-md, train-ner-md,
finalize-dep-md, assemble-core-md, evaluate-md, finalize-meta-md,
compare-md, package-md, smoke-md), same corpus/architecture as sm plus
the fa_floret static vector table (50k rows x 300d, 400k Persian
documents).
- configs/fa_dep_news_md.cfg, configs/fa_ner_md.cfg: byte-identical to the
sm configs except include_static_vectors, isolating what the vectors buy.
- scripts/unpack_vectors.py: extracts a floret wheel's vectors-only
pipeline into a directory --paths.vectors can point at.
- scripts/finalize_pipeline.py: --size now accepts md (floret source +
vectors note), plus lg/trf (used by the pending Colab notebook on
colab-lg-trf-training; lg's vectors note is generated from the trained
model's actual vector table shape since that tier is still being
iterated on).
- scripts/compare_tiers.py: sm vs md metrics diff.
- docs/MODELS.md, README.md, README.fa.md: md tier results, fa_ent_news_md
package row, and the analysis of why floret helps NER recall.
- docs/CONTRIBUTING-GUIDE.md, README.md, README.fa.md: fixed every
documented pip install URL. spacy huggingface-hub push names the
uploaded wheel '<name>-any-py3-none-any.whl'; 'any' is not a valid PEP
440 version, so current pip rejects it. Re-uploaded a correctly
versioned copy of every published wheel (fa_core_news_sm, fa_dep_news_sm,
fa_ent_news_md) to the Hub and repointed the docs at that filename.
Mohamad Fazeli