fa_core_news_sm: Persian spaCy pipeline from UD_Persian-PerDT + ParsTwiNER
No trained Persian pipeline exists for spaCy: spacy.load("fa_core_news_sm") has
never worked. This builds one from openly-licensed data so the result can actually
be redistributed.
Test scores (held-out splits): TAG 95.96, POS 96.24, MORPH 96.29, LEMMA 97.91,
UAS 89.69, LAS 85.15, ENTS_F 67.22. ~9,250 words/s, 13 MB wheel. 1h27m on 4 CPU
cores, no GPU.
Corpus choices, with evidence:
- UD_Persian-PerDT (CC BY-SA 4.0) over Seraji: 3.7x more tokens (452k vs 121k) and
Seraji has no PROPN tag at all. hazm's own spaCy parser used PerDT too.
- ParsTwiNER (MIT) for NER. ARMAN/PEYMA/NSURL are research-only; spaCy never
shipped the 2018 Persian models precisely because of corpus licensing. Cost: NER
is trained on tweets, so it is the weak component.
- --merge-subtokens: measured token F 0.9887 vs 0.9823 split. Without it, 1.5% of
gold token boundaries are unreachable by the tokenizer we ship.
- morphologizer + trainable_lemmatizer instead of the English attribute_ruler +
rule lemmatizer, because UD gives gold UPOS/FEATS/lemmas to train and measure on.
The ner component embeds its own tok2vec rather than using a Tok2VecListener, so it
can be sourced into the core pipeline after being trained on a separate corpus.
Also documents an upstream bug: spacy/lang/fa/syntax_iterators.py matches ClearNLP
labels (dobj, pobj, nsubjpass, attr, dative) that do not exist in UD, so
doc.noun_chunks returns bare head nouns (1.31 vs 2.77 tokens/chunk). Patch and
tests in docs/upstream/fa-noun-chunks.md.
This commit is contained in:
commit
92fc1c3002
|
|
@ -0,0 +1,10 @@
|
|||
# Downloaded corpora — reproduce with `spacy project assets` (checksums in project.yml)
|
||||
assets/
|
||||
# Generated artifacts
|
||||
corpus/
|
||||
training/
|
||||
metrics/
|
||||
packages/
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
|
@ -0,0 +1,164 @@
|
|||
# fa_core_news_sm — a Persian pipeline for spaCy
|
||||
|
||||
There is no trained Persian pipeline for spaCy. `spacy.load("fa_core_news_sm")` has never
|
||||
worked; `spacy.blank("fa")` gives you a tokenizer and stop words and nothing else. This
|
||||
project builds the missing pipeline from openly-licensed data, using spaCy's own tooling, so
|
||||
the result can actually be redistributed.
|
||||
|
||||
- **What the four English pipelines are, and what the four Persian equivalents should be:**
|
||||
[`docs/MODELS.md`](docs/MODELS.md)
|
||||
- **How models get contributed/published in the spaCy ecosystem, and what upstream `fa`
|
||||
already has:** [`docs/CONTRIBUTING-GUIDE.md`](docs/CONTRIBUTING-GUIDE.md)
|
||||
- **The build itself:** [`project.yml`](project.yml)
|
||||
|
||||
## The short version
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Pipeline | `fa_core_news_sm` — tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner |
|
||||
| Syntax/morphology data | [UD_Persian-PerDT](https://github.com/UniversalDependencies/UD_Persian-PerDT) (PerUDT v1.0) — 29,107 sentences, **CC BY-SA 4.0** |
|
||||
| NER data | [ParsTwiNER](https://github.com/overfit-ir/parstwiner) — 7,667 tweets, **MIT** |
|
||||
| Language data | `spacy/lang/fa` upstream (its stop word list comes from hazm) |
|
||||
| Licence of the result | CC BY-SA 4.0 (inherited from the treebank) |
|
||||
| Hardware | 4-core CPU. No GPU needed. |
|
||||
|
||||
### Results
|
||||
|
||||
Trained and evaluated on this laptop (4-core i5-7200U, CPU only, 1h27m for the UD
|
||||
components, ~25 min for NER). Scores are on the **held-out test splits**, produced by
|
||||
`spacy benchmark accuracy` and stored in `metrics/`.
|
||||
|
||||
| Metric | `fa_core_news_sm` | reference |
|
||||
| --- | --- | --- |
|
||||
| `TOKEN_ACC` / `TOKEN_F` | 99.96 / 99.11 | |
|
||||
| `TAG_ACC` (XPOS) | **95.96** | |
|
||||
| `POS_ACC` (UPOS) | **96.24** | |
|
||||
| `MORPH_ACC` | 96.29 | |
|
||||
| `LEMMA_ACC` | **97.91** | |
|
||||
| `SENTS_F` | 99.25 | |
|
||||
| `DEP_UAS` | **89.69** | hazm+ParsBERT: 92.46 |
|
||||
| `DEP_LAS` | **85.15** | hazm+ParsBERT: 89.34 |
|
||||
| `ENTS_P` / `ENTS_R` / `ENTS_F` | 74.77 / 61.06 / **67.22** | |
|
||||
| Speed | ~9,250 words/s (CPU) | |
|
||||
| Wheel size | 13 MB | `en_core_web_sm`: 12 MB |
|
||||
|
||||
Per-entity F: `LOC` 73.9, `PER` 69.1, `NAT` 63.2, `ORG` 59.3, `POG` 41.2, `EVE` 30.0.
|
||||
|
||||
Read these honestly:
|
||||
|
||||
- **Parsing is 4.2 LAS behind hazm's parser**, which is the expected gap between a 13 MB
|
||||
CPU model with hash embeddings and a fine-tuned ParsBERT. It is the same corpus and the
|
||||
same spaCy parser architecture, so the comparison is fair, and it sets the target for the
|
||||
future `trf` tier.
|
||||
- **NER is the weak component.** 67 F reflects three compounding handicaps: an `sm` model
|
||||
with no static vectors, a 233k-token training corpus, and a genre mismatch (trained on
|
||||
tweets, most users will run it on prose). `EVE` and `POG` are near-useless. This is the
|
||||
price of using the only MIT-licensed Persian NER corpus that exists.
|
||||
- **Everything else is competitive with the English `sm` pipeline** (`en_core_web_sm`:
|
||||
TAG 97, LAS 90, ENTS_F 84 — on a much larger and cleaner corpus).
|
||||
|
||||
Reproduce: `.venv/bin/python -m spacy project run all`.
|
||||
|
||||
### Install the built pipeline
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pip install packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl
|
||||
```
|
||||
|
||||
```python
|
||||
import spacy
|
||||
nlp = spacy.load("fa_core_news_sm")
|
||||
doc = nlp("دانشگاه تهران در سال ۱۳۱۳ تأسیس شد.")
|
||||
print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
|
||||
print(doc.ents) # (دانشگاه تهران, ORG)
|
||||
```
|
||||
|
||||
### Why not hazm's own models
|
||||
|
||||
hazm is the reference Persian NLP toolkit and it *does* publish spaCy-format pipelines on the
|
||||
HF Hub, so it was the obvious starting point. It does not survive contact:
|
||||
|
||||
- Its trainable models are pycrfsuite CRFs (`hazm/sequence_tagger.py`). There is no
|
||||
`config.cfg` and no `spacy train` anywhere in the repo — the `Spacy*` classes only download
|
||||
pretrained pipelines.
|
||||
- Those pretrained pipelines are three *single-task* models (`transformer + tagger`,
|
||||
`transformer + parser`, `transformer + chunker`), each `version: 0.0.0` with an empty
|
||||
`license` field, pinned to spaCy 3.6. Using all three means three ParsBERT forward passes
|
||||
over the same text and no shared `Doc`.
|
||||
- Its tokenizer is deliberately incompatible with UD tokenization: the normaliser fuses ZWNJ
|
||||
affixes and `join_verb_parts()` glues multi-word verb chains into single tokens.
|
||||
- Most corpora it reads (Bijankhan, Peykare, Hamshahri, raw PerDT) are gated behind
|
||||
`peykaregan.ir` / `dadegan.ir` under research-only terms.
|
||||
|
||||
What hazm *does* give us: confirmation of the corpus choice — hazm's own spaCy parser was
|
||||
trained on `modified_fa_perdt-ud-train.spacy`, i.e. the same treebank we use — plus the stop
|
||||
word list already vendored into `spacy/lang/fa`. Full analysis in
|
||||
[`docs/MODELS.md`](docs/MODELS.md) §4.
|
||||
|
||||
### Why licensing is the load-bearing constraint
|
||||
|
||||
spaCy's maintainers state that Persian models trained back in 2018 were never published
|
||||
*because of corpus licensing* (spaCy discussion #8233, after PR #2797 added `fa` tokenizer
|
||||
support). The standard Persian NER corpora — ARMAN, PEYMA, NSURL — are all "research use
|
||||
only", and wrapping them in an Apache-2.0 toolkit does not launder that. ParsTwiNER (MIT) is
|
||||
the only redistributable Persian NER corpus we could verify, which is why the NER component is
|
||||
trained on tweets. That trade-off is recorded in the model's `meta.json["notes"]`.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Python 3.12
|
||||
python -m venv .venv
|
||||
.venv/bin/python -m pip install -U pip
|
||||
.venv/bin/python -m pip install "spacy>=3.8,<3.9" spacy-lookups-data
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
Everything is driven by [`project.yml`](project.yml):
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m spacy project assets # download + checksum the corpora
|
||||
.venv/bin/python -m spacy project run all # inspect -> convert -> train -> assemble -> evaluate -> package
|
||||
```
|
||||
|
||||
Individual steps:
|
||||
|
||||
| Command | What it does |
|
||||
| --- | --- |
|
||||
| `inspect` | annotation coverage of the treebanks (`scripts/inspect_treebanks.py`) |
|
||||
| `convert-ud` | CoNLL-U → `DocBin` with `--merge-subtokens`, plus the tokenizer-agreement report |
|
||||
| `convert-ner` | unpack ParsTwiNER, IOB2 → `DocBin` |
|
||||
| `debug-data` | `spacy debug data` on both corpora before spending CPU |
|
||||
| `train-core` | tagger + morphologizer + trainable_lemmatizer + parser on PerDT |
|
||||
| `train-ner` | standalone `ner` with its own embedded tok2vec on ParsTwiNER |
|
||||
| `assemble` | source `ner` into the core pipeline, write full `meta.json` (`scripts/assemble_core.py`) |
|
||||
| `evaluate` | `spacy benchmark accuracy` on both held-out test sets |
|
||||
| `package` | build the wheel + sdist |
|
||||
| `smoke` | run the pipeline over real Persian text and print every annotation layer |
|
||||
|
||||
The two training runs are independent and can run concurrently — each is single-threaded.
|
||||
|
||||
## Design decisions worth knowing before you touch anything
|
||||
|
||||
1. **`--merge-subtokens`.** spaCy has no multiword-token layer, and PerDT splits pronominal
|
||||
clitics (`پدرم` → `پدر` + `م`). Measured on dev: merging gives token F 0.9887 vs 0.9823 for
|
||||
the split version, at the cost of 34 composite XPOS tags on 1.5% of tokens. Merging wins
|
||||
because otherwise 1.5% of gold tokens are boundaries the shipped tokenizer can never
|
||||
produce. Numbers: `scripts/tokenization_report.py`.
|
||||
2. **`ner` carries its own tok2vec.** A `Tok2VecListener` only resolves inside the pipeline it
|
||||
was trained in, so a listener-based component cannot be sourced into another pipeline.
|
||||
`configs/fa_ner_sm.cfg` embeds the tok2vec instead — the same design as `en_core_web_sm`.
|
||||
3. **`morphologizer` + `trainable_lemmatizer` instead of `attribute_ruler` + rule lemmatizer.**
|
||||
The English pipelines derive UPOS from PTB tags by rule because OntoNotes has no UPOS. UD
|
||||
gives us gold UPOS, FEATS and lemmas, so we train on them and get real `pos_acc`,
|
||||
`morph_acc` and `lemma_acc` numbers instead of unmeasurable rule coverage.
|
||||
4. **PerDT, not Seraji.** 3.7× more tokens, and Seraji has no `PROPN` tag at all.
|
||||
|
||||
## Roadmap
|
||||
|
||||
`md`/`lg` need floret vectors trained on Persian Wikipedia + OSCAR (see
|
||||
`spacy-vectors-builder`); floret rather than classic fastText because Persian's ZWNJ usage is
|
||||
inconsistent and explodes the surface vocabulary. `trf` needs a rented GPU and should use
|
||||
`HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) rather than ParsBERT, whose model card
|
||||
carries no licence. `senter` is one extra training run away. Details in `docs/MODELS.md` §2.
|
||||
|
|
@ -0,0 +1,231 @@
|
|||
# fa_core_news_sm — UD components (tagger, morphologizer, trainable_lemmatizer, parser).
|
||||
#
|
||||
# Generated with:
|
||||
# spacy init config configs/fa_core_news_sm.cfg --lang fa \
|
||||
# --pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency
|
||||
#
|
||||
# Deviations from the generated defaults:
|
||||
# - eval_frequency 200 -> 400 (dev set is only 146 docs; 200 wastes CPU on a 4-core box)
|
||||
# The `ner` component is trained separately (configs/fa_ner_sm.cfg) from a different
|
||||
# corpus and merged in by scripts/assemble_core.py.
|
||||
|
||||
[paths]
|
||||
train = null
|
||||
dev = null
|
||||
vectors = null
|
||||
init_tok2vec = null
|
||||
|
||||
[system]
|
||||
gpu_allocator = null
|
||||
seed = 0
|
||||
|
||||
[nlp]
|
||||
lang = "fa"
|
||||
pipeline = ["tok2vec", "tagger", "morphologizer", "trainable_lemmatizer", "parser"]
|
||||
batch_size = 1000
|
||||
disabled = []
|
||||
before_creation = null
|
||||
after_creation = null
|
||||
after_pipeline_creation = null
|
||||
|
||||
[corpora]
|
||||
|
||||
[training]
|
||||
dev_corpus = "corpora.dev"
|
||||
train_corpus = "corpora.train"
|
||||
seed = ${system.seed}
|
||||
gpu_allocator = ${system.gpu_allocator}
|
||||
dropout = 0.1
|
||||
accumulate_gradient = 1
|
||||
patience = 1600
|
||||
max_epochs = 0
|
||||
max_steps = 20000
|
||||
eval_frequency = 400
|
||||
frozen_components = []
|
||||
annotating_components = []
|
||||
before_to_disk = null
|
||||
before_update = null
|
||||
|
||||
[initialize]
|
||||
vectors = ${paths.vectors}
|
||||
init_tok2vec = ${paths.init_tok2vec}
|
||||
vocab_data = null
|
||||
lookups = null
|
||||
before_init = null
|
||||
after_init = null
|
||||
|
||||
[components]
|
||||
|
||||
[pretraining]
|
||||
|
||||
[nlp.tokenizer]
|
||||
@tokenizers = "spacy.Tokenizer.v1"
|
||||
|
||||
[nlp.vectors]
|
||||
@vectors = "spacy.Vectors.v1"
|
||||
|
||||
[corpora.train]
|
||||
@readers = "spacy.Corpus.v1"
|
||||
path = ${paths.train}
|
||||
max_length = 0
|
||||
gold_preproc = false
|
||||
limit = 0
|
||||
augmenter = null
|
||||
|
||||
[corpora.dev]
|
||||
@readers = "spacy.Corpus.v1"
|
||||
path = ${paths.dev}
|
||||
max_length = 0
|
||||
gold_preproc = false
|
||||
limit = 0
|
||||
augmenter = null
|
||||
|
||||
[training.optimizer]
|
||||
@optimizers = "Adam.v1"
|
||||
beta1 = 0.9
|
||||
beta2 = 0.999
|
||||
L2_is_weight_decay = true
|
||||
L2 = 0.01
|
||||
grad_clip = 1.0
|
||||
use_averages = false
|
||||
eps = 1e-08
|
||||
learn_rate = 0.001
|
||||
|
||||
[training.batcher]
|
||||
@batchers = "spacy.batch_by_words.v1"
|
||||
discard_oversize = false
|
||||
tolerance = 0.2
|
||||
get_length = null
|
||||
|
||||
[training.logger]
|
||||
@loggers = "spacy.ConsoleLogger.v1"
|
||||
progress_bar = false
|
||||
|
||||
[training.score_weights]
|
||||
tag_acc = 0.25
|
||||
pos_acc = 0.12
|
||||
tag_micro_p = null
|
||||
tag_micro_r = null
|
||||
tag_micro_f = null
|
||||
morph_acc = 0.12
|
||||
morph_per_feat = null
|
||||
lemma_acc = 0.25
|
||||
dep_uas = 0.12
|
||||
dep_las = 0.12
|
||||
dep_las_per_type = null
|
||||
sents_p = null
|
||||
sents_r = null
|
||||
sents_f = 0.0
|
||||
|
||||
[initialize.tokenizer]
|
||||
|
||||
[initialize.components]
|
||||
|
||||
[components.tok2vec]
|
||||
factory = "tok2vec"
|
||||
|
||||
[components.tagger]
|
||||
factory = "tagger"
|
||||
label_smoothing = 0.05
|
||||
overwrite = false
|
||||
neg_prefix = "!"
|
||||
|
||||
[components.morphologizer]
|
||||
factory = "morphologizer"
|
||||
label_smoothing = 0.05
|
||||
overwrite = true
|
||||
extend = false
|
||||
|
||||
[components.trainable_lemmatizer]
|
||||
factory = "trainable_lemmatizer"
|
||||
backoff = "orth"
|
||||
min_tree_freq = 3
|
||||
overwrite = false
|
||||
top_k = 1
|
||||
|
||||
[components.parser]
|
||||
factory = "parser"
|
||||
moves = null
|
||||
update_with_oracle_cut_size = 100
|
||||
learn_tokens = false
|
||||
min_action_freq = 30
|
||||
|
||||
[training.batcher.size]
|
||||
@schedules = "compounding.v1"
|
||||
start = 100
|
||||
stop = 1000
|
||||
compound = 1.001
|
||||
t = 0.0
|
||||
|
||||
[components.tok2vec.model]
|
||||
@architectures = "spacy.Tok2Vec.v2"
|
||||
|
||||
[components.tagger.model]
|
||||
@architectures = "spacy.Tagger.v2"
|
||||
nO = null
|
||||
normalize = false
|
||||
|
||||
[components.tagger.scorer]
|
||||
@scorers = "spacy.tagger_scorer.v1"
|
||||
|
||||
[components.morphologizer.model]
|
||||
@architectures = "spacy.Tagger.v2"
|
||||
nO = null
|
||||
normalize = false
|
||||
|
||||
[components.morphologizer.scorer]
|
||||
@scorers = "spacy.morphologizer_scorer.v1"
|
||||
|
||||
[components.trainable_lemmatizer.model]
|
||||
@architectures = "spacy.Tagger.v2"
|
||||
nO = null
|
||||
normalize = false
|
||||
|
||||
[components.trainable_lemmatizer.scorer]
|
||||
@scorers = "spacy.lemmatizer_scorer.v1"
|
||||
|
||||
[components.parser.model]
|
||||
@architectures = "spacy.TransitionBasedParser.v2"
|
||||
state_type = "parser"
|
||||
extra_state_tokens = false
|
||||
hidden_width = 128
|
||||
maxout_pieces = 3
|
||||
use_upper = true
|
||||
nO = null
|
||||
|
||||
[components.parser.scorer]
|
||||
@scorers = "spacy.parser_scorer.v1"
|
||||
|
||||
[components.tok2vec.model.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
|
||||
rows = [5000, 1000, 2500, 2500]
|
||||
include_static_vectors = false
|
||||
|
||||
[components.tok2vec.model.encode]
|
||||
@architectures = "spacy.MaxoutWindowEncoder.v2"
|
||||
width = 96
|
||||
depth = 4
|
||||
window_size = 1
|
||||
maxout_pieces = 3
|
||||
|
||||
[components.tagger.model.tok2vec]
|
||||
@architectures = "spacy.Tok2VecListener.v1"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
upstream = "*"
|
||||
|
||||
[components.morphologizer.model.tok2vec]
|
||||
@architectures = "spacy.Tok2VecListener.v1"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
upstream = "*"
|
||||
|
||||
[components.trainable_lemmatizer.model.tok2vec]
|
||||
@architectures = "spacy.Tok2VecListener.v1"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
upstream = "*"
|
||||
|
||||
[components.parser.model.tok2vec]
|
||||
@architectures = "spacy.Tok2VecListener.v1"
|
||||
width = ${components.tok2vec.model.encode.width}
|
||||
upstream = "*"
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
# fa NER component, CPU size (sm).
|
||||
#
|
||||
# Trained on ParsTwiNER (MIT) — a different corpus from the UD treebank that trains
|
||||
# the rest of fa_core_news_sm, so this is a standalone run whose `ner` component is
|
||||
# later sourced into the core pipeline by scripts/assemble_core.py.
|
||||
#
|
||||
# Deliberate deviation from `spacy init config --pipeline ner`: the tok2vec is
|
||||
# EMBEDDED inside components.ner.model instead of being a separate `tok2vec`
|
||||
# component with a Tok2VecListener. A listener can only resolve inside the pipeline
|
||||
# it was trained in; embedding makes the component self-contained and therefore
|
||||
# sourceable. This is the same design as en_core_web_sm, whose `ner` "has its own
|
||||
# independent internal tok2vec" (https://spacy.io/models#design).
|
||||
|
||||
[paths]
|
||||
train = null
|
||||
dev = null
|
||||
vectors = null
|
||||
init_tok2vec = null
|
||||
|
||||
[system]
|
||||
gpu_allocator = null
|
||||
seed = 0
|
||||
|
||||
[nlp]
|
||||
lang = "fa"
|
||||
pipeline = ["ner"]
|
||||
batch_size = 1000
|
||||
disabled = []
|
||||
before_creation = null
|
||||
after_creation = null
|
||||
after_pipeline_creation = null
|
||||
|
||||
[nlp.tokenizer]
|
||||
@tokenizers = "spacy.Tokenizer.v1"
|
||||
|
||||
[nlp.vectors]
|
||||
@vectors = "spacy.Vectors.v1"
|
||||
|
||||
[components]
|
||||
|
||||
[components.ner]
|
||||
factory = "ner"
|
||||
moves = null
|
||||
update_with_oracle_cut_size = 100
|
||||
incorrect_spans_key = null
|
||||
|
||||
[components.ner.model]
|
||||
@architectures = "spacy.TransitionBasedParser.v2"
|
||||
state_type = "ner"
|
||||
extra_state_tokens = false
|
||||
hidden_width = 64
|
||||
maxout_pieces = 2
|
||||
use_upper = true
|
||||
nO = null
|
||||
|
||||
[components.ner.model.tok2vec]
|
||||
@architectures = "spacy.Tok2Vec.v2"
|
||||
|
||||
[components.ner.model.tok2vec.embed]
|
||||
@architectures = "spacy.MultiHashEmbed.v2"
|
||||
width = ${components.ner.model.tok2vec.encode.width}
|
||||
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
|
||||
rows = [5000, 1000, 2500, 2500]
|
||||
include_static_vectors = false
|
||||
|
||||
[components.ner.model.tok2vec.encode]
|
||||
@architectures = "spacy.MaxoutWindowEncoder.v2"
|
||||
width = 96
|
||||
depth = 4
|
||||
window_size = 1
|
||||
maxout_pieces = 3
|
||||
|
||||
[components.ner.scorer]
|
||||
@scorers = "spacy.ner_scorer.v1"
|
||||
|
||||
[corpora]
|
||||
|
||||
[corpora.train]
|
||||
@readers = "spacy.Corpus.v1"
|
||||
path = ${paths.train}
|
||||
max_length = 0
|
||||
gold_preproc = false
|
||||
limit = 0
|
||||
augmenter = null
|
||||
|
||||
[corpora.dev]
|
||||
@readers = "spacy.Corpus.v1"
|
||||
path = ${paths.dev}
|
||||
max_length = 0
|
||||
gold_preproc = false
|
||||
limit = 0
|
||||
augmenter = null
|
||||
|
||||
[training]
|
||||
dev_corpus = "corpora.dev"
|
||||
train_corpus = "corpora.train"
|
||||
seed = ${system.seed}
|
||||
gpu_allocator = ${system.gpu_allocator}
|
||||
dropout = 0.1
|
||||
accumulate_gradient = 1
|
||||
patience = 1600
|
||||
max_epochs = 0
|
||||
max_steps = 20000
|
||||
eval_frequency = 400
|
||||
frozen_components = []
|
||||
annotating_components = []
|
||||
before_to_disk = null
|
||||
before_update = null
|
||||
|
||||
[training.optimizer]
|
||||
@optimizers = "Adam.v1"
|
||||
beta1 = 0.9
|
||||
beta2 = 0.999
|
||||
L2_is_weight_decay = true
|
||||
L2 = 0.01
|
||||
grad_clip = 1.0
|
||||
use_averages = false
|
||||
eps = 1e-08
|
||||
learn_rate = 0.001
|
||||
|
||||
[training.batcher]
|
||||
@batchers = "spacy.batch_by_words.v1"
|
||||
discard_oversize = false
|
||||
tolerance = 0.2
|
||||
get_length = null
|
||||
|
||||
[training.batcher.size]
|
||||
@schedules = "compounding.v1"
|
||||
start = 100
|
||||
stop = 1000
|
||||
compound = 1.001
|
||||
t = 0.0
|
||||
|
||||
[training.logger]
|
||||
@loggers = "spacy.ConsoleLogger.v1"
|
||||
progress_bar = false
|
||||
|
||||
[training.score_weights]
|
||||
ents_f = 1.0
|
||||
ents_p = 0.0
|
||||
ents_r = 0.0
|
||||
ents_per_type = null
|
||||
|
||||
[initialize]
|
||||
vectors = ${paths.vectors}
|
||||
init_tok2vec = ${paths.init_tok2vec}
|
||||
vocab_data = null
|
||||
lookups = null
|
||||
before_init = null
|
||||
after_init = null
|
||||
|
||||
[initialize.tokenizer]
|
||||
|
||||
[initialize.components]
|
||||
|
||||
[pretraining]
|
||||
|
|
@ -0,0 +1,239 @@
|
|||
# Contribution guidelines: getting a Persian pipeline into the spaCy ecosystem
|
||||
|
||||
Everything below is sourced from spaCy's own primary docs/repos (URLs inline). Read this
|
||||
before writing code, because **where** a contribution goes decides how it must be shaped.
|
||||
|
||||
## 0. The one thing to internalise
|
||||
|
||||
spaCy splits Persian support into two *completely different* contribution surfaces:
|
||||
|
||||
| Surface | What it is | Where it lives | How you contribute |
|
||||
| --- | --- | --- | --- |
|
||||
| **Language data** (`fa`) | Hand-written rules: tokenizer exceptions, stop words, `LIKE_NUM`, punctuation, noun-chunk iterator | `spacy/lang/fa/*.py` inside the spaCy repo | Normal PR to `explosion/spaCy` |
|
||||
| **Trained pipeline** (`fa_core_news_sm`) | Statistical weights + `config.cfg` + `meta.json`, shipped as a pip wheel | `explosion/spacy-models` releases | **You cannot.** Publish it yourself (PyPI / HF Hub) and get it listed in spaCy Universe |
|
||||
|
||||
`spacy/lang/fa` **already exists upstream**. What does not exist is any trained `fa` pipeline.
|
||||
So this project is a *publishing* project, not an upstream-PR project — with optional
|
||||
upstream PRs for the language-data gaps we find along the way.
|
||||
|
||||
## 1. Policy, verbatim
|
||||
|
||||
From <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md>:
|
||||
|
||||
- **No CLA.** There is no contributor licence agreement section; contribution is an ordinary
|
||||
GitHub PR governed by the Contributor Covenant Code of Conduct v1.4.
|
||||
- Inclusion philosophy — this is the sentence that decides everything:
|
||||
> "Our philosophy is to prefer a smaller core library. […] If you're looking to implement a
|
||||
> new spaCy feature, starting with a custom component package is usually the best strategy.
|
||||
> […] And if it works well, we can always integrate it into the core library later."
|
||||
- The only sanctioned route for non-trivial additions is the *Publishing spaCy extensions and
|
||||
plugins* section:
|
||||
> "An extension or plugin should add substantial functionality, be well-documented and
|
||||
> open-source. It should be available for users to download and install as a Python package
|
||||
> – for example via PyPi."
|
||||
> "Once your extension is published, you can open a PR to suggest it for the Universe page."
|
||||
- `CONTRIBUTING.md` contains **zero** mention of submitting trained models. Neither does
|
||||
<https://github.com/explosion/spacy-models> — its README only documents `compatibility.json`
|
||||
as "the source of spaCy's internal compatibility check, performed when you run the download
|
||||
command". There is no PR template, issue label, or documented process for a third party to
|
||||
land a pipeline in `spacy download`.
|
||||
**Conclusion: the `spacy download` index is an Explosion-only release channel.**
|
||||
(`[INFERENCE]` from the absence of any process across all three primary sources.)
|
||||
- `https://spacy.io/usage/adding-languages` no longer exists; it redirects to
|
||||
<https://spacy.io/usage/linguistic-features#language-data>. The `BaseDefaults` /
|
||||
`Language` contract is documented at <https://spacy.io/api/language>.
|
||||
|
||||
### Code rules that apply to a `spacy/lang/fa` PR
|
||||
|
||||
- `black` formatting, `flake8` clean, type hints where practical.
|
||||
- Tests go in `spacy/tests/lang/fa/`; regression tests use `@pytest.mark.issue(N)`.
|
||||
- Touching `.pyx` means the reviewer must rebuild: `python setup.py build_ext --inplace`.
|
||||
- Language data must be *pure data + rules*: no model downloads, no network, no heavy deps.
|
||||
|
||||
## 2. Three publishing routes for the trained pipeline
|
||||
|
||||
1. **Hugging Face Hub** — Explosion's own tool, the path of least resistance
|
||||
(<https://github.com/explosion/spacy-huggingface-hub>):
|
||||
```bash
|
||||
pip install spacy-huggingface-hub
|
||||
huggingface-cli login
|
||||
python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build wheel
|
||||
python -m spacy huggingface-hub push packages/fa_core_news_sm-3.8.0/dist/fa_core_news_sm-3.8.0-py3-none-any.whl --org <org>
|
||||
```
|
||||
Users then `pip install https://huggingface.co/<org>/fa_core_news_sm/resolve/main/fa_core_news_sm-any-py3-none-any.whl`.
|
||||
2. **PyPI / self-hosted wheel** — `spacy package … --build sdist,wheel` then `twine upload`,
|
||||
or attach the wheel to a GitHub Release. See <https://spacy.io/api/cli#package>.
|
||||
3. **spaCy Universe listing** — lists the package on spacy.io but hosts nothing. Per
|
||||
<https://github.com/explosion/spaCy/blob/master/website/UNIVERSE.md>: fork `explosion/spaCy`,
|
||||
append an entry to `website/meta/universe.json`, open a PR. Required fields:
|
||||
`id`, `title`, `slogan`, `description`, `github`, `pip`, `code_example`, `code_language`,
|
||||
`url`, `thumb`, `image`, `author`, `author_links`, `category`, `tags`.
|
||||
Checklist: must be **open-source with a user-friendly license** and "at least somewhat documented".
|
||||
|
||||
**Route we take:** HF Hub for the artifact + a Universe PR once metrics are respectable.
|
||||
|
||||
## 3. Naming and versioning conventions (non-negotiable if you want to look official)
|
||||
|
||||
From <https://spacy.io/models#conventions> and the `spacy-models` README:
|
||||
|
||||
```
|
||||
[lang]_[type]_[genre]_[size] e.g. fa_core_news_sm
|
||||
```
|
||||
|
||||
| Slot | Allowed values | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `type` | `core` | tagger + parser + lemmatizer + NER |
|
||||
| | `dep` | tagger + parser + lemmatizer, **no NER** |
|
||||
| | `ent` | NER only |
|
||||
| | `sent` | sentence segmentation only |
|
||||
| `genre` | `news`, `web`, `wiki` | text domain of the training corpus |
|
||||
| `size` | `sm` | no static vectors |
|
||||
| | `md` | ~20k unique vectors / ~500k keys (or 50k floret rows) |
|
||||
| | `lg` | ~500k vectors (or 200k floret rows) |
|
||||
| | `trf` | transformer, no static vectors |
|
||||
|
||||
**Version numbers are `a.b.c` = spaCy-major . spaCy-minor . model-revision.** Trained against
|
||||
spaCy 3.8 → the package version starts at `3.8.0`. Retraining on new data bumps `c`.
|
||||
Do **not** invent `0.1.0`-style versions; that is exactly how hazm's HF pipelines ended up with
|
||||
`version: 0.0.0` and `spacy_version: >=3.6.0,<3.7.0`, which is unusable metadata.
|
||||
|
||||
### `meta.json` fields that matter (<https://spacy.io/api/data-formats#meta>)
|
||||
|
||||
`lang`, `name`, `version`, `spacy_version` (range), `description`, `author`, `email`, `url`,
|
||||
`license`, **`sources`** (list of `{name, url, author, license}` — this is where treebank
|
||||
provenance and its CC BY-SA obligation must be recorded), `requirements` (extra pip deps
|
||||
injected into the generated `setup.cfg`), `vectors`, `pipeline`, `labels`, `performance`
|
||||
(auto-filled by `spacy train`), `speed`, `spacy_git_version`.
|
||||
|
||||
Note: in v3, `meta.json` "isn't used to construct the language class and pipeline anymore" —
|
||||
`config.cfg` is the single source of truth for loading. `meta.json` is metadata + packaging.
|
||||
|
||||
## 4. Canonical training workflow (what Explosion actually runs)
|
||||
|
||||
Template: <https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud>
|
||||
(`project.yml` + `configs/default.cfg`). Directory layout `assets / corpus / configs /
|
||||
training / metrics / packages` is the convention — this repo follows it.
|
||||
|
||||
```bash
|
||||
# 1. assets: clone the UD treebank
|
||||
git clone https://github.com/UniversalDependencies/UD_Persian-PerDT assets/UD_Persian-PerDT
|
||||
|
||||
# 2. corpus: CoNLL-U -> binary DocBin
|
||||
python -m spacy convert assets/.../fa_perdt-ud-train.conllu corpus/ \
|
||||
--converter conllu --n-sents 10 --merge-subtokens
|
||||
|
||||
# 3. config
|
||||
python -m spacy init config configs/default.cfg --lang fa \
|
||||
--pipeline tagger,morphologizer,trainable_lemmatizer,parser --optimize efficiency
|
||||
python -m spacy init fill-config partial.cfg configs/default.cfg # completes defaults
|
||||
|
||||
# 4. (md/lg only) vectors
|
||||
python -m spacy init vectors fa cc.fa.300.vec vectors/ --mode floret --prune 200000
|
||||
|
||||
# 5. train
|
||||
python -m spacy train configs/default.cfg --output training/ --gpu-id -1 \
|
||||
--paths.train corpus/train.spacy --paths.dev corpus/dev.spacy
|
||||
|
||||
# 6. evaluate on held-out test
|
||||
python -m spacy benchmark accuracy training/model-best corpus/test.spacy --output metrics/test.json
|
||||
# ('spacy evaluate' is now just an alias for 'benchmark accuracy')
|
||||
|
||||
# 7. package
|
||||
python -m spacy package training/model-best packages --name core_news_sm --version 3.8.0 --build sdist,wheel
|
||||
```
|
||||
|
||||
`spacy assemble` builds a pipeline from a config **without training** — useful for a
|
||||
rule-only artifact (tokenizer + lookup lemmatizer + stop words).
|
||||
|
||||
### Flags worth knowing on `spacy convert`
|
||||
|
||||
- `--converter conllu` — explicit is better than `auto`.
|
||||
- `--n-sents N` — how many sentences per `Doc`; 10 is the Explosion default and gives the
|
||||
parser/senter cross-sentence context.
|
||||
- `--merge-subtokens` — spaCy has **no multi-word-token layer**. Persian UD treebanks use MWT
|
||||
ranges for clitics (`پدرم` → `پدر` + `م`), so this flag decides whether gold tokens are
|
||||
clitic-split or fused. See `docs/MODELS.md` for the tokenisation decision.
|
||||
- `--morphology` — appends morph features to the tag (only if you *aren't* using a morphologizer).
|
||||
|
||||
### How the four size tiers differ mechanically
|
||||
|
||||
| Tier | Embedding source | Config difference |
|
||||
| --- | --- | --- |
|
||||
| `sm` | hash embeddings only | `spacy.MultiHashEmbed` with `include_static_vectors = false`, `spacy.MaxoutWindowEncoder` width 96, everything else a `spacy.Tok2VecListener` on the shared `tok2vec` |
|
||||
| `md` / `lg` | + static vectors | same CNN, `include_static_vectors = true`, `[initialize] vectors = <dir built by init vectors>`; `lg` just has a bigger table |
|
||||
| `trf` | contextual | replace `tok2vec` with a `transformer` component; every other component listens via `TransformerListener` instead of `Tok2VecListener` |
|
||||
|
||||
`spacy pretrain` (Tok2Vec LM-style pretraining on raw text → `[initialize] init_tok2vec`) is
|
||||
orthogonal to the tier and optional.
|
||||
|
||||
## 5. Upstream `spacy/lang/fa` — current state and gaps
|
||||
|
||||
Contents of <https://github.com/explosion/spaCy/tree/master/spacy/lang/fa>:
|
||||
|
||||
| File | Size | What it gives us |
|
||||
| --- | --- | --- |
|
||||
| `__init__.py` | 1.3 KB | `PersianDefaults`: tokenizer exceptions, `TOKENIZER_SUFFIXES`, `LEX_ATTRS`, `SYNTAX_ITERATORS`, `STOP_WORDS`, `writing_system = {"direction": "rtl", "has_case": False, "has_letters": True}`; registers a **`lemmatizer` factory defaulting to `mode="rule"`** |
|
||||
| `tokenizer_exceptions.py` | 64.9 KB | Large generated compound-verb / enclitic exception table |
|
||||
| `generate_verbs_exc.py` | 14.8 KB | Dev script that generates the above |
|
||||
| `stop_words.py` | 3.8 KB | ~500 stop words, comment says **"Stop words from HAZM package"** |
|
||||
| `lex_attrs.py` | 1.4 KB | `LIKE_NUM` only (Persian numerals + `ام`/`ین` suffixes) |
|
||||
| `punctuation.py` | 508 B | `TOKENIZER_SUFFIXES` only — **no prefixes, no infixes** |
|
||||
| `syntax_iterators.py` | 1.6 KB | `noun_chunks()`, needs a trained parser to do anything |
|
||||
|
||||
Tests: `spacy/tests/lang/fa/` has only `test_noun_chunks.py`. No tokenizer, lemmatizer, or
|
||||
stop-word tests.
|
||||
|
||||
`spacy-lookups-data` (MIT) **already ships Persian rule-lemmatizer tables**:
|
||||
|
||||
| File | Size |
|
||||
| --- | --- |
|
||||
| `fa_lemma_exc.json` | 1.68 MB |
|
||||
| `fa_lemma_index.json` | 171 KB |
|
||||
| `fa_lemma_rules.json` | 882 B |
|
||||
| `fa_source.txt` | "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" |
|
||||
|
||||
Missing there: `fa_lemma_lookup.json` (no lookup-mode table), `fa_lexeme_norm.json`,
|
||||
`fa_license.txt` (Catalan has one; Persian's Seraji-derived provenance is undocumented).
|
||||
|
||||
**Gap list for a real `fa_core_news_*`:**
|
||||
|
||||
1. No trained artefacts at all — no weights, no `config.cfg`, no `meta.json`, no entry in
|
||||
`compatibility.json`. `spacy.load("fa_core_news_sm")` fails today.
|
||||
2. No word vectors for `fa` → `md`/`lg` need vectors built from scratch.
|
||||
3. No NER data or labels anywhere in `spacy/lang/fa` → `core` (which implies NER) needs an
|
||||
external annotated corpus.
|
||||
4. Rule lemmatizer needs `token.pos` from a tagger/morphologizer → chicken-and-egg until the
|
||||
tagger exists (or use a `trainable_lemmatizer`, which learns edit trees and needs no tables).
|
||||
5. No Persian-specific prefix/infix rules — ZWNJ (U+200C) is only handled implicitly through
|
||||
the verb-exception table.
|
||||
6. Zero tokenizer test coverage for a 65 KB exception table.
|
||||
|
||||
Items 5–7 are legitimate upstream PR material; items 1–4 are this project's job.
|
||||
|
||||
## 6. Prior art we must not duplicate badly
|
||||
|
||||
hazm publishes **spaCy-format** Persian pipelines on the HF Hub (MIT-ish, but `license` field
|
||||
left empty), each a single-task pipeline built on ParsBERT:
|
||||
|
||||
| HF repo | Pipeline | Reported | Trained on (`[paths]` in its `config.cfg`) |
|
||||
| --- | --- | --- | --- |
|
||||
| `roshan-research/hazm-parsbert-postagger` | `transformer, tagger` | `tag_acc` 0.9862 | `data_train_98_rs10.spacy` (hazm's own EZ-augmented tagset: `NOUN,EZ`, `ADJ,EZ`, …) |
|
||||
| `roshan-research/hazm-bert-dependency-parser` | `transformer, parser` | `dep_uas` 0.9246 / `dep_las` 0.8934 | `modified_fa_perdt-ud-train.spacy` — **UD_Persian-PerDT** |
|
||||
| `roshan-research/hazm-parsbert-chunker` | `transformer, tagger` (IOB chunk tags) | `tag_acc` 0.9618 | hazm chunk data |
|
||||
|
||||
Both use `spacy-transformers` `TransformerModel.v3` on
|
||||
`HooshvareLab/bert-base-parsbert-uncased`, `strided_spans` window 128 / stride 96,
|
||||
`spacy_version >= 3.6.0,<3.7.0`.
|
||||
|
||||
What they get wrong, and what we fix:
|
||||
|
||||
- Three separate pipelines instead of one `core` pipeline → users pay for three BERT forward
|
||||
passes and cannot share a `Doc`.
|
||||
- `version: 0.0.0`, empty `license`/`author`/`sources`, `name: "pipeline"` → violates every
|
||||
convention in §3 and makes redistribution legally murky.
|
||||
- Transformer-only, so no CPU-friendly tier at all.
|
||||
- No lemmatizer, no morphologizer, no NER, no vectors.
|
||||
- Pinned to spaCy 3.6; unusable on 3.8 without retraining.
|
||||
|
||||
The valuable, reusable finding: **hazm's own parser recipe is "UD_Persian-PerDT + spaCy
|
||||
`TransitionBasedParser`"**, which independently confirms the corpus choice in `docs/MODELS.md`.
|
||||
|
|
@ -0,0 +1,219 @@
|
|||
# The Persian pipelines: what to build, from what, and why
|
||||
|
||||
## 1. What "the 4 English pipelines" actually are
|
||||
|
||||
They are **one pipeline design at four embedding budgets**, not four different products.
|
||||
Every one of them has the same components; the only real axis of variation is where token
|
||||
representations come from.
|
||||
|
||||
| | `en_core_web_sm` | `en_core_web_md` | `en_core_web_lg` | `en_core_web_trf` |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Size on disk | 12 MB | 31 MB | 382 MB | 436 MB |
|
||||
| Embeddings | hash embeddings only | 685k keys / 20k vectors (300d) | 685k keys / 343k vectors (300d) | `roberta-base`, 768d contextual |
|
||||
| Components | tok2vec, tagger, parser, senter, attribute_ruler, lemmatizer, ner | same | same | **transformer**, tagger, parser, attribute_ruler, lemmatizer, ner |
|
||||
| `TAG_ACC` | 0.97 | 0.97 | 0.97 | 0.98 |
|
||||
| `DEP_UAS` / `LAS` | 0.92 / 0.90 | 0.92 / 0.90 | 0.92 / 0.90 | 0.95 / 0.94 |
|
||||
| `ENTS_F` | 0.84 | 0.85 | 0.86 | 0.90 |
|
||||
| Training data | OntoNotes 5 (+ ClearNLP dep conversion, WordNet 3.0) | + Explosion vectors (OSCAR 2109 + Wikipedia + OpenSubtitles + WMT News Crawl) | same | OntoNotes 5 + roberta-base |
|
||||
|
||||
Read the accuracy table honestly: **static vectors buy almost nothing for tagging and
|
||||
parsing** (identical to two decimals) and ~1–2 F on NER. The transformer buys ~4 LAS and
|
||||
~6 NER F, at 36× the size and a GPU requirement. That ordering dictates the roadmap below.
|
||||
|
||||
Sources: <https://spacy.io/models/en>.
|
||||
|
||||
## 2. Target: the four Persian pipelines
|
||||
|
||||
Naming follows `[lang]_[type]_[genre]_[size]` (<https://spacy.io/models#conventions>).
|
||||
`core` = tagger + parser + lemmatizer + NER. Genre is `news`, after the dominant genre of
|
||||
UD_Persian-PerDT (its README lists "news fiction nonfiction academic web blog", and spaCy
|
||||
labels comparable treebank-trained pipelines such as `de_core_news_sm` as `news`).
|
||||
|
||||
| Pipeline | Components | Embeddings | Trainable on this hardware? |
|
||||
| --- | --- | --- | --- |
|
||||
| **`fa_core_news_sm`** | tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner | hash embeddings | **yes — this is what we train now** |
|
||||
| `fa_core_news_md` | same | floret vectors, 50k rows | yes, but vectors must be trained first (CPU-days on fa Wikipedia + OSCAR) |
|
||||
| `fa_core_news_lg` | same | floret vectors, 200k rows | same as md, bigger table |
|
||||
| `fa_core_news_trf` | transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner | `HooshvareLab/roberta-fa-zwnj-base` (Apache-2.0) | **no** — 2 GB VRAM (GTX 940MX) cannot fine-tune a 125M-param encoder; needs rented GPU |
|
||||
|
||||
One deviation from the English design, deliberate: Persian gets a **`morphologizer`**
|
||||
(UPOS + morphological features) and a **`trainable_lemmatizer`** instead of English's
|
||||
`attribute_ruler` + rule `lemmatizer`. Reasons:
|
||||
|
||||
- The English pipelines use `attribute_ruler` because OntoNotes gives them PTB `tag`s and
|
||||
they *derive* UPOS from tags by rule. UD treebanks give UPOS and FEATS as gold data —
|
||||
training a morphologizer on them is strictly more information, and Persian morphology
|
||||
(Number, Person, Tense, Mood, Voice, Polarity, PronType) is worth predicting.
|
||||
- Persian rule-lemmatizer tables *do* exist in `spacy-lookups-data` (`fa_lemma_exc.json`
|
||||
1.68 MB, `fa_lemma_index.json`, `fa_lemma_rules.json`, derived from Seraji's treebank), but
|
||||
a rule lemmatizer's accuracy is unmeasurable against the corpus it was extracted from and it
|
||||
needs `token.pos` to work at all. `trainable_lemmatizer` learns edit trees from PerDT's gold
|
||||
lemmas and reports a real `lemma_acc`. PerDT yields **1,908 edit trees** with 100% lemma
|
||||
coverage — plenty.
|
||||
|
||||
`senter` is intentionally omitted from v1: it is a separately trained component that ships
|
||||
*disabled by default* in the English pipelines, and the parser already produces sentence
|
||||
boundaries. Adding it later requires only one extra training run.
|
||||
|
||||
## 3. Resource inventory (everything checked for license)
|
||||
|
||||
### 3.1 Treebanks — the tagger / morphologizer / lemmatizer / parser data
|
||||
|
||||
| Treebank | Sents | Tokens | License | LEMMA | FEATS | XPOS | PROPN? | Verdict |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| **UD_Persian-PerDT** (PerUDT v1.0) | 29,107 | ~509k | **CC BY-SA 4.0** | converted + corrections | converted + corrections | manual native (34 types) | **yes** | **CHOSEN** |
|
||||
| UD_Persian-Seraji | 5,997 | ~152k | CC BY-SA 4.0 | manual native | manual native | manual native (30 types) | **no** | secondary / cross-eval |
|
||||
| UD_Persian-PUD | 1,000 | — | CC BY-SA 4.0 | — | — | none | — | test-only, parallel corpus |
|
||||
| UD_Persian-IPerUDT | tiny | — | CC BY-SA 4.0 | — | — | none | — | grammar examples, unusable |
|
||||
|
||||
Measured locally with `scripts/inspect_treebanks.py` (train splits):
|
||||
|
||||
```
|
||||
file sents tokens MWT empty lemma% feats% UPOS XPOS DEP
|
||||
fa_perdt-ud-train.conllu 26196 452496 6508 0 100.0 57.7 16 34 34
|
||||
fa_seraji-ud-train.conllu 4798 121067 1117 0 100.0 65.0 15 30 39
|
||||
```
|
||||
|
||||
Why PerDT over Seraji:
|
||||
|
||||
1. **3.7× more training tokens** (452k vs 121k). At `sm` size, data is the binding constraint.
|
||||
2. **Seraji has no `PROPN`** — proper nouns are tagged `NOUN`. A pipeline that cannot mark
|
||||
proper nouns is crippled for exactly the downstream tasks people use spaCy for.
|
||||
3. hazm independently chose PerDT for its own spaCy dependency parser — its
|
||||
`config.cfg` names `modified_fa_perdt-ud-train.spacy` as the training set. Converging on
|
||||
the same corpus makes our numbers comparable to theirs.
|
||||
|
||||
Seraji's advantages (richer manual FEATS, fully manual lemmas) are real; it is the natural
|
||||
cross-evaluation set and a candidate for a future concatenated-corpus run. The two use
|
||||
different XPOS inventories, so naive concatenation would corrupt the `tag` label space.
|
||||
|
||||
### 3.2 NER — the one place with a licensing minefield
|
||||
|
||||
| Dataset | Labels | Size | License | Usable? |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **ParsTwiNER** | PER, ORG, LOC, NAT, POG, EVENT | 7,667 tweets / 233k tokens | **MIT** (verified via GitHub API on `overfit-ir/parstwiner`) | **CHOSEN** |
|
||||
| ARMAN (PersianNER) | 6 classes | 250k tokens | academic research only | no |
|
||||
| PEYMA | 7 classes | 302k tokens | "free for research purposes", no OSS licence | no |
|
||||
| NSURL-2019 Task 7 | PEYMA tagset | ~1M tokens | no explicit licence | no |
|
||||
| HooshvareLab merged ParsNER | 10 classes | ARMAN+PEYMA+WikiANN | inherits ARMAN/PEYMA restrictions; HF repo gated (401) | no |
|
||||
|
||||
This is not pedantry. **spaCy's own maintainers state that Persian models trained back in
|
||||
2018 were never published precisely because of corpus licensing** (spaCy discussion #8233,
|
||||
following PR #2797 which added only `spacy.blank("fa")` tokenizer support). Repeating that
|
||||
mistake would waste the whole exercise: a pipeline you cannot legally redistribute is not a
|
||||
pipeline, it is a local file.
|
||||
|
||||
Consequence to state plainly in the model card: **the NER component is trained on Twitter
|
||||
text while the rest of the pipeline is trained on edited prose.** Expect NER to degrade on
|
||||
formal news text relative to what an ARMAN/PEYMA-trained model would score. That is the
|
||||
price of a redistributable artifact.
|
||||
|
||||
### 3.3 Vectors (for md / lg)
|
||||
|
||||
| Option | License | Note |
|
||||
| --- | --- | --- |
|
||||
| **floret vectors** trained via `spacy-vectors-builder` (MIT tooling) on fa Wikipedia + OSCAR | corpus-dependent | **recommended.** Subword + Bloom-hash embeddings: bounded table size, zero OOV |
|
||||
| fastText `cc.fa.300` | CC BY-SA 3.0 | quick fallback; classic word table, large and OOV-prone |
|
||||
|
||||
Floret is the right call for Persian specifically. Persian surface forms explode through
|
||||
suffixation *and* through inconsistent ZWNJ (U+200C) usage — the same word appears as
|
||||
`میرود` / `میرود` / `می رود` in real text. A classic word-vector table has a miss for every
|
||||
variant; floret's subword hashing covers all of them. spaCy already ships floret vectors for
|
||||
Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian for the same reason.
|
||||
|
||||
### 3.4 Transformer encoders (for trf)
|
||||
|
||||
| Model | Arch | License | Note |
|
||||
| --- | --- | --- | --- |
|
||||
| **`HooshvareLab/roberta-fa-zwnj-base`** | RoBERTa-base | **Apache-2.0** | recommended: licensed, ZWNJ-aware, smallest of the credible options |
|
||||
| `FacebookAI/xlm-roberta-base` | XLM-R base, 278M | MIT | licensed but larger |
|
||||
| `PartAI/TookaBERT-Base` | BERT-base | Apache-2.0 | licensed |
|
||||
| `m3hrdadfi/albert-fa-base-v2` | ALBERT-base-v2 | Apache-2.0 | licensed, smallest |
|
||||
| `HooshvareLab/bert-base-parsbert-uncased` | BERT-base, ~162M | **no licence on the card** | what hazm used; redistribution risk |
|
||||
| `sbunlp/fabert` | BERT-base, 124M | **no licence on the card** | redistribution risk |
|
||||
|
||||
All of these are BERT/RoBERTa/XLM-R/ALBERT, so all are supported by both
|
||||
`spacy-transformers` and `spacy-curated-transformers` (the latter supports exactly
|
||||
ALBERT / BERT / CamemBERT / RoBERTa / XLM-RoBERTa).
|
||||
|
||||
Note the trap: hazm's own pipelines use ParsBERT, which has **no license statement**. Copying
|
||||
that choice would reintroduce the redistribution problem that sank the 2018 attempt.
|
||||
|
||||
## 4. What already exists (and why it is not enough)
|
||||
|
||||
- **No trained spaCy Persian pipeline exists.** Zero `persian` / `farsi` / `fa_` hits in
|
||||
spaCy's `website/meta/universe.json`. `spacy.load("fa_core_news_sm")` has never worked.
|
||||
- **hazm** ships three *single-task* spaCy pipelines on the HF Hub:
|
||||
`hazm-parsbert-postagger` (`tag_acc` 0.9862, hazm's own EZ-augmented tagset),
|
||||
`hazm-bert-dependency-parser` (`dep_uas` 0.9246 / `dep_las` 0.8934, trained on PerDT),
|
||||
`hazm-parsbert-chunker` (`tag_acc` 0.9618). Each is `transformer + one component`,
|
||||
`version: 0.0.0`, empty `license`/`author`/`sources`, pinned to spaCy 3.6. Using all three
|
||||
means three separate BERT forward passes over the same text and no shared `Doc`.
|
||||
- **hazm's training code is not reusable.** Its trainable models are pycrfsuite CRFs
|
||||
(`hazm/sequence_tagger.py`); there is no `config.cfg` or `spacy train` anywhere in the repo.
|
||||
Its `Spacy*` classes only *download* the HF pipelines above.
|
||||
- **hazm's tokenizer is actively incompatible** with UD gold tokenization: `Normalizer`'s
|
||||
`AFFIX_SPACING_PATTERNS` fuse ZWNJ affixes and `WordTokenizer.join_verb_parts()` glues
|
||||
multi-word verb chains into single underscore-joined tokens. Training against PerDT with
|
||||
hazm's tokenizer would misalign tokens systematically.
|
||||
- **DadmaTools** (Apache-2.0 code) emits spaCy-compatible `Doc` objects but is not a loadable
|
||||
spaCy pipeline package, and its NER wraps ARMAN/PEYMA — the restricted data again.
|
||||
|
||||
So what hazm genuinely contributes to this project is **one validated design decision**
|
||||
(PerDT is the corpus) and **the stop-word list already vendored into `spacy/lang/fa`**.
|
||||
Everything else is built with spaCy-native tooling.
|
||||
|
||||
## 5. Decision: train `fa_core_news_sm` first
|
||||
|
||||
Not md/lg: those need floret vectors trained from scratch on Wikipedia+OSCAR (CPU-days) and
|
||||
buy ~0.00 tag/dep accuracy in the English reference numbers.
|
||||
Not trf: 2 GB of VRAM cannot fine-tune a 125M-param encoder, and renting a GPU should wait
|
||||
until the CPU pipeline proves the data plumbing is right.
|
||||
`sm` is also the tier every other tier is validated against: md/lg/trf reuse the exact same
|
||||
corpus conversion, config skeleton and evaluation harness.
|
||||
|
||||
### Tokenization decision, settled with a measurement
|
||||
|
||||
spaCy has no multi-word-token layer, so CoNLL-U MWT ranges (Persian pronominal clitics and
|
||||
copulas: `پدرم` = `پدر` + `م`) must either be merged into one token or kept split. Measured on
|
||||
the PerDT dev set with `scripts/tokenization_report.py`, comparing gold boundaries against
|
||||
`spacy.blank("fa")`'s tokenizer:
|
||||
|
||||
| `spacy convert` mode | token P | token R | token F | XPOS types | merge artefacts |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `--merge-subtokens` | 0.9860 | 0.9914 | **0.9887** | 68 | 34 composite tags on 1.49% of tokens; 1.49% lemmas contain a space |
|
||||
| plain (clitics split) | 0.9875 | 0.9772 | 0.9823 | 35 | none |
|
||||
|
||||
**Chosen: `--merge-subtokens`** (also what Explosion's `tagger_parser_ud` template does).
|
||||
Rationale: without it, 1.5% of gold tokens are boundaries the shipped tokenizer can never
|
||||
produce, so those tokens are permanently unlearnable and unpredictable at runtime. With it,
|
||||
every gold token is reachable; the cost is confined to rare composite XPOS tags such as
|
||||
`N_IANM_PR_JOPER` (noun + enclitic pronoun) — which are, at least, informative — and 1.5% of
|
||||
lemmas that come out as two words.
|
||||
|
||||
The better long-term fix is Persian clitic-splitting suffix rules in `spacy/lang/fa`, which
|
||||
would be an upstream PR, not a model change. Recorded in `docs/CONTRIBUTING-GUIDE.md` §5.
|
||||
|
||||
### Final composition of `fa_core_news_sm`
|
||||
|
||||
| Component | Trained on | Metric | Test score |
|
||||
| --- | --- | --- | --- |
|
||||
| `tok2vec` | shared, PerDT | — | — |
|
||||
| `tagger` (XPOS) | PerDT, 90 labels | `tag_acc` | 95.96 |
|
||||
| `morphologizer` (UPOS + FEATS) | PerDT, 298 labels | `pos_acc` / `morph_acc` | 96.24 / 96.29 |
|
||||
| `trainable_lemmatizer` | PerDT, 1,908 edit trees | `lemma_acc` | 97.91 |
|
||||
| `parser` | PerDT, 34 deprels | `dep_uas` / `dep_las` | 89.69 / 85.15 |
|
||||
| `ner` (own internal tok2vec) | ParsTwiNER, 6 labels | `ents_p/r/f` | 74.77 / 61.06 / 67.22 |
|
||||
|
||||
Training cost on the target hardware (4-core i5-7200U, no GPU): **1h27m** for the UD
|
||||
components (early-stopped at step 10,800; best checkpoint step 9,200) and ~25 min for NER
|
||||
(best at step 4,000). Both runs are single-threaded, so they were run concurrently.
|
||||
Inference: ~9,250 words/s. Wheel: 13 MB.
|
||||
|
||||
The `--merge-subtokens` artefacts predicted above are visible in the shipped model exactly as
|
||||
expected — `کتابهایش` ("his/her books") comes out as one token tagged `N_IANM_PR_JOPER` with
|
||||
lemma `کتاب او`. Worth knowing before you consume `token.lemma_` downstream.
|
||||
|
||||
Sources recorded in `meta.json`, both with their licenses, per
|
||||
<https://spacy.io/api/data-formats#meta>. CC BY-SA 4.0 on PerDT means the packaged pipeline
|
||||
must carry attribution and share-alike notice — handled in `scripts/assemble_core.py`.
|
||||
|
|
@ -0,0 +1,223 @@
|
|||
# Upstream PR candidate: `spacy/lang/fa/syntax_iterators.py` uses non-UD dependency labels
|
||||
|
||||
## The bug
|
||||
|
||||
`spacy/lang/fa/syntax_iterators.py` matches these dependency labels as noun-phrase heads:
|
||||
|
||||
```python
|
||||
labels = ["nsubj", "dobj", "nsubjpass", "pcomp", "pobj", "dative", "appos", "attr", "ROOT"]
|
||||
```
|
||||
|
||||
`dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They are not
|
||||
Universal Dependencies relations, and **every Persian treebank is UD** (`UD_Persian-PerDT`,
|
||||
`UD_Persian-Seraji`, `UD_Persian-PUD`, `UD_Persian-IPerUDT`). Any trained `fa` pipeline must
|
||||
therefore emit UD labels, so five of the nine labels are dead code and `doc.noun_chunks`
|
||||
silently returns bare head nouns.
|
||||
|
||||
Compare `spacy/lang/fr/syntax_iterators.py` and `spacy/lang/es/syntax_iterators.py`, which
|
||||
use UD labels (`nsubj`, `nsubj:pass`, `obj`, `obl`, `nmod`, `appos`, `ROOT`) because their
|
||||
treebanks are UD too. `spacy/lang/de` legitimately uses TIGER labels (`sb`, `oa`, `nk`, …)
|
||||
because the German pipelines are trained on TIGER. Persian has no such excuse.
|
||||
|
||||
Second, smaller problem: the iterator yields `word.left_edge.i` → `word.i + 1`, i.e. it only
|
||||
ever expands **left**. Persian noun phrases expand **right** through ezafe:
|
||||
`رئیس انجمن جراحان قلب ایران` ("the head of the Iranian society of heart surgeons") is one NP
|
||||
whose head is the leftmost token. Left-only expansion truncates it to `رئیس`.
|
||||
|
||||
## Evidence
|
||||
|
||||
Measured with `scripts/check_noun_chunks.py` on the `UD_Persian-PerDT` dev split (146 docs)
|
||||
parsed by this project's trained pipeline:
|
||||
|
||||
```
|
||||
shipped noun_chunks: 1503 chunks, 1.31 tokens/chunk
|
||||
proposed noun_chunks: 5300 chunks, 2.77 tokens/chunk
|
||||
|
||||
deprels on NOUN/PROPN/PRON tokens in dev (top 15):
|
||||
nmod 2894 -
|
||||
obl 1401 -
|
||||
nsubj 1326 shipped
|
||||
compound:lvc 1294 -
|
||||
obl:arg 1095 -
|
||||
obj 973 -
|
||||
conj 583 shipped
|
||||
flat:name 389 -
|
||||
ROOT 101 shipped
|
||||
xcomp 89 -
|
||||
appos 51 shipped
|
||||
nsubj:pass 38 -
|
||||
```
|
||||
|
||||
1.31 tokens per chunk is the tell: the shipped iterator is returning single head nouns.
|
||||
`nmod` (2894), `obl` (1401), `obl:arg` (1095) and `obj` (973) — the four most common
|
||||
noun-bearing relations after `nsubj` — are all unreachable.
|
||||
|
||||
Live text:
|
||||
|
||||
```
|
||||
رئیس انجمن جراحان قلب ایران تأکید کرد که امکانات پیشرفته در ایران وجود دارد.
|
||||
shipped : ['رئیس', 'امکانات']
|
||||
proposed: ['رئیس انجمن جراحان قلب ایران', 'امکانات پیشرفته', 'ایران']
|
||||
|
||||
دانشگاه تهران بزرگترین دانشگاه ایران است.
|
||||
shipped : ['دانشگاه']
|
||||
proposed: ['دانشگاه تهران', 'ایران']
|
||||
```
|
||||
|
||||
## Proposed patch
|
||||
|
||||
Swap in UD labels, expand right through modifier chains, and drop a leading `ADP`/`CCONJ`
|
||||
exactly as `fr`/`es` already do:
|
||||
|
||||
```diff
|
||||
--- a/spacy/lang/fa/syntax_iterators.py
|
||||
+++ b/spacy/lang/fa/syntax_iterators.py
|
||||
@@
|
||||
def noun_chunks(doclike: Union[Doc, Span]) -> Iterator[Tuple[int, int, int]]:
|
||||
"""
|
||||
Detect base noun phrases from a dependency parse. Works on both Doc and Span.
|
||||
"""
|
||||
- labels = [
|
||||
- "nsubj",
|
||||
- "dobj",
|
||||
- "nsubjpass",
|
||||
- "pcomp",
|
||||
- "pobj",
|
||||
- "dative",
|
||||
- "appos",
|
||||
- "attr",
|
||||
- "ROOT",
|
||||
- ]
|
||||
+ # Persian pipelines are trained on Universal Dependencies treebanks
|
||||
+ # (UD_Persian-PerDT, UD_Persian-Seraji), so these are UD relations.
|
||||
+ labels = [
|
||||
+ "nsubj",
|
||||
+ "nsubj:pass",
|
||||
+ "obj",
|
||||
+ "iobj",
|
||||
+ "obl",
|
||||
+ "obl:arg",
|
||||
+ "nmod",
|
||||
+ "appos",
|
||||
+ "vocative",
|
||||
+ "ROOT",
|
||||
+ ]
|
||||
+ # Persian noun phrases grow to the right through ezafe constructions.
|
||||
+ post_modifiers = [
|
||||
+ "nmod",
|
||||
+ "nmod:poss",
|
||||
+ "amod",
|
||||
+ "det",
|
||||
+ "nummod",
|
||||
+ "flat",
|
||||
+ "flat:name",
|
||||
+ "flat:num",
|
||||
+ "fixed",
|
||||
+ "compound",
|
||||
+ ]
|
||||
doc = doclike.doc # Ensure works on both Doc and Span.
|
||||
|
||||
if not doc.has_annotation("DEP"):
|
||||
raise ValueError(Errors.E029)
|
||||
|
||||
np_deps = [doc.vocab.strings.add(label) for label in labels]
|
||||
+ np_modifs = {doc.vocab.strings.add(label) for label in post_modifiers}
|
||||
conj = doc.vocab.strings.add("conj")
|
||||
np_label = doc.vocab.strings.add("NP")
|
||||
+ adp_pos = doc.vocab.strings.add("ADP")
|
||||
+ cconj_pos = doc.vocab.strings.add("CCONJ")
|
||||
prev_end = -1
|
||||
for i, word in enumerate(doclike):
|
||||
if word.pos not in (NOUN, PROPN, PRON):
|
||||
continue
|
||||
# Prevent nested chunks from being produced
|
||||
if word.left_edge.i <= prev_end:
|
||||
continue
|
||||
- if word.dep in np_deps:
|
||||
- prev_end = word.i
|
||||
- yield word.left_edge.i, word.i + 1, np_label
|
||||
- elif word.dep == conj:
|
||||
+ if word.dep == conj:
|
||||
head = word.head
|
||||
while head.dep == conj and head.head.i < head.i:
|
||||
head = head.head
|
||||
- # If the head is an NP, and we're coordinated to it, we're an NP
|
||||
- if head.dep in np_deps:
|
||||
- prev_end = word.i
|
||||
- yield word.left_edge.i, word.i + 1, np_label
|
||||
+ # If the head is an NP, and we're coordinated to it, we're an NP
|
||||
+ if head.dep not in np_deps:
|
||||
+ continue
|
||||
+ elif word.dep not in np_deps:
|
||||
+ continue
|
||||
+ # Expand right through the ezafe / modifier chain.
|
||||
+ right = word
|
||||
+ for child in word.rights:
|
||||
+ if child.dep in np_modifs:
|
||||
+ right = child.right_edge
|
||||
+ else:
|
||||
+ break
|
||||
+ start, end = word.left_edge.i, max(word.i, right.i) + 1
|
||||
+ # A leading preposition or coordinator is not part of the NP.
|
||||
+ while start < word.i and doc[start].pos in (adp_pos, cconj_pos):
|
||||
+ start += 1
|
||||
+ if end <= prev_end:
|
||||
+ continue
|
||||
+ prev_end = end - 1
|
||||
+ yield start, end, np_label
|
||||
```
|
||||
|
||||
The working implementation lives in `scripts/check_noun_chunks.py::proposed_noun_chunks`.
|
||||
|
||||
## Test to add
|
||||
|
||||
`spacy/tests/lang/fa/test_noun_chunks.py` currently has one test (a hand-built `Doc`, 296 B).
|
||||
Add UD-label coverage:
|
||||
|
||||
```python
|
||||
def test_fa_noun_chunks_ezafe(fa_vocab):
|
||||
# رئیس انجمن جراحان — "head of the surgeons' society"
|
||||
words = ["رئیس", "انجمن", "جراحان", "آمد"]
|
||||
heads = [3, 0, 1, 3]
|
||||
deps = ["nsubj", "nmod", "nmod", "ROOT"]
|
||||
pos = ["NOUN", "NOUN", "NOUN", "VERB"]
|
||||
doc = Doc(fa_vocab, words=words, heads=heads, deps=deps, pos=pos)
|
||||
assert [c.text for c in doc.noun_chunks] == ["رئیس انجمن جراحان"]
|
||||
|
||||
|
||||
def test_fa_noun_chunks_drops_leading_adp(fa_vocab):
|
||||
words = ["در", "ایران", "بود"]
|
||||
heads = [1, 2, 2]
|
||||
deps = ["case", "obl", "ROOT"]
|
||||
pos = ["ADP", "NOUN", "VERB"]
|
||||
doc = Doc(fa_vocab, words=words, heads=heads, deps=deps, pos=pos)
|
||||
assert [c.text for c in doc.noun_chunks] == ["ایران"]
|
||||
```
|
||||
|
||||
## Other `spacy/lang/fa` gaps found while building this pipeline
|
||||
|
||||
Ordered by how much they cost a real Persian pipeline:
|
||||
|
||||
1. **Clitic splitting.** The `fa` tokenizer cannot split pronominal enclitics or the enclitic
|
||||
copula (`پدرم` → `پدر` + `م`, `ساکتند` → `ساکت` + `ند`), which UD treebanks annotate as
|
||||
multiword tokens. Measured on PerDT dev: gold-vs-tokenizer token F is 0.9823 when clitics
|
||||
are kept split, and 1.49% of tokens are affected. Persian-specific `TOKENIZER_SUFFIXES`
|
||||
entries for the enclitic set would remove the need for `--merge-subtokens` and eliminate
|
||||
the composite XPOS tags it produces.
|
||||
2. **`punctuation.py` defines only `TOKENIZER_SUFFIXES`** — no `TOKENIZER_PREFIXES`, no
|
||||
`TOKENIZER_INFIXES`. ZWNJ (U+200C) is handled only implicitly through the 65 KB generated
|
||||
verb-exception table. The missing infix rules bite on numerics: `spacy/lang/fa/examples.py`
|
||||
ships the sentence `دیروز علی به من ۲۰۰۰.۱﷼ پول نقد داد.`, and the trained pipeline splits
|
||||
`۲۰۰۰.۱﷼` into `۲۰۰۰` + `.` + `۱﷼`, with the stray `.` promoted to a sentence boundary —
|
||||
one input sentence comes out as three. `LIKE_NUM` in `lex_attrs.py` recognises Persian
|
||||
digits, but no tokenizer rule keeps a Persian decimal or a currency sign attached.
|
||||
3. **No tokenizer tests at all** for `fa` — `spacy/tests/lang/fa/` contains only
|
||||
`test_noun_chunks.py`, guarding none of the 65 KB exception table.
|
||||
4. **`spacy-lookups-data` has no `fa_license.txt`**, although `fa_source.txt` records that the
|
||||
lemma tables were "extracted from Mojgan Seraji's Persian Universal Dependencies Corpus" —
|
||||
which is CC BY-SA 4.0. Catalan ships a `ca_license.txt`; Persian should too.
|
||||
5. **No `fa_lemma_lookup.json`** — only rule-mode lemmatizer assets exist, so
|
||||
`mode="lookup"` is unavailable for Persian.
|
||||
|
||||
Items 1–3 are self-contained code PRs. Item 4 is a licence-hygiene PR against
|
||||
`spacy-lookups-data` and matters for anyone redistributing a Persian pipeline.
|
||||
|
|
@ -0,0 +1,197 @@
|
|||
title: "fa_core_news_sm"
|
||||
description: >
|
||||
A CPU-sized Persian (fa) core pipeline for spaCy 3.8: tagger (XPOS), morphologizer
|
||||
(UPOS + FEATS), trainable lemmatizer, dependency parser and NER.
|
||||
|
||||
UD components are trained on UD_Persian-PerDT (PerUDT v1.0, CC BY-SA 4.0). The NER
|
||||
component is trained separately on ParsTwiNER (MIT) and merged in, because no
|
||||
redistributably-licensed Persian NER corpus shares a genre with the treebank. See
|
||||
docs/MODELS.md for the full source/licence analysis and docs/CONTRIBUTING-GUIDE.md
|
||||
for how this gets published.
|
||||
|
||||
Run everything with: `spacy project run all`
|
||||
|
||||
vars:
|
||||
lang: "fa"
|
||||
package_name: "core_news_sm"
|
||||
package_version: "3.8.0"
|
||||
treebank: "fa_perdt"
|
||||
# -1 = CPU. A GTX 940MX (2 GB) is not worth the transfer overhead for an sm pipeline.
|
||||
gpu: -1
|
||||
n_sents: 10
|
||||
|
||||
directories:
|
||||
- "assets"
|
||||
- "corpus"
|
||||
- "configs"
|
||||
- "scripts"
|
||||
- "training"
|
||||
- "metrics"
|
||||
- "packages"
|
||||
|
||||
assets:
|
||||
- dest: "assets/ud/fa_perdt-ud-train.conllu"
|
||||
url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-train.conllu"
|
||||
checksum: "f5a8ba901a776b4fd1941ecadcc6d506"
|
||||
description: "UD_Persian-PerDT train split (CC BY-SA 4.0)"
|
||||
- dest: "assets/ud/fa_perdt-ud-dev.conllu"
|
||||
url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-dev.conllu"
|
||||
checksum: "f103020da7c1e917aafb8a8321f4cb84"
|
||||
description: "UD_Persian-PerDT dev split (CC BY-SA 4.0)"
|
||||
- dest: "assets/ud/fa_perdt-ud-test.conllu"
|
||||
url: "https://raw.githubusercontent.com/UniversalDependencies/UD_Persian-PerDT/master/fa_perdt-ud-test.conllu"
|
||||
checksum: "b62a66994cef2c50f7e524a1471102d8"
|
||||
description: "UD_Persian-PerDT test split (CC BY-SA 4.0)"
|
||||
- dest: "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
|
||||
url: "https://github.com/overfit-ir/parstwiner/releases/download/v1.0.0/ParsTwiNER_corpus_v1.0.0.zip"
|
||||
checksum: "54615340d76c4d51b8f54751b07a278d"
|
||||
description: "ParsTwiNER Persian Twitter NER corpus, IOB2 (MIT)"
|
||||
|
||||
workflows:
|
||||
all:
|
||||
- inspect
|
||||
- convert-ud
|
||||
- convert-ner
|
||||
- debug-data
|
||||
- train-core
|
||||
- train-ner
|
||||
- assemble
|
||||
- evaluate
|
||||
- finalize-meta
|
||||
- package
|
||||
- smoke
|
||||
|
||||
commands:
|
||||
- name: "inspect"
|
||||
help: "Report annotation coverage of the downloaded treebank(s)"
|
||||
script:
|
||||
- "python scripts/inspect_treebanks.py assets/ud"
|
||||
deps:
|
||||
- "assets/ud/fa_perdt-ud-train.conllu"
|
||||
- "scripts/inspect_treebanks.py"
|
||||
|
||||
- name: "convert-ud"
|
||||
help: >
|
||||
CoNLL-U -> DocBin. --merge-subtokens fuses multiword-token clitics into single
|
||||
tokens; docs/MODELS.md §5 has the measurement that justifies it.
|
||||
script:
|
||||
- "python -m spacy convert assets/ud/${vars.treebank}-ud-train.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens"
|
||||
- "python -m spacy convert assets/ud/${vars.treebank}-ud-dev.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens"
|
||||
- "python -m spacy convert assets/ud/${vars.treebank}-ud-test.conllu corpus/merged --converter conllu --n-sents ${vars.n_sents} --merge-subtokens"
|
||||
# Reproduce the measurement in docs/MODELS.md §5: convert the dev split WITHOUT
|
||||
# merging and compare both against the tokenizer we actually ship.
|
||||
- "python -m spacy convert assets/ud/${vars.treebank}-ud-dev.conllu corpus/split --converter conllu --n-sents ${vars.n_sents}"
|
||||
- "python scripts/tokenization_report.py corpus/merged/${vars.treebank}-ud-dev.spacy corpus/split/${vars.treebank}-ud-dev.spacy"
|
||||
deps:
|
||||
- "assets/ud/${vars.treebank}-ud-train.conllu"
|
||||
- "assets/ud/${vars.treebank}-ud-dev.conllu"
|
||||
- "assets/ud/${vars.treebank}-ud-test.conllu"
|
||||
outputs:
|
||||
- "corpus/merged/${vars.treebank}-ud-train.spacy"
|
||||
- "corpus/merged/${vars.treebank}-ud-dev.spacy"
|
||||
- "corpus/merged/${vars.treebank}-ud-test.spacy"
|
||||
|
||||
- name: "convert-ner"
|
||||
help: "Unpack ParsTwiNER and convert its IOB2 files to DocBin"
|
||||
script:
|
||||
- "python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner"
|
||||
- "python -m spacy convert assets/ner/train.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
|
||||
- "python -m spacy convert assets/ner/dev.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
|
||||
- "python -m spacy convert assets/ner/test.txt corpus/ner --converter ner --n-sents ${vars.n_sents} --lang ${vars.lang}"
|
||||
deps:
|
||||
- "assets/ner/ParsTwiNER_corpus_v1.0.0.zip"
|
||||
- "scripts/extract_parstwiner.py"
|
||||
outputs:
|
||||
- "corpus/ner/train.spacy"
|
||||
- "corpus/ner/dev.spacy"
|
||||
- "corpus/ner/test.spacy"
|
||||
|
||||
- name: "debug-data"
|
||||
help: "Validate both corpora against their configs before burning CPU on training"
|
||||
script:
|
||||
- "python -m spacy debug data configs/fa_core_news_sm.cfg --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy"
|
||||
- "python -m spacy debug data configs/fa_ner_sm.cfg --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy"
|
||||
deps:
|
||||
- "corpus/merged/${vars.treebank}-ud-train.spacy"
|
||||
- "corpus/ner/train.spacy"
|
||||
- "configs/fa_core_news_sm.cfg"
|
||||
- "configs/fa_ner_sm.cfg"
|
||||
|
||||
- name: "train-core"
|
||||
help: "Train tok2vec + tagger + morphologizer + trainable_lemmatizer + parser on PerDT"
|
||||
script:
|
||||
- "python -m spacy train configs/fa_core_news_sm.cfg --output training/core --paths.train corpus/merged/${vars.treebank}-ud-train.spacy --paths.dev corpus/merged/${vars.treebank}-ud-dev.spacy --gpu-id ${vars.gpu}"
|
||||
deps:
|
||||
- "corpus/merged/${vars.treebank}-ud-train.spacy"
|
||||
- "corpus/merged/${vars.treebank}-ud-dev.spacy"
|
||||
- "configs/fa_core_news_sm.cfg"
|
||||
outputs:
|
||||
- "training/core/model-best"
|
||||
|
||||
- name: "train-ner"
|
||||
help: "Train the standalone NER component (own internal tok2vec) on ParsTwiNER"
|
||||
script:
|
||||
- "python -m spacy train configs/fa_ner_sm.cfg --output training/ner --paths.train corpus/ner/train.spacy --paths.dev corpus/ner/dev.spacy --gpu-id ${vars.gpu}"
|
||||
deps:
|
||||
- "corpus/ner/train.spacy"
|
||||
- "corpus/ner/dev.spacy"
|
||||
- "configs/fa_ner_sm.cfg"
|
||||
outputs:
|
||||
- "training/ner/model-best"
|
||||
|
||||
- name: "assemble"
|
||||
help: "Source the trained ner into the core pipeline and write full meta.json"
|
||||
script:
|
||||
- "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version}"
|
||||
deps:
|
||||
- "training/core/model-best"
|
||||
- "training/ner/model-best"
|
||||
- "scripts/assemble_core.py"
|
||||
outputs:
|
||||
- "training/fa_core_news_sm"
|
||||
|
||||
- name: "evaluate"
|
||||
help: "Score the assembled pipeline on both held-out test sets"
|
||||
script:
|
||||
- "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/merged/${vars.treebank}-ud-test.spacy --output metrics/ud-test.json --gpu-id ${vars.gpu}"
|
||||
- "python -m spacy benchmark accuracy training/fa_core_news_sm corpus/ner/test.spacy --output metrics/ner-test.json --gpu-id ${vars.gpu}"
|
||||
deps:
|
||||
- "training/fa_core_news_sm"
|
||||
- "corpus/merged/${vars.treebank}-ud-test.spacy"
|
||||
- "corpus/ner/test.spacy"
|
||||
outputs:
|
||||
- "metrics/ud-test.json"
|
||||
- "metrics/ner-test.json"
|
||||
|
||||
- name: "finalize-meta"
|
||||
help: >
|
||||
Re-assemble, this time folding the test scores into meta.json["performance"].
|
||||
Separate from `assemble` because the scores can only exist after `evaluate`, and
|
||||
`evaluate` needs an assembled pipeline to score. Cheap: it only copies models.
|
||||
script:
|
||||
- "python scripts/assemble_core.py training/core/model-best training/ner/model-best training/fa_core_news_sm --version ${vars.package_version} --ud-metrics metrics/ud-test.json --ner-metrics metrics/ner-test.json"
|
||||
deps:
|
||||
- "metrics/ud-test.json"
|
||||
- "metrics/ner-test.json"
|
||||
- "scripts/assemble_core.py"
|
||||
|
||||
- name: "package"
|
||||
help: "Build the installable wheel + sdist"
|
||||
script:
|
||||
- "python -m spacy package training/fa_core_news_sm packages --name ${vars.package_name} --version ${vars.package_version} --build sdist,wheel --force"
|
||||
deps:
|
||||
- "training/fa_core_news_sm"
|
||||
outputs:
|
||||
- "packages/${vars.lang}_${vars.package_name}-${vars.package_version}"
|
||||
|
||||
- name: "smoke"
|
||||
help: "Load the packaged pipeline and run it over real Persian text"
|
||||
script:
|
||||
- "python scripts/smoke_test.py training/fa_core_news_sm"
|
||||
deps:
|
||||
- "training/fa_core_news_sm"
|
||||
|
||||
- name: "clean"
|
||||
help: "Drop corpora, training runs and metrics (keeps downloaded assets)"
|
||||
script:
|
||||
- "rm -rf corpus/merged corpus/split corpus/ner training metrics packages"
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
# Build environment for fa_core_news_sm. Python 3.12.
|
||||
spacy>=3.8,<3.9
|
||||
# Persian rule-lemmatizer tables (fa_lemma_exc/index/rules). Not used by the trained
|
||||
# pipeline (we train an edit-tree lemmatizer) but required if you want to compare against
|
||||
# the rule-based lemmatizer, and pulled in by `spacy init config` validation paths.
|
||||
spacy-lookups-data>=1.0.5
|
||||
# Publishing to the HF Hub (docs/CONTRIBUTING-GUIDE.md §2). Optional.
|
||||
# spacy-huggingface-hub>=0.0.10
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
"""Merge the separately-trained NER component into the UD pipeline and write full metadata.
|
||||
|
||||
The two components cannot be trained together: the UD components come from
|
||||
UD_Persian-PerDT (CC BY-SA 4.0, edited prose) and `ner` comes from ParsTwiNER (MIT,
|
||||
tweets). `ner` was therefore configured with its own embedded tok2vec (see
|
||||
configs/fa_ner_sm.cfg) so it can be sourced into another pipeline without a dangling
|
||||
Tok2VecListener — the same design as en_core_web_sm.
|
||||
|
||||
Run it twice: once right after training (no metrics yet), and again after
|
||||
`spacy benchmark accuracy` has produced metrics/*.json so `meta.json["performance"]`
|
||||
reflects the assembled pipeline on the held-out test sets.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python scripts/assemble_core.py training/core/model-best training/ner/model-best \\
|
||||
training/fa_core_news_sm --version 3.8.0 \\
|
||||
[--ud-metrics metrics/ud-test.json] [--ner-metrics metrics/ner-test.json]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import spacy
|
||||
|
||||
DESCRIPTION = (
|
||||
"Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, "
|
||||
"trainable_lemmatizer, parser, ner."
|
||||
)
|
||||
|
||||
SOURCES = [
|
||||
{
|
||||
"name": "UD_Persian-PerDT (PerUDT v1.0)",
|
||||
"url": "https://github.com/UniversalDependencies/UD_Persian-PerDT",
|
||||
"author": (
|
||||
"Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian"
|
||||
),
|
||||
"license": "CC BY-SA 4.0",
|
||||
},
|
||||
{
|
||||
"name": "ParsTwiNER",
|
||||
"url": "https://github.com/overfit-ir/parstwiner",
|
||||
"author": "MohammadMahdi Aghajani, AliAkbar Badri, Hamid Beigy et al. (Overfit-IR)",
|
||||
"license": "MIT",
|
||||
},
|
||||
{
|
||||
"name": "spaCy lang/fa language data (stop words originally from HAZM)",
|
||||
"url": "https://github.com/explosion/spaCy/tree/master/spacy/lang/fa",
|
||||
"author": "Explosion and spaCy contributors",
|
||||
"license": "MIT",
|
||||
},
|
||||
]
|
||||
|
||||
# CC BY-SA 4.0 on the treebank propagates to anything derived from it.
|
||||
NOTES = (
|
||||
"The tagger, morphologizer, trainable_lemmatizer and parser are trained on "
|
||||
"UD_Persian-PerDT, which is licensed CC BY-SA 4.0; this pipeline is therefore "
|
||||
"distributed under CC BY-SA 4.0 with attribution to the treebank authors. "
|
||||
"The ner component is trained on ParsTwiNER (MIT), a Twitter corpus, so entity "
|
||||
"recognition is weaker on formal/edited prose than on social media text. "
|
||||
"Multiword tokens in the treebank (pronominal clitics, enclitic copulas) were merged "
|
||||
"with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite "
|
||||
"(e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space."
|
||||
)
|
||||
|
||||
|
||||
def load_metrics(path):
|
||||
if not path:
|
||||
return {}
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
print(f" (no metrics at {p}, skipping)")
|
||||
return {}
|
||||
return json.loads(p.read_text(encoding="utf8"))
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("core", help="trained UD pipeline (training/core/model-best)")
|
||||
ap.add_argument("ner", help="trained NER pipeline (training/ner/model-best)")
|
||||
ap.add_argument("output", help="destination directory for the assembled pipeline")
|
||||
ap.add_argument("--version", default="3.8.0")
|
||||
ap.add_argument("--ud-metrics", default=None)
|
||||
ap.add_argument("--ner-metrics", default=None)
|
||||
args = ap.parse_args()
|
||||
|
||||
nlp = spacy.load(args.core)
|
||||
ner_nlp = spacy.load(args.ner)
|
||||
if "ner" in nlp.pipe_names:
|
||||
nlp.remove_pipe("ner")
|
||||
nlp.add_pipe("ner", source=ner_nlp)
|
||||
print(f"pipeline: {nlp.pipe_names}")
|
||||
print(f"ner labels: {sorted(nlp.get_pipe('ner').labels)}")
|
||||
|
||||
ud = load_metrics(args.ud_metrics)
|
||||
ner = load_metrics(args.ner_metrics)
|
||||
performance = dict(nlp.meta.get("performance", {}))
|
||||
for key in ("token_acc", "token_p", "token_r", "token_f", "tag_acc", "pos_acc",
|
||||
"morph_acc", "lemma_acc", "dep_uas", "dep_las", "sents_p", "sents_r",
|
||||
"sents_f"):
|
||||
if key in ud:
|
||||
performance[key] = ud[key]
|
||||
for key in ("ents_p", "ents_r", "ents_f"):
|
||||
if key in ner:
|
||||
performance[key] = ner[key]
|
||||
if "ents_per_type" in ner:
|
||||
performance["ents_per_type"] = ner["ents_per_type"]
|
||||
if "dep_las_per_type" in ud:
|
||||
performance["dep_las_per_type"] = ud["dep_las_per_type"]
|
||||
|
||||
nlp.meta.update(
|
||||
{
|
||||
"lang": "fa",
|
||||
"name": "core_news_sm",
|
||||
"version": args.version,
|
||||
"description": DESCRIPTION,
|
||||
"author": "",
|
||||
"email": "",
|
||||
"url": "",
|
||||
"license": "CC BY-SA 4.0",
|
||||
"sources": SOURCES,
|
||||
"notes": NOTES,
|
||||
"performance": performance,
|
||||
}
|
||||
)
|
||||
|
||||
out = Path(args.output)
|
||||
nlp.to_disk(out)
|
||||
print(f"wrote {out}")
|
||||
print(json.dumps(performance, indent=2, ensure_ascii=False)[:1200])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,137 @@
|
|||
"""Evidence for the upstream `spacy/lang/fa/syntax_iterators.py` bug.
|
||||
|
||||
The shipped Persian `noun_chunks` iterator matches these dependency labels:
|
||||
|
||||
nsubj, dobj, nsubjpass, pcomp, pobj, dative, appos, attr, ROOT
|
||||
|
||||
`dobj`, `nsubjpass`, `pobj`, `dative` and `attr` are ClearNLP/English labels. They do not
|
||||
exist in Universal Dependencies, and every Persian treebank is UD. So on a UD-trained `fa`
|
||||
pipeline the iterator only ever fires on `nsubj`, `appos`, `ROOT` and `conj`.
|
||||
|
||||
It also expands each chunk to `word.left_edge`, which is wrong for Persian: ezafe
|
||||
constructions put modifiers to the RIGHT of the head noun (`رئیس انجمن جراحان قلب ایران`),
|
||||
so left-only expansion truncates the phrase to the head.
|
||||
|
||||
This script quantifies both problems against the dev corpus and prints a side-by-side
|
||||
comparison with the proposed UD implementation (see docs/upstream/fa-noun-chunks.md).
|
||||
|
||||
Usage:
|
||||
.venv/bin/python scripts/check_noun_chunks.py training/core/model-best \\
|
||||
corpus/merged/fa_perdt-ud-dev.spacy
|
||||
"""
|
||||
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
import spacy
|
||||
from spacy.lang.fa.syntax_iterators import noun_chunks as shipped_noun_chunks
|
||||
from spacy.symbols import NOUN, PRON, PROPN
|
||||
from spacy.tokens import DocBin
|
||||
|
||||
# Labels a UD-trained parser can actually emit, as heads of a base noun phrase.
|
||||
UD_LABELS = [
|
||||
"nsubj",
|
||||
"nsubj:pass",
|
||||
"obj",
|
||||
"iobj",
|
||||
"obl",
|
||||
"obl:arg",
|
||||
"nmod",
|
||||
"appos",
|
||||
"vocative",
|
||||
"ROOT",
|
||||
]
|
||||
# Relations that continue a Persian noun phrase to the right of its head.
|
||||
UD_POST_MODIFIERS = ["nmod", "nmod:poss", "amod", "flat", "flat:name", "flat:num",
|
||||
"fixed", "compound", "det", "nummod"]
|
||||
|
||||
|
||||
def proposed_noun_chunks(doclike):
|
||||
"""UD-label implementation with right-side ezafe expansion."""
|
||||
doc = doclike.doc
|
||||
if not doc.has_annotation("DEP"):
|
||||
raise ValueError("requires a dependency parse")
|
||||
np_deps = {doc.vocab.strings.add(label) for label in UD_LABELS}
|
||||
np_modifs = {doc.vocab.strings.add(label) for label in UD_POST_MODIFIERS}
|
||||
np_label = doc.vocab.strings.add("NP")
|
||||
conj = doc.vocab.strings.add("conj")
|
||||
adp_pos = doc.vocab.strings.add("ADP")
|
||||
cconj_pos = doc.vocab.strings.add("CCONJ")
|
||||
prev_end = -1
|
||||
for word in doclike:
|
||||
if word.pos not in (NOUN, PROPN, PRON):
|
||||
continue
|
||||
if word.left_edge.i <= prev_end:
|
||||
continue
|
||||
head_dep = word.dep
|
||||
if head_dep == conj:
|
||||
head = word.head
|
||||
while head.dep == conj and head.head.i < head.i:
|
||||
head = head.head
|
||||
if head.dep not in np_deps:
|
||||
continue
|
||||
elif head_dep not in np_deps:
|
||||
continue
|
||||
# Expand right through ezafe / modifier chains.
|
||||
right = word
|
||||
for child in word.rights:
|
||||
if child.dep in np_modifs:
|
||||
right = child.right_edge
|
||||
else:
|
||||
break
|
||||
start, end = word.left_edge.i, max(word.i, right.i) + 1
|
||||
# Mirror fr/es: a leading preposition or coordinator is not part of the NP.
|
||||
while start < word.i and doc[start].pos in (adp_pos, cconj_pos):
|
||||
start += 1
|
||||
if end <= prev_end:
|
||||
continue
|
||||
prev_end = end - 1
|
||||
yield start, end, np_label
|
||||
|
||||
|
||||
def spans(doc, iterator):
|
||||
return [doc[s:e].text for s, e, _ in iterator(doc)]
|
||||
|
||||
|
||||
def main():
|
||||
model, corpus = sys.argv[1], sys.argv[2]
|
||||
nlp = spacy.load(model)
|
||||
docs = list(DocBin().from_disk(corpus).get_docs(nlp.vocab))
|
||||
|
||||
shipped_total = proposed_total = 0
|
||||
shipped_tokens = proposed_tokens = 0
|
||||
dep_hist = Counter()
|
||||
for doc in docs:
|
||||
for t in doc:
|
||||
if t.pos in (NOUN, PROPN, PRON):
|
||||
dep_hist[t.dep_] += 1
|
||||
for s, e, _ in shipped_noun_chunks(doc):
|
||||
shipped_total += 1
|
||||
shipped_tokens += e - s
|
||||
for s, e, _ in proposed_noun_chunks(doc):
|
||||
proposed_total += 1
|
||||
proposed_tokens += e - s
|
||||
|
||||
print(f"gold dev corpus: {len(docs)} docs")
|
||||
print(f"shipped noun_chunks: {shipped_total:>6} chunks, "
|
||||
f"{shipped_tokens / max(shipped_total, 1):.2f} tokens/chunk")
|
||||
print(f"proposed noun_chunks: {proposed_total:>6} chunks, "
|
||||
f"{proposed_tokens / max(proposed_total, 1):.2f} tokens/chunk")
|
||||
print("\ndeprels on NOUN/PROPN/PRON tokens in dev (top 15):")
|
||||
for dep, n in dep_hist.most_common(15):
|
||||
reachable = "shipped" if dep in {"nsubj", "appos", "conj", "ROOT", "root"} else "-"
|
||||
print(f" {dep:<14}{n:>7} {reachable}")
|
||||
|
||||
print("\nside by side on live text:")
|
||||
for text in [
|
||||
"رئیس انجمن جراحان قلب ایران تأکید کرد که امکانات پیشرفته در ایران وجود دارد.",
|
||||
"دانشگاه تهران بزرگترین دانشگاه ایران است.",
|
||||
]:
|
||||
doc = nlp(text)
|
||||
print(f"\n {text}")
|
||||
print(f" shipped : {spans(doc, shipped_noun_chunks)}")
|
||||
print(f" proposed: {spans(doc, proposed_noun_chunks)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
"""Unpack the ParsTwiNER release zip into flat IOB2 files.
|
||||
|
||||
The archive carries macOS resource-fork junk (`__MACOSX/._*`) that would confuse
|
||||
`spacy convert`, so only the three real splits are extracted.
|
||||
|
||||
Usage: .venv/bin/python scripts/extract_parstwiner.py assets/ner/ParsTwiNER_corpus_v1.0.0.zip assets/ner
|
||||
"""
|
||||
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
SPLITS = ("train.txt", "dev.txt", "test.txt")
|
||||
|
||||
|
||||
def main():
|
||||
archive = Path(sys.argv[1])
|
||||
dest = Path(sys.argv[2])
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(archive) as zf:
|
||||
names = set(zf.namelist())
|
||||
missing = [s for s in SPLITS if s not in names]
|
||||
if missing:
|
||||
raise SystemExit(f"{archive}: missing expected members {missing}")
|
||||
for split in SPLITS:
|
||||
zf.extract(split, dest)
|
||||
path = dest / split
|
||||
sents = path.read_text(encoding="utf8").strip().split("\n\n")
|
||||
tokens = sum(1 for line in path.open(encoding="utf8") if line.strip())
|
||||
print(f"{path}: {len(sents)} sentences, {tokens} tokens")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
"""Report annotation coverage of the UD Persian treebanks in assets/ud/.
|
||||
|
||||
Usage: .venv/bin/python scripts/inspect_treebanks.py [assets/ud]
|
||||
"""
|
||||
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def read_conllu(path):
|
||||
sent = []
|
||||
for line in path.open(encoding="utf8"):
|
||||
line = line.rstrip("\n")
|
||||
if not line:
|
||||
if sent:
|
||||
yield sent
|
||||
sent = []
|
||||
continue
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
sent.append(line.split("\t"))
|
||||
if sent:
|
||||
yield sent
|
||||
|
||||
|
||||
def stats(path):
|
||||
s = {
|
||||
"sents": 0,
|
||||
"toks": 0,
|
||||
"mwt": 0,
|
||||
"empty": 0,
|
||||
"lemma": 0,
|
||||
"feats": 0,
|
||||
"nonproj_root": 0,
|
||||
}
|
||||
upos, xpos, dep = Counter(), Counter(), Counter()
|
||||
for sent in read_conllu(path):
|
||||
s["sents"] += 1
|
||||
roots = 0
|
||||
for c in sent:
|
||||
if "-" in c[0]:
|
||||
s["mwt"] += 1
|
||||
continue
|
||||
if "." in c[0]:
|
||||
s["empty"] += 1
|
||||
continue
|
||||
s["toks"] += 1
|
||||
upos[c[3]] += 1
|
||||
xpos[c[4]] += 1
|
||||
dep[c[7]] += 1
|
||||
if c[2] not in ("_", ""):
|
||||
s["lemma"] += 1
|
||||
if c[5] != "_":
|
||||
s["feats"] += 1
|
||||
if c[7] == "root":
|
||||
roots += 1
|
||||
if roots != 1:
|
||||
s["nonproj_root"] += 1
|
||||
return s, upos, xpos, dep
|
||||
|
||||
|
||||
def main():
|
||||
root = Path(sys.argv[1] if len(sys.argv) > 1 else "assets/ud")
|
||||
agg = {}
|
||||
print(
|
||||
f"{'file':<30}{'sents':>8}{'tokens':>10}{'MWT':>7}{'empty':>7}"
|
||||
f"{'lemma%':>8}{'feats%':>8}{'UPOS':>6}{'XPOS':>6}{'DEP':>5}"
|
||||
)
|
||||
for path in sorted(root.glob("*.conllu")):
|
||||
s, upos, xpos, dep = stats(path)
|
||||
agg[path.name] = (s, upos, xpos, dep)
|
||||
print(
|
||||
f"{path.name:<30}{s['sents']:>8}{s['toks']:>10}{s['mwt']:>7}{s['empty']:>7}"
|
||||
f"{100 * s['lemma'] / s['toks']:>8.1f}{100 * s['feats'] / s['toks']:>8.1f}"
|
||||
f"{len(upos):>6}{len(xpos):>6}{len(dep):>5}"
|
||||
)
|
||||
|
||||
for name, (s, upos, xpos, dep) in agg.items():
|
||||
if not name.endswith("train.conllu"):
|
||||
continue
|
||||
print(f"\n=== {name} ===")
|
||||
print(f"UPOS ({len(upos)}): {' '.join(sorted(upos))}")
|
||||
print(f"XPOS ({len(xpos)}): {' '.join(sorted(xpos))}")
|
||||
print(f"DEPREL ({len(dep)}): {' '.join(sorted(dep))}")
|
||||
print(f"sentences with != 1 root: {s['nonproj_root']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
"""Run the assembled pipeline over real Persian text and print every annotation layer.
|
||||
|
||||
This is the end-to-end check that the artifact actually works: tokenizer -> tagger ->
|
||||
morphologizer -> lemmatizer -> parser -> ner -> noun_chunks.
|
||||
|
||||
Usage: .venv/bin/python scripts/smoke_test.py training/fa_core_news_sm
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
import spacy
|
||||
from spacy.lang.fa.examples import sentences as FA_EXAMPLES
|
||||
|
||||
EXTRA = [
|
||||
# ZWNJ-heavy verb forms, an enclitic pronoun, and named entities.
|
||||
"دانشگاه تهران در سال ۱۳۱۳ تأسیس شد و بزرگترین دانشگاه ایران است.",
|
||||
"کتابهایش را روی میز گذاشت و به سرعت از خانه بیرون رفت.",
|
||||
"شرکت ایران خودرو اعلام کرد که تولید خود را افزایش میدهد.",
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
path = sys.argv[1] if len(sys.argv) > 1 else "training/fa_core_news_sm"
|
||||
nlp = spacy.load(path)
|
||||
print(f"loaded {nlp.meta['lang']}_{nlp.meta['name']} {nlp.meta['version']}")
|
||||
print(f"pipeline: {nlp.pipe_names}")
|
||||
|
||||
for text in list(FA_EXAMPLES) + EXTRA:
|
||||
doc = nlp(text)
|
||||
print("\n" + "=" * 78)
|
||||
print(text)
|
||||
print(f"{'TEXT':<16}{'LEMMA':<16}{'UPOS':<7}{'TAG':<14}{'DEP':<14}HEAD")
|
||||
for t in doc:
|
||||
print(
|
||||
f"{t.text:<16}{t.lemma_:<16}{t.pos_:<7}{t.tag_:<14}"
|
||||
f"{t.dep_:<14}{t.head.text}"
|
||||
)
|
||||
print(f"morph[0]: {doc[0].morph}")
|
||||
print(f"sents: {[s.text for s in doc.sents]}")
|
||||
print(f"ents: {[(e.text, e.label_) for e in doc.ents]}")
|
||||
print(f"noun_chunks: {[c.text for c in doc.noun_chunks]}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
"""Compare gold tokenisation against spaCy's rule-based `fa` tokenizer.
|
||||
|
||||
Answers the only question that decides `spacy convert --merge-subtokens` for Persian:
|
||||
how often can the tokenizer we ship at runtime reproduce the gold token boundaries?
|
||||
|
||||
Usage: .venv/bin/python scripts/tokenization_report.py corpus/merged/fa_perdt-ud-dev.spacy [...]
|
||||
"""
|
||||
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
import spacy
|
||||
from spacy.tokens import DocBin
|
||||
|
||||
|
||||
def offsets(doc):
|
||||
return {(t.idx, t.idx + len(t.text)) for t in doc}
|
||||
|
||||
|
||||
def report(path, nlp):
|
||||
gold_docs = list(DocBin().from_disk(path).get_docs(nlp.vocab))
|
||||
tp = gold_n = pred_n = 0
|
||||
tags = Counter()
|
||||
multi_lemma = 0
|
||||
tokens = 0
|
||||
for gold in gold_docs:
|
||||
pred = nlp.make_doc(gold.text)
|
||||
g, p = offsets(gold), offsets(pred)
|
||||
tp += len(g & p)
|
||||
gold_n += len(g)
|
||||
pred_n += len(p)
|
||||
for t in gold:
|
||||
tokens += 1
|
||||
tags[t.tag_] += 1
|
||||
if " " in t.lemma_:
|
||||
multi_lemma += 1
|
||||
precision = tp / pred_n
|
||||
recall = tp / gold_n
|
||||
f = 2 * precision * recall / (precision + recall)
|
||||
composite = {t: n for t, n in tags.items() if "_" in t and t.count("_") > 1}
|
||||
print(f"\n== {path}")
|
||||
print(f"docs {len(gold_docs)} gold tokens {gold_n} predicted tokens {pred_n}")
|
||||
print(f"token P {precision:.4f} R {recall:.4f} F {f:.4f}")
|
||||
print(f"tag types {len(tags)}")
|
||||
print(f"tags containing >1 underscore (merge artefacts): {len(composite)}"
|
||||
f" covering {sum(composite.values())} tokens"
|
||||
f" ({100 * sum(composite.values()) / tokens:.2f}%)")
|
||||
if composite:
|
||||
top = ", ".join(f"{t}={n}" for t, n in Counter(composite).most_common(8))
|
||||
print(f" most frequent: {top}")
|
||||
print(f"lemmas containing a space (merge artefacts): {multi_lemma}"
|
||||
f" ({100 * multi_lemma / tokens:.2f}%)")
|
||||
|
||||
|
||||
def main():
|
||||
nlp = spacy.blank("fa")
|
||||
for arg in sys.argv[1:]:
|
||||
report(Path(arg), nlp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
Reference in New Issue