Modernbert Release Fixes

[warner-benjamin]

This PR resolves two issues with ModernBERT so it will be working for release in just under an hour.

First, ModernBERT has a final head layer ModernBertPredictionHead with a pretrained Linear layer and LayerNorm . In downstream heads, these two layers are loaded to ModernBertPoolingHead. I readded ModernBertPoolingHead to ModernBertForTokenClassification so TokenClassification models will load the pretrained layer weights.

Second, #35235 appears to have broken the SDPA/Eager paths by changing how the RoPE layer ModernBERT inherited from works. In the interest of expediency, I unmoduarized it so we can have a working ModernBERT implementation on all paths.

cc @ArthurZucker @tomaarsen

[tomaarsen]

The rotary embedding solution is a temporary fix - I'll introduce a modular-approved one soon, but we have a release to hit.

I didn't realise the ForTokenClassification model also requires the dense -> act -> norm -> drop, I thought it was only drop like with BERT. My apologies, thanks for the quick fix.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks 🤗

[sileod]

Hi, thanks, is there a plan to support multiplechoice ?

[stefan-it]

Have you already road-tested the token classification implementation? E.g. using the ModernBERT Large model only yields to ~91% F-Score on CoNLL-2003 (Test set, I tried some batch size and learning rate configurations), in two different frameworks (Transformers and Flair). I am not sure if ModernBERT needs really special hyper-parameters or if there are fixes needed in the architecture 🤔

I tested it with b5a557e.

[KoichiYasuoka]

Hi @stefan-it I've tried to test ModernBertForTokenClassification with ModernBERT-base and UD_English-EWT and got 97% F1-Score on UPOS (see my report written in Japanese). During my test I found that ModernBertForTokenClassification lacked get_input_embeddings() and made PR #35422. And I seem the tokenizer of ModernBERT-base is slightly difficult to handle POS-tagging (than BERT's one) and "classifier_dropout": 0.0 seems too small.