lfoppiano/MatTPUSciBERT

Material SciBERT (TPU): Improving language understanding in materials science

Work in progress

Introduction

SciBERT-based model pre-trained with materials-science scientific full text.

Authors

Luca Foppiano Pedro Ortiz Suarez

TLDR

• Collected full text from ~700,000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM-PF/en/), a dataset called ScienceCorpus (SciCorpus)
• We added to the SciBERT vocabulary (32k tokens) 100 domain-specific unknown words extracted from SciCorpus with a keyword modeller (KeyBERT)
• Starting conditions: original SciBERT weights
• Pre-trained the MatTPUSciBERT model on Google Cloud with a TPU (Tensor Processing Unit) as follows:
  • 800,000 steps with batch_size 256, max_seq_length 512
  • 100,000 steps with batch_size 2048, max_seq_length 128
• Fine-tuned and tested on NER for superconductors (https://github.com/lfoppiano/grobid-superconductors) and physical quantities (https://github.com/kermitt2/grobid-quantities)

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("lfoppiano/MatTPUSciBERT")
model = AutoModelForMaskedLM.from_pretrained("lfoppiano/MatTPUSciBERT")

The repository ships both PyTorch (pytorch_model.bin, model.safetensors) and the original TensorFlow checkpoint (bert_model.ckpt.*).

Related work

BERT Implementations

• BERT (the original) https://arxiv.org/abs/1810.04805
• RoBERTa (Re-implementation by Facebook) https://arxiv.org/abs/1907.11692

Relevant models

• SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert
• MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K papers limited to 4 MS families http://github.com/m3rg-iitd/matscibert
• MaterialBERT: Not yet published
• MatBERT (CEDER): BERT, from scratch, 2M documents on materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT
• BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/

Results

Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft).

NER Superconductors

Model	Precision	Recall	F1
SciBERT (baseline)	81.62%	84.23%	82.90%
MatSciBERT (Gupta)	81.45%	84.36%	82.88%
MatTPUSciBERT	82.13%	85.15%	83.61%
MatBERT (Ceder)	81.25%	83.99%	82.60%
BatteryScibert-cased	81.09%	84.14%	82.59%

NER Quantities

Model	Precision	Recall	F1
SciBERT (baseline)	88.73%	86.76%	87.73%
MatSciBERT (Gupta)	84.98%	90.12%	87.47%
MatTPUSciBERT	88.62%	86.33%	87.46%
MatBERT (Ceder)	85.08%	89.93%	87.44%
BatteryScibert-cased	85.02%	89.30%	87.11%

References

This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers

Acknowledgements

TBA