Graph Extraction from Books with Domain-Adapted Pretrained Models

End-to-end NLP framework for extracting dynamic, chapter-level character interaction networks from literary novels and predicting continuous pairwise character affinity scores in $[0, 1]$ using domain-adapted Longformer regression models with SHAP interpretability.

📄 Full Paper: Graph Extraction from Books with Domain-Adapted Pretrained Models (report.pdf)

---

Authors

Bocconi University — Milan, Italy (Equal contribution)

Author	GitHub	Email
Tommaso Aiello	@AielloTommaso	tommaso.aiello@studbocconi.it
Alessandro Bottardi	@alessandrobottardi	alessandro.bottardi@studbocconi.it
Gregorio Ceria	@gregorioceria	gregorio.ceria@studbocconi.it
Niccolò Pasi	@npasi	niccolo.pasi@studbocconi.it

---

Overview

Computational literary studies traditionally represent character relationships as static, book-level networks, collapsing an entire narrative into a single graph and relying solely on co-occurrence counts. Such static representations fail to capture how bonds evolve across chapters or whether interactions are affectionate, cooperative, neutral, or hostile.

This repository implements a five-stage pipeline that models evolving character affinity over time:

1. Chapter Splitting & BookNLP Processing: Raw novels are segmented into chapters via multi-criteria boundary validation (prefixed headings, Roman/Arabic enumerations, progressive ordering checks) and processed through the full BookNLP pipeline (entity recognition, quote attribution, supersense tagging, event detection, and coreference resolution).
2. Character Resolution: Coreference clusters are filtered per chapter, narrator/weak chains are pruned, in-quote pronouns are reassigned to speakers, and extracted character pairs are aligned against ground-truth inventories via alias expansion and fuzzy matching.
3. Lexical & Statistical Edge Scoring: Undirected chapter-level character graphs are enriched with sentence-level Jaccard co-occurrence ($w_{ij} = \frac{|S_i \cap S_j|}{|S_i \cup S_j|}$), smoothed normalized PMI, curated 5-class interaction verb lexicons (affective, cooperative, dialogue, conflict, hostility), and VADER sentiment fallback.
4. Temporally Conditioned Dataset Construction: Across 44 annotated English novels, each (book, chapter, char_a, char_b) instance pairs co-occurrence narrative sentences (tagged with <e1> / <e2> global-attention entity markers) with the previous chapter's affinity score, allowing the model to update relationship states over narrative time.
5. Longformer Affinity Regression & SHAP Analysis: We compare a Pretrained Longformer Baseline against two fine-tuned variants (Longformer MSE Regressor and Longformer Pearson + Weighted-MSE Regressor) and explain model behavior using exact 3-coalition SHAP attributions and top-$k$ token masking ($\Delta_{10}$).

flowchart LR
    A["Raw Book (.txt)"] --> B["Stage 1: Chapter Splitter & BookNLP"]
    B --> C["Stage 2: Character & Coref Resolution"]
    C --> D["Stage 3: Edge Scoring (PMI + Verb Lexicon + VADER)"]
    D --> E["Stage 4: Temporal Conditioning (Prev-Chapter Score + <e1>/<e2>)"]
    E --> F["Stage 5: Longformer Affinity Regressor & SHAP Analysis"]

Loading

---

Key Results

Quantitative Evaluation (44 Annotated English Novels)

We evaluate models on pointwise error (MSE), linear and rank correlation (Pearson, Spearman), chapter-level top-10 ranking agreement (nDCG@10), and strong-relationship detection (F1@strong):

Method	MSE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$	nDCG@10 $\uparrow$	F1@strong $\uparrow$
Longformer Pretrained Baseline	0.034	$-0.037$	$-0.041$	0.817	$0.183$
Longformer MSE Regressor	$0.039$	0.083	0.028	$0.814$	0.195
Longformer Pearson Regressor	$0.038$	$-0.124$	$-0.171$	$0.771$	$0.139$

• Longformer MSE Regressor is the strongest overall configuration: it is the only variant achieving positive Pearson and Spearman correlations with ground truth and the highest F1@strong ($0.195$), while maintaining competitive chapter-level ranking (nDCG@10 = $0.814$).
• While the Pretrained Baseline achieves a slightly lower MSE ($0.034$), its negative correlations reflect score compression toward the center of the $[0, 1]$ scale rather than meaningful relational discrimination.

SHAP Interpretability Analysis

Exact Shapley attributions across the three input regions (Previous Score, Character Names, Narrative Text) and top-10 gradient-token removal ($\Delta_{10}$) confirm that MSE fine-tuning shifts the model away from near-constant mid-scale outputs ($\Delta_{10} = 0.003 \rightarrow 0.137$) toward active temporal conditioning and narrative evidence:

Method	Previous (%)	Characters (%)	Narrative (%)	Top-10 Removal $\Delta_{10}$ $\uparrow$
Longformer Pretrained Baseline	$39.3%$	$12.7%$	$48.0%$	$0.003$
Longformer MSE Regressor	$61.5%$	$11.7%$	$26.9%$	0.137

Qualitative Network Comparison (Jude the Obscure, Chapters 5–6)

Fine-tuning with MSE produces sharper chapter-to-chapter contrast and less score compression than the pretrained baseline:

Chapter	Pretrained Longformer Baseline	MSE-Fine-Tuned Longformer
Chapter 5
Chapter 6

---

Repository Structure

NLP-Graph/
├── report.pdf                          # Full academic paper (PDF)
├── README.md                           # Project documentation
├── requirements.txt                    # Python 3.11 dependencies
├── pkg_resources.py                    # Lightweight compatibility shim for BookNLP
├── data/                               # Tracked directory skeleton (.gitkeep only; heavy artifacts gitignored)
│   ├── raw/                            # Raw book .txt inputs
│   ├── dataset/                        # ground_truth/, interactions/, texts/
│   ├── output/                         # chapters/ and booknlp/ intermediate outputs
│   ├── splits/                         # Book-level train/val/test split definitions
│   ├── training/                       # Processed JSONL datasets for Longformer
│   ├── models/                         # Cached BookNLP & fine-tuned Longformer checkpoints
│   ├── predictions/                    # Per-book predicted affinity CSVs across models
│   └── evaluation/                     # Aggregate metrics, comparisons, and diagnostics
├── results_and_plots/                  # Committed experimental figures, distributions, and sample graphs
│   ├── Sample_Improvement/             # Qualitative comparison graphs (Jude the Obscure, Ch. 5-6)
│   ├── Sample_output_graph/            # Per-chapter and max-change graphs for test books (95, 99, 110)
│   ├── training_testing_results/       # 2-way and 3-way evaluation plots across splits
│   ├── sample_book_improvements/       # Validation book progression comparisons
│   └── histplots/                      # Ground-truth vs. predicted score and delta distributions
└── src/
    ├── chapter_splitter.py             # Multi-criteria chapter boundary detection & validation
    ├── run_booknlp_per_chapter.py      # Per-chapter BookNLP execution & caching
    ├── run_booknlp_only.py             # Full-book BookNLP runner
    ├── step1_split_only.py             # CLI step 1: split raw book into validated chapters
    ├── step2_character_list_per_chapter.py      # CLI step 2: extract character inventories & aliases
    ├── step3_predicates_between_characters.py   # CLI step 3: extract pairwise verb predicates
    ├── step4_sentences_by_character_pair.py     # CLI step 4: extract pairwise co-occurrence sentences
    ├── split_full_book_to_chapters.py  # Byte-offset chapter assignment from full-book BookNLP
    ├── graph_builder.py                # BookNLP wrapper, Windows/Transformers patches, graph construction
    ├── relation_lexicon.py             # 5-class curated interaction verb dictionary
    ├── relation_scorer.py              # Hybrid PMI, verb lexicon, and VADER edge scorer
    ├── build_book_graphs.py            # Chapter-level NetworkX graph builder
    ├── run_pipeline_all_books.py       # Batch runner across multiple books
    ├── validate_chapter_alignment.py   # Ground-truth vs. extracted chapter count verification
    ├── audit_dataset_mismatch.py       # Dataset coverage and character resolution diagnostics
    ├── explore_graphs.ipynb            # Interactive notebook for inspecting chapter graphs
    └── longformer_pipeline/            # Supervised Longformer affinity modeling & interpretability
        ├── build_dataset.py            # Fuzzy entity matching & JSONL dataset generation
        ├── input_builder.py            # Entity marker (<e1>/<e2>) & prior-score prefix formatting
        ├── scorer.py                   # Longformer architecture with global attention on entity tokens
        ├── train.py                    # Fine-tuning with MSE or composite Pearson + Weighted-MSE loss
        ├── inference.py                # Sequential chapter inference with absent-pair carry-forward
        ├── batch_inference.py          # Multi-book inference runner
        ├── evaluate.py                 # MSE, Pearson, Spearman, nDCG@10, and F1@strong evaluation
        ├── explain_shap.py             # Exact 3-region Shapley values & top-10 token masking delta
        ├── diagnose_score_compression.py # Prediction variance and compression diagnostics
        └── plot_*.py / report_*.py     # Graph rendering and metric comparison scripts

---

Installation

Use Python 3.11 (required for compatibility with booknlp>=1.0.7.1, transformers==4.30.0, and tokenizers==0.13.3).

macOS / Linux

git clone https://github.com/npasi/NLP-Graph.git
cd NLP-Graph

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip setuptools
python -m pip install -r requirements.txt
python -m spacy download en_core_web_sm

Windows (PowerShell)

git clone https://github.com/npasi/NLP-Graph.git
cd NLP-Graph

py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip setuptools
python -m pip install -r requirements.txt
python -m spacy download en_core_web_sm

Optional: Local Model Cache

To keep HuggingFace and BookNLP weights self-contained inside data/hf_cache:

export HF_HOME="$PWD/data/hf_cache"
export TRANSFORMERS_CACHE="$PWD/data/hf_cache"
export XDG_CACHE_HOME="$PWD/data/hf_cache"

---

Usage

Part 1: Chapter Splitting & BookNLP Graph Extraction

Place a raw novel text file under data/raw/<book_id>.txt (e.g., data/raw/46.txt):

# 1. Split novel into chapters
python src/step1_split_only.py --input data/raw/46.txt --book-id 46 --log-level INFO

# 2. Run BookNLP per chapter (entity, quote, supersense, event, coref)
python src/run_booknlp_per_chapter.py --book-id 46 --model-size big \
    --pipeline entity,quote,supersense,event,coref

# 3. Extract per-chapter characters, predicates, and co-occurrence sentences
python src/step2_character_list_per_chapter.py    --book-id 46
python src/step3_predicates_between_characters.py --book-id 46
python src/step4_sentences_by_character_pair.py   --book-id 46

# 4. Build weighted chapter graphs (PMI + curated verb lexicon + VADER)
python -m src.run_pipeline_all_books --book 46

Part 2: Longformer Affinity Fine-Tuning, Inference & SHAP

Run Longformer scripts with PYTHONPATH=src:

export PYTHONPATH=src

# 1. Build book-split training/validation/test JSONL datasets
python -m longformer_pipeline.build_dataset \
    --gt-dir data/dataset/ground_truth \
    --interactions-dir data/dataset/interactions \
    --output-dir data/training \
    --texts-dir data/dataset/texts

# 2. Fine-tune Longformer Regressor (MSE or Pearson composite loss)
python -m longformer_pipeline.train \
    --train-path data/training/train.jsonl \
    --val-path   data/training/val.jsonl \
    --output-dir data/models/longformer_prevprefix_full_3ep_mse \
    --epochs 3 --batch-size 4 --learning-rate 2e-5

# 3. Run sequential chapter-level inference on a test book
python -m longformer_pipeline.inference \
    --book-id 110 \
    --interactions-dir data/dataset/interactions \
    --model-dir data/models/longformer_prevprefix_full_3ep_mse/best \
    --output-csv data/predictions/prevprefix_test_3ep_mse/110_predicted.csv

# 4. Compute evaluation metrics, comparison plots, and SHAP attributions
python -m longformer_pipeline.evaluate
python -m longformer_pipeline.explain_shap
python -m longformer_pipeline.plot_two_chapters_batch
python -m longformer_pipeline.plot_all_chapters_batch

---

Technical Notes: BookNLP Compatibility Patches

src/graph_builder.py transparently applies two runtime monkey-patches to booknlp==1.0.7.1:

1. Cross-platform path handling: Replaces hardcoded model_file.split("/")[-1] calls in entity_tagger.py, litbank_coref.py, and bert_qa.py with os.path.basename so HuggingFace model resolution works seamlessly on Windows as well as macOS/Linux.
2. Checkpoint buffer compatibility: Reloads BERT state dictionaries with strict=False so legacy bert.embeddings.position_ids buffers in upstream BookNLP checkpoints do not raise errors on newer transformers releases.

---

Citation & License

If you reference this project or report, please cite:

@techreport{aiello2026graphextraction,
  title       = {Graph Extraction from Books with Domain-Adapted Pretrained Models},
  author      = {Aiello, Tommaso and Bottardi, Alessandro and Ceria, Gregorio and Pasi, Niccol{\`o}},
  institution = {Bocconi University},
  year        = {2026}
}

Academic research project developed at Bocconi University (Course: Language Technology, 2025/2026). BookNLP © David Bamman et al.; LitBank, VADER, and Longformer are used under their respective open-source licenses.