This tutorial explains how to use the manifestoberta large language model for automating the annotation of text data with the manifesto codebook. It shows how to use python for importing your text dataset, downloading the model from huggingface and clean the output into a dataset ready for further analyses. You also find some exemplary descriptive analyses for a closer look into how well the prediction performed on your data and how the categories are distributed.
Setup
First, you need to install the python libraries needed for running the model. Run this code only if you have not installed these libraries yet.
Next, load/import the libraries.
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import pandas as pd
import os
import numpy as np
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt
from docx import Document
import nltk
# Download the punkt tokenizer for sentence splitting
nltk.download('punkt')
nltk.download('punkt_tab')
from nltk.tokenize import sent_tokenizeSet your working directory. It should be the directory where your data set is stored.
Importing text data
First, you need to transform your text data (e.g. a word or pdf file) into a data set (e.g. .csv or .xlsx)- so it has to be split into sentences. Here, we take the example of the Labour Party’s manifesto of the 2024 election in the United Kingdom. The example document is in word/docx format.
Importantly, if you use manifestos for substantial analyses, they might need some more preprocessing than we do here. Our usual routines in the Manifesto Project include manual tagging of which parts of the manifesto are not to be coded (such as headings, tables of content, text in the margins). You can either perform this manually beforehand as well, if you have word documents. Or, if you have a large set of pdf data, you can also opt for a more automated approach after the sentence splitting, i.e. removing tables of contents via regular expressions or removing sentences with less then three words etc.
For looking at our example manifesto of Labour 2024, we first write a function to read a Word document and extract text.
def read_word_file(file_path):
doc = Document(file_path)
full_text = []
for para in doc.paragraphs:
full_text.append(para.text)
return '\n'.join(full_text)Next, we write a function to split the text into sentences and create a DataFrame.
def text_to_dataframe(text):
sentences = sent_tokenize(text)
df = pd.DataFrame({'id': range(1, len(sentences) + 1), 'text': sentences})
return dfWe can then define the file path to our text file, read it with our abovedefined function and split it into a dataframe with sentences as rows.
# Path to your Word file
file_path = '51320_2024.docx'
# Read the Word file
text = read_word_file(file_path)
# Convert text to DataFrame
df = text_to_dataframe(text)
# Resetting the index to make sure it is integer-based
df.reset_index(drop=True, inplace=True)The dataset should at least have a) a running ID variable, b) a variable containing the text to be classified, e.g. split into sentences.
Load Model from Huggingface
There are two manifestoberta models ready for use. The sentence model classifies statements into one of the 56 different substantial categories available in the Handbook 4 coding scheme. The context model variant additionally incorporates the surrounding sentences of a statement to improve the classification results compared to the sentence model version. The context model is superior in performance. However, its use is only advised when there is context to a text, e.g. when the texts to be classified are sentences or paragraphs split from larger texts. When the texts are standalone, such as social media posts, the sentence model is the preferred choice. Here, we download the context model from Hugging Face, because we want to classify a manifesto - so sentences do have a context: their surrounding sentences.
model = AutoModelForSequenceClassification.from_pretrained("manifesto-project/manifestoberta-xlm-roberta-56policy-topics-context-2025-1-1", trust_remote_code = True)Then, we import the AutoTokenizer class from the Hugging Face Transformers library. The tokenizer is an essential component in natural language processing as it converts raw text into a format that the model can process.
We use the from_pretrained method to load a pre-trained tokenizer specifically for the “xlm-roberta-large” model. By loading this tokenizer, we ensure that the text is tokenized in a way that is consistent with how the model was trained.
Next, we have to check whether a GPU is available for the model to run, and tell it to use a CPU otherwise:
Define a function to get top n (here: top 3) classes and probabilities for each class
Classes, for manifestoberta, are the 56 manifesto categories. For each text in your dataset, the model will calculate a probability with which it can be assigned to each of these categories. The “top 3 classes” then are the three most likely categories the model predicts for your text.
def get_top_classes(logits, top_n=3):
probabilities = torch.softmax(logits, dim=1)[0].tolist()
classes_and_probs = {model.config.id2label[index]: round(probability * 100, 2) for index, probability in enumerate(probabilities)}
top_classes = dict(sorted(classes_and_probs.items(), key=lambda item: item[1], reverse=True)[:top_n])
return top_classesRun manifestoberta on your dataset: The context model
At first, we create two empty lists, which we fill gradually with our predictions for each texts. To use the context model correctly, we first have to preprocess our text data. The function we define for that makes sure that a window of 200/300 (depending on the model version) tokens is correctly constructed around the focus sentence to be classified. Context is created in equal parts from the previous and following sentences, gradually filling until 200 or 300 tokens are reached.
In the next step, the preprocessed inputs are passed to the model. The model processes these inputs and outputs logits, which are raw prediction scores for each possible class. The logits represent the unnormalized probabilities of each class. Then, we extract the top predicted classes from the logits and format them into a comma-separated string. The same is done for the associated probabilities, which are converted into percentage strings. These formatted strings are then appended to lists that store all the predicted classes and their probabilities for later use.
Preprocessing steps for the model to handle the context window
# Define function to build context inputs for manifestoberta classification
def build_sentencepair_inputs_from_texts(df, tokenizer,
max_length_sentence: int,
max_length_paragraph: int,
is_roberta: bool = True):
"""
Build sentence-pair inputs in the same format as used during training.
Returns:
input_ids_list: list of padded input-id sequences
attention_mask_list: list of corresponding attention masks
Input format for RoBERTa-style models:
[CLS] focus_sentence [SEP] [SEP] context [SEP]
Input format for non-RoBERTa-style models:
[CLS] focus_sentence [SEP] context [SEP]
The context is built from previous and following sentences.
"""
# Get special token ids from the tokenizer.
# For roberta, cls_token_id is usually <s>, sep_token_id is usually </s>.
cls_token = getattr(tokenizer, "cls_token_id", None) or getattr(tokenizer, "bos_token_id", None)
sep_token = getattr(tokenizer, "sep_token_id", None) or getattr(tokenizer, "eos_token_id", None)
pad_token = getattr(tokenizer, "pad_token_id", tokenizer.eos_token_id)
# 1. Tokenize all sentences without adding special tokens
tokenized_sentences = []
for txt in df['text'].fillna("").astype(str).tolist():
enc = tokenizer(txt, add_special_tokens=False, truncation=False)
tokenized_sentences.append(enc["input_ids"])
n = len(tokenized_sentences)
# 2. Helper function to build context tokens for one sentence
def build_context_tokens(i, max_ctx_tokens):
"""
Build a context window around sentence i.
The function first adds previous sentences, then following sentences,
until the maximum context token budget is reached.
"""
ctx_tokens = []
# Add previous sentences, starting from the nearest one.
for j in range(i-1, -1, -1):
s = tokenized_sentences[j]
# If the full sentence fits, prepend it.
if len(ctx_tokens) + len(s) <= max_ctx_tokens:
ctx_tokens = s + ctx_tokens
# If it does not fit, take only the last tokens of that sentence.
else:
remaining = max_ctx_tokens - len(ctx_tokens)
if remaining > 0:
ctx_tokens = s[-remaining:] + ctx_tokens
break
# Add following sentences, starting from the nearest one.
for j in range(i+1, n):
s = tokenized_sentences[j]
# If the full sentence fits, append it.
if len(ctx_tokens) + len(s) <= max_ctx_tokens:
ctx_tokens = ctx_tokens + s
# If it does not fit, take only the first tokens of that sentence.
else:
remaining = max_ctx_tokens - len(ctx_tokens)
if remaining > 0:
ctx_tokens = ctx_tokens + s[:remaining]
break
return ctx_tokens
# 3. Define token budgets
# One token is reserved from the paragraph budget because of special-token handling.
max_len_paragraph = max_length_paragraph - 1
# roberta-style sentence-pair inputs need three special tokens around the focus sentence:
# [CLS] sentence [SEP] [SEP] context [SEP]
# Non-roberta models usually use:
# [CLS] sentence [SEP] context [SEP]
max_len_sentence = (max_length_sentence - 3) if is_roberta else (max_length_sentence - 2)
max_length_overall = (max_len_sentence + max_len_paragraph + 4) if is_roberta else (max_len_sentence + max_len_paragraph + 3)
# 4. Build final padded input sequences and attention masks
input_ids_list = []
attention_mask_list = []
for i in range(n):
# Truncate the focus sentence to its token budget.
focus = tokenized_sentences[i][:max_len_sentence]
# Build context from surrounding sentences.
context_tokens = build_context_tokens(i, max_len_paragraph)
# Construct the final model input.
if is_roberta:
# roberta sentence pair
# [CLS] focus [SEP] [SEP] context [SEP]
final = []
if cls_token is not None:
final.append(cls_token)
final += focus + ([sep_token] if sep_token is not None else []) + ([sep_token] if sep_token is not None else []) + context_tokens + ([sep_token] if sep_token is not None else [])
else:
# Standard sentence pair:
# [CLS] focus [SEP] context [SEP]
final = []
if cls_token is not None:
final.append(cls_token)
final += focus + ([sep_token] if sep_token is not None else []) + context_tokens + ([sep_token] if sep_token is not None else [])
# Truncate in case the sequence is still too long.
if len(final) > max_length_overall:
final = final[:max_length_overall]
# Pad sequence to the required maximum length.
pad_len = max_length_overall - len(final)
final_padded = final + [pad_token] * pad_len
mask = [1 if tok != pad_token else 0 for tok in final_padded]
input_ids_list.append(final_padded)
attention_mask_list.append(mask)
return input_ids_list, attention_mask_list# Define input length parameters
# Adjust parameters depending on the model version
# (e.g., 2024-1 uses max_length_paragraph = 200 but 2025-1 uses 300, see Hugging Face model card)
max_length_sentence = 100
max_length_paragraph = 300
is_roberta = True
# Create input_ids and attention masks for all rows in df
input_ids_list, attention_mask_list = build_sentencepair_inputs_from_texts(
df,
tokenizer,
max_length_sentence=max_length_sentence,
max_length_paragraph=max_length_paragraph,
is_roberta=is_roberta
)# Define inference function
logits_list = []
predicted_classes = []
predicted_probabilities = []
# Set model to evaluation mode.
# This disables dropout and other training-specific behavior.
model.eval()
# Prepare id-to-label mapping
if hasattr(model.config, "id2label") and model.config.id2label:
try:
id2label = {int(k): v for k, v in model.config.id2label.items()}
except Exception:
id2label = model.config.id2label
else:
id2label = None
def get_top_classes(logits, top_k=3):
"""
Convert model logits into probabilities and return the top-k classes.
Args:
logits: raw model output of shape [1, number_of_classes]
top_k: number of top classes to return
Returns:
Dictionary with class labels as keys and probabilities in percent as values.
Example:
{
"per504": 72.31,
"per503": 14.88,
"per506": 5.21
}
"""
# Convert logits to probabilities.
probs = torch.softmax(logits, dim=-1)[0]
# Get the top-k probabilities and their class indices.
top_probs, top_indices = torch.topk(probs, k=top_k)
top_classes = {}
for prob, idx in zip(
top_probs.detach().cpu().tolist(),
top_indices.detach().cpu().tolist()
):
# Convert class index to class label if id2label exists
if id2label is not None:
label = id2label[int(idx)]
else:
label = str(int(idx))
# Store probability as percentage.
top_classes[label] = round(prob * 100, 2)
return top_classes# Loop through all rows and classify each sentence
for pos, (index, row) in enumerate(df.iterrows()):
# Use pos instead of index because DataFrame indices are not always 0, 1, 2, ...
# input_ids_list and attention_mask_list are normal Python lists, so they need positional indexing
final_padded = input_ids_list[pos]
attention_mask = attention_mask_list[pos]
# Convert input_ids and attention_mask to PyTorch tensors
# The additional brackets create a batch dimension: [sequence_length] -> [1, sequence_length]
input_ids_tensor = torch.tensor(
[final_padded],
dtype=torch.long
).to(device)
attention_mask_tensor = torch.tensor(
[attention_mask],
dtype=torch.long
).to(device)
# Create the input dictionary expected by Hugging Face models
inputs = {
"input_ids": input_ids_tensor,
"attention_mask": attention_mask_tensor
}
# Run model inference
# torch.no_grad() disables gradient calculation and saves memory
with torch.no_grad():
logits = model(**inputs).logits
# Optional: store raw logits for later analysis
logits_list.append(logits.detach().cpu().numpy()[0])
# Get top-3 classes and probabilities
top_classes = get_top_classes(logits, top_k=3)
# Convert top classes and probabilities to comma-separated strings
top_classes_str = ', '.join(top_classes.keys())
top_probs_str = ', '.join([str(prob) + '%' for prob in top_classes.values()])
# Store results
predicted_classes.append(top_classes_str)
predicted_probabilities.append(top_probs_str)Then, we can add the predicted classes and probabilities to the DataFrame.
Observe dataset classified by the context model
After successfully having run the context model on our Labour 2024 manifesto, now, we can observe the dataset with the newly created columns:
Clean Output
For further working with the model output, it is helpful to create three separate columns from the predicted classes and their probabilities (and convert the latter to numeric values).
# Creating a DataFrame
df = pd.DataFrame(df)
# Separate "Predicted Classes" into "class_1", "class_2", "class_3"
df[['class_1', 'class_2', 'class_3']] = df['Predicted Classes'].str.split(', ', expand=True)
# Separate "Predicted Probabilities" into "prob_class_1", "prob_class_2", "prob_class_3"
df[['prob_class_1', 'prob_class_2', 'prob_class_3']] = df['Predicted Probabilities'].str.split(', ', expand=True)
# Remove '%' and convert to numeric
df[['prob_class_1', 'prob_class_2', 'prob_class_3']] = df[['prob_class_1', 'prob_class_2', 'prob_class_3']].map(lambda x: float(x.replace('%', '')))Examine predicted classes and probabilities
Next, you can examine how the three predicted classes and their probabilities are distributed in your data.
# Calculate some descriptive statistics
# Function to calculate confidence intervals
def confidence_interval(data, confidence=0.95):
n = len(data)
mean = np.mean(data)
sem = stats.sem(data) # Standard error of the mean
margin = sem * stats.t.ppf((1 + confidence) / 2., n - 1)
return mean - margin, mean + margin
# Descriptive statistics
descriptives = {
"mean_prob_class_1": df['prob_class_1'].mean(),
"median_prob_class_1": df['prob_class_1'].median(),
"min_prob_class_1": df['prob_class_1'].min(),
"max_prob_class_1": df['prob_class_1'].max(),
"sd_prob_class_1": df['prob_class_1'].std(),
"ci_lower_prob_class_1": confidence_interval(df['prob_class_1'])[0],
"ci_upper_prob_class_1": confidence_interval(df['prob_class_1'])[1],
"mean_prob_class_2": df['prob_class_2'].mean(),
"median_prob_class_2": df['prob_class_2'].median(),
"min_prob_class_2": df['prob_class_2'].min(),
"max_prob_class_2": df['prob_class_2'].max(),
"sd_prob_class_2": df['prob_class_2'].std(),
"ci_lower_prob_class_2": confidence_interval(df['prob_class_2'])[0],
"ci_upper_prob_class_2": confidence_interval(df['prob_class_2'])[1],
"mean_prob_class_3": df['prob_class_3'].mean(),
"median_prob_class_3": df['prob_class_3'].median(),
"min_prob_class_3": df['prob_class_3'].min(),
"max_prob_class_3": df['prob_class_3'].max(),
"sd_prob_class_3": df['prob_class_3'].std(),
"ci_lower_prob_class_3": confidence_interval(df['prob_class_3'])[0],
"ci_upper_prob_class_3": confidence_interval(df['prob_class_3'])[1],
"n_total": len(df)
}Likewise, we can calculate the distribution for each manifesto category. Here, we take the category with the highest probability (top-1 class) as an example.
# Group by 'class_1' and calculate the statistics
grouped = df.groupby('class_1').agg(
mean_prob_class_1=('prob_class_1', 'mean'),
median_prob_class_1=('prob_class_1', 'median'),
min_prob_class_1=('prob_class_1', 'min'),
max_prob_class_1=('prob_class_1', 'max'),
sd_prob_class_1=('prob_class_1', 'std'),
n=('prob_class_1', 'count')
).reset_index()
# Calculate confidence intervals if n >= 3, else set to NaN
grouped['ci_lower_prob_class_1'] = grouped.apply(
lambda row: confidence_interval(df[df['class_1'] == row['class_1']]['prob_class_1'])[0] if row['n'] >= 3 else np.nan,
axis=1
)
grouped['ci_upper_prob_class_1'] = grouped.apply(
lambda row: confidence_interval(df[df['class_1'] == row['class_1']]['prob_class_1'])[1] if row['n'] >= 3 else np.nan,
axis=1
)Heat map
You can use these to create a heat map of the predicted probabilities for class 1. We therefore reshape the data frame into long format first.
descriptives_by_category_long = grouped.melt(
id_vars=['class_1'],
value_vars=['mean_prob_class_1', 'ci_lower_prob_class_1', 'ci_upper_prob_class_1'],
var_name='Statistic',
value_name='Value'
)
# Map labels
statistic_labels = {
'mean_prob_class_1': 'Mean',
'ci_lower_prob_class_1': 'Lower CI',
'ci_upper_prob_class_1': 'Upper CI'
}
# Pivot the DataFrame for the heatmap
pivot_table = descriptives_by_category_long.pivot(index='class_1', columns='Statistic', values='Value')
# Plotting the heatmap
plt.figure(figsize=(22, 10))## <Figure size 2200x1000 with 0 Axes>heatmap = sns.heatmap(
pivot_table,
annot=True,
fmt=".2f",
cmap=sns.color_palette("YlGnBu", as_cmap=True),
cbar_kws={'label': 'Value'}
)
# Customizing the plot
heatmap.set_xticklabels([statistic_labels.get(label.get_text(), label.get_text()) for label in heatmap.get_xticklabels()])## [Text(0.5, 0, 'Lower CI'), Text(1.5, 0, 'Upper CI'), Text(2.5, 0, 'Mean')]## (array([0.5, 1.5, 2.5]), [Text(0.5, 0, 'Lower CI'), Text(1.5, 0, 'Upper CI'), Text(2.5, 0, 'Mean')])## (array([ 0.5, 1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5, 9.5, 10.5,
## 11.5, 12.5, 13.5, 14.5, 15.5, 16.5, 17.5, 18.5, 19.5, 20.5, 21.5,
## 22.5, 23.5, 24.5, 25.5, 26.5, 27.5, 28.5, 29.5, 30.5, 31.5, 32.5,
## 33.5, 34.5, 35.5, 36.5, 37.5, 38.5, 39.5, 40.5]), [Text(0, 0.5, '101 - Foreign Special Relationships: Positive'), Text(0, 1.5, '104 - Military: Positive'), Text(0, 2.5, '105 - Military: Negative'), Text(0, 3.5, '106 - Peace'), Text(0, 4.5, '107 - Internationalism: Positive'), Text(0, 5.5, '108 - European Community/Union or Latin America Integration: Positive'), Text(0, 6.5, '109 - Internationalism: Negative'), Text(0, 7.5, '110 - European Community/Union or Latin America Integration: Negative'), Text(0, 8.5, '201 - Freedom and Human Rights'), Text(0, 9.5, '202 - Democracy'), Text(0, 10.5, '204 - Constitutionalism: Negative'), Text(0, 11.5, '301 - Decentralisation: Positive'), Text(0, 12.5, '303 - Governmental and Administrative Efficiency'), Text(0, 13.5, '304 - Political Corruption'), Text(0, 14.5, '305 - Political Authority'), Text(0, 15.5, '402 - Incentives: Positive'), Text(0, 16.5, '403 - Market Regulation'), Text(0, 17.5, '404 - Economic Planning'), Text(0, 18.5, '405 - Corporatism/ Mixed Economy'), Text(0, 19.5, '406 - Protectionism: Positive'), Text(0, 20.5, '407 - Protectionism: Negative'), Text(0, 21.5, '410 - Economic Growth: Positive'), Text(0, 22.5, '411 - Technology and Infrastructure: Positive'), Text(0, 23.5, '412 - Controlled Economy'), Text(0, 24.5, '413 - Nationalisation'), Text(0, 25.5, '414 - Economic Orthodoxy'), Text(0, 26.5, '415 - Marxist Analysis: Positive'), Text(0, 27.5, '416 - Anti-Growth Economy and Sustainability'), Text(0, 28.5, '501 - Environmental Protection'), Text(0, 29.5, '502 - Culture: Positive'), Text(0, 30.5, '503 - Equality: Positive'), Text(0, 31.5, '504 - Welfare State Expansion'), Text(0, 32.5, '506 - Education Expansion'), Text(0, 33.5, '601 - National Way of Life: Positive'), Text(0, 34.5, '602 - National Way of Life: Negative'), Text(0, 35.5, '603 - Traditional Morality: Positive'), Text(0, 36.5, '605 - Law and Order'), Text(0, 37.5, '606 - Civic Mindedness: Positive'), Text(0, 38.5, '701 - Labour Groups: Positive'), Text(0, 39.5, '703 - Agriculture and Farmers'), Text(0, 40.5, '704 - Middle Class and Professional Groups')])## Text(0.5, 80.5815972222222, '')## Text(245.7222222222222, 0.5, '')## Text(0.5, 1.0, 'Predicted Probabilities for Class 1')
Check difference of probabilities
Additionally, you can check how close the probabilities of the three predicted classes are. The more similar the probabilities, the less reliable the prediction. Therefore, we define a function that checks if the difference between the first predicted class (i.e., class with the highest probability) and the third predicted class (i.e., class with the lowest probability) is smaller than some threshold you define (here, we take a threshold of 6 percentage points as an example).
Aggregate sentence-per-sentence codings to saliences
If you are interested “filling” missing data points of the Manifesto Dataset, or calculate salience in general, of course you need a measure of the percentage that each category makes up of all sentences (Note: Sentences, not quasi-sentences here!). For that, we need to count the sentences per category and divide them each by the total number of sentences. We therefore take the class_1 variable, so the category manifestoberta considers the most likely, as the basis. If you add it to existing manifesto data, remember that you need to add the meta information (country, date, etc.) on the data points you created with manifestoberta.
# Calculate the total number of rows
total_rows = len(df)
# Group by 'class_1' and calculate the relative frequency
relative_freq = df['class_1'].value_counts() / total_rows*100
# Convert the series to a DataFrame with one row
relative_freq_df = pd.DataFrame([relative_freq]).reset_index(drop=True)
# Adjust column names to Marpor logic
relative_freq_df.columns = [
f"per{col.split('-')[0].strip()}" if ' - ' in col else f"per{col.strip()}"
for col in relative_freq_df.columns
]We then have a dataset similar to the logic of the manifesto dataset:
Save Output
Lastly, you can save the resulting dataset in any format you like, for instance as a csv file.
Run manifestoberta on your dataset: The sentence model
If you prefer to run a sentence model, the steps would be very similar to the ones above, with slight modifications.
Loading the sentence model and a tokenizer:
model = AutoModelForSequenceClassification.from_pretrained("manifesto-project/manifestoberta-xlm-roberta-56policy-topics-sentence-2025-1-1")
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")Define the top_n classes function in the same way as for the context model:
def get_top_classes(logits, top_n=3):
probabilities = torch.softmax(logits, dim=1)[0].tolist()
classes_and_probs = {model.config.id2label[index]: round(probability * 100, 2) for index, probability in enumerate(probabilities)}
top_classes = dict(sorted(classes_and_probs.items(), key=lambda item: item[1], reverse=True)[:top_n])
return top_classesAnd lastly, run the sentence model (including tokenizing and appending the logits as described above):
predicted_classes = []
predicted_probabilities = []
for index, row in df.iterrows():
text_input = row['text'] # name of your text variable
inputs = tokenizer(text_input,
return_tensors="pt",
max_length=200,
padding="max_length",
truncation=True
)
logits = model(**inputs).logits
top_classes = get_top_classes(logits)
# Get top classes and probabilities
top_classes_str = ', '.join(top_classes.keys())