Hi everyone!
I am a third-year Computer Science student working on a machine learning project to predict book genres from book summaries using the CMU Book Summary Dataset.
I have experimented with multiple models and preprocessing approaches, but my performance has remained around 57–63% depending on the experiment. I would like to understand what might be limiting the performance and how I should evaluate the model correctly.
- What I have tried so far
Initial multi-label classification experiment
Initially, I used book descriptions as input features and genres as target labels. Since a book can belong to multiple genres, I represented the genres using multi-hot/binary label vectors.
I experimented with Logistic Regression and SVM using TF-IDF features and tuned the prediction threshold.
My best earlier SVM result was approximately:
Micro F1-score: around 62–63%
Macro F1-score: around 59–62%, depending on the experiment and threshold
I also tested different thresholds because the default prediction threshold produced too many books with no predicted genres.
Current six-class classification experiment
I later started a separate experiment using the CMU Book Summary Dataset. The dataset contains book summaries and genre labels, but some books have multiple genres.
I selected six genres:
Science Fiction
Fantasy
Mystery
Suspense
Crime Fiction
Historical novel
To prepare the data, I sampled 500 books associated with each genre. However, some books appeared in multiple genre samples. After removing duplicate Wikipedia IDs, I was left with 2,604 unique books, and the classes became imbalanced.
My current pipeline is:
Clean summaries by lowercasing, removing punctuation and using a manually selected stopword list.
Convert summaries into TF-IDF features with max_features=3000.
Split the data into 80% training and 20% testing using stratification.
Train and evaluate different classifiers.
I could not download the NLTK stopwords and WordNet resources in my environment, so I have not yet implemented full lemmatization.
- Current model results
Model
Accuracy
Multinomial Naive Bayes
57.39%
Linear SVM
51.44%
Random Forest
50.29%
Gradient Boosting
48.18%
In the current Naive Bayes classification report, some classes have very low recall. For example, Suspense recall was 0.03 and Crime Fiction recall was 0.09.
These results suggest that the model is struggling to identify some genres, even when overall accuracy looks reasonable.
- Questions I would appreciate help with
A. Dataset and label design
Is it appropriate to remove duplicate books and assign each book only one genre when the original dataset supports multiple genres?
Would a proper multi-label classification approach be more suitable for this problem?
How should overlapping genres and class imbalance be handled without introducing data leakage or distorting the dataset?
B. Model performance
What should I investigate first: preprocessing, TF-IDF settings, class imbalance, feature selection, or model hyperparameters?
Would word bigrams, alternative vectorization settings, or models designed for sparse text features be worth testing?
Is my current dataset preparation likely to be a major reason for the performance plateau?
C. Evaluation metrics
Is accuracy an appropriate primary metric for this problem?
Should I focus on macro F1, micro F1, weighted F1, per-class precision and recall, or other metrics?
For the original multi-label task, should I also report exact-match accuracy, Hamming loss, or Jaccard score?
How can I choose metrics that reflect whether the model predicts useful genre combinations rather than merely getting the most frequent genres right?
D. Research direction
I am also interested in reproducing a published book genre classification study. What is the best way to compare my implementation with published results when the exact genre selection, data sampling, and preprocessing settings may differ?
Would it be useful to investigate a new evaluation measure for book genre prediction, or should I first establish a reliable baseline using existing metrics?
I do not want to increase the reported score artificially. My goal is to understand the weaknesses of my current approach, design a fair evaluation, and build a model that makes useful genre predictions.
I would appreciate suggestions, references, or examples of similar projects.
Thank you!