F1 Score

The F1 score is the harmonic mean of precision and recall, giving a single metric that balances false positives and false negatives. It ranges from 0 to 1, where 1 is perfect precision and recall. Operators encounter F1 when evaluating classification models, especially on imbalanced datasets where accuracy is misleading. For example, a model that always predicts the majority class may have high accuracy but F1 near 0. F1 is commonly reported in benchmarks for tasks like text classification or named entity recognition.

When fine-tuning a BERT model for sentiment analysis on a dataset with 90% positive and 10% negative reviews, accuracy might be 90% if the model predicts positive always. F1 score for the negative class would be 0. A good model should have F1 > 0.8 for both classes. In practice, operators monitor F1 during training via validation splits.

In Hugging Face Transformers, after training a classifier, you compute F1 using from sklearn.metrics import f1_score on predictions vs labels. In llama.cpp, when evaluating a model on a classification task, you might calculate F1 from the output logits. In Ollama, you can script evaluation by sending prompts and comparing responses to ground truth.