Learning in Public: Machine Learning Zoomcamp - Week 4

Evaluation Metrics for Classification

Metric is a function that compares the predictions with the actual values and outputs a single number that tells how good the predictions are.

Inputs are predictions and true values. Outputs are a single number.

Different metrics look at different aspects of the predictions.

Data Preparation

This week's work used the same telecom churn data set as last week. We began this week by reimporting the data and going through the data preparation steps in a systematic manner.

After removing spaces and making everything lowercase, I set up the training, validation, and test groups. After the validation framework was set up, I deleted the target churn column, so that the data would not be trained on answers already there.

The Decision Threshold in Machine Learning

The decision threshold in machine learning is a crucial factor in classifying data into binary classes.

It is the value that determines how likely an instance is to be classified as a positive or negative class based on the model's output probabilities.

For example, in binary classification, a model may output a probability score between 0 and 1, and if the score exceeds the threshold, the instance is classified as positive; otherwise, it is classified as negative.

Issue with the Target Variable Being Wiped Out on Subsequent Runs

Just before training the model, I encountered a Python ValueError relating to the y_train data array. On inquiry, Gemini spotted that y_train was all zeros, and it should be ones and zeros, for binary classification. It also pointed out:

• Incorrect Target Column
• Target Leakage / Accidental Filtering
• Random State / Shuffling

None of these recommendations got rid of the ValueError, though it was helpful that Gemini pointed out that y_train was entirely zeros. When it became clear that of the target churn values had been set zero, I turned to Claude.

Claude had a difference of opinion, stating "Gemini’s suggestions about random_state and shuffling do not apply here. Your split code is correct; the target column was already all zeros before the split ran."

Claude then came back with the following useful advice: " On the first run, churn holds the strings 'yes' and 'no', and the line produces the correct 0s and 1s. If the cell runs again without df being reloaded from the CSV, churn already holds integers. No integer equals the string 'yes', so every comparison is False and the whole column becomes 0. Nothing raises an error at that point; the target is silently wiped."

The default threshold is often set to 0.5, but this value may not be optimal for all scenarios, especially in cases with imbalanced classes or where the cost of false negatives outweighs the cost of false positives.

ROC Curves

The Receiver Operating Characteristic (ROC) Curve is a graphical plot that illustrates the performance of a binary classifier model at varying threshold values. Though originally for two-class scenarios, it can be generalised to multiple classes.

The ROC Curve is a plot of the True Positive Rate versus the False Positive Rate across all classification thresholds. The curve visualises the trade-off between detecting positives and avoiding false alarms. It is used to evaluate the performance of binary classifiers.

Conclusion

At this stage of my journey into machine learning, it is clear to me that I am being hand held. Each one of these concepts is PhD level stuff, but to get to the end of the lesson I am being told what to do, and generally why it works, but all the mind-bending maths and academic heavy lifting is skipped.

Fair enough; but what it means is that I will need to circle back to each of these in depth, to know both how and why they work, and what to do when results are unexpected.