16 NOV 2026
Provide machine learning predictions relating to jobs and employment in technology and computers utilising a large set of survey data.
The model predicts the annual salary of a professional developer in the UK or the EU from their profile (country, experience, role, organisation size, languages, and so on), using the Stack Overflow Developer Survey for 2023, 2024 and 2025.
The problem is that job seekers are generally distressed, being under both time and financial pressures to find a job. Job seekers are also the target of a great deal of marketing, which offers "common sense" jobseeker advice which may or may not be objectively true.
This project seeks to address that problem by providing a clear and objective method for evaluating jobs and jobseekers. Having such a tool, jobseekers can make a more well informed determination regarding their own prospects in the job market.
Target: log_comp, the logarithm of ConvertedCompYearly (annual compensation in USD). Metric: RMSE on the log scale (lower is better), with MAE in USD as a more readable companion.
Install the dependencies first with pip install -r requirements.txt (see the README). The survey files are downloaded once and reused on later runs.
[[[Statement and screenshots]]]
Three data sets into one. The questions and answer options change from year to year, so harmonise.py maps each year onto one shared set of columns:
It keeps respondents in the EU-27 and the United Kingdom.
It maps DevType labels from all three years onto one shared role list.
It reduces the multi-select Employment field to a single category.
It converts year values like 'Less than 1 year' and 'More than 50 years' to numbers.
It turns the 20 languages common to all three years into binary lang_* columns.
The model is for working developers, so we keep professional developers who are employed or self-employed and who report a plausible annual salary (USD 5,000 to 500,000). Salaries are strongly right-skewed, so the target is the logarithm of the salary.
The regression equivalent of the lecture's "risk ratio" tables: each group's median salary relative to the overall median.
Mutual information tends to favour high-cardinality features such as Country.
A quick, untuned gradient-boosting model, used only to rank features. It is not one of the candidate models.
The probe is trained and scored inside the training portion of the data. The same random_state=1 splits are used in section 5, so the probe never sees the test set that the final model is scored on.
A note on the "Re-encoding the categorical variables" stage from Week 6 In the credit-scoring dataset, categorical columns such as status, home, marital, records, and job are stored as integer codes (1 = rent, 2 = owner, and so on). The re-encoding step uses .map() to turn those codes back into readable strings. That matters for two reasons. DictVectorizer() one-hot encodes string values but passes numbers through unchanged. A model would also wrongly read marital = 4 as being "greater than" marital = 2.
The Stack Overflow survey data never had this problem. Country, EdLevel, OrgSize, RemoteWork and the others are stored as strings from the start, and harmonise.py has already done the more demanding version of this work (section 3). So the step has, in effect, already been done.
Lower-case and replace spaces with underscores in column names and in all text values. This must be applied after harmonise(), and identically in predict.py.
ConvertedCompYearly is the untransformed target, so leaving it in the features would let the model cheat, just as leaving status in would in Week 6. ResponseId and the columns used only for filtering must also stay out. Selecting only the features list in prepare() below guarantees this.
The Stack Overflow data uses real NaNs rather than placeholder numbers such as Week 6's 99999999, so there is nothing to translate, but we do need to decide how to fill them. The course pattern of fillna(0) works for the numeric columns. For categorical columns, filling with 'unknown' is clearer and avoids mixing numbers into text columns, which pandas 3 also warns against.
The feature matrix must have no missing values and must not contain the target or the response ID.
The Week 6 credit-scoring target has two classes, ok and default, which makes it a classification problem. Here, the target log_comp is a continuous quantity, which makes it a regression problem. The changes from the Week 6 notebook are:
| Week 6 classification | This project: regression |
| --- | --- |
| DecisionTreeClassifier, RandomForestClassifier | DecisionTreeRegressor, RandomForestRegressor |
| predict_proba(X)[:, 1] | predict(X) |
| roc_auc_score (higher is better) | RMSE on the log scale, plus MAE in USD (lower is better) |
| Gini or entropy impurity | Squared error (variance) as the impurity |
The rest of the Week 6 method carries over unchanged: comparing training and validation scores to detect overfitting, tuning max_depth and min_samples_leaf, and reading the tree with export_text.
A model that predicts the training mean for every row. Any useful model must beat this.
Linear regression with regularisation, as in Weeks 2 and 3, giving an interpretable linear model. We tune the regularisation strength alpha.
An unrestricted tree memorises the training data, so we expect severe overfitting.
We tune max_depth and min_samples_leaf together, as in the coursework.
The top of the tuned tree shows which features it splits on first.
An ensemble of decision trees, as in the rest of Week 6. Following the coursework, we first tune max_depth together with the number of trees, then tune min_samples_leaf.
Each model is shown with its best tuned parameters, scored on the same validation set. The test set has not been used yet.
Selection rationale: Decision made based on metrics given.
Following the Week 6 recipe: combine the training and validation data into df_full_train, fit a new DictVectorizer, retrain the selected model with its tuned parameters, and score it once on the test set.
If the test RMSE is close to the validation RMSE, the model has generalised: it learned patterns that hold on data it has never seen.
The training logic from this section is exported to train.py, which saves dv and final_model for the web service in predict.py.