HospitalCancerClassification/HospitalCancerClassification.ipynb

680 lines
22 KiB
Text

{
"cells": [
{
"cell_type": "markdown",
"id": "c01ffe8d-3916-478f-a9c5-f3b480e5e293",
"metadata": {},
"source": [
"# Copyright Rayyan Hodges, TAFE NSW, Gelos Enterprises, Indigo Community Services and Health Hub © 2026"
]
},
{
"cell_type": "markdown",
"id": "b9f47b2c-6f01-479a-a87b-e698effc6edf",
"metadata": {},
"source": [
"# Contact: rayyan.hodges@studytafensw.edu.au, rayyan.hodges@gelosmail.com.au indigoCSHH@hmail.com"
]
},
{
"cell_type": "markdown",
"id": "c49336b7-8ec3-4a18-8590-4cd6c02917cb",
"metadata": {},
"source": [
"# Program Name: HopsitalCancerClassification.ipynb"
]
},
{
"cell_type": "markdown",
"id": "59691122-a537-4615-8a65-f408eb6fcffb",
"metadata": {},
"source": [
"# Purpose: To train and evaluate an ML model to automate the task of classifying clients as high, medium or low risk of developing cancer."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "1092545b-8be3-487b-87df-d8829e0a00eb",
"metadata": {},
"outputs": [],
"source": [
"# Import required libraries\n",
"import joblib\n",
"import numpy\n",
"import pandas as pd\n",
"import sklearn # importing the scikit-learn project, different name due to syntax rules \n",
"import scipy\n",
"import threadpoolctl"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "3a05014a-200c-4bc6-9667-568898547852",
"metadata": {},
"outputs": [],
"source": [
"# Set appropriate display options in Pandas\n",
"pd.set_option('display.max_rows', 10000)\n",
"pd.set_option('display.max_columns', 10000)\n",
"pd.set_option('display.width', 10000)"
]
},
{
"cell_type": "markdown",
"id": "95ffdbd6-b7e7-41b9-990f-932a05cae2e6",
"metadata": {},
"source": [
"# Identify any datapoints with missing labels and remove them"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "10d93e7f-9ba8-45a4-831d-7ba48046fea7",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.int64(10)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load the data set into memory and define it for further analysis.\n",
"df = pd.read_csv(\"data.csv\")\n",
"# Check for missing values in the target column, count them, and display them in a total number.\n",
"df['cancer_risk'].isnull().sum()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "3434f6bb-7cdb-4434-a948-1dc7d9e7d260",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"np.int64(0)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Remove the affected rows with missing data\n",
"df = df.dropna(subset=['cancer_risk'])\n",
"# Verify removal of said invalid data.\n",
"df['cancer_risk'].isnull().sum()"
]
},
{
"cell_type": "markdown",
"id": "44a9b1bc-0c6c-4e9e-95a7-03d3108b696b",
"metadata": {},
"source": [
"# Categorise ages as required."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "a8887198-0dbd-471d-bd1e-836d34def83a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>age_group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>51</td>\n",
" <td>45-54</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>92</td>\n",
" <td>85+</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>14</td>\n",
" <td>0-14</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>71</td>\n",
" <td>65-74</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>60</td>\n",
" <td>55-64</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age age_group\n",
"0 51 45-54\n",
"1 92 85+\n",
"2 14 0-14\n",
"3 71 65-74\n",
"4 60 55-64"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Define age bins and labels for the groups\n",
"# Bins meaning boundaries \n",
"bins = [0, 14, 24, 34, 44, 54, 64, 74, 84, 120]\n",
"labels = ['0-14', '15-24', '25-34', '35-44', \n",
" '45-54', '55-64', '65-74', '75-84', '85+']\n",
"\n",
"# Categorise ages into age groups\n",
"df['age_group'] = pd.cut(df['age'], bins=bins, labels=labels, right=True)\n",
"\n",
"# Display the results in a table\n",
"df[['age', 'age_group']].head()\n"
]
},
{
"cell_type": "markdown",
"id": "b2751d18-2580-453a-b0d9-70bc956f73cc",
"metadata": {},
"source": [
"# Standardise numerical features using appropriate scaling techniques."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "f9beff06-ce08-479d-b63d-56ca8aaf6338",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>bmi</th>\n",
" <th>blood_pressure</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>1.000000e+03</td>\n",
" <td>1.000000e+03</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>1.456613e-16</td>\n",
" <td>5.879741e-16</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>1.000500e+00</td>\n",
" <td>1.000500e+00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>-1.684915e+00</td>\n",
" <td>-1.703831e+00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>-8.885765e-01</td>\n",
" <td>-8.621512e-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>3.871853e-03</td>\n",
" <td>-1.077505e-02</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>8.860227e-01</td>\n",
" <td>8.687217e-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>1.733849e+00</td>\n",
" <td>1.775369e+00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" bmi blood_pressure\n",
"count 1.000000e+03 1.000000e+03\n",
"mean 1.456613e-16 5.879741e-16\n",
"std 1.000500e+00 1.000500e+00\n",
"min -1.684915e+00 -1.703831e+00\n",
"25% -8.885765e-01 -8.621512e-01\n",
"50% 3.871853e-03 -1.077505e-02\n",
"75% 8.860227e-01 8.687217e-01\n",
"max 1.733849e+00 1.775369e+00"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Import the scaling component from the SciKit Learning Library\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"# Select numerical features from the datasheet.\n",
"numerical_features = ['bmi', 'blood_pressure']\n",
"\n",
"# Initialise scaler component\n",
"scaler = StandardScaler()\n",
"\n",
"# Apply scaling and transformation to standardise the numbers.\n",
"df[numerical_features] = scaler.fit_transform(df[numerical_features])\n",
"\n",
"# Display results from the scaling and transformation.\n",
"df[numerical_features].describe()\n"
]
},
{
"cell_type": "markdown",
"id": "2616a37d-bffd-4a8d-b648-24673a543e7d",
"metadata": {},
"source": [
"# Encode categorical data using appropriate encoding techniques."
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "8bc14115-e855-4233-9058-eff0ce5c9386",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>age</th>\n",
" <th>family_history</th>\n",
" <th>alcohol_consumption</th>\n",
" <th>bmi</th>\n",
" <th>blood_pressure</th>\n",
" <th>cholesterol_level</th>\n",
" <th>travelled_overseas</th>\n",
" <th>number_of_children</th>\n",
" <th>cancer_risk</th>\n",
" <th>age_group_15-24</th>\n",
" <th>age_group_25-34</th>\n",
" <th>age_group_35-44</th>\n",
" <th>age_group_45-54</th>\n",
" <th>age_group_55-64</th>\n",
" <th>age_group_65-74</th>\n",
" <th>age_group_75-84</th>\n",
" <th>age_group_85+</th>\n",
" <th>smoking_history_former</th>\n",
" <th>smoking_history_never</th>\n",
" <th>diet_good</th>\n",
" <th>diet_poor</th>\n",
" <th>exercise_frequency_low</th>\n",
" <th>exercise_frequency_moderate</th>\n",
" <th>exercise_frequency_none</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>51</td>\n",
" <td>no</td>\n",
" <td>low</td>\n",
" <td>0.484421</td>\n",
" <td>1.631857</td>\n",
" <td>226.4</td>\n",
" <td>no</td>\n",
" <td>5</td>\n",
" <td>moderate</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>92</td>\n",
" <td>no</td>\n",
" <td>moderate</td>\n",
" <td>-1.492695</td>\n",
" <td>1.399135</td>\n",
" <td>170.9</td>\n",
" <td>yes</td>\n",
" <td>2</td>\n",
" <td>low</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>14</td>\n",
" <td>no</td>\n",
" <td>none</td>\n",
" <td>0.525611</td>\n",
" <td>0.569092</td>\n",
" <td>279.9</td>\n",
" <td>yes</td>\n",
" <td>4</td>\n",
" <td>high</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>71</td>\n",
" <td>yes</td>\n",
" <td>low</td>\n",
" <td>1.665199</td>\n",
" <td>-0.101925</td>\n",
" <td>280.7</td>\n",
" <td>yes</td>\n",
" <td>0</td>\n",
" <td>moderate</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>60</td>\n",
" <td>no</td>\n",
" <td>none</td>\n",
" <td>-1.451505</td>\n",
" <td>0.650544</td>\n",
" <td>212.8</td>\n",
" <td>yes</td>\n",
" <td>5</td>\n",
" <td>moderate</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" age family_history alcohol_consumption bmi blood_pressure cholesterol_level travelled_overseas number_of_children cancer_risk age_group_15-24 age_group_25-34 age_group_35-44 age_group_45-54 age_group_55-64 age_group_65-74 age_group_75-84 age_group_85+ smoking_history_former smoking_history_never diet_good diet_poor exercise_frequency_low exercise_frequency_moderate exercise_frequency_none\n",
"0 51 no low 0.484421 1.631857 226.4 no 5 moderate False False False True False False False False False True False True False False True\n",
"1 92 no moderate -1.492695 1.399135 170.9 yes 2 low False False False False False False False True False True False True False False False\n",
"2 14 no none 0.525611 0.569092 279.9 yes 4 high False False False False False False False False False False False False False False True\n",
"3 71 yes low 1.665199 -0.101925 280.7 yes 0 moderate False False False False False True False False True False False True False False False\n",
"4 60 no none -1.451505 0.650544 212.8 yes 5 moderate False False False False True False False False True False False False False False True"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Identify categorical features from the dataset file to be referenced.\n",
"categorical_features = ['age_group', 'smoking_history', 'diet', 'exercise_frequency']\n",
"\n",
"# Apply one-hot encoding to convert the text categories into dummy numerical categories (specifically true or false)\n",
"df_encoded = pd.get_dummies(df, columns=categorical_features, drop_first=True)\n",
"\n",
"# Print the results to ensure the categorical features have been converted to variables such as true or false correctly.\n",
"df_encoded.head()\n"
]
},
{
"cell_type": "markdown",
"id": "3b6926b7-31d0-4558-9695-c16570087104",
"metadata": {},
"source": [
"# Divide and randomise the dataset into training"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "a5174be9-c40b-457e-afd5-1ec97c70ed0a",
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"((693, 23), (148, 23), (149, 23))"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"# Remove rows with missing target labels (specifically, the 'cancer_risk' label, as it has missing data) \n",
"df_encoded = df_encoded.dropna(subset=['cancer_risk'])\n",
"\n",
"#Import the splitting model from the scikit-learn model.\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# Separate features and target\n",
"X = df_encoded.drop('cancer_risk', axis=1)\n",
"y = df_encoded['cancer_risk']\n",
"\n",
"# First split: 70% training, 30% temp\n",
"X_train, X_temp, y_train, y_temp = train_test_split(\n",
" X, y, test_size=0.30, random_state=42, stratify=y\n",
")\n",
"\n",
"# Second split: 15% validation, 15% test\n",
"X_val, X_test, y_val, y_test = train_test_split(\n",
" X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp\n",
")\n",
"\n",
"# Print the results of said sizes to ensure it is working correctly.\n",
"X_train.shape, X_val.shape, X_test.shape\n"
]
},
{
"cell_type": "markdown",
"id": "b4c21ba9-b730-4150-8b38-dfb5b00f87c1",
"metadata": {},
"source": [
"# Select the most important features in the training dataset that may influence cancer risk, dropping 2 unnecessary features. - Rayyan Hodges"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "c94e2161-e95d-4a84-864d-bc1388bd4d48",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(693, 21)"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"# Drop two unnecessary features/columns that aren't relevant to cancer risk evaluation and wouldn't affect if someone would be likely to have cancer.\n",
"X_train = X_train.drop(['travelled_overseas', 'number_of_children'], axis=1)\n",
"X_val = X_val.drop(['travelled_overseas', 'number_of_children'], axis=1)\n",
"X_test = X_test.drop(['travelled_overseas', 'number_of_children'], axis=1)\n",
"\n",
"# Print the results to allow the user to determine the changes have worked\n",
"X_train.shape\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c0707c69-dfd9-4536-a21a-39db9ec108e6",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}