DECISION TREES for Risk Assessment One of the great advantages of decision trees is their interpretability. The rules learnt for classification are easy for a person to follow, unli

  

DECISION TREES for Risk Assessment

One of the great advantages of decision trees is their interpretability. The rules learnt for classification are easy for a person to follow, unlike the opaque "black box" of many other methods, such as neural networks. We demonstrate the utility of this using a German credit data set. You can read a description of this dataset at the UCI site. The task is to predict whether a loan approval is good or bad credit risk based on 20 attributes. We've simplified the data set somewhat, particularly making attribute names and values more meaningful.

1. Download the credit_Dataset.arff dataset and load it to Weka.

2. (5 Points) When presented with a dataset, it is usually a good idea to visualise it first. Go to the Visualise tab. Click on any of the scatter plots to open a new window which shows the scatter plot for two selected attributes. Try visualising a scatter plot of age and duration. Do you notice anything unusual? You can click on any data point to display all it's values.

3. (5 Points) In the previous point you should have found a data point, which seems to be corrupted, as some of its values are nonsensical. Even a single point like this can significantly affect the performance of a classifier. How do you think it would affect Decision trees? A good way to check this is to test the performance of each classifier before and after removing this datapoint.

4. (10 Points) To remove this instance from the dataset we will use a filter. We want to remove all instances, where the age of an applicant is lower than 0 years, as this suggests that the instance is corrupted. In the Preprocess tab click on Choose in the Filter pane. Select filters > unsupervised > instance > RemoveWithValues. Click on the text of this filter to change the parameters. Set the attribute index to 13 (Age) and set the split point at 0. Click Ok to set the parameters and Apply to apply the filter to the data. Visualise the data again to verify that the invalid data point was removed.

5. (20 Points) On the Classify tab, select the Percentage split test option and change its value to 90%. This way, we will train the classifiers using 90% of the training data and evaluate their performance on the remaining 10%. First, train a decision tree classifier with default options. Select classifiers > trees > J48 and click Start. J48 is the Weka implementation of the C4.5 algorithm, which uses the normalized information gain criterion to build a decision tree for classification.

6. (20 Points) After training the classifier, the full decision tree is output for your perusal; you may need to scroll up for this. The tree may also be viewed in graphical form by right-clicking in the Result list and selecting Visualize tree; unfortunately this format is very cluttered for large trees. Such a tree accentuates one of the strengths of decision tree algorithms: they produce classifiers which are understandable to humans. This can be an important asset in real life applications (people are seldom prepared to do what a computer program tells them if there is no clear explanation). Observe the output of the classifier and try to answer the following questions:

o How would you assess the performance of the classifier? Is the Percentage of Correctly Classified Instances a sufficient measure in this case? Why? Hint: check the number of good and bad cases in the test sample, using the confusion matrix. Each column of the matrix represents the instances in a predicted class, while each row represents the instances in an actual class. For example let us define an experiment from P positive instances and N negative instances. The four outcomes can be formulated in a 2 by 2 contingency table or confusion matrix. One benefit of a confusion matrix is that it is easy to see if the system is confusing two classes (i.e. commonly mislabeling one as another).

o Looking at the decision tree itself, are the rules it applies sensible? Are there any branches which appear absurd? At what depth of the tree? What does this suggest?
Hint: Check the rules applied after following the paths: (a) CheckingAccount = <0, Foreign = yes, Duration >11, Job = skilled, OtherDebtors = none, Duration <= 30 and (b) CheckingAccount = <0, Foreign = yes, Duration >11, Job = unskilled.

o How does the decision tree deal with classification in the case where there are zero instances in the training set corresponding to that particular path in the tree (e.g. those leaf nodes that have (0:0))?

7. (20 Points) Now, explore the effect of the confidenceFactor option. You can find this by clicking on the Classifer name (to the right of the Choose button on the Classify tab). On the Classifier options window, click on the More button to find out what the confidence factor controls. Try the values 0.1, 0.2, 0.3 and 0.5. What is the performance of the classifier at each case? Did you expect this given your observations in the previous questions? Why do you think this happens?

8. (20 Points) Suppose that it is worse to classify a customer as good when they are bad, than it is to classify a customer as bad when they are good. Which value would you pick for the confidence factor? Which performance measure would you base your decision on?

9.  (20 Points)Finally we will create a random decision forest and compare the performance of this classifier to that of the decision tree and the decision stump. The random decision forest is an ensemble classifier that consists of many decision trees and outputs the class that is the mode of the class's output by individual trees. Again set the test option Percentage split to 90%. Select classifiers > trees > RandomForest and hit Start. Again, observe the output. How high can you get the performance of the classifier by changing the number of trees (numTrees) parameter? How does the random decision forest compare performance wise to the decision tree and decision stump?

Assignment

(100 Points)

CYBR 7240 – Cyber Analytics and Intelligence Module 03: Decision Trees

DECISION TREES for Risk Assessment

One of the great advantages of decision trees is their interpretability. The rules learnt for classification are easy for a person to follow, unlike the opaque "black box" of many other methods, such as neural networks. We demonstrate the utility of this using a German credit data set. You can read a description of this dataset at the UCI site. The task is to predict whether a loan approval is good or bad credit risk based on 20 attributes. We've simplified the data set somewhat, particularly making attribute names and values more meaningful.

1. Download the credit_Dataset.arff dataset and load it to Weka.

2. (5 Points) When presented with a dataset, it is usually a good idea to visualise it first. Go to the Visualise tab. Click on any of the scatter plots to open a new window which shows the scatter plot for two selected attributes. Try visualising a scatter plot of age and duration. Do you notice anything unusual? You can click on any data point to display all it's values.

3. (5 Points) In the previous point you should have found a data point, which seems to be corrupted, as some of its values are nonsensical. Even a single point like this can significantly affect the performance of a classifier. How do you think it would affect Decision trees? A good way to check this is to test the performance of each classifier before and after removing this datapoint.

4. (10 Points) To remove this instance from the dataset we will use a filter. We want to remove all instances, where the age of an applicant is lower than 0 years, as this suggests that the instance is corrupted. In the Preprocess tab click on Choose in the Filter pane. Select filters > unsupervised > instance > RemoveWithValues. Click on the text of this filter to change the parameters. Set the attribute index to 13 (Age) and set the split point at 0. Click Ok to set the parameters and Apply to apply the filter to the data. Visualise the data again to verify that the invalid data point was removed.

5. (20 Points) On the Classify tab, select the Percentage split test option and change its value to 90%. This way, we will train the classifiers using 90% of the training data and evaluate their performance on the remaining 10%. First, train a decision tree classifier with default options. Select classifiers > trees > J48 and click Start. J48 is the Weka implementation of the C4.5 algorithm, which uses the normalized information gain criterion to build a decision tree for classification.

6. (20 Points) After training the classifier, the full decision tree is output for your perusal; you may need to scroll up for this. The tree may also be viewed in graphical form by right-clicking in the Result list and selecting Visualize tree; unfortunately this format is very cluttered for large trees. Such a tree accentuates one of the strengths of decision tree algorithms: they produce classifiers which are understandable to humans. This can be an important asset in real life applications (people are seldom prepared to do what a computer program tells them if there is no clear explanation). Observe the output of the classifier and try to answer the following questions:

· How would you assess the performance of the classifier? Is the Percentage of Correctly Classified Instances a sufficient measure in this case? Why? Hint: check the number of good and bad cases in the test sample, using the confusion matrix. Each column of the matrix represents the instances in a predicted class, while each row represents the instances in an actual class. For example let us define an experiment from P positive instances and N negative instances. The four outcomes can be formulated in a 2 by 2 contingency table or confusion matrix. One benefit of a confusion matrix is that it is easy to see if the system is confusing two classes (i.e. commonly mislabeling one as another).

· Looking at the decision tree itself, are the rules it applies sensible? Are there any branches which appear absurd? At what depth of the tree? What does this suggest? Hint: Check the rules applied after following the paths: (a) CheckingAccount = <0, Foreign = yes, Duration >11, Job = skilled, OtherDebtors = none, Duration <= 30 and (b) CheckingAccount = <0, Foreign = yes, Duration >11, Job = unskilled.

· How does the decision tree deal with classification in the case where there are zero instances in the training set corresponding to that particular path in the tree (e.g. those leaf nodes that have (0:0))?

7. (20 Points) Now, explore the effect of the confidenceFactor option. You can find this by clicking on the Classifer name (to the right of the Choose button on the Classify tab). On the Classifier options window, click on the More button to find out what the confidence factor controls. Try the values 0.1, 0.2, 0.3 and 0.5. What is the performance of the classifier at each case? Did you expect this given your observations in the previous questions? Why do you think this happens?

8. (20 Points) Suppose that it is worse to classify a customer as good when they are bad, than it is to classify a customer as bad when they are good. Which value would you pick for the confidence factor? Which performance measure would you base your decision on?

9. (20 Points)Finally we will create a random decision forest and compare the performance of this classifier to that of the decision tree and the decision stump. The random decision forest is an ensemble classifier that consists of many decision trees and outputs the class that is the mode of the class's output by individual trees. Again set the test option Percentage split to 90%. Select classifiers > trees > RandomForest and hit Start. Again, observe the output. How high can you get the performance of the classifier by changing the number of trees (numTrees) parameter? How does the random decision forest compare performance wise to the decision tree and decision stump?

Deliverable:

· Your report including the screenshots of your implementation for each section and the results.

,

<td

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.

Related Posts

All Rights Reserved Terms and Conditions
College pals.com Privacy Policy 2010-2018

@RELATION GermanCredit
@ATTRIBUTE CheckingAccount {<0 <200 >=200 none}
@ATTRIBUTE Duration NUMERIC
@ATTRIBUTE CreditHistory {ok ok_at_this_bank ok_til_now past_delays critical}
@ATTRIBUTE Purpose {car_new car_used furniture television appliances repairs education vacation retraining business others}
@ATTRIBUTE CreditAmount NUMERIC
@ATTRIBUTE SavingsAccount {<100 <500 <1000 >=1000 unknown}
@ATTRIBUTE YearsEmployed {unemployed <1 <4 <7 >=7}
@ATTRIBUTE InstallmentRate NUMERIC
@ATTRIBUTE PersonalStatus {male_divorced female male_single male_married female_single}
@ATTRIBUTE OtherDebtors {none co guarantor}
@ATTRIBUTE ResidentSince NUMERIC
@ATTRIBUTE Property {real_estate savings car unknown}
@ATTRIBUTE Age NUMERIC
@ATTRIBUTE OtherPlans {bank stores none}
@ATTRIBUTE Housing {rent own free}
@ATTRIBUTE NumCreditsAtBank NUMERIC
@ATTRIBUTE Job {unemployed unskilled skilled management}
@ATTRIBUTE Dependents NUMERIC
@ATTRIBUTE Telephone {no yes}
@ATTRIBUTE Foreign {yes no}
@ATTRIBUTE Approve {good bad}
@DATA
<0 0 ok furniture -1000000000 unknown unemployed -10000 female_single none 0 car -293 none own -1000 skilled -185 yes no good
<0 6 critical television 1169 unknown >=7 4 male_single none 4 real_estate 67 none own 2 skilled 1 yes yes good
<200 48 ok_til_now television 5951 <100 <4 2 female none 2 real_estate 22 none own 1 skilled 1 no yes bad
none 12 critical education 2096 <100 <7 2 male_single none 3 real_estate 49 none own 1 unskilled 2 no yes good
<0 42 ok_til_now furniture 7882 <100 <7 2 male_single guarantor 4 savings 45 none free 1 skilled 2 no yes good
<0 24 past_delays car_new 4870 <100 <4 3 male_single none 4 unknown 53 none free 2 skilled 2 no yes bad
none 36 ok_til_now education 9055 unknown <4 2 male_single none 4 unknown 35 none free 1 unskilled 2 yes yes good
none 24 ok_til_now furniture 2835 <1000 >=7 3 male_single none 4 savings 53 none own 1 skilled 1 no yes good
<200 36 ok_til_now car_used 6948 <100 <4 2 male_single none 2 car 35 none rent 1 management 1 yes yes good
none 12 ok_til_now television 3059 >=1000 <7 2 male_divorced none 4 real_estate 61 none own 1 unskilled 1 no yes good
<200 30 critical car_new 5234 <100 unemployed 4 male_married none 2 car 28 none own 2 management 1 no yes bad
<200 12 ok_til_now car_new 1295 <100 <1 3 female none 1 car 25 none rent 1 skilled 1 no yes bad
<0 48 ok_til_now business 4308 <100 <1 3 female none 4 savings 24 none rent 1 skilled 1 no yes bad
<200 12 ok_til_now television 1567 <100 <4 1 female none 1 car 22 none own 1 skilled 1 yes yes good
<0 24 critical car_new 1199 <100 >=7 4 male_single none 4 car 60 none own 2 unskilled 1 no yes bad
<0 15 ok_til_now car_new 1403 <100 <4 2 female none 4 car 28 none rent 1 skilled 1 no yes good
<0 24 ok_til_now television 1282 <500 <4 4 female none 2 car 32 none own 1 unskilled 1 no yes bad
none 24 critical television 2424 unknown >=7 4 male_single none 4 savings 53 none own 2 skilled 1 no yes good
<0 30 ok business 8072 unknown <1 2 male_single none 3 car 25 bank own 3 skilled 1 no yes good
<200 24 ok_til_now car_used 12579 <100 >=7 4 female none 2 unknown 44 none free 1 management 1 yes yes bad
none 24 ok_til_now television 3430 <1000 >=7 3 male_single none 2 car 31 none own 1 skilled 2 yes yes good
none 9 critical car_new 2134 <100 <4 4 male_single none 4 car 48 none own 3 skilled 1 yes yes good
<0 6 ok_til_now television 2647 <1000 <4 2 male_single none 3 real_estate 44 none rent 1 skilled 2 no yes good
<0 10 critical car_new 2241 <100 <1 1 male_single none 3 real_estate 48 none rent 2 unskilled 2 no no good
<200 12 critical car_used 1804 <500 <1 3 male_single none 4 savings 44 none own 1 skilled 1 no yes good
none 10 critical furniture 2069 unknown <4 2 male_married none 1 car 26 none own 2 skilled 1 no no good
<0 6 ok_til_now furniture 1374 <100 <4 1 male_single none 2 real_estate 36 bank own 1 unskilled 1 yes yes good
none 6 ok television 426 <100 >=7 4 male_married none 4 car 39 none own 1 unskilled 1 no yes good
>=200 12 ok_at_this_bank television 409 >=1000 <4 3 female none 3 real_estate 42 none rent 2 skilled 1 no yes good
<200 7 ok_til_now television 2415 <100 <4 3 male_single guarantor 2 real_estate 34 none own 1 skilled 1 no yes good
<0 60 past_delays business 6836 <100 >=7 3 male_single none 4 unknown 63 none own 2 skilled 1 yes yes bad
<200 18 ok_til_now business 1913 >=1000 <1 3 male_married none 3 real_estate 36 bank own 1 skilled 1 yes yes good
<0 24 ok_til_now furniture 4020 <100 <4 2 male_single none 2 car 27 stores own 1 skilled 1 no yes good
<200 18 ok_til_now car_new 5866 <500 <4 2 male_single none 2 car 30 none own 2 skilled 1 yes yes good
none 12 critical business 1264 unknown >=7 4 male_single none 4 unknown 57 none rent 1 unskilled 1 no yes good
>=200 12 ok_til_now furniture 1474 <100 <1 4 female none 1 savings 33 bank own 1 management 1 yes yes good
<200 45 critical television 4746 <100 <1 4 male_single none 2 savings 25 none own 2 unskilled 1 no yes bad
none 48 critical education 6110 <100 <4 1 male_single none 3 unknown 31 bank free 1 skilled 1 yes yes good
>=200 18 ok_til_now television 2100 <100 <4 4 male_single co 2 real_estate 37 stores own 1 skilled 1 no yes bad
>=200 10 ok_til_now appliances 1225 <100 <4 2 male_single none 2 car 37 none own 1 skilled 1 yes yes good
<200 9 ok_til_now television 458 <100 <4 4 male_single none 3 real_estate 24 none own 1 skilled 1 no yes good
none 30 ok_til_now television 2333 <1000 >=7 4 male_single none 2 car 30 bank own 1 management 1 no yes good
<200 12 ok_til_now television 1158 <1000 <4 3 male_divorced none 1 car 26 none own 1 skilled 1 yes yes good
<200 18 past_delays repairs 6204 <100 <4 2 male_single none 4 real_estate 44 none own 1 unskilled 2 yes yes good
<0 30 critical car_used 6187 <500 <7 1 male_married none 4 car 24 none rent 2 skilled 1 no yes good
<0 48 critical car_used 6143 <100 >=7 4 female none 4 unknown 58 stores free 2 unskilled 1 no yes bad
none 11 critical car_new 1393 <100 <1 4 female none 4 car 35 none own 2 management 1 no yes good
none 36 ok_til_now television 2299 <1000 >=7 4 male_single none 4 car 39 none own 1 skilled 1 no yes good
<0 6 ok_til_now car_used 1352 <1000 unemployed 1 female none 2 savings 23 none rent 1 unemployed 1 yes yes good
none 11 critical car_new 7228 <100 <4 1 male_single none 4 savings 39 none own 2 unskilled 1 no yes good
none 12 ok_til_now television 2073 <500 <4 4 female co 2 real_estate 28 none own 1 skilled 1 no yes good
<200 24 past_delays furniture 2333 unknown <1 4 male_single none 2 savings 29 bank own 1 unskilled 1 no yes good
<200 27 past_delays car_used 5965 <100 >=7 1 male_single none 2 car 30 none own 2 management 1 yes yes good
none 12 ok_til_now television 1262 <100 <4 3 male_single none 2 car 25 none own 1 skilled 1 no yes good
none 18 ok_til_now car_used 3378 unknown <4 2 male_single none 1 savings 31 none own 1 skilled 1 yes yes good
<200 36 past_delays car_new 2225 <100 >=7 4 male_single none 4 unknown 57 bank free 2 skilled 1 yes yes bad
none 6 ok_at_this_bank car_new 783 unknown <4 1 male_single guarantor 2 real_estate 26 stores own 1 unskilled 2 no yes good
<200 12 ok_til_now television 6468 unknown unemployed 2 male_single none 1 unknown 52 none own 1 management 1 yes yes bad
none 36 critical television 9566 <100 <4 2 female none 2 car 31 stores own 2 skilled 1 no yes good
>=200 18 ok_til_now car_new 1961 <100 >=7 3 female none 2 car 23 none own 1 management 1 no yes good
<0 36 critical furniture 6229 <100 <1 4 female co 4 unknown 23 none rent 2 unskilled 1 yes yes bad
<200 9 ok_til_now business 1391 <100 <4 2 male_married none 1 real_estate 27 bank own 1 skilled 1 yes yes good
<200 15 critical television 1537 unknown >=7 4 male_single guarantor 4 real_estate 50 none own 2 skilled 1 yes yes good
<200 36 ok business 1953 <100 >=7 4 male_single none 4 unknown 61 none free 1 management 1 yes yes bad
<200 48 ok business 14421 <100 <4 2 male_single none 2 car 25 none own 1 skilled 1 yes yes bad
none 24 ok_til_now television 3181 <100 <1 4 female none 4 savings 26 none own 1 skilled 1 yes yes good
none 27 ok_til_now repairs 5190 unknown >=7 4 male_single none 4 savings 48 none own 4 skilled 2 yes yes good
none 12 ok_til_now television 2171 <100 <1 2 female none 2 car 29 bank own 1 skilled 1 no yes good
<200 12 ok_til_now car_new 1007 >=1000 <4 4 male_married none 1 real_estate 22 none own 1 skilled 1 no yes good
none 36 ok_til_now education 1819 <100 <4 4 male_single none 4 unknown 37 stores free 1 skilled 1 yes yes bad
none 36 ok_til_now television 2394 unknown <4 4 female none 4 car 25 none own 1 skilled 1 no yes good
none 36 ok_til_now car_used 8133 <100 <4 1 female none 2 savings 30 bank own 1 skilled 1 no yes good
none 7 critical television 730 unknown >=7 4 male_single none 2 savings 46 none rent 2 unskilled 1 yes yes good
<0 8 critical others 1164 <100 >=7 3 male_single none 4 unknown 51 bank free 2 management 2 yes yes good
<200 42 critical business 5954 <100 <7 2 female none 1 real_estate 41 bank own 2 unskilled 1 no yes good
<0 36 ok_til_now education 1977