ࡱ> PO}( / 00DTimes New Roman43v 0( 0DArial Narrowan43v 0( 0" DSymbolarrowan43v 0( 00DWingdingsowan43v 0( 0 ` .  @n?" dd@  @@`` Ht     c $ f@uʚ;2Nʚ; g42d2dv 0ppp@ <4!d!d` 0L 4<4dddd` 0L 4<4BdBd` 0L 4___PPT9h` H?&+ Dr. RiggsO =Classification & ID3Dr. Riggs Spring 2004Classification ProblemGiven a some number of observed features Predict an unobserved feature (the  class ) Example: Given features of a borrower Predict whether he will default An interesting problem is learning rules from exampleslVZ Z=ZZ7ZV =7  Example Dataa ?id ?size ?color ?shape ?class (item 1 medium blue brick yes) (item 2 small red sphere yes) (item 3 large green pillar yes) (item 4 large green sphere yes) (item 5 small red wedge no) (item 6 large red wedge no) (item 7 large red pillar no)4*Z8Z#8b!3Distinguish ALL By Size Distinguish {2 5} By Shape Distinguish {3 4 6 7 } By COLOR Considerations Are the examples enough? The examples must be enough to tell the classes apart This is an unsolvable question Are the rules the most efficient? We could have made other choices What should we uses to compare choices?v6 "!(6 " !( EntropyhMeasures  disorder Def: n H(m1..mn) = - Pr(mi) * lg( Pr(mi ) ) i=1 Example (entropy of learning set): Messages (m1& m7) : Y Y Y Y N N N Pr(Y) = 4/7 Pr(N) = 3/7 H = - [ 4/7*lg 4/7 + 3/7*lg(3/7) - [ .571*-.243 + .429*-.368] = .985FZ0Z+0Z +Z#ZZ1   * #a  &!  o 1 GainIf a set is partitioned by a feature into subsets the gain in entropy is: Original_entropy - the_weighted_sum_of_subclass_entropies Eg: Partition ALL = {1,2,3,4,5,6,7} by COLOR {blue 1 } {red 2 5 6 7} {green 3 4} partition =>{blue Y} {red Y N N N} {green Y Y} map H(blue)=0 H(red)= .811 H(green)=0 GAIN(color) = H(all) - |ss|/|all|*H(ss) ss=red,green,blue = .985  ( 1/7*0 + 4/7*.811 + 2/7*0) = .522>2A--01  @)- (,01 P A Distinguish All By Color Distinguish All By Shape ID3Given: a learning set (LS) examples w/ features & outcome (class) Use each (feature,value) to partition the LS Calculate H for each partition Pf,v Calculate the gain for each feature Original H  | Pf,v | / |LS| * H(Pf,v) v Partition by the feature with highest gain Apply ID3 to any subsets Pf,v with H>0*" 'v" NR" 'N $           "   E  b"  ` ` ̙33` 333MMM` ff3333f` f` f` 3>?" dd@,|?" dd@   " @ ` n?" dd@   @@``PR    @ ` ` p>> jb(    6@j "`  T Click to edit Master title style! !$  0l "  RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S  0Tq "``  X*  0v "`   Z*  0{ "`   Z*H  0޽h ? ̙33 Classes0 zr (    0E P    P*    0K     R*  d  c $ ?    0F  @  RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S  6R `P   P*    6X `   R*  H  0޽h ? ̙33@ $(  $ $ 0f P    X*  $ 0Pk     Z*  $ 6n `P   X*  $ 6r `   Z* H $ 0޽h ? ̙33   $(   r  S .p  r  S / `    H  0޽h ? ̙33  0$(  r  S H=`   r  S >  H  0޽h ? ̙33  @$(  r  S LB`   r  S |D  H  0޽h ? ̙33  P(  r  S XM`     < (class ?id yes)<<H  0޽h ? ̙33  JB` ,(  ,x , c $z`    , <} -{2 5} = size: small , <X  sphere , <  wedge , <` <  6{2 }f  , < \  6{5 }f  , <l $  CClass:: Y NP  RB  ,@ s *D&p ` RB  , s *D& ` , <80 a ,$D0 yI(feature ?id size small) (feature ?id shape sphere) => (class ?id yes)JJ , <|0  ,$D0 xH(feature ?id size small) (feature ?id shape wedge) => (class ?id no)IIH , 0޽h ? ̙33F  p 0(  0x 0 c $`    0 < /{ 3 4 6 7 } = size: large  0 < > green 0 <  red 0 <ঌP Vp  :{3 4 } f 0 <쪌   < { 6 7 } f  0 <L E  VBC: Y Y N NRB  0@ s *D  6 RB  0 s *D `  0 <xt `,$D0 yI(feature ?id size large) (feature ?id color green) => (class ?id yes)JJ 0 < |,$D0 vF(feature ?id size large) (feature ?id color red) => (class ?id no)GGH 0 0޽h ? ̙33  4$(  4r 4 S `   r 4 S T  H 4 0޽h ? ̙33  g_8(  8r 8 S Œ`   r 8 S XÌ   8 <dnj0 _ lg = log2 H 8 0޽h ? ̙33  <$(  <r < S x`   r < S 4  H < 0޽h ? ̙33      (  x  c $`     <- ){1,2,3,4,5,6,7} = All  < & blue  <V O   red  < & green  < V  {1}  <`   Y{ 2 5 6 7 },   <0 |P   { 3 4 }   < `  T@map: Y Y N N N Y Y`   00 P,$D 0 F H: 0 -1/4lg1/4 -3/4lg3/4 0 wH= 0 + 4/7*(.5+.31) + 0 = .81 Gain = .985 - .464 = .5216C],D*.RB  @ s *DV  v RB   s *DV v RB  s *DV ` H  0޽h ? ̙33   [ S @ (  @x @ c $h`    @ <P!   *{1,2,3,4,5,6,7} = All @ <H$ brick @ <&  sphere @ <4*p  wedge @ <. V  {1} @ <,1    { 2 4 }  @ <4 |  { 5 6 }  @ <8   JC: Y Y Y N N Y N  @ 0x;0 ,$D 0 2 H: 0 0 0 1 wH= 0 + 2/7*0 + 2/7 *0 + 2/7*1 =.286 Gain = .985 -.286 = .699*Uib" + )RB  @@ s *DV  v RB  @@ s *DVp ` RB @ s *DV `  @ <dF pillar @ <I ,  {3 7}RB @ s *Dp  H @ 0޽h ? ̙33  D:(  Dr D S LN`    D S R  "p`PpH D 0޽h ? ̙33rD%/13}5q+U`>GPO@ABCDEFHIJKLMNQRoot EntrydO)Current UserGSummaryInformation(7PowerPoint Document(mDocumentSummaryInformation8?