7 Feature Selection and extraction 7.1 Selecting an Initial Set of Features 331 7.2 Procedure for Feature Selection 7.3 Feature Selection Using Support Vector machine 7.3.1 Backward or Forward Feature Selection 7.3.2 Support Vector Machine-Based Feature Selection 7.3.3 Feature Selection by Cross-Validation 7.4 Feature extraction 339 References 340 8 Clustering 8.1 Domain Description 8.2 Extension to Clustering References 351 9 Maximum-Margin Multilayer Neural Networks 9.1 Approach 353 9.2 Three-Layer Neural Networks 9.3 CARVE Algorithm 9.4 Determination of Hidden-Layer Hyperplanes 9.4.1 Rotation of Hyperplanes 9.4.2 Training Algorithm 362 9.5 Determination of Output-Layer Hyperplanes 9. 6 Determination of Parameter values 9.7 Performance Evaluation 364 10 Maximum-Margin Fuzzy Classifiers 10.1 Kernel Fuzzy Classifiers with Ellipsoidal Regions 10.1.1 Conventional Fuzzy Classifiers with dal regions 10.1.2 Extension to a Feature Space 369 10.1. 4 Maximizing Margins 10.1.5 Performance evaluation 10.2 Fuzzy Classifiers with Polyhedral Regions 10.2.1 Training Methods 383 10.2.2 Performance evaluation 11 Function Approximation 11.1 Optimal Hyperplanes 11.2 LI Soft-Margin Support Vector Regressors 11.3 L2 Soft-Margin Support Vector Regressors 11.4 Model selection 11.5 Training Methods 103Contents xvii 7 Feature Selection and Extraction . . . . . . . . . . . . . . . . . . . . . . . . . 331 7.1 Selecting an Initial Set of Features . . . . . . . . . . . . . . . . . . . . . . . . 331 7.2 Procedure for Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 332 7.3 Feature Selection Using Support Vector Machines . . . . . . . . . . 333 7.3.1 Backward or Forward Feature Selection . . . . . . . . . . . . . 333 7.3.2 Support Vector Machine-Based Feature Selection . . . . . 336 7.3.3 Feature Selection by Cross-Validation . . . . . . . . . . . . . . . 337 7.4 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 8 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 8.1 Domain Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 8.2 Extension to Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 9 Maximum-Margin Multilayer Neural Networks . . . . . . . . . . . 353 9.1 Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 9.2 Three-Layer Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 9.3 CARVE Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 9.4 Determination of Hidden-Layer Hyperplanes . . . . . . . . . . . . . . . 358 9.4.1 Rotation of Hyperplanes . . . . . . . . . . . . . . . . . . . . . . . . . . 359 9.4.2 Training Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 9.5 Determination of Output-Layer Hyperplanes . . . . . . . . . . . . . . . 363 9.6 Determination of Parameter Values . . . . . . . . . . . . . . . . . . . . . . . 363 9.7 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 10 Maximum-Margin Fuzzy Classifiers . . . . . . . . . . . . . . . . . . . . . . . 367 10.1 Kernel Fuzzy Classifiers with Ellipsoidal Regions . . . . . . . . . . . 368 10.1.1 Conventional Fuzzy Classifiers with Ellipsoidal Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 10.1.2 Extension to a Feature Space . . . . . . . . . . . . . . . . . . . . . . 369 10.1.3 Transductive Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 10.1.4 Maximizing Margins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375 10.1.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 10.2 Fuzzy Classifiers with Polyhedral Regions . . . . . . . . . . . . . . . . . 382 10.2.1 Training Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 10.2.2 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11 Function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 11.1 Optimal Hyperplanes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 11.2 L1 Soft-Margin Support Vector Regressors . . . . . . . . . . . . . . . . . 399 11.3 L2 Soft-Margin Support Vector Regressors . . . . . . . . . . . . . . . . . 401 11.4 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 11.5 Training Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403