A Multiple Objective Classifier Selection Methodology for Real World Problems

Real world engineering problems often involve selecting between alternatives. As cyber-physical systems (CPS) become more prevalent in society, including smart cities, intelligent transportation networks, smart homes, etc., these alternatives will increasingly not only include variations in hardware but also in software. Stakeholders will have to decide between competing designs consisting of varying sensor sets and varying machine learning models. Ensuring that these selections are well guided and include multiple objectives such as performance, cost, and sensitivity is paramount. Many of these CPS will include machine learning classification models. The receiver operating characteristic (ROC) analysis is useful for selecting between alternative classification models based on performance and sensitivity, however, this analysis fails to treat classification models as a system of hardware and software, a CPS. Therefore, the ROC analysis ignores additional stakeholder objectives such cost and reliability. This thesis presents methods to extend the ROC analysis to include an additional objective of system cost. The methodology is demonstrated on three real world data sets that include additional cost information. The presented methodology is shown to drastically reduce the stakeholder decision space and guide stakeholder decisions using performance, cost, and sensitivity objectives.