Bioch, J.C.

Estimating the Market Share Attraction Model using Support Vector Regressions (Article)

2010-09-01T00:00:00Z

We propose to estimate the parameters of the Market Share Attraction Model (Cooper and Nakanishi, 1988; Fok and Franses, 2004) in a novel way by using a nonparametric technique for function estimation called Support Vector Regressions (SVR) (Smola, 1996; Vapnik, 1995). Traditionally, the parameters of the Market Share Attraction Model are estimated via a Maximum Likelihood (ML) procedure, assuming that the data are drawn from a conditional Gaussian distribution. However, if the distribution is unknown, Ordinary Least Squares (OLS) estimation may seriously fail (Vapnik, 1982). One way to tackle this problem is to introduce a linear loss function over the errors and a penalty on the magnitude of model coefficients. This leads to qualities such as robustness to outliers and avoidance of the problem of overfitting. This kind of estimation forms the basis of the SVR technique, which, as we will argue, makes it a good candidate for estimating the Market Share Attraction Model. We test the SVR approach to predict (the evolution of) the market shares of 36 car brands simultaneously and report promising results.

SVM-Maj: a majorization approach to linear support vector machines with different hinge errors (Research Paper)

2007-11-01T00:00:00Z

Support vector machines (SVM) are becoming increasingly popular for the prediction of a binary dependent variable. SVMs perform very well with respect to competing techniques. Often, the solution of an SVM is obtained by switching to the dual. In this paper, we stick to the primal support vector machine (SVM) problem, study its effective aspects, and propose varieties of convex loss functions such as the standard for SVM with the absolute hinge error as well as the quadratic hinge and the Huber hinge errors. We present an iterative majorization algorithm that minimizes each of the adaptations. In addition, we show that many of the features of an SVM are also obtained by an optimal scaling approach to regression. We illustrate this with an example from the literature and do a comparison of different methods on several empirical data sets.

Instance-Based penalization techniques for classification (Research Paper)

2007-01-06T00:00:00Z

Several instance-based large-margin classi¯ers have recently been put forward in the literature: Support Hyperplanes, Nearest Convex Hull classifier, and Soft Nearest Neighbor. We examine those techniques from a common fit-versus-complexity framework and study the links be- tween them. Finally, we compare the performance of these techniques vis-a-vis each other and other standard classification methods.

Estimating the market share attraction model using support vector regressions. (Research Paper)

2007-01-01T00:00:00Z

We propose to estimate the parameters of the Market Share Attraction Model (Cooper & Nakanishi, 1988; Fok & Franses, 2004) in a novel way by using a non-parametric technique for function estimation called Support Vector Regressions (SVR) (Vapnik, 1995; Smola, 1996). Traditionally, the parameters of the Market Share Attraction Model are estimated via a Maximum Likelihood (ML) procedure, assuming that the data are drawn from a conditional Gaussian distribution. However, if the distribution is unknown, ML estimation may seriously fail (Vapnik, 1982). One way to tackle this problem is to introduce a linear loss function over the errors and a penalty on the magnitude of model coefficients. This leads to qualities such as robustness to outliers and avoidance of the problem of over¯tting. This kind of estimation forms the basis of the SVR technique, which, as we will argue, makes it a good candidate for solving the Market Share Attraction Model. We test the SVR approach to predict (the evolution of) the market shares of 36 car brands simultaneously and report stronger results than when using a ML estimation procedure.

Nearest convex hull classification (Research Paper)

2006-12-01T00:00:00Z

Consider the classification task of assigning a test object to one of two or more possible groups, or classes. An intuitive way to proceed is to assign the object to that class, to which the distance is minimal. As a distance measure to a class, we propose here to use the distance to the convex hull of that class. Hence the name Nearest Convex Hull (NCH) classification for the method. Convex-hull overlap is handled through the introduction of slack variables and kernels. In spirit and computationally the method is therefore close to the popular Support Vector Machine (SVM) classifier. Advantages of the NCH classifier are its robustness to outliers, good regularization properties and relatively easy handling of multi-class problems. We compare the performance of NCH against state-of-art techniques and report promising results.

Nonlinear support vector machines through iterative majorization and I-splines (Research Paper)

2006-07-19T00:00:00Z

To minimize the primal support vector machine (SVM) problem, we propose to use iterative majorization. To do so, we propose to use it- erative majorization. To allow for nonlinearity of the predictors, we use (non)monotone spline transformations. An advantage over the usual ker- nel approach in the dual problem is that the variables can be easily inter- preted. We illustrate this with an example from the literature.

Classification with support hyperplanes (Research Paper)

2006-07-19T00:00:00Z

A new classification method is proposed, called Support Hy- perplanes (SHs). To solve the binary classification task, SHs consider the set of all hyperplanes that do not make classification mistakes, referred to as semi-consistent hyperplanes. A test object is classified using that semi-consistent hyperplane, which is farthest away from it. In this way, a good balance between goodness-of-fit and model complexity is achieved, where model complexity is proxied by the distance between a test object and a semi-consistent hyperplane. This idea of complexity resembles the one imputed in the width of the so-called margin between two classes, which arises in the context of Support Vector Machine learning. Class overlap can be handled via the introduction of kernels and/or slack vari- ables. The performance of SHs against standard classifiers is promising on several widely-used empirical data sets.

Solving and interpreting binary classification problems in marketing with SVMs (Research Paper)

2005-11-09T00:00:00Z

Marketing problems often involve inary classification of customers into ``buyers'' versus ``non-buyers'' or ``prefers brand A'' versus ``prefers brand B''. These cases require binary classification models such as logistic regression, linear, and quadratic discriminant analysis. A promising recent technique for the binary classification problem is the Support Vector Machine (Vapnik (1995)), which has achieved outstanding results in areas ranging from Bioinformatics to Finance. In this paper, we compare the performance of the Support Vector Machine against standard binary classification techniques on a marketing data set and elaborate on the interpretation of the obtained results.

Induction of Ordinal Decision Trees (Research Paper)

2003-02-10T00:00:00Z

This paper focuses on the problem of monotone decision trees from the point of view of the multicriteria decision aid methodology (MCDA). By taking into account the preferences of the decision maker, an attempt is made to bring closer similar research within machine learning and MCDA. The paper addresses the question how to label the leaves of a tree in a way that guarantees the monotonicity of the resulting tree. Two approaches are proposed for that purpose - dynamic and static labeling which are also compared experimentally. The paper further considers the problem of splitting criteria in the con- text of monotone decision trees. Two criteria from the literature are com- pared experimentally - the entropy criterion and the number of con criterion - in an attempt to find out which one fits better the specifics of the monotone problems and which one better handles monotonicity noise.

Monotone Decision Trees and Noisy Data (Research Paper)

2002-06-17T00:00:00Z

The decision tree algorithm for monotone classification presented in [4, 10] requires strictly monotone data sets. This paper addresses the problem of noise due to violation of the monotonicity constraints and proposes a modification of the algorithm to handle noisy data. It also presents methods for controlling the size of the resulting trees while keeping the monotonicity property whether the data set is monotone or not.

Modular Decomposition of Boolean Functions (Research Paper)

2002-04-08T00:00:00Z

Modular decomposition is a thoroughly investigated topic in many areas such as switching theory, reliability theory, game theory and graph theory. Most appli- cations can be formulated in the framework of Boolean functions. In this paper we give a uni_ed treatment of modular decomposition of Boolean functions based on the idea of generalized Shannon decomposition. Furthermore, we discuss some new results on the complexity of modular decomposition. We propose an O(mn)- algorithm for the recognition of a modular set of a monotone Boolean function f with m prime implicants and n variables. Using this result we show that the the computation of the modular closure of a set can be done in time O(mn2). On the other hand, we prove that the recognition problem for general Boolean functions is coNP-complete.

Version Spaces and Generalized Monotone Boolean Functions (Research Paper)

2002-03-19T00:00:00Z

We consider generalized monotone functions f: X --> {0,1} defined for an arbitrary binary relation <= on X by the property x <= y implies f(x) <= f(y). These include the standard monotone (or positive) Boolean functions, regular Boolean functions and other interesting functions as special cases. It is shown that a class of functions is closed under conjunction and disjunction (i.e., a distributive lattice) if and only if it is the class of monotone functions with respect to some quasi-order. Subsequently, we consider the monoid of all conjunctive operators on a set and show that this monoid is algebraically isomorphic to the monoid of all binary relations on this set. In this development, two operators, positive content and positive closure, play an important role. The results are then applied to the version space of all monotone hypotheses of a set of binary examples also called the class of all monotone extensions of a partially defined Boolean function, to clarify its lattice theoretic properties.

The Algorithmic Complexity of Modular Decomposition (Research Paper)

2001-06-14T00:00:00Z

Modular decomposition is a thoroughly investigated topic in many areas such as switching theory, reliability theory, game theory and graph theory. We propose an O(mn)-algorithm for the recognition of a modular set of a monotone Boolean function f with m prime implicants and n variables. Using this result we show that the computation of the modular closure of a set can be done in time O(mn2). On the other hand, we prove that the recognition problem for general Boolean func tions is NP-complete. Moreover, we introduce the so called generalized Shannon decomposition of a Boolean functions as an efficient tool for proving theorems on Boolean function decompositions.

Bankruptcy Prediction with Rough Sets (Research Paper)

2001-02-22T00:00:00Z

The bankruptcy prediction problem can be considered an or dinal classification problem. The classical theory of Rough Sets describes objects by discrete attributes, and does not take into account the order- ing of the attributes values. This paper proposes a modification of the Rough Set approach applicable to monotone datasets. We introduce re- spectively the concepts of monotone discernibility matrix and monotone (object) reduct. Furthermore, we use the theory of monotone discrete functions developed earlier by the first author to represent and to com- pute decision rules. In particular we use monotone extensions, decision lists and dualization to compute classification rules that cover the whole input space. The theory is applied to the bankruptcy prediction problem.

Mining frequent intemsets in memory-resident databases (Research Paper)

2000-12-05T00:00:00Z

Due to the present-day memory sizes, a memory-resident database has become a practical option. Consequently, new methods designed to mining in such databases are desirable. In the case of disk-resident databases, breadth-first search methods are commonly used. We propose a new algorithm, based upon depth-first search in a set-enumeration tree. For memory-resident databases, this method turns out to be superior to breadth-first search.

Quasi-monotone decision trees for ordinal classification (Research Paper)

1998-01-01T00:00:00Z

In many classification problems the domains of the attributes and the classes are linearly ordered. Since the known decision tree methods generate non-monotone trees, these methods are not suitable for monotone classification problems. We already provided order-preserving tree-generation algorithms for multi-attribute classification problems with k linearly ordered classes in a previous paper. For real-world datasets it is important to consider approximate solutions to handle problems like speed, tree-size and noise. In this report we develop a new decision tree algorithm that generates quasi-monotone decision trees. This algorithm outperforms classical algorithms such as those of Quinlan with respect to prediction, and beats algorithms that generate strictly monotone decision trees with respect to speed. This report contains proofs of all presented results.

Dualisation, decision lists and identification of monotone discrete functions (Research Paper)

1998-01-01T00:00:00Z

Many data-analysis algorithms in machine learning, datamining and a variety of other disciplines essentially operate on discrete multi-attribute data sets. By means of discretisation or binarisation also numerical data sets can be successfully analysed. Therefore, in this paper we view/introduce the theory of (partially defined) discrete functions as an important theoretical tool for the analysis of multi-attribute data sets. In particular we study monotone (partially defined) discrete functions. Compared with the theory of Boolean functions relatively little is known about (partially defined) monotone discrete functions. It appears that decision lists are useful for the representation of monotone discrete functions. Since dualisation is an important tool in the theory of (monotone) Boolean functions, we study the interpretation and properties of the dual of a (monotone) binary or discrete function. We also introduce the dual of a pseudo-Boolean function. The results are used to investigate extensions of partially defined monotone discrete functions and the identification of monotone discrete functions. In particular we present a polynomial time algorithm for the identification of so-called stable discrete functions.

Monotone Decision Trees (Research Paper)

1997-01-01T00:00:00Z

EUR-FEW-CS-97-07 Title Monotone decision trees Author(s) R. Potharst J.C. Bioch T. Petter Abstract In many classification problems the domains of the attributes and the classes are linearly ordered. Often, classification must preserve this ordering: this is called monotone classification. Since the known decision tree methods generate non-monotone trees, these methods are not suitable for monotone classification problems. In this report we provide a number of order-preserving tree-generation algorithms for multi-attribute classification problems with k linearly ordered classes.

Bivariate decision trees (Research Paper)

1996-01-01T00:00:00Z

Decision trees with tests based on a single variable, as produced by methods such as ID3, C4.5 etc., often require a large number of tests to achieve an acceptable accuracy. This makes interpretation of these trees, which is an important reason for their use, disputable. Recently, a number of methods for constructing decision trees with multivariate tests have been presented. Multivariate decision trees are often smaller and more accurate than univariate trees; however, the use of linear combinations of the variables may result in trees that are hard to interpret. In this paper we consider trees with test based on combinations of at most two variables. We show that bivariate decision trees are an interesting alternative to both uni- and multivariate trees.

Classification using Bayesian neural nets (Research Paper)

1995-01-01T00:00:00Z

Recently, Bayesian methods have been proposed for neural networks to solve regression and classification problems. These methods claim to overcome some difficulties encountered in the standard approach such as overfitting. However, an implementation of the full Bayesian approach to neural networks as suggested in the literature applied to classification problems is not easy. In fact we are not aware of applications of the full approach to real world classification problems. In this paper we discuss how the Bayesian framework can improve the predictive performance of neural networks. We demonstrate the effects of this approach by an implementation of the full Bayesian framework applied to three real world classification problems. We also discuss the idea of calibration to measure the predictive performance.

On the use of simple classifiers for the initialisation of one-hidden-layer neural nets (Research Paper)

1995-01-01T00:00:00Z

In this report we discuss the use of two simple classifiers to initialise the input-to-hidden layer of a one-hidden-layer neural network. These classifiers divide the input space in convex regions that can be represented by membership functions. These functions are then used to determine the weights of the first layer of a feedforward network.

On the statistical mechanics of (un)constrained stochastic Hopfield and 'elastic' neural networks (Research Paper)

1994-01-01T00:00:00Z

Stochastic binary Hopfield models are viewed from the angle of statistical mechanics. After an analysis of the unconstrained model using mean field theory, a similar investigation is applied to a constrained model yielding comparable general explicit formulas of the free energy. Conditions are given for which some of the free energy expressions are Lyapunov functions of the corresponding differential equations. Both stochastic models appear to coincide with a specific continuous model. Physically, the models are related to spin and Potts glass models. Also, a `complementary' free energy function of both the unconstrained and the constrained model is derived. The analysis culminates in a very general framework for analyzing constrained and unconstrained Hopfield neural networks: the stationary points of the corresponding free energy appears to coincide exactly with the set of equilibrium conditions of the corresponding continuous Hopfield neural network. Moreover, the relationship with `elastic net' algorithms is analyzed: it is proved that this class of algorithms cannot be derived from the theory of statistical mechanics (as sometimes is supposed), but should be considered as a special `penalty method', namely as one with dynamical penalty weights. We mention some experimental results and discuss implications for the use of the various models in resolving constrained optimization problems.

Constrained optimization with a continuous Hopfield-Lagrange model (Research Paper)

1993-01-01T00:00:00Z

In this paper, a generalized Hopfield model with continuous neurons using Lagrange multipliers, originally introduced Wacholder, Han &Mann [1989], is thoroughly analysed. We have termed the model the Hopfield-Lagrange model. It can be used to resolve constrained optimization problems. In the theoretical part, we present a simple explanation of a fundamental energy term of the continuous Hopfield model. This term has caused some confusion as reported in Takefuji [1992]. It led to some misinterpretations which will be corrected. Next, a new Lyapunov function is derived which, under some dynamical conditions, guarantees stability of the the system. We explain why a certain type of frequently used quadratic constraints can degenerate the Hopfield-Lagrange model to a penalty method. Furthermore, a difficulty is described which may arise if the method is applied to problems with `hard constraints'. The theoretical results suggest a method of using the Hopfield- Lagrange model. This method is described and applied to several problems like Weighted Matching, Crossbar Switch Scheduling and the Travelling Salesman Problem. The relevant theoretical results are applied and compared to the computational ones. Various formulations of the constraints are tried, of which one is a new approach, where a multiplier is used for every single constraint.

Simple improvements of a simple solution for inverting resolution (Research Paper)

1991-01-01T00:00:00Z

In this paper we address some simple improvements of the algorithm of Rouveirol and Puget [1989] for inverting resolution. Their approach is based on automatic change of representation called flattening and unflattening of clauses in a logic program. This enables a simple implementation of operators, such as Absorption, presented in Muggleton and Buntine [1988]. Unfortunately both the algorithms of MB and RP are incomplete. We analyze the reasons of the incompleteness of the RP algorithm and present an improved Absorption operator. It appears that flat tree epresentations of clauses and predicate calculus with equality provide an appropriate context for these matters.