Yes, that’s an interesting point about hard SVMs — I guess that follows from the fact that SVMs maximise the margin; it’s pretty clear that “duplicating” a data point has no effect whatsoever on the margin.

Interestingly, a number of psycholinguists (e.g., Bybee) suggest that humans pay far more attention to type frequency rather than token frequency when learning morphology; i.e., multiple presentations of the same data item doesn’t help human learning, rather we need to see different variants instantiating the same principle.

But type-based learning isn’t unique to SVMs — learning the parameters in the upper levels of hierarchical Bayesian models is type-based in a similar way.

I also find the interpretation of SVMs as a robust classifier (Xu, et al. JMLR 2009) to be intriguing. In any case, I think there is a lot more going on in SVMs than just a convex approximation to a probabilistic model.

P.S. I use Platt scaling all the time. However, I don’t see how that gives us a probabilistic interpretation of SVMs. I think it just tells us that the discriminant function computed by an SVM is a useful input to a logistic regression.

But SVMs do have a probabilistic interpretation, i.e., SVMs are designed to minimise the expected loss. There’s a certain attraction to this — in many applications this is precisely what we want to do.

Also, there is a close relationship between the Perceptron and log-linear “MaxEnt” classifier models. Specifically, if you estimate a log-linear classifier with stochastic gradient descent (i.e., mini-batch of size 1) and make a Viterbi approximation (i.e., assume that the expected feature values are the feature values of the mode label) then you obtain the Perceptron update rule! Perhaps even more surprisingly, in SGD, L2 regularisation becomes multiplicative weight decay and L1 regularisation becomes subtractive weight decay! (This isn’t too surprising after a bit of thought).

Probabilistic methods make sense when the component being learned is buried inside a large system, and you seek a modular solution. Probabilities provide a good intermediate representation that can be converted (in a principled way) to optimize many other loss functions. Of course if you are willing to perform end-to-end training of the entire system (in the style of deep neural networks), then you don’t need modularity and you can just use the task-specific loss.

I don’t know of a probabilistic interpretation of SVMs, but I haven’t looked. As usually stated, SVMs (and perceptrons) give you a classification machine, but don’t model either the probabilities of the predictors or the outcome categories. They’re more like the discriminative models than the generative ones if you work by analogy to logistic regression. After all, MAP estimates for logistic regression and the usual estimates for SVMs look the same when you fit logistic regression with a point estimate by minimizing log loss and fit an SVM by minimizing hinge loss.