Hi Prof. Ng! Thank you for your amazing course of machine learning on Coursera. I am working on a research subject inside image processing which uses machine learning tools, e.g., the SVM. Before taking your machine learning course, I simply used these machine learning tools without really getting to understand them (more or less like a black box to me). I really appreciate your course and the very clear explanations from the basics to widely used techniques of machine learning. The course has cleared up many my previous confusions. I have the following questions, and I'd really appreciate it if you could kindly share your answers/opinions/suggestions with us.
The curse of dimensionality. In practice, I sometimes encountered the problem of overfitting. According to your course, one solution to overfitting problems is to collect more samples for training the classifier. You also mentioned "sometimes it's not who has the best algorithm that wins; it's who has the most data." I understand the importance of data, especially we are now in the era of big data. Some people say "the number of training samples should be at least, in general, ten times the dimensionality of the feature". However, often, we don't have that much data (or it is very expensive to collect a large amount data). Moreover, when the dimensionality of the feature is very large (say, 30000), the cost of the extraction of features and the training are very expensive (more than I can afford). On the other side, for example, the SVM is often used in the case that the dimensionality of the feature is much larger than the number of training samples. Vapnik also pointed out in his classical book The Nature of Statistical Learning Theory that the support vector technique in many cases allows us to overcome the curse of dimensionality. What's your opinion on this problem?
What's the main big gap/difference in your opinion between the research community and the industry? What're your suggestions for people who just got the Ph.D. to search for a research position in industry?