Purpose: Ovarian cancer is one of the most common gynecological cancers in women. It is difficult to accurately and objectively diagnose benign and malignant ovarian tumors using ultrasound and other tests. Hence, there is an imperative need to develop a computer-aided diagnostic (CAD) system for ovarian tumor classification in order to reduce patient anxiety and the cost of unnecessary biopsies. In this paper, we present an automatic CAD system for the detection of benign and malignant ovarian tumors using advanced image processing and data mining techniques. Materials and Methods: In the proposed system, Hu’s invariant moments, Gabor transform parameters and entropies are first extracted from the acquired ultrasound images. Significant features are then used to train a probabilistic neural network (PNN) classifier for classifying the images into benign and malignant categories. The model parameter (σ) for which the PNN classifier performs the best is identified using a genetic algorithm (GA). Results: The proposed system was validated using 1300 benign images and 1300 malignant images, obtained from 10 patients with a benign disease and 10 with a malignant disease. We used 23 statistically significant (p < 0.0001) features. By evaluating the classifier using a ten-fold cross-validation technique, we were able to achieve an average classification accuracy of 99.8 %, sensitivity of 99.2 % and specificity of 99.6 % with a σ of 0.264. Conclusion: The proposed system is automated and hence is more objective, can be easily deployed in any computer, is fast and accurate and can act as an adjunct tool in helping physicians make a confident call about the nature of the ovarian tumor under evaluation.