In this paper, islanding detection in a hybrid distributed generation (DG) system is analyzed by the use of hyperbolic S-transform (HST), timetime transform, and mathematical morphology methods. The merits of these methods are thoroughly compared against commonly adopted wavelet transform (WT) and S-transform (ST) techniques, as a new contribution to earlier studies. The hybrid DG system consists of photovoltaic and wind energy systems connected to the grid within the IEEE 30-bus system. Negative sequence component of the voltage signal is extracted at the point of common coupling and passed through the above-mentioned techniques. The efficacy of the proposed methods is also compared by an energy-based technique with proper threshold selection to accurately detect the islanding phenomena. Further, to augment the accuracy of the result, the classification is done using support vector machine (SVM) to distinguish islanding from other power quality (PQ) disturbances. The results demonstrate effective performance and feasibility of the proposed techniques for islanding detection under both noise-free and noisy environments, and also in the presence of harmonics.