Prostate cancer (PCa) is a heterogeneous disease with multifocal origin. In current clinical care, the Gleason scoring system is the well-established diagnosis by microscopic evaluation of the tissue from trans-rectal ultrasound (TRUS) guided biopsies. Nevertheless, the sensitivity and specificity in detecting PCa can range from 40 to 50% for conventional TRUS B-mode imaging. Tissue elasticity is associated with the disease progression and elastography technique has recently shown promise in aiding PCa diagnosis. However, many cancer foci in the prostate gland has very small size less than 1 mm and those detected by medical elastography were larger than 2 mm. Hereby, we introduce optical coherence elastography (OCE) to quantify the prostate stiffness with high resolution in the magnitude of 10 μm. Following our feasibility study of 10 patients reported previously, we recruited 60 more patients undergoing 12-core TRUS guided biopsies for suspected PCa with a total of 720 biopsies. The stiffness of cancer tissue was approximately 57.63% higher than that of benign ones. Using histology as reference standard and cut-off threshold of 600kPa, the data analysis showed sensitivity and specificity of 89.6% and 99.8% respectively. The method also demonstrated potential in characterising different grades of PCa based on the change of tissue morphology and quantitative mechanical properties. In conclusion, quantitative OCE can be a reliable technique to identify PCa lesion and differentiate indolent from aggressive cancer.