This study examined the predictive validity of in-clerkship evaluation methods to identify medical students who have insufficient knowledge. Study subjects were 124 third-year medical students at the Uniformed Services University. Insufficient knowledge was defined by: (1) a clerkship 'pre-test' score one standard deviation below the mean or lower; or (2) any teacher verbally rating a student's general knowledge as 'marginal' or less; or (3) a student did not pass Step One of the United States Medical Licensing Examination (USMLE). We determined sensitivity and specificity using a standard score of </= 300 on the end of clerkship National Board of Medical Examiners (NBME) subject examination in medicine as the outcome variable. Sixteen students scored </= 300 on the NBME examination. The sensitivity of the 'pre-test' or verbal comments alone was 44% (seven of 16 students). By combining methods, 11 students were identified, for a sensitivity of 69%. The specificity of all methods was > 90%. Using USMLE Step One pass-fail performance did not improve sensitivity. Combining a 'pre-test' and instructors' formal evaluation session comments improves the early identification of students with insufficient knowledge, allowing for formative feedback and remediation during the clerkship.