Climate shocks are predicted to increase in magnitude and frequency as the climate changes, notably impacting poor and vulnerable communities across the Tropics. The urgency to better understand and improve communities’ resilience is reflected in international agreements such as the Paris Agreement and the multiplication of adaptation research and action programs. In turn, the need for collecting and communicating evidence on the climate resilience of communities has increasingly drawn questions concerning how to assess resilience. While empirical case studies are often used to delve into the context-specific nature of resilience, synthesizing results is essential to produce generalizable findings at the scale at which policies are designed. Yet datasets, methods and modalities that enable cross-case analyses that draw from individual local studies are still rare in climate resilience literature. We use empirical case studies on the impacts of El Niño on smallholder households from five countries to test the application of quantitative data aggregation for policy recommendation. We standardized data into an aggregated dataset to explore how key demographic factors affected the impact of climate shocks, modeled as crop loss. We find that while cross-study results partially align with the findings from the individual projects and with theory, several challenges associated with quantitative aggregation remain when examining complex, contextual and multi-dimensional concepts such as resilience. We conclude that future exercises synthesizing cross-site empirical evidence in climate resilience could accelerate research to policy impact by using mixed methods, focusing on specific landscapes or regional scales, and facilitating research through the use of shared frameworks and learning exercises.