KDM4/JMJD2 are H3K9- and H3K36- specific demethylases, which are considered promising therapeutic targets for the treatment of acute myeloid leukemia (AML) harboring MLL-translocations. Here, we investigate the long-term effects of depleting KDM4 activity on normal hematopoiesis to probe potential side effects of continuous inhibition of these enzymes. Utilizing conditional Kdm4a/Kdm4b/Kdm4c triple-knockout mice we show that KDM4 activity is required for hematopoietic stem cell (HSC) maintenance in vivo The knockout of the KDM4 demethylases leads to accumulation of H3K9me3 on transcription start sites and the corresponding downregulation of expression of several genes in hematopoietic stem cells. We show that two of these genes, Taf1band Nom1, are essential for the maintenance of hematopoietic cells. Taken together, our results show that the KDM4 demethylases are required for the expression of genes essential for the long-term maintenance of normal hematopoiesis.