Speech-enabled smart speakers are common devices used for numerous tasks in everyday life. While speech-enabled technologies are widespread, using one's voice as a computing modality introduces new accessibility challenges for people with speech disfluencies such as stammering (also known as stuttering). This paper investigates the smart speaker user experiences of people who stammer over three weeks. We conducted diary studies and semi-structured interviews with 11 individuals to identify their daily routines, difficulties with successful interactions, and strategies to overcome these barriers. Our analysis demonstrates key factors such as device location, its affordances, and the structure of commands had a strong impact on user experience. Participants highlighted different linguistic strategies to try and overcome interaction difficulties and discussed the potential of using smart speakers for speech and language therapy. We emphasise the need to further understand the experiences of people who stammer in smart speaker design to increase their accessibility.