The smart grid (SG) is a modern power system that leverages digital technologies to enhance the generation, delivery, and consumption of electric power. Reinforcement learning (RL) plays an important role in SG by helping make smart decisions. However, RL faces challenges such as sparse rewards, poor generalization, and difficulty in representing and interpreting regulatory constraints. Large language models (LLMs) offer new opportunities to address these challenges by understanding natural language, leveraging external knowledge, and enhancing reasoning. This paper presents a comprehensive review of LLM-enhanced RL for SG. It first analyzes the key challenges of RL and introduces how LLMs help enhance the performance, including a taxonomy based on the integration patterns and functions. It then reviews applications of LLM-enhanced RL for SG, with a focus on energy management, operational control, electricity market, and hardware design. Finally, it discusses research challenges and future directions of LLM-enhanced RL for SG. This paper aims to provide a clear framework for researchers to apply LLMs in RL and to promote the application of RL for SG.