The modern power systems face challenges, including high proportions of uncertain renewable energy, rapid dynamics of power electronics, and decentralized control among multiple entities. Digital development has enabled power grids to integrate numerous edge devices equipped with sensing and computing capabilities, aiming to exploit edge data to enhance grid observability, controllability, and resilience. However, much of potential value of edge data remains unexploited with traditional architecture and methods. Therefore, we explore the potential of leveraging large language models (LLMs) to fully exploit edge data in modern power systems. An intelligent, scalable, and efficient three-layer architecture is proposed to align the capabilities of LLMs with the constraints of edge scenarios. Supporting technologies are reviewed for each layer, including multimodal data fusion, lightweight collaborative inference, and closed-loop control. To validate the proposed architecture, we provide three representative scenarios for preliminary exploration: virtual power plant (VPP) dispatch, intelligent substation inspection, and contingency management, illustrating how LLMs can unlock the value of edge data. We conclude by identifying key technical challenges and outlining future research directions for building modern power systems by LLM-based exploitation of edge data.