The rapid advancement of deepfake technology has enabled the creation of highly realistic forged face images or videos. While deepfake technology adds entertainment to people’s lives, it also poses a potential threat to social security. Deepfake detection is a crucial technology for identifying forged images. However, existing deep learning-based models for deepfake detection often overlook subtle forged traces. To solve this problem, we propose a Progressive Attention Network (PANet). The PANet incorporates two attention modules, namely the Efficient Multi-Scale Attention Module (EMAM) and the Spatial and Channel Attention Module (SCAM), in a progressive manner. The EMAM focuses on crucial facial regions, such as the eyes, nose, and mouth, rather than the entire face. The SCAM facilitates fine-grained feature extraction. Experimental results demonstrate that the proposed method achieves state-of-the-art results on deepfake detection datasets.