In this study, we propose a machine‐learning‐based approach to identify the modal parameters of the output‐only data for structural health monitoring (SHM) that makes full use of the characteristic of independence of modal responses and the principle of machine learning. By taking advantage of the independent feature of each mode, we use the principle of unsupervised learning, turning the training process of the neural network into the process of modal separation. A self‐coding neural network is designed to identify the structural modal parameters from the vibration data of structures. The mixture signals, that is, the structural response data, are used as the input of the neural network. Then, we use a complex loss function to restrict the training process of the neural network, making the output of the third layer the modal responses we want, and the weights of the last two layers are mode shapes. The neural …