Abstract: The application of Smoothed Particle Hydrodynamics (SPH) in solving the Shallow Water Equations (SWE) holds significant potential. This potential is particularly evident in the field of ocean numerical simulation. However, issues related to open boundaries in the SWE-SPH method remain unresolved. This study addresses the reflection problem at open boundaries in existing SWE-SPH models. It proposes a new non-reflective boundary treatment method based on Flather's theory of gravity wave open boundaries. The performance of the new boundary condition is evaluated using three classic numerical simulation cases: steady flow over a bump, propagation of sea level perturbations in a flat-bottom channel with two open ends, and circular cylinder. Comparisons are made with the characteristic open boundary conditions from the original method. Results indicate that the new boundary conditions effectively reduce reflections, allow disturbances to propagate outward, and facilitate the implementation of external flow conditions specified at the model's open boundaries. Furthermore, the SWE-SPH model has been applied to simulate the Okushiri tsunami. The simulation results were then compared with those from the finite volume method. This comparison demonstrates the model's applicability and effectiveness in handling complex depth and transcritical flow simulations, thus expanding the scope of ocean numerical simulation problems that can be addressed.