Thanks for the file. Which shows up here as a 2023 file.
The Connector can't be alone; it must be a Simulation object.
Anyway, I would suggest caching Cloth and using the Surface Deformer here.
Often, a Constraint is suggested for Buttons, but I have not found a stable setup with Simulation with it. I.e. I assume there is something, but I need to explore it more.
Let me know if that works.
Here is an example done in 2024
All the best