Hi MFVis,

I hope I get your target correctly.

What you like are two independent systems that work together anyway. Correct.

Here is a suggestion

I baked the Voronoi Fracture setup into an Alembic and de-activated the initial Voronoi Fracture setup.
Voronoi Fracture Object selected: Object Manager> Object> Bake Alembic.

Since the motion of the sphere is short, I just traced it. While moving and matching the position, I keyframed the start and end of it, then one in the middle, and perhaps some more.
In the F-Curve Editor, you will see a specific Curve now. That should work already just fine.

In the example, I took an F-Curve. Snapshot, and deleted all keyframes except the first and the last, then called up the Snapshot and used only two keyframes to shape the F-Curve. It is smoother that way.

My best wishes for your project