Past is past.
It is your choice and mostly your will to make things different.
You have one life.
It's you or them.
The notion of "Time" and what implies on our existence frightens you to death but time is not the enemy: you are. It's all in your head.
Try harder even if it hurts inside. If it's worth it, you must go all the way.