The start of the big bang?

You have to take into account forces like gravity into place. Things can sling shot around eachother and collisions can slow down large mes. The universe has had billions of years to do all of this. But on a larger scale, yes everything is moving from a single point