The problem is your server not the video file Whatever it is , it's pretty slow and there may be some other problem too. Basically you should not serve mp4 videos from a web server because they are often slow at output bandwidth and most of them are shared hosting setups. The best way for you to host your video files ( in my opinion) is on Amazon S3.
I can confirm that when I put your video file inside my Amazon S3 it plays back really well and you can jump back and forward without that error,