Without meaning to be rude, if you don't know the answer you are a long way away from your goals. This is not something we can say "do this" and you will be right. The answer is not possible to give because there is no one answer and there are a lot of trade offs, it;'s like how long is a piece of string or what is the best car.
To add to that, all compression methods are "loss-y." Anything that reduces the file size will reduce quality. The question is whether a viewer will be able to detect the loss of quality. When you're preparing a video for web viewing, you can get away with quite a bit of compression without it being detectable.
Definitely MP4. It can now be played on virtually every platform. The only exceptions are IE8 and Firefox under WinXP, and with JW Player, you can fall back to Flash for those. But you still need only one file. You can pump almost anything through Handbrake with default settings, plus check "Web optimized," and get an MP4 that will play anywhere.
JW Player doesn't do any processing of the file at all. It's just a steering script, feeding the video to the browser and supplying its own controls, playlists, etc.