How does the game determine which ship in the fleet gets attacked?

attack/(defense+hp) i believe. damage will reduce the denominator, thereby causing damaged ships to be attacked over undamaged ones of the same type.

Porcupine is correct. Toward that end if you have the space to spare on your hulls, you can use defenses to manipulate the targetting order. And if a ship gets damaged that will raise its priority to be targetted.

In order to counter this, I usually send my fleets with 2 or 3 extra ships (that won't fit in the fleet). Then, when one ship in the fleet gets damaged enough to be killed in the next encounter, I swap it out for one of the extras.

yea, that is also how you get high experience lvls on your ship. reserves are a must

It always seems to me that the AI attacks the ship that posses the biggest threat and that can kill quickest. Like in the begining when i put 1 beam attack on my fighters and a stinger on my heavy fighters. It always goes after the heavy fighters first.

That is essentially correct, but the AI also weighs the enemy's hit points and defenses against it's attack.

A tiny hull fighter has 6 hp, so a 1-beam attack with no defenses scores 1/(6+0)=0.1667 for targetting.
A small hull (10hp) stinger scores 2/(10+0)=0.200. The largest number gets targetted, which is why they attack your heavy stinger fighters before your light beam fighters.

You can use this to your advantage, if you have developed a good defensive system. Instead of spending all kinds of money and space and time adding it to every ship, build one or two ships in each fleet that are heavily defended, but still have the highest target number. That requires both a high defense and a good offense, so it can be a technologically superior vessel, flying with older, cheaply produced ships.