Well, smaller ships have lesser HP, and lesser space, thus lesser ATT/DEF and most likely the game will consider them to be weaker; and in a fleet they will be attacked first. Actually I found it always more usefull to concentrate to get bigger ships to survive the battles, upgrading them to the same design to refill HP's, or giving them some rest in orbit. With time, they'll gain levels, and become increasingly strong. That's then that type of ship I'd update to most modern weapons usually.
While we are on the subject, if I have a fleet of two small warships, an armed star base, and a few unarmed, transport-hull troop carriers in the same square, how does the game decide what gets attacked first when the enemy sends a fleet into that square? Is the same sort of formula used?
Hard to say. Usually "specialized" ships that carry some kind of module are targetted, starbases, also unarmed ships. The AI will also target maxed-out fleets or SB's even when it's clear he won't survive that battle. The routines for this are somewhat strange....