There are a few techs like that. Check out the the last couple beam weapon techs leading up to Doomray, for example.

I consider them 'transition techs'. Like you said, they are 'bleeding edge' techs that are the dead-end of their line and have trade-offs - but they lead to something much better. In your case it is the Hyperwarp drives, which go up in speed and/or down in size (and sizemod) from the Mk V.

The Mk V is 1 larger in size with the same sizemod of 6 vs. the Mk IV, with a speed increase of 1 over the Mk IV. That is a 33% increase in both speed and size, but the cost goes down from 18 to 10. Isn't that acceptable? (my numbers come from DL - I have no idea if these have changed with DA or ToA)

Since the size vs. sizemod is more a factor on larger ships, and you have figured this out with your own calculations, you should probably use the Mk V only on tiny and small hulls and stick with the Mk IV for larger hulls.

It is not something that should be tweaked in the unmodded game. But you can always tweak it yourself, to your own liking.

Changes occured because there were players who would make troop transports with speeds over 40. Those players in turn would completely destroy an empire by invading all of their victim's planets in a single turn. Back then when an empire was defeated, all owned ships would disappear instead of joining another empire or turning to a life of piracy. As such, it was an easy way to win since any enemy ship you could avoid a fight with (namely those not orbiting a planet), you never had to worry about.

Suffice to say, engines were nerfed in DA to make it possible for the AI to compete against human players who would use such a cheap tactic.