You need to be at war and hold a diplomatic and military advantage and it also helps to have the Galactic Bazaar (makes your trades appear 50% better to the AI).
Then you offer a peace treaty and the dialog should be green indicating that the treaty will be accepted. At that point start demanding more and more as part of your trade until the dialog turns red indicating the deal will not be accepted. At that point back off until it turns green again. Usually you finish with money which allows fine granularity on the total value of the trade.
You can ask for and sometimes receive all kinds of things. Trade goods, tech, ships, starbases, even planets and of course cash. How much you can get away with is all dependent on military might ratios and diplomatic advantage (and Galactic Bazaar).
In certain situations you can even get the AI to trade away every last planet they own, however this happens only when an AI is pretty much close to death anyway. The advantage of "killing off" an AI in this manner is that because you get everything in exchange for a peace treaty you don't get credit (blame is perhaps the better word) for killing them off so you don't get labelled with the negative approval factor "tendency to conquer others". It's usually important to avoid this label because it can be a significant (negative) factor in future dealings with other AI's.
