To start with I was talking to Rich about baking and he imparted some of the wisdom from his own quest for the perfect bake. As I this was a hard surface project I took Rich's advice by setting the smoothing groups up so that anything with a harsh angle was set to be a hard edge on the low poly and then separated the UVs so that the different smoothing groups weren't welded together on the UV. It seems that most things talking about baking explain this as the way to do things including Scott Homer's presentation that was everywhere last week if you're interested in this stuff this presentation is awesome to study! Go study!
This technique however didn't lead to the desired result, I can't say why... computers are evil masterminds of rationality so I must have done something not quite right. You can see that the edges are unrealistically sharp and in comparison to the high poly the baked version just doesn't look right. So I continued to play around, got fed up and set everything to one smoothing group as I had done before and that ended with a blobby mess.
|Ignore that smoothing error on the right side of the low poly I'm working on that.|
During these series of experimentations I modified the shape of the low poly to fit better with the sculpted high poly, I took off corners that were too harsh before etc. I'm now in the process of going through everything and re-baking and adjusting the low poly I have a feeling that this should be worth the time I've spent on it. Unfortunately the normal maps themselves look like they've got worse but their effect has got better...