"Why are such different diets successful for different people?"
The answer isn't obvious, but it is simple. Success has different criteria. Someone with food allergies will have different priorities than someone simply trying to lose weight.
My main priorities are weight-loss, sustainability and gut health, and I chose the Paleo Diet.
The Paleo Diet seems to have demonstrable success in the following parameters:
- Food Allergies
- Brain Function Disorders
- Gut Health
- Weight Correction
- Increased Insulin Sensitivity
- Cortisol Correction
- Joint Health
- Blood Lipids and Triglycerides
The Atkins Diet, and other Very Low Carb Diets by comparison does quite well on several of these points, such as helping epilepsy and insulin sensitivity, but isn't geared for food allergies as much. Some of those Atkins-brand candy bars really blew up my stomach thanks to freaky sugar alcohols and other Frankenfoods. (But I did lose weight).
Why is this type of analysis important?
First, people will do better if they set goals. Goals require choosing metrics for success. Example: I want to lower my fasting blood sugar, so I will try the Paleo Diet, limit carb intake, and test this in 2 months.
This leads to the second point. These points are testable. As this study shows, the Paleo Diet seems to out perform the Mediterranean Diet at increasing insulin sensitivity; this suggests a mechanistic cause that could help you accomplish that goal.
The USDA's MyPyramid plan, on the other hand, has a chief stated goal to "educate people about the 2005 Dietary Guidelines for Americans".
MyPyramid even claims that the "Stone Age Diet" is healthy because it is "low in fat"; this is false. We have no evidence that fats from wild, healthy animals is anything but healthy even in large doses, and most of us Stone-Agers like a nice fatty chunk of grass-fed steak.
It is healthy because it optimizes the above parameters, but none of those are related to fat intake.
Think of what vectors you wish to improve and try to make an eating and lifestyle plan that accomplishes them. Prioritize these vectors and set up realistic and frank methods for testing whether your plan is working.