There are no large community-based studies examining the association of body size vs. body fat with vitamin D status. Association of serum 25-hydroxyvitamin D (25OHD) with body weight and subcategories of body weight defined by fat mass were evaluated in a large, free living population. Out of a total of 29,235 subjects from the 2008–2010 Korean National Health and Nutrition Examination Survey, the relevant data included 6458 subjects over 50 years of age who were analyzed cross-sectionally. Serum 25OHD concentrations were compared in men (n = 3164) and in women (n = 3294) by tertiles of body weight and body fat mass, as measured by Dual-energy X-ray Absorptiometry (DXA) within sex-specific tertiles of body weight. Serum 25OHD was weakly inversely correlated with body weight in the men and the women after adjustment for age (r = −0.075 and −0.073, respectively, p < 0.001 for both). Within each tertile of body weight, serum 25OHD decreased progressively as fat mass increased in men. This pattern was similar in the women but not consistently significant. Whereas body weight predicted a small decrease in serum 25OHD in the men and the women, greater adiposity, for any given weight, predicted larger decreases in the men, but not consistently in women.