So many things these days are condensed into sound bites, for general consumption. Your mileage may vary.
Like you, I drink milk. Your two gallons a week wouldn’t last for 2 days in my fridge!
In western societies, many people eat too much, and too much of that food is too high in fat. The result is that many/most people are too fat. Not all, but most. Most people will benefit from eating lower fat food. Changing from full fat whole milk to lower fat varieties can be beneficial for most people, particularly if the drink a lot of milk.
Fat is a consideration for its high calorie issues, but there is also the inner health question to be dealt with. While you might not be fat on the outside, your intake of fat, including from milk, may be causing arterial blockages, or at least the beginnings of them. Lower fat intake including from milk, may help with heart and arterial health, even if you’re not fat on the outside.
The only way to know whether fat intake, with or without milk intake, is an issue for you is to get tested. See you doctor and get some blood tests. The results may of may not lead to other tests, but get tested and use the information to help you decide whether whole milk or lower fat version is for you. There is no one answer that is right for everyone.
In a similar way, my salt intake is an issue for me. I have several heart problems, plus high exercise blood pressure. Many people get high blood pressure from a high salt intake. Telling people with heart or blood pressure problems to reduce their salt intake is seen as a no-brainer by most people. Most people with blood pressure problems are put on beta blockers. My cardiologist tried me on beta blockers, but the side effects were not acceptable in my case. I’m on a completely different form of medication. My tested sodium levels are very low, and I’m not a salt lover, so I take very little salt with food. On top of that, my meds actually strip salt from my system. I actually have to force myself to eat more salt than I feel comfortable eating! I’ve had a CT angiogram of my heart and my cardiologist has said my heart arteries are “better than textbook”! He explained that in the textbooks, the arteries are always shown with at least some plaque/cholesterol buildup, and mine show absolutely none. Extra salt is not an issue for me.
Interestingly, when I got my meds from the pharmacy one time, there was a pamphlet saying that because I was on blood pressure meds, that I should limit/reduce my salt intake! I contacted the publisher about it, explained the situation, and recommended they seek advice from both my cardiologist and the pill manufacturer. To their credit, they did just that. They have since revised their pamphlet to say with beta blockers you should reduce salt intake. They no longer provide the brochure for people who get my medication.
My point is two fold. Get tested to find out where you stand. Eat to suit your situation, not that of the average population.