WeightLoss question i appreciated the advice!?

So i just started taking things out, like junk food, and fast food. But when i crave something i start to feel like binging. I'm not sure if taking everything out is such a good idea. I mean since I'm starting. Would it be wise to eat in moderation. I always eat junk food and it's very hard to stop eating it. I can cut soft drinks, and fast food. But junk food is hard. I been thinking of not depriving myself. Just get a small amount and that's it. I'm counting calories so i guess i could incorporate that? What do you think. As for exercise that going good. Just my stopping eating junk food is hard. : / Advice would be much appreciated.