It goes down when you make things that have a crop symbol with with an x through it. So if you see on an iron mine, it has 3 of them, so when you make an iron mine the crop production goes down by 3.

Thats why even when you have plenty of crop resources for what you need, you still have to build more to offset thecost of the other things you build.