Should I Remove Immature Fruit From My Orange Tree?

Q. I have an orange tree I planted nine years ago.  It is about 4 feet high and only produced about 2 oranges a year.  Last winter it did have some freeze damage. I trimmed and fertilized it with iron and nitrogen and watered it once a week with a hose instead of the drip system. It is looking real good and has about 40 pea sized oranges on it.  Should I cull some of them as I don't think the tree can handle that many?