Arithmophobia: Fear of numbers
All of us have dreaded going to math class at one time or another, so we can't even imagine how awful it would be for those who have arithmophobia.
Ironically, the fear itself could originate in math class. If you always do poorly no matter how hard you try, have a math teacher that dislikes you, or have been embarrassed in the class, the fear could originate.
Occasionally, children will be beaten by their parents if they do not perform well on tests, which creates a trauma associated with math and numbers.
Even after graduation, the fear would follow in the form of phone numbers or simple equations you might run into.
There is also the fear of specific numbers that are considered to be "unlucky" or even evil, such as 666 or 13. These fears are often proliferated in certain cultures and religions, or even the media on occasion.