Do not worry about your difficulties in Mathematics. I can assure you mine are still greater. How can Albert Einstein have said this if, in...

The simple answer to your question is that Einstein was
referring to his struggles at learning math when he was younger.  Of course, he
eventually mastered the subject on a very high level and was able to use it as a tool in
his work as a physicist.

The deeper answer is that
mathematics always remained a difficult subject for Einstein because he recognized how
profound and difficult it truly is.  When a scholar truly recognizes the complexity of
his chosen field of study, it becomes harder, not
easier. 

For example, when you first learned about the
Civil War in 4th or 5th grade, you were probably taught that it was all about slavery. 
This makes it easy to understand the conflict about the North and South.  The problem is
that it is a very imcomplete picture of the truth.  When a scholar studies the Civil
War, he realizes that the truth is much more complex--and therefore, much more difficult
to understand.