As long as age isnt an issue neither for you nor him i say go for it! If you love each other, want to be around each other and you dont have serious problems in your relationship, there is absolutely no reason for others to try and patronize you into doing something you dont want to do. Live this situation to the fullest, enjoy it. If its meant to be it will be. If not you will have the experience by your side to continue to better things. Being 24 doesnt mean youre brainless. 24 is just a number as are 19 and 49. If you manage to close that 'gap' in between and youre sure that this is the right man for you(of course he must think the same for you)age shouldnt matter.
I hope this helps