Technically, there is no upper limit for the amount of memory you can specify. However, no matter how much memory you specify, the 32 bit version of yEd cannot use more than 2GB of memory. (That is not a yEd specific restriction, though, but is true for all 32 bit applications.) If you need more than 2GB memory, you will have to use the 64 bit version on a 64 bit operating system (and of course your computer must have enough RAM available).
That said, why do you think there is an upper limit of 3.7GB?
Moreover, which version of yEd, Java and what operating system are you using?