Having 4 GB memory doesn't always mean you can allocate 4GB buffers. First you should look at the actually usable memory. Read this for more details about "usable" memory.
However several other factors can reduce big buffers allocation. The most frequent is the "memory fragmentation" (frequently bitmap allocations and releases). Look here:
Anyway, if ImageEn cannot allocate a single big buffer then it allocates a set of buffers representing each row. It works only when LegacyBitmap = false and you don't use Canvases (TCanvas or TIECanvas).