Ps counts the mappings for the framebuffer as memory. Especially on the FFB where a number of different mappings of the device address space is used to optimize access this can cause large amounts of memory, but not physical memory, to be mapped and shown by ps.
It's not unusual for the FFB+ (Creator3D) to show a 500MB process size for the X server.
Solaris 2.3 FCS also has a number of Xsun memory leaks when using the SX. Get the SX patches or upgrade to a later release of Solaris 2.x.
/usr/proc/bin/pmap (new in 2.5) will show the exact mappings used by Xsun. "pmap -x" (new in 2.6) will even show how much of each mapping is shared/private resident, etc. If Xsun has a memory leak, you'll see a huge "[heap]" with pmap.