If I have 8GB of RAM in a server and I run 4 instances of SQL Express,
will the total memory limit used by SQL Server be 1GB or 4GB?
Each instance can use up to 1GB of memory for the buffer pool. Each instance can use a bit more than 1GB in total because not all memory allocations go via the buffer pool. In your case, the maximum memory used by the four instances for buffer pool would be 4GB.
To confirm, I started two instances of the SQL Server 2008 Express Database Engine, performed some activity to load up the (separate) buffer pools, and then looked at per-instance memory utilization in a number of ways, for example using DBCC MEMORYSTATUS or by counting the number of buffers using the sys.dm_os_buffer_descriptors DMV.
The physical memory usage numbers below were obtained using simultaneous queries against the system DMV sys.dm_os_process_memory on each instance of the database engine:
FROM sys.dm_os_process_memory AS dopm;
║ Instance1 ║ Instance2 ║
║ 1102872 ║ 1059812 ║
Each of these is slightly in excess of 1GB as total physical memory usage includes more than just buffer pool usage, as mentioned previously.
Would it be advisable to run multiple instances like this to enable
each database to make better use of resources (assuming that the
server has plenty of resources)?
If the databases on each instance are functionally independent then it is at least workable to run multiple instances of Express in this way, though you would need to pay careful attention to configuration and maintenance may be made more complex.
You might be be better served using another edition, such as the fully-featured (and very cheap) Developer Edition, if the intended usage matches the licence). You would need to say much more about the circumstances to get a clear answer on this point.