Wine memory limitation for malloc?

Adept · Post by **Adept** » Mon Jul 21, 2008 12:26 pm

Why win32 application under Wine can allocate only 2000 MB of memory, but same programm translated by gcc can get 3800 MB.
(OS/Compilers - WinXP/Visual C 6.0 32 bit, and openSUSE 11 x64/gcc -m32)

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

main(int argc, char **argv)
{unsigned long n,m,i; int *p;
i=4095; if (argc == 2) i = (size_t) strtoul(argv[1], NULL, 0);
m=i*1024*1024; while( (p=(int*)malloc(m)) ==0 ) m-=1024l*1024l;
printf("Allocated %u MB ( %u bytes )\n",m/1024l/1024l,m);
getchar(); free(p); exit(0);
}

vitamin · Post by **vitamin** » Tue Jul 22, 2008 12:18 am

Adept wrote:Why win32 application under Wine can allocate only 2000 MB of memory,

Correct, that's all you get with 32-bit windows. Everything else reserved for the system.

Gert van den Berg · Post by **Gert van den Berg** » Tue Jul 22, 2008 12:50 am

On Tue, Jul 22, 2008 at 7:18 AM, vitamin <[email protected]> wrote:

Adept wrote:
Why win32 application under Wine can allocate only 2000 MB of memory,
Correct, that's all you get with 32-bit windows. Everything else reserved for the system.

Actually up to 3GB in certain cases under Windows NT 5.x...
http://www.microsoft.com/whdc/system/pl ... AEmem.mspx

Gert van den Berg

Adept · Post by **Adept** » Tue Jul 22, 2008 2:35 am

Gert van den Berg wrote:On Tue, Jul 22, 2008 at 7:18 AM, vitamin <[email protected]> wrote:
Adept wrote:
Why win32 application under Wine can allocate only 2000 MB of memory,
Correct, that's all you get with 32-bit windows. Everything else reserved for the system.
Actually up to 3GB in certain cases under Windows NT 5.x...
http://www.microsoft.com/whdc/system/pl ... AEmem.mspx

Gert van den Berg

BTW, PAE options supported and Windows XP.

I think, it's bad idea for Wine, to emulate Windows limitations.
If Wine+linux can give more memory to my application, it's good reason (for me, as Windows user) to format Windows and install Linux+Wine.

mikolajz · Post by **mikolajz** » Tue Jul 22, 2008 5:42 am

There are apps that won't work if there is more than 2GB memory (e.g. they may treat this value as signed and fail to start informing the user that it requires at least 16MB of RAM on a computer with 3GB RAM). That's why Microsoft introduced a header flag that allows the developer to mark the executable as able to handle such an amount of memory. Wine also implements this behaviour. When compiling the program with Visual C, you should try the linker option /LARGEADDRESSAWARE:YES . AFAIK this puts that flag in the header.

As it was said, on 32-bit Windows, in the default configuration, this flag won't change anything as the high 2GB of address space is reserved for the system (and it's not possible to allocate more memory with malloc than the available address space). It's possible to make 3GB of address space available to the application with the /3gb switch in boot.ini. AFAIR, the 3GB address space is the default on Linux and may be even increased by distributions. On 64-bit Windows and Linux, the whole 4GB of address space will be visible to applicaitons with such a flag.

I may also add that this problem is different than the problem addressed by the PAE. The PAE allows to give more than 4GB of physical memory to all the processes in the system. The problem here is how much of the address space is available to a single process.

Adept · Post by **Adept** » Tue Jul 22, 2008 8:31 am

mikolajz wrote:There are apps that won't work if there is more than 2GB memory (e.g. they may treat this value as signed and fail to start informing the user that it requires at least 16MB of RAM on a computer with 3GB RAM). That's why Microsoft introduced a header flag that allows the developer to mark the executable as able to handle such an amount of memory. Wine also implements this behaviour. When compiling the program with Visual C, you should try the linker option /LARGEADDRESSAWARE:YES . AFAIK this puts that flag in the header.

As it was said, on 32-bit Windows, in the default configuration, this flag won't change anything as the high 2GB of address space is reserved for the system (and it's not possible to allocate more memory with malloc than the available address space). It's possible to make 3GB of address space available to the application with the /3gb switch in boot.ini. AFAIR, the 3GB address space is the default on Linux and may be even increased by distributions. On 64-bit Windows and Linux, the whole 4GB of address space will be visible to applicaitons with such a flag.

I may also add that this problem is different than the problem addressed by the PAE. The PAE allows to give more than 4GB of physical memory to all the processes in the system. The problem here is how much of the address space is available to a single process.

It's great idea!
Test program:
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
main(int argc, char **argv)
{unsigned long n,m,N,i; int *p,*p1,*p2,*p3; clock_t time1,time2;
i=4095; if (argc == 2) i = (size_t) strtoul(argv[1], NULL, 0); n=0; N=0;
m=i*1024*1024; while( (p=(int*)malloc(m)) ==0 ) m-=1024l*1024l;
printf("Allocated %u MB ( %u bytes )\n",m/1024l/1024l,m); n=m; N=m;
m=i*1024*1024; while( (p1=(int*)malloc(m)) ==0 ) m-=1024l*1024l;
printf("Allocated %u MB ( %u bytes )\n",m/1024l/1024l,m); N+=m;
m=i*1024*1024; while( (p2=(int*)malloc(m)) ==0 ) m-=1024l*1024l;
printf("Allocated %u MB ( %u bytes )\n",m/1024l/1024l,m); N+=m;
m=i*1024*1024; while( (p3=(int*)malloc(m)) ==0 ) m-=1024l*1024l;
printf("Allocated %u MB ( %u bytes )\n",m/1024l/1024l,m); N+=m;
printf("Total allocated %u MB ( %u bytes )\n",N/1024l/1024l,m);
}

Results:
Computer: Core 2Duo, 4GB physical memory.
OS - openSUSE 11 x64.

Win32 console application compiled _without_ linker option /LARGEADDRESSAWARE. Visual C 6.0. Started under wineconsole.
Allocated 1529 MB
Allocated 439 MB
Allocated 45 MB
Allocated 15 MB
Total allocated 2028 MB

Win32 console application compiled with linker option /LARGEADDRESSAWARE. Visual C 6.0. Started under wineconsole.
Allocated 1529 MB
Allocated 1024 MB
Allocated 439 MB
Allocated 45 MB
Total allocated 3037 MB

Linux 32-bit application compiled gcc with option -m32
Allocated 3836 MB
Allocated 128 MB
Allocated 123 MB
Allocated 4 MB
Total allocated 4091 MB

Good option /LARGEADDRESSAWARE, but 1GB unused

Quote from wine sources. File vxd.c:
>> /*
>> * Note: Win32s reserves 0GB - 2GB for Win 3.1 and uses 2GB - 4GB as
>> * sparse memory arena. We do it the other way around, since
>> * we have to reserve 3GB - 4GB for Linux, and thus use
>> * 0GB - 3GB as sparse memory arena.
>> *
>> * FIXME: What about other OSes ?
>> */
>> context->Ecx = W32S_WINE2APP(0x00000000);
>> context->Edx = W32S_WINE2APP(0xbfffffff);

1GB reserve for Lunix?! IMHO, not more 256MB!

mikolajz · Post by **mikolajz** » Tue Jul 22, 2008 10:15 am

The comment in vxd.c is old and so it Win32s. The important part is in dlls/ntdll/virtual.c - there is there an ADDRESS_SPACE_LIMIT of 3GB. There is a comment that this is a Windows limit, but AFAIK it is not present on x64 Windows, so probably it can be changed for LARGE_ADDRESS_AWARE programs, to allow them to use all the address space. Maybe we should query the host OS for it?

You could try to compile Wine with this limit set to 0 and check if this help with your program.

downer · Post by **downer** » Tue Jul 22, 2008 8:09 pm

I'll try also, but please let us know how it goes. I ran into maximum memory problems too!

Gert van den Berg · Post by **Gert van den Berg** » Wed Jul 23, 2008 4:49 am

On Tue, Jul 22, 2008 at 12:42 PM, mikolajz <[email protected]> wrote:

As it was said, on 32-bit Windows, in the default configuration, this flag won't change anything as the high 2GB of address space is reserved for the system (and it's not possible to allocate more memory with malloc than the available address space). It's possible to make 3GB of address space available to the application with the /3gb switch in boot.ini. AFAIR, the 3GB address space is the default on Linux and may be even increased by distributions. On 64-bit Windows and Linux, the whole 4GB of address space will be visible to applicaitons with such a flag.

I wonder if the CONFIG_VMSPLIT* options in the kernel is relevant? I
know Wine complains if they are changed...

Just a note: If I remember correctly Linux x86 can use the full 64GB
supported by PAE, with a smaller per-process limit... (x86_64 supports
a lot more off course...)

I may also add that this problem is different than the problem addressed by the PAE. The PAE allows to give more than 4GB of physical memory to all the processes in the system. The problem here is how much of the address space is available to a single process.

Windows XP x86 will not use the full 4GB (as a total address space for
all processes) even if PAE is enabled since Service Pack 2...
http://support.microsoft.com/kb/888137

mikolajz · Post by **mikolajz** » Wed Jul 23, 2008 12:25 pm

It's written in a comment that this 3GB limit is a Windows limit, so maybe it's from the the Win9x memory layout (http://support.microsoft.com/kb/q125691/). I don't know why the 3GB would be special in the NT memory layout. I would need to check that but I think we can change this in NT mode. It would be best to take the vmsplit value from the kernel, but I don't know yet how it can be queried at runtime.

Windows XP x86 will not use the full 4GB (as a total address space for
all processes) even if PAE is enabled since Service Pack 2...

It's getting a bit off-topis, but AFAIK the PAE was enabled in SP2 only because it is required to use the NX bit (no-execute memory, a.k.a. XD, DEP). To have true PAE support you need to have Windows Server Enterprise (or a 64-bit OS).

Adept · Post by **Adept** » Thu Jul 24, 2008 4:17 pm

mikolajz wrote:The comment in vxd.c is old and so it Win32s. The important part is in dlls/ntdll/virtual.c - there is there an ADDRESS_SPACE_LIMIT of 3GB. There is a comment that this is a Windows limit, but AFAIK it is not present on x64 Windows, so probably it can be changed for LARGE_ADDRESS_AWARE programs, to allow them to use all the address space. Maybe we should query the host OS for it?

You could try to compile Wine with this limit set to 0 and check if this help with your program.

Great! It's work!
I change in dlls/ntdll/virtual.c two lines:

# define ADDRESS_SPACE_LIMIT ((void *)0xc0000000) /* top of the total available address space */
# define USER_SPACE_LIMIT ((void *)0x7fff0000) /* top of the user address space */

to

# define ADDRESS_SPACE_LIMIT 0 /* no limit needed on other platforms */
# define USER_SPACE_LIMIT 0 /* no limit needed on other platforms */

and compile Wine. As a result it possible to allocate about 3600 MB for windows application (openSUSE x64 +Wine 1.0)

downer · Post by **downer** » Mon Jul 28, 2008 8:09 am

Do I need x64 Linux, or need to change more? When I change just what's listed on i686 bigmem, I get

Code: Select all

make[2]: Entering directory `/home/username/Desktop/wine-1.1.2/dlls/ntdll'
gcc -c -I. -I. -I../../include -I../../include  -D__WINESRC__ -D_NTSYSTEM_ -D_RE                                                             ENTRANT -fPIC -Wall -pipe -fno-strict-aliasing -Wdeclaration-after-statement -Ww                                                             rite-strings -Wtype-limits -Wpointer-arith  -g -O2  -o actctx.o actctx.c
actctx.c: In function ‘RtlQueryInformationActivationContext’:
actctx.c:2620: internal compiler error: Segmentation fault

Thanks for any help.

downer · Post by **downer** » Tue Jul 29, 2008 1:44 pm

(Debian Testing with gcc 4.3)
Will larger memory support be standard with Wine in the future? I only ask because I don't want to go back to Windows just for larger memory support.

vitamin · Post by **vitamin** » Tue Jul 29, 2008 2:23 pm

downer wrote:(Debian Testing with gcc 4.3)
Will larger memory support be standard with Wine in the future? I only ask because I don't want to go back to Windows just for larger memory support.

Unlikely. Wine conforms to what windows does and it's limitations, or it will break LOTS of applications.
Also Wine is NOT 64-bit and won't be in the near future.

vitamin · Post by **vitamin** » Tue Jul 29, 2008 2:25 pm

downer wrote:actctx.c internal compiler error: Segmentation fault[/code]

Thanks for any help.

That's compile problem not Wine. You most likely need

Code: Select all

ulimit -u unlimited

before compiling Wine.

downer · Post by **downer** » Tue Jul 29, 2008 4:07 pm

That's compile problem not Wine. You most likely need
Code:
ulimit -u unlimited
before compiling Wine.

Thanks for the fast reply. It compiles without error now!

Adept · Post by **Adept** » Wed Jul 30, 2008 3:04 am

vitamin wrote:
downer wrote:(Debian Testing with gcc 4.3)
Will larger memory support be standard with Wine in the future? I only ask because I don't want to go back to Windows just for larger memory support.
Unlikely. Wine conforms to what windows does and it's limitations, or it will break LOTS of applications.
Also Wine is NOT 64-bit and won't be in the near future.

IMHO, it's possible to make some user defined option for skip memory limits. For example:
$ wine -BIG_MEM myprog.exe