'Not enough memory' error on larger input file

[eniw]

I've successful used wine with a 32bit Windows command line executable called msconvert.exe, it's a scientific tool used to convert an input binary data file (.raw) into a readable XML format (.mzXML).

I've used it on input files from ranging from 20MB to 400MB without any problems.

However given a larger data file (over 700MB), I get 'Not enough memory' errors from wine. The error happens under a couple seconds after I run the command. From monitoring top, the wine command does not go above 1.0% memory usage in those few seconds. I've tried varying the amount of RAM from 2GB up to 8GB, the error still occurs in seconds each time. I've verified I can natively convert those files in Windows to rule out the app being defective.

Linux OS: CentOS 6.5 x86_64
wine: wine-1.7.26

For some input files, there seems to be an element of randomness to the not enough memory error as I actually observed the exe finish successfully sometimes when repeatedly running the same command immediately after getting an error. On the successful run, top indicates the exe is using less than 15% of system memory.

I've included such a sample terminal session below, it's for a 732MB input raw file. Notice I get an error, I run it again and it succeeds, then I run it again and I get an error again. Note that for very large data files (I tried a 1.5GB input file), it always seems to give me the not enough memory error. So that 732MB file seems to be on the borderline of what input file can work.

Any suggestions on what I can do to troubleshoot and fix this? Thanks!

Sample code of the same wine command run 3 times successively, 2 fail, 1 succeeds.

Code: Select all

[sammi@new-host-44 data]$ wine ~/bin/msconvert.exe --mzXML -z --filter "peakPicking true 1-" testfile.raw 
fixme:thread:SetThreadStackGuarantee (0x32fc04): stub
err:ole:CoGetContextToken apartment not initialised
fixme:heap:HeapSetInformation (nil) 1 (nil) 0
format: mzXML 
    m/z: Compression-Zlib, 64-bit
    intensity: Compression-Zlib, 32-bit
    rt: Compression-Zlib, 64-bit
ByteOrder_LittleEndian
 indexed="true"
outputPath: .
extension: .mzXML
contactFilename: 

filters:
  peakPicking true 1-
  
filenames:
  testfile.raw
  
processing file: testfile.raw
fixme:ntdll:NtLockFile I/O completion on lock not implemented yet
Error writing run 1 in "testfile.raw":
Not enough memory.

fixme:process:FlushProcessWriteBuffers : stub


[sammi@new-host-44 data]$ wine ~/bin/msconvert.exe --mzXML -z --filter "peakPicking true 1-" testfile.raw 
fixme:thread:SetThreadStackGuarantee (0x32fc04): stub
err:ole:CoGetContextToken apartment not initialised
fixme:heap:HeapSetInformation (nil) 1 (nil) 0
format: mzXML 
    m/z: Compression-Zlib, 64-bit
    intensity: Compression-Zlib, 32-bit
    rt: Compression-Zlib, 64-bit
ByteOrder_LittleEndian
 indexed="true"
outputPath: .
extension: .mzXML
contactFilename: 

filters:
  peakPicking true 1-
  
filenames:
  testfile.raw
  
processing file: testfile.raw
fixme:ntdll:NtLockFile I/O completion on lock not implemented yet
writing output file: .\testfile.mzXML

fixme:process:FlushProcessWriteBuffers : stub
##NOTE: IT SUCCEEDS HERE IN WRITING THE OUTPUT MZXML FILE##


[sammi@new-host-44 data]$ wine ~/bin/msconvert.exe --mzXML -z --filter "peakPicking true 1-" testfile.raw 
fixme:thread:SetThreadStackGuarantee (0x32fc04): stub
err:ole:CoGetContextToken apartment not initialised
fixme:heap:HeapSetInformation (nil) 1 (nil) 0
format: mzXML 
    m/z: Compression-Zlib, 64-bit
    intensity: Compression-Zlib, 32-bit
    rt: Compression-Zlib, 64-bit
ByteOrder_LittleEndian
 indexed="true"
outputPath: .
extension: .mzXML
contactFilename: 

filters:
  peakPicking true 1-
  
filenames:
  testfile.raw
  
processing file: testfile.raw
fixme:ntdll:NtLockFile I/O completion on lock not implemented yet
Error writing run 1 in "testfile.raw":
Not enough memory.

fixme:process:FlushProcessWriteBuffers : stub
[sammi@new-host-44 data]$ 

[oiaohm]

eniw I hope what you are processing requires vendor parts. You need to install "libpwiz-tools" package for you distribution or the static package from http://proteowizard.sourceforge.net/downloads.shtml . msconvert is a native tool I know it is short a few things in the cross platform form but if what you are doing is not requiring any of the missing parts from the native tools you are just harming self without good cause.


Wine should not be given Unix paths unless you want to risk oddities.

Code: Select all

wine start /unix ~/bin/msconvert.exe

should have been used instead of.

Code: Select all

wine ~/bin/msconvert.exe

Using the start command can alter application performance quite a bit.

eniw something to be aware of is that the file is compressed by 64 bit mode compression. Have you tested that the same files don't explode under windows using 32 bit version. This might not be a wine bug but an application so resulting in application attempting to allocate more than the allowed space for a 32 bit windows application.

Some of the randomness will be decompression vs processing. There are other factors to wine as well. wineserver a background process does not end straight away when application does. wineserver sets up sections of wine memory space that might not be freed from the prior run when the next run happens. wineserver -k before rerunning might make following cases not fail. 32 bit application in wine still has 32 bit memory limits.

Anything past 4G of memory will not help a 32 bit application from windows. Windows applications don't natively PAE. Yes this is an application where someone need to work on the 64 bit version of wine so the 64 bit version of it works.

[eniw]

Thanks for the reply oiaohm. Yes I install the Windows tar.bz2 (includes vendor reader support) from in linux from the proteowizard site.

In order to read in the input files, msconvert uses vendor libraries which are only available in Windows format, so it's not possible to do this conversion with the linux version of msconvert. I must use wine with the Windows version. Here's note from the proteowizard wine page regarding why I use the 32bit Windows version: You will need to use wine in 32bit emulation mode (basically a 32bit wine even if you are on a 64bit architecture). This is because the .Net and Visual Studio libraries are required and these libraries are installed using a 32 bit installer, even for the 64 bit versions of these libraries. Please also be aware that not all proprietary vendor libraries that are distributed with ProtoWizard are 64 bit hence the 64 bit version of msconvert cannot be used to convert most proprietary files. Source: http://tools.proteomecenter.org/wiki/in ... nvert_Wine

I actually never have used the wine start /unix command prefix to run windows exes in linux as the stuff I ran worked without it. I just tried wine start /unix from terminal in gnome desktop (keep in mind msconvert is a command line tool) and it seems to redirect the output of the program to some X window which flashes open for a second or two then closes so I can't see anything. More importantly, with start /unix, msconvert no longer seems to work as it does not produce the expected output file for files I know that work. There's no program output I can see so it's hard to determine what the problem is.

If I'm connecting via SSH without X11 forwarding, I get this output for a file that works without the start /unix:

Code: Select all

[sammi@new-host-44 data]$ wine start /unix ~/bin/msconvert.exe --mzXML -z --filter "peakPicking true 1-" workingtestfile.raw 
fixme:heap:HeapSetInformation (nil) 1 (nil) 0
fixme:process:SetProcessDEPPolicy (1): stub
fixme:heap:HeapSetInformation (nil) 1 (nil) 0
fixme:process:SetProcessShutdownParameters (00000380, 00000000): partial stub.
[sammi@new-host-44 data]$ Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
fixme:thread:SetThreadStackGuarantee (0x33fc04): stub
                                                     err:ole:CoGetContextToken apartment not initialised
                                                                                                        fixme:heap:HeapSetInformation (nil) 1 (nil) 0
            fixme:process:FlushProcessWriteBuffers : stub

So there's no msconvert program output at all there like there was before. Note the above command works without the start /unix, but with it I get returned to the command prompt and the command continues to print out output to my command line for a few seconds (Application tried to create a windows...). I don't get the output file.

Not sure what's happening here with wine not working when I use start /unix.

I've vertified the files I'm testing work using msconvert in native Windows XP 32-bit. The output file itself actually is smaller than the input file when using msconvert. As an example, the 1.5GB input file produces a 700MB file when I do it in Windows.

[oiaohm]

eniw so the program is really ouch.

I guess you Linux system has swapfile/swap-partition large enough. For mapped to memory this can be a issue. Some people install Linux systems without a swap file or swap partition because they believe they have enough ram. Few odd cases programs will fail due to not having swap wine can be one of them.

start /unix should function close to the same except start up a console window and this can close quickly. I forgot this.

There is another way of skinning the same cat.

Code: Select all

wine cmd /c 'c:\windows\notepad.exe' arguements...

The c:\windows\notepad.exe will have to be replaced what ever

Code: Select all

winepath -w ~/bin/msconvert.exe

gives. You can also use wine --whatever winepath -w gives--

Both "start" and "cmd" create a nice little stall to allow wine to get itself setup before attempting your program. Its just the case if the first application off the bat is quick to be heavy and wine is setting up its environment and being heavy trouble sometimes happen.

Code: Select all

wine cmd

Alone and let wine get running then run msconvert .exe from there may also get around the issue.

eniw just to be aware when you feed a Unix path as application into wine bad thing sometimes happen because of how things are mapped. Straight after wine without start or cmd you may not have given wine enough time to be ready for a huge operation. Little like waking you up and asking you to answer a complex problem it does not work well. Running msconvert under windows this would not happen at the same time the system is init state you will not see the login screen until windows is complete init.

Also start and cmd will consume a little memory. If it background activity of wine both will help. If it a true case that its a memory limit in wine both will make it worse.

eniw now if it still not able to do it right using "wine cmd" and waiting a little (5 min)before doing the msconvert.exe open a bug as there has to be something about wine that its not allocating enough memory. In fact just on the past limit files should now fail if it a case of a real wine limit.

SSH without X11 forwarding without fake X11 will not run wine start commands. ssh you have more complex instructions to skin cat. wine cmd does support sleep.

eniw this is a case we will risk making stuff worse to get a better clue what is going wrong.