Wine and 32-bit applications virtual address space

Questions about Wine on Linux
Locked
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Wine and 32-bit applications virtual address space

Post by zpangwin »

Hello, first post here but I have what I think is probably a little bit tougher question... Does the Windows 32-bit applications virtual address space stuff apply to Wine or does Wine manage memory differently than when the app runs on Windows natively?

System Info:
OS: Linux Mint 19.1 x64 (with plans to reinstall with 19.2 in the near future)
RAM: 32 GB
CPU: AMD FX-990
OS and Game partitions are using ext4 format.

Wine Info:
I have 2 (relevant) wine prefixes: the proton one setup by Steam and another one I setup via Lutris. I was primarily trying to run the utility described below through the Lutris one, but I am asking my question in a more general context. I am not at home and will try to fill in actual versions later but everything should be the latest...

System Wine: latest Wine-staging from PPA (Not at home... I think 4.14?)
System Winetricks: latest from github source master branch as of yesterday morning
Wine Arch: 64-bit
OS Settings: I think it was either set to win7 or possibly undefined. Will confirm when I get home.

Proton: I think it is latest (pretty sure my last protondb submission said 4.2 a couple weeks back)
Proton Arch: 64-bit

Background:
I was looking at modding Fallout 3 (steam) via an instance of Vortex that I have running in wine via lutris. I am currently in the process of setting up the core / dependent stuff and in several (windows) tutorials for setting up FO3 with better graphics I have come across mention of the nexus mod Large Address Aware Enabler for FO3. After extracting and copying it over to my lutris prefix with a copy of Fallout3.exe, I tried kicking off the utility with:

Code: Select all

env WINEPREFIX=$(pwd) wine wineconsole
cd C:\3gb_enabler
START.bat
but I got an error (I think it was something like ShellExecuteEx failed or something like that). Also tried launching C:/windows/system32/cmd.exe with wine but it never came up so I figured wineconsole was probably as good as I was going to get.

Then I started wondering if there was a way to do the same thing as the utility from a purely bash + shell utils approach since the mod's doc says:
I am releasing this application as it may help with crashing and performance issues with Fallout 3, particularly when using my larger texture packs. The utility was originally programmed and released by MadBoris, for the game Supreme Commander. I (NMC) have simply tweaked the program to work with Fallout 3. Read all about the original utility here: http://forums.gaspowered.com/viewtopic.php?t=2382 (note: I have relinked to an archived version since the original is long gone).


While I do have access to a Windows partition to run this from, I am trying hard to avoid it (as I am aiming to completely sever ties with Windows by Jan 2020). Anyway, I have been trying to search and determine if any of this 32-bit application virtual address space stuff is even relevant in Wine?. Apologies if my google foo is off today but I did spend a bit of time looking and haven't been able to find much with regards to this topic and wine... and what I could find was a bit dated.

Guesswork so far:
In Link #1, mikolajz says that "AFAIR, the 3GB address space is the default on Linux and may be even increased by distributions. On 64-bit Windows and Linux, the whole 4GB of address space will be visible to applications with such a flag". He later mentions some bits of code but I couldn't tell definitively how wine handles this and didn't want to necro the post. Also, the solutions there seemed to be suggestions for making kernel / wine options where this would apply across the board -- that's great if it exists but if not I would be content with applying memory options to specific launch instance (e.g. wine --some-flag-to-override-3gb-limit C:/foo/myapp.exe) or else running/creating a script/utility that would permanently set the /3GB and /LARGEADDRESSAWARE:YES flags on a pre-compiled exe for which I do not have the source code.

Link #2 from wiki.tesnexus.com says that the limitation "also applies to Operating Systems (OS) that are acting 'Windows-like', such as the Apple MacIntosh with OSX and Unix style platforms using 'WINE'. " but does not go into any technical details specific to Linux/Wine that I could see.

Link #3 - seems to suggest there is a regedit setting for VIDEO mem but not for application mem.

Link #4 - TBH, I just found this and haven't had a chance to test it but it seems to be suggesting that setting my wine config to winxp might make this work. When I get a chance, will update if this fixes.

References:
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Re: Wine and 32-bit applications virtual address space

Post by zpangwin »

Sorry, not seeing any option to edit my previous post. Wanted to confirm that the versions I had guesstimated earlier were correct and to provide a summarized version which I should have done in my OP.

Summary:

A few questions:
1. Can anyone verify if Wine on 64-bit system with 4+GB RAM will automatically ignore windows 32-bit exe headers with /LARGEADDRESSAWARENESS (LLA) set to No at compile time (the default). In other words, does Wine always let 32-bit exe's on 64-bit systems use more than the default 2GB?

2. If answer to #1 is "No", is there any option in Wine to do so, either for a specific application or for the entire wine prefix?

3. If both #1 and #2 are "No", is anyone aware with a method to update the header LLA / 3GB flags from Linux without needing a Windows image? (note: I tried installing vc2010express via winetricks and then using the DUMPBIN and EDITBIN exes from winconsole but they would just exit without doing anything).

Update to my wine settings:

Code: Select all

$ wine --version
wine-4.14 (Staging)

$ winetricks --version
20190615-next - sha256sum: 1526e5e169345ebfd55ca7ef56cd5ebb002ea58c5721e35ba053911fb8bc3d51

$ find /gaming/steam/steamapps/common -maxdepth 1 -iname '*Proton*'
/gaming/steam/steamapps/common/Proton 4.2
User avatar
dimesio
Moderator
Moderator
Posts: 13201
Joined: Tue Mar 25, 2008 10:30 pm

Re: Wine and 32-bit applications virtual address space

Post by dimesio »

1. No
2. Not in vanilla Wine, but according to https://bugs.winehq.org/show_bug.cgi?id=34658#c24, Proton has an option to do that.
3. There are various Windows tools to change the LARGEADDRESSAWARE flag in PE executables. One that apparently works in Wine is mentioned in https://bugs.winehq.org/show_bug.cgi?id=33858.
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Re: Wine and 32-bit applications virtual address space

Post by zpangwin »

dimesio wrote:1. No
2. Not in vanilla Wine, but according to https://bugs.winehq.org/show_bug.cgi?id=34658#c24, Proton has an option to do that.
3. There are various Windows tools to change the LARGEADDRESSAWARE flag in PE executables. One that apparently works in Wine is mentioned in https://bugs.winehq.org/show_bug.cgi?id=33858.
Thanks for the fast reply! This is exactly was I was looking for :D

Will check those out and reply back if I find anything helpful to add for others who might stumble on this later looking for the same thing.
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Re: Wine and 32-bit applications virtual address space

Post by zpangwin »

Quick Update for anyone else coming in from google:

I have been successfully using the third option mentioned above: creating a 32-bit wine prefix, copying in both the ntcore executable and the game executable, running the ntcore app then copying it back. In the case of Fallout3, when I did a binary comparison between the exe modified by ntcore in wine vs the one modified in Windows 7 using nexus mods utility I reference in my original post both were identical -- even running sha256sum gave the same output for both.

For those who wish to be lazy / see actual examples (possibly future me)*:
note that this assumes you have wine and winetricks installed already and that you know to change the paths/exe name before running the commands

Code: Select all

#You should change these paths before running!
NAME_OF_EXE_TO_PATCH="Fallout3.exe";
PARENT_DIR_OF_EXE_TO_PATCH="/path/to/your/SteamOrGOG/Fallout 3 goty";
PARENT_DIR_OF_WINE_PREFIXES="/media/games/lutris";

#path to the prefix we will create
WINE_32PFX_DIR="${PARENT_DIR_OF_WINE_PREFIXES}/ntcore_4gb_patcher";
NTCORE_DIR="${WINE_32PFX_DIR}/drive_c/ntcore_4gb_patch";

mkdir -p "${WINE_32PFX_DIR}" 2>/dev/null;
env WINEPREFIX="${WINE_32PFX_DIR}" WINEARCH=win32 wine wineboot;

#not sure if any of these are actually needed; was just trying to cover a few bases
#if you get errors you can probably ignore or at most have mono/some dot net version
env WINEPREFIX="${WINE_32PFX_DIR}" winetricks cmd comctl32 corefonts dotnet45 msxml6 vcrun2017 win7;
mkdir -p "${NTCORE_DIR}";
cd "${PARENT_DIR_OF_EXE_TO_PATCH}";

#make a backup first then copy to working dir (NTCORE_DIR)
cp -a "${NAME_OF_EXE_TO_PATCH}" "${NAME_OF_EXE_TO_PATCH}.original";
cp -a -t "${NTCORE_DIR}" "${NAME_OF_EXE_TO_PATCH}";
cd "${NTCORE_DIR}";

wget https://ntcore.com/files/4gb_patch.zip;
unzip 4gb_patch.zip;
rm 4gb_patch.zip;

#open either wine console or cmd. I prefer cmd but have seen some cases where it didn't work.
env WINEPREFIX="${WINE_32PFX_DIR}" wine cmd;

#example: wineconsole - not needed if cmd works for you
#env WINEPREFIX="${WINE_32PFX_DIR}" wine wineconsole;
Then from the cmd/wineconsole window:
note: change the executable name as needed (I don't believe linux shell vars work in cmd tho)

Code: Select all

cd /d "C:\ntcore_4gb_patch"
4gb_patch.exe Fallout3.exe
exit
Finally, back in the original terminal window (or a new one where the paths have been redefined):

Code: Select all

# copy the modified exe back to the game directory
cd "${NTCORE_DIR}";
cp -a -t "${PARENT_DIR_OF_EXE_TO_PATCH}" "${NAME_OF_EXE_TO_PATCH}";

# confirm that it was modified
# expected result: checksums of x.exe and x.exe.original should be different
cd "${PARENT_DIR_OF_EXE_TO_PATCH}";
sha256sum "${NAME_OF_EXE_TO_PATCH}"*;


== THAT SAID ==
I realize this is probably not a desirable long-term solution (especially for Linux) as:
  • It relies on a closed-source, third-party utility that may or may not have non-free licensing and may or may not have limited availability at some point depending on things like the author's discretion, hosting, your monthly data cap, etc.
  • It requires a process which must be repeated for each exe rather than simply setting an envvar / passed flag / config setting / etc. I could see where this might be confusing or difficult to manually apply in some scenarios especially if one exe calls another as the user may not be aware of secondary calls, whereas an option in wine could theoretically be made to work for all calls generated from that particular instance of wine.
The good news is that some folks out there already appear to be thinking in this direction. In addition to what is mentioned in the 2nd link that dimesio shared (PROTON_FORCE_LARGE_ADDRESS_AWARE=1 option in latest Proton versions after around Dec 2018), I also saw in the latest Lutris release notes here that they mention "WINE_LARGE_ADDRESS_AWARE is now set to 1 when DXVK or D9VK is enabled (only works with lutris-provided builds) to workaround crashes in 32-bit games." I have not tested this new feature yet but it sounds exciting (at least to me).

I did not get any matches when I grepped for "WINE_LARGE_ADDRESS_AWARE" against the official wine stable or wine-staging sources. I later found an issue tracker note for lutris that says "It should also be noted that this envar only works for lutris, ge and tkg builds. Neither vanilla wine or wine-staging support it." I assume ge is referring to one of the GloriousEggroll builds. I am not sure what tkg builds are or on the likeliness of the new flag making it back upstream to the official wine/wine-staging repos but would definitely be interested if that were to happen.
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Re: Wine and 32-bit applications virtual address space

Post by zpangwin »

Another quick update:

In evaluating whether I want to play FO3 directly vs. running the FO3 story in FO:NV (via Tales of Two Wastes mod), I noticed that the ntcore tool mentioned above does not appear to work correctly for FO:NV (via Wine) despite same thing working fine for FO3. This is probably a pretty decent reason for the community to consider implementing LLA overrides directly in wine as opposed to relying on various patching tools (assuming we really want to be able to run completely independently without needing windows). The ntcore tool appeared to work the same as for FO3 when I ran it in wine/cmd, but upon replacing the exe in the game dir and trying to run via steamwine, I kept getting a steam pop-up saying the game failed to start (hadn't even got to mods / fose_loader yet; just vanilla with patched exe). Reverting the exe to the (unpatched) backup I was able to play again.

Anyway, my current recommendation for quick and easy Wine-based FO:NV 4gb patching is to use the FNV 4GB Patcher by RoyBatterian instead since that one specifically mentions that it is "Linux/WINE compatible" (worked fine for me first try). I ran it in the same wine prefix I used for ntcore util. After patching, I was able to run with nvse_loader.exe.

For the long game of wine-/proton-based implementations....

I have yet to test PROTON_FORCE_LARGE_ADDRESS_AWARE=1 / WINE_LARGE_ADDRESS_AWARE=1 options. I assume for the Proton one, you just set it in steam > library > game > properties > launch options as

Code: Select all

PROTON_FORCE_LARGE_ADDRESS_AWARE=1 %command%
BUT... since I had already downloaded/installed all of my bethesda games under steamwine in lutris before I even started messing with LLA patching, I probably won't be evaluating that one anytime soon. But since I'm already using lutris, the WINE_LARGE_ADDRESS_AWARE=1 sounds fairly easy to setup. If I am understanding correctly, then I just need to:
  • Make sure Lutris v0.5.3 or newer is installed (done)
  • Click the Lutris button (orange button, top-left of app) > Manage Runners > scroll to Wine > Click the green installer button > Install one of the latest "lutris, ge [or] tkg builds" (done... although it was mostly the "shotgun approach" lol)
  • Right-click game > Configure > Runner options > Wine version: select one of the latest "lutris, ge [or] tkg builds" > then enable either DXVK or D9VK.
Now I probably need to confirm that the specific wine version I select in that last step has the change I want before I try actually testing it out. But the part I'm stumped on is how do I go about verifying if the LLA option is working or not?

The only thing I can think of for testing is:
1. Keep going with the patched version for now until I get a stable but highly modded setup that I'm happy with
2. Find some program/write a script to log memory usage by process
3. Using the mods to create a high memory load, compare mem usage under FNV 4GB patched exe vs. vanilla exe (no LLA option) vs. vanilla exe (LLA option)
4. ???
5. Profit (aka enjoy)
haarp
Level 2
Level 2
Posts: 17
Joined: Sun May 18, 2008 1:22 pm

Re: Wine and 32-bit applications virtual address space

Post by haarp »

I've noticed that Proton's and Lutris' Wine seem to have flags to enable LAA. After some investigation, I found the patch that enables this:

https://github.com/Frogging-Family/wine ... /LAA.patch
https://github.com/Frogging-Family/wine ... ging.patch

with that, it should be possible to set WINE_LARGE_ADDRESS_AWARE=1 as an env variable and remove the need to patch binaries with 4gb_patch.exe. I haven't tested it myself yet, tho.
User avatar
zpangwin
Level 1
Level 1
Posts: 6
Joined: Sun Aug 25, 2019 10:46 am

Re: Wine and 32-bit applications virtual address space

Post by zpangwin »

haarp wrote: Wed Aug 05, 2020 5:04 am I've noticed that Proton's and Lutris' Wine seem to have flags to enable LAA. After some investigation, I found the patch that enables this:

https://github.com/Frogging-Family/wine ... /LAA.patch
https://github.com/Frogging-Family/wine ... ging.patch

with that, it should be possible to set WINE_LARGE_ADDRESS_AWARE=1 as an env variable and remove the need to patch binaries with 4gb_patch.exe. I haven't tested it myself yet, tho.
I think you are correct and I see the System Options tab has a section for adding Environment Variables to a Lutris Runner but I have not tested it yet either. Have mostly been using Steam/Proton lately but I never did finish my FO:NV playthrough so I might go back and test this at some point in the future.

I guess for testing, I could first drop to virtual terminal (Ctrl+Alt+F2) and monitor the memory of vanilla game (via htop or logging ps output etc) over the course of 30-60 mins. then close, add lots of mods and relaunch. confirm via ps / pgrep that the environment variable is set correctly. then just continue monitoring to see if it goes over the 2GB threshold. Not really sure how else to verify other than see it use more than 2GB mem.

Will try to report back if I get around to testing
Locked