Debugging wine under qemu-i386 on a Raspberry Pi

Questions about Wine on Linux
Locked
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

I've got wine 6.0 running win32 programs on Raspberry Pi (latest rasberry pi OS) in an i386 chroot jail of debian buster built with debootstrap.

I'm using qemu in linux-user mode, but had to custom-compile qemu-5.2.0, relocating the text segment to 0xa0000000 to keep qemu code out of memory used by wine, but it may be possible to achieve the same effect using the -B and -R options to qemu instead.

I still need to disable the preloader (by renaming it) because it crashes. More accurately, the executed program crashes. The preloader itself completes OK but someting about the memory map created causes problems within the qemu/wine combination. Running a normal program like 'ls' under has no negative effect on the run progrm, though. Mind, i can't get the preloader to find wine_main_preload_info in arbitrary executables I build, so there are other things different about the way wine is linked that i'm not accounting for.

It works pretty well, but i've found 2 16-bit programs that crash wine consistently. Unfortunately one of them is an installer for a program I particularly wanted to test. Note that neither program crashes on the same wine build run under a debian chroot jail on an i386 ubuntu VM. Other win16 programs do seem to work.

I want to debug this, but i'm running into limitations at every level.

1) winedbg won't run properly under QEMU due to lack of ptrace support
2) there doesn't seem to be a way to use winedbg with target remote <IP:port> to connect to qemu's built in debug capability
3) GDB run against qemu's debug port can't see any symbols when it gets a SEGV, loading the wine excutable with file doesn't help. Neither does loading ntdll.so or winevdm.exe.so
4) qemu strace and exception reporting flags show nothing other than the SEGV. The crash is not in a qemu syscall function.
5) WINEDEBUG=+all shows the last call to NtCreateEvent but that doesn't seem to be the source of the SEGV. I could be wrong.
6) Disassembly at the exception address shows code that seems completely suspect, as if we branched to an invalid code location.

Note that qemu itself does not crash, it generates a SEGV for the target program. A qemu crash I could debug in GDB.

Does anybody have any suggestions of how to proceed with investigating this?
nru1979
Level 2
Level 2
Posts: 16
Joined: Thu Nov 05, 2020 3:23 am

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by nru1979 »

I do not know if it will answer your question, but I have a little experience debootstrapping:

- Debootstrapping on a buster/buster often generates bus error crashes. It is a problem of qemu memory allocation on this system. Only workaround I have found is to install a stretch/stretch.

- Debootstrapping has a poor sound support. I have noticed that some runnables that stop or crash would run fine if I winetricks sound=disabled. The only way I found to get some sound is a pulseaudio client-server.

- Don't forget 3G/1G if you have an early Pi (I think it is not necessary on a Pi4). I have uploaded 3G+/1G tar.gz for 3B+ on this blog post: https://thepigamer.blogspot.com/2021/01 ... es-on.html Search vercingetorix in the blog post.

- If your final objective is to make things run on your Pi, you will probably have much more success trying box86. It is an integrated x86 emulator which does not need debootstrapping. (A bit like exagear). It is very alive and almost has an update every day. I have been able to run Fallout 2, HoMM4, HoMM3, RPG Maker VX Ace (win32api) and loads of other goodies without problem and with sound. Check ptitSeb's git, its worth it!
https://github.com/ptitSeb/box86

Don't dump Qemu anyways, because for things like winetricks, I use Qemu (I winetricks all my wineprefixes inside qemu even the ones I use in box86)

I hope that can help, but good luck anyways!
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

I'm using a pi4 so the 3G/1G memory split shouldnt be an issue.

Thanks for the suggestion of using box86- I'll try it out.
However, i'd still like to figure out how to debug my qemu setup.

I now know that the preloader somehow causes one of the initial mmaps to be created from 0x10000-0x7d400000 instead of 0x00010000-0x68000000, which interferes with the loading of ntdll.so at 0x7bc00000.
I have no idea why because qemu-i386 -strace reports the mmap2 syscall correctly as:

10294 mmap2(0x00110000,1743716352,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED|MAP_NORESERVE,-1,0) = 0x00110000

which is no different to mmap2 on a native i386 system:

mmap2(0x110000, 1743716352, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x110000

I dont think the preloader has anything to do with my segfault, as the applicationa which crash in qemu work fine without the preloader in a native i386 kernel.
nru1979
Level 2
Level 2
Posts: 16
Joined: Thu Nov 05, 2020 3:23 am

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by nru1979 »

Maybe addr2line -e /usr/bin/qemu-user-static -a 0xxxxxxxx (being segfault adress)
and addr2line -e /usr/bin/wine -a ....
can give you a hint?

if not, box86 allows preloader to be present, and you might get another addr2line
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

Just for completeness, i tried box86 with wine and although it's wonderfully fast for win32 apps, every win16 app I try fails with a series of:
modify_ldt: Invalid argument
errors, followed by:
wine: Unhandled page fault on execute access to 03270083 at address 03270083 (thread 0088), starting debugger...
(the addresses vary).

I am led to believe that box86 does not support 16-bit apps.

I will try addr2line.
nru1979
Level 2
Level 2
Posts: 16
Joined: Thu Nov 05, 2020 3:23 am

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by nru1979 »

Hello,

You might want to raise an issue on Github for that 16 bit problem.

ptitSeb is very active and box86 has made terrific progress for the past 2 months that I started to follow it.

As for the addr2line, you will have to raise an issue (on qemu's git?) too. it is an information for the developper.

If you are lucky enough, addr2line will give you an obvious link to a driver or a setting. But that is quite rare.

Good luck anyways!
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

Sorry, ptitSeb has confirmed that there is no 16-bit support in box86 at present (and it's not being worked on currently).
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

OK a little update. I've managed to use /proc/<pid>/maps and add-symbol-file in gdb to get a partial backtrace.
I had to manually find the mmaped executables/libraries in the maps and add the offset in gdb using the -o option to add-symbol-file.
This is with a remote GDB connection to qemu.

The 'weird' diassembly was the machine code for _dl_sysinfo_int80().

It just doesn't make much sense to me:

(gdb) bt
#0 0x40802092 in _dl_sysinfo_int80 ()
#1 0x40951d53 in __GI___select (timeout=0x0, exceptfds=0x0, writefds=0x0, readfds=0x0, nfds=0)
at ../sysdeps/unix/sysv/linux/select.c:41
#2 __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:37
#3 0x40a92632 in NtDelayExecution (alertable=<optimized out>, timeout=<optimized out>) at dlls/ntdll/unix/sync.c:1328
#4 0x40a6218e in __wine_syscall_dispatcher ()
#5 0x7bc0bb0c in ?? ()
#6 0x40d905e6 in __wine_spec_pe_header ()
#7 0x0013c91b in ?? ()
#8 0x40d91bcb in __wine_spec_pe_header ()
#9 0x00000000 in ?? ()

The problem is, looking at the wine source for NtDelayExecution, I just don't understand how select(0, NULL, NULL, NULL, NULL) can segfault. Unless the segfault was generated by another thread while this one was paused in select() and qemu displatched the SEGV to the wrong/a different thread??
nru1979
Level 2
Level 2
Posts: 16
Joined: Thu Nov 05, 2020 3:23 am

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by nru1979 »

Out of my competence,
Best track I still have would be to report a bug on qemu. I do not think this is a wine issue...
See launchpad: https://bugs.launchpad.net/qemu/

Good luck
Dirk Niggemann
Level 1
Level 1
Posts: 8
Joined: Tue Jan 21, 2020 4:04 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by Dirk Niggemann »

OK, an update.

It's a QEMU bug. The backtrace is misleading. The SEGV was delivered to the paused thread. The other thread is running in 16-bit segmented mode, which confuses GDB. However, when you disassemble the running 16-bit program, it turns out the faulting intruction is CD 21 (int 21h). i.e. an attempt by a 16-bit program to access DOS services. This should result in a GPF SEGV which wine catches in the structured exception handler and processes as a call to its internal int21 handler in krnl386.exe16.

The problem is that QEMU doesn't always populate the sigcontext struct returned in a signal handler correctly. For certain cases (and this seems to be specific to faults in 16-bit mode) it leaves the trapno field populated with -1, which confuses wine into simply aborting the 16-bit thread rather than calling the int21 handler.

You can sort-of fix this in qemu by forcing the trapno field to be set to a GPF exception if it's left at -1 in setup_sigcontext() (linux-user/i386/signal.c) but that's a pretty dirty workaround to some underlying flaw in QEMUs exception logic for i386.
WheezyE
Level 1
Level 1
Posts: 6
Joined: Sat Dec 19, 2020 9:53 pm

Re: Debugging wine under qemu-i386 on a Raspberry Pi

Post by WheezyE »

Hey Dirk. I recently ran into the wine memory allocation problems that you mentioned with qemu-i386-static chroot on an RPi4. Running `wineboot` gives me "preloader: Warning: failed to reserve range 00010000-00110000" and "vm.mmap_min_addr=0" wine bug fixes inside my chroot don't work to resolve them. Would there be any chance I could get a script or instructions on how to custom-compile qemu with those memory addresses open like you did?

I'm not quite sure what you meant by the -B or -R options in qemu (I don't see these options under `./configure --help`). I've been trying for a few years to get Winlink (32 bit) running on a Raspberry Pi for ham radio purposes (low power portable email over radio) and would love to know more about your endeavors.

I don't want to hijack your thread, but my github project page is github.com/WheezyE/Winelink if you wanted to reply there in an issue tracker.

In case your 16 bit installers are still giving you heck, I also know of a trick where you can install everything in a wineprefix on another computer, tar the prefix on that computer (to preserve symlinks), copy to a flashdrive, then copy to your Pi & untar the wineprefix there. This "transplanting" or "side-loading" can get around the installer issues while still letting you run the software on your Pi inside your chroot.
Locked