As a maintainer, how to deal with a non-reproducible test?

Post issues, bugs, and feature requests about the various WineHQ websites to this forum.
Locked
thePalindrome
Newbie
Newbie
Posts: 1
Joined: Wed Feb 27, 2019 5:59 am

As a maintainer, how to deal with a non-reproducible test?

Post by thePalindrome » Wed Feb 27, 2019 6:05 am

The game that I became maintainer of recently got a test submitted giving it a bronze rating for severe issues, but when I attempted to verify the results, I could not.

The three things I noticed that could explain it are:
  • They were using 4.0, and I was on 4.1
    They were using Proprietary Intel drivers (less important because...)
    They are using OSX, and I am on Linux
Assuming the issue isn't just the first one, how should I process it? My current plan is to submit, then immediately post my own results, but I have some concerns that this may be a "wine-with-intel" bug or a "wine-with-osx" bug, neither of which I can properly test for.

I appreciate your guys' help,
Alex

User avatar
dimesio
Moderator
Moderator
Posts: 12692
Joined: Tue Mar 25, 2008 10:30 pm

Re: As a maintainer, how to deal with a non-reproducible tes

Post by dimesio » Wed Feb 27, 2019 7:04 am

All three of the factors you list could account for the difference; that's why we ask for that information. As maintainer, you are not expected to reproduce all problems. Each test report represents that person's experience, and nothing more. Simply check that the form is correctly filled out, sufficiently detailed, internally consistent, and adheres to the test report guidelines. If it is, accept it.

Note that while maintainers are expected to periodically submit their own reports, it is not your job to try to suppress ratings you cannot reproduce because of differences in the underlying system. If the performance in 4.0 on a Mac is bronze, Mac users need to know that.

Locked