Correct wrong character set?

[cnbiz850]

A Wine'd program lets me create/edit some script for it to execute. All I need to do is click on 'Edit Script', then it pops up its own little editor for me to type in. Strange things is after I finished editing something and save, it complains that some character is wrong. I can't see/find anything wrong. I saved it anyway and reopen it without changing anything, then the program is OK with it and the script actually executes fine. Seems most likely it has to do with line-change character because if I just edit some letters on a same line, it seems OK. Anyone please advise what to do to correct the problem?

[DaVince]

Are you editing UNIX files in a Windows program? In that case, it might be adding the "carriage return" character at the end of every line once you save it. Why? Well, here's why.

Text files in Windows and Linux have a very minor difference: the way they store "newlines". Linux uses the LF character (line feed). Windows uses TWO characters to indicate a line break: CR (carriage return) and LF (line feed).

Usually, when you try editing a text file that has only LF characters as newlines, Windows apps will behave differently depending on whether they support "seeing" this character:
- Notepad doesn't recognize them as newlines at all and glues all text together;
- Some editors recognize it correctly, then convert to CR+LF once you save the file;
- Some other editors recognize the file correctly and let you save back to whatever format you were using (CR+LF, just LF, or just CR in case of Macs).

Now, I'm not sure about this, but I think what is happening is you have an editor of the middle type - and it warns you about those UNIX-type newlines and converts them to Windows-type newlines.

[cnbiz850]

Thanks very much. It makes a lot of sense.

The editor it provides doesn't correct the problem when I save. The reason I know is because the script doesn't execute after my initial save. I have to reopen the script and save it again to get away with the problem. It seems the editor corrects it on the reopen of the script, because when clicking on "check grammar" at that time it shows OK.

Is there a way to add the additional characters during editing (put in the right line-change character for Windows)? See I am using Linux and the editor is from the Windows program running via Wine on the Linux system.

[Martin Gregorie]

On Fri, 2010-04-16 at 07:27 -0500, DaVince wrote:

Are you editing UNIX files in a Windows program? In that case, it
might be adding the "carriage return" character at the end of every
line once you save it. Why? Well, here's why.

Text files in Windows and Linux have a very minor difference: the way
they store "newlines". Linux uses the LF character (line feed).
Windows uses TWO characters to indicate a line break: CR (carriage
return) and LF (line feed).

Usually, when you try editing a text file that has only LF characters
as newlines, Windows apps will behave differently depending on whether
they support "seeing" this character:
- Notepad doesn't recognize them as newlines at all and glues all
text together;
- Some editors recognize it correctly, then convert to CR+LF once
you save the file;
- Some other editors recognize the file correctly and let you save
back to whatever format you were using (CR+LF, just LF, or just
CR in case of Macs).

If, for some reason you want to use a Windows editor under wine to edit
both Windows and Linux files I can recommend PFE (the Programmers File
Editor, http://www.lancs.ac.uk/staff/steveb/cpaap/pfe/ ). Its a good,
well featured text file editor that can edit as many files
simultaneously as you'd ever need as well as acting as a primitive IDE,
but the reason I'm mentioning it is that it can edit both Windows and
Unix/Linux files and use the appropriate newline for each file.

If you don't mind using different editors but want them all to work just
the same then microEmacs (aka uemacs) is the answer. Its available for
many operating systems including Linux, DOS (where it runs in a console
window) and Windows, is readily customised and has the same look and
feel everywhere. http://www.aquest.com/emacs.htm It suits me so well
that I install it everywhere rather than learning a new editor. It is
written in C and has no relation to emacs apart from the name!


Martin

[cnbiz850]

Thanks for tips regarding different editors to use.

Well, in my case, I have to use the editor provided by this Windows program, because it then saves it in its native format (not ASCII). I won't somehow I can type the CRLF on my Linux keyboard.

A philosophical question: should Wine actually be the one to handle this issue because that is what makes possible to run Windows programs on a Linux system?

[Martin Gregorie]

On Fri, 2010-04-16 at 08:23 -0500, cnbiz850 wrote:

Is there a way to add the additional characters during editing (put in
the right line-change character for Windows)? See I am using Linux
and the editor is from the Windows program running via Wine on the
Linux system.

Yes. There are two command line utilities, dos2unix and unix2dos that
convert the newlines used in the file. If you supply a file name as the
only argument the file is converted in situ.

They can do rather more than that too - see the manpage for details.


Martin

[Martin Gregorie]

On Fri, 2010-04-16 at 08:49 -0500, cnbiz850 wrote:

Thanks for tips regarding different editors to use.

Well, in my case, I have to use the editor provided by this Windows
program, because it then saves it in its native format (not ASCII). I
won't somehow I can type the CRLF on my Linux keyboard.

A philosophical question: should Wine actually be the one to handle
this issue because that is what makes possible to run Windows programs
on a Linux system?

Well, you can go that way and use the wordpad that comes with wine, but
there are a lot of better editors out there.

PFE and uemacs can both be configured to set the tab width, decide
whether to convert tabs to spaces, whether to use auto indentation and
what newline to use depending on the file extension.

I suppose you know that changing the $EDITOR environment variable you
can alter the default editor? If you were feeling really cute you could
set $EDITOR to run a shell script that looks at file details such as the
extension or which directory its in and uses that information to decide
which editor to use - including Windows editors under wine.

Also, its worthwhile learning at least the basics of using vi. Not that
its such a wonderful editor, but it can often dig you out of a bad hole
because it can cope with almost any terminal type and can be used with
just the printable text keys plus Enter, Esc and Ctrl. Last but not
least, it is always available on machines running Linux or some form of
Unix.


Martin

[cnbiz850]

Martin,

Thanks for the response.

But sorry that was not what I was asking.

The Windows program I am running with Wine provides an editor of its own. I need to use that because only it then can save into the proprietary (non-ASCII) format the program needs to use. Of course I could use other editors to edit but will have to copy/paste the text into the program's editor. A question for that is whether the CRLF get copied/pasted correctly.

I hope if there is a way for me to use the Linux keyboard to type correctly into the program's editor.

[Martin Gregorie]

On Fri, 2010-04-16 at 10:20 -0500, cnbiz850 wrote:

The Windows program I am running with Wine provides an editor of its
own. I need to use that because only it then can save into the
proprietary (non-ASCII) format the program needs to use.

Yes, I understood that, but not how the editor is launched: is it
launched by the program that needs the special file content or
separately?

If the latter, my script suggestion might help. As an example, this
(tested) script normally edits the file with gedit but if the file is in
$HOME/utils/scripts it uses vi. The magic on line 6 works because
"readlink -f" returns the absolute name of the file and dirname chops
off the file name, leaving just the directory path:

========================= tricked ==============================
#!/bin/bash
if [ -z "$1" ]
then
gedit $1
else
path=$(dirname $(readlink -f $1))
if [ "$path" == "$HOME/utils/scripts" ]
then
vi $1
else
gedit $1
fi
fi
====================== end of tricked ==============================

It runs vi regardless of whether you type something like:

tricked utils/scripts/installscript

or this:

cd utils/scripts
tricked installscript

It does the same regardless of whether you're using the editor to create
a new file or edit an existing one.

If no filename is given it always runs gedit.

I hope if there is a way for me to use the Linux keyboard to type
correctly into the program's editor.

Why not? AFAIK the keyboard generates the same output regardless of
whether the keystroke consumer is a Linus program or a Windows program
running under wine.


Martin

[cnbiz850]

Martin,

Thanks for all the good suggestions.

Unfortunately, the editor is launched from within the Windows program.

The use of an outside editor in Linux that automatically puts the Windows required CRLF might be a good idea. Then I copy/paste the text into the editor provided by the program. I guess the copy/paste would also copy/paste CRLF. The question is what editor is the best choice for that.

My philosophical question again: should Wine take the responsibility in putting the correct newline character? Seems to me that would solve the whole thing.

[DaVince]

If your file is not actually written in any sort of normal text format, it wouldn't write any CR/LF line encodings either as those are special characters in the ASCII (and base Unicode) character set.

Unless the resulting file will be a binary file with some internal ASCII/Unicode strings for the text...

[cnbiz850]

If your file is not actually written in any sort of normal text format, it wouldn't write any CR/LF line encodings either as those are special characters in the ASCII (and base Unicode) character set.

Unless the resulting file will be a binary file with some internal ASCII/Unicode strings for the text...

Understood.

Maybe I should put it this way. The script I edit is a formated language (like C). Before it can be executed, the Windows program - by using its provided editor - somehow processes/compiles it into its needed proprietary file format. I think at the time of the processing/compiling, it checks the ASCII text I put in and verifies that it is in the legal format. So that is when it complains about the CRLF related problem.

Through our discussion here, I take it that because I run the Windows program through Wine, the newline character I type in is a LF rather than the required CRLF, then the program complains about it. Guess I am more and more inclined that Wine should take the responsibility in putting in the right newline character.

Think about how the LF character get into the editor of my Wine'd program. When I type 'Enter', it is interpreted as a LF character by Linux and the LF is then passed to Wine, Wine then pass it down to the editor. In this process, the CR required in Windows is nowhere to be generated, and so the editor complains when it tries to process/compile the script. Is this the right thinking?

[Martin Gregorie]

On Fri, 2010-04-16 at 20:39 -0500, cnbiz850 wrote:

Maybe I should put it this way. The script I edit is a formated
language (like C). Before it can be executed, the Windows program -
by using its provided editor - somehow processes/compiles it into its
needed proprietary file format.

So your editor is more than just an editor: it is a compiler as well and
saves both the ASCII text file and the compiler output.

I think at the time of the processing/compiling, it checks the ASCII
text I put in and verifies that it is in the legal format. So that is
when it complains about the CRLF related problem.

Or more likely, it doesn't recognise the LF as a newline.

Through our discussion here, I take it that because I run the Windows
program through Wine, the newline character I type in is a LF rather
than the required CRLF, then the program complains about it.

Not exactly. The keyboard emits a scan code which MS operating systems
keyboard drivers pass to the program as CR and the Linux keyboard driver
passes as LF. Most DOS/Windows programs won't recognise LF as a newline,
but often treat it as an ordinary non-printing character. If you're
using C a further library function ensures that this always matches '\n'
on input and that on output '\n' becomes CRLF or LF. Other languages
will do something similar.

Its actually quite a mess on the MS side, with newline being CR from the
keyboard but CRLF when read from a file and always being written as
CRLF.

Guess I am more and more inclined that Wine should take the
responsibility in putting in the right newline character.

I'd expect that Wine would handle the conversion of keystrokes but would
pass files unchanged. Anything else would cause major problems. Think of
the effects if it converted LF to CRLF in an exe or other binary file.
IOW it would have to be able to distinguish text files from other file
types with 100% accuracy which would be nearly impossible.

I'd suggest that your best bet is a simple script like this:

#!/bin/sh
dos2unix $1
gedit $1 # Substitute your favourite Linux editor here
unix2dos $1

The double conversion is a good idea because some Linux editors don't
treat CRLF as a simple newline. BTW, you can run a file through unix2dos
more than once without damaging it.

Another thought: have you looked for a Linux equivalent to your
editor/compiler? There may be one. For example, the Parallax STAMP
program editor does pretty much what you're describing: it holds program
source as a DOS text file but downloads a tokenised form into the STAMP
to be run. There is an equivalent Linux development package consisting
of a tokeniser and a downloader/debugger. I haven't used it, but I
imagine that it uses standard Linux text editors to write programs.


Martin

[cnbiz850]

Anyone disagree that Wine should take care of adding in the right newline characters (CRLF) in a Wine'd editor (like for instance, Notepad)?

The reason I propose that is because anyone who uses a wine'd editor to edit something intends to use the outcome in the Wine'd (Windows) environment. As Windows require CRLF as the newline, Wine simply passing through a Linux LF into the editor is not acceptable.

[vitamin]

Wine should take care of adding in the right newline characters (CRLF) in a Wine'd editor (like for instance, Notepad)?

No, they shouldn't convert anything. That's broken by design. Only MS can create programs that automatically alter files without even asking you. They of course ask if you want to write magically modified files back.

[cnbiz850]

Wine should take care of adding in the right newline characters (CRLF) in a Wine'd editor (like for instance, Notepad)?

No, they shouldn't convert anything. That's broken by design. Only MS can create programs that automatically alter files without even asking you. They of course ask if you want to write magically modified files back.

It's not a real matter of conversion, but of to what format.

First, Wine is designed to run Windows programs. Regardless how good/bad MS Windows is, Wine should try to be just the same as the that.

Second, in the case of editing/creating something in a Wine'd editor, the ultimate question to be answered to whether the file being edited is a Linux file or a Windows file. The obvious answer is that the outcome from a Wine'd editor should be a Windows file - why would someone use a Wine'd editor to edit a non-Windows file.

[vitamin]

Second, in the case of editing/creating something in a Wine'd editor

It already works at is should. Go create a file in Wine's notepad and look at it. It will have lines terminated with CRLF.

[cnbiz850]

Second, in the case of editing/creating something in a Wine'd editor

It already works at is should. Go create a file in Wine's notepad and look at it. It will have lines terminated with CRLF.

OK. But I guess it is a function of Notepad rather than Wine itself. I have used two editors that both come with Windows based stock analysis programs. One is more tolerate without complaining about this issue. But the other (the subject of this thread) has issues with it. The conclusion we can draw from this is that in this particular aspect, the Wine'd environment is not on par as it should be.

[Martin Gregorie]

On Sun, 2010-04-25 at 19:57 -0500, cnbiz850 wrote:

It's not a real matter of conversion, but of to what format.

It gets worse than that, because what your program sees when keys are
pressed also depends what DOS/Windows compiler you're using:

- the DJGPP port of gcc offers near-perfect source code compatibility
with the Linux port of gcc but its support libraries return LF if
ENTER is typed and zero if function or cursor keys are typed.

- Borland C 4.5 is almost totally incompatible with gcc at the source
code level (it uses non-standard keyboard access functions) but it
does return CR for ENTER. The only way to read function and cursor
key presses is to look at the keyboard flags register and retrieve
and use scan codes.

I'm certain there are other variations generated by other compilers, but
these are the DOS/Windows compilers I know. Anyway, that should be
enough to give you the picture...


Martin