Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id GAA04344 for ietf-xml-mime-bks; Fri, 24 Mar 2000 06:26:15 -0800 (PST) Received: from prserv.net (out1.prserv.net [32.97.166.31]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id GAA04340 for ; Fri, 24 Mar 2000 06:26:13 -0800 (PST) Received: from t3knz.attglobal.net ([210.88.161.179]) by prserv.net (out1) with SMTP id <2000032414280525202g3emde>; Fri, 24 Mar 2000 14:28:05 +0000 Message-Id: <200003241427.AA02055@t3knz.attglobal.net> From: MURATA Makoto Date: Fri, 24 Mar 2000 23:27:55 +0900 To: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 In-Reply-To: References: MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.10 Content-Type: text/plain; charset=us-ascii Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: In message "Re: Some text that may be useful for the update of RFC 2376", Rick Jelliffe wrote... >> HTTP already has the accept-charset field. I do not understand your claim. > >If I am using a DOM parser, I cannot ask it "what encodings do you >support?" If I am using SAX in Java, I can assume that the encodings >underlying Java are the ones available. I don't recall any C or C++ XML >parser that exposes this information: I don't think Expat does, for >example. Although such information is described in their manuals, I do not think that they have any APIs for "what encodings do you support?". I agree. >If my browser cannot ask its XML processor "what encodings do you >support?" in order to perform content negotiation for XML, then either the >poor user must configure it themselves or the HTTP code has to take on the >responsibility for providing transcoding services itself (perhaps not a >bad thing for the future). And configuration has to be done >application-by-application: for example James Clark's vanilla Expat did >not accept Big5, so every XML application built on it was pretty unusable >for Traditional Chinese here. Ideally, XML processors should silently provide the accept-charset field. On top of document entities, an XML processor may silently fetch external parsed entities, external parameter entities, and external DTD subsets. (I know expat doesn't, but other parsers do.) Since application programmers cannot control such fetching, the best solution is hardcode the accept-charset field in the XML processor. Certainly, the person who writes the XML processor knows which encoding is supported. (It would be great if we can register callback routines for unsupported charsets.) > >Even if you are using DOM (or >SAX) it is quite possible that the system integrator has chosen to use a >different implementation from the one which the software developer used. >So you cannot ask DOM, and the programmer cannot be sure which >implementation is being used. Right. But we can always assume that the same UCS characters will be received. >So it seems to me that content negotiation of character encoding for XML >is a bit of a myth: it requires that the user test applications rather >than it being transparent. If content negotiation is hardcoded in XML processors, application programmers do not have to worry. >That is an unreasonable and unworkable >requirement. At the moment, the browser has to guess which encodings are >available, or the poor user has to test if the local encoding is >supported. (I suppose systems could also have some automated system which >requested a big5 XML file and then tried to parse it. Not really >elegant. Presumably the XML file would have to be sourced internally. ) I am afraid that I do not fully understand your claim. Could you try again? >For content negotiation of MIME types, a browser knows which content-types >have handlers. But it doesn't know this information for character-encoding >for the XML applications it has. That is why I Probably, you sent this mail before you finished the last sentence. ---- MURATA Makoto muraw3c@attglobal.net Received: by ns.secondary.com (8.9.3/8.9.3) id JAA13627 for ietf-xml-mime-bks; Thu, 23 Mar 2000 09:10:57 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id JAA13623 for ; Thu, 23 Mar 2000 09:10:56 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id MAA08084; Thu, 23 Mar 2000 12:12:49 -0500 Message-ID: <38DA5072.61415F6F@w3.org> Date: Thu, 23 Mar 2000 18:12:18 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 References: <38D903E1.7E6BC41F@w3.org> <200003230919.AA02031@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > > In message "Re: Some text that may be useful for the update of RFC 2376", > Chris Lilley wrote... > > >Thus, for any XML file which is not encoded in US-ASCII, text/xml is an > >inappropriate choice of MIME type. Silent data corruption can and will > >occur. > > If I use UTF-8 or UTF-16 and provide the charset parameter, no data > corruption will occur. Provided that the fallback to text/plain does not occur, which cannot be guaranteed. > >For all international xml files (noting that in this context, the USA is > >international too due to the widespread use of Spanish, and the wide numbe > >rof other languages in use), a type such as application/xml is the correct > >choice, unless ther eis a more specific non-text type available. > > If an XML document is readable for casual users, satisfies > restrictions of the top-level media type "text", and does not require > special dispatching, "text/xml" is the most appropriate media type. Thats a lot of ifs. -- Chris Received: by ns.secondary.com (8.9.3/8.9.3) id IAA12658 for ietf-xml-mime-bks; Thu, 23 Mar 2000 08:11:46 -0800 (PST) Received: from gate.sinica.edu.tw (gate.sinica.edu.tw [140.109.4.130]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id IAA12654 for ; Thu, 23 Mar 2000 08:11:44 -0800 (PST) Received: from localhost by gate.sinica.edu.tw (8.9.3/8.9.3) with ESMTP id AAA22642 for ; Fri, 24 Mar 2000 00:13:34 +0800 (CST) Date: Fri, 24 Mar 2000 00:13:33 +0800 (CST) From: Rick Jelliffe X-Sender: ricko@gate To: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 In-Reply-To: <200003170736.AA01945@t3knz.attglobal.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: On Fri, 17 Mar 2000, MURATA Makoto wrote: > Rick Jelliffe wrote... > >We need a way to ensure end-to-end integrity. > > I do not agree. Why? By "we" I mean my employers and I, not "we" as in ietf-xml-mime. We need to be able to send and receive Big5-encoded XML where we have no control over the behaviour of intermediate systems. > >(and, in any case, > >there is no mechanism currently for an XML parser to feed information > >about which encodings it accepts to the HTTP system to set up the > >preferences in the first place.) > > HTTP already has the accept-charset field. I do not understand your claim. If I am using a DOM parser, I cannot ask it "what encodings do you support?" If I am using SAX in Java, I can assume that the encodings underlying Java are the ones available. I don't recall any C or C++ XML parser that exposes this information: I don't think Expat does, for example. If my browser cannot ask its XML processor "what encodings do you support?" in order to perform content negotiation for XML, then either the poor user must configure it themselves or the HTTP code has to take on the responsibility for providing transcoding services itself (perhaps not a bad thing for the future). And configuration has to be done application-by-application: for example James Clark's vanilla Expat did not accept Big5, so every XML application built on it was pretty unusable for Traditional Chinese here. Even if you are using DOM (or SAX) it is quite possible that the system integrator has chosen to use a different implementation from the one which the software developer used. So you cannot ask DOM, and the programmer cannot be sure which implementation is being used. So it seems to me that content negotiation of character encoding for XML is a bit of a myth: it requires that the user test applications rather than it being transparent. That is an unreasonable and unworkable requirement. At the moment, the browser has to guess which encodings are available, or the poor user has to test if the local encoding is supported. (I suppose systems could also have some automated system which requested a big5 XML file and then tried to parse it. Not really elegant. Presumably the XML file would have to be sourced internally. ) For content negotiation of MIME types, a browser knows which content-types have handlers. But it doesn't know this information for character-encoding for the XML applications it has. That is why I Rick Jelliffe Received: by ns.secondary.com (8.9.3/8.9.3) id GAA09850 for ietf-xml-mime-bks; Thu, 23 Mar 2000 06:01:05 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id GAA09846 for ; Thu, 23 Mar 2000 06:01:04 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id JAA21493; Thu, 23 Mar 2000 09:02:36 -0500 Message-ID: <38DA23F5.1C519480@w3.org> Date: Thu, 23 Mar 2000 15:02:29 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Meida types and stylesheet linking References: <38D8FD3C.E5ADD63C@w3.org> <200003230219.AA02016@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > > In message "Re: Meida types and stylesheet linking", > Chris Lilley wrote... > >Can you point me to some documentation that says that this usage of a MIME > >type label, outside of a content-type header, is somehow invalid or > >incorrect? > > In RFC 2045, we have the following: > > The purpose of the Content-Type field is to describe the data > contained in the body fully enough that the receiving user agent can > pick an appropriate agent or mechanism to present the data to the > user, or otherwise deal with the data in an appropriate manner. The > value in this field is called a media type. > > So, if something is not specified in the content-type field, it is > not a media type :-) No, sorry, the paragraph you quote does not establish commutativity. The valie in that field is a media type, yes. That clearly establishes that it is an error to put something else in that field. It does not establish that it is an error to use that type elsewhere in other contexts, and that is what people have been doing to good effect. The alternative being to invent a new and parallel labelling scheme, which is just a waste. So the answer to my original question above is "no, I cannot" -- Chris Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id BAA29074 for ietf-xml-mime-bks; Thu, 23 Mar 2000 01:21:17 -0800 (PST) Received: from prserv.net (out4.prserv.net [32.97.166.34]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id BAA29069 for ; Thu, 23 Mar 2000 01:21:16 -0800 (PST) Received: from t3knz.attglobal.net ([210.88.161.172]) by prserv.net (out4) with SMTP id <2000032309231023902likc8e>; Thu, 23 Mar 2000 09:23:11 +0000 Message-Id: <200003230923.AA02033@t3knz.attglobal.net> Date: Thu, 23 Mar 2000 18:23:03 +0900 To: Chris Lilley Cc: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 From: MURATA Makoto In-Reply-To: <38D906A7.8837412F@w3.org> References: <38D906A7.8837412F@w3.org> MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.10 Content-Type: text/plain; charset=us-ascii Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: In message "Re: Some text that may be useful for the update of RFC 2376", Chris Lilley wrote... > >> Unfortunately, we do not have "fairly good state of encoding declaration >> of XML files". People generate XML documents by XSLT or their own programs, >> and fail to specify the correct charset. > >That is not a problem. Such files will not be well formed and thus, will >fail toparse. You are saying that the omission of the charset parameter is a problem, and that incorrect encoding PIs are not problems. I do not know why. Many Japanese users have failed to specify correct encoding PIs, and many Japanese programmers have failed to generate them correctly. I also heard that users in developing countries copy ISO-8859-1 HTML files and mistakenly put incorrect meta tags. The same thing will happen to XML. In-band encoding is not free from errors. >> I think that you are not paying attention to other textual format. > >Oh I am, but not on this list where it is off topic. In RFC 2318 (text/css) you co-authored, the charset parameter is described as below: The syntax of CSS is expressed in US-ASCII, but a CSS file can contain strings which may use any Unicode character. Any charset that is a superset of US-ASCII may be used; US-ASCII, iso-8859-X and utf-8 are recommended. RFC 2616 (HTTP) is a draft standard and defined the default as below: The "charset" parameter is used with some media types to define the character set (section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value. See section 3.4.1 for compatibility problems. Thus, the default value of the charset parameter of text/css is ISO-8859-1. I know that CSS recommendations are different. But in the realm of IETF, the default value is ISO-8859-1. >> I would >> like XML to be a good citizen of the WWW and to establish a good practise > >As would I. I don't consider the propogation of known faults to be "good >practice". Sorry, but I have to trust W3C I18N WG, etc. >> The charset parameter >> is not a historical requirement. Rather, it is the right solution, >> which is just about to take off. I think that we are wasting our >> limited resources by repeating old discussion rather than doing more >> implemenations. > >You consistently fail to address the issue of file system processing of >XML, and instead characterise all opposition to your proposal as "time >wasting". I will be happy to characterise it as that once you have given a >satisfactory response to the questions I pose. The long-term goal is to make file systems of operating systems to provide the charset parameter. Encoding declarations are tentative solutions. I am not insisting on my proposal. I am insisting on the rough consensus achived in the past. Since the I18N WG asked the XML Syntax WG not to change the precedence of the charset parameter, I am extremely reluctant to do such changes. Up to now, the only change I can support is to mandate the charset parameter of text/xml. >> Since XML processors support UTF-8 and UTF-16, transcoding from Unicode to >> legacy encodings does not look very attractive. > >I agree that such transcoding is unattractive, but you seem to want to bias >the XML MIME specification to supporting such transcoding whatever the cost >to other sorts of processing. The only "other sorts of processing" I can imagine is to provide the charset parameter. I understand that it is not very easy at present, but WWW servers are getting better. You think that the cost of developing and using XML-aware transcoders and the cost of inventing different in-band encoding for different textual formats is not a big deal. I do not agree. >However, something that converts an XML file from 8859-1 to UTF-8 and >leaves the endoding declaration saying 8859-1 is not useful. It has not >generated XML. It has made a thing which will fail to parse. Since the charset parameter is now authoritative, such documents will parse. ---- MURATA Makoto muraw3c@attglobal.net Received: by ns.secondary.com (8.9.3/8.9.3) id BAA29066 for ietf-xml-mime-bks; Thu, 23 Mar 2000 01:21:14 -0800 (PST) Received: from prserv.net (out4.prserv.net [32.97.166.34]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id BAA29062 for ; Thu, 23 Mar 2000 01:21:14 -0800 (PST) Received: from t3knz.attglobal.net ([210.88.161.172]) by prserv.net (out4) with SMTP id <2000032309230823902likc7e>; Thu, 23 Mar 2000 09:23:09 +0000 Message-Id: <200003230919.AA02031@t3knz.attglobal.net> Date: Thu, 23 Mar 2000 18:19:50 +0900 To: Chris Lilley Cc: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 From: MURATA Makoto In-Reply-To: <38D903E1.7E6BC41F@w3.org> References: <38D903E1.7E6BC41F@w3.org> MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.10 Content-Type: text/plain; charset=us-ascii Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: In message "Re: Some text that may be useful for the update of RFC 2376", Chris Lilley wrote... >Thus, for any XML file which is not encoded in US-ASCII, text/xml is an >inappropriate choice of MIME type. Silent data corruption can and will >occur. If I use UTF-8 or UTF-16 and provide the charset parameter, no data corruption will occur. >For all international xml files (noting that in this context, the USA is >international too due to the widespread use of Spanish, and the wide numbe >rof other languages in use), a type such as application/xml is the correct >choice, unless ther eis a more specific non-text type available. If an XML document is readable for casual users, satisfies restrictions of the top-level media type "text", and does not require special dispatching, "text/xml" is the most appropriate media type. Cheers, ---- MURATA Makoto muraw3c@attglobal.net Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id BAA29060 for ietf-xml-mime-bks; Thu, 23 Mar 2000 01:21:13 -0800 (PST) Received: from prserv.net (out4.prserv.net [32.97.166.34]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id BAA29056 for ; Thu, 23 Mar 2000 01:21:11 -0800 (PST) Received: from t3knz.attglobal.net ([210.88.161.172]) by prserv.net (out4) with SMTP id <2000032309230623902likc6e>; Thu, 23 Mar 2000 09:23:07 +0000 Message-Id: <200003230922.AA02032@t3knz.attglobal.net> Date: Thu, 23 Mar 2000 18:22:07 +0900 To: Chris Lilley Cc: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 From: MURATA Makoto In-Reply-To: <38D902E0.8E2DA0E9@w3.org> References: <38D902E0.8E2DA0E9@w3.org> MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.10 Content-Type: text/plain; charset=us-ascii Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: In message "Re: Some text that may be useful for the update of RFC 2376", Chris Lilley wrote... > >> I think that such a transcoder is very helpful because it works for >> all textual formats and also because it is very efficient. > >No, it is not helpful, because it makes like a lot more difficult for >everyone else and leads to data corruption. Apparently, Martin and I do not agree with you. >Incidentally I don't see an answer to my question about what such an >XML-unaweare transcoder would do when converting down from UTF-8 or UTF-16 >to some 8-bit charset withall the unrepresentable characters. Since it >doesn't know XML itcan't use NCRS. What does it do, silently replace these >characters with question marks? And that is somehow OK? In the case of XML, conversion from Unicode to legacy encodings is not very useful. Even when such conversion is requested, transcoders can give up transcoding, when they encounter something unrepresentable. >> >> The charset parameter is such a solution. >> > >> >It is one such solution. There are better ones, and indeed a much better >> >one in the XML specification. >> >> It works only for XML. It is not bad, when the MIME header is not available. >> But when it is available, we must rely on the charset parameter. > >For text/*, yes, we have to. Luckily there is application/* and model/* and >image/* and so forth for people using XML who care about data integrity and >don't want cheap text processing tools playing fast and loose with their >data. I believe that "always the charset parameter" is the recommendation shown in RFC 2130 and the public page of W3C I18N WG. In my message "History of the charset issue", I tried to summarize my understanding of the history. I am unable to ignore the consensus of W3C I18N WG, W3C XML Syntax WG, and W3C XML SIG&WG, and the recommendation shown in RFC 2130. >> You are advocating different in-band encoding signatures for different >> formats. I think that this is a significant burden to users and speficiation >> developers. > >You are advocating different out-of-band or in-band or mixed signatures for >different protocols. The long-term goal is to make file systems of OS aware of the charset parameter. Editors know the charset, and they store the charset information in the file system. This info is then passed to WWW servers and further passed to WWW browsers. The charset info is completely hidden from users and everything is automatic. There will be no data corruption. As of today, we need in-band signature and some tricks to keep out-of-band signature and in-band signature consistent. Most modern WWW servers provide the charset parameter. We only have to encourage them without repeading old arguments. > A solution that requires every "save as" of an XML >file to rewrite the (incorrect, but overridded by a MIME charset parameter) >encoding declaration, which was only incorrect because one of your "I know >how to fiddle with all text files" transcoders silently broke it in the >first place. This places, as you say, an intolerable burden on users. You are confusing XML-unaware transcoders and XML-aware programs which save XML documents into files. >One of the things about XML, which differs from HTML, is typical patterns >of use. XML treansmitted over HTTP ius likely to be extensively manipulated >from the filesystem of both the server and the client, a common operation >which your proposal makes much more difficult, just to allow people who >write simple text processing tools to not add XML support. As a trade off, >i hope it is obvious to everyone else why this is such a bad idea. I agree on the first and second sentence and completely disagree with the last sentence. ---- MURATA Makoto muraw3c@attglobal.net Received: by ns.secondary.com (8.9.3/8.9.3) id VAA17163 for ietf-xml-mime-bks; Wed, 22 Mar 2000 21:45:22 -0800 (PST) Received: from sh.w3.mag.keio.ac.jp (sh.w3.mag.keio.ac.jp [133.27.194.41]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id VAA17154 for ; Wed, 22 Mar 2000 21:45:19 -0800 (PST) Received: from enoshima (dhcp-100-224.mag.keio.ac.jp [133.27.195.224]) by sh.w3.mag.keio.ac.jp (8.9.3/3.7W) with ESMTP id OAA08218; Thu, 23 Mar 2000 14:46:37 +0900 (JST) Message-Id: <4.2.0.58.J.20000323143207.0331d9c0@sh.w3.mag.keio.ac.jp> X-Sender: duerst@sh.w3.mag.keio.ac.jp X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.0.58.J Date: Thu, 23 Mar 2000 14:45:48 +0900 To: Tim Bray , John Cowan From: "Martin J. Duerst" Subject: Re: UTF-16, the BOM, and media types Cc: MURATA Makoto , ietf-xml-mime@imc.org In-Reply-To: <3.0.32.20000322135249.0295ba70@pop.intergate.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: At 00/03/22 13:52 -0800, Tim Bray wrote: >At 04:34 PM 3/22/00 -0500, John Cowan wrote: > >> Section 4.3.3 of XML 1.0 says > >> "Entities encoded in UTF-16 must begin with the Byte Order Mark described > >> by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO WIDTH NO-BREAK > >> SPACE character, #xFEFF)." > > > >That describes entities encoded in the charset called "UTF-16". It says > >nothing about entities encoded in the charsets "UTF-16BE" and "UTF-16LE" > >or for that matter charset "x-focs". > >Yep, if you hold your head at just the right angle, and don't think of >the word "rhinocerous", you can convince yourself that the 16[BL]E >encodings are really different things entirely, just happen to share >a few characters with That Other Encoding's name, just close personal >friends, etc... Well, we know they are closely related, but what do processors do? No XML processor is supposed or even allowed to assume that e.g. iso-8859-1 and iso-8859-15 are closely related, or that e.g. iso-8859-1 and windows-1252 are even more closely related. There is no way for a processor to figure out. Trying to guess at that level in a non-interactive environment is doomed to fail. Trying to guess on prefixes of names is of course crazy. So if something comes in with a label of UTF-16BE, then an XML processor can either say 'sorry, don't know UTF-16BE', or it can know it and interpret it accordingly. Every XML processor has to understand UTF-16, but supporting UTF-16LE is not required. If you don't like UTF-16LE for XML, just don't support it. And please note the following erratum to the XML spec: http://www.w3.org/XML/xml-19980210-errata#E44 New: 00 3C ## ##, 00 25 ## ##, 00 20 ## ##, 00 09 ## ##, 00 0D ## ## or 00 0A ## ##: Big-endian UTF-16 or ISO-10646-UCS-2. Note that, absent an encoding declaration, these cases are strictly speaking in error. 3C 00 ## ##, 25 00 ## ##, 20 00 ## ##, 09 00 ## ##, 0D 00 ## ## or 0A 00 ## ##: Little-endian UTF-16 or ISO-10646-UCS-2. Note that, absent an encoding declaration, these cases are strictly speaking in error. old: 00 3C 00 3F: UTF-16, big-endian, no Byte Order Mark (and thus, strictly speaking, in error) 3C 00 3F 00: UTF-16, little-endian, no Byte Order Mark (and thus, strictly speaking, in error) The new text is quite a bit clearer. But if it's not clear enough, then we'll have to make it even clearer. Regards, Martin. Received: by ns.secondary.com (8.9.3/8.9.3) id VAA17162 for ietf-xml-mime-bks; Wed, 22 Mar 2000 21:45:22 -0800 (PST) Received: from sh.w3.mag.keio.ac.jp (sh.w3.mag.keio.ac.jp [133.27.194.41]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id VAA17155 for ; Wed, 22 Mar 2000 21:45:19 -0800 (PST) Received: from enoshima (dhcp-100-224.mag.keio.ac.jp [133.27.195.224]) by sh.w3.mag.keio.ac.jp (8.9.3/3.7W) with ESMTP id OAA08215; Thu, 23 Mar 2000 14:46:35 +0900 (JST) Message-Id: <4.2.0.58.J.20000323141220.0331ede0@sh.w3.mag.keio.ac.jp> X-Sender: duerst@sh.w3.mag.keio.ac.jp X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.0.58.J Date: Thu, 23 Mar 2000 14:48:02 +0900 To: Tim Bray , John Cowan , MURATA Makoto From: "Martin J. Duerst" Subject: Re: UTF-16, the BOM, and media types Cc: ietf-xml-mime@imc.org In-Reply-To: <3.0.32.20000322130928.015296f0@pop.intergate.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: I'm surprised this comes up again. At 00/03/22 13:09 -0800, Tim Bray wrote: >At 02:57 PM 3/22/00 -0500, John Cowan wrote: > >> UTF-16le and UTF-16be cannot be used for XML. XML mandates > >> the BOM for utf-16. Meanwhile, utf-16le and utf-16be cannot > >> have the BOM. More about this, see RFC 2781. > > > >I do not understand this from the text of XML 1.0. Clause 4.3.3 only says > >that if there is no encoding declaration, then either: > >Section 4.3.3 of XML 1.0 says > "Entities encoded in UTF-16 must begin with the Byte Order Mark described > by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO WIDTH NO-BREAK > SPACE character, #xFEFF)." > >Thus in my view the RFC is correct, Sorry, which RFC? If you mean RFC 2781, then I just checked and I didn't find the string 'XML' there. And it's not supposed to show up. RFCs about character encodings are not supposed to say things about various document formats, and document formats, for the largest part, are not supposed to say something about specific character encodings. >and thus 16BE and 16LE are not useful >for XML. It is good practice, whenever you store anything in UTF-16, to >put a BOM in, and XML makes that good practice compulsory, which is pretty >painless since it seems that virtually all software that writes UTF-16 does >so anyhow. The cost of a BOM is zilch. The benefit in data survival in the >face of stupid byte order tricks (yes, they still happen), is immense. > >Martin Duerst, a smart guy whom I respect, invested several hours in >trying to convince me that the 16[BL]E variants with forbidden-BOM had >some real-world justification, Well, that was mainly because you insisted that they needed to be forbidden unless there was some real-world justification. But that's not the real issue. The real issue is that each spec stays with it's business. XML does that most of the time, and requiring a BOM for UTF-16 in the XML spec makes sense in the context of requiring all XML processors to accept UTF-8 and UTF-16 even without any encoding information. Otherwise, it wouldn't always be possible to distinguish them. Apart from that, it does not make sense to use the XML spec to try to legislate on any character encodings. It would be very surprising if e.g. the XML spec said that EUC-JP is okay, but Shift_JIS is not okay, or Shift_JIS is okay, except for half-width kana, and so on. What you think is suitable for XML is another thing, that can go into tutorials, books, and so on. Some people would claim that only UTF-8 is suitable, others would claim whatever they want. Some get it right, and others get it wrong. It's not for the spec to judge. Regards, Martin. P.S.: If you wonder where UTF-16BE/LE could be of use in the context of XML, there was recently a discussion in the XML Signature WG about the use of XPath. The BOM confused a lot of people on that group. Received: by ns.secondary.com (8.9.3/8.9.3) id SAA04284 for ietf-xml-mime-bks; Wed, 22 Mar 2000 18:23:33 -0800 (PST) Received: from prserv.net (out2.prserv.net [32.97.166.32]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id SAA04278 for ; Wed, 22 Mar 2000 18:23:32 -0800 (PST) Received: from t3knz.attglobal.net ([210.88.161.2]) by prserv.net (out2) with SMTP id <2000032302251322901cm4sle>; Thu, 23 Mar 2000 02:25:14 +0000 Message-Id: <200003230219.AA02016@t3knz.attglobal.net> Date: Thu, 23 Mar 2000 11:19:18 +0900 To: Chris Lilley Cc: ietf-xml-mime@imc.org Subject: Re: Meida types and stylesheet linking From: MURATA Makoto In-Reply-To: <38D8FD3C.E5ADD63C@w3.org> References: <38D8FD3C.E5ADD63C@w3.org> MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.10 Content-Type: text/plain; charset=us-ascii Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: In message "Re: Meida types and stylesheet linking", Chris Lilley wrote... >Can you point me to some documentation that says that this usage of a MIME >type label, outside of a content-type header, is somehow invalid or >incorrect? In RFC 2045, we have the following: The purpose of the Content-Type field is to describe the data contained in the body fully enough that the receiving user agent can pick an appropriate agent or mechanism to present the data to the user, or otherwise deal with the data in an appropriate manner. The value in this field is called a media type. So, if something is not specified in the content-type field, it is not a media type :-) ---- MURATA Makoto muraw3c@attglobal.net Received: by ns.secondary.com (8.9.3/8.9.3) id NAA27175 for ietf-xml-mime-bks; Wed, 22 Mar 2000 13:51:10 -0800 (PST) Received: from smtp.gatewaymail.net (IDENT:root@[207.34.179.250]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id NAA27171 for ; Wed, 22 Mar 2000 13:51:09 -0800 (PST) Received: from FRITZ (00-10-4b-22-27-db.bconnected.net [209.53.11.246]) by smtp.gatewaymail.net (8.9.3/8.9.3) with SMTP id NAA18264; Wed, 22 Mar 2000 13:51:28 -0800 Message-Id: <3.0.32.20000322135249.0295ba70@pop.intergate.ca> X-Sender: tbray@pop.intergate.ca X-Mailer: Windows Eudora Pro Version 3.0 (32) Date: Wed, 22 Mar 2000 13:52:51 -0800 To: John Cowan From: Tim Bray Subject: Re: UTF-16, the BOM, and media types Cc: MURATA Makoto , ietf-xml-mime@imc.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: At 04:34 PM 3/22/00 -0500, John Cowan wrote: >> Section 4.3.3 of XML 1.0 says >> "Entities encoded in UTF-16 must begin with the Byte Order Mark described >> by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO WIDTH NO-BREAK >> SPACE character, #xFEFF)." > >That describes entities encoded in the charset called "UTF-16". It says >nothing about entities encoded in the charsets "UTF-16BE" and "UTF-16LE" >or for that matter charset "x-focs". Yep, if you hold your head at just the right angle, and don't think of the word "rhinocerous", you can convince yourself that the 16[BL]E encodings are really different things entirely, just happen to share a few characters with That Other Encoding's name, just close personal friends, etc... But I just don't get it. This feels perverse. I repeat, anyone using any flavor of UTF-16 can (and usually does) put in a BOM, if only on a belt-and-suspenders basis; and for this reason, de facto, the 16[BL]E media types, which forbid this practice, are simply not in practical terms usable for XML. I'm not denying that these things exist. Just asserting that the RFC is correct in saying they don't work with XML. -Tim Received: by ns.secondary.com (8.9.3/8.9.3) id NAA27112 for ietf-xml-mime-bks; Wed, 22 Mar 2000 13:47:35 -0800 (PST) Received: from mauve.innosoft.com (DSL107-055.brandx.net [209.55.107.55]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id NAA27101 for ; Wed, 22 Mar 2000 13:47:33 -0800 (PST) From: ned.freed@INNOSOFT.COM Received: from MAUVE.INNOSOFT.COM by MAUVE.INNOSOFT.COM (PMDF V6.1-1 #35243) id <01JNBW2V7NN40000IV@MAUVE.INNOSOFT.COM> for ietf-xml-mime@imc.org; Wed, 22 Mar 2000 13:49:21 -0800 (PST) Date: Wed, 22 Mar 2000 13:47:51 -0800 (PST) Subject: Re: Finishing(!) the XML-tagging discussion In-reply-to: "Your message dated Wed, 22 Mar 2000 20:00:39 +0000" <4.2.2.20000322195726.00bfa2c0@pop.dial.pipex.com> To: Graham Klyne Cc: "Martin J. Duerst" , ietf-xml-mime@imc.org Message-id: <01JNC3GA5EDC0000IV@MAUVE.INNOSOFT.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed References: <01JNAEGY3OOG00004D@MAUVE.INNOSOFT.COM> <4.2.2.20000321081348.00a75320@pop.dial.pipex.com> <4.2.0.58.J.20000322115620.03467dc0@sh.w3.mag.keio.ac.jp> Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: > At 11:59 AM 3/22/00 +0900, Martin J. Duerst wrote: > >should definitely be > > > >>accept-features: (&((language=de),(language=fr))) > A nit, before this example becomes received reality: the way to express > this in CONNEG is: > accept-features: (| (language=de) (language=fr) ) > (i.e. use logical OR, fewer brackets in this case, and no comma between > options. The basic syntax is lifted from LDAP search filters.) Um, the point of the example was to have something that _doesn't_ exist. Hence the use of AND, not OR. Ned Received: by ns.secondary.com (8.9.3/8.9.3) id NAA26889 for ietf-xml-mime-bks; Wed, 22 Mar 2000 13:32:18 -0800 (PST) Received: from mail.reutershealth.com (mail.reutershealth.com [204.243.9.36]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id NAA26885 for ; Wed, 22 Mar 2000 13:32:17 -0800 (PST) Received: from reutershealth.com (IDENT:cowan@skunk.reutershealth.com [204.243.9.153]) by mail.reutershealth.com (Pro-8.9.3/8.9.3) with ESMTP id QAA19198; Wed, 22 Mar 2000 16:34:03 -0500 (EST) Message-ID: <38D93C7A.7EB52152@reutershealth.com> Date: Wed, 22 Mar 2000 16:34:50 -0500 From: John Cowan Organization: Reuters Health Information X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: Tim Bray CC: MURATA Makoto , ietf-xml-mime@imc.org Subject: Re: UTF-16, the BOM, and media types References: <3.0.32.20000322130928.015296f0@pop.intergate.ca> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: Tim Bray wrote: > Section 4.3.3 of XML 1.0 says > "Entities encoded in UTF-16 must begin with the Byte Order Mark described > by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO WIDTH NO-BREAK > SPACE character, #xFEFF)." That describes entities encoded in the charset called "UTF-16". It says nothing about entities encoded in the charsets "UTF-16BE" and "UTF-16LE" or for that matter charset "x-focs". > It is good practice, whenever you store anything in UTF-16, to > put a BOM in, I don't deny it. But the letter of the specification permits UTF-16[BL]E, as long as an explicit encoding declaration is present. > and XML makes that good practice compulsory, I am not convinced. > Martin Duerst, a smart guy whom I respect, invested several hours in > trying to convince me that the 16[BL]E variants with forbidden-BOM had > some real-world justification, but I forget what it is... The main issue is that people actually do use them. Charset names, like media types, are intended to permit labeling of what actually exists. The existence of a charset name does not mean that anybody thinks the corresponding charset is a Good Thing. -- Schlingt dreifach einen Kreis um dies! || John Cowan Schliesst euer Aug vor heiliger Schau, || http://www.reutershealth.com Denn er genoss vom Honig-Tau, || http://www.ccil.org/~cowan Und trank die Milch vom Paradies. -- Coleridge (tr. Politzer) Received: by ns.secondary.com (8.9.3/8.9.3) id NAA26461 for ietf-xml-mime-bks; Wed, 22 Mar 2000 13:08:44 -0800 (PST) Received: from smtp.gatewaymail.net (IDENT:root@[207.34.179.250]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id NAA26456 for ; Wed, 22 Mar 2000 13:08:43 -0800 (PST) Received: from FRITZ (00-10-4b-22-27-db.bconnected.net [209.53.11.246]) by smtp.gatewaymail.net (8.9.3/8.9.3) with SMTP id NAA18175; Wed, 22 Mar 2000 13:08:27 -0800 Message-Id: <3.0.32.20000322130928.015296f0@pop.intergate.ca> X-Sender: tbray@pop.intergate.ca X-Mailer: Windows Eudora Pro Version 3.0 (32) Date: Wed, 22 Mar 2000 13:09:49 -0800 To: John Cowan , MURATA Makoto From: Tim Bray Subject: UTF-16, the BOM, and media types Cc: ietf-xml-mime@imc.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: At 02:57 PM 3/22/00 -0500, John Cowan wrote: >> UTF-16le and UTF-16be cannot be used for XML. XML mandates >> the BOM for utf-16. Meanwhile, utf-16le and utf-16be cannot >> have the BOM. More about this, see RFC 2781. > >I do not understand this from the text of XML 1.0. Clause 4.3.3 only says >that if there is no encoding declaration, then either: Section 4.3.3 of XML 1.0 says "Entities encoded in UTF-16 must begin with the Byte Order Mark described by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO WIDTH NO-BREAK SPACE character, #xFEFF)." Thus in my view the RFC is correct, and thus 16BE and 16LE are not useful for XML. It is good practice, whenever you store anything in UTF-16, to put a BOM in, and XML makes that good practice compulsory, which is pretty painless since it seems that virtually all software that writes UTF-16 does so anyhow. The cost of a BOM is zilch. The benefit in data survival in the face of stupid byte order tricks (yes, they still happen), is immense. Martin Duerst, a smart guy whom I respect, invested several hours in trying to convince me that the 16[BL]E variants with forbidden-BOM had some real-world justification, but I forget what it is... and I remain convinced that they are simply not suitable for use with XML. -Tim Received: by ns.secondary.com (8.9.3/8.9.3) id MAA25573 for ietf-xml-mime-bks; Wed, 22 Mar 2000 12:18:27 -0800 (PST) Received: from hose.pipex.net (hose.news.pipex.net [158.43.128.58]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id MAA25569 for ; Wed, 22 Mar 2000 12:18:25 -0800 (PST) Received: from GK-VAIO (userk545.uk.uudial.com [193.149.70.121]) by hose.pipex.net (Postfix) with ESMTP id 4CE8D451D; Wed, 22 Mar 2000 20:20:14 +0000 (GMT) Message-Id: <4.2.2.20000322195726.00bfa2c0@pop.dial.pipex.com> X-Sender: maiw03@pop.dial.pipex.com X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.2 Date: Wed, 22 Mar 2000 20:00:39 +0000 To: "Martin J. Duerst" From: Graham Klyne Subject: Re: Finishing(!) the XML-tagging discussion Cc: ietf-xml-mime@imc.org In-Reply-To: <4.2.0.58.J.20000322115620.03467dc0@sh.w3.mag.keio.ac.jp> References: <01JNAEGY3OOG00004D@MAUVE.INNOSOFT.COM> <"Your message dated Tue, 21 Mar 2000 08:26:34 -0500" <200003211326.IAA19007@astro.cs.utk.edu> <4.2.2.20000321081348.00a75320@pop.dial.pipex.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: At 11:59 AM 3/22/00 +0900, Martin J. Duerst wrote: >should definitely be > >>accept-features: (&((language=de),(language=fr))) A nit, before this example becomes received reality: the way to express this in CONNEG is: accept-features: (| (language=de) (language=fr) ) (i.e. use logical OR, fewer brackets in this case, and no comma between options. The basic syntax is lifted from LDAP search filters.) #g ------------ Graham Klyne (GK@ACM.ORG) Received: by ns.secondary.com (8.9.3/8.9.3) id LAA25180 for ietf-xml-mime-bks; Wed, 22 Mar 2000 11:54:50 -0800 (PST) Received: from mail.reutershealth.com (mail.reutershealth.com [204.243.9.36]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id LAA25176 for ; Wed, 22 Mar 2000 11:54:49 -0800 (PST) Received: from reutershealth.com (IDENT:cowan@skunk.reutershealth.com [204.243.9.153]) by mail.reutershealth.com (Pro-8.9.3/8.9.3) with ESMTP id OAA18233; Wed, 22 Mar 2000 14:56:40 -0500 (EST) Message-ID: <38D925A5.48188F12@reutershealth.com> Date: Wed, 22 Mar 2000 14:57:25 -0500 From: John Cowan Organization: Reuters Health Information X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.5-15 i686) X-Accept-Language: en MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 References: <200003221650.AA02009@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > UTF-16le and UTF-16be cannot be used for XML. XML mandates > the BOM for utf-16. Meanwhile, utf-16le and utf-16be cannot > have the BOM. More about this, see RFC 2781. I do not understand this from the text of XML 1.0. Clause 4.3.3 only says that if there is no encoding declaration, then either: a BOM is present, and the encoding is UTF-16, or no BOM is present, and the encoding is UTF-8. If a proper encoding declaration is present, then any charset may be used; however, parsers are only required to handle UTF-8 and UTF-16. (In practice, all parsers known to me also accept US-ASCII and ISO-8859-1.) For example, a file beginning with the characters encoded in Finagle's Own Character Set is perfectly legal, and will be parsed successfully by any parser with an x-focs conversion table. This is true even if x-focs is a multi-byte character set. I see absolutely no reason why UTF-16BE and UTF-16LE should be excluded from the list of acceptable charsets. It is true that Appendix F claims that a text beginning with the bytes 00 3C 00 3F or 3C 00 3F 00 is "strictly speaking, in error", but Appendix F is marked "non-normative", and this text is qualified in E44 anyhow. > I see no reasons for preserving byte sequences. We only have to > preserve XML information sets. Almost, since strictly speaking the charset is part of the information set. > Existing programming languages do not support Unicode very well, as > I see it. Except Java, Javascript, Ada 95, Dylan .... -- Schlingt dreifach einen Kreis um dies! || John Cowan Schliesst euer Aug vor heiliger Schau, || http://www.reutershealth.com Denn er genoss vom Honig-Tau, || http://www.ccil.org/~cowan Und trank die Milch vom Paradies. -- Coleridge (tr. Politzer) Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id JAA22818 for ietf-xml-mime-bks; Wed, 22 Mar 2000 09:43:29 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id JAA22814 for ; Wed, 22 Mar 2000 09:43:28 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id MAA12822; Wed, 22 Mar 2000 12:45:16 -0500 Message-ID: <38D906A7.8837412F@w3.org> Date: Wed, 22 Mar 2000 18:45:11 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 References: <38CF9D82.F4420ED4@w3.org> <200003221650.AA02008@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > > In message "Re: Some text that may be useful for the update of RFC 2376", > Chris Lilley wrote... > >and attempt to > >stretch this to make loose and wooly the current, fairly good state of > >encoding declaration of XML files? > > Unfortunately, we do not have "fairly good state of encoding declaration > of XML files". People generate XML documents by XSLT or their own programs, > and fail to specify the correct charset. That is not a problem. Such files will not be well formed and thus, will fail toparse. > Encoding PIs are not bad when > the MIME header is absent. But mistakes do happen. > > >Several people have pointed out that I am focussing on XML here. I would > >refer them to the name and scope of the mailing list. > > I think that you are not paying attention to other textual format. Oh I am, but not on this list where it is off topic. > I would > like XML to be a good citizen of the WWW and to establish a good practise As would I. I don't consider the propogation of known faults to be "good practice". > >Incidentally, XML is probably not best described as a textual format. It is > >a data format, which can among other things be used to describe > >international text. I am aware that the text/* media types have some > >historical requirements regarding 'character set'; this is sufficient that > >my opinion is that text/* should not be used for XML in general. > >Application/xml has no such problems (though it seems that people propose > >to propogate these problems there). > > I think that many XML documents are readable for casual users and that > the top-level type "text" is most appropriate. As long as they only use US-ASCII. > The charset parameter > is not a historical requirement. Rather, it is the right solution, > which is just about to take off. I think that we are wasting our > limited resources by repeating old discussion rather than doing more > implemenations. You consistently fail to address the issue of file system processing of XML, and instead characterise all opposition to your proposal as "time wasting". I will be happy to characterise it as that once you have given a satisfactory response to the questions I pose. > >It is possible for example to take a payload of image/svg-xml and alter it > >from UTF-16 to ISO-8859-15 (this would entail rewriting the encoding > >declaration and insertion of NCRs for any characters outside the repertoire > >of 8859-15). I would be most upset, as would every decoder on the planet, > >if the same conversion was performed on image/png. > > Since XML processors support UTF-8 and UTF-16, transcoding from Unicode to > legacy encodings does not look very attractive. I agree that such transcoding is unattractive, but you seem to want to bias the XML MIME specification to supporting such transcoding whatever the cost to other sorts of processing. > What is needed is the > other way around: conversion from legacy encodings to Unicode. Such transcoders > do not need character references by numbers. Thanks. That is the first time that I saw you limit these "all text" transcoders to somewhere that they might at least be useful and be able to represent all the characters. However, something that converts an XML file from 8859-1 to UTF-8 and leaves the endoding declaration saying 8859-1 is not useful. It has not generated XML. It has made a thing which will fail to parse. A transcoder that *knows* is converting from (list of legacy 8-bit charsets) to (UTF-8 or UTF-16) can always do the right thing by always emitting an XML declaration without an encodingdeclareation (or better, one thatsauys which of UTF-8 or UTF-16 is used). This is one line of code. So then all it needs to do is strip out any existing XML declaration. That is pretty trivial, too. I mean, grep -v will do in a pinch ;-) -- Chris Received: by ns.secondary.com (8.9.3/8.9.3) id JAA22557 for ietf-xml-mime-bks; Wed, 22 Mar 2000 09:31:36 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id JAA22553 for ; Wed, 22 Mar 2000 09:31:35 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id MAA11222; Wed, 22 Mar 2000 12:33:26 -0500 Message-ID: <38D903E1.7E6BC41F@w3.org> Date: Wed, 22 Mar 2000 18:33:21 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 References: <38CE7404.316E69DB@w3.org> <200003221650.AA02006@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > In message "Re: Some text that may be useful for the update of RFC 2376", > Chris Lilley wrote... > >That was what I was suggesting, changing it for text/xml only. > > If we consider e-mail and fallback to text/plain, there is not > much we can do to text/xml. We can mandate the charset parameter. > That's all what we can do. We cannot change the default. Correct. The text/plain fallback for HTTP and for email differs, and thus, only the intersection of these two defaults can be relied upon. Thus, for any XML file which is not encoded in US-ASCII, text/xml is an inappropriate choice of MIME type. Silent data corruption can and will occur. For all international xml files (noting that in this context, the USA is international too due to the widespread use of Spanish, and the wide numbe rof other languages in use), a type such as application/xml is the correct choice, unless ther eis a more specific non-text type available. -- Chris Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id JAA22500 for ietf-xml-mime-bks; Wed, 22 Mar 2000 09:28:21 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id JAA22495 for ; Wed, 22 Mar 2000 09:28:19 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id MAA10571; Wed, 22 Mar 2000 12:29:08 -0500 Message-ID: <38D902E0.8E2DA0E9@w3.org> Date: Wed, 22 Mar 2000 18:29:04 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Some text that may be useful for the update of RFC 2376 References: <38CE65A3.1BFE9ECF@w3.org> <200003221650.AA02005@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > > In message "Re: Some text that may be useful for the update of RFC 2376", > Chris Lilley wrote... > >Yes - such behaviour is clearly broken. Since a transcoder is changing many > >or all the other bytes in the file, expecting it to also correctly update > >the encoding declaration rather than leaving it broken is not asking too > >much. > > I think that such a transcoder is very helpful because it works for > all textual formats and also because it is very efficient. No, it is not helpful, because it makes like a lot more difficult for everyone else and leads to data corruption. Incidentally I don't see an answer to my question about what such an XML-unaweare transcoder would do when converting down from UTF-8 or UTF-16 to some 8-bit charset withall the unrepresentable characters. Since it doesn't know XML itcan't use NCRS. What does it do, silently replace these characters with question marks? And that is somehow OK? What about if this makes the file no longer well formed - that is OK too? And all to save half an hour of developer time in a transcoder, at the expense of silently corrupted data and lots more work for developers of tools and for users, to patch up the errors that such tools introduce. This is a really bad idea! > >> The charset parameter is such a solution. > > > >It is one such solution. There are better ones, and indeed a much better > >one in the XML specification. > > It works only for XML. It is not bad, when the MIME header is not available. > But when it is available, we must rely on the charset parameter. For text/*, yes, we have to. Luckily there is application/* and model/* and image/* and so forth for people using XML who care about data integrity and don't want cheap text processing tools playing fast and loose with their data. > >> We should not try to bend > >> specifications only to invent an ad-hoc solution for a particular format. > > > >I can only agree with that sentence by replacing "format" with "protocol". > > You are advocating different in-band encoding signatures for different > formats. I think that this is a significant burden to users and speficiation > developers. You are advocating different out-of-band or in-band or mixed signatures for different protocols. A solution that requires every "save as" of an XML file to rewrite the (incorrect, but overridded by a MIME charset parameter) encoding declaration, which was only incorrect because one of your "I know how to fiddle with all text files" transcoders silently broke it in the first place. This places, as you say, an intolerable burden on users. One of the things about XML, which differs from HTML, is typical patterns of use. XML treansmitted over HTTP ius likely to be extensively manipulated from the filesystem of both the server and the client, a common operation which your proposal makes much more difficult, just to allow people who write simple text processing tools to not add XML support. As a trade off, i hope it is obvious to everyone else why this is such a bad idea. -- Chris Received: by ns.secondary.com (8.9.3/8.9.3) id JAA22042 for ietf-xml-mime-bks; Wed, 22 Mar 2000 09:03:16 -0800 (PST) Received: from tux.w3.org (IDENT:root@tux.w3.org [18.29.0.27]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id JAA22037 for ; Wed, 22 Mar 2000 09:03:15 -0800 (PST) Received: from w3.org (IDENT:root@localhost [127.0.0.1]) by tux.w3.org (8.9.3/8.9.3) with ESMTP id MAA05349; Wed, 22 Mar 2000 12:05:05 -0500 Message-ID: <38D8FD3C.E5ADD63C@w3.org> Date: Wed, 22 Mar 2000 18:05:00 +0100 From: Chris Lilley Organization: W3C X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en,fr MIME-Version: 1.0 To: MURATA Makoto CC: ietf-xml-mime@imc.org Subject: Re: Meida types and stylesheet linking References: <38D8CE54.6CC45E61@w3.org> <200003221446.AA01978@t3knz.attglobal.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-xml-mime@mail.imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: MURATA Makoto wrote: > > In message "Re: Meida types and stylesheet linking", > Chris Lilley wrote... > >The media type is being used as a > >label. But it is not being used as a value for the Content-Type mail or > >HTTP header, so is not asserting that the entire resource is of this type. > > Historically, HTML has (ab)used media types even when no MIME entities are > involved. For example, embedded stylesheets represented by