Subject: RE: [Syslog] Sec 6.1: Truncation

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 9 Jan 2006 10:44:38 -0500

Rainer:

I agree - this is better than a convoluted rule. 

I think we only have any business in defining truncation for relays.  For collectors, we have tried to stay away from describing how messages are stored.  

For relays, I think it would be useful to state that relay can't just drop arbitrary message parts. Your statements about "some parts ... are lost" may be interpreted that way. 

I would recommend that we state that any truncation must happen at the end of the message, which I think is what truncation means to a lot of people anyway. This would prevent an implementation which prefers to throw out STRUCTURED-DATA before the MSG content.  A consistent behavior is useful for interop and, in particular, may help in dealing with security issues. 

Thanks,
Anton. 

> -----Original Message-----
> From: Rainer Gerhards [mailto:[email-masked]] 
> Sent: Monday, January 09, 2006 3:21 AM
> To: Anton Okmianski (aokmians)
> Subject: RE: [Syslog] Sec 6.1: Truncation
> 
> Anton, Darren,
> 
> I agree that the truncation rule is probably not really 
> useful, even confusing. I think it is hard to predict for any 
> potential message if the more interesting content is in 
> STRUCTURED-DATA or in the MSG part.
> For example, with our current SD-IDs, I'd prefer to trunctate 
> them instead of MSG. Obviously, the case is different for 
> your LINKDOWN sample. I also agree with Darren that 
> truncation probably happens on the transport layer, the 
> application may not even see the full message.
> 
> My conclusion, however, is slightly different: I recommend 
> now that we remove truncation rules from -protocol. We should 
> just say that truncation might happen and that in this case 
> some parts of the message are lost - what is at the 
> discretion of the receiver.
> 
> Rainer
> 
> > -----Original Message-----
> > From: [email-masked]
> > [mailto:[email-masked]] On Behalf Of Anton Okmianski 
> > (aokmians)
> > Sent: Friday, January 06, 2006 9:48 PM
> > To: [email-masked]
> > Subject: [Syslog] Sec 6.1: Truncation
> > 
> > Rainer and all:
> > 
> > I started reading draft #16. Since we are revisiting 
> everything... I 
> > am not very comfortable with the current truncation rules.
> > 
> > "Receivers SHOULD follow this order of preference when it comes to 
> > truncation:
> > 
> >  1) No truncation
> >  2) Truncation by dropping SD-ELEMENTs
> >  3) If 2) not sufficient, truncate MSG"
> > 
> > I don't think that this is a good recommendation.  I would 
> assume that 
> > in many cases people would try to tokenize more important data into 
> > structured data and use message text as a secondary user-friendly 
> > description. For example, for LINK_DOWN message, I would probably 
> > encode link ID in the structured elements as this is something that 
> > should be readily available for receivers. The MSGID could be 
> > "LINK_DOWN" and the MSG text may simply be "Link down".  If you 
> > truncate the structured data, it makes it harder.
> > 
> > I also think, in general it is useful to put more important data 
> > first, which is another reason for putting more valuable data into 
> > structured data in a more compact way.
> > 
> > Additionally, structured data can be used to provide 
> message length or 
> > digest, which can help receiver to determine if message was 
> truncated.
> > 
> > Also, I think this statement is very convoluted:
> > 
> > "Please note that it is possible that the MSG field is truncated 
> > without dropping any SD-PARAMS.  This is the case if a 
> message with an 
> > empty STRUCTURED-DATA field must be truncated."
> > 
> > I think I understand what you are driving at, but I don't see it as 
> > adding any requirements or clarification.
> > 
> > This sentence is not clear although I know what you are 
> trying to say:
> > 
> > "The limits below are minimum maximum lengths, not maximum length."
> > 
> > I propose replacing the entire section 6.1 with this text:
> > 
> > "Syslog message limits are dictated by the syslog transport 
> mapping in 
> > use. Each transport mapping MUST define the minimum 
> required message 
> > length support. Any syslog transport mapping MUST support 
> messages of 
> > up to and including 480 octets in length.
> > 
> > Any syslog receiver MUST be able to accept messages of up to and 
> > including 480 octets in length.  All receiver 
> implementations SHOULD 
> > be able to accept messages of up to and including 2048 octets in 
> > length. Receivers MAY receive messages larger than 2048 octets in 
> > length. If a receiver receives a message with a length 
> larger than it 
> > supports, the receiver MAY discard the message or truncate the 
> > payload.
> > 
> > If truncation is performed by the receiver, it MUST first 
> truncate the 
> > MSG field as necessary to meet the supported length limit. If 
> > truncation of the entire MSG field is not sufficient, then 
> > additionally, the STRUCTURED-DATA field MUST be truncated 
> by removing 
> > one or more SD-ELEMENT fields. A minimum number of 
> SD-ELEMENT fields 
> > MUST be truncated starting from the end as necessary to meet the 
> > supported length limit. SD-ELEMENT field can't be truncated 
> partially.
> > If all SD-ELEMENT fields are removed, NILVALUE MUST be 
> specified for 
> > STRUCTURED-DATA field. Truncation of HEADER
> > field MUST NOT be performed."   
> > 
> > BTW, in your text or mine, what happens if message is malformed?  A 
> > proxy won't be able to truncate it properly then. We don't want to 
> > prevent it from truncating it in some way and sending the message 
> > further, I would think.  At least you will see something at 
> the final 
> > destination, which maybe more useful than nothing. If we just made 
> > truncation a simple take the first X octets out of Y 
> octets, it would 
> > not be an issue, but then proxy would be allowed to turn a 
> well-formed
> > message into malformed message upon truncation.   
> > 
> > What do you think?
> > 
> > Thanks,
> > Anton. 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Syslog mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/syslog
> > 
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: "Rainer Gerhards" <[email-masked]>
Date: Mon, 9 Jan 2006 16:48:41 +0100

> I think we only have any business in defining truncation for 
> relays.  For collectors, we have tried to stay away from 
> describing how messages are stored.  
> 
> For relays, I think it would be useful to state that relay 
> can't just drop arbitrary message parts. Your statements 
> about "some parts ... are lost" may be interpreted that way. 

Actually, this was what I meant ;) [I saw a number of use cases where it
would make sense to strip some known-not-so-relavant SD-IDs to be
strippedd], but ...
> 
> I would recommend that we state that any truncation must 
> happen at the end of the message, which I think is what 
> truncation means to a lot of people anyway. This would 
> prevent an implementation which prefers to throw out 
> STRUCTURED-DATA before the MSG content.  A consistent 
> behavior is useful for interop and, in particular, may help 
> in dealing with security issues.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
... this is more important. I now agree with your point.

As a side-note, we had the idea that relay operations may become a
separate document, so I would prefer not to dig too deep into relay
behaviour. To specify what you recommend, this is not necessary, so this
is not really a discussion topic here.

Rainer

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Wed, 11 Jan 2006 12:18:00 -0500

Rainer: 

Thanks for the update. Comments below. 

> I have now changed section 6.1 to:
> 
> ###
> 6.1.  Message Length
> 
>    Syslog message size limits are dictated by the syslog transport
>    mapping in use.  There is no upper limit per se.  

These two sentences are contradictory.  I'd remove the last one.  The maximum limit can be dictated by a transport mapping, like in the case of UDP. 

> Each transport
>    mapping MUST define the minimum required message length 
> support.  Any
>    syslog transport mapping MUST support messages of up to 
> and including
>    480 octets in length.
> 
>    length.  If a receiver receives a message with a length larger than
>    it supports, the receiver MAY discard the message or truncate the
>    payload.
> 
>    If a receiver truncates messages, the truncation MUST occur at the
>    end of the message.  UTF-8 encoding and STRUCTURED-DATA 
> MUST be kept
>    valid during truncation.  

You need to be clear what you mean by keeping the UTF-8 encoding. Do you mean that octets should not be truncated in a way which corrupts the last character (which may have multiple octets)?

It is probably possible to detect such corruption by looking at the first bit of the last character and making sure it is not 1, if I recall UTF-8 encoding correctly. If it is 1, drop the last octet.  Check the new last one and do the same until you find one with first bit set to 0. 

It seems that to ensure that the receiver would need to be pretty smart. I wonder if it is a problem.  Another question is whether or not validation like this is more appropriate at the higher layer, where every UTF character may be validated anyway. 

> SD-ELEMENTs MUST NOT partly be truncated.
>    If an SD-ELEMENT is to be truncated, the whole SD-ELEMENT MUST be
>    deleted.  If the last SD-ELEMENT of a message is deleted, the
>    STRUCTURED-DATA field MUST be changed to NILVALUE.
> ###

I thought the last train of thought was to do a dumb cutover of octets at the end. Darren mentioned this is what you will likely get at the application layer. Proposed rules (although simpler than before) would still demand quite a bit of handling for messages that exceed the max size supported by receiver.  I now wonder if implementors would really bother to implement all that logic for the case of messages of sizes they are not configured to handle.  

After all the trouble of validating and fixing the message which exceeds normative size for receiver, all you'd get is a truncated message, which will be well-formed syslog message after truncation, but may not be well-formed as far as consuming application is concerned.  

What do you guys think?

Thanks,
Anton. 

> 
> I have explicitly stated that there is no intrinsic upper 
> size limit. I did this, because we had so much 
> confusion/misunderstanding on that fact in the past. I've 
> also added some details on truncation. The rest is as 
> suggested by Anton :)
> 
> Please review and comment.
> 
> Rainer
> 
> > [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> > Sent: Monday, January 09, 2006 4:49 PM
> > Cc: [email-masked]
> > > I think we only have any business in defining truncation 
> for relays.  
> > > For collectors, we have tried to stay away from describing how 
> > > messages are stored.
> > > 
> > > For relays, I think it would be useful to state that relay can't 
> > > just drop arbitrary message parts. Your statements about 
> "some parts 
> > > ... are lost" may be interpreted that way.
> > 
> > Actually, this was what I meant ;) [I saw a number of use 
> cases where 
> > it would make sense to strip some known-not-so-relavant 
> SD-IDs to be 
> > strippedd], but ...
> > > 
> happen at 
> > > the end of the message, which I think is what truncation 
> means to a 
> > > lot of people anyway. This would prevent an implementation which 
> > > prefers to throw out STRUCTURED-DATA before the MSG content.  A 
> > > consistent behavior is useful for interop and, in particular, may 
> > > help in dealing with security issues.
> > behaviour. To specify what you recommend, this is not necessary, so 
> > this is not really a discussion topic here.
> > 
> > Rainer
> > > 
> > > Thanks,
> > > Anton. 
> > > 
> > > > I agree that the truncation rule is probably not really useful, 
> > > > even confusing. I think it is hard to predict for any potential 
> > > > message if the more interesting content is in 
> STRUCTURED-DATA or 
> > > > in the MSG part.
> > > > For example, with our current SD-IDs, I'd prefer to 
> trunctate them 
> > > > instead of MSG. Obviously, the case is different for 
> your LINKDOWN 
> > > > sample. I also agree with Darren that truncation 
> probably happens 
> > > > on the transport layer, the application may not even 
> see the full 
> > > > message.
> > > > 
> > > > My conclusion, however, is slightly different: I recommend now 
> > > > that we remove truncation rules from -protocol. We 
> should just say 
> > > > that truncation might happen and that in this case some 
> parts of 
> > > > the message are lost - what is at the discretion of the 
> receiver.
> > > > 
> > > > Rainer
> > > >

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Thu, 12 Jan 2006 12:24:58 -0500

Banzsi:

I agree truncation does not solve the issue - that's why I don't want to over-design it.  

Splitting is an option I'd leave to application-layer running above syslog. It is not precluded. Just a matter of another RFC with extra sd-elements. 

If we do fragmentation in syslog transport/protocol, we are re-inventing IP & TCP. We went down the path of defining special sd-elements for this once, but decided that in most cases increasing max acceptable message size on receivers and using appropriate transport is a better solution for many deployments.  

I'd propose we do truncation as raw octet cut-over, but define an optional sd-element for msg-length (in octets?), which people can use to know if there was truncation. However, this would mean we would allow a proxy to legitimately send malformed messages. Well, a receiver has to handle those somehow coming from the original sender anyway. It is not ideal, whichever way you slice it. 

Anton.  

> [mailto:[email-masked]] On Behalf Of Balazs Scheidler
> Sent: Thursday, January 12, 2006 5:07 AM
> To: Rainer Gerhards
> Cc: [email-masked]
> On Thu, 2006-01-12 at 10:45 +0100, Rainer Gerhards wrote:
> > Anton & all,
> > 
> > You have good points and I have to admit I am still 
> thinking what is 
> > the best way. I would appreciate if some other WG members could 
> > express their thoughts...
> 
> My pragmatic view is that overly long messages should be 
> split instead of truncated. Of course splitting rules are 
> similar to truncating rules in a sense, but the question of 
> generating the syslog header also comes up, e.g.
> 
> <16>Jun 12 13:45:54 host app[12345]: This is a too long message
> 
> should become:
> 
> <16>Jun 12 13:45:54 host app[12345]: This is a too...
> <16>Jun 12 13:45:54 host app[12345]: long message
> 
> This way we don't lose information while still limiting the 
> message size. Of course this will still confuse log analysis 
> applications but it can be solved simply by lifting the 
> message size limit if that is configurable in the syslog application.
> 
> Maybe we should indicate in an SD-ID that message truncation 
> happened so there would be no ambiguity.
> 
> Another question whether we allow relays to modify the SD-ID 
> part of the message or it must be done by the sender alone?
> 
> --
> Bazsi
> 
> 
>