Subject: RE: [Syslog-sec] I-D ACTION:draft-ietf-syslog-transport-udp-02.txt

From: "Moehrke, John (GE Healthcare)" <[email-masked]>
Date: Fri, 10 Sep 2004 07:27:59 -0500

Rainer,

I apologize for not reviewing the syslog documents and commenting. I
can't claim anything other than moderate understanding of any of the
syslog specifications. But you mention healthcare requirements. I
presume you are talking about the IHE efforts. There are a few items
that we have identified:

A) Max message size (you pointed it out.) We called for 32768. Our
newest schema is even more detailed, and I don't think there is a single
message that would fit in 512 bytes. The other factor is that we figured
that syslog servers/relays are usually on real operating systems and
thus ip frag/reassembly was not unreasonable (or should be designed to
handle IP frag/reassembly). The message creators would be self-imposed,
thus a medical device vendor would need to self-design a system that
either made small messages, or included IP fragmentation. (Please don't
add fragmentation anywhere other than network layer).
B) We need UTF-8. I think you got that covered now. We figured that the
BSD syslog was silent and thus just carried <bytes>. Since UTF-8 is
externally bytes, we figured it would work. 
C) We are using XML, thus there is a question about the new-line
characters. I am not sure if syslog requires that a single event be made
up of a single line of text. If this is the case then we will have to
collapse the new-line characters. This isn't a problem, as it still is
well-formed xml. But it is something that we didn't think about until
lately.

When we threw our document on the wall, we expected if these assumptions
were not going to work, that someone or some implementation would force
the issue. Even if it didn't come up, we also are just "Profiling" and
not "standardizing". Thus a product can be conformant to the standard
and not conformant to our profile. With a little effort they could be
conformant to our profile as well. Thus it was just a set of rules to
ensure that our needs were met and that interoperability could happen.

John 

_______________________________________________
Syslog-sec mailing list
[email-masked]
http://www.employees.org/mailman/listinfo/syslog-sec

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 10 Sep 2004 15:18:38 +0200

Hi List,

I think there is a lot of wisdom and obviously a lot of experience in
what David says. So even though I have strong concerns that a guaranteed
size of just 480 octets is not wise, I would accept it if nobody else
objects violently.

Rainer

> -----Original Message-----
> From: David B Harrington [mailto:[email-masked]] 
> Sent: Friday, September 10, 2004 3:06 PM
> To: Rainer Gerhards; 'Anton Okmianski'; [email-masked]
> Subject: RE: [Syslog-sec] I-D 
> ACTION:draft-ietf-syslog-transport-udp-02.txt
> 
> Hi,
> 
> Please keep in mind that MUST is used for an absolute requirement of
> the specification, i.e. the specification won't work if you don't
> support this. MUST should NOT be used to set some arbitrary size
> selected by the authors as looking like a nice round number that could
> be useful in many circumstances. It sounds like you are trying to
> create a MUST.
> 
> I recommend that you set the 480 as a RECOMMENDED minimum size, but
> also RECOMMEND 16k or 64k be supported to make it possible to send
> larger messages. You could also identify some expected use cases, such
> as the IHE case to demonstrate that potential users might expect
> support for larger sizes.
> 
> You express a concern that if 480 is the recommended minimum, then
> implementors will only implement that size, not a more useful size.
> The market can decide whether those implementations that only support
> 480 will actually be used widely. I suspect only free
> 480-implementations would get much usage, and you get what you pay
> for. Better implementations, both free and commercial, will probably
> be preferred by operators, and therefore vendors. It shouldn't be the
> author's decision what sizes MUST be supported based on their own
> preferences and their company's use cases.
> 
> 
> Dbh
> 
> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Rainer
> Gerhards
> Sent: Friday, September 10, 2004 4:10 AM
> To: Anton Okmianski; [email-masked]
> Subject: RE: [Syslog-sec] I-D
> ACTION:draft-ietf-syslog-transport-udp-02.txt
> 
> Anton, 
> 
> > > - In 3.4, the draft says:
> > > 
> > > ######
> > >    All implementations MUST support sending and receiving syslog
> > >    messages up to and including the size which does not require
> > >    fragmentation (507 bytes for IPv4 and 1191 bytes for IPv6).
> This
> > >    size excludes the overhead of the syslog transport and UDP/IP
> > >    headers.  Support for larger messages is encouraged.
> > Implementors
> > >    SHOULD clearly state the maximum supported message size in
> > >    documentation.
> > > ######
> > > 
> > > Actually, all implementations MUST support a message size of
> 65,000 
> > > bytes, because this is the required minimum in -protocol 4.1. We 
> > > agreed on this minimum after long discussions on the message size.
> 
> > > I'd actually prefer not to re-iterate this. But if we must, let me
> 
> > > say that a guaranteed message size of 507 bytes looks far too low
> to 
> > > me.
> > > 
> > > Please note that the -protcol requirement meant that all 
> > > implementations MUST support fragmentation.
> > > 
> > > Sorry I didn't spot this in the -01 version...
> > > 
> > > I've also noticed that in -protocol, I have limited that max
> message 
> > > size to 16,000,000 bytes in -protocol. It might make sense to sync
> 
> > > this. I can go up with my number.
> > 
> > Indeed we discussed this at length, but my recollection of
> conclusions 
> > was different. :)  Let's revisit.  We decided (I thought) that we 
> > can't force transport implementors into supporting fragmentation as
> it 
> > was too complicated. Plus, we can't them into supporting the maximum
> 
> > message size as the common denominator may be too small.
> 
> I've tried to find something definitive in the mailing list archive.
> Actually, I couldn't come up with such a thing... So it looks like
> there were some private conversations or we both had a wrong
> understanding of concensus. Anyhow, let's try to finalize it now.
> 
> I thought that the concensus was that we can not force implementors to
> support very large messages. Thus, the minimum-to-be-supported size
> should be just 64K. Please remember that the IHE healthcare initiative
> tries to use syslog for auditing and has a size requirement of at
> least 32K. So there is some industry need for larger size messages.
> > 
> > Hence, only the size that can be transported without fragmentation 
> > must be supported by transport implementors.
> 
> This means we are going down with the actual message limit from 1K
> (which already was very small) to just 480 octets. Given the
> additional overhead we have introduced, there is not much left for the
> acutal message.
> 
> Of course, you can argue that it is just the minimum size that MUST be
> supported. But I think if we face reality, that minimum will soon
> become the maximum supported by most implementations. I am very
> concerend that such a short guaranteed message size really makes
> sense... If we really take that route, we should eventually be bold
> enough to go back to 1K without fragmentation, even though we have
> some reliability concerns. At least that seems to have worked for
> several years now. I don't think we will get much acceptance for a
> standard that leaves around 400 bytes for the actual message (not
> counting meta data in structured ids....).
> 
> > However, a full
> > transport implementation will support fragmentation and size of up
> to
> > 16777216 bytes.   
> > 
> > I think what we need is a statement in -protocol that it supports a 
> > message size up to 16777216, but transports MAY impose a different 
> > limit.
> 
> That's already there ;)
> 
> > 
> > It was mentioned that some devices will physically not be able to 
> > handle large messages, where "large" can be quite a small number.
> > Restricting all messages to this common denominator is not good, so
> we 
> > have a multi-tier support system. People are not likely to support 
> > streaming of message parts in their implementations, so available 
> > memory will be an issue.
> > 
> > Administrators will choose the appropriate implementations. Should
> we 
> > add a statement that transport implementations MUST document maximum
> 
> > supported message size?
> > 
> > I know it is less than optimal, but requiring all transports to 
> > support some huge size is not a good idea either IMO.
> > 
> > You also had an idea for doing fragmentation within the -protocol by
> 
> > upper layer. I originally liked it.  But later people commented that
> 
> > this was improper layering and shifted responsibilities of transport
> 
> > layer up the stack.  I agree with this.  It would have introduced 
> > potential fragmentation on several layers and we would have still 
> > forced the higher layer to deal with UDP sizing constraints.
> > 
> > I agree this issue deserves discussion. Let's have it and put it bed
> 
> > for good.
> > 
> > > I do not fully understand what you mean in 4.2 by:
> > > 
> > > #####
> > >  No concurrent port reuse on the same host is
> > >    allowed.
> > > #####
> > > 
> > > It's probably just my English, but I do not get it. I'd appreciate
> 
> > > if you could elaborate a little.
> > 
> > What this refers to is the use of SO_REUSEPORT socket option which 
> > allows multiple sockets to be bound to the same port.  I guess it 
> > would not be a problem if the two different consumers of the socket 
> > (maybe threads) use different MessageId numbers.  And this is stated
> 
> > as requirement elsewhere: all message parts must be send using the
> > same MessageId and source port among other things.   
> > 
> > Following your suggestion, we already allow multiple different ports
> 
> > to be used concurrent within a single process, so I have no problem 
> > with removing this restriction. It is too implementation-specific 
> > anyway.  What do you think?
> 
> I agree. I simply did not fully understand what you were saying ;)
> Eventually the sentence could be a bit more verbose - but this might
> just have been my personal problem ;)
> 
> Rainer

---

From: "Anton Okmianski" <[email-masked]>
Date: Fri, 10 Sep 2004 14:21:21 -0400

David and all:

> Please keep in mind that MUST is used for an absolute 
> requirement of the specification, i.e. the specification 
> won't work if you don't support this. MUST should NOT be used 
> to set some arbitrary size selected by the authors as looking 
> like a nice round number that could be useful in many 
> circumstances. It sounds like you are trying to create a MUST.
> 
> I recommend that you set the 480 as a RECOMMENDED minimum 
> size, but also RECOMMEND 16k or 64k be supported to make it 
> possible to send larger messages. 

Yes, the ~507 bytes minimum size support requirement in transport-udp
is semi-arbitrary. It is the maximum size we think can be easily
supported (ie without fragmentation). You are asking why can it be say
300 bytes if I have a small device?  But I am concerned with
interoperability issues if there is no ANY minimum supported size?
Don't we need at least basic minimum baseline?

Maybe in the spirit of "be strict in what you send and liberal in what
you accept", some minimum supported size requirement should only be
for the receiver?  The sending library can have whatever size support
it wants.  This way we can support small devices sending small
messages. 

Thanks,
Anton. 

> You could also identify 
> some expected use cases, such as the IHE case to demonstrate 
> that potential users might expect support for larger sizes.

> You express a concern that if 480 is the recommended minimum, 
> then implementors will only implement that size, not a more 
> useful size. The market can decide whether those 
> implementations that only support 480 will actually be used 
> widely. I suspect only free 480-implementations would get 
> much usage, and you get what you pay for. Better 
> implementations, both free and commercial, will probably be 
> preferred by operators, and therefore vendors. It shouldn't 
> be the author's decision what sizes MUST be supported based 
> on their own preferences and their company's use cases.
> 
> 
> Dbh
> 
> [mailto:[email-masked]] On Behalf Of 
> Rainer Gerhards
> I've tried to find something definitive in the mailing list 
> archive. Actually, I couldn't come up with such a thing... So 
> it looks like there were some private conversations or we 
> both had a wrong understanding of concensus. Anyhow, let's 
> try to finalize it now.
> 
> I thought that the concensus was that we can not force 
> implementors to support very large messages. Thus, the 
> minimum-to-be-supported size should be just 64K. Please 
> remember that the IHE healthcare initiative tries to use 
> syslog for auditing and has a size requirement of at least 
> 32K. So there is some industry need for larger size messages.
> > 
> This means we are going down with the actual message limit 
> from 1K (which already was very small) to just 480 octets. 
> Given the additional overhead we have introduced, there is 
> not much left for the acutal message.
> 
> Of course, you can argue that it is just the minimum size 
> that MUST be supported. But I think if we face reality, that 
> minimum will soon become the maximum supported by most 
> implementations. I am very concerend that such a short 
> guaranteed message size really makes sense... If we really 
> take that route, we should eventually be bold enough to go 
> back to 1K without fragmentation, even though we have some 
> reliability concerns. At least that seems to have worked for 
> several years now. I don't think we will get much acceptance 
> for a standard that leaves around 400 bytes for the actual 
> message (not counting meta data in structured ids....).
> 
> > However, a full

> I agree. I simply did not fully understand what you were 
> saying ;) Eventually the sentence could be a bit more verbose 
> - but this might just have been my personal problem ;)
> 
> Rainer

---

From: "Anton Okmianski" <[email-masked]>
Date: Fri, 10 Sep 2004 16:45:12 -0400

David:

Which draft is the target of your recommendations?  Syslog-protocol or
transport-udp? What restrictions/recommendations do you think each
should have?

I am not sure I see a clear distinction between troubleshooting and
device/network status messages to warrant two different levels of
message size recommendations. The uses cases overlap.   

Could you explain what is the point of providing max-message size in
the syslog message?  What would the server do if it gets it from the
client?  

I can understand the motivation for ability to learn max supported
message size of the receiver through SNMP. It could be useful, but
relays complicate things. The sender will still know nothing about
end-to-end capabilities of delivery of messages of certain size (well,
it does not even know the intended final receiver).  If we use this
information on a next-hop basis, then we have to assume relay can
transform messages the same way the original sender can. This is
something we wanted to avoid to support security signatures.     

Thanks,
Anton. 

> Sent: Friday, September 10, 2004 11:20 AM
> To: 'Moehrke, John (GE Healthcare)'; 'Rainer Gerhards'; 
> 'Anton Okmianski'; [email-masked]
> I recommend that we set three pairs of recommendations, based 
> on the use cases:
> 
> Use Case #1 - Troubleshooting: 
> 
> A small UDP-based message is important for device and network
> **troubleshooting** in a reliability-challenged network, such 
> as one under DDOS or physical combat situations, where a 
> fragmented message may have greater difficulty getting 
> delivered, and TCP has difficulty establishing sessions. 
> 
> 1a) receiver - Recommend the 480 size as the minimum 
> recommended size for carrying rather small messages for this 
> use case. 
> 
> 1b) sender - We should specify what the resulting payload 
> size is after subtracting the syslog overhead, and recommend 
> that message generators not exceed that length for messages 
> reporting device failures or other conditions expected in a 
> reliability-challenged network.
> 
> Use Case #2 - Device/network status reporting:
> 
> Status reporting in a normally-operating network will 
> typically benefit from larger message sizes. I think it 
> reasonable to think that a 1k message is large enough to 
> contain "hints" about the nature of a problem to help an 
> operator understand what steps are needed to resolve the problem.
>  
> 2a) reciever - RECOMMEND 1k be supported as a minimum message 
> size to make it possible to send more information in 
> messages, to provide more useful information to the reciever. 
> 
> 2b) sender - RECOMMEND messages not exceed 1k for most 
> situations. Message generators should be encouraged to avoid 
> bloat in their messages.
> 
> Use case #3 - application usage
> 
> [soap]
> I have difficulty understanding why messages or 32k and 64k 
> are really necessary - it strikes me that a short message or 
> an SNMP trap could contain a URL to a web page or a file that 
> contains information of that size to be communicated. Just 
> because syslog CAN be used to transmit large messages doesn't 
> mean it SHOULD be used for messages of that size. Expecting 
> all vendors to implement syslog buffers of that size, when 
> other means are available for huge messages, seems like a bad 
> use of the syslog protocol. [end soap]
> 
> I expect the number of use cases that demand this size log 
> message will be limited, but there are obviously people who 
> do want messages this size, so we should address the
recommendations:
> 
> 3a) reciever - I suggest we recommend 16k as the recommended 
> minimum size and 64k as the recommended minimum-maximum size, 
> i.e recommend at least 64k buffers. 16k is large enough to 
> hold a fair amount of info, and 64k might be a limit run into 
> in a compiler or malloc() implementation. Being able to 
> support messages of at least 64k would be a good thing, for 
> those implementations that want to support such use cases.
> 
> 3b) sender - many implementations will not be able to receive 
> greater than 64k messages, so senders should not send more than 64k.
> 
> I recommend that either the message contain a max-message 
> size indicator in the message (e.g. see SNMNPv3), or that the 
> syslog MIB contain an object that specifies the maxiumum size 
> supported by a receiver, so a sender can know not to send  
> larger messages.
> 
> David Harrington
> [email-masked]
> 
> 
> 
> 
> 
> From: Moehrke, John (GE Healthcare) [mailto:[email-masked]]

> Sent: Friday, September 10, 2004 9:57 AM
> To: Rainer Gerhards; [email-masked]; Anton Okmianski; 
> [email-masked]
> I truly don't understand why syslog cares about max-size as 
> defined in IP. It can say: "Due to a protocol binding to udp, 
> big messages may get fragmented.  This fragmentation 
> increases the risk of the message not getting delivered." The 
> only reason for syslog to use any SHOULD/MUST language on 
> this subject is to assure a syslog user that a 'syslog' 
> implementation will be able to create messages with a payload 
> of n size, and that if this message gets through the gauntlet 
> that is udp/ip, it will be consumed without buffer size 
> rejects. Going over UDP, you can't give any form of assurance 
> that it will be delivered. This lack of assurance is simply 
> made worse if the udp packet is big.
> 
> On the max-size issue... What IHE is concerned with is 
> profiling a use-case that is forbidden by standards or not 
> implementable given current systems. Thus we are more 
> interested in the syslog-max-size, than the assurance issue.  
> If we truly can't expect a syslog implementation (client and 
> server) to handle a message bigger than 480 octets; then we 
> must abandon the use of udp-syslog. We have no choice but to 
> go with Reliable syslog. We will have to go to 
> Reliable-syslog, not for the reliability aspect, but because 
> we know our payload won't be rejected simply because it is 
> big. In our design of the schema we understood that a message 
> might not get there if we are bound to udp-syslog. We 
> understood that the larger the message the bigger this 
> un-reliability is.
> 
> OSI layers are GOOD...
> 
> John
> 
> From: [email-masked]
> [mailto:[email-masked]] On Behalf 
> Of Rainer Gerhards
> Sent: Friday, September 10, 2004 8:19 AM
> To: [email-masked]; Anton Okmianski; [email-masked]
> I think there is a lot of wisdom and obviously a lot of 
> experience in what David says. So even though I have strong 
> concerns that a guaranteed size of just 480 octets is not 
> wise, I would accept it if nobody else objects violently.
> 
> Rainer
> 

> > > > I'd actually prefer not to re-iterate this. But if we must,
let
> me
> archive. 

> > I thought that the concensus was that we can not force
implementors
> > Of course, you can argue that it is just the minimum size that
MUST
> be 

> > > I think what we need is a statement in -protocol that it
supports
> a 
> > > You also had an idea for doing fragmentation within the
-protocol
> by

---

From: "Moehrke, John (GE Healthcare)" <[email-masked]>
Date: Sat, 11 Sep 2004 21:40:54 -0500

Anton,

Yes, I am all for removing the restriction and at most making it a
recommendation. I really think that the client side is simply a choice
of tool (kind of), the standard should place no rules here. I do want to
be assured that a receiver can handle my message, and thus I would
understand a receiver side recommendation. 

I know healthcare (IHE) has a large footprint schema, but I think that
32K is comfortable. Our schema may seem excessive but we are including
who, what, where, when and why in detail. We are indeed pointing at many
objects indirectly, but of course healthcare is very committed to
Globally Unique Identifiers.

John