From lehmann@ulticom.com Tue Dec 2 16:03:58 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA10839 for ; Tue, 2 Dec 2003 16:03:57 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ARHgn-0000ru-00 for sctp-impl-archive@ietf.org; Tue, 02 Dec 2003 16:04:09 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ARHgm-0000r2-00 for sctp-impl-archive@ietf.org; Tue, 02 Dec 2003 16:04:09 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hB2L2nw5018787; Tue, 2 Dec 2003 13:02:49 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB2L26Ks017354 for sctp-impl-filtered; Tue, 2 Dec 2003 13:02:08 -0800 (PST) Message-Id: <200312022102.hB2L26Ks017354@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070398925-17352-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Path.Max.Retrans - inactive when reaches or exceeds? List-Id: sctp-impl To: SCTP Implementors From: David Lehmann X-Nails-Approved: lehmann@ulticom.com Date: Tue, 02 Dec 2003 15:49:03 -0500 This is a multi-part message in MIME format... ------------=_1070398925-17352-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Sections 8.2 and 8.3 seem to contradict each other. 8.2 says to mark the destination inactive when the error count _exceeds_ Path.Max.Retrans and 8.3 says to do it when it _reaches_ Path.Max.Retrans. 8.2 Path Failure Detection... Each time the T3-rtx timer expires on any address, or when a HEARTBEAT sent to an idle address is not acknowledged within a RTO, the error counter of that destination address will be incremented. When the value in the error counter exceeds the protocol parameter 'Path.Max.Retrans' of that destination address, the endpoint should mark the destination transport address as inactive, and a notification SHOULD be sent to the upper layer. 8.3 Path Heartbeat... The endpoint should increment the respective error counter of the destination transport address each time a HEARTBEAT is sent to that address and not acknowledged within one RTO. When the value of this counter reaches the protocol parameter ' Path.Max.Retrans', the endpoint should mark the corresponding destination address as inactive if it is not so marked, and may also optionally report to the upper layer the change of reachability of this destination address. Which is correct? -- David Lehmann Ulticom, Inc. AOL/Yahoo IM: davidULCM 1020 Briggs Road 1-856-787-2729 Mt. Laurel, NJ 08054 USA ------------=_1070398925-17352-0-- From rrs@cisco.com Tue Dec 2 17:08:22 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA15974 for ; Tue, 2 Dec 2003 17:08:22 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ARIh8-0002xK-00 for sctp-impl-archive@ietf.org; Tue, 02 Dec 2003 17:08:34 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ARIh7-0002vg-00 for sctp-impl-archive@ietf.org; Tue, 02 Dec 2003 17:08:34 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 02 Dec 2003 14:08:21 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hB2M7Rjq021744; Tue, 2 Dec 2003 14:07:28 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB2M67Ks018167 for sctp-impl-filtered; Tue, 2 Dec 2003 14:06:09 -0800 (PST) Message-Id: <200312022206.hB2M67Ks018167@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070402767-18159-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path.Max.Retrans - inactive when reaches or exceeds? List-Id: sctp-impl To: David Lehmann From: "Randall Stewart (cisco)" Cc: SCTP Implementors X-Nails-Approved: rrs@cisco.com Date: Tue, 02 Dec 2003 16:05:59 -0600 This is a multi-part message in MIME format... ------------=_1070402767-18159-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary David Lehmann wrote: > Sections 8.2 and 8.3 seem to contradict each other. > 8.2 says to mark the destination inactive when the error > count _exceeds_ Path.Max.Retrans and 8.3 says > to do it when it _reaches_ Path.Max.Retrans. > > 8.2 Path Failure Detection... > Each time the T3-rtx timer expires on any address, or when a > HEARTBEAT sent to an idle address is not acknowledged within a RTO, > the error counter of that destination address will be incremented. > When the value in the error counter exceeds the protocol parameter > 'Path.Max.Retrans' of that destination address, the endpoint should > mark the destination transport address as inactive, and a > notification SHOULD be sent to the upper layer. > > 8.3 Path Heartbeat... > The endpoint should increment the respective error counter of the > destination transport address each time a HEARTBEAT is sent to that > address and not acknowledged within one RTO. > > When the value of this counter reaches the protocol parameter ' > Path.Max.Retrans', the endpoint should mark the corresponding > destination address as inactive if it is not so marked, and may also > optionally report to the upper layer the change of reachability of > this destination address. > > Which is correct? > We will need to fix this in the next I-G issuance... as far as which is correct... I don't think it really makes that much difference. Reaches makes more sense to me.. but I don't think it matters.. maybe others could give their opinion.. R -- Randall R. Stewart ITD Cisco Systems Inc. rrs@cisco.com 815-342-5222(cell) or 815-477-2127(office) ------------=_1070402767-18159-0-- From rohde@exp-math.uni-essen.de Wed Dec 3 07:59:45 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13409 for ; Wed, 3 Dec 2003 07:59:45 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ARWbn-00003F-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 07:59:59 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ARWbm-00002m-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 07:59:58 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 03 Dec 2003 05:00:09 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hB3CwuAt022924; Wed, 3 Dec 2003 04:58:56 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB3CwUKs029965 for sctp-impl-filtered; Wed, 3 Dec 2003 04:58:32 -0800 (PST) Message-Id: <200312031258.hB3CwUKs029965@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070456309-29957-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Path mtu discovery List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Sebastian Rohde X-Nails-Approved: rohde@exp-math.uni-essen.de Date: 03 Dec 2003 13:54:00 +0100 This is a multi-part message in MIME format... ------------=_1070456309-29957-0 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: binary Hi, I am currently working on the sctplib to implement path mtu discovery. Unfortunately I have a problem with that ;) Please let me give you an example of what i am thinking about: Let's assume somebody sends a large Message with the DF-Bit set - After that but before we receive the icmp frag needed we send a Message containing a data chunk which is - along with all headers - the size of the current known path mtu. When this message is lost due to a lower path mtu and a frag needed is received the path mtu for this path will be adjusted and all further data chunks will be bundled to fit within this new mtu. The problem is now that we already have bundled at least one data chunk at a size that exceeds the current path mtu. I don't think that rebundling would be an option as we would run into trouble with the continuity of tsns. On the other hand at least the linux kernel does not allow to send a larger sized packet as he remembers from the new path mtu and returns an error as long as path mtu discovery is enabled. An Option would be to retransmit the packets which are too big without the df bit set and accept that this message will be fragmented. How do others handle this problem? Another question related to this would be whether to send all packets with df set or just selected packets. If doing the latter with ipv6 the linux kernel does not allow(as least as far as i know) to send messages greater than the minimum link mtu 1280 which is really undesirable. I am currently thinking about a solution which sends every packet with df bit set and only if I have to send a packet larger than the current path mtu(resulting from the scenario above) I will disable the pmtud and let the linux kernel fragment the packet into his know path mtu size or for ipv6 in 1280 sized packets. I am not working very long with these internals of sctp and linux so please excuse and correct me if i got something wrong ;-). Regards, Sebastian Rohde ------------=_1070456309-29957-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 3 17:12:42 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA12070 for ; Wed, 3 Dec 2003 17:12:42 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ARfEu-00033r-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 17:12:56 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1ARfEu-00032X-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 17:12:56 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 03 Dec 2003 14:12:50 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB3MBTiN029290; Wed, 3 Dec 2003 14:11:30 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB3MAMKs007127 for sctp-impl-filtered; Wed, 3 Dec 2003 14:10:24 -0800 (PST) Message-Id: <200312032210.hB3MAMKs007127@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070489421-7125-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path mtu discovery List-Id: sctp-impl To: Sebastian Rohde From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 3 Dec 2003 22:29:36 +0100 This is a multi-part message in MIME format... ------------=_1070489421-7125-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Sebastian, see my comments below. Best regards Michael On Dec 3, 2003, at 1:54 PM, Sebastian Rohde wrote: > Hi, > > I am currently working on the sctplib to implement path mtu discovery. > > Unfortunately I have a problem with that ;) > Please let me give you an example of what i am thinking about: > Let's assume somebody sends a large Message with the DF-Bit set - After > that but before we receive the icmp frag needed we send a Message > containing a data chunk which is - along with all headers - the size of > the current known path mtu. > When this message is lost due to a lower path mtu and a frag needed is > received the path mtu for this path will be adjusted and all further > data chunks will be bundled to fit within this new mtu. > I'm not sure what you mean by bundling here. You can change the bundling with each packet you send. The problem is if you send a packet consisting of a common header and one large DATA chunk only. And now your path MTU shrinks. You have only two choices: 1. Send the packet again and do not set the DF bit and make use if IP fragmentation. 2. Use PR-SCTP to tell the peer to drop the packet and segment the user data into more DATA chunks and send this again. This might not be possible because your peer does not support PR-SCTP or the upper layer can not handle the possible reordering of the user data. > The problem is now that we already have bundled at least one data chunk > at a size that exceeds the current path mtu. I don't think that > rebundling would be an option as we would run into trouble with the > continuity of tsns. > On the other hand at least the linux kernel does not allow to send a > larger sized packet as he remembers from the new path mtu and returns > an > error as long as path mtu discovery is enabled. > I do not see the problem here. You should not set this option because the sctplib runs on multiple platforms and you should not rely on a feature only available on one platform. The sctplib should handle path MTU discovery by itself only by having a raw IP socket for ICMP and having a callback for the ICMP messages. > An Option would be to retransmit the packets which are too big without > the df bit set and accept that this message will be fragmented. > > How do others handle this problem? > > Another question related to this would be whether to send all packets > with df set or just selected packets. If doing the latter with ipv6 the > linux kernel does not allow(as least as far as i know) to send messages > greater than the minimum link mtu 1280 which is really undesirable. > > > I am currently thinking about a solution which sends every packet with > df bit set and only if I have to send a packet larger than the current > path mtu(resulting from the scenario above) I will disable the pmtud > and > let the linux kernel fragment the packet into his know path mtu size or > for ipv6 in 1280 sized packets. > i would strongly suggest not to make use of any feature in the linux kernel. Just use a raw socket to receive ICMP packets. This works on all platforms. > I am not working very long with these internals of sctp and linux so > please excuse and correct me if i got something wrong ;-). > > Regards, > Sebastian Rohde > > > > > ------------=_1070489421-7125-0-- From rrs@cisco.com Wed Dec 3 18:33:39 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16049 for ; Wed, 3 Dec 2003 18:33:39 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ARgVF-0004Dn-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 18:33:53 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1ARgVF-0004Cs-00 for sctp-impl-archive@ietf.org; Wed, 03 Dec 2003 18:33:53 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB3NWeiN020154; Wed, 3 Dec 2003 15:32:41 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB3NW2Ks008164 for sctp-impl-filtered; Wed, 3 Dec 2003 15:32:04 -0800 (PST) Message-Id: <200312032332.hB3NW2Ks008164@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070494322-8162-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path mtu discovery List-Id: sctp-impl To: Michael Tuexen From: "Randall Stewart (cisco)" Cc: Sebastian Rohde , sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Wed, 03 Dec 2003 17:31:55 -0600 This is a multi-part message in MIME format... ------------=_1070494322-8162-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael: I posted earlier.. It did not go to the list due to an error in the filtering script which Don as now fixed... Michael Tuexen wrote: > Sebastian, > > see my comments below. > > Best regards > Michael > > On Dec 3, 2003, at 1:54 PM, Sebastian Rohde wrote: > >> Hi, >> >> I am currently working on the sctplib to implement path mtu discovery. >> >> Unfortunately I have a problem with that ;) >> Please let me give you an example of what i am thinking about: >> Let's assume somebody sends a large Message with the DF-Bit set - After >> that but before we receive the icmp frag needed we send a Message >> containing a data chunk which is - along with all headers - the size of >> the current known path mtu. >> When this message is lost due to a lower path mtu and a frag needed is >> received the path mtu for this path will be adjusted and all further >> data chunks will be bundled to fit within this new mtu. >> > I'm not sure what you mean by bundling here. You can change the bundling > with each packet you send. The problem is if you send a packet consisting > of a common header and one large DATA chunk only. And now your path MTU > shrinks. You have only two choices: > 1. Send the packet again and do not set the DF bit and make use if IP > fragmentation. The above is the standard way you handle this in SCTP or TCP... KAME does it this way... > 2. Use PR-SCTP to tell the peer to drop the packet and segment the > user data into more DATA chunks and send this again. This might not > be possible because your peer does not support PR-SCTP or the upper > layer > can not handle the possible reordering of the user data. unless of course the packet is PR-SCTP .. and both sides support it as you illustrate above :-> > >> The problem is now that we already have bundled at least one data chunk >> at a size that exceeds the current path mtu. I don't think that >> rebundling would be an option as we would run into trouble with the >> continuity of tsns. >> On the other hand at least the linux kernel does not allow to send a >> larger sized packet as he remembers from the new path mtu and returns an >> error as long as path mtu discovery is enabled. >> > I do not see the problem here. You should not set this option because > the sctplib runs on multiple platforms and you should not rely on a > feature only available on one platform. The sctplib should handle > path MTU discovery by itself only by having a raw IP socket for ICMP > and having a callback for the ICMP messages. We had this working in the original ref-impl that Q and I did.. we had both the raw IP socket and a raw ICMP socket.. you have to look at a few more packets and only pay attention to the SCTP ones.. but it should work fine :-> R >> An Option would be to retransmit the packets which are too big without >> the df bit set and accept that this message will be fragmented. >> >> How do others handle this problem? >> >> Another question related to this would be whether to send all packets >> with df set or just selected packets. If doing the latter with ipv6 the >> linux kernel does not allow(as least as far as i know) to send messages >> greater than the minimum link mtu 1280 which is really undesirable. >> >> >> I am currently thinking about a solution which sends every packet with >> df bit set and only if I have to send a packet larger than the current >> path mtu(resulting from the scenario above) I will disable the pmtud and >> let the linux kernel fragment the packet into his know path mtu size or >> for ipv6 in 1280 sized packets. >> > i would strongly suggest not to make use of any feature in the linux > kernel. > Just use a raw socket to receive ICMP packets. This works on all > platforms. > >> I am not working very long with these internals of sctp and linux so >> please excuse and correct me if i got something wrong ;-). >> >> Regards, >> Sebastian Rohde >> >> >> >> >> > > -- Randall R. Stewart ITD Cisco Systems Inc. rrs@cisco.com 815-342-5222(cell) or 815-477-2127(office) ------------=_1070494322-8162-0-- From rohde@exp-math.uni-essen.de Mon Dec 8 05:24:03 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA03973 for ; Mon, 8 Dec 2003 05:24:03 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATIYq-0000Uk-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 05:24:16 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1ATIYq-0000UP-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 05:24:16 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB8AN6iN012710; Mon, 8 Dec 2003 02:23:07 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB8AM0Ks001281 for sctp-impl-filtered; Mon, 8 Dec 2003 02:22:02 -0800 (PST) Message-Id: <200312081022.hB8AM0Ks001281@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070878920-1279-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path mtu discovery List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Sebastian Rohde X-Nails-Approved: rohde@exp-math.uni-essen.de Date: 08 Dec 2003 11:17:28 +0100 This is a multi-part message in MIME format... ------------=_1070878920-1279-0 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: binary Thank you for your answers. I was out of office on thursday and friday so I was unable answer these days. Please see my comments below. > > I'm not sure what you mean by bundling here. You can change the bundling > > with each packet you send. The problem is if you send a packet consisting > > of a common header and one large DATA chunk only. And now your path MTU > > shrinks. You have only two choices: > > 1. Send the packet again and do not set the DF bit and make use if IP > > fragmentation. This was what i meant ;) > > The above is the standard way you handle this in SCTP or TCP... KAME does > it this way... > > > 2. Use PR-SCTP to tell the peer to drop the packet and segment the > > user data into more DATA chunks and send this again. This might not > > be possible because your peer does not support PR-SCTP or the upper > > layer > > can not handle the possible reordering of the user data. I was thinking about that but as you said - we would not be anymore ordered in this case. > > > unless of course the packet is PR-SCTP .. and both sides support it as you > illustrate above :-> > > > > >> The problem is now that we already have bundled at least one data chunk > >> at a size that exceeds the current path mtu. I don't think that > >> rebundling would be an option as we would run into trouble with the > >> continuity of tsns. > >> On the other hand at least the linux kernel does not allow to send a > >> larger sized packet as he remembers from the new path mtu and returns an > >> error as long as path mtu discovery is enabled. > >> As far as I know there is no option to just set the df bit in the linux kernel - You can only enable PMTU Discovery or disable it. Normally this is no problem but when using ipv6 this becomes a small problem in a scenario as stated below. > > I do not see the problem here. You should not set this option because > > the sctplib runs on multiple platforms and you should not rely on a > > feature only available on one platform. The sctplib should handle > > path MTU discovery by itself only by having a raw IP socket for ICMP > > and having a callback for the ICMP messages. > > > We had this working in the original ref-impl that Q and I did.. we had > both the raw IP socket and a raw ICMP socket.. you have to > look at a few more packets and only pay attention to the SCTP > ones.. but it should work fine :-> This is the way I was handling it(And I will go on handling it this way ;)) before i discovered the problem and searched for good alternatives. > > R > > >> An Option would be to retransmit the packets which are too big without > >> the df bit set and accept that this message will be fragmented. > >> > >> How do others handle this problem? > >> > >> Another question related to this would be whether to send all packets > >> with df set or just selected packets. If doing the latter with ipv6 the > >> linux kernel does not allow(as least as far as i know) to send messages > >> greater than the minimum link mtu 1280 which is really undesirable. > >> > >> > >> I am currently thinking about a solution which sends every packet with > >> df bit set and only if I have to send a packet larger than the current > >> path mtu(resulting from the scenario above) I will disable the pmtud and > >> let the linux kernel fragment the packet into his know path mtu size or > >> for ipv6 in 1280 sized packets. > >> > > i would strongly suggest not to make use of any feature in the linux > > kernel. > > Just use a raw socket to receive ICMP packets. This works on all > > platforms. The point here was that with ipv6 the linux kernel works in some circumstances against the own implementation ;-(. Originally we planned not to send every packet with the df bit set but only one large packet in a given timeframe to allow a better handling of firewalls which block all icmp traffic - The linux kernel does in fact work against this with ipv6 as it automatically fragments every packet which is larger than 1280 when the IPV6_PMTUDISC_DONT sockoption is set. I think for ipv6 I will send every packet with the df bit set and if a packet gets to big ipv6 I will switch that off... Sebastian ------------=_1070878920-1279-0-- From Michael.Tuexen@micmac.franken.de Mon Dec 8 06:53:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA06401 for ; Mon, 8 Dec 2003 06:53:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATJx5-0001hc-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 06:53:23 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1ATJx5-0001hA-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 06:53:23 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 08 Dec 2003 11:53:35 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB8BqOiN012525; Mon, 8 Dec 2003 03:52:24 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB8BpoKs002694 for sctp-impl-filtered; Mon, 8 Dec 2003 03:51:52 -0800 (PST) Message-Id: <200312081151.hB8BpoKs002694@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070884310-2692-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path mtu discovery List-Id: sctp-impl To: Sebastian Rohde From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 8 Dec 2003 12:11:11 +0100 This is a multi-part message in MIME format... ------------=_1070884310-2692-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Sebastian, see my comments below. Best regards Michael On Dec 8, 2003, at 11:17 AM, Sebastian Rohde wrote: > Thank you for your answers. I was out of office on thursday and friday > so I was unable answer these days. > Please see my comments below. > >>> I'm not sure what you mean by bundling here. You can change the >>> bundling >>> with each packet you send. The problem is if you send a packet >>> consisting >>> of a common header and one large DATA chunk only. And now your path >>> MTU >>> shrinks. You have only two choices: >>> 1. Send the packet again and do not set the DF bit and make use if IP >>> fragmentation. > This was what i meant ;) > >> >> The above is the standard way you handle this in SCTP or TCP... KAME >> does >> it this way... >> >>> 2. Use PR-SCTP to tell the peer to drop the packet and segment the >>> user data into more DATA chunks and send this again. This might >>> not >>> be possible because your peer does not support PR-SCTP or the >>> upper >>> layer >>> can not handle the possible reordering of the user data. > I was thinking about that but as you said - we would not be anymore > ordered in this case. >> This is not a general solution. >> >> unless of course the packet is PR-SCTP .. and both sides support it >> as you >> illustrate above :-> >> >>> >>>> The problem is now that we already have bundled at least one data >>>> chunk >>>> at a size that exceeds the current path mtu. I don't think that >>>> rebundling would be an option as we would run into trouble with the >>>> continuity of tsns. >>>> On the other hand at least the linux kernel does not allow to send a >>>> larger sized packet as he remembers from the new path mtu and >>>> returns an >>>> error as long as path mtu discovery is enabled. >>>> > As far as I know there is no option to just set the df bit in the linux > kernel - You can only enable PMTU Discovery or disable it. Normally > this > is no problem but when using ipv6 this becomes a small problem in a > scenario as stated below. You can use a raw IP socket an set it by yourself. Using the IP_HDRINCL option you have to provide the complete IP-header by your own. So setting the bit is no problem. And it works on all platforms currently supported by the sctplib. >>> I do not see the problem here. You should not set this option because >>> the sctplib runs on multiple platforms and you should not rely on a >>> feature only available on one platform. The sctplib should handle >>> path MTU discovery by itself only by having a raw IP socket for ICMP >>> and having a callback for the ICMP messages. >> >> >> We had this working in the original ref-impl that Q and I did.. we had >> both the raw IP socket and a raw ICMP socket.. you have to >> look at a few more packets and only pay attention to the SCTP >> ones.. but it should work fine :-> > This is the way I was handling it(And I will go on handling it this way > ;)) before i discovered the problem and searched for good alternatives. >> >> R >> >>>> An Option would be to retransmit the packets which are too big >>>> without >>>> the df bit set and accept that this message will be fragmented. >>>> >>>> How do others handle this problem? >>>> >>>> Another question related to this would be whether to send all >>>> packets >>>> with df set or just selected packets. If doing the latter with ipv6 >>>> the >>>> linux kernel does not allow(as least as far as i know) to send >>>> messages >>>> greater than the minimum link mtu 1280 which is really undesirable. >>>> >>>> >>>> I am currently thinking about a solution which sends every packet >>>> with >>>> df bit set and only if I have to send a packet larger than the >>>> current >>>> path mtu(resulting from the scenario above) I will disable the >>>> pmtud and >>>> let the linux kernel fragment the packet into his know path mtu >>>> size or >>>> for ipv6 in 1280 sized packets. >>>> >>> i would strongly suggest not to make use of any feature in the linux >>> kernel. >>> Just use a raw socket to receive ICMP packets. This works on all >>> platforms. > The point here was that with ipv6 the linux kernel works in some > circumstances against the own implementation ;-(. Originally we planned > not to send every packet with the df bit set but only one large packet > in a given timeframe to allow a better handling of firewalls which > block > all icmp traffic - The linux kernel does in fact work against this with > ipv6 as it automatically fragments every packet which is larger than > 1280 when the IPV6_PMTUDISC_DONT sockoption is set. Please keep in mind that the sctplib is not Linux only. You should provide a solution for all OSes. You can use an optimized version for a particular OS but I would prefer a more general solution. This is and will always be a prototype implementation and any special handling on a per OS basis has to be tested separately. > > > I think for ipv6 I will send every packet with the df bit set and if a > packet gets to big ipv6 I will switch that off... > > Sebastian > ------------=_1070884310-2692-0-- From rrs@cisco.com Mon Dec 8 16:15:49 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA03319 for ; Mon, 8 Dec 2003 16:15:49 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATSjc-0005lr-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 16:16:04 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ATSjb-0005kr-00 for sctp-impl-archive@ietf.org; Mon, 08 Dec 2003 16:16:03 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 08 Dec 2003 13:17:20 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hB8LF2At019727; Mon, 8 Dec 2003 13:15:03 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB8LEeKs009955 for sctp-impl-filtered; Mon, 8 Dec 2003 13:14:42 -0800 (PST) Message-Id: <200312082114.hB8LEeKs009955@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070918080-9947-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path mtu discovery List-Id: sctp-impl To: Sebastian Rohde From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Mon, 08 Dec 2003 15:14:58 -0600 This is a multi-part message in MIME format... ------------=_1070918080-9947-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Sebastian Rohde wrote: >Thank you for your answers. I was out of office on thursday and friday >so I was unable answer these days. >Please see my comments below. > > > >>>I'm not sure what you mean by bundling here. You can change the bundling >>>with each packet you send. The problem is if you send a packet consisting >>>of a common header and one large DATA chunk only. And now your path MTU >>>shrinks. You have only two choices: >>>1. Send the packet again and do not set the DF bit and make use if IP >>>fragmentation. >>> >>> >This was what i meant ;) > > > >>The above is the standard way you handle this in SCTP or TCP... KAME does >>it this way... >> >> >> >>>2. Use PR-SCTP to tell the peer to drop the packet and segment the >>> user data into more DATA chunks and send this again. This might not >>> be possible because your peer does not support PR-SCTP or the upper >>>layer >>> can not handle the possible reordering of the user data. >>> >>> >I was thinking about that but as you said - we would not be anymore >ordered in this case. > > >>unless of course the packet is PR-SCTP .. and both sides support it as you >>illustrate above :-> >> >> >> >>>>The problem is now that we already have bundled at least one data chunk >>>>at a size that exceeds the current path mtu. I don't think that >>>>rebundling would be an option as we would run into trouble with the >>>>continuity of tsns. >>>>On the other hand at least the linux kernel does not allow to send a >>>>larger sized packet as he remembers from the new path mtu and returns an >>>>error as long as path mtu discovery is enabled. >>>> >>>> >>>> >As far as I know there is no option to just set the df bit in the linux >kernel - You can only enable PMTU Discovery or disable it. Normally this >is no problem but when using ipv6 this becomes a small problem in a >scenario as stated below. > > >>>I do not see the problem here. You should not set this option because >>>the sctplib runs on multiple platforms and you should not rely on a >>>feature only available on one platform. The sctplib should handle >>>path MTU discovery by itself only by having a raw IP socket for ICMP >>>and having a callback for the ICMP messages. >>> >>> >>We had this working in the original ref-impl that Q and I did.. we had >>both the raw IP socket and a raw ICMP socket.. you have to >>look at a few more packets and only pay attention to the SCTP >>ones.. but it should work fine :-> >> >> >This is the way I was handling it(And I will go on handling it this way >;)) before i discovered the problem and searched for good alternatives. > > >>R >> >> >> >>>>An Option would be to retransmit the packets which are too big without >>>>the df bit set and accept that this message will be fragmented. >>>> >>>>How do others handle this problem? >>>> >>>>Another question related to this would be whether to send all packets >>>>with df set or just selected packets. If doing the latter with ipv6 the >>>>linux kernel does not allow(as least as far as i know) to send messages >>>>greater than the minimum link mtu 1280 which is really undesirable. >>>> >>>> >>>>I am currently thinking about a solution which sends every packet with >>>>df bit set and only if I have to send a packet larger than the current >>>>path mtu(resulting from the scenario above) I will disable the pmtud and >>>>let the linux kernel fragment the packet into his know path mtu size or >>>>for ipv6 in 1280 sized packets. >>>> >>>> >>>> >>>i would strongly suggest not to make use of any feature in the linux >>>kernel. >>>Just use a raw socket to receive ICMP packets. This works on all >>>platforms. >>> >>> >The point here was that with ipv6 the linux kernel works in some >circumstances against the own implementation ;-(. Originally we planned >not to send every packet with the df bit set but only one large packet >in a given timeframe to allow a better handling of firewalls which block >all icmp traffic - The linux kernel does in fact work against this with >ipv6 as it automatically fragments every packet which is larger than >1280 when the IPV6_PMTUDISC_DONT sockoption is set. > > >I think for ipv6 I will send every packet with the df bit set and if a >packet gets to big ipv6 I will switch that off... > >Sebastian > > > > Sebastian: IPv6 does NOT have a DF bit. What happens is the IPv6 layer will fragment for you.. you just need to pay attention to path mtu changes and set your MTU accordingly... R ------------=_1070918080-9947-0-- From kacheong.poon@sun.com Tue Dec 9 04:56:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA19989 for ; Tue, 9 Dec 2003 04:56:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATebQ-0004eX-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 04:56:24 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1ATebP-0004e4-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 04:56:23 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB99tDiN005746; Tue, 9 Dec 2003 01:55:13 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB99sZKs019725 for sctp-impl-filtered; Tue, 9 Dec 2003 01:54:37 -0800 (PST) Message-Id: <200312090954.hB99sZKs019725@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070963675-19723-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Kacheong Poon X-Nails-Approved: kacheong.poon@sun.com Date: Tue, 09 Dec 2003 01:54:42 -0800 This is a multi-part message in MIME format... ------------=_1070963675-19723-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary I'd like to gather opinions about making sctp_peeloff() work in one-one style SCTP socket. The semantics is creating a new socket for a particular stream of an association. The stream number will be passed in the place of the assoc_id in the current sctp_peeloff(). Do people think that their apps can benefit from this? Thanks. -- K. Poon. kacheong.poon@sun.com ------------=_1070963675-19723-0-- From rrs@cisco.com Tue Dec 9 06:38:05 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA22950 for ; Tue, 9 Dec 2003 06:38:04 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATgC2-0005wn-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 06:38:18 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ATgC2-0005wd-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 06:38:18 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hB9BbMBN026224; Tue, 9 Dec 2003 03:37:22 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9BaTKs021292 for sctp-impl-filtered; Tue, 9 Dec 2003 03:36:31 -0800 (PST) Message-Id: <200312091136.hB9BaTKs021292@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070969789-21290-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Tue, 09 Dec 2003 05:36:44 -0600 This is a multi-part message in MIME format... ------------=_1070969789-21290-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong Poon wrote: > I'd like to gather opinions about making sctp_peeloff() > work in one-one style SCTP socket. The semantics is > creating a new socket for a particular stream of an > association. The stream number will be passed in the > place of the assoc_id in the current sctp_peeloff(). > > Do people think that their apps can benefit from this? > Thanks. > Hmm... There does seem to be some interest (at least I have been contacted privately several times) on making it so that you can read from a particular stream... I don't think I would be opposed to this... it will be a bit of code for me, but has you have said .. its just code... and it may well be quite useful... Let's see what other developers think... I could go for it :-> R ------------=_1070969789-21290-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 9 13:01:56 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05944 for ; Tue, 9 Dec 2003 13:01:56 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATmBY-0004Ug-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 13:02:12 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ATmBX-0004So-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 13:02:11 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 09 Dec 2003 10:03:38 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hB9I19At010600; Tue, 9 Dec 2003 10:01:10 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9I0ZKs026284 for sctp-impl-filtered; Tue, 9 Dec 2003 10:00:37 -0800 (PST) Message-Id: <200312091800.hB9I0ZKs026284@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070992835-26282-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Kacheong Poon X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Tue, 9 Dec 2003 18:19:14 +0100 This is a multi-part message in MIME format... ------------=_1070992835-26282-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Hi all, i think having a fd for a stream is a real problem. If the user 'decides' not to read from a particular stream because there is a separate thread/process for it which is blocked the receiver can not forward the cumulative TSN and therefore blocks the whole association. The receiver should always return in the receive call the (part of) a DATA chunk with lowest TSN. Best regards Michael On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: > Kacheong Poon wrote: > >> I'd like to gather opinions about making sctp_peeloff() >> work in one-one style SCTP socket. The semantics is >> creating a new socket for a particular stream of an >> association. The stream number will be passed in the >> place of the assoc_id in the current sctp_peeloff(). >> >> Do people think that their apps can benefit from this? >> Thanks. >> > Hmm... There does seem to be some interest (at least > I have been contacted privately several times) on making > it so that you can read from a particular stream... > > I don't think I would be opposed to this... it will be a bit > of code for me, but has you have said .. its just code... and > it may well be quite useful... > > Let's see what other developers think... I could go for it :-> > > R > ------------=_1070992835-26282-0-- From rrs@cisco.com Tue Dec 9 13:11:24 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06256 for ; Tue, 9 Dec 2003 13:11:24 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATmKh-0004bv-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 13:11:40 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1ATmKh-0004bM-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 13:11:39 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 09 Dec 2003 10:12:48 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hB9IAnZ0014926; Tue, 9 Dec 2003 10:10:49 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9I9AKs026399 for sctp-impl-filtered; Tue, 9 Dec 2003 10:09:12 -0800 (PST) Message-Id: <200312091809.hB9I9AKs026399@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1070993350-26397-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com, Kacheong Poon X-Nails-Approved: rrs@cisco.com Date: Tue, 09 Dec 2003 12:09:22 -0600 This is a multi-part message in MIME format... ------------=_1070993350-26397-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael: Interesting point... but is not the problem NOT that the cum-TSN cannot be forwarded... you can always ack the data even though the receiving stream is holding off reading.. but the issue is more how does one manage the rwnd in this situation... I think this is where you are going... If socket a-stream-1 stops reading and socket a-stream-2 is reading.. then it is possible that a-stream-1 will hold all of the rwnd and thus block the association... Since we have no way to apportion the rwnd per stream this is a problem.. R Michael Tuexen wrote: > Hi all, > > i think having a fd for a stream is a real problem. If the > user 'decides' not to read from a particular stream because > there is a separate thread/process for it which is blocked > the receiver can not forward the cumulative TSN and therefore > blocks the whole association. > > The receiver should always return in the receive call the > (part of) a DATA chunk with lowest TSN. > > Best regards > Michael > > On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: > >> Kacheong Poon wrote: >> >>> I'd like to gather opinions about making sctp_peeloff() >>> work in one-one style SCTP socket. The semantics is >>> creating a new socket for a particular stream of an >>> association. The stream number will be passed in the >>> place of the assoc_id in the current sctp_peeloff(). >>> >>> Do people think that their apps can benefit from this? >>> Thanks. >>> >> Hmm... There does seem to be some interest (at least >> I have been contacted privately several times) on making >> it so that you can read from a particular stream... >> >> I don't think I would be opposed to this... it will be a bit >> of code for me, but has you have said .. its just code... and >> it may well be quite useful... >> >> Let's see what other developers think... I could go for it :-> >> >> R >> > > ------------=_1070993350-26397-0-- From cait@asomi.com Tue Dec 9 15:02:58 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA10924 for ; Tue, 9 Dec 2003 15:02:58 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATo4e-0006f7-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 15:03:13 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1ATo4e-0006eE-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 15:03:12 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hB9K24iN006745; Tue, 9 Dec 2003 12:02:05 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9K1gKs027820 for sctp-impl-filtered; Tue, 9 Dec 2003 12:01:44 -0800 (PST) Message-Id: <200312092001.hB9K1gKs027820@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071000102-27818-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Caitlin Bestler Cc: sctp-impl@external.cisco.com X-Nails-Approved: cait@asomi.com Date: Tue, 9 Dec 2003 12:15:03 -0600 This is a multi-part message in MIME format... ------------=_1071000102-27818-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 9, 2003, at 3:54 AM, Kacheong Poon wrote: > I'd like to gather opinions about making sctp_peeloff() > work in one-one style SCTP socket. The semantics is > creating a new socket for a particular stream of an > association. The stream number will be passed in the > place of the assoc_id in the current sctp_peeloff(). > > Do people think that their apps can benefit from this? > Thanks. There is a major benefit to being able to use stream specific buffers when reading. But with the sockets API that has to be weighed against the overhead of using poll/select to find out which stream has supplied the buffer. If you migrate to more of an asynchronous interface, I believe the ideal is to allow receive buffers to be pre-posted on a per-stream basis, but to receive the completions on a consolidated socket. In terms of a VIA/IB/RDMA style interface, each stream has its own Send and Receive Queues, but the entire association has a single Completion Queue. Personally, I believe there is more value in trying to add that type of functionality to the API than in running a classic synchronous socket for each stream. However, I would agree that there are uses for both. In particular, having to provide the target buffer for a read *before* knowing what stream the next message is for has always struck me as awkward. -- Caitlin Bestler - cait@asomi.com - http://asomi.com/ http://asomi.com/CaitlinBestlerPublicPgpKey.html ------------=_1071000102-27818-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 9 15:09:17 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11744 for ; Tue, 9 Dec 2003 15:09:16 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AToAl-0006l8-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 15:09:31 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AToAk-0006kg-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 15:09:30 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hB9K8brX017727; Tue, 9 Dec 2003 12:08:37 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9K8MKs027913 for sctp-impl-filtered; Tue, 9 Dec 2003 12:08:24 -0800 (PST) Message-Id: <200312092008.hB9K8MKs027913@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071000502-27911-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Kacheong Poon X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Tue, 9 Dec 2003 20:27:07 +0100 This is a multi-part message in MIME format... ------------=_1071000502-27911-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Randy, you are right, that was what I was thinking about. It came to my mind when i was thinking about receiver procedure of stream scheduling. This is a method to overcome the limitation of not having an rwnd per stream. This can then be handled completely by the sender and does not require any change of the protocol. However, it seems not to be a good idea to write emails having during a lab session... Best regards Michael On Dec 9, 2003, at 7:09 PM, Randall Stewart (cisco) wrote: > Michael: > > Interesting point... but is not the problem NOT > that the cum-TSN cannot be forwarded... you can > always ack the data even though the receiving stream > is holding off reading.. but the issue is more how > does one manage the rwnd in this situation... I think > this is where you are going... If socket a-stream-1 stops > reading and socket a-stream-2 is reading.. then it is possible > that a-stream-1 will hold all of the rwnd and thus block the > association... Since we have no way to apportion the rwnd per > stream this is a problem.. > > R > > Michael Tuexen wrote: > >> Hi all, >> >> i think having a fd for a stream is a real problem. If the >> user 'decides' not to read from a particular stream because >> there is a separate thread/process for it which is blocked >> the receiver can not forward the cumulative TSN and therefore >> blocks the whole association. >> >> The receiver should always return in the receive call the >> (part of) a DATA chunk with lowest TSN. >> >> Best regards >> Michael >> >> On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: >> >>> Kacheong Poon wrote: >>> >>>> I'd like to gather opinions about making sctp_peeloff() >>>> work in one-one style SCTP socket. The semantics is >>>> creating a new socket for a particular stream of an >>>> association. The stream number will be passed in the >>>> place of the assoc_id in the current sctp_peeloff(). >>>> >>>> Do people think that their apps can benefit from this? >>>> Thanks. >>>> >>> Hmm... There does seem to be some interest (at least >>> I have been contacted privately several times) on making >>> it so that you can read from a particular stream... >>> >>> I don't think I would be opposed to this... it will be a bit >>> of code for me, but has you have said .. its just code... and >>> it may well be quite useful... >>> >>> Let's see what other developers think... I could go for it :-> >>> >>> R >>> >> >> > > > ------------=_1071000502-27911-0-- From cait@asomi.com Tue Dec 9 16:57:04 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19114 for ; Tue, 9 Dec 2003 16:57:03 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATpr4-0001fk-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 16:57:18 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1ATpr4-0001ep-00 for sctp-impl-archive@ietf.org; Tue, 09 Dec 2003 16:57:18 -0500 Received: from sj-core-3.cisco.com (171.68.223.137) by sj-iport-5.cisco.com with ESMTP; 09 Dec 2003 21:57:43 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hB9LuMrX008321; Tue, 9 Dec 2003 13:56:22 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hB9LtmKs029267 for sctp-impl-filtered; Tue, 9 Dec 2003 13:55:50 -0800 (PST) Message-Id: <200312092155.hB9LtmKs029267@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071006947-29265-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Caitlin Bestler Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" , Kacheong Poon X-Nails-Approved: cait@asomi.com Date: Tue, 9 Dec 2003 15:51:53 -0600 This is a multi-part message in MIME format... ------------=_1071006947-29265-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > Hi all, > > i think having a fd for a stream is a real problem. If the > user 'decides' not to read from a particular stream because > there is a separate thread/process for it which is blocked > the receiver can not forward the cumulative TSN and therefore > blocks the whole association. > > The receiver should always return in the receive call the > (part of) a DATA chunk with lowest TSN. > That is one more strong point on the "gotchas" of attempting to do per-stream processing with a synchronous API. I still think that per-stream buffering is very useful, but it needs to be done in a way where the failure to provide a buffer for a given stream prevents a Data Chunk from being delivered to the application. With RDMA style interfaces the solution to this problem is very simple, streams MUST pre-post sufficient buffers. Failure to do so will result in the stream being reset (which allows the TSN to advance). But that is inconsistent with socket semantics, where data is held "somewhere" until the application asks for it. Given that applications SHOULD take delivery of all data within an association promptly then having "per stream" sockets is indeed dangerous. That suggests that stream specific processing is best achieved through a callback or RDMA style interface. -- Caitlin Bestler - cait@asomi.com - http://asomi.com/ http://asomi.com/CaitlinBestlerPublicPgpKey.html ------------=_1071006947-29265-0-- From torbjorn.andersson@kau.se Wed Dec 10 04:04:48 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA04886 for ; Wed, 10 Dec 2003 04:04:48 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU0H1-00061E-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 04:04:47 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU0H1-00060x-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 04:04:47 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBA93QZ0028358; Wed, 10 Dec 2003 01:03:27 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBA92nKs007827 for sctp-impl-filtered; Wed, 10 Dec 2003 01:02:51 -0800 (PST) Message-Id: <200312100902.hBA92nKs007827@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071046969-7825-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: From: =?iso-8859-1?Q?Torbj=F6rn_Andersson?= X-Nails-Approved: torbjorn.andersson@kau.se Date: Wed, 10 Dec 2003 10:01:06 +0100 This is a multi-part message in MIME format... ------------=_1071046969-7825-0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: binary I'm testing the SCTP-kame implementation and I have some specific questions about the SCTP send buffer in Kame. I want to know how the send buffer behaves. The reason to why I want to know this is that I'm studying the total transmission time for individual messages. I guess someone reading this mailing list know this. How big is it? Is it tuned to the cwnd? Is the size of the send buffer dependent of the congestion state, slowstart/fast recovery/congestion avoidance etc? Is it chunk or byte limited? Best Regards, Torbjörn -------------------------------------------------------------- | Torbjörn Andersson phone: +46 54 700 11 61 | Computer Science fax: +46 54 700 18 28 | Karlstad University URL: http://www.cs.kau.se/~tora --------------------------------------------------------------- ------------=_1071046969-7825-0-- From kacheong.poon@sun.com Wed Dec 10 05:07:08 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06073 for ; Wed, 10 Dec 2003 05:07:08 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU1FM-0006g7-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:07:08 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU1FM-0006f0-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:07:08 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 10 Dec 2003 02:08:45 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBAA6JAt016109; Wed, 10 Dec 2003 02:06:20 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAA68Ks008634 for sctp-impl-filtered; Wed, 10 Dec 2003 02:06:10 -0800 (PST) Message-Id: <200312101006.hBAA68Ks008634@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071050767-8632-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Kacheong Poon X-Nails-Approved: kacheong.poon@sun.com Date: Wed, 10 Dec 2003 02:07:05 -0800 This is a multi-part message in MIME format... ------------=_1071050767-8632-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Randall Stewart (cisco) wrote: > Michael: > > Interesting point... but is not the problem NOT > that the cum-TSN cannot be forwarded... you can > always ack the data even though the receiving stream > is holding off reading.. but the issue is more how > does one manage the rwnd in this situation... I think > this is where you are going... If socket a-stream-1 stops > reading and socket a-stream-2 is reading.. then it is possible > that a-stream-1 will hold all of the rwnd and thus block the > association... Since we have no way to apportion the rwnd per > stream this is a problem.. This can be an issue. If an app chooses to peel off a stream, it has to be careful in handling the individual stream-socket to avoid blocking of other streams. But this also opens up the socket interface for people to play with streams scheduling. Any idea on how to make it safer? > Michael Tuexen wrote: > >> Hi all, >> >> i think having a fd for a stream is a real problem. If the >> user 'decides' not to read from a particular stream because >> there is a separate thread/process for it which is blocked >> the receiver can not forward the cumulative TSN and therefore >> blocks the whole association. >> >> The receiver should always return in the receive call the >> (part of) a DATA chunk with lowest TSN. >> >> Best regards >> Michael >> >> On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: >> >>> Kacheong Poon wrote: >>> >>>> I'd like to gather opinions about making sctp_peeloff() >>>> work in one-one style SCTP socket. The semantics is >>>> creating a new socket for a particular stream of an >>>> association. The stream number will be passed in the >>>> place of the assoc_id in the current sctp_peeloff(). >>>> >>>> Do people think that their apps can benefit from this? >>>> Thanks. >>>> >>> Hmm... There does seem to be some interest (at least >>> I have been contacted privately several times) on making >>> it so that you can read from a particular stream... >>> >>> I don't think I would be opposed to this... it will be a bit >>> of code for me, but has you have said .. its just code... and >>> it may well be quite useful... >>> >>> Let's see what other developers think... I could go for it :-> >>> >>> R >>> >> >> > > -- K. Poon. kacheong.poon@sun.com ------------=_1071050767-8632-0-- From kacheong.poon@sun.com Wed Dec 10 05:14:43 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06342 for ; Wed, 10 Dec 2003 05:14:42 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU1Mh-0006n2-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:14:43 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AU1Mg-0006mw-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:14:42 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBAADnBN002937; Wed, 10 Dec 2003 02:13:50 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAADfKs008734 for sctp-impl-filtered; Wed, 10 Dec 2003 02:13:43 -0800 (PST) Message-Id: <200312101013.hBAADfKs008734@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071051221-8732-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Kacheong Poon X-Nails-Approved: kacheong.poon@sun.com Date: Wed, 10 Dec 2003 02:14:16 -0800 This is a multi-part message in MIME format... ------------=_1071051221-8732-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin Bestler wrote: > There is a major benefit to being able to use stream > specific buffers when reading. > > But with the sockets API that has to be weighed against > the overhead of using poll/select to find out which > stream has supplied the buffer. > > If you migrate to more of an asynchronous interface, > I believe the ideal is to allow receive buffers to be > pre-posted on a per-stream basis, but to receive the > completions on a consolidated socket. In terms of a > VIA/IB/RDMA style interface, each stream has its own > Send and Receive Queues, but the entire association > has a single Completion Queue. One catch is that this kind of schemes, event completion queue, normally align with file descriptor. This means that after buffers are pre-posted to a socket, the completion event is delivered to the same socket. The reason is simply that the framework is also used by other file descriptor, not just socket. I agree that different implementations may have different architectures. Can people comment on whether the above file descriptor alignment is common on different platforms' event completion framework? If this is true, then peeling off a stream from a socket fits what you mentioned above. > Personally, I believe there is more value in trying > to add that type of functionality to the API than in > running a classic synchronous socket for each stream. > However, I would agree that there are uses for both. > > In particular, having to provide the target buffer > for a read *before* knowing what stream the next > message is for has always struck me as awkward. > > -- K. Poon. kacheong.poon@sun.com ------------=_1071051221-8732-0-- From ajung@exp-math.uni-essen.de Wed Dec 10 05:24:08 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA06453 for ; Wed, 10 Dec 2003 05:24:08 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU1Vo-0006t2-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:24:08 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AU1Vn-0006sZ-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 05:24:08 -0500 Received: from sj-core-3.cisco.com (171.68.223.137) by sj-iport-5.cisco.com with ESMTP; 10 Dec 2003 10:24:42 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBAANErX024686; Wed, 10 Dec 2003 02:23:15 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAAN6Ks008859 for sctp-impl-filtered; Wed, 10 Dec 2003 02:23:08 -0800 (PST) Message-Id: <200312101023.hBAAN6Ks008859@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071051786-8857-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Andreas Jungmaier X-Nails-Approved: ajung@exp-math.uni-essen.de Date: Wed, 10 Dec 2003 11:17:46 +0100 This is a multi-part message in MIME format... ------------=_1071051786-8857-0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: binary -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi folks, to add up to the discussion: in cases where there is only an outgoing stream, but no incoming stream, applications can only send, not receive on a "stream-socket". In the opposite case, an application could only receive, but not send. I don't know if that's a problem, since that can happen as well with some streams within an SCTP association. However, I don't really like this idea. Best regards, Andreas On Wednesday 10 December 2003 11:07, Kacheong Poon wrote: > Randall Stewart (cisco) wrote: > > Michael: > > > > Interesting point... but is not the problem NOT > > that the cum-TSN cannot be forwarded... you can > > always ack the data even though the receiving stream > > is holding off reading.. but the issue is more how > > does one manage the rwnd in this situation... I think > > this is where you are going... If socket a-stream-1 stops > > reading and socket a-stream-2 is reading.. then it is possible > > that a-stream-1 will hold all of the rwnd and thus block the > > association... Since we have no way to apportion the rwnd per > > stream this is a problem.. > > This can be an issue. If an app chooses to peel off a > stream, it has to be careful in handling the individual > stream-socket to avoid blocking of other streams. But > this also opens up the socket interface for people to play > with streams scheduling. > > Any idea on how to make it safer? > > > Michael Tuexen wrote: > >> Hi all, > >> > >> i think having a fd for a stream is a real problem. If the > >> user 'decides' not to read from a particular stream because > >> there is a separate thread/process for it which is blocked > >> the receiver can not forward the cumulative TSN and therefore > >> blocks the whole association. > >> > >> The receiver should always return in the receive call the > >> (part of) a DATA chunk with lowest TSN. > >> > >> Best regards > >> Michael > >> > >> On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: > >>> Kacheong Poon wrote: > >>>> I'd like to gather opinions about making sctp_peeloff() > >>>> work in one-one style SCTP socket. The semantics is > >>>> creating a new socket for a particular stream of an > >>>> association. The stream number will be passed in the > >>>> place of the assoc_id in the current sctp_peeloff(). > >>>> > >>>> Do people think that their apps can benefit from this? > >>>> Thanks. > >>> > >>> Hmm... There does seem to be some interest (at least > >>> I have been contacted privately several times) on making > >>> it so that you can read from a particular stream... > >>> > >>> I don't think I would be opposed to this... it will be a bit > >>> of code for me, but has you have said .. its just code... and > >>> it may well be quite useful... > >>> > >>> Let's see what other developers think... I could go for it :-> > >>> > >>> R - -- Dipl.-Ing. Andreas Jungmaier Computer Networking Technology Group University of Duisburg-Essen http://www.exp-math.uni-essen.de/~ajung PGP Key-ID: D382 4AC0 PGP Key-Fingerprint: 228C 7C2C 3381 2DD4 9998 2812 5C0B 0B04 D382 4AC0 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQE/1vLKXAsLBNOCSsARAjVDAKDRlPy3Hkg8KUFbu31DjUFJ5ca1sQCglS3Z P5d8fnU5Jq3z+E+nB0Fn4IM= =/AL3 -----END PGP SIGNATURE----- ------------=_1071051786-8857-0-- From rrs@cisco.com Wed Dec 10 07:37:45 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA09986 for ; Wed, 10 Dec 2003 07:37:45 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU3b8-0000uN-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 07:37:46 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU3b7-0000tv-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 07:37:46 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBACZcAt015598; Wed, 10 Dec 2003 04:35:38 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBACYxKs010911 for sctp-impl-filtered; Wed, 10 Dec 2003 04:35:01 -0800 (PST) Message-Id: <200312101235.hBACYxKs010911@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071059699-10909-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: =?ISO-8859-1?Q?Torbj=F6rn_Andersson?= From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Wed, 10 Dec 2003 06:35:08 -0600 This is a multi-part message in MIME format... ------------=_1071059699-10909-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Torbjorn: Yes, there are a couple of us that know... in fact I am behind on getting a couple of patches in too :-< But as to your questions... Torbjörn Andersson wrote: >I'm testing the SCTP-kame implementation and I have some specific >questions about the SCTP send buffer in Kame. I want to know how the >send buffer behaves. The reason to why I want to know this is that I'm >studying the total transmission time for individual messages. I guess >someone reading this mailing list know this. > >How big is it? > The current send buffer and receive window are defaulted to the maximum socket buffer minus a bit... the actual code is in sctp_init() in sctp_usrreq.c and the lines look like: sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + MCLBYTES)); sctp_sendspace = min((min(SB_MAX, sb_max_adj)), ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT)); /* Now for the recv window, should we take the same * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX min * above. For now I will just copy. */ sctp_recvspace = sctp_sendspace; SB_MAX is set in sys/socketvar.h In order to get real high performance (GigE type speed) you will need to set this up higher from the shipped default (256*1024). My machine that I do high bandwidth testing on has its set to (2048 * 1024) You also can change this by doing sysctl since both the sendspace and recvspace are sysctl variables (at least in freebsd). >Is it tuned to the cwnd? > cwnd is tied to the network and may get limited by your send buffer and peers rwnd but it will never influence the value of your send space. >Is the size of the send buffer dependent of the congestion state, >slowstart/fast recovery/congestion avoidance etc? > Nope... these are unrelated items .. cwnd deals with the associations view of the network, sendspace deals with how much outstanding data a app can write down the socket buffer... You can also change the value down or up by doing a setsockopt() on the SND_BUF... >Is it chunk or byte limited? > There is also an additional limit placed on an implementation as well.. number of chunks... This is tuned so that if you do a WHOLE bunch of 1 byte sends for example: while(1) send(1 byte); You will block when you hit a threshold of chunks.. otherwise you would quickly exhaust the number of mbufs in the system which will have dire consequences... This value is setup to either be the max of (SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters) nmbclusters is tuneable in the config file.. .while the chunks on queue is a hardcoded #define currently set to 512 Hope that all helps.. R > >Best Regards, Torbjörn > >-------------------------------------------------------------- >| Torbjörn Andersson phone: +46 54 700 11 61 >| Computer Science fax: +46 54 700 18 28 >| Karlstad University URL: http://www.cs.kau.se/~tora >--------------------------------------------------------------- > > > > > ------------=_1071059699-10909-0-- From torbjorn.andersson@kau.se Wed Dec 10 09:24:00 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12599 for ; Wed, 10 Dec 2003 09:24:00 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU5Fx-0002lo-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 09:24:01 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AU5Fw-0002lA-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 09:24:01 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-5.cisco.com with ESMTP; 10 Dec 2003 14:24:36 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBAEN3BN003383; Wed, 10 Dec 2003 06:23:04 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAEMiKs012319 for sctp-impl-filtered; Wed, 10 Dec 2003 06:22:46 -0800 (PST) Message-Id: <200312101422.hBAEMiKs012319@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071066164-12317-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: SV: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: From: =?iso-8859-1?Q?Torbj=F6rn_Andersson?= Cc: "'Randall Stewart \(cisco\)'" X-Nails-Approved: torbjorn.andersson@kau.se Date: Wed, 10 Dec 2003 15:21:04 +0100 This is a multi-part message in MIME format... ------------=_1071066164-12317-0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: binary Thanks for your quick answer. Your answer has moved my focus from the send buffer to the blocking behavior of the socket. When does SCTP-block? Is it dependent of the state of the association? > -----Ursprungligt meddelande----- > Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] > Skickat: den 10 december 2003 13:35 > Till: Torbjörn Andersson > Kopia: sctp-impl@external.cisco.com > Ämne: Re: Question regarding the SCTP-kame implementation > > Torbjorn: > > Yes, there are a couple of us that know... in fact I am > behind on getting a couple of patches in too :-< > > But as to your questions... > > Torbjörn Andersson wrote: > > >I'm testing the SCTP-kame implementation and I have some specific > >questions about the SCTP send buffer in Kame. I want to know how the > >send buffer behaves. The reason to why I want to know this is that I'm > >studying the total transmission time for individual messages. I guess > >someone reading this mailing list know this. > > > >How big is it? > > > The current send buffer and receive window are > defaulted to the maximum socket buffer minus a bit... the > actual code is in sctp_init() in sctp_usrreq.c and the lines > look like: > > sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + > MCLBYTES)); > sctp_sendspace = min((min(SB_MAX, sb_max_adj)), > ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT)); > /* Now for the recv window, should we take the same > * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX min > * above. For now I will just copy. > */ > sctp_recvspace = sctp_sendspace; > > SB_MAX is set in sys/socketvar.h > > In order to get real high performance (GigE type speed) you will need > to set this up higher from the shipped default (256*1024). My machine > that I do high bandwidth testing on has its set to (2048 * 1024) > > You also can change this by doing sysctl since both the sendspace and > recvspace > are sysctl variables (at least in freebsd). > > > >Is it tuned to the cwnd? > > > cwnd is tied to the network and may get limited by your send > buffer and peers rwnd but it will never influence the value > of your send space. > > >Is the size of the send buffer dependent of the congestion state, > >slowstart/fast recovery/congestion avoidance etc? > > > Nope... these are unrelated items .. cwnd deals with > the associations view of the network, sendspace deals > with how much outstanding data a app can write down > the socket buffer... > > You can also change the value down or up by doing a > setsockopt() on the SND_BUF... > > > > >Is it chunk or byte limited? > > > There is also an additional limit placed on an implementation > as well.. number of chunks... This is tuned so that if you > do a WHOLE bunch of 1 byte sends for example: > > while(1) > send(1 byte); > > You will block when you hit a threshold of chunks.. otherwise > you would quickly exhaust the number of mbufs in the system > which will have dire consequences... > > This value is setup to either be the max of > (SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters) > > nmbclusters is tuneable in the config file.. .while the chunks on queue > is a hardcoded #define currently set to 512 > > Hope that all helps.. > > R > > > > >Best Regards, Torbjörn > > > >-------------------------------------------------------------- > >| Torbjörn Andersson phone: +46 54 700 11 61 > >| Computer Science fax: +46 54 700 18 28 > >| Karlstad University URL: http://www.cs.kau.se/~tora > >--------------------------------------------------------------- > > > > > > > > > > > ------------=_1071066164-12317-0-- From rrs@cisco.com Wed Dec 10 10:20:20 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16133 for ; Wed, 10 Dec 2003 10:20:20 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU68T-0003sG-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 10:20:21 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU68T-0003s5-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 10:20:21 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBAFJLAt004412; Wed, 10 Dec 2003 07:19:21 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAFIrKs013071 for sctp-impl-filtered; Wed, 10 Dec 2003 07:18:55 -0800 (PST) Message-Id: <200312101518.hBAFIrKs013071@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071069533-13069-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: SV: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: =?ISO-8859-1?Q?Torbj=F6rn_Andersson?= From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Wed, 10 Dec 2003 09:19:03 -0600 This is a multi-part message in MIME format... ------------=_1071069533-13069-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary It depends on a couple of things you need to answer for me... 1) Do you have blocking IO on? FNBIO I think is the socket opt 2) What model are you using 1-1 or 1-M? Generally the 1-1 model will block at connect()/accept() and when the send buffer is full... if blocking IO is on (I think the default)... If you do non-blocking IO then thats a different story.. oh and one of the bugs that is still in patch prepartion as to do with blocking io.. there is a problem in the 1-1 model when you do non-blocking io.. connect will fail.. I fixed this a while ago but just kept forgetting to get the right cvs diffs to kame.. its in the in_proto.c and in6_proto.c files (the 1-1 model needs PR_CONNREQUIRED) R Torbjörn Andersson wrote: >Thanks for your quick answer. Your answer has moved my focus from the >send buffer to the blocking behavior of the socket. > >When does SCTP-block? > Is it dependent of the state of the association? > > > > > >>-----Ursprungligt meddelande----- >>Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] >>Skickat: den 10 december 2003 13:35 >>Till: Torbjörn Andersson >>Kopia: sctp-impl@external.cisco.com >>Ämne: Re: Question regarding the SCTP-kame implementation >> >>Torbjorn: >> >>Yes, there are a couple of us that know... in fact I am >>behind on getting a couple of patches in too :-< >> >>But as to your questions... >> >>Torbjörn Andersson wrote: >> >> >> >>>I'm testing the SCTP-kame implementation and I have some specific >>>questions about the SCTP send buffer in Kame. I want to know how the >>>send buffer behaves. The reason to why I want to know this is that >>> >>> >I'm > > >>>studying the total transmission time for individual messages. I guess >>>someone reading this mailing list know this. >>> >>>How big is it? >>> >>> >>> >>The current send buffer and receive window are >>defaulted to the maximum socket buffer minus a bit... the >>actual code is in sctp_init() in sctp_usrreq.c and the lines >>look like: >> >> sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + >>MCLBYTES)); >> sctp_sendspace = min((min(SB_MAX, sb_max_adj)), >> ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT)); >> /* Now for the recv window, should we take the same >> * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX min >> * above. For now I will just copy. >> */ >> sctp_recvspace = sctp_sendspace; >> >>SB_MAX is set in sys/socketvar.h >> >>In order to get real high performance (GigE type speed) you will need >>to set this up higher from the shipped default (256*1024). My machine >>that I do high bandwidth testing on has its set to (2048 * 1024) >> >>You also can change this by doing sysctl since both the sendspace and >>recvspace >>are sysctl variables (at least in freebsd). >> >> >> >> >>>Is it tuned to the cwnd? >>> >>> >>> >>cwnd is tied to the network and may get limited by your send >>buffer and peers rwnd but it will never influence the value >>of your send space. >> >> >> >>>Is the size of the send buffer dependent of the congestion state, >>>slowstart/fast recovery/congestion avoidance etc? >>> >>> >>> >>Nope... these are unrelated items .. cwnd deals with >>the associations view of the network, sendspace deals >>with how much outstanding data a app can write down >>the socket buffer... >> >>You can also change the value down or up by doing a >>setsockopt() on the SND_BUF... >> >> >> >> >> >>>Is it chunk or byte limited? >>> >>> >>> >>There is also an additional limit placed on an implementation >>as well.. number of chunks... This is tuned so that if you >>do a WHOLE bunch of 1 byte sends for example: >> >>while(1) >> send(1 byte); >> >>You will block when you hit a threshold of chunks.. otherwise >>you would quickly exhaust the number of mbufs in the system >>which will have dire consequences... >> >>This value is setup to either be the max of >>(SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters) >> >>nmbclusters is tuneable in the config file.. .while the chunks on >> >> >queue > > >>is a hardcoded #define currently set to 512 >> >>Hope that all helps.. >> >>R >> >> >> >>>Best Regards, Torbjörn >>> >>>-------------------------------------------------------------- >>>| Torbjörn Andersson phone: +46 54 700 11 61 >>>| Computer Science fax: +46 54 700 18 28 >>>| Karlstad University URL: http://www.cs.kau.se/~tora >>>--------------------------------------------------------------- >>> >>> >>> >>> >>> >>> >>> > > > > > ------------=_1071069533-13069-0-- From kacheong.poon@sun.com Wed Dec 10 10:34:38 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16912 for ; Wed, 10 Dec 2003 10:34:38 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU6MK-0004C5-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 10:34:40 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU6MJ-0004Bl-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 10:34:40 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBAFXjAt017398; Wed, 10 Dec 2003 07:33:46 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAFXRKs013265 for sctp-impl-filtered; Wed, 10 Dec 2003 07:33:29 -0800 (PST) Message-Id: <200312101533.hBAFXRKs013265@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071070407-13263-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Kacheong Poon X-Nails-Approved: kacheong.poon@sun.com Date: Wed, 10 Dec 2003 07:33:24 -0800 This is a multi-part message in MIME format... ------------=_1071070407-13263-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Andreas Jungmaier wrote: > in cases where there is only an outgoing stream, but no > incoming stream, applications can only send, not receive > on a "stream-socket". > In the opposite case, an application could only receive, > but not send. Can you elaborate on the above? I don't quite get the issue. If there is only one stream, I guess an app will not do a sctp_peeloff(). A stream-socket is still bi-directional. The only difference is that it is a socket which only deals with one stream of an association. The original socket deals with all the other streams of an association except those which have been sctp_peeloff(). > I don't know if that's a problem, since that can happen > as well with some streams within an SCTP association. > > However, I don't really like this idea. -- K. Poon. kacheong.poon@sun.com ------------=_1071070407-13263-0-- From torbjorn.andersson@kau.se Wed Dec 10 11:41:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA20884 for ; Wed, 10 Dec 2003 11:41:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU7Oi-0005sY-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 11:41:12 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU7Og-0005TW-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 11:41:10 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBAGeIAt016507; Wed, 10 Dec 2003 08:40:19 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAGdqKs014106 for sctp-impl-filtered; Wed, 10 Dec 2003 08:39:54 -0800 (PST) Message-Id: <200312101639.hBAGdqKs014106@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071074394-14104-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: SV: SV: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: "'Randall Stewart \(cisco\)'" From: =?iso-8859-1?Q?Torbj=F6rn_Andersson?= Cc: X-Nails-Approved: torbjorn.andersson@kau.se Date: Wed, 10 Dec 2003 17:38:16 +0100 This is a multi-part message in MIME format... ------------=_1071074394-14104-0 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: binary I append a file to this message, which contain the results of an experiment I have performed. The experiment measured individual ULP message times, which include the time of getting through the send buffer, router buffer (12 segements) and network (25ms). Further retransmissions and reordering at the receiver can delay the messages. The experiment does not have any background traffic on the test network. Each message contains 500 bytes of data and they are sent as fast as possible. A 1-1 SCTP socket is used at the sender and the socket is configured as blocking (default). Now, one can see in the figure that the message times increase differently in four different phases. I was actually expecting one maybe two different phases until the buffer of the SCTP association becomes dominant and everything levels out. But it does not look like that and I believed first that there was some intelligent buffer management at the sender. But the send buffer is statically configured. The next thing I though could explain the different message increase speeds was the blocking algorithm. Does anyone have any input to this? > -----Ursprungligt meddelande----- > Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] > Skickat: den 10 december 2003 16:19 > Till: Torbjörn Andersson > Kopia: sctp-impl@external.cisco.com > Ämne: Re: SV: Question regarding the SCTP-kame implementation > > It depends on a couple of things you need to answer for me... > > 1) Do you have blocking IO on? FNBIO I think is the socket opt > 2) What model are you using 1-1 or 1-M? > > Generally the 1-1 model will block at connect()/accept() and > when the send buffer is full... if blocking IO is on (I think the > default)... If you do non-blocking IO then thats a different > story.. oh and one of the bugs that is still in patch prepartion as > to do with blocking io.. there is a problem in the 1-1 model > when you do non-blocking io.. connect will fail.. I fixed this > a while ago but just kept forgetting to get the right > cvs diffs to kame.. its in the in_proto.c and in6_proto.c files > (the 1-1 model needs PR_CONNREQUIRED) > > > R > > Torbjörn Andersson wrote: > > >Thanks for your quick answer. Your answer has moved my focus from the > >send buffer to the blocking behavior of the socket. > > > >When does SCTP-block? > > Is it dependent of the state of the association? > > > > > > > > > > > >>-----Ursprungligt meddelande----- > >>Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] > >>Skickat: den 10 december 2003 13:35 > >>Till: Torbjörn Andersson > >>Kopia: sctp-impl@external.cisco.com > >>Ämne: Re: Question regarding the SCTP-kame implementation > >> > >>Torbjorn: > >> > >>Yes, there are a couple of us that know... in fact I am > >>behind on getting a couple of patches in too :-< > >> > >>But as to your questions... > >> > >>Torbjörn Andersson wrote: > >> > >> > >> > >>>I'm testing the SCTP-kame implementation and I have some specific > >>>questions about the SCTP send buffer in Kame. I want to know how the > >>>send buffer behaves. The reason to why I want to know this is that > >>> > >>> > >I'm > > > > > >>>studying the total transmission time for individual messages. I guess > >>>someone reading this mailing list know this. > >>> > >>>How big is it? > >>> > >>> > >>> > >>The current send buffer and receive window are > >>defaulted to the maximum socket buffer minus a bit... the > >>actual code is in sctp_init() in sctp_usrreq.c and the lines > >>look like: > >> > >> sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + > >>MCLBYTES)); > >> sctp_sendspace = min((min(SB_MAX, sb_max_adj)), > >> ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT)); > >> /* Now for the recv window, should we take the same > >> * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX min > >> * above. For now I will just copy. > >> */ > >> sctp_recvspace = sctp_sendspace; > >> > >>SB_MAX is set in sys/socketvar.h > >> > >>In order to get real high performance (GigE type speed) you will need > >>to set this up higher from the shipped default (256*1024). My machine > >>that I do high bandwidth testing on has its set to (2048 * 1024) > >> > >>You also can change this by doing sysctl since both the sendspace and > >>recvspace > >>are sysctl variables (at least in freebsd). > >> > >> > >> > >> > >>>Is it tuned to the cwnd? > >>> > >>> > >>> > >>cwnd is tied to the network and may get limited by your send > >>buffer and peers rwnd but it will never influence the value > >>of your send space. > >> > >> > >> > >>>Is the size of the send buffer dependent of the congestion state, > >>>slowstart/fast recovery/congestion avoidance etc? > >>> > >>> > >>> > >>Nope... these are unrelated items .. cwnd deals with > >>the associations view of the network, sendspace deals > >>with how much outstanding data a app can write down > >>the socket buffer... > >> > >>You can also change the value down or up by doing a > >>setsockopt() on the SND_BUF... > >> > >> > >> > >> > >> > >>>Is it chunk or byte limited? > >>> > >>> > >>> > >>There is also an additional limit placed on an implementation > >>as well.. number of chunks... This is tuned so that if you > >>do a WHOLE bunch of 1 byte sends for example: > >> > >>while(1) > >> send(1 byte); > >> > >>You will block when you hit a threshold of chunks.. otherwise > >>you would quickly exhaust the number of mbufs in the system > >>which will have dire consequences... > >> > >>This value is setup to either be the max of > >>(SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters) > >> > >>nmbclusters is tuneable in the config file.. .while the chunks on > >> > >> > >queue > > > > > >>is a hardcoded #define currently set to 512 > >> > >>Hope that all helps.. > >> > >>R > >> > >> > >> > >>>Best Regards, Torbjörn > >>> > >>>-------------------------------------------------------------- > >>>| Torbjörn Andersson phone: +46 54 700 11 61 > >>>| Computer Science fax: +46 54 700 18 28 > >>>| Karlstad University URL: http://www.cs.kau.se/~tora > >>>--------------------------------------------------------------- > >>> > >>> > >>> > >>> > >>> > >>> > >>> > > > > > > > > > > > ------------=_1071074394-14104-0 Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: binary %!PS-Adobe-2.0 EPSF-2.0 %%Title: MsgDelay_SCTP(0ms)+O_0TCP.eps %%Creator: gnuplot 3.7 patchlevel 3 %%CreationDate: Mon Dec 8 09:16:26 2003 %%DocumentFonts: (atend) %%BoundingBox: 50 50 410 302 %%Orientation: Portrait %%EndComments /gnudict 256 dict def gnudict begin /Color true def /Solid false def /gnulinewidth 5.000 def /userlinewidth gnulinewidth def /vshift -46 def /dl {10 mul} def /hpt_ 31.5 def /vpt_ 31.5 def /hpt hpt_ def /vpt vpt_ def /M {moveto} bind def /L {lineto} bind def /R {rmoveto} bind def /V {rlineto} bind def /vpt2 vpt 2 mul def /hpt2 hpt 2 mul def /Lshow { currentpoint stroke M 0 vshift R show } def /Rshow { currentpoint stroke M dup stringwidth pop neg vshift R show } def /Cshow { currentpoint stroke M dup stringwidth pop -2 div vshift R show } def /UP { dup vpt_ mul /vpt exch def hpt_ mul /hpt exch def /hpt2 hpt 2 mul def /vpt2 vpt 2 mul def } def /DL { Color {setrgbcolor Solid {pop []} if 0 setdash } {pop pop pop Solid {pop []} if 0 setdash} ifelse } def /BL { stroke userlinewidth 2 mul setlinewidth } def /AL { stroke userlinewidth 2 div setlinewidth } def /UL { dup gnulinewidth mul /userlinewidth exch def dup 1 lt {pop 1} if 10 mul /udl exch def } def /PL { stroke userlinewidth setlinewidth } def /LTb { BL [] 0 0 0 DL } def /LTa { AL [1 udl mul 2 udl mul] 0 setdash 0 0 0 setrgbcolor } def /LT0 { PL [] 1 0 0 DL } def /LT1 { PL [4 dl 2 dl] 0 1 0 DL } def /LT2 { PL [2 dl 3 dl] 0 0 1 DL } def /LT3 { PL [1 dl 1.5 dl] 1 0 1 DL } def /LT4 { PL [5 dl 2 dl 1 dl 2 dl] 0 1 1 DL } def /LT5 { PL [4 dl 3 dl 1 dl 3 dl] 1 1 0 DL } def /LT6 { PL [2 dl 2 dl 2 dl 4 dl] 0 0 0 DL } def /LT7 { PL [2 dl 2 dl 2 dl 2 dl 2 dl 4 dl] 1 0.3 0 DL } def /LT8 { PL [2 dl 2 dl 2 dl 2 dl 2 dl 2 dl 2 dl 4 dl] 0.5 0.5 0.5 DL } def /Pnt { stroke [] 0 setdash gsave 1 setlinecap M 0 0 V stroke grestore } def /Dia { stroke [] 0 setdash 2 copy vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath stroke Pnt } def /Pls { stroke [] 0 setdash vpt sub M 0 vpt2 V currentpoint stroke M hpt neg vpt neg R hpt2 0 V stroke } def /Box { stroke [] 0 setdash 2 copy exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath stroke Pnt } def /Crs { stroke [] 0 setdash exch hpt sub exch vpt add M hpt2 vpt2 neg V currentpoint stroke M hpt2 neg 0 R hpt2 vpt2 V stroke } def /TriU { stroke [] 0 setdash 2 copy vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath stroke Pnt } def /Star { 2 copy Pls Crs } def /BoxF { stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath fill } def /TriUF { stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath fill } def /TriD { stroke [] 0 setdash 2 copy vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath stroke Pnt } def /TriDF { stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath fill} def /DiaF { stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath fill } def /Pent { stroke [] 0 setdash 2 copy gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath stroke grestore Pnt } def /PentF { stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath fill grestore } def /Circle { stroke [] 0 setdash 2 copy hpt 0 360 arc stroke Pnt } def /CircleF { stroke [] 0 setdash hpt 0 360 arc fill } def /C0 { BL [] 0 setdash 2 copy moveto vpt 90 450 arc } bind def /C1 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc closepath fill vpt 0 360 arc closepath } bind def /C2 { BL [] 0 setdash 2 copy moveto 2 copy vpt 90 180 arc closepath fill vpt 0 360 arc closepath } bind def /C3 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 180 arc closepath fill vpt 0 360 arc closepath } bind def /C4 { BL [] 0 setdash 2 copy moveto 2 copy vpt 180 270 arc closepath fill vpt 0 360 arc closepath } bind def /C5 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc 2 copy moveto 2 copy vpt 180 270 arc closepath fill vpt 0 360 arc } bind def /C6 { BL [] 0 setdash 2 copy moveto 2 copy vpt 90 270 arc closepath fill vpt 0 360 arc closepath } bind def /C7 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 270 arc closepath fill vpt 0 360 arc closepath } bind def /C8 { BL [] 0 setdash 2 copy moveto 2 copy vpt 270 360 arc closepath fill vpt 0 360 arc closepath } bind def /C9 { BL [] 0 setdash 2 copy moveto 2 copy vpt 270 450 arc closepath fill vpt 0 360 arc closepath } bind def /C10 { BL [] 0 setdash 2 copy 2 copy moveto vpt 270 360 arc closepath fill 2 copy moveto 2 copy vpt 90 180 arc closepath fill vpt 0 360 arc closepath } bind def /C11 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 180 arc closepath fill 2 copy moveto 2 copy vpt 270 360 arc closepath fill vpt 0 360 arc closepath } bind def /C12 { BL [] 0 setdash 2 copy moveto 2 copy vpt 180 360 arc closepath fill vpt 0 360 arc closepath } bind def /C13 { BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc closepath fill 2 copy moveto 2 copy vpt 180 360 arc closepath fill vpt 0 360 arc closepath } bind def /C14 { BL [] 0 setdash 2 copy moveto 2 copy vpt 90 360 arc closepath fill vpt 0 360 arc } bind def /C15 { BL [] 0 setdash 2 copy vpt 0 360 arc closepath fill vpt 0 360 arc closepath } bind def /Rec { newpath 4 2 roll moveto 1 index 0 rlineto 0 exch rlineto neg 0 rlineto closepath } bind def /Square { dup Rec } bind def /Bsquare { vpt sub exch vpt sub exch vpt2 Square } bind def /S0 { BL [] 0 setdash 2 copy moveto 0 vpt rlineto BL Bsquare } bind def /S1 { BL [] 0 setdash 2 copy vpt Square fill Bsquare } bind def /S2 { BL [] 0 setdash 2 copy exch vpt sub exch vpt Square fill Bsquare } bind def /S3 { BL [] 0 setdash 2 copy exch vpt sub exch vpt2 vpt Rec fill Bsquare } bind def /S4 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt Square fill Bsquare } bind def /S5 { BL [] 0 setdash 2 copy 2 copy vpt Square fill exch vpt sub exch vpt sub vpt Square fill Bsquare } bind def /S6 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt vpt2 Rec fill Bsquare } bind def /S7 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt vpt2 Rec fill 2 copy vpt Square fill Bsquare } bind def /S8 { BL [] 0 setdash 2 copy vpt sub vpt Square fill Bsquare } bind def /S9 { BL [] 0 setdash 2 copy vpt sub vpt vpt2 Rec fill Bsquare } bind def /S10 { BL [] 0 setdash 2 copy vpt sub vpt Square fill 2 copy exch vpt sub exch vpt Square fill Bsquare } bind def /S11 { BL [] 0 setdash 2 copy vpt sub vpt Square fill 2 copy exch vpt sub exch vpt2 vpt Rec fill Bsquare } bind def /S12 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill Bsquare } bind def /S13 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill 2 copy vpt Square fill Bsquare } bind def /S14 { BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill 2 copy exch vpt sub exch vpt Square fill Bsquare } bind def /S15 { BL [] 0 setdash 2 copy Bsquare fill Bsquare } bind def /D0 { gsave translate 45 rotate 0 0 S0 stroke grestore } bind def /D1 { gsave translate 45 rotate 0 0 S1 stroke grestore } bind def /D2 { gsave translate 45 rotate 0 0 S2 stroke grestore } bind def /D3 { gsave translate 45 rotate 0 0 S3 stroke grestore } bind def /D4 { gsave translate 45 rotate 0 0 S4 stroke grestore } bind def /D5 { gsave translate 45 rotate 0 0 S5 stroke grestore } bind def /D6 { gsave translate 45 rotate 0 0 S6 stroke grestore } bind def /D7 { gsave translate 45 rotate 0 0 S7 stroke grestore } bind def /D8 { gsave translate 45 rotate 0 0 S8 stroke grestore } bind def /D9 { gsave translate 45 rotate 0 0 S9 stroke grestore } bind def /D10 { gsave translate 45 rotate 0 0 S10 stroke grestore } bind def /D11 { gsave translate 45 rotate 0 0 S11 stroke grestore } bind def /D12 { gsave translate 45 rotate 0 0 S12 stroke grestore } bind def /D13 { gsave translate 45 rotate 0 0 S13 stroke grestore } bind def /D14 { gsave translate 45 rotate 0 0 S14 stroke grestore } bind def /D15 { gsave translate 45 rotate 0 0 S15 stroke grestore } bind def /DiaE { stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath stroke } def /BoxE { stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath stroke } def /TriUE { stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath stroke } def /TriDE { stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath stroke } def /PentE { stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath stroke grestore } def /CircE { stroke [] 0 setdash hpt 0 360 arc stroke } def /Opaque { gsave closepath 1 setgray fill grestore 0 setgray closepath } def /DiaW { stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V Opaque stroke } def /BoxW { stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V Opaque stroke } def /TriUW { stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V Opaque stroke } def /TriDW { stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V Opaque stroke } def /PentW { stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat Opaque stroke grestore } def /CircW { stroke [] 0 setdash hpt 0 360 arc Opaque stroke } def /BoxFill { gsave Rec 1 setgray fill grestore } def /Symbol-Oblique /Symbol findfont [1 0 .167 1 0 0] makefont dup length dict begin {1 index /FID eq {pop pop} {def} ifelse} forall currentdict end definefont pop end %%EndProlog gnudict begin gsave 50 50 translate 0.050 0.050 scale 0 setgray newpath (Helvetica) findfont 140 scalefont setfont 1.000 UL LTb 1.000 UL LTa 714 420 M 6248 0 V 1.000 UL LTb 714 420 M 63 0 V 6185 0 R -63 0 V 630 420 M ( 0) Rshow 1.000 UL LTa 714 1056 M 6248 0 V 1.000 UL LTb 714 1056 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 0.2) Rshow 1.000 UL LTa 714 1692 M 6248 0 V 1.000 UL LTb 714 1692 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 0.4) Rshow 1.000 UL LTa 714 2328 M 6248 0 V 1.000 UL LTb 714 2328 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 0.6) Rshow 1.000 UL LTa 714 2964 M 6248 0 V 1.000 UL LTb 714 2964 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 0.8) Rshow 1.000 UL LTa 714 3600 M 6248 0 V 1.000 UL LTb 714 3600 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 1) Rshow 1.000 UL LTa 714 4236 M 6248 0 V 1.000 UL LTb 714 4236 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 1.2) Rshow 1.000 UL LTa 714 4872 M 6248 0 V 1.000 UL LTb 714 4872 M 63 0 V 6185 0 R -63 0 V -6269 0 R ( 1.4) Rshow 714 420 M 0 63 V 0 4389 R 0 -63 V 714 280 M ( 0) Cshow 1962 420 M 0 63 V 0 4389 R 0 -63 V 0 -4529 R ( 200) Cshow 3211 420 M 0 63 V 0 4389 R 0 -63 V 0 -4529 R ( 400) Cshow 4459 420 M 0 63 V 0 4389 R 0 -63 V 0 -4529 R ( 600) Cshow 5707 420 M 0 63 V 0 4389 R 0 -63 V 0 -4529 R ( 800) Cshow 6956 420 M 0 63 V 0 4389 R 0 -63 V 0 -4529 R ( 1000) Cshow 1.000 UL LTb 714 420 M 6248 0 V 0 4452 V -6248 0 V 714 420 L 140 2646 M currentpoint gsave translate 90 rotate 0 0 M (Message delay \(seconds\)) Cshow grestore 3838 70 M (Message Id) Cshow 0.500 UP 1.000 UL LT0 6343 4739 M (Average) Rshow 720 538 Pls 726 605 Pls 733 605 Pls 739 640 Pls 745 710 Pls 751 710 Pls 758 776 Pls 764 777 Pls 770 844 Pls 776 844 Pls 783 911 Pls 789 911 Pls 795 981 Pls 801 981 Pls 808 1048 Pls 814 1048 Pls 820 1115 Pls 826 1115 Pls 833 1185 Pls 839 1185 Pls 845 1252 Pls 851 1252 Pls 858 1319 Pls 864 1319 Pls 870 1386 Pls 876 1386 Pls 883 1456 Pls 889 1456 Pls 895 1523 Pls 901 1523 Pls 907 1590 Pls 914 1590 Pls 920 1660 Pls 926 1660 Pls 932 1727 Pls 939 1727 Pls 945 1794 Pls 951 1794 Pls 957 1861 Pls 964 1861 Pls 970 1931 Pls 976 1931 Pls 982 1998 Pls 989 1998 Pls 995 2065 Pls 1001 2065 Pls 1007 2135 Pls 1014 2135 Pls 1020 2202 Pls 1026 2202 Pls 1032 2269 Pls 1039 2269 Pls 1045 2370 Pls 1051 2372 Pls 1057 2374 Pls 1064 2376 Pls 1070 2380 Pls 1076 2439 Pls 1082 2439 Pls 1089 2441 Pls 1095 2441 Pls 1101 2443 Pls 1107 2443 Pls 1113 2445 Pls 1120 2445 Pls 1126 2449 Pls 1132 2449 Pls 1138 2508 Pls 1145 2508 Pls 1151 2509 Pls 1157 2509 Pls 1163 2511 Pls 1170 2511 Pls 1176 2513 Pls 1182 2513 Pls 1188 2516 Pls 1195 2516 Pls 1201 2575 Pls 1207 2575 Pls 1213 2576 Pls 1220 2576 Pls 1226 2578 Pls 1232 2578 Pls 1238 2580 Pls 1245 2580 Pls 1251 2583 Pls 1257 2583 Pls 1263 2643 Pls 1270 2643 Pls 1276 2645 Pls 1282 2645 Pls 1288 2647 Pls 1294 2647 Pls 1301 2649 Pls 1307 2649 Pls 1313 2653 Pls 1319 2653 Pls 1326 2710 Pls 1332 2710 Pls 1338 2713 Pls 1344 2713 Pls 1351 2715 Pls 1357 2715 Pls 1363 2717 Pls 1369 2717 Pls 1376 2720 Pls 1382 2720 Pls 1388 2779 Pls 1394 2779 Pls 1401 2780 Pls 1407 2780 Pls 1413 2782 Pls 1419 2782 Pls 1426 2782 Pls 1432 2782 Pls 1438 2783 Pls 1444 2783 Pls 1451 2783 Pls 1457 2785 Pls 1463 2785 Pls 1469 2786 Pls 1475 2786 Pls 1482 2787 Pls 1488 2787 Pls 1494 2845 Pls 1500 2845 Pls 1507 2847 Pls 1513 2847 Pls 1519 2849 Pls 1525 2850 Pls 1532 2851 Pls 1538 2852 Pls 1544 2852 Pls 1550 2853 Pls 1557 2855 Pls 1563 2855 Pls 1569 2913 Pls 1575 2913 Pls 1582 2916 Pls 1588 2916 Pls 1594 2917 Pls 1600 2917 Pls 1607 2918 Pls 1613 2919 Pls 1619 2920 Pls 1625 2921 Pls 1632 2922 Pls 1638 2923 Pls 1644 2923 Pls 1650 2925 Pls 1657 2925 Pls 1663 2981 Pls 1669 2981 Pls 1675 2982 Pls 1681 2982 Pls 1688 2983 Pls 1694 2983 Pls 1700 2986 Pls 1706 2986 Pls 1713 2987 Pls 1719 2987 Pls 1725 2988 Pls 1731 2988 Pls 1738 2989 Pls 1744 2989 Pls 1750 2990 Pls 1756 2990 Pls 1763 2992 Pls 1769 2992 Pls 1775 3049 Pls 1781 3049 Pls 1788 3050 Pls 1794 3050 Pls 1800 3052 Pls 1806 3052 Pls 1813 3055 Pls 1819 3055 Pls 1825 3055 Pls 1831 3055 Pls 1838 3056 Pls 1844 3056 Pls 1850 3056 Pls 1856 3056 Pls 1862 3057 Pls 1869 3057 Pls 1875 3059 Pls 1881 3059 Pls 1887 3116 Pls 1894 3116 Pls 1900 3117 Pls 1906 3117 Pls 1912 3119 Pls 1919 3119 Pls 1925 3121 Pls 1931 3121 Pls 1937 3122 Pls 1944 3122 Pls 1950 3123 Pls 1956 3124 Pls 1962 3125 Pls 1969 3125 Pls 1975 3125 Pls 1981 3125 Pls 1987 3125 Pls 1994 3125 Pls 2000 3185 Pls 2006 3185 Pls 2012 3186 Pls 2019 3186 Pls 2025 3188 Pls 2031 3188 Pls 2037 3190 Pls 2043 3190 Pls 2050 3191 Pls 2056 3191 Pls 2062 3192 Pls 2068 3192 Pls 2075 3192 Pls 2081 3192 Pls 2087 3194 Pls 2093 3194 Pls 2100 3195 Pls 2106 3195 Pls 2112 3252 Pls 2118 3252 Pls 2125 3253 Pls 2131 3253 Pls 2137 3255 Pls 2143 3255 Pls 2150 3257 Pls 2156 3257 Pls 2162 3259 Pls 2168 3259 Pls 2175 3259 Pls 2181 3259 Pls 2187 3259 Pls 2193 3259 Pls 2200 3261 Pls 2206 3261 Pls 2212 3262 Pls 2218 3262 Pls 2225 3320 Pls 2231 3320 Pls 2237 3323 Pls 2243 3323 Pls 2249 3324 Pls 2256 3324 Pls 2262 3325 Pls 2268 3325 Pls 2274 3325 Pls 2281 3325 Pls 2287 3325 Pls 2293 3325 Pls 2299 3325 Pls 2306 3325 Pls 2312 3325 Pls 2318 3325 Pls 2324 3325 Pls 2331 3325 Pls 2337 3325 Pls 2343 3325 Pls 2349 3325 Pls 2356 3325 Pls 2362 3325 Pls 2368 3325 Pls 2374 3326 Pls 2381 3326 Pls 2387 3326 Pls 2393 3326 Pls 2399 3326 Pls 2406 3326 Pls 2412 3326 Pls 2418 3326 Pls 2424 3326 Pls 2430 3326 Pls 2437 3327 Pls 2443 3327 Pls 2449 3328 Pls 2455 3328 Pls 2462 3329 Pls 2468 3329 Pls 2474 3389 Pls 2480 3390 Pls 2487 3391 Pls 2493 3391 Pls 2499 3392 Pls 2505 3393 Pls 2512 3393 Pls 2518 3393 Pls 2524 3394 Pls 2530 3395 Pls 2537 3395 Pls 2543 3395 Pls 2549 3395 Pls 2555 3395 Pls 2562 3396 Pls 2568 3396 Pls 2574 3398 Pls 2580 3398 Pls 2587 3456 Pls 2593 3456 Pls 2599 3456 Pls 2605 3456 Pls 2611 3460 Pls 2618 3460 Pls 2624 3462 Pls 2630 3462 Pls 2636 3462 Pls 2643 3462 Pls 2649 3462 Pls 2655 3462 Pls 2661 3464 Pls 2668 3465 Pls 2674 3465 Pls 2680 3465 Pls 2686 3492 Pls 2693 3492 Pls 2699 3492 Pls 2705 3492 Pls 2711 3516 Pls 2718 3518 Pls 2724 3519 Pls 2730 3519 Pls 2736 3519 Pls 2743 3519 Pls 2749 3520 Pls 2755 3521 Pls 2761 3521 Pls 2768 3522 Pls 2774 3522 Pls 2780 3523 Pls 2786 3524 Pls 2793 3525 Pls 2799 3525 Pls 2805 3525 Pls 2811 3525 Pls 2817 3525 Pls 2824 3525 Pls 2830 3525 Pls 2836 3525 Pls 2842 3525 Pls 2849 3525 Pls 2855 3525 Pls 2861 3525 Pls 2867 3525 Pls 2874 3525 Pls 2880 3525 Pls 2886 3525 Pls 2892 3525 Pls 2899 3525 Pls 2905 3525 Pls 2911 3525 Pls 2917 3525 Pls 2924 3525 Pls 2930 3525 Pls 2936 3525 Pls 2942 3525 Pls 2949 3525 Pls 2955 3525 Pls 2961 3525 Pls 2967 3525 Pls 2974 3525 Pls 2980 3525 Pls 2986 3525 Pls 2992 3525 Pls 2998 3525 Pls 3005 3525 Pls 3011 3525 Pls 3017 3525 Pls 3023 3525 Pls 3030 3525 Pls 3036 3525 Pls 3042 3525 Pls 3048 3525 Pls 3055 3525 Pls 3061 3525 Pls 3067 3525 Pls 3073 3525 Pls 3080 3525 Pls 3086 3525 Pls 3092 3525 Pls 3098 3525 Pls 3105 3525 Pls 3111 3525 Pls 3117 3525 Pls 3123 3526 Pls 3130 3526 Pls 3136 3526 Pls 3142 3526 Pls 3148 3526 Pls 3155 3526 Pls 3161 3526 Pls 3167 3526 Pls 3173 3526 Pls 3179 3526 Pls 3186 3526 Pls 3192 3526 Pls 3198 3526 Pls 3204 3526 Pls 3211 3526 Pls 3217 3526 Pls 3223 3526 Pls 3229 3526 Pls 3236 3526 Pls 3242 3526 Pls 3248 3526 Pls 3254 3526 Pls 3261 3526 Pls 3267 3526 Pls 3273 3526 Pls 3279 3526 Pls 3286 3526 Pls 3292 3526 Pls 3298 3526 Pls 3304 3526 Pls 3311 3526 Pls 3317 3526 Pls 3323 3526 Pls 3329 3526 Pls 3336 3526 Pls 3342 3526 Pls 3348 3526 Pls 3354 3526 Pls 3361 3526 Pls 3367 3526 Pls 3373 3526 Pls 3379 3526 Pls 3385 3526 Pls 3392 3526 Pls 3398 3526 Pls 3404 3526 Pls 3410 3526 Pls 3417 3526 Pls 3423 3526 Pls 3429 3526 Pls 3435 3526 Pls 3442 3526 Pls 3448 3526 Pls 3454 3526 Pls 3460 3526 Pls 3467 3526 Pls 3473 3526 Pls 3479 3526 Pls 3485 3526 Pls 3492 3526 Pls 3498 3526 Pls 3504 3526 Pls 3510 3526 Pls 3517 3526 Pls 3523 3526 Pls 3529 3526 Pls 3535 3526 Pls 3542 3526 Pls 3548 3526 Pls 3554 3526 Pls 3560 3526 Pls 3566 3526 Pls 3573 3526 Pls 3579 3526 Pls 3585 3526 Pls 3591 3526 Pls 3598 3526 Pls 3604 3526 Pls 3610 3526 Pls 3616 3526 Pls 3623 3526 Pls 3629 3526 Pls 3635 3526 Pls 3641 3526 Pls 3648 3526 Pls 3654 3526 Pls 3660 3526 Pls 3666 3526 Pls 3673 3526 Pls 3679 3526 Pls 3685 3526 Pls 3691 3526 Pls 3698 3526 Pls 3704 3526 Pls 3710 3526 Pls 3716 3526 Pls 3723 3526 Pls 3729 3526 Pls 3735 3526 Pls 3741 3526 Pls 3747 3526 Pls 3754 3527 Pls 3760 3527 Pls 3766 3527 Pls 3772 3528 Pls 3779 3528 Pls 3785 3528 Pls 3791 3528 Pls 3797 3528 Pls 3804 3528 Pls 3810 3528 Pls 3816 3528 Pls 3822 3528 Pls 3829 3528 Pls 3835 3528 Pls 3841 3528 Pls 3847 3528 Pls 3854 3529 Pls 3860 3529 Pls 3866 3529 Pls 3872 3529 Pls 3879 3529 Pls 3885 3529 Pls 3891 3529 Pls 3897 3529 Pls 3904 3529 Pls 3910 3529 Pls 3916 3529 Pls 3922 3529 Pls 3929 3529 Pls 3935 3529 Pls 3941 3529 Pls 3947 3529 Pls 3953 3529 Pls 3960 3529 Pls 3966 3529 Pls 3972 3529 Pls 3978 3529 Pls 3985 3529 Pls 3991 3529 Pls 3997 3529 Pls 4003 3529 Pls 4010 3529 Pls 4016 3529 Pls 4022 3529 Pls 4028 3529 Pls 4035 3529 Pls 4041 3530 Pls 4047 3530 Pls 4053 3530 Pls 4060 3532 Pls 4066 3532 Pls 4072 3532 Pls 4078 3532 Pls 4085 3533 Pls 4091 3533 Pls 4097 3551 Pls 4103 3559 Pls 4110 3559 Pls 4116 3559 Pls 4122 3559 Pls 4128 3587 Pls 4134 3588 Pls 4141 3590 Pls 4147 3591 Pls 4153 3592 Pls 4159 3592 Pls 4166 3592 Pls 4172 3592 Pls 4178 3592 Pls 4184 3592 Pls 4191 3592 Pls 4197 3592 Pls 4203 3592 Pls 4209 3592 Pls 4216 3592 Pls 4222 3592 Pls 4228 3592 Pls 4234 3592 Pls 4241 3592 Pls 4247 3592 Pls 4253 3592 Pls 4259 3592 Pls 4266 3592 Pls 4272 3592 Pls 4278 3592 Pls 4284 3592 Pls 4291 3592 Pls 4297 3592 Pls 4303 3592 Pls 4309 3592 Pls 4315 3592 Pls 4322 3592 Pls 4328 3592 Pls 4334 3592 Pls 4340 3592 Pls 4347 3592 Pls 4353 3592 Pls 4359 3592 Pls 4365 3592 Pls 4372 3592 Pls 4378 3592 Pls 4384 3592 Pls 4390 3592 Pls 4397 3592 Pls 4403 3592 Pls 4409 3592 Pls 4415 3592 Pls 4422 3592 Pls 4428 3592 Pls 4434 3592 Pls 4440 3592 Pls 4447 3592 Pls 4453 3592 Pls 4459 3592 Pls 4465 3592 Pls 4472 3592 Pls 4478 3592 Pls 4484 3592 Pls 4490 3592 Pls 4497 3592 Pls 4503 3592 Pls 4509 3592 Pls 4515 3592 Pls 4521 3592 Pls 4528 3592 Pls 4534 3592 Pls 4540 3592 Pls 4546 3592 Pls 4553 3592 Pls 4559 3592 Pls 4565 3592 Pls 4571 3592 Pls 4578 3592 Pls 4584 3592 Pls 4590 3592 Pls 4596 3592 Pls 4603 3592 Pls 4609 3592 Pls 4615 3592 Pls 4621 3592 Pls 4628 3592 Pls 4634 3592 Pls 4640 3592 Pls 4646 3592 Pls 4653 3592 Pls 4659 3592 Pls 4665 3592 Pls 4671 3592 Pls 4678 3592 Pls 4684 3592 Pls 4690 3592 Pls 4696 3592 Pls 4702 3592 Pls 4709 3593 Pls 4715 3593 Pls 4721 3593 Pls 4727 3593 Pls 4734 3593 Pls 4740 3593 Pls 4746 3593 Pls 4752 3593 Pls 4759 3593 Pls 4765 3593 Pls 4771 3593 Pls 4777 3593 Pls 4784 3593 Pls 4790 3593 Pls 4796 3593 Pls 4802 3593 Pls 4809 3593 Pls 4815 3593 Pls 4821 3593 Pls 4827 3594 Pls 4834 3594 Pls 4840 3594 Pls 4846 3595 Pls 4852 3595 Pls 4859 3595 Pls 4865 3595 Pls 4871 3595 Pls 4877 3595 Pls 4883 3595 Pls 4890 3595 Pls 4896 3595 Pls 4902 3595 Pls 4908 3595 Pls 4915 3595 Pls 4921 3595 Pls 4927 3595 Pls 4933 3595 Pls 4940 3595 Pls 4946 3595 Pls 4952 3595 Pls 4958 3595 Pls 4965 3595 Pls 4971 3595 Pls 4977 3595 Pls 4983 3595 Pls 4990 3595 Pls 4996 3595 Pls 5002 3595 Pls 5008 3595 Pls 5015 3595 Pls 5021 3595 Pls 5027 3595 Pls 5033 3595 Pls 5040 3595 Pls 5046 3595 Pls 5052 3595 Pls 5058 3595 Pls 5065 3595 Pls 5071 3596 Pls 5077 3596 Pls 5083 3596 Pls 5089 3596 Pls 5096 3596 Pls 5102 3596 Pls 5108 3596 Pls 5114 3596 Pls 5121 3596 Pls 5127 3596 Pls 5133 3596 Pls 5139 3596 Pls 5146 3596 Pls 5152 3596 Pls 5158 3596 Pls 5164 3596 Pls 5171 3596 Pls 5177 3596 Pls 5183 3596 Pls 5189 3596 Pls 5196 3596 Pls 5202 3596 Pls 5208 3596 Pls 5214 3596 Pls 5221 3596 Pls 5227 3596 Pls 5233 3596 Pls 5239 3596 Pls 5246 3596 Pls 5252 3596 Pls 5258 3596 Pls 5264 3596 Pls 5270 3596 Pls 5277 3596 Pls 5283 3596 Pls 5289 3596 Pls 5295 3596 Pls 5302 3596 Pls 5308 3596 Pls 5314 3596 Pls 5320 3596 Pls 5327 3596 Pls 5333 3596 Pls 5339 3596 Pls 5345 3596 Pls 5352 3596 Pls 5358 3596 Pls 5364 3596 Pls 5370 3596 Pls 5377 3596 Pls 5383 3596 Pls 5389 3596 Pls 5395 3596 Pls 5402 3596 Pls 5408 3596 Pls 5414 3596 Pls 5420 3597 Pls 5427 3597 Pls 5433 3597 Pls 5439 3597 Pls 5445 3598 Pls 5451 3598 Pls 5458 3598 Pls 5464 3599 Pls 5470 3599 Pls 5476 3599 Pls 5483 3599 Pls 5489 3600 Pls 5495 3622 Pls 5501 3622 Pls 5508 3629 Pls 5514 3629 Pls 5520 3629 Pls 5526 3629 Pls 5533 3663 Pls 5539 3663 Pls 5545 3663 Pls 5551 3663 Pls 5558 3663 Pls 5564 3663 Pls 5570 3664 Pls 5576 3664 Pls 5583 3688 Pls 5589 3688 Pls 5595 3696 Pls 5601 3697 Pls 5608 3697 Pls 5614 3697 Pls 5620 3729 Pls 5626 3729 Pls 5633 3729 Pls 5639 3730 Pls 5645 3730 Pls 5651 3730 Pls 5657 3730 Pls 5664 3730 Pls 5670 3732 Pls 5676 3734 Pls 5682 3756 Pls 5689 3756 Pls 5695 3764 Pls 5701 3764 Pls 5707 3764 Pls 5714 3764 Pls 5720 3796 Pls 5726 3796 Pls 5732 3796 Pls 5739 3796 Pls 5745 3796 Pls 5751 3797 Pls 5757 3797 Pls 5764 3798 Pls 5770 3798 Pls 5776 3800 Pls 5782 3800 Pls 5789 3800 Pls 5795 3800 Pls 5801 3801 Pls 5807 3801 Pls 5814 3803 Pls 5820 3803 Pls 5826 3822 Pls 5832 3822 Pls 5838 3834 Pls 5845 3834 Pls 5851 3834 Pls 5857 3834 Pls 5863 3863 Pls 5870 3863 Pls 5876 3864 Pls 5882 3866 Pls 5888 3866 Pls 5895 3866 Pls 5901 3866 Pls 5907 3866 Pls 5913 3867 Pls 5920 3867 Pls 5926 3867 Pls 5932 3868 Pls 5938 3870 Pls 5945 3870 Pls 5951 3871 Pls 5957 3871 Pls 5963 3873 Pls 5970 3873 Pls 5976 3893 Pls 5982 3893 Pls 5988 3900 Pls 5995 3900 Pls 6001 3900 Pls 6007 3900 Pls 6013 3930 Pls 6019 3933 Pls 6026 3933 Pls 6032 3933 Pls 6038 3933 Pls 6044 3933 Pls 6051 3934 Pls 6057 3934 Pls 6063 3935 Pls 6069 3935 Pls 6076 3936 Pls 6082 3937 Pls 6088 3938 Pls 6094 3938 Pls 6101 3940 Pls 6107 3940 Pls 6113 3960 Pls 6119 3960 Pls 6126 3966 Pls 6132 3966 Pls 6138 3967 Pls 6144 3967 Pls 6151 3970 Pls 6157 3970 Pls 6163 3973 Pls 6169 4000 Pls 6176 4000 Pls 6182 4000 Pls 6188 4000 Pls 6194 4002 Pls 6201 4002 Pls 6207 4003 Pls 6213 4003 Pls 6219 4005 Pls 6225 4005 Pls 6232 4006 Pls 6238 4006 Pls 6244 4008 Pls 6250 4008 Pls 6257 4027 Pls 6263 4034 Pls 6269 4034 Pls 6275 4034 Pls 6282 4034 Pls 6288 4066 Pls 6294 4066 Pls 6300 4067 Pls 6307 4067 Pls 6313 4067 Pls 6319 4067 Pls 6325 4069 Pls 6332 4069 Pls 6338 4071 Pls 6344 4071 Pls 6350 4073 Pls 6357 4073 Pls 6363 4074 Pls 6369 4074 Pls 6375 4105 Pls 6382 4105 Pls 6388 4105 Pls 6394 4105 Pls 6400 4136 Pls 6406 4136 Pls 6413 4136 Pls 6419 4136 Pls 6425 4138 Pls 6431 4138 Pls 6438 4140 Pls 6444 4140 Pls 6450 4141 Pls 6456 4142 Pls 6463 4143 Pls 6469 4145 Pls 6475 4145 Pls 6481 4172 Pls 6488 4172 Pls 6494 4172 Pls 6500 4203 Pls 6506 4203 Pls 6513 4203 Pls 6519 4203 Pls 6525 4206 Pls 6531 4206 Pls 6538 4206 Pls 6544 4206 Pls 6550 4210 Pls 6556 4210 Pls 6563 4270 Pls 6569 4270 Pls 6575 4271 Pls 6581 4271 Pls 6587 4273 Pls 6594 4274 Pls 6600 4275 Pls 6606 4278 Pls 6612 4278 Pls 6619 4338 Pls 6625 4338 Pls 6631 4341 Pls 6637 4341 Pls 6644 4343 Pls 6650 4343 Pls 6656 4346 Pls 6662 4346 Pls 6669 4406 Pls 6675 4406 Pls 6681 4409 Pls 6687 4409 Pls 6694 4411 Pls 6700 4411 Pls 6706 4413 Pls 6712 4413 Pls 6719 4474 Pls 6725 4474 Pls 6731 4476 Pls 6737 4476 Pls 6744 4478 Pls 6750 4478 Pls 6756 4480 Pls 6762 4480 Pls 6769 4541 Pls 6775 4541 Pls 6781 4544 Pls 6787 4544 Pls 6793 4547 Pls 6800 4547 Pls 6806 4549 Pls 6812 4549 Pls 6818 4611 Pls 6825 4611 Pls 6831 4614 Pls 6837 4614 Pls 6843 4615 Pls 6850 4615 Pls 6856 4617 Pls 6862 4617 Pls 6868 4676 Pls 6875 4676 Pls 6881 4679 Pls 6887 4679 Pls 6893 4682 Pls 6900 4682 Pls 6906 4683 Pls 6912 4683 Pls 6918 4684 Pls 6925 4684 Pls 6931 4743 Pls 6937 4747 Pls 6943 4749 Pls 6950 4750 Pls 6956 4753 Pls 6610 4739 Pls 0.500 UP 1.000 UL LT1 6343 4599 M (Min) Rshow 720 537 Crs 726 604 Crs 733 604 Crs 739 639 Crs 745 708 Crs 751 708 Crs 758 775 Crs 764 775 Crs 770 842 Crs 776 842 Crs 783 909 Crs 789 909 Crs 795 978 Crs 801 978 Crs 808 1045 Crs 814 1045 Crs 820 1112 Crs 826 1112 Crs 833 1182 Crs 839 1182 Crs 845 1249 Crs 851 1249 Crs 858 1315 Crs 864 1315 Crs 870 1382 Crs 876 1382 Crs 883 1452 Crs 889 1452 Crs 895 1519 Crs 901 1519 Crs 907 1585 Crs 914 1585 Crs 920 1655 Crs 926 1655 Crs 932 1722 Crs 939 1722 Crs 945 1789 Crs 951 1789 Crs 957 1855 Crs 964 1855 Crs 970 1925 Crs 976 1925 Crs 982 1992 Crs 989 1992 Crs 995 2059 Crs 1001 2059 Crs 1007 2129 Crs 1014 2129 Crs 1020 2195 Crs 1026 2195 Crs 1032 2262 Crs 1039 2262 Crs 1045 2366 Crs 1051 2372 Crs 1057 2372 Crs 1064 2372 Crs 1070 2378 Crs 1076 2436 Crs 1082 2436 Crs 1089 2439 Crs 1095 2439 Crs 1101 2439 Crs 1107 2439 Crs 1113 2442 Crs 1120 2442 Crs 1126 2445 Crs 1132 2445 Crs 1138 2502 Crs 1145 2502 Crs 1151 2509 Crs 1157 2509 Crs 1163 2509 Crs 1170 2509 Crs 1176 2509 Crs 1182 2509 Crs 1188 2515 Crs 1195 2515 Crs 1201 2575 Crs 1207 2575 Crs 1213 2575 Crs 1220 2575 Crs 1226 2575 Crs 1232 2575 Crs 1238 2576 Crs 1245 2576 Crs 1251 2582 Crs 1257 2582 Crs 1263 2642 Crs 1270 2642 Crs 1276 2642 Crs 1282 2642 Crs 1288 2645 Crs 1294 2645 Crs 1301 2645 Crs 1307 2645 Crs 1313 2651 Crs 1319 2651 Crs 1326 2709 Crs 1332 2709 Crs 1338 2712 Crs 1344 2712 Crs 1351 2712 Crs 1357 2712 Crs 1363 2712 Crs 1369 2712 Crs 1376 2718 Crs 1382 2718 Crs 1388 2779 Crs 1394 2779 Crs 1401 2779 Crs 1407 2779 Crs 1413 2779 Crs 1419 2779 Crs 1426 2780 Crs 1432 2780 Crs 1438 2780 Crs 1444 2780 Crs 1451 2780 Crs 1457 2780 Crs 1463 2780 Crs 1469 2785 Crs 1475 2785 Crs 1482 2785 Crs 1488 2785 Crs 1494 2845 Crs 1500 2845 Crs 1507 2845 Crs 1513 2845 Crs 1519 2846 Crs 1525 2846 Crs 1532 2849 Crs 1538 2849 Crs 1544 2852 Crs 1550 2852 Crs 1557 2852 Crs 1563 2852 Crs 1569 2912 Crs 1575 2912 Crs 1582 2912 Crs 1588 2912 Crs 1594 2916 Crs 1600 2916 Crs 1607 2916 Crs 1613 2916 Crs 1619 2917 Crs 1625 2919 Crs 1632 2922 Crs 1638 2922 Crs 1644 2922 Crs 1650 2922 Crs 1657 2923 Crs 1663 2980 Crs 1669 2980 Crs 1675 2980 Crs 1681 2980 Crs 1688 2982 Crs 1694 2982 Crs 1700 2982 Crs 1706 2982 Crs 1713 2983 Crs 1719 2983 Crs 1725 2987 Crs 1731 2987 Crs 1738 2988 Crs 1744 2988 Crs 1750 2989 Crs 1756 2989 Crs 1763 2989 Crs 1769 2989 Crs 1775 3049 Crs 1781 3049 Crs 1788 3049 Crs 1794 3049 Crs 1800 3049 Crs 1806 3049 Crs 1813 3052 Crs 1819 3052 Crs 1825 3055 Crs 1831 3055 Crs 1838 3055 Crs 1844 3055 Crs 1850 3055 Crs 1856 3055 Crs 1862 3056 Crs 1869 3056 Crs 1875 3057 Crs 1881 3057 Crs 1887 3114 Crs 1894 3114 Crs 1900 3114 Crs 1906 3114 Crs 1912 3119 Crs 1919 3119 Crs 1925 3119 Crs 1931 3119 Crs 1937 3120 Crs 1944 3120 Crs 1950 3122 Crs 1956 3122 Crs 1962 3122 Crs 1969 3122 Crs 1975 3125 Crs 1981 3125 Crs 1987 3125 Crs 1994 3125 Crs 2000 3182 Crs 2006 3182 Crs 2012 3184 Crs 2019 3184 Crs 2025 3185 Crs 2031 3185 Crs 2037 3189 Crs 2043 3189 Crs 2050 3189 Crs 2056 3189 Crs 2062 3190 Crs 2068 3190 Crs 2075 3192 Crs 2081 3192 Crs 2087 3192 Crs 2093 3192 Crs 2100 3192 Crs 2106 3192 Crs 2112 3250 Crs 2118 3250 Crs 2125 3250 Crs 2131 3250 Crs 2137 3252 Crs 2143 3252 Crs 2150 3256 Crs 2156 3256 Crs 2162 3258 Crs 2168 3258 Crs 2175 3258 Crs 2181 3258 Crs 2187 3259 Crs 2193 3259 Crs 2200 3259 Crs 2206 3259 Crs 2212 3259 Crs 2218 3259 Crs 2225 3319 Crs 2231 3319 Crs 2237 3320 Crs 2243 3320 Crs 2249 3320 Crs 2256 3320 Crs 2262 3320 Crs 2268 3320 Crs 2274 3320 Crs 2281 3320 Crs 2287 3320 Crs 2293 3320 Crs 2299 3320 Crs 2306 3320 Crs 2312 3320 Crs 2318 3320 Crs 2324 3320 Crs 2331 3320 Crs 2337 3320 Crs 2343 3320 Crs 2349 3320 Crs 2356 3320 Crs 2362 3320 Crs 2368 3320 Crs 2374 3325 Crs 2381 3325 Crs 2387 3325 Crs 2393 3325 Crs 2399 3325 Crs 2406 3325 Crs 2412 3326 Crs 2418 3326 Crs 2424 3326 Crs 2430 3326 Crs 2437 3326 Crs 2443 3326 Crs 2449 3327 Crs 2455 3327 Crs 2462 3329 Crs 2468 3329 Crs 2474 3386 Crs 2480 3386 Crs 2487 3387 Crs 2493 3387 Crs 2499 3389 Crs 2505 3389 Crs 2512 3392 Crs 2518 3392 Crs 2524 3392 Crs 2530 3395 Crs 2537 3395 Crs 2543 3395 Crs 2549 3395 Crs 2555 3395 Crs 2562 3395 Crs 2568 3395 Crs 2574 3396 Crs 2580 3396 Crs 2587 3450 Crs 2593 3450 Crs 2599 3450 Crs 2605 3450 Crs 2611 3457 Crs 2618 3457 Crs 2624 3462 Crs 2630 3462 Crs 2636 3462 Crs 2643 3462 Crs 2649 3462 Crs 2655 3462 Crs 2661 3462 Crs 2668 3462 Crs 2674 3465 Crs 2680 3465 Crs 2686 3486 Crs 2693 3486 Crs 2699 3486 Crs 2705 3486 Crs 2711 3509 Crs 2718 3514 Crs 2724 3517 Crs 2730 3517 Crs 2736 3517 Crs 2743 3517 Crs 2749 3517 Crs 2755 3517 Crs 2761 3517 Crs 2768 3517 Crs 2774 3517 Crs 2780 3517 Crs 2786 3520 Crs 2793 3523 Crs 2799 3523 Crs 2805 3523 Crs 2811 3523 Crs 2817 3523 Crs 2824 3523 Crs 2830 3523 Crs 2836 3523 Crs 2842 3523 Crs 2849 3523 Crs 2855 3523 Crs 2861 3523 Crs 2867 3523 Crs 2874 3523 Crs 2880 3523 Crs 2886 3523 Crs 2892 3523 Crs 2899 3523 Crs 2905 3523 Crs 2911 3523 Crs 2917 3523 Crs 2924 3523 Crs 2930 3523 Crs 2936 3523 Crs 2942 3523 Crs 2949 3523 Crs 2955 3523 Crs 2961 3523 Crs 2967 3523 Crs 2974 3523 Crs 2980 3523 Crs 2986 3523 Crs 2992 3523 Crs 2998 3523 Crs 3005 3523 Crs 3011 3523 Crs 3017 3523 Crs 3023 3523 Crs 3030 3523 Crs 3036 3523 Crs 3042 3523 Crs 3048 3523 Crs 3055 3523 Crs 3061 3523 Crs 3067 3523 Crs 3073 3523 Crs 3080 3523 Crs 3086 3523 Crs 3092 3523 Crs 3098 3523 Crs 3105 3523 Crs 3111 3523 Crs 3117 3523 Crs 3123 3523 Crs 3130 3523 Crs 3136 3523 Crs 3142 3523 Crs 3148 3523 Crs 3155 3523 Crs 3161 3523 Crs 3167 3523 Crs 3173 3523 Crs 3179 3523 Crs 3186 3523 Crs 3192 3523 Crs 3198 3523 Crs 3204 3523 Crs 3211 3523 Crs 3217 3523 Crs 3223 3523 Crs 3229 3523 Crs 3236 3523 Crs 3242 3523 Crs 3248 3523 Crs 3254 3523 Crs 3261 3523 Crs 3267 3523 Crs 3273 3523 Crs 3279 3523 Crs 3286 3523 Crs 3292 3523 Crs 3298 3523 Crs 3304 3523 Crs 3311 3523 Crs 3317 3523 Crs 3323 3523 Crs 3329 3523 Crs 3336 3523 Crs 3342 3523 Crs 3348 3523 Crs 3354 3523 Crs 3361 3523 Crs 3367 3523 Crs 3373 3523 Crs 3379 3523 Crs 3385 3523 Crs 3392 3523 Crs 3398 3523 Crs 3404 3523 Crs 3410 3523 Crs 3417 3523 Crs 3423 3523 Crs 3429 3523 Crs 3435 3523 Crs 3442 3523 Crs 3448 3523 Crs 3454 3523 Crs 3460 3523 Crs 3467 3523 Crs 3473 3523 Crs 3479 3523 Crs 3485 3523 Crs 3492 3523 Crs 3498 3523 Crs 3504 3523 Crs 3510 3523 Crs 3517 3523 Crs 3523 3523 Crs 3529 3523 Crs 3535 3523 Crs 3542 3523 Crs 3548 3523 Crs 3554 3523 Crs 3560 3523 Crs 3566 3523 Crs 3573 3523 Crs 3579 3523 Crs 3585 3523 Crs 3591 3523 Crs 3598 3523 Crs 3604 3523 Crs 3610 3523 Crs 3616 3523 Crs 3623 3523 Crs 3629 3523 Crs 3635 3523 Crs 3641 3523 Crs 3648 3523 Crs 3654 3523 Crs 3660 3523 Crs 3666 3523 Crs 3673 3523 Crs 3679 3523 Crs 3685 3523 Crs 3691 3523 Crs 3698 3523 Crs 3704 3523 Crs 3710 3523 Crs 3716 3523 Crs 3723 3523 Crs 3729 3523 Crs 3735 3523 Crs 3741 3524 Crs 3747 3524 Crs 3754 3524 Crs 3760 3524 Crs 3766 3526 Crs 3772 3526 Crs 3779 3526 Crs 3785 3526 Crs 3791 3526 Crs 3797 3526 Crs 3804 3526 Crs 3810 3526 Crs 3816 3526 Crs 3822 3526 Crs 3829 3526 Crs 3835 3526 Crs 3841 3526 Crs 3847 3526 Crs 3854 3526 Crs 3860 3526 Crs 3866 3526 Crs 3872 3526 Crs 3879 3526 Crs 3885 3526 Crs 3891 3526 Crs 3897 3526 Crs 3904 3526 Crs 3910 3526 Crs 3916 3526 Crs 3922 3526 Crs 3929 3526 Crs 3935 3526 Crs 3941 3526 Crs 3947 3526 Crs 3953 3526 Crs 3960 3526 Crs 3966 3526 Crs 3972 3526 Crs 3978 3526 Crs 3985 3526 Crs 3991 3526 Crs 3997 3527 Crs 4003 3528 Crs 4010 3528 Crs 4016 3528 Crs 4022 3529 Crs 4028 3529 Crs 4035 3529 Crs 4041 3529 Crs 4047 3529 Crs 4053 3529 Crs 4060 3529 Crs 4066 3529 Crs 4072 3532 Crs 4078 3532 Crs 4085 3532 Crs 4091 3532 Crs 4097 3546 Crs 4103 3559 Crs 4110 3559 Crs 4116 3559 Crs 4122 3559 Crs 4128 3583 Crs 4134 3583 Crs 4141 3583 Crs 4147 3587 Crs 4153 3590 Crs 4159 3590 Crs 4166 3590 Crs 4172 3590 Crs 4178 3590 Crs 4184 3590 Crs 4191 3590 Crs 4197 3590 Crs 4203 3590 Crs 4209 3590 Crs 4216 3590 Crs 4222 3590 Crs 4228 3590 Crs 4234 3590 Crs 4241 3590 Crs 4247 3590 Crs 4253 3590 Crs 4259 3590 Crs 4266 3590 Crs 4272 3590 Crs 4278 3590 Crs 4284 3590 Crs 4291 3590 Crs 4297 3590 Crs 4303 3590 Crs 4309 3590 Crs 4315 3590 Crs 4322 3590 Crs 4328 3590 Crs 4334 3590 Crs 4340 3590 Crs 4347 3590 Crs 4353 3590 Crs 4359 3590 Crs 4365 3590 Crs 4372 3590 Crs 4378 3590 Crs 4384 3590 Crs 4390 3590 Crs 4397 3590 Crs 4403 3590 Crs 4409 3590 Crs 4415 3590 Crs 4422 3590 Crs 4428 3590 Crs 4434 3590 Crs 4440 3590 Crs 4447 3590 Crs 4453 3590 Crs 4459 3590 Crs 4465 3590 Crs 4472 3590 Crs 4478 3590 Crs 4484 3590 Crs 4490 3590 Crs 4497 3590 Crs 4503 3590 Crs 4509 3590 Crs 4515 3590 Crs 4521 3590 Crs 4528 3590 Crs 4534 3590 Crs 4540 3590 Crs 4546 3590 Crs 4553 3590 Crs 4559 3590 Crs 4565 3590 Crs 4571 3590 Crs 4578 3590 Crs 4584 3590 Crs 4590 3590 Crs 4596 3590 Crs 4603 3590 Crs 4609 3590 Crs 4615 3590 Crs 4621 3590 Crs 4628 3590 Crs 4634 3590 Crs 4640 3590 Crs 4646 3590 Crs 4653 3590 Crs 4659 3590 Crs 4665 3590 Crs 4671 3590 Crs 4678 3590 Crs 4684 3590 Crs 4690 3590 Crs 4696 3590 Crs 4702 3590 Crs 4709 3590 Crs 4715 3590 Crs 4721 3590 Crs 4727 3590 Crs 4734 3590 Crs 4740 3590 Crs 4746 3590 Crs 4752 3590 Crs 4759 3590 Crs 4765 3590 Crs 4771 3590 Crs 4777 3590 Crs 4784 3590 Crs 4790 3590 Crs 4796 3590 Crs 4802 3590 Crs 4809 3591 Crs 4815 3591 Crs 4821 3591 Crs 4827 3591 Crs 4834 3593 Crs 4840 3593 Crs 4846 3593 Crs 4852 3593 Crs 4859 3593 Crs 4865 3593 Crs 4871 3593 Crs 4877 3593 Crs 4883 3593 Crs 4890 3593 Crs 4896 3593 Crs 4902 3593 Crs 4908 3593 Crs 4915 3593 Crs 4921 3593 Crs 4927 3593 Crs 4933 3593 Crs 4940 3593 Crs 4946 3593 Crs 4952 3593 Crs 4958 3593 Crs 4965 3593 Crs 4971 3593 Crs 4977 3593 Crs 4983 3593 Crs 4990 3593 Crs 4996 3593 Crs 5002 3593 Crs 5008 3593 Crs 5015 3593 Crs 5021 3593 Crs 5027 3593 Crs 5033 3593 Crs 5040 3593 Crs 5046 3593 Crs 5052 3593 Crs 5058 3593 Crs 5065 3593 Crs 5071 3593 Crs 5077 3593 Crs 5083 3593 Crs 5089 3593 Crs 5096 3593 Crs 5102 3593 Crs 5108 3593 Crs 5114 3593 Crs 5121 3593 Crs 5127 3593 Crs 5133 3593 Crs 5139 3593 Crs 5146 3593 Crs 5152 3593 Crs 5158 3593 Crs 5164 3593 Crs 5171 3593 Crs 5177 3593 Crs 5183 3593 Crs 5189 3593 Crs 5196 3593 Crs 5202 3593 Crs 5208 3593 Crs 5214 3593 Crs 5221 3593 Crs 5227 3593 Crs 5233 3593 Crs 5239 3593 Crs 5246 3593 Crs 5252 3593 Crs 5258 3593 Crs 5264 3593 Crs 5270 3593 Crs 5277 3593 Crs 5283 3593 Crs 5289 3593 Crs 5295 3593 Crs 5302 3593 Crs 5308 3593 Crs 5314 3593 Crs 5320 3593 Crs 5327 3593 Crs 5333 3593 Crs 5339 3593 Crs 5345 3593 Crs 5352 3593 Crs 5358 3593 Crs 5364 3593 Crs 5370 3593 Crs 5377 3593 Crs 5383 3593 Crs 5389 3594 Crs 5395 3594 Crs 5402 3594 Crs 5408 3594 Crs 5414 3594 Crs 5420 3595 Crs 5427 3595 Crs 5433 3595 Crs 5439 3596 Crs 5445 3596 Crs 5451 3596 Crs 5458 3596 Crs 5464 3596 Crs 5470 3598 Crs 5476 3598 Crs 5483 3599 Crs 5489 3599 Crs 5495 3619 Crs 5501 3619 Crs 5508 3629 Crs 5514 3629 Crs 5520 3629 Crs 5526 3629 Crs 5533 3660 Crs 5539 3660 Crs 5545 3661 Crs 5551 3661 Crs 5558 3661 Crs 5564 3661 Crs 5570 3661 Crs 5576 3661 Crs 5583 3685 Crs 5589 3685 Crs 5595 3696 Crs 5601 3696 Crs 5608 3696 Crs 5614 3696 Crs 5620 3727 Crs 5626 3727 Crs 5633 3727 Crs 5639 3727 Crs 5645 3727 Crs 5651 3728 Crs 5657 3728 Crs 5664 3728 Crs 5670 3728 Crs 5676 3731 Crs 5682 3752 Crs 5689 3752 Crs 5695 3762 Crs 5701 3762 Crs 5707 3762 Crs 5714 3762 Crs 5720 3794 Crs 5726 3794 Crs 5732 3794 Crs 5739 3794 Crs 5745 3794 Crs 5751 3794 Crs 5757 3794 Crs 5764 3794 Crs 5770 3797 Crs 5776 3797 Crs 5782 3798 Crs 5789 3800 Crs 5795 3800 Crs 5801 3800 Crs 5807 3800 Crs 5814 3801 Crs 5820 3801 Crs 5826 3819 Crs 5832 3819 Crs 5838 3832 Crs 5845 3832 Crs 5851 3832 Crs 5857 3832 Crs 5863 3860 Crs 5870 3860 Crs 5876 3860 Crs 5882 3863 Crs 5888 3864 Crs 5895 3864 Crs 5901 3864 Crs 5907 3864 Crs 5913 3864 Crs 5920 3864 Crs 5926 3864 Crs 5932 3864 Crs 5938 3867 Crs 5945 3867 Crs 5951 3870 Crs 5957 3870 Crs 5963 3870 Crs 5970 3870 Crs 5976 3889 Crs 5982 3889 Crs 5988 3899 Crs 5995 3899 Crs 6001 3899 Crs 6007 3899 Crs 6013 3927 Crs 6019 3930 Crs 6026 3930 Crs 6032 3931 Crs 6038 3931 Crs 6044 3931 Crs 6051 3931 Crs 6057 3931 Crs 6063 3931 Crs 6069 3933 Crs 6076 3934 Crs 6082 3934 Crs 6088 3935 Crs 6094 3935 Crs 6101 3937 Crs 6107 3937 Crs 6113 3955 Crs 6119 3955 Crs 6126 3965 Crs 6132 3965 Crs 6138 3966 Crs 6144 3966 Crs 6151 3966 Crs 6157 3966 Crs 6163 3972 Crs 6169 3997 Crs 6176 3997 Crs 6182 3997 Crs 6188 3997 Crs 6194 3998 Crs 6201 3998 Crs 6207 4001 Crs 6213 4001 Crs 6219 4003 Crs 6225 4003 Crs 6232 4004 Crs 6238 4004 Crs 6244 4004 Crs 6250 4004 Crs 6257 4022 Crs 6263 4032 Crs 6269 4032 Crs 6275 4032 Crs 6282 4032 Crs 6288 4064 Crs 6294 4064 Crs 6300 4064 Crs 6307 4064 Crs 6313 4067 Crs 6319 4067 Crs 6325 4067 Crs 6332 4067 Crs 6338 4070 Crs 6344 4070 Crs 6350 4071 Crs 6357 4071 Crs 6363 4071 Crs 6369 4071 Crs 6375 4102 Crs 6382 4102 Crs 6388 4102 Crs 6394 4102 Crs 6400 4134 Crs 6406 4134 Crs 6413 4134 Crs 6419 4134 Crs 6425 4134 Crs 6431 4134 Crs 6438 4137 Crs 6444 4137 Crs 6450 4140 Crs 6456 4140 Crs 6463 4140 Crs 6469 4144 Crs 6475 4144 Crs 6481 4169 Crs 6488 4169 Crs 6494 4169 Crs 6500 4200 Crs 6506 4200 Crs 6513 4201 Crs 6519 4201 Crs 6525 4201 Crs 6531 4201 Crs 6538 4201 Crs 6544 4201 Crs 6550 4207 Crs 6556 4207 Crs 6563 4267 Crs 6569 4267 Crs 6575 4268 Crs 6581 4268 Crs 6587 4268 Crs 6594 4271 Crs 6600 4271 Crs 6606 4274 Crs 6612 4274 Crs 6619 4334 Crs 6625 4334 Crs 6631 4337 Crs 6637 4337 Crs 6644 4337 Crs 6650 4337 Crs 6656 4344 Crs 6662 4344 Crs 6669 4404 Crs 6675 4404 Crs 6681 4404 Crs 6687 4404 Crs 6694 4404 Crs 6700 4404 Crs 6706 4410 Crs 6712 4410 Crs 6719 4471 Crs 6725 4471 Crs 6731 4471 Crs 6737 4471 Crs 6744 4474 Crs 6750 4474 Crs 6756 4477 Crs 6762 4477 Crs 6769 4537 Crs 6775 4537 Crs 6781 4537 Crs 6787 4537 Crs 6793 4544 Crs 6800 4544 Crs 6806 4547 Crs 6812 4547 Crs 6818 4607 Crs 6825 4607 Crs 6831 4608 Crs 6837 4608 Crs 6843 4611 Crs 6850 4611 Crs 6856 4614 Crs 6862 4614 Crs 6868 4672 Crs 6875 4672 Crs 6881 4674 Crs 6887 4674 Crs 6893 4674 Crs 6900 4674 Crs 6906 4681 Crs 6912 4681 Crs 6918 4681 Crs 6925 4681 Crs 6931 4741 Crs 6937 4741 Crs 6943 4747 Crs 6950 4747 Crs 6956 4750 Crs 6610 4599 Crs 0.500 UP 1.000 UL LT2 6343 4459 M (Max) Rshow 720 544 Star 726 610 Star 733 610 Star 739 645 Star 745 715 Star 751 715 Star 758 782 Star 764 782 Star 770 848 Star 776 848 Star 783 915 Star 789 915 Star 795 985 Star 801 985 Star 808 1052 Star 814 1052 Star 820 1119 Star 826 1119 Star 833 1188 Star 839 1188 Star 845 1255 Star 851 1255 Star 858 1322 Star 864 1322 Star 870 1389 Star 876 1389 Star 883 1458 Star 889 1458 Star 895 1531 Star 901 1531 Star 907 1598 Star 914 1598 Star 920 1668 Star 926 1668 Star 932 1735 Star 939 1735 Star 945 1801 Star 951 1801 Star 957 1868 Star 964 1868 Star 970 1938 Star 976 1938 Star 982 2005 Star 989 2005 Star 995 2072 Star 1001 2072 Star 1007 2142 Star 1014 2142 Star 1020 2208 Star 1026 2208 Star 1032 2275 Star 1039 2275 Star 1045 2372 Star 1051 2372 Star 1057 2379 Star 1064 2379 Star 1070 2382 Star 1076 2439 Star 1082 2439 Star 1089 2443 Star 1095 2444 Star 1101 2445 Star 1107 2445 Star 1113 2448 Star 1120 2448 Star 1126 2452 Star 1132 2452 Star 1138 2509 Star 1145 2509 Star 1151 2512 Star 1157 2512 Star 1163 2515 Star 1170 2515 Star 1176 2515 Star 1182 2515 Star 1188 2518 Star 1195 2518 Star 1201 2576 Star 1207 2576 Star 1213 2579 Star 1220 2579 Star 1226 2582 Star 1232 2582 Star 1238 2582 Star 1245 2582 Star 1251 2585 Star 1257 2585 Star 1263 2645 Star 1270 2645 Star 1276 2648 Star 1282 2648 Star 1288 2649 Star 1294 2649 Star 1301 2652 Star 1307 2652 Star 1313 2655 Star 1319 2655 Star 1326 2712 Star 1332 2712 Star 1338 2715 Star 1344 2715 Star 1351 2719 Star 1357 2719 Star 1363 2719 Star 1369 2719 Star 1376 2722 Star 1382 2722 Star 1388 2779 Star 1394 2779 Star 1401 2785 Star 1407 2785 Star 1413 2785 Star 1419 2785 Star 1426 2785 Star 1432 2785 Star 1438 2786 Star 1444 2786 Star 1451 2786 Star 1457 2786 Star 1463 2786 Star 1469 2787 Star 1475 2787 Star 1482 2788 Star 1488 2788 Star 1494 2846 Star 1500 2846 Star 1507 2852 Star 1513 2852 Star 1519 2852 Star 1525 2852 Star 1532 2852 Star 1538 2852 Star 1544 2855 Star 1550 2855 Star 1557 2856 Star 1563 2856 Star 1569 2916 Star 1575 2916 Star 1582 2919 Star 1588 2919 Star 1594 2919 Star 1600 2919 Star 1607 2919 Star 1613 2922 Star 1619 2922 Star 1625 2922 Star 1632 2923 Star 1638 2925 Star 1644 2925 Star 1650 2925 Star 1657 2925 Star 1663 2982 Star 1669 2982 Star 1675 2985 Star 1681 2985 Star 1688 2986 Star 1694 2986 Star 1700 2989 Star 1706 2989 Star 1713 2989 Star 1719 2989 Star 1725 2989 Star 1731 2989 Star 1738 2989 Star 1744 2989 Star 1750 2992 Star 1756 2992 Star 1763 2992 Star 1769 2992 Star 1775 3052 Star 1781 3052 Star 1788 3055 Star 1794 3055 Star 1800 3055 Star 1806 3055 Star 1813 3055 Star 1819 3055 Star 1825 3055 Star 1831 3055 Star 1838 3057 Star 1844 3057 Star 1850 3057 Star 1856 3057 Star 1862 3059 Star 1869 3059 Star 1875 3065 Star 1881 3065 Star 1887 3120 Star 1894 3120 Star 1900 3120 Star 1906 3120 Star 1912 3122 Star 1919 3122 Star 1925 3122 Star 1931 3122 Star 1937 3125 Star 1944 3125 Star 1950 3125 Star 1956 3125 Star 1962 3125 Star 1969 3125 Star 1975 3126 Star 1981 3126 Star 1987 3126 Star 1994 3126 Star 2000 3189 Star 2006 3189 Star 2012 3190 Star 2019 3190 Star 2025 3190 Star 2031 3190 Star 2037 3192 Star 2043 3192 Star 2050 3192 Star 2056 3192 Star 2062 3192 Star 2068 3192 Star 2075 3192 Star 2081 3192 Star 2087 3195 Star 2093 3195 Star 2100 3196 Star 2106 3196 Star 2112 3257 Star 2118 3257 Star 2125 3257 Star 2131 3257 Star 2137 3258 Star 2143 3258 Star 2150 3259 Star 2156 3259 Star 2162 3259 Star 2168 3259 Star 2175 3259 Star 2181 3259 Star 2187 3259 Star 2193 3259 Star 2200 3262 Star 2206 3262 Star 2212 3262 Star 2218 3262 Star 2225 3325 Star 2231 3325 Star 2237 3325 Star 2243 3325 Star 2249 3325 Star 2256 3325 Star 2262 3326 Star 2268 3326 Star 2274 3326 Star 2281 3326 Star 2287 3326 Star 2293 3326 Star 2299 3326 Star 2306 3326 Star 2312 3326 Star 2318 3326 Star 2324 3326 Star 2331 3326 Star 2337 3326 Star 2343 3326 Star 2349 3326 Star 2356 3326 Star 2362 3326 Star 2368 3326 Star 2374 3326 Star 2381 3326 Star 2387 3326 Star 2393 3326 Star 2399 3326 Star 2406 3326 Star 2412 3327 Star 2418 3327 Star 2424 3327 Star 2430 3327 Star 2437 3329 Star 2443 3329 Star 2449 3329 Star 2455 3329 Star 2462 3329 Star 2468 3329 Star 2474 3392 Star 2480 3392 Star 2487 3392 Star 2493 3393 Star 2499 3393 Star 2505 3393 Star 2512 3393 Star 2518 3395 Star 2524 3395 Star 2530 3395 Star 2537 3395 Star 2543 3395 Star 2549 3396 Star 2555 3396 Star 2562 3398 Star 2568 3398 Star 2574 3401 Star 2580 3401 Star 2587 3457 Star 2593 3457 Star 2599 3457 Star 2605 3457 Star 2611 3462 Star 2618 3462 Star 2624 3462 Star 2630 3462 Star 2636 3462 Star 2643 3462 Star 2649 3463 Star 2655 3463 Star 2661 3465 Star 2668 3465 Star 2674 3466 Star 2680 3466 Star 2686 3493 Star 2693 3493 Star 2699 3493 Star 2705 3493 Star 2711 3519 Star 2718 3520 Star 2724 3520 Star 2730 3520 Star 2736 3520 Star 2743 3520 Star 2749 3523 Star 2755 3526 Star 2761 3526 Star 2768 3526 Star 2774 3526 Star 2780 3526 Star 2786 3526 Star 2793 3526 Star 2799 3526 Star 2805 3526 Star 2811 3526 Star 2817 3526 Star 2824 3526 Star 2830 3526 Star 2836 3526 Star 2842 3526 Star 2849 3526 Star 2855 3526 Star 2861 3526 Star 2867 3526 Star 2874 3526 Star 2880 3526 Star 2886 3526 Star 2892 3526 Star 2899 3526 Star 2905 3526 Star 2911 3526 Star 2917 3526 Star 2924 3526 Star 2930 3526 Star 2936 3526 Star 2942 3526 Star 2949 3526 Star 2955 3526 Star 2961 3526 Star 2967 3526 Star 2974 3526 Star 2980 3526 Star 2986 3526 Star 2992 3526 Star 2998 3526 Star 3005 3526 Star 3011 3526 Star 3017 3526 Star 3023 3526 Star 3030 3526 Star 3036 3526 Star 3042 3526 Star 3048 3526 Star 3055 3526 Star 3061 3526 Star 3067 3526 Star 3073 3526 Star 3080 3526 Star 3086 3526 Star 3092 3526 Star 3098 3526 Star 3105 3526 Star 3111 3526 Star 3117 3526 Star 3123 3526 Star 3130 3526 Star 3136 3526 Star 3142 3526 Star 3148 3526 Star 3155 3526 Star 3161 3526 Star 3167 3526 Star 3173 3526 Star 3179 3526 Star 3186 3526 Star 3192 3526 Star 3198 3526 Star 3204 3526 Star 3211 3526 Star 3217 3526 Star 3223 3526 Star 3229 3526 Star 3236 3526 Star 3242 3526 Star 3248 3526 Star 3254 3526 Star 3261 3526 Star 3267 3526 Star 3273 3526 Star 3279 3526 Star 3286 3526 Star 3292 3526 Star 3298 3526 Star 3304 3526 Star 3311 3526 Star 3317 3526 Star 3323 3526 Star 3329 3526 Star 3336 3526 Star 3342 3526 Star 3348 3526 Star 3354 3526 Star 3361 3526 Star 3367 3526 Star 3373 3526 Star 3379 3526 Star 3385 3526 Star 3392 3526 Star 3398 3526 Star 3404 3526 Star 3410 3526 Star 3417 3526 Star 3423 3526 Star 3429 3526 Star 3435 3526 Star 3442 3526 Star 3448 3526 Star 3454 3526 Star 3460 3526 Star 3467 3526 Star 3473 3526 Star 3479 3526 Star 3485 3526 Star 3492 3526 Star 3498 3526 Star 3504 3526 Star 3510 3526 Star 3517 3527 Star 3523 3527 Star 3529 3527 Star 3535 3527 Star 3542 3527 Star 3548 3527 Star 3554 3527 Star 3560 3527 Star 3566 3527 Star 3573 3527 Star 3579 3527 Star 3585 3527 Star 3591 3527 Star 3598 3527 Star 3604 3527 Star 3610 3527 Star 3616 3527 Star 3623 3527 Star 3629 3527 Star 3635 3527 Star 3641 3527 Star 3648 3527 Star 3654 3527 Star 3660 3527 Star 3666 3527 Star 3673 3527 Star 3679 3527 Star 3685 3527 Star 3691 3527 Star 3698 3527 Star 3704 3527 Star 3710 3527 Star 3716 3527 Star 3723 3527 Star 3729 3527 Star 3735 3527 Star 3741 3527 Star 3747 3527 Star 3754 3528 Star 3760 3528 Star 3766 3529 Star 3772 3529 Star 3779 3529 Star 3785 3529 Star 3791 3529 Star 3797 3529 Star 3804 3529 Star 3810 3529 Star 3816 3529 Star 3822 3529 Star 3829 3530 Star 3835 3530 Star 3841 3530 Star 3847 3530 Star 3854 3530 Star 3860 3530 Star 3866 3530 Star 3872 3530 Star 3879 3530 Star 3885 3530 Star 3891 3530 Star 3897 3530 Star 3904 3530 Star 3910 3530 Star 3916 3530 Star 3922 3530 Star 3929 3530 Star 3935 3530 Star 3941 3530 Star 3947 3530 Star 3953 3530 Star 3960 3530 Star 3966 3530 Star 3972 3530 Star 3978 3530 Star 3985 3530 Star 3991 3530 Star 3997 3530 Star 4003 3530 Star 4010 3530 Star 4016 3530 Star 4022 3530 Star 4028 3530 Star 4035 3530 Star 4041 3530 Star 4047 3531 Star 4053 3531 Star 4060 3532 Star 4066 3532 Star 4072 3532 Star 4078 3533 Star 4085 3537 Star 4091 3537 Star 4097 3552 Star 4103 3559 Star 4110 3560 Star 4116 3560 Star 4122 3560 Star 4128 3593 Star 4134 3593 Star 4141 3593 Star 4147 3593 Star 4153 3593 Star 4159 3593 Star 4166 3593 Star 4172 3593 Star 4178 3593 Star 4184 3593 Star 4191 3593 Star 4197 3593 Star 4203 3593 Star 4209 3593 Star 4216 3593 Star 4222 3593 Star 4228 3593 Star 4234 3593 Star 4241 3593 Star 4247 3593 Star 4253 3593 Star 4259 3593 Star 4266 3593 Star 4272 3593 Star 4278 3593 Star 4284 3593 Star 4291 3593 Star 4297 3593 Star 4303 3593 Star 4309 3593 Star 4315 3593 Star 4322 3593 Star 4328 3593 Star 4334 3593 Star 4340 3593 Star 4347 3593 Star 4353 3593 Star 4359 3593 Star 4365 3593 Star 4372 3593 Star 4378 3593 Star 4384 3593 Star 4390 3593 Star 4397 3593 Star 4403 3593 Star 4409 3593 Star 4415 3593 Star 4422 3593 Star 4428 3593 Star 4434 3593 Star 4440 3593 Star 4447 3593 Star 4453 3593 Star 4459 3593 Star 4465 3593 Star 4472 3593 Star 4478 3593 Star 4484 3593 Star 4490 3593 Star 4497 3593 Star 4503 3593 Star 4509 3593 Star 4515 3593 Star 4521 3593 Star 4528 3593 Star 4534 3593 Star 4540 3593 Star 4546 3593 Star 4553 3593 Star 4559 3593 Star 4565 3593 Star 4571 3593 Star 4578 3593 Star 4584 3593 Star 4590 3593 Star 4596 3593 Star 4603 3593 Star 4609 3593 Star 4615 3593 Star 4621 3593 Star 4628 3593 Star 4634 3593 Star 4640 3593 Star 4646 3593 Star 4653 3593 Star 4659 3593 Star 4665 3593 Star 4671 3593 Star 4678 3593 Star 4684 3593 Star 4690 3593 Star 4696 3593 Star 4702 3593 Star 4709 3593 Star 4715 3593 Star 4721 3593 Star 4727 3593 Star 4734 3593 Star 4740 3593 Star 4746 3593 Star 4752 3593 Star 4759 3593 Star 4765 3593 Star 4771 3593 Star 4777 3593 Star 4784 3593 Star 4790 3593 Star 4796 3593 Star 4802 3593 Star 4809 3594 Star 4815 3594 Star 4821 3594 Star 4827 3594 Star 4834 3595 Star 4840 3595 Star 4846 3596 Star 4852 3596 Star 4859 3596 Star 4865 3596 Star 4871 3596 Star 4877 3596 Star 4883 3596 Star 4890 3596 Star 4896 3596 Star 4902 3596 Star 4908 3596 Star 4915 3596 Star 4921 3596 Star 4927 3596 Star 4933 3596 Star 4940 3596 Star 4946 3596 Star 4952 3596 Star 4958 3596 Star 4965 3596 Star 4971 3596 Star 4977 3596 Star 4983 3596 Star 4990 3596 Star 4996 3596 Star 5002 3596 Star 5008 3596 Star 5015 3596 Star 5021 3596 Star 5027 3596 Star 5033 3596 Star 5040 3596 Star 5046 3596 Star 5052 3596 Star 5058 3596 Star 5065 3596 Star 5071 3596 Star 5077 3596 Star 5083 3596 Star 5089 3596 Star 5096 3596 Star 5102 3596 Star 5108 3596 Star 5114 3596 Star 5121 3596 Star 5127 3596 Star 5133 3596 Star 5139 3596 Star 5146 3596 Star 5152 3596 Star 5158 3596 Star 5164 3596 Star 5171 3596 Star 5177 3596 Star 5183 3596 Star 5189 3596 Star 5196 3596 Star 5202 3596 Star 5208 3596 Star 5214 3596 Star 5221 3596 Star 5227 3596 Star 5233 3596 Star 5239 3596 Star 5246 3596 Star 5252 3596 Star 5258 3596 Star 5264 3596 Star 5270 3596 Star 5277 3596 Star 5283 3596 Star 5289 3596 Star 5295 3596 Star 5302 3596 Star 5308 3596 Star 5314 3596 Star 5320 3596 Star 5327 3596 Star 5333 3596 Star 5339 3596 Star 5345 3596 Star 5352 3596 Star 5358 3596 Star 5364 3596 Star 5370 3596 Star 5377 3596 Star 5383 3596 Star 5389 3596 Star 5395 3596 Star 5402 3596 Star 5408 3596 Star 5414 3597 Star 5420 3597 Star 5427 3597 Star 5433 3598 Star 5439 3598 Star 5445 3599 Star 5451 3599 Star 5458 3599 Star 5464 3599 Star 5470 3599 Star 5476 3600 Star 5483 3600 Star 5489 3601 Star 5495 3622 Star 5501 3622 Star 5508 3629 Star 5514 3629 Star 5520 3629 Star 5526 3629 Star 5533 3664 Star 5539 3664 Star 5545 3664 Star 5551 3664 Star 5558 3664 Star 5564 3664 Star 5570 3671 Star 5576 3671 Star 5583 3689 Star 5589 3689 Star 5595 3702 Star 5601 3702 Star 5608 3702 Star 5614 3702 Star 5620 3730 Star 5626 3730 Star 5633 3730 Star 5639 3731 Star 5645 3731 Star 5651 3731 Star 5657 3731 Star 5664 3731 Star 5670 3737 Star 5676 3737 Star 5682 3762 Star 5689 3762 Star 5695 3769 Star 5701 3769 Star 5707 3769 Star 5714 3769 Star 5720 3797 Star 5726 3797 Star 5732 3797 Star 5739 3797 Star 5745 3798 Star 5751 3798 Star 5757 3798 Star 5764 3800 Star 5770 3800 Star 5776 3801 Star 5782 3801 Star 5789 3801 Star 5795 3801 Star 5801 3801 Star 5807 3801 Star 5814 3807 Star 5820 3807 Star 5826 3828 Star 5832 3829 Star 5838 3839 Star 5845 3839 Star 5851 3839 Star 5857 3839 Star 5863 3867 Star 5870 3867 Star 5876 3867 Star 5882 3867 Star 5888 3867 Star 5895 3868 Star 5901 3868 Star 5907 3868 Star 5913 3868 Star 5920 3870 Star 5926 3870 Star 5932 3870 Star 5938 3871 Star 5945 3871 Star 5951 3877 Star 5957 3877 Star 5963 3877 Star 5970 3877 Star 5976 3898 Star 5982 3898 Star 5988 3906 Star 5995 3906 Star 6001 3906 Star 6007 3906 Star 6013 3934 Star 6019 3934 Star 6026 3934 Star 6032 3934 Star 6038 3934 Star 6044 3934 Star 6051 3937 Star 6057 3937 Star 6063 3937 Star 6069 3937 Star 6076 3937 Star 6082 3940 Star 6088 3943 Star 6094 3943 Star 6101 3943 Star 6107 3943 Star 6113 3965 Star 6119 3965 Star 6126 3968 Star 6132 3968 Star 6138 3972 Star 6144 3972 Star 6151 3975 Star 6157 3975 Star 6163 3978 Star 6169 4001 Star 6176 4001 Star 6182 4004 Star 6188 4004 Star 6194 4004 Star 6201 4004 Star 6207 4004 Star 6213 4004 Star 6219 4007 Star 6225 4007 Star 6232 4010 Star 6238 4010 Star 6244 4010 Star 6250 4010 Star 6257 4032 Star 6263 4039 Star 6269 4039 Star 6275 4039 Star 6282 4039 Star 6288 4067 Star 6294 4067 Star 6300 4067 Star 6307 4067 Star 6313 4068 Star 6319 4068 Star 6325 4073 Star 6332 4073 Star 6338 4073 Star 6344 4073 Star 6350 4074 Star 6357 4074 Star 6363 4077 Star 6369 4077 Star 6375 4109 Star 6382 4109 Star 6388 4109 Star 6394 4109 Star 6400 4137 Star 6406 4137 Star 6413 4138 Star 6419 4138 Star 6425 4141 Star 6431 4141 Star 6438 4141 Star 6444 4144 Star 6450 4144 Star 6456 4144 Star 6463 4146 Star 6469 4146 Star 6475 4146 Star 6481 4176 Star 6488 4176 Star 6494 4176 Star 6500 4204 Star 6506 4204 Star 6513 4204 Star 6519 4204 Star 6525 4210 Star 6531 4210 Star 6538 4210 Star 6544 4210 Star 6550 4211 Star 6556 4211 Star 6563 4271 Star 6569 4271 Star 6575 4277 Star 6581 4277 Star 6587 4277 Star 6594 4277 Star 6600 4277 Star 6606 4280 Star 6612 4280 Star 6619 4341 Star 6625 4341 Star 6631 4347 Star 6637 4347 Star 6644 4347 Star 6650 4347 Star 6656 4347 Star 6662 4347 Star 6669 4408 Star 6675 4408 Star 6681 4414 Star 6687 4414 Star 6694 4414 Star 6700 4414 Star 6706 4414 Star 6712 4414 Star 6719 4480 Star 6725 4480 Star 6731 4481 Star 6737 4481 Star 6744 4481 Star 6750 4481 Star 6756 4481 Star 6762 4481 Star 6769 4547 Star 6775 4547 Star 6781 4547 Star 6787 4547 Star 6793 4550 Star 6800 4550 Star 6806 4551 Star 6812 4551 Star 6818 4617 Star 6825 4617 Star 6831 4617 Star 6837 4617 Star 6843 4617 Star 6850 4617 Star 6856 4620 Star 6862 4620 Star 6868 4678 Star 6875 4678 Star 6881 4684 Star 6887 4684 Star 6893 4684 Star 6900 4684 Star 6906 4684 Star 6912 4684 Star 6918 4687 Star 6925 4687 Star 6931 4744 Star 6937 4751 Star 6943 4751 Star 6950 4754 Star 6956 4760 Star 6610 4459 Star stroke grestore end showpage %%Trailer %%DocumentFonts: Helvetica ------------=_1071074394-14104-0-- From xtang@qnx.com Wed Dec 10 14:05:06 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA21728 for ; Wed, 10 Dec 2003 12:36:34 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AU8GK-0005ET-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 12:36:36 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AU8GJ-0004kb-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 12:36:35 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 10 Dec 2003 09:37:55 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBAHZjZ0023060; Wed, 10 Dec 2003 09:35:46 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAHZZKs014819 for sctp-impl-filtered; Wed, 10 Dec 2003 09:35:37 -0800 (PST) Message-Id: <200312101735.hBAHZZKs014819@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071077735-14817-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Xiaodan Tang X-Nails-Approved: xtang@qnx.com Date: Wed, 10 Dec 2003 11:56:13 -0500 This is a multi-part message in MIME format... ------------=_1071077735-14817-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Instead of "peel_off()" a stream, how about allow sctp_rcvmsg() to be able to receive from a perticular steam_id ? As of now, we can sctp_sendmsg() to a specefic stream, but not receive on it. -xtang Andreas Jungmaier wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Hi folks, > >to add up to the discussion: > >in cases where there is only an outgoing stream, but no >incoming stream, applications can only send, not receive >on a "stream-socket". >In the opposite case, an application could only receive, >but not send. > >I don't know if that's a problem, since that can happen >as well with some streams within an SCTP association. > >However, I don't really like this idea. > >Best regards, >Andreas > >On Wednesday 10 December 2003 11:07, Kacheong Poon wrote: > > >>Randall Stewart (cisco) wrote: >> >> >>>Michael: >>> >>>Interesting point... but is not the problem NOT >>>that the cum-TSN cannot be forwarded... you can >>>always ack the data even though the receiving stream >>>is holding off reading.. but the issue is more how >>>does one manage the rwnd in this situation... I think >>>this is where you are going... If socket a-stream-1 stops >>>reading and socket a-stream-2 is reading.. then it is possible >>>that a-stream-1 will hold all of the rwnd and thus block the >>>association... Since we have no way to apportion the rwnd per >>>stream this is a problem.. >>> >>> >>This can be an issue. If an app chooses to peel off a >>stream, it has to be careful in handling the individual >>stream-socket to avoid blocking of other streams. But >>this also opens up the socket interface for people to play >>with streams scheduling. >> >>Any idea on how to make it safer? >> >> >> >>>Michael Tuexen wrote: >>> >>> >>>>Hi all, >>>> >>>>i think having a fd for a stream is a real problem. If the >>>>user 'decides' not to read from a particular stream because >>>>there is a separate thread/process for it which is blocked >>>>the receiver can not forward the cumulative TSN and therefore >>>>blocks the whole association. >>>> >>>>The receiver should always return in the receive call the >>>>(part of) a DATA chunk with lowest TSN. >>>> >>>>Best regards >>>>Michael >>>> >>>>On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: >>>> >>>> >>>>>Kacheong Poon wrote: >>>>> >>>>> >>>>>>I'd like to gather opinions about making sctp_peeloff() >>>>>>work in one-one style SCTP socket. The semantics is >>>>>>creating a new socket for a particular stream of an >>>>>>association. The stream number will be passed in the >>>>>>place of the assoc_id in the current sctp_peeloff(). >>>>>> >>>>>>Do people think that their apps can benefit from this? >>>>>>Thanks. >>>>>> >>>>>> >>>>>Hmm... There does seem to be some interest (at least >>>>>I have been contacted privately several times) on making >>>>>it so that you can read from a particular stream... >>>>> >>>>>I don't think I would be opposed to this... it will be a bit >>>>>of code for me, but has you have said .. its just code... and >>>>>it may well be quite useful... >>>>> >>>>>Let's see what other developers think... I could go for it :-> >>>>> >>>>>R >>>>> >>>>> > >- -- >Dipl.-Ing. Andreas Jungmaier >Computer Networking Technology Group >University of Duisburg-Essen >http://www.exp-math.uni-essen.de/~ajung >PGP Key-ID: D382 4AC0 > >PGP Key-Fingerprint: 228C 7C2C 3381 2DD4 9998 2812 5C0B 0B04 D382 4AC0 >-----BEGIN PGP SIGNATURE----- >Version: GnuPG v1.2.3 (GNU/Linux) > >iD8DBQE/1vLKXAsLBNOCSsARAjVDAKDRlPy3Hkg8KUFbu31DjUFJ5ca1sQCglS3Z >P5d8fnU5Jq3z+E+nB0Fn4IM= >=/AL3 >-----END PGP SIGNATURE----- > > > ------------=_1071077735-14817-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 10 16:38:55 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19734 for ; Wed, 10 Dec 2003 16:38:55 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUC2p-0002fp-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:38:56 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AUC2p-0002fY-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:38:55 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBALc0At022181; Wed, 10 Dec 2003 13:38:00 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBALbUKs017891 for sctp-impl-filtered; Wed, 10 Dec 2003 13:37:32 -0800 (PST) Message-Id: <200312102137.hBALbUKs017891@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071092250-17883-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 10 Dec 2003 21:56:07 +0100 This is a multi-part message in MIME format... ------------=_1071092250-17883-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Dear all, the idea on which streams are based on is to minimize the HOL problem. For the receiver it should not be important on which stream a message was received, it is only important that the receiver receives the messages in an order appropriate for the upper layer. That is the reason why there is no per stream flow control. If you need this you can setup separate associations. Regarding stream scheduling: The point is that the receiver always provides the packet with the lowest TSN to the user. This opens the rwnd. The scheduling takes place at the sender. The sender can assign the TSN at a very late point and can therefore provide a fair usage of the association between the streams. the fairness depends on the scheduling function. This is useful (some think necessary) for usage in the SS7 environment. I'm working on a paper describing that stuff. Best regards Michael On Dec 10, 2003, at 11:07 AM, Kacheong Poon wrote: > Randall Stewart (cisco) wrote: >> Michael: >> Interesting point... but is not the problem NOT >> that the cum-TSN cannot be forwarded... you can >> always ack the data even though the receiving stream >> is holding off reading.. but the issue is more how >> does one manage the rwnd in this situation... I think >> this is where you are going... If socket a-stream-1 stops >> reading and socket a-stream-2 is reading.. then it is possible >> that a-stream-1 will hold all of the rwnd and thus block the >> association... Since we have no way to apportion the rwnd per >> stream this is a problem.. > > This can be an issue. If an app chooses to peel off a > stream, it has to be careful in handling the individual > stream-socket to avoid blocking of other streams. But > this also opens up the socket interface for people to play > with streams scheduling. > > Any idea on how to make it safer? > > > >> Michael Tuexen wrote: >>> Hi all, >>> >>> i think having a fd for a stream is a real problem. If the >>> user 'decides' not to read from a particular stream because >>> there is a separate thread/process for it which is blocked >>> the receiver can not forward the cumulative TSN and therefore >>> blocks the whole association. >>> >>> The receiver should always return in the receive call the >>> (part of) a DATA chunk with lowest TSN. >>> >>> Best regards >>> Michael >>> >>> On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: >>> >>>> Kacheong Poon wrote: >>>> >>>>> I'd like to gather opinions about making sctp_peeloff() >>>>> work in one-one style SCTP socket. The semantics is >>>>> creating a new socket for a particular stream of an >>>>> association. The stream number will be passed in the >>>>> place of the assoc_id in the current sctp_peeloff(). >>>>> >>>>> Do people think that their apps can benefit from this? >>>>> Thanks. >>>>> >>>> Hmm... There does seem to be some interest (at least >>>> I have been contacted privately several times) on making >>>> it so that you can read from a particular stream... >>>> >>>> I don't think I would be opposed to this... it will be a bit >>>> of code for me, but has you have said .. its just code... and >>>> it may well be quite useful... >>>> >>>> Let's see what other developers think... I could go for it :-> >>>> >>>> R >>>> >>> >>> > > > -- > > K. Poon. > kacheong.poon@sun.com > > ------------=_1071092250-17883-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 10 16:42:46 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19860 for ; Wed, 10 Dec 2003 16:42:46 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUC6Z-0002kB-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:42:47 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUC6Z-0002jy-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:42:47 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-3.cisco.com with ESMTP; 10 Dec 2003 13:44:29 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBALfwZ0014486; Wed, 10 Dec 2003 13:41:59 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBALfnKs017949 for sctp-impl-filtered; Wed, 10 Dec 2003 13:41:51 -0800 (PST) Message-Id: <200312102141.hBALfnKs017949@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071092509-17947-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 10 Dec 2003 21:59:37 +0100 This is a multi-part message in MIME format... ------------=_1071092509-17947-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin, I think you are using the streams in a way they are (initially) not intended for. The problem is that you have only a per association receiver window, not a per stream. So it is very likely that one processing entity will block the other. On the other hand i do not see the usage of this. If you need a demultiplexer of some sort you might want to do it above SCTP. Best regards Michael On Dec 10, 2003, at 11:14 AM, Kacheong Poon wrote: > Caitlin Bestler wrote: > >> There is a major benefit to being able to use stream >> specific buffers when reading. >> But with the sockets API that has to be weighed against >> the overhead of using poll/select to find out which >> stream has supplied the buffer. >> If you migrate to more of an asynchronous interface, >> I believe the ideal is to allow receive buffers to be >> pre-posted on a per-stream basis, but to receive the >> completions on a consolidated socket. In terms of a >> VIA/IB/RDMA style interface, each stream has its own >> Send and Receive Queues, but the entire association >> has a single Completion Queue. > > One catch is that this kind of schemes, event completion > queue, normally align with file descriptor. This means > that after buffers are pre-posted to a socket, the > completion event is delivered to the same socket. The > reason is simply that the framework is also used by > other file descriptor, not just socket. I agree that > different implementations may have different architectures. > > Can people comment on whether the above file descriptor > alignment is common on different platforms' event completion > framework? If this is true, then peeling off a stream from > a socket fits what you mentioned above. > >> Personally, I believe there is more value in trying >> to add that type of functionality to the API than in >> running a classic synchronous socket for each stream. >> However, I would agree that there are uses for both. >> In particular, having to provide the target buffer >> for a read *before* knowing what stream the next >> message is for has always struck me as awkward. > > > -- > > K. Poon. > kacheong.poon@sun.com > > ------------=_1071092509-17947-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 10 16:45:53 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA20047 for ; Wed, 10 Dec 2003 16:45:53 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUC9a-0002nH-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:45:54 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUC9Z-0002mw-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:45:53 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-5.cisco.com with ESMTP; 10 Dec 2003 21:46:34 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBALj8At000446; Wed, 10 Dec 2003 13:45:08 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBALiuKs017992 for sctp-impl-filtered; Wed, 10 Dec 2003 13:44:58 -0800 (PST) Message-Id: <200312102144.hBALiuKs017992@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071092695-17990-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 10 Dec 2003 22:03:06 +0100 This is a multi-part message in MIME format... ------------=_1071092695-17990-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, see my comments below. Best regards Michael On Dec 10, 2003, at 4:33 PM, Kacheong Poon wrote: > Andreas Jungmaier wrote: > >> in cases where there is only an outgoing stream, but no incoming >> stream, applications can only send, not receive >> on a "stream-socket". >> In the opposite case, an application could only receive, >> but not send. > > Can you elaborate on the above? I don't quite get the issue. > If there is only one stream, I guess an app will not do > a sctp_peeloff(). A stream-socket is still bi-directional. > The only difference is that it is a socket which only > deals with one stream of an association. The original socket > deals with all the other streams of an association except > those which have been sctp_peeloff(). > So I assume that the fds will be used by different processing entities (processes / threads). What do you gain? How do you handle the receiver window problem? We do not have per stream flow control. >> I don't know if that's a problem, since that can happen >> as well with some streams within an SCTP association. >> However, I don't really like this idea. > > -- > > K. Poon. > kacheong.poon@sun.com > > ------------=_1071092695-17990-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 10 16:51:24 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA20363 for ; Wed, 10 Dec 2003 16:51:24 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUCEv-0002wK-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:51:25 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUCEu-0002tV-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:51:24 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-3.cisco.com with ESMTP; 10 Dec 2003 13:53:06 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBALoeZ0022289; Wed, 10 Dec 2003 13:50:40 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBALoXKs018073 for sctp-impl-filtered; Wed, 10 Dec 2003 13:50:35 -0800 (PST) Message-Id: <200312102150.hBALoXKs018073@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071093033-18065-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Xiaodan Tang From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 10 Dec 2003 22:09:44 +0100 This is a multi-part message in MIME format... ------------=_1071093033-18065-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary The is the same problem: You call a blocking sctp_recvmsg for stream 0. You get/have a lot of messages for stream 1 which fill up the receiver window. Now you are in a deadlock situation. The only way is on the receiving side accept the user data with the lowest TSN involved. Best regards Michael On Dec 10, 2003, at 5:56 PM, Xiaodan Tang wrote: > Instead of "peel_off()" a stream, how about allow sctp_rcvmsg() to be > able to > receive from a perticular steam_id ? As of now, we can sctp_sendmsg() > to a > specefic stream, but not receive on it. > > -xtang > > Andreas Jungmaier wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hi folks, >> >> to add up to the discussion: >> in cases where there is only an outgoing stream, but no incoming >> stream, applications can only send, not receive >> on a "stream-socket". >> In the opposite case, an application could only receive, >> but not send. >> >> I don't know if that's a problem, since that can happen >> as well with some streams within an SCTP association. >> >> However, I don't really like this idea. >> >> Best regards, >> Andreas >> >> On Wednesday 10 December 2003 11:07, Kacheong Poon wrote: >> >>> Randall Stewart (cisco) wrote: >>> >>>> Michael: >>>> >>>> Interesting point... but is not the problem NOT >>>> that the cum-TSN cannot be forwarded... you can >>>> always ack the data even though the receiving stream >>>> is holding off reading.. but the issue is more how >>>> does one manage the rwnd in this situation... I think >>>> this is where you are going... If socket a-stream-1 stops >>>> reading and socket a-stream-2 is reading.. then it is possible >>>> that a-stream-1 will hold all of the rwnd and thus block the >>>> association... Since we have no way to apportion the rwnd per >>>> stream this is a problem.. >>>> >>> This can be an issue. If an app chooses to peel off a >>> stream, it has to be careful in handling the individual >>> stream-socket to avoid blocking of other streams. But >>> this also opens up the socket interface for people to play >>> with streams scheduling. >>> >>> Any idea on how to make it safer? >>> >>> >>>> Michael Tuexen wrote: >>>> >>>>> Hi all, >>>>> >>>>> i think having a fd for a stream is a real problem. If the >>>>> user 'decides' not to read from a particular stream because >>>>> there is a separate thread/process for it which is blocked >>>>> the receiver can not forward the cumulative TSN and therefore >>>>> blocks the whole association. >>>>> >>>>> The receiver should always return in the receive call the >>>>> (part of) a DATA chunk with lowest TSN. >>>>> >>>>> Best regards >>>>> Michael >>>>> >>>>> On Dec 9, 2003, at 12:36 PM, Randall Stewart (cisco) wrote: >>>>> >>>>>> Kacheong Poon wrote: >>>>>> >>>>>>> I'd like to gather opinions about making sctp_peeloff() >>>>>>> work in one-one style SCTP socket. The semantics is >>>>>>> creating a new socket for a particular stream of an >>>>>>> association. The stream number will be passed in the >>>>>>> place of the assoc_id in the current sctp_peeloff(). >>>>>>> >>>>>>> Do people think that their apps can benefit from this? >>>>>>> Thanks. >>>>>>> >>>>>> Hmm... There does seem to be some interest (at least >>>>>> I have been contacted privately several times) on making >>>>>> it so that you can read from a particular stream... >>>>>> >>>>>> I don't think I would be opposed to this... it will be a bit >>>>>> of code for me, but has you have said .. its just code... and >>>>>> it may well be quite useful... >>>>>> >>>>>> Let's see what other developers think... I could go for it :-> >>>>>> >>>>>> R >>>>>> >> >> - -- Dipl.-Ing. Andreas Jungmaier Computer Networking >> Technology Group University of Duisburg-Essen >> http://www.exp-math.uni-essen.de/~ajung PGP Key-ID: D382 4AC0 >> PGP Key-Fingerprint: 228C 7C2C 3381 2DD4 9998 2812 5C0B 0B04 D382 >> 4AC0 >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.2.3 (GNU/Linux) >> >> iD8DBQE/1vLKXAsLBNOCSsARAjVDAKDRlPy3Hkg8KUFbu31DjUFJ5ca1sQCglS3Z >> P5d8fnU5Jq3z+E+nB0Fn4IM= >> =/AL3 >> -----END PGP SIGNATURE----- >> >> > ------------=_1071093033-18065-0-- From Michael.Tuexen@micmac.franken.de Wed Dec 10 16:55:23 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA20518 for ; Wed, 10 Dec 2003 16:55:22 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUCIl-00030e-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:55:23 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUCIl-0002yo-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 16:55:23 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBALscZ0025809; Wed, 10 Dec 2003 13:54:39 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBALsRKs018124 for sctp-impl-filtered; Wed, 10 Dec 2003 13:54:29 -0800 (PST) Message-Id: <200312102154.hBALsRKs018124@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071093267-18122-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: Michael Tuexen Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" , Kacheong Poon X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Wed, 10 Dec 2003 22:13:22 +0100 This is a multi-part message in MIME format... ------------=_1071093267-18122-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin, see my comments below. Best regards Michael On Dec 9, 2003, at 10:51 PM, Caitlin Bestler wrote: > > Michael Tuexen wrote: > >> Hi all, >> >> i think having a fd for a stream is a real problem. If the >> user 'decides' not to read from a particular stream because >> there is a separate thread/process for it which is blocked >> the receiver can not forward the cumulative TSN and therefore >> blocks the whole association. >> >> The receiver should always return in the receive call the >> (part of) a DATA chunk with lowest TSN. >> > > > That is one more strong point on the "gotchas" of attempting > to do per-stream processing with a synchronous API. > > I still think that per-stream buffering is very useful, but > it needs to be done in a way where the failure to provide a > buffer for a given stream prevents a Data Chunk from being > delivered to the application. > You can do per stream buffering on the sender side easily... > With RDMA style interfaces the solution to this problem is > very simple, streams MUST pre-post sufficient buffers. Failure > to do so will result in the stream being reset (which allows > the TSN to advance). What does reset a stream mean in RFC 2960? > > But that is inconsistent with socket semantics, where data > is held "somewhere" until the application asks for it. > This is how it is done in the socketapi currently. > Given that applications SHOULD take delivery of all data > within an association promptly then having "per stream" > sockets is indeed dangerous. That suggests that stream > specific processing is best achieved through a callback > or RDMA style interface. > > > -- > Caitlin Bestler - cait@asomi.com - http://asomi.com/ > http://asomi.com/CaitlinBestlerPublicPgpKey.html > ------------=_1071093267-18122-0-- From xtang@qnx.com Wed Dec 10 17:24:35 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21727 for ; Wed, 10 Dec 2003 17:24:35 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUCl2-0003bX-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:24:36 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AUCl1-0003bO-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:24:35 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBAMNPiN014096; Wed, 10 Dec 2003 14:23:25 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAMNHKs018493 for sctp-impl-filtered; Wed, 10 Dec 2003 14:23:19 -0800 (PST) Message-Id: <200312102223.hBAMNHKs018493@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071094997-18491-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Xiaodan Tang Cc: sctp-impl@external.cisco.com X-Nails-Approved: xtang@qnx.com Date: Wed, 10 Dec 2003 16:44:50 -0500 This is a multi-part message in MIME format... ------------=_1071094997-18491-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary I would expect this could be an application error. It should always have other threads waiting on other stream. If you are single thread application, you should never call stream specific sctp_recvmsg()? Regards -Xiaodan Michael Tuexen wrote: >The is the same problem: You call a blocking sctp_recvmsg for stream 0. >You get/have a lot of messages for stream 1 which fill up the receiver >window. Now you are in a deadlock situation. >The only way is on the receiving side accept the user data with the >lowest TSN involved. > >Best regards >Michael > >On Dec 10, 2003, at 5:56 PM, Xiaodan Tang wrote: > > > >>Instead of "peel_off()" a stream, how about allow sctp_rcvmsg() to be >>able to >>receive from a perticular steam_id ? As of now, we can sctp_sendmsg() >> >> > > > >>to a >>specefic stream, but not receive on it. >> >>-xtang >> >> ------------=_1071094997-18491-0-- From cait@asomi.com Wed Dec 10 17:45:25 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA22655 for ; Wed, 10 Dec 2003 17:45:23 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUD5A-0003xK-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:45:24 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUD5A-0003x1-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:45:24 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-5.cisco.com with ESMTP; 10 Dec 2003 22:41:11 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBAMdhZ0016626; Wed, 10 Dec 2003 14:39:43 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAMdTKs018705 for sctp-impl-filtered; Wed, 10 Dec 2003 14:39:31 -0800 (PST) Message-Id: <200312102239.hBAMdTKs018705@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071095969-18703-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Caitlin Bestler Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" , Kacheong Poon X-Nails-Approved: cait@asomi.com Date: Wed, 10 Dec 2003 16:36:07 -0600 This is a multi-part message in MIME format... ------------=_1071095969-18703-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 10, 2003, at 3:13 PM, Michael Tuexen wrote: >> With RDMA style interfaces the solution to this problem is >> very simple, streams MUST pre-post sufficient buffers. Failure >> to do so will result in the stream being reset (which allows >> the TSN to advance). > What does reset a stream mean in RFC 2960? A "stream reset" occurs in the layer above RFC 2960 providing local API services. The fact that a stream is "reset" means that all incoming chunks for that stream will be promptly placed in the bit bucket (for inbound) and no new chunks may be generated (for outbound). For PR-SCTP associations it probably also means abandoning any pending outbound chunks. In socket terms I guess that would mean that the socket associated with the stream(s) was in an error state. For the RDDP mapping to SCTP it means that the stream pair is not associated with an RDMA endpoint: hence all input is dropped and no output can be generated. Those semantics could be adopted to any QP/CQ style interface, whether or not RDMA capabilities were supported. In any case, the fact that a stream is "reset" would have no meaning to the software layers dealing with SCTP itself (as opposed to RDDP and/or sockets). -- Caitlin Bestler - cait@asomi.com - http://asomi.com/ http://asomi.com/CaitlinBestlerPublicPgpKey.html ------------=_1071095969-18703-0-- From cait@asomi.com Wed Dec 10 17:50:27 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA22839 for ; Wed, 10 Dec 2003 17:50:27 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUDA4-000434-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:50:28 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUDA3-00042W-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 17:50:27 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBAMlEZ0023279; Wed, 10 Dec 2003 14:47:15 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBAMl3Ks018807 for sctp-impl-filtered; Wed, 10 Dec 2003 14:47:05 -0800 (PST) Message-Id: <200312102247.hBAMl3Ks018807@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071096423-18805-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Caitlin Bestler Cc: sctp-impl@external.cisco.com, Kacheong Poon X-Nails-Approved: cait@asomi.com Date: Wed, 10 Dec 2003 16:45:10 -0600 This is a multi-part message in MIME format... ------------=_1071096423-18805-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 10, 2003, at 2:56 PM, Michael Tuexen wrote: > Dear all, > > the idea on which streams are based on is to minimize > the HOL problem. For the receiver it should not be important > on which stream a message was received, it is only > important that the receiver receives the messages in an > order appropriate for the upper layer. > > I disagree. Each stream can have a very distinct application specific meaning. It might be an MPEG Elementary Stream, one file transfer in an HTTP or FTP session, an audio stream as opposed to a video stream (non-MPEG), etc. In all of these cases, having to receive the message into a holding buffer and *then* put it into the "proper" buffer would be extra work for the application. In many cases successive messages on a given stream will be expected to be placed in successive buffers on the receiver, for example. If I have been asked to transfer a 20 MB file on one stream I do not want to have to read it all into my own memory just so I can send it as a single message. However the receive may well have a single 20 MB buffer waiting for the file. A lot of this depends on whether you view each message as being primarily a "transaction" or whether it might just as easily be a checkpointed data transfer. For control/transaction oriented messages I would tend to agree with you that the receiver should be processing the message with the lowest TSN first. But I don't think that is necessarily a universal rule. -- Caitlin Bestler - cait@asomi.com - http://asomi.com/ http://asomi.com/CaitlinBestlerPublicPgpKey.html ------------=_1071096423-18805-0-- From rrs@cisco.com Wed Dec 10 18:17:33 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA25291 for ; Wed, 10 Dec 2003 18:17:33 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUDaI-0004hO-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 18:17:34 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUDaI-0004go-00 for sctp-impl-archive@ietf.org; Wed, 10 Dec 2003 18:17:34 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBANGpAt003289; Wed, 10 Dec 2003 15:16:51 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBANGaKs019190 for sctp-impl-filtered; Wed, 10 Dec 2003 15:16:38 -0800 (PST) Message-Id: <200312102316.hBANGaKs019190@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071098196-19188-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Xiaodan Tang From: "Randall Stewart (cisco)" Cc: Michael Tuexen , sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Wed, 10 Dec 2003 17:16:43 -0600 This is a multi-part message in MIME format... ------------=_1071098196-19188-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Xiaodan Tang wrote: > I would expect this could be an application error. It should always > have other threads waiting on other stream. If you are single thread > application, you should never call > stream specific sctp_recvmsg()? Ok, I write a multi-threaded app.. and somewhere along the line a sender is pumping out messages faster than a thread wants to read the messages.. the other threads want the data at a faster rate... the slow one will block the faster consumers.. this is the crux of the problem.. and your restrictions above do not solve it. R > > > Regards > -Xiaodan > > Michael Tuexen wrote: > >> The is the same problem: You call a blocking sctp_recvmsg for stream 0. >> You get/have a lot of messages for stream 1 which fill up the receiver >> window. Now you are in a deadlock situation. >> The only way is on the receiving side accept the user data with the >> lowest TSN involved. >> >> Best regards >> Michael >> >> On Dec 10, 2003, at 5:56 PM, Xiaodan Tang wrote: >> >> >> >>> Instead of "peel_off()" a stream, how about allow sctp_rcvmsg() to >>> be able to >>> receive from a perticular steam_id ? As of now, we can sctp_sendmsg() >>> >> >> >> >> >>> to a >>> specefic stream, but not receive on it. >>> >>> -xtang >>> >> > > ------------=_1071098196-19188-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 07:53:49 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA28315 for ; Thu, 11 Dec 2003 07:53:48 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUQKD-0002V9-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 07:53:49 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUQKD-0002Ts-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 07:53:49 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 12:54:33 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBBCqciN028648; Thu, 11 Dec 2003 04:52:38 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBCq2Ks029915 for sctp-impl-filtered; Thu, 11 Dec 2003 04:52:04 -0800 (PST) Message-Id: <200312111252.hBBCq2Ks029915@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071147122-29913-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: xtang@qnx.com From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 13:23:11 +0100 This is a multi-part message in MIME format... ------------=_1071147122-29913-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Hi Xiaoan, even if the other thread reads the messages for stream 1 the rwnd can not be grown. So you are in the deadlock situation even in the case were almost all messages are read. Best regards Michael On Dec 10, 2003, at 10:44 PM, Xiaodan Tang wrote: > I would expect this could be an application error. It should always > have other threads waiting on other stream. If you are single thread > application, you should never call > stream specific sctp_recvmsg()? > > Regards > -Xiaodan > > Michael Tuexen wrote: > >> The is the same problem: You call a blocking sctp_recvmsg for stream >> 0. >> You get/have a lot of messages for stream 1 which fill up the receiver >> window. Now you are in a deadlock situation. >> The only way is on the receiving side accept the user data with the >> lowest TSN involved. >> >> Best regards >> Michael >> >> On Dec 10, 2003, at 5:56 PM, Xiaodan Tang wrote: >> >> >>> Instead of "peel_off()" a stream, how about allow sctp_rcvmsg() to >>> be able to >>> receive from a perticular steam_id ? As of now, we can >>> sctp_sendmsg() >>> >> >> >>> to a >>> specefic stream, but not receive on it. >>> >>> -xtang >>> > > ------------=_1071147122-29913-0-- From rrs@cisco.com Thu Dec 11 07:55:31 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA28403 for ; Thu, 11 Dec 2003 07:55:31 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUQLs-0002XX-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 07:55:32 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUQLr-0002Wf-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 07:55:31 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 12:56:17 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBBCsVAt013654; Thu, 11 Dec 2003 04:54:32 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBCsCKs029954 for sctp-impl-filtered; Thu, 11 Dec 2003 04:54:14 -0800 (PST) Message-Id: <200312111254.hBBCsCKs029954@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071147251-29952-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: SV: SV: Question regarding the SCTP-kame implementation List-Id: sctp-impl To: =?ISO-8859-1?Q?Torbj=F6rn_Andersson?= From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Thu, 11 Dec 2003 06:54:17 -0600 This is a multi-part message in MIME format... ------------=_1071147251-29952-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Torbjörn Andersson wrote: >I append a file to this message, which contain the results of an >experiment I have performed. >The experiment measured individual ULP message times, which include the >time of getting through the send buffer, router buffer (12 segements) >and network (25ms). Further retransmissions and reordering at the >receiver can delay the messages. The experiment does not have any >background traffic on the test network. > >Each message contains 500 bytes of data and they are sent as fast as >possible. A 1-1 SCTP socket is used at the sender and the socket is >configured as blocking (default). > >Now, one can see in the figure that the message times increase >differently in four different phases. I was actually expecting one maybe >two different phases until the buffer of the SCTP association becomes >dominant and everything levels out. >But it does not look like that and I believed first that there was some >intelligent buffer management at the sender. But the send buffer is >statically configured. >The next thing I though could explain the different message increase >speeds was the blocking algorithm. > >Does anyone have any input to this? > I am not sure what would cause a 4 stage change.. I would expect two stages like you...hmmm could loss have entered in here.. that would cause a jump down from slow start to ca... Might I also make one suggestion... If you enable SCTP_STAT_LOGGING you can also enable: SCTP_CWND_LOGGING SCTP_BLK_LOGGING SCTP_FR_LOGGING SCTP_MAP_LOGGING (note you must enable stat logging to get any of the other ones to compile )... So.. what does this gain you? CWND_LOGGING will log the congestion window events BLK_LOGGING will log when you block due to buffer full conditions FR_LOGGING will log retransmissions (I think also timeouts) MAP_LOGGING is probably least interesting .. it logs mapping array movement .. the receiver side bit map manipulations. Now You can enable one or any combination of the logging.. either manually add it to opt_sctp.h or I think the changes have gone in with the last kame patch that let you add it directly to the config file... i.e. options SCTP_STAT_LOGGING options SCTP_CWND_LOGGING will turn on the cwnd logging... if you want to do it in the opt_sctp.h file #define SCTP_STAT_LOGGING 1 #define SCTP_CWND_LOGGING 1 . The log defaults to 80,000 events. You can expand this larger if that is not enough for the transfer you are doing by changing SCTP_STAT_LOG_SIZE in sctp_constants.h If you are interested in doing this let me know ... I have a log pulling/printing set of utilities I can send you that can then pull and display the log information. This logging is quite low overhead being a array of ints that gets manipulated to generate the log... I have used it for debugging quite effectively since doing a SCTP_DEBUG generates a lot more information and slows things down... Let me know if you want the utilities... R > > > >>-----Ursprungligt meddelande----- >>Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] >>Skickat: den 10 december 2003 16:19 >>Till: Torbjörn Andersson >>Kopia: sctp-impl@external.cisco.com >>Ämne: Re: SV: Question regarding the SCTP-kame implementation >> >>It depends on a couple of things you need to answer for me... >> >>1) Do you have blocking IO on? FNBIO I think is the socket opt >>2) What model are you using 1-1 or 1-M? >> >>Generally the 1-1 model will block at connect()/accept() and >>when the send buffer is full... if blocking IO is on (I think the >>default)... If you do non-blocking IO then thats a different >>story.. oh and one of the bugs that is still in patch prepartion as >>to do with blocking io.. there is a problem in the 1-1 model >>when you do non-blocking io.. connect will fail.. I fixed this >>a while ago but just kept forgetting to get the right >>cvs diffs to kame.. its in the in_proto.c and in6_proto.c files >>(the 1-1 model needs PR_CONNREQUIRED) >> >> >>R >> >>Torbjörn Andersson wrote: >> >> >> >>>Thanks for your quick answer. Your answer has moved my focus from the >>>send buffer to the blocking behavior of the socket. >>> >>>When does SCTP-block? >>> Is it dependent of the state of the association? >>> >>> >>> >>> >>> >>> >>> >>>>-----Ursprungligt meddelande----- >>>>Från: Randall Stewart (cisco) [mailto:rrs@cisco.com] >>>>Skickat: den 10 december 2003 13:35 >>>>Till: Torbjörn Andersson >>>>Kopia: sctp-impl@external.cisco.com >>>>Ämne: Re: Question regarding the SCTP-kame implementation >>>> >>>>Torbjorn: >>>> >>>>Yes, there are a couple of us that know... in fact I am >>>>behind on getting a couple of patches in too :-< >>>> >>>>But as to your questions... >>>> >>>>Torbjörn Andersson wrote: >>>> >>>> >>>> >>>> >>>> >>>>>I'm testing the SCTP-kame implementation and I have some specific >>>>>questions about the SCTP send buffer in Kame. I want to know how >>>>> >>>>> >the > > >>>>>send buffer behaves. The reason to why I want to know this is that >>>>> >>>>> >>>>> >>>>> >>>I'm >>> >>> >>> >>> >>>>>studying the total transmission time for individual messages. I >>>>> >>>>> >guess > > >>>>>someone reading this mailing list know this. >>>>> >>>>>How big is it? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>The current send buffer and receive window are >>>>defaulted to the maximum socket buffer minus a bit... the >>>>actual code is in sctp_init() in sctp_usrreq.c and the lines >>>>look like: >>>> >>>> sb_max_adj = (u_long)((u_quad_t)(SB_MAX) * MCLBYTES / (MSIZE + >>>>MCLBYTES)); >>>> sctp_sendspace = min((min(SB_MAX, sb_max_adj)), >>>> ((nmbclusters/2) * SCTP_DEFAULT_MAXSEGMENT)); >>>> /* Now for the recv window, should we take the same >>>> * amount? or should I do 1/2 the SB_MAX instead in the SB_MAX >>>> >>>> >min > > >>>> * above. For now I will just copy. >>>> */ >>>> sctp_recvspace = sctp_sendspace; >>>> >>>>SB_MAX is set in sys/socketvar.h >>>> >>>>In order to get real high performance (GigE type speed) you will >>>> >>>> >need > > >>>>to set this up higher from the shipped default (256*1024). My >>>> >>>> >machine > > >>>>that I do high bandwidth testing on has its set to (2048 * 1024) >>>> >>>>You also can change this by doing sysctl since both the sendspace >>>> >>>> >and > > >>>>recvspace >>>>are sysctl variables (at least in freebsd). >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Is it tuned to the cwnd? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>cwnd is tied to the network and may get limited by your send >>>>buffer and peers rwnd but it will never influence the value >>>>of your send space. >>>> >>>> >>>> >>>> >>>> >>>>>Is the size of the send buffer dependent of the congestion state, >>>>>slowstart/fast recovery/congestion avoidance etc? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>Nope... these are unrelated items .. cwnd deals with >>>>the associations view of the network, sendspace deals >>>>with how much outstanding data a app can write down >>>>the socket buffer... >>>> >>>>You can also change the value down or up by doing a >>>>setsockopt() on the SND_BUF... >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Is it chunk or byte limited? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>There is also an additional limit placed on an implementation >>>>as well.. number of chunks... This is tuned so that if you >>>>do a WHOLE bunch of 1 byte sends for example: >>>> >>>>while(1) >>>> send(1 byte); >>>> >>>>You will block when you hit a threshold of chunks.. otherwise >>>>you would quickly exhaust the number of mbufs in the system >>>>which will have dire consequences... >>>> >>>>This value is setup to either be the max of >>>>(SCTP_ASOC_MAX_CHUNKS_ON_QUEUE, nmbclusters) >>>> >>>>nmbclusters is tuneable in the config file.. .while the chunks on >>>> >>>> >>>> >>>> >>>queue >>> >>> >>> >>> >>>>is a hardcoded #define currently set to 512 >>>> >>>>Hope that all helps.. >>>> >>>>R >>>> >>>> >>>> >>>> >>>> >>>>>Best Regards, Torbjörn >>>>> >>>>>-------------------------------------------------------------- >>>>>| Torbjörn Andersson phone: +46 54 700 11 61 >>>>>| Computer Science fax: +46 54 700 18 28 >>>>>| Karlstad University URL: http://www.cs.kau.se/~tora >>>>>--------------------------------------------------------------- >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> >>> >>> >>> > > > > ------------=_1071147251-29952-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 08:14:19 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA29050 for ; Thu, 11 Dec 2003 08:14:18 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUQe3-0002zM-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 08:14:19 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUQdt-0002yZ-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 08:14:09 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-2.cisco.com with ESMTP; 11 Dec 2003 05:15:37 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBBDDJAt023556; Thu, 11 Dec 2003 05:13:20 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBDD1Ks000199 for sctp-impl-filtered; Thu, 11 Dec 2003 05:13:03 -0800 (PST) Message-Id: <200312111313.hBBDD1Ks000199@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071148381-197-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Kacheong Poon X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 13:32:17 +0100 This is a multi-part message in MIME format... ------------=_1071148381-197-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin, I see what you mean, but SCTP was not designed to do this, that seems be to the reason why does not fit now. What you really would need is a receiver buffer management on a per stream base. This comes down to a flow control on a per stream base. This would allow what you want to do: call receive on one stream providing 20MB for a large file and call receive on another stream with 1KB for a short message and do on. But I do know how well this fits in OSes were you still need to copy from kernel to userland. On most OSes you have the partial delivery API to handle large user messages. But my main point is: for supporting different receive buffers for different streams you also need different flow controls for different streams. Do you agree with this statement? Then there is the next problem: Can we extend SCTP (or even do we need to extend) to provide a per stream flow control. Some guys thought about this already and the result was, that it is not possible to do as an extension. Of course, SCTP could have been designed to provide that feature. But it was not done that way. Best regards Michael On Dec 10, 2003, at 11:45 PM, Caitlin Bestler wrote: > > On Dec 10, 2003, at 2:56 PM, Michael Tuexen wrote: > >> Dear all, >> >> the idea on which streams are based on is to minimize >> the HOL problem. For the receiver it should not be important >> on which stream a message was received, it is only >> important that the receiver receives the messages in an >> order appropriate for the upper layer. >> >> > I disagree. Each stream can have a very distinct application > specific meaning. It might be an MPEG Elementary Stream, > one file transfer in an HTTP or FTP session, an audio > stream as opposed to a video stream (non-MPEG), etc. > > In all of these cases, having to receive the message > into a holding buffer and *then* put it into the "proper" > buffer would be extra work for the application. > > In many cases successive messages on a given stream will > be expected to be placed in successive buffers on the > receiver, for example. If I have been asked to transfer > a 20 MB file on one stream I do not want to have to read > it all into my own memory just so I can send it as a single > message. However the receive may well have a single 20 MB > buffer waiting for the file. > > A lot of this depends on whether you view each message > as being primarily a "transaction" or whether it might > just as easily be a checkpointed data transfer. For > control/transaction oriented messages I would tend to > agree with you that the receiver should be processing > the message with the lowest TSN first. But I don't > think that is necessarily a universal rule. > > -- > Caitlin Bestler - cait@asomi.com - http://asomi.com/ > http://asomi.com/CaitlinBestlerPublicPgpKey.html > > ------------=_1071148381-197-0-- From thomas.brown@tekelec.com Thu Dec 11 09:39:28 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02132 for ; Thu, 11 Dec 2003 09:39:27 -0500 (EST) From: thomas.brown@tekelec.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AURyS-00056L-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 09:39:28 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AURyS-000566-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 09:39:28 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 11 Dec 2003 06:40:57 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBBEccZ0028678; Thu, 11 Dec 2003 06:38:38 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBEcJKs001374 for sctp-impl-filtered; Thu, 11 Dec 2003 06:38:21 -0800 (PST) Message-Id: <200312111438.hBBEcJKs001374@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071153499-1372-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: SCTP Compliance Testing Tools List-Id: sctp-impl To: sctp-impl@external.cisco.com X-Nails-Approved: thomas.brown@tekelec.com Date: Thu, 11 Dec 2003 08:05:43 -0500 This is a multi-part message in MIME format... ------------=_1071153499-1372-0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: binary We are looking for software tools to verify that an SCTP protocol stack is compliant to the SCTP RFC. Hopefully if such a tool is available it would support both an interactive and automated interface for testing. We are interested in learning about both commercially available solutions and also tools from other sources. If anyone can provide information on this topic it would be greatly appreciated. ------------=_1071153499-1372-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 10:39:06 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA07177 for ; Thu, 11 Dec 2003 10:39:06 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUSuB-0006vS-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 10:39:07 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUSuB-0006uZ-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 10:39:07 -0500 Received: from sj-core-3.cisco.com (171.68.223.137) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 15:39:56 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBBFc8rX018484; Thu, 11 Dec 2003 07:38:09 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBFbrKs002155 for sctp-impl-filtered; Thu, 11 Dec 2003 07:37:55 -0800 (PST) Message-Id: <200312111537.hBBFbrKs002155@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071157072-2153-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: SCTP Compliance Testing Tools List-Id: sctp-impl To: thomas.brown@tekelec.com From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 15:56:25 +0100 This is a multi-part message in MIME format... ------------=_1071157072-2153-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Hi Thomas, I have released an SCTP packet generator which allows you to send and receive SCTP packets. It is programmable in scheme, a lisp dialect. There are also several testcases I have written, but these are not released (yet). The testtool can be downloaded from http://www.sctp.de. Best regards Michael On Dec 11, 2003, at 2:05 PM, thomas.brown@tekelec.com wrote: > > We are looking for software tools to verify that an SCTP protocol > stack is > compliant to the SCTP RFC. Hopefully if such a tool is available it > would > support both an interactive and automated interface for testing. We > are > interested in learning about both commercially available solutions and > also > tools from other sources. > > If anyone can provide information on this topic it would be greatly > appreciated. > > ------------=_1071157072-2153-0-- From kacheong.poon@sun.com Thu Dec 11 11:25:40 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08483 for ; Thu, 11 Dec 2003 11:25:40 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUTdF-0007mS-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:25:41 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUTdF-0007m7-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:25:41 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBBGOmBN009091; Thu, 11 Dec 2003 08:24:48 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBGOYKs002743 for sctp-impl-filtered; Thu, 11 Dec 2003 08:24:36 -0800 (PST) Message-Id: <200312111624.hBBGOYKs002743@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071159873-2741-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Thu, 11 Dec 2003 08:22:48 -0800 This is a multi-part message in MIME format... ------------=_1071159873-2741-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > the idea on which streams are based on is to minimize > the HOL problem. For the receiver it should not be important > on which stream a message was received, it is only > important that the receiver receives the messages in an > order appropriate for the upper layer. I guess the point of having a socket for individual stream is not related to what SCTP the protocol tries to do. It is more about how SCTP apps want to use the API. So it seems to me that the dicussion should be focused on whether app programmers like to have such as interface. Then the second question is what kind of caveats this interface may have. Also note that the stream-socket can both send and receive. We should also consider the usefulness of having stream-socket for sending. The point you raised is that if the app does not behave, one stream can block the other in receive. This is a caveat. > That is the reason why there is no per stream flow control. > If you need this you can setup separate associations. Note that flow control is not the question here. The new API is not an attempt to change the protocol. It is not about an app tries to do flow control on a per stream basis. > Regarding stream scheduling: > The point is that the receiver always provides the packet > with the lowest TSN to the user. This opens the rwnd. > The scheduling takes place at the sender. The sender can > assign the TSN at a very late point and can therefore > provide a fair usage of the association between the streams. > the fairness depends on the scheduling function. This is > useful (some think necessary) for usage in the SS7 environment. > I'm working on a paper describing that stuff. From the app's point of view, having different sockets for different streams allow them to separate task handling easier. It has nothing to do with how SCTP assigns TSNs. The way an app will see this is that it just needs to setup one association but can have multiple data streams controlled individually. This is "like" opening multiple TCP connections. Now whether SCTP can provide identical semantics is another issue, we know that it cannot because of the issue rasied. But is it good enough so that a well bahaved app can make use of it. This is the question we need to ask and answer. What you described above is really in the SCTP protocol level. What the stream scheduling I mentioned was about how an app could schedule different threads in handling send/receive of different streams. I think they are separate issues. -- K. Poon. kacheong.poon@sun.com ------------=_1071159873-2741-0-- From kacheong.poon@sun.com Thu Dec 11 11:27:26 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08541 for ; Thu, 11 Dec 2003 11:27:26 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUTev-00000n-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:27:25 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUTeu-00000e-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:27:24 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-2.cisco.com with ESMTP; 11 Dec 2003 08:28:54 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBBGQbAt011522; Thu, 11 Dec 2003 08:26:37 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBGQPKs002774 for sctp-impl-filtered; Thu, 11 Dec 2003 08:26:27 -0800 (PST) Message-Id: <200312111626.hBBGQPKs002774@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071159985-2772-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Thu, 11 Dec 2003 08:25:05 -0800 This is a multi-part message in MIME format... ------------=_1071159985-2772-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > I think you are using the streams in a way they are (initially) > not intended for. The problem is that you have only a > per association receiver window, not a per stream. So > it is very likely that one processing entity will block > the other. It seems that we are really discussing two different but related issues. Your point of view is from the protocol level. But we are talking about an app which tries to do the right thing. And the question is whether it is useful to provide such an API for those apps to do their jobs. > On the other hand i do not see the usage of this. If you need a > demultiplexer of some sort you might want to do it above SCTP. The API is above SCTP. We are not talking about changing the protocol. -- K. Poon. kacheong.poon@sun.com ------------=_1071159985-2772-0-- From kacheong.poon@sun.com Thu Dec 11 11:33:06 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08746 for ; Thu, 11 Dec 2003 11:33:06 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUTkS-00006X-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:33:08 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUTkR-00005y-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:33:07 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 16:33:57 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBBGWBiN020061; Thu, 11 Dec 2003 08:32:11 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBGVxKs002856 for sctp-impl-filtered; Thu, 11 Dec 2003 08:32:01 -0800 (PST) Message-Id: <200312111632.hBBGVxKs002856@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071160318-2854-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Thu, 11 Dec 2003 08:29:35 -0800 This is a multi-part message in MIME format... ------------=_1071160318-2854-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > So I assume that the fds will be used by different processing > entities (processes / threads). What do you gain? How do you > handle the receiver window problem? We do not have per stream > flow control. The gain is the ability to use a socket, along with all the functionalities an OS provides to socket, to send/receive on a stream. The app using this has to behave. And that's the question I asked, is there a way to help an app to behave so that this stream-socket feature can be safer. Maybe other people can comment. -- K. Poon. kacheong.poon@sun.com ------------=_1071160318-2854-0-- From kacheong.poon@sun.com Thu Dec 11 11:44:41 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09110 for ; Thu, 11 Dec 2003 11:44:40 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUTve-0000Qi-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:44:42 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUTve-0000Q5-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 11:44:42 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-3.cisco.com with ESMTP; 11 Dec 2003 08:46:30 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBBGhsBN000086; Thu, 11 Dec 2003 08:43:55 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBGhgKs003006 for sctp-impl-filtered; Thu, 11 Dec 2003 08:43:44 -0800 (PST) Message-Id: <200312111643.hBBGhgKs003006@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071161022-3004-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: Caitlin Bestler , sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Thu, 11 Dec 2003 08:42:04 -0800 This is a multi-part message in MIME format... ------------=_1071161022-3004-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > But my main point is: for supporting different receive buffers > for different streams you also need different flow controls for > different streams. Do you agree with this statement? It really depends. I think what Caitlin's point is that an app does not have control over on putting data from one particular stream to one particular pool of buffers. For example, in the current socket API, an app gives a buffer the recvmsg(). It does not know which stream's data will be put into that buffer. With stream-socket, the app can control this as the buffer given to recvmsg() will only be filled with data from a particular stream. This has nothing to do with flow control. The flow control issue comes up at the protocol level. > Then there is the next problem: Can we extend SCTP (or even do we > need to extend) to provide a per stream flow control. Some guys > thought about this already and the result was, that it is not > possible to do as an extension. Of course, SCTP could have been > designed to provide that feature. But it was not done that way. Again, the API is not about changing SCTP. This needs to be emphasized. -- K. Poon. kacheong.poon@sun.com ------------=_1071161022-3004-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 15:36:39 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA21404 for ; Thu, 11 Dec 2003 15:36:39 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUXY8-0007WH-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:36:40 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AUXY7-0007Uh-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:36:40 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBBKZUrX006597; Thu, 11 Dec 2003 12:35:30 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBKZCKs005946 for sctp-impl-filtered; Thu, 11 Dec 2003 12:35:14 -0800 (PST) Message-Id: <200312112035.hBBKZCKs005946@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071174912-5944-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 20:54:02 +0100 This is a multi-part message in MIME format... ------------=_1071174912-5944-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, see my comments below. Best regards Michael On Dec 11, 2003, at 5:22 PM, Kacheong Poon wrote: > Michael Tuexen wrote: > >> the idea on which streams are based on is to minimize >> the HOL problem. For the receiver it should not be important >> on which stream a message was received, it is only >> important that the receiver receives the messages in an >> order appropriate for the upper layer. > > I guess the point of having a socket for individual stream > is not related to what SCTP the protocol tries to do. It > is more about how SCTP apps want to use the API. So it > seems to me that the dicussion should be focused on whether > app programmers like to have such as interface. Then the > second question is what kind of caveats this interface may > have. Also note that the stream-socket can both send and > receive. We should also consider the usefulness of having > stream-socket for sending. > Streams are not bidirectional. So how will you send? > The point you raised is that if the app does not behave, > one stream can block the other in receive. This is a caveat. > >> That is the reason why there is no per stream flow control. >> If you need this you can setup separate associations. > > Note that flow control is not the question here. The new > API is not an attempt to change the protocol. It is not > about an app tries to do flow control on a per stream basis. > >> Regarding stream scheduling: >> The point is that the receiver always provides the packet >> with the lowest TSN to the user. This opens the rwnd. >> The scheduling takes place at the sender. The sender can >> assign the TSN at a very late point and can therefore >> provide a fair usage of the association between the streams. >> the fairness depends on the scheduling function. This is >> useful (some think necessary) for usage in the SS7 environment. >> I'm working on a paper describing that stuff. > > From the app's point of view, having different sockets for > different streams allow them to separate task handling easier. > It has nothing to do with how SCTP assigns TSNs. The way an > app will see this is that it just needs to setup one association > but can have multiple data streams controlled individually. This > is "like" opening multiple TCP connections. Now whether SCTP can > provide identical semantics is another issue, we know that it > cannot because of the issue rasied. But is it good enough so that > a well bahaved app can make use of it. This is the question we > need to ask and answer. This is my point: people want something like multiple TCP connections in one entity and might think about about an association with n streams in each direction such they can use them a bidirectional streams. But the BIG difference is that the TCP connections have separate buffer management and the receive buffer for the SCTP association is shared. So my answer to this is: It is not good enough. > > What you described above is really in the SCTP protocol level. > What the stream scheduling I mentioned was about how an app could > schedule different threads in handling send/receive of different > streams. I think they are separate issues. This is really my point. When you write a multithreaded application you may want to block threads intentionally by using semaphores or other techniques. But the receive calls should be independent. And this is not the case. > > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071174912-5944-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 15:39:44 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA21891 for ; Thu, 11 Dec 2003 15:39:44 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUXb7-0007eZ-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:39:45 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUXb7-0007cy-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:39:45 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBBKcsAt007988; Thu, 11 Dec 2003 12:38:55 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBKccKs005995 for sctp-impl-filtered; Thu, 11 Dec 2003 12:38:40 -0800 (PST) Message-Id: <200312112038.hBBKccKs005995@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071175118-5993-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 20:57:11 +0100 This is a multi-part message in MIME format... ------------=_1071175118-5993-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, see my comments below. Best regards Michael On Dec 11, 2003, at 5:25 PM, Kacheong Poon wrote: > Michael Tuexen wrote: > >> I think you are using the streams in a way they are (initially) >> not intended for. The problem is that you have only a >> per association receiver window, not a per stream. So >> it is very likely that one processing entity will block >> the other. > > It seems that we are really discussing two different but > related issues. Your point of view is from the protocol > level. But we are talking about an app which tries to > do the right thing. And the question is whether it is > useful to provide such an API for those apps to do their > jobs. > What kind of function calls are you thinking about, what are the semantic and how will you map this to the protocol layer? My point is that you will promise independent receive calls which will result in my mapping to the protocol layer to a deadlock situation. But possibly there is another way map your functions to the protocol. >> On the other hand i do not see the usage of this. If you need a >> demultiplexer of some sort you might want to do it above SCTP. > > The API is above SCTP. We are not talking about changing > the protocol. > OK. > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071175118-5993-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 15:41:12 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA22062 for ; Thu, 11 Dec 2003 15:41:12 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUXcX-0007hU-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:41:13 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUXcX-0007ga-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:41:13 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-2.cisco.com with ESMTP; 11 Dec 2003 12:42:46 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBBKeMBN018296; Thu, 11 Dec 2003 12:40:23 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBKeDKs006022 for sctp-impl-filtered; Thu, 11 Dec 2003 12:40:15 -0800 (PST) Message-Id: <200312112040.hBBKeDKs006022@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071175213-6020-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 20:58:27 +0100 This is a multi-part message in MIME format... ------------=_1071175213-6020-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Could you provide a simple example how you would use this feature? Best regards Michael On Dec 11, 2003, at 5:29 PM, Kacheong Poon wrote: > Michael Tuexen wrote: > >> So I assume that the fds will be used by different processing >> entities (processes / threads). What do you gain? How do you >> handle the receiver window problem? We do not have per stream >> flow control. > > The gain is the ability to use a socket, along with all the > functionalities an OS provides to socket, to send/receive > on a stream. The app using this has to behave. And that's > the question I asked, is there a way to help an app to behave > so that this stream-socket feature can be safer. Maybe other > people can comment. > > > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071175213-6020-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 15:44:15 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA22207 for ; Thu, 11 Dec 2003 15:44:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUXfO-0007k9-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:44:11 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUXfO-0007jz-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 15:44:10 -0500 Received: from sj-core-3.cisco.com (171.68.223.137) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 20:45:03 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBBKhJrX012097; Thu, 11 Dec 2003 12:43:20 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBKhEKs006066 for sctp-impl-filtered; Thu, 11 Dec 2003 12:43:16 -0800 (PST) Message-Id: <200312112043.hBBKhEKs006066@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071175393-6064-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Caitlin Bestler X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 21:02:04 +0100 This is a multi-part message in MIME format... ------------=_1071175393-6064-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, Assuming that you can use recv_message_from_stream(stream, buffer) to get data from a particular stream, how will you handle the problem 'at the protocol level'? If you are not doing it right there you will have problems at the upper layer. Best regards Michael On Dec 11, 2003, at 5:42 PM, Kacheong Poon wrote: > Michael Tuexen wrote: > >> But my main point is: for supporting different receive buffers >> for different streams you also need different flow controls for >> different streams. Do you agree with this statement? > > It really depends. I think what Caitlin's point is that an app > does not have control over on putting data from one particular > stream to one particular pool of buffers. For example, in the current > socket API, an app gives a buffer the recvmsg(). It does not know > which stream's data will be put into that buffer. With stream-socket, > the app can control this as the buffer given to recvmsg() will > only be filled with data from a particular stream. This has nothing > to do with flow control. The flow control issue comes up at the > protocol level. > >> Then there is the next problem: Can we extend SCTP (or even do we >> need to extend) to provide a per stream flow control. Some guys >> thought about this already and the result was, that it is not >> possible to do as an extension. Of course, SCTP could have been >> designed to provide that feature. But it was not done that way. > > Again, the API is not about changing SCTP. This needs to be > emphasized. > > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071175393-6064-0-- From rrs@cisco.com Thu Dec 11 16:32:21 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA26397 for ; Thu, 11 Dec 2003 16:32:21 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUYQ2-0001ss-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 16:32:22 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUYQ1-0001sH-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 16:32:22 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBBLVTAt000808; Thu, 11 Dec 2003 13:31:30 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBLVLKs006684 for sctp-impl-filtered; Thu, 11 Dec 2003 13:31:23 -0800 (PST) Message-Id: <200312112131.hBBLVLKs006684@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071178281-6682-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: The poll response List-Id: sctp-impl To: SCTP Implementors From: "Randall Stewart (cisco)" X-Nails-Approved: rrs@cisco.com Date: Thu, 11 Dec 2003 15:31:23 -0600 This is a multi-part message in MIME format... ------------=_1071178281-6682-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Dear all: I have finally gotten to compiling all the responses I have received on implementations of SCTP. You can find it at: http://www.sctp.org Follow the implementations tab on the left and you should get my feeble first attempt at putting together a page listing the known implementations. If you know of something that is inacurate.. or missing please send me an email. Happy SCTPing R --- Randall Stewart ITD - Transport Technologies ------------=_1071178281-6682-0-- From cait@asomi.com Thu Dec 11 16:36:05 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA26649 for ; Thu, 11 Dec 2003 16:36:05 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUYTe-0001zd-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 16:36:06 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUYTe-0001ye-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 16:36:06 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-3.cisco.com with ESMTP; 11 Dec 2003 13:37:59 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBBLZBBN014747; Thu, 11 Dec 2003 13:35:12 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBLZ2Ks006741 for sctp-impl-filtered; Thu, 11 Dec 2003 13:35:04 -0800 (PST) Message-Id: <200312112135.hBBLZ2Ks006741@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071178501-6739-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: SCTP Implementors From: Caitlin Bestler Cc: Michael Tuexen , Kacheong Poon X-Nails-Approved: cait@asomi.com Date: Thu, 11 Dec 2003 15:24:27 -0600 This is a multi-part message in MIME format... ------------=_1071178501-6739-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote: > Michael Tuexen wrote: > >> But my main point is: for supporting different receive buffers >> for different streams you also need different flow controls for >> different streams. Do you agree with this statement? > > It really depends. I think what Caitlin's point is that an app > does not have control over on putting data from one particular > stream to one particular pool of buffers. For example, in the current > socket API, an app gives a buffer the recvmsg(). It does not know > which stream's data will be put into that buffer. With stream-socket, > the app can control this as the buffer given to recvmsg() will > only be filled with data from a particular stream. This has nothing > to do with flow control. The flow control issue comes up at the > protocol level. > >> Then there is the next problem: Can we extend SCTP (or even do we >> need to extend) to provide a per stream flow control. Some guys >> thought about this already and the result was, that it is not >> possible to do as an extension. Of course, SCTP could have been >> designed to provide that feature. But it was not done that way. > > Again, the API is not about changing SCTP. This needs to be > emphasized. Absolutely correct. The issues here have nothing to do with the end-to-end semantics SCTP, but rather with the semantics of the socket interface. Michael' key objection, as I understand it, is that any stream specific receive will lead to laggard messages that cannot be delivered. Fundamentally, the API layer is expected to deliver the message with the oldest TSN first because that will enable advancing the receive window. That is a valid concern. But you can achieve stream-specific buffering without causing the receive window to be tardy. RDDP's use of SCTP (for RDMA services) relies on per stream buffering for untagged messages and strictly upon ULP flow control. SCTP flow control is used solely to manage network congestion, and theoretically for SCTP layer buffering (although the RDDP adaptation of SCTP uses only unordered delivery, thereby enabling the SCTP layer to function without its own buffers. It can place directly from the IP layer buffers to the ULP buffers). Therefore the key conflict here is not between per-stream receive buffering and SCTP, but between per-stream receive buffering and normal socket semantics. Applications have gotten used to having the socket interface provide critical ULP flow control: pause receiver until message from peer arrives, pause transmitter until peer is ready to receive more messages. Inherently, the socket layer will hold any messages that it allowed to be sent that the ULP is not ready to receive. Michael is assuming that because SCTP does not have per- stream flow control that per stream buffering will not work. What is more accurate, is that it means that inferring per-stream flow control from the sockets interface will not work. To successfully provide per-stream buffering, an API must have the following rules: The receiver, through some mechanism, must have informed the sender of the maximum number of messages that can be in-flight. For most RDMA applications this is a pre-arranged or negotiated number of credits. Credits are reduced when a message is sent, and incremented when the receiver replies. For non-RDMA applications it would probably work to increase credits as prior messages were acked. When any authorized message is received, the API layer MUST be able to forward it to a ULP buffer immediately. This implies a callback that says "here, this messge is yours -- do something with it NOW", and/or pre-posting of receive buffers. With pre-posting, an unauthorized message MAY be discarded, placing the stream in a "terminated" state. Essentially, if the ULP fails to provide sufficient buffers the API layer would substitute the bit bucket as the destination. The key is that advance the receive window must never be held up because the application failed to ask for buffer on a specific stream. I do not see how you can extend the socket paradigm to match these rules, but APIs based upon callbacks and/or RQ/CQs would handle this easily. Obviously all streams within an association should be related. They are sharing the association's bandwidth, and probably resources that do not share across process boundaries easily. But there are still many examples of applications that could naturally be using different streams to concurrently receive data into different target buffers. FTP, HTTP, email, streaming media, SAN and NAS come to mind quickly. ------------=_1071178501-6739-0-- From Michael.Tuexen@micmac.franken.de Thu Dec 11 17:29:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29223 for ; Thu, 11 Dec 2003 17:29:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUZJ1-0003cR-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:29:11 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AUZJ1-0003b9-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:29:11 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBBMSNBN008459; Thu, 11 Dec 2003 14:28:24 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBMRxKs007408 for sctp-impl-filtered; Thu, 11 Dec 2003 14:28:01 -0800 (PST) Message-Id: <200312112228.hBBMRxKs007408@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071181679-7406-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: Michael Tuexen Cc: SCTP Implementors , Kacheong Poon X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Thu, 11 Dec 2003 22:47:15 +0100 This is a multi-part message in MIME format... ------------=_1071181679-7406-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin, see my comments below. Best regards Michael On Dec 11, 2003, at 10:24 PM, Caitlin Bestler wrote: > > On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote: > >> Michael Tuexen wrote: >> >>> But my main point is: for supporting different receive buffers >>> for different streams you also need different flow controls for >>> different streams. Do you agree with this statement? >> >> It really depends. I think what Caitlin's point is that an app >> does not have control over on putting data from one particular >> stream to one particular pool of buffers. For example, in the current >> socket API, an app gives a buffer the recvmsg(). It does not know >> which stream's data will be put into that buffer. With stream-socket, >> the app can control this as the buffer given to recvmsg() will >> only be filled with data from a particular stream. This has nothing >> to do with flow control. The flow control issue comes up at the >> protocol level. >> >>> Then there is the next problem: Can we extend SCTP (or even do we >>> need to extend) to provide a per stream flow control. Some guys >>> thought about this already and the result was, that it is not >>> possible to do as an extension. Of course, SCTP could have been >>> designed to provide that feature. But it was not done that way. >> >> Again, the API is not about changing SCTP. This needs to be >> emphasized. > > > Absolutely correct. The issues here have nothing to do with the > end-to-end semantics SCTP, but rather with the semantics of the > socket interface. > > Michael' key objection, as I understand it, is that any stream > specific receive will lead to laggard messages that cannot be > delivered. Fundamentally, the API layer is expected to deliver > the message with the oldest TSN first because that will enable > advancing the receive window. > > That is a valid concern. But you can achieve stream-specific > buffering without causing the receive window to be tardy. > > RDDP's use of SCTP (for RDMA services) relies on per stream > buffering for untagged messages and strictly upon ULP flow > control. SCTP flow control is used solely to manage network > congestion, and theoretically for SCTP layer buffering > (although the RDDP adaptation of SCTP uses only unordered > delivery, thereby enabling the SCTP layer to function > without its own buffers. It can place directly from the > IP layer buffers to the ULP buffers). > > Therefore the key conflict here is not between per-stream > receive buffering and SCTP, but between per-stream receive > buffering and normal socket semantics. > > Applications have gotten used to having the socket interface > provide critical ULP flow control: pause receiver until > message from peer arrives, pause transmitter until peer > is ready to receive more messages. > > Inherently, the socket layer will hold any messages that > it allowed to be sent that the ULP is not ready to receive. > > Michael is assuming that because SCTP does not have per- > stream flow control that per stream buffering will not work. > What is more accurate, is that it means that inferring > per-stream flow control from the sockets interface will > not work. > > To successfully provide per-stream buffering, an API must > have the following rules: > > The receiver, through some mechanism, must have > informed the sender of the maximum number of messages > that can be in-flight. For most RDMA applications this > is a pre-arranged or negotiated number of credits. Credits > are reduced when a message is sent, and incremented when > the receiver replies. For non-RDMA applications it would > probably work to increase credits as prior messages were > acked. What does 'the receiver replies' means. Sending a SACK? I have the impression that you are running a thin layer on top of SCTP doing the ACKs, is that right? > > When any authorized message is received, the API layer > MUST be able to forward it to a ULP buffer immediately. > This implies a callback that says "here, this messge is > yours -- do something with it NOW", and/or pre-posting of > receive buffers. > > With pre-posting, an unauthorized message MAY be discarded, > placing the stream in a "terminated" state. Essentially, if > the ULP fails to provide sufficient buffers the API layer > would substitute the bit bucket as the destination. > > The key is that advance the receive window must never be held > up because the application failed to ask for buffer on a specific > stream. I do not see how you can extend the socket paradigm > to match these rules, but APIs based upon callbacks and/or RQ/CQs > would handle this easily. > > Obviously all streams within an association should be related. > They are sharing the association's bandwidth, and probably resources > that do not share across process boundaries easily. But there are > still many examples of applications that could naturally be using > different streams to concurrently receive data into different > target buffers. FTP, HTTP, email, streaming media, SAN and NAS > come to mind quickly. > > ------------=_1071181679-7406-0-- From cait@asomi.com Thu Dec 11 17:32:15 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29465 for ; Thu, 11 Dec 2003 17:32:15 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUZM0-0003mV-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:32:16 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AUZLz-0003ky-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:32:15 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 11 Dec 2003 22:31:46 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBBMV6iN026739; Thu, 11 Dec 2003 14:31:06 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBMUtKs007453 for sctp-impl-filtered; Thu, 11 Dec 2003 14:30:57 -0800 (PST) Message-Id: <200312112230.hBBMUtKs007453@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071181855-7451-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Caitlin Bestler Cc: SCTP Implementors , Kacheong Poon X-Nails-Approved: cait@asomi.com Date: Thu, 11 Dec 2003 16:06:15 -0600 This is a multi-part message in MIME format... ------------=_1071181855-7451-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 11, 2003, at 3:47 PM, Michael Tuexen wrote: >> To successfully provide per-stream buffering, an API must >> have the following rules: >> >> The receiver, through some mechanism, must have >> informed the sender of the maximum number of messages >> that can be in-flight. For most RDMA applications this >> is a pre-arranged or negotiated number of credits. Credits >> are reduced when a message is sent, and incremented when >> the receiver replies. For non-RDMA applications it would >> probably work to increase credits as prior messages were >> acked. > What does 'the receiver replies' means. Sending a SACK? I have > the impression that you are running a thin layer on top of SCTP > doing the ACKs, is that right? For RDDP over SCTP what is meant by "the receiver replies" is that the ULP sends a response message back to its peer. So the type of behavior I am talking about can be implemented with an API layer, it does not require a protocol layer. RDDP already comes very close to that, in that there are no RDDP messages, merely wrappers around messages supplied from its ULP. But you could build an RQ/CQ and/or callback interface directly to SCTP and it would be no more of a protocol layer than sockets. ------------=_1071181855-7451-0-- From bidulock@openss7.org Thu Dec 11 17:57:37 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA00456 for ; Thu, 11 Dec 2003 17:57:36 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AUZkY-0004gf-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:57:38 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AUZkX-0004fb-00 for sctp-impl-archive@ietf.org; Thu, 11 Dec 2003 17:57:37 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBBMucrX010583; Thu, 11 Dec 2003 14:56:38 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBBMuIKs007777 for sctp-impl-filtered; Thu, 11 Dec 2003 14:56:20 -0800 (PST) Message-Id: <200312112256.hBBMuIKs007777@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071183378-7775-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: "Brian F. G. Bidulock" Cc: SCTP Implementors , Michael Tuexen , Kacheong Poon X-Nails-Approved: bidulock@openss7.org Date: Thu, 11 Dec 2003 15:52:36 -0700 This is a multi-part message in MIME format... ------------=_1071183378-7775-0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: binary Caitlin, Per-stream flow control can be effectively performed by both the receiver and the transmitter. How each proportions its receive and transmit buffer is up to it. The end-point implementation can even track the number of oustanding bytes per stream and proportion its local transmit buffer in accordance with the remote end's receive buffer if it wants (full flow control). The only protocol difficulty I know of is when messages are being fragmented and MUST be sent with consecutive TSNs, blocking all other streams off of the wire and the rwnd, introducing HOL blocking again. Relaxing that constraint, as is easily done for most implementations, would free the protocol from this defficiency in design. If you don't permit a read from a stream, you will just force the user to flock and peek the socket, something like: mutex-lock- loop: no-wait-poll flock -peek- -read-if-correct-sid- else cond-signal funlock cond-wait-mutex-unlock wakeup-mutex-locked Which could more easily be performed in the kernel, after all socket sendmsg recvmsg functions are supposed to be thread-safe as well a respecting mandatory POSIX file locking. Well, if it won't be done by the kernel, the sockets library will wind up doing it. But why do it so different than TCP or standard SOCK_SEQPACKET? --brian On Thu, 11 Dec 2003, Caitlin Bestler wrote: > > On Dec 11, 2003, at 10:42 AM, Kacheong Poon wrote: > > > Michael Tuexen wrote: > > > >> But my main point is: for supporting different receive buffers > >> for different streams you also need different flow controls for > >> different streams. Do you agree with this statement? > > > > It really depends. I think what Caitlin's point is that an app > > does not have control over on putting data from one particular > > stream to one particular pool of buffers. For example, in the current > > socket API, an app gives a buffer the recvmsg(). It does not know > > which stream's data will be put into that buffer. With stream-socket, > > the app can control this as the buffer given to recvmsg() will > > only be filled with data from a particular stream. This has nothing > > to do with flow control. The flow control issue comes up at the > > protocol level. > > > >> Then there is the next problem: Can we extend SCTP (or even do we > >> need to extend) to provide a per stream flow control. Some guys > >> thought about this already and the result was, that it is not > >> possible to do as an extension. Of course, SCTP could have been > >> designed to provide that feature. But it was not done that way. > > > > Again, the API is not about changing SCTP. This needs to be > > emphasized. > > > Absolutely correct. The issues here have nothing to do with the > end-to-end semantics SCTP, but rather with the semantics of the > socket interface. > > Michael' key objection, as I understand it, is that any stream > specific receive will lead to laggard messages that cannot be > delivered. Fundamentally, the API layer is expected to deliver > the message with the oldest TSN first because that will enable > advancing the receive window. > > That is a valid concern. But you can achieve stream-specific > buffering without causing the receive window to be tardy. > > RDDP's use of SCTP (for RDMA services) relies on per stream > buffering for untagged messages and strictly upon ULP flow > control. SCTP flow control is used solely to manage network > congestion, and theoretically for SCTP layer buffering > (although the RDDP adaptation of SCTP uses only unordered > delivery, thereby enabling the SCTP layer to function > without its own buffers. It can place directly from the > IP layer buffers to the ULP buffers). > > Therefore the key conflict here is not between per-stream > receive buffering and SCTP, but between per-stream receive > buffering and normal socket semantics. > > Applications have gotten used to having the socket interface > provide critical ULP flow control: pause receiver until > message from peer arrives, pause transmitter until peer > is ready to receive more messages. > > Inherently, the socket layer will hold any messages that > it allowed to be sent that the ULP is not ready to receive. > > Michael is assuming that because SCTP does not have per- > stream flow control that per stream buffering will not work. > What is more accurate, is that it means that inferring > per-stream flow control from the sockets interface will > not work. > > To successfully provide per-stream buffering, an API must > have the following rules: > > The receiver, through some mechanism, must have > informed the sender of the maximum number of messages > that can be in-flight. For most RDMA applications this > is a pre-arranged or negotiated number of credits. Credits > are reduced when a message is sent, and incremented when > the receiver replies. For non-RDMA applications it would > probably work to increase credits as prior messages were > acked. > > When any authorized message is received, the API layer > MUST be able to forward it to a ULP buffer immediately. > This implies a callback that says "here, this messge is > yours -- do something with it NOW", and/or pre-posting of > receive buffers. > > With pre-posting, an unauthorized message MAY be discarded, > placing the stream in a "terminated" state. Essentially, if > the ULP fails to provide sufficient buffers the API layer > would substitute the bit bucket as the destination. > > The key is that advance the receive window must never be held > up because the application failed to ask for buffer on a specific > stream. I do not see how you can extend the socket paradigm > to match these rules, but APIs based upon callbacks and/or RQ/CQs > would handle this easily. > > Obviously all streams within an association should be related. > They are sharing the association's bandwidth, and probably resources > that do not share across process boundaries easily. But there are > still many examples of applications that could naturally be using > different streams to concurrently receive data into different > target buffers. FTP, HTTP, email, streaming media, SAN and NAS > come to mind quickly. > > -- Brian F. G. Bidulock ¦ The reasonable man adapts himself to the ¦ bidulock@openss7.org ¦ world; the unreasonable one persists in ¦ http://www.openss7.org/ ¦ trying to adapt the world to himself. ¦ ¦ Therefore all progress depends on the ¦ ¦ unreasonable man. -- George Bernard Shaw ¦ ------------=_1071183378-7775-0-- From ma-kun@kozuka.jp Mon Dec 15 01:25:27 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA11716 for ; Mon, 15 Dec 2003 01:25:27 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVmAX-0004HE-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 01:25:25 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AVmAX-0004Gt-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 01:25:25 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBF6O4Z0002661; Sun, 14 Dec 2003 22:24:07 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBF6N9Ks009762 for sctp-impl-filtered; Sun, 14 Dec 2003 22:23:11 -0800 (PST) Message-Id: <200312150623.hBF6N9Ks009762@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071469389-9760-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: [RFC] Chage looking PCB(TCB) code List-Id: sctp-impl To: SCTP Implementors From: KOZUKA Masahiro X-Nails-Approved: ma-kun@kozuka.jp Date: Mon, 15 Dec 2003 15:21:18 +0900 This is a multi-part message in MIME format... ------------=_1071469389-9760-0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: binary Hi. I promote a following change. This change is related to sctp_pcbfind() and sctp_findassociation_addr_sa(). sctp_pcbfind() will find at TCP pool too only if find_tcp_pool true, sctp_findassociation_addr_sa() and so on. This leads to 2 merits. 1) sctp_getcred() or sctp6_getcred will be able to return the proper cred. Before this change, it returns the cred only on the basis of local address. Without this change, sctp_getcred() must call sctp_findassociation_addr_sa() and sctp_tcb_special_locate(), but sctp_tcb_special_locate() not exported. And I think this way is not beautiful. With this change, sctp_findassociation_addr_sa() will return the correct endpoint on TCP pool even if there is no accepting endpoint, so that sctp_tcb_special_locate() not needed. 2) The associastions which we sctp_peeroff(2) or accept(2) (ie. on TCP pool) will be found more quickly than other associations. Before this change, the associations on TCP pool have overheads. For looking up an association on TCP pool, sctp_pcbfind() return a listening endpoint and sctp_findassociation_ep_addr() call sctp_tcb_special_locate(). But sctp_findassociation_addr_sa() can call sctp_tcb_special_locate() directly because sctp_findassociation_addr_sa() have the information about local and remote. Of course, other associations will be processed unfaily but the way in which we process separate associastions in preference to others isn't so bad, I think. Comments? Regards, Masahiro. -- diff -u -r sctp.kame.20031215/netinet/sctp_pcb.c sctp.kame/netinet/sctp_pcb.c --- sctp.kame.20031215/netinet/sctp_pcb.c Sat Dec 13 23:28:19 2003 +++ sctp.kame/netinet/sctp_pcb.c Sun Dec 14 05:34:37 2003 @@ -657,7 +657,7 @@ struct sctp_inpcb * -sctp_pcb_findep(struct sockaddr *nam) +sctp_pcb_findep(struct sockaddr *nam, int find_tcp_pool) { /* * First we check the hash table to see if someone has this port @@ -707,7 +707,7 @@ * may NOT be the correct one but the sctp_findassociation_ep_addr * has further code to look at all TCP models. */ - if (ep == NULL) { + if (ep == NULL && find_tcp_pool) { int i; #ifdef SCTP_DEBUG if (sctp_debug_on & SCTP_DEBUG_PCB1) { @@ -748,11 +748,25 @@ struct sctp_tcb * sctp_findassociation_addr_sa(struct sockaddr *to, struct sockaddr *from, struct sctp_inpcb **inp, - struct sctp_nets **netp) + struct sctp_nets **netp, + int find_tcp_pool) { struct sctp_inpcb *ep; +#ifdef SCTP_TCP_MODEL_SUPPORT + struct sctp_tcb *tcb; - ep = sctp_pcb_findep(to); + if (find_tcp_pool) { + if (inp != NULL) { + tcb = sctp_tcb_special_locate(inp, from, to, netp); + } else { + tcb = sctp_tcb_special_locate(&ep, from, to, netp); + } + if (tcb != NULL) { + return tcb; + } + } +#endif + ep = sctp_pcb_findep(to, 0); if (inp != NULL) { *inp = ep; } @@ -938,9 +949,7 @@ struct sctp_nets **netp, uint32_t vtag) { -#ifdef SCTP_TCP_MODEL_SUPPORT - int to_tcp_pool; -#endif + int find_tcp_pool; struct ip *iph; struct sctphdr *sh; struct sctp_chunkhdr *chdr; @@ -1005,40 +1014,31 @@ } } + find_tcp_pool = 0; +#ifdef SCTP_TCP_MODEL_SUPPORT + chdr = (struct sctp_chunkhdr *)((caddr_t)sh + + sizeof(struct sctphdr)); + if ((chdr->chunk_type != SCTP_INITIATION) && + (chdr->chunk_type != SCTP_INITIATION_ACK) && + (chdr->chunk_type != SCTP_COOKIE_ACK) && + (chdr->chunk_type != SCTP_COOKIE_ECHO)) + /* Other chunk types go to the tcp pool. */ + find_tcp_pool = 1; +#endif if (inp) { - ret = sctp_findassociation_addr_sa(to, from, inp, netp); + ret = sctp_findassociation_addr_sa(to, from, + inp, netp, find_tcp_pool); linp = *inp; } else { - ret = sctp_findassociation_addr_sa(to, from, &linp, netp); + ret = sctp_findassociation_addr_sa(to, from, + &linp, netp, find_tcp_pool); } #ifdef SCTP_DEBUG if (sctp_debug_on & SCTP_DEBUG_PCB1) { printf("ret:%p linp:%p\n", ret, linp); } #endif -#ifdef SCTP_TCP_MODEL_SUPPORT - chdr = (struct sctp_chunkhdr *)((caddr_t)sh + - sizeof(struct sctphdr)); - if ((chdr->chunk_type == SCTP_INITIATION) || - (chdr->chunk_type == SCTP_INITIATION_ACK) || - (chdr->chunk_type == SCTP_COOKIE_ACK) || - (chdr->chunk_type == SCTP_COOKIE_ECHO)) { - /* These chunk types go back to the main - * pool and can't go to the tcp pool. - */ - to_tcp_pool = 0; - } else { - to_tcp_pool = 1; - } - if (to_tcp_pool) { - if (ret == NULL) { - ret = sctp_tcb_special_locate(inp, from, to, netp); - } - if (ret && *inp) { - return (ret); - } - } -#endif if ((ret == NULL) && (linp)) { /* Found a EP but not this address */ #ifdef SCTP_DEBUG @@ -1612,20 +1613,9 @@ if (p == NULL) return (error); - - - lep = sctp_pcb_findep(addr); + lep = sctp_pcb_findep(addr, 0); /* not find at TCP pool. */ if (lep != NULL) { - /* If it is NOT in the TCP pool then it exists */ - if ((lep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL) == 0) { - return (EADDRNOTAVAIL); - } else - /* - * In this case it IS only found in the TCP - * pool which means the main listner went away - * so it is now ok to re-bind. - */ - lep = NULL; + return (EADDRNOTAVAIL); } if (bindall) { /* verify that no lport is not used by a singleton */ diff -u -r sctp.kame.20031215/netinet/sctp_pcb.h sctp.kame/netinet/sctp_pcb.h --- sctp.kame.20031215/netinet/sctp_pcb.h Sat Dec 13 23:28:21 2003 +++ sctp.kame/netinet/sctp_pcb.h Sun Dec 14 05:24:33 2003 @@ -376,7 +386,7 @@ struct sctp_nets *sctp_findnet(struct sctp_tcb *, struct sockaddr *); -struct sctp_inpcb *sctp_pcb_findep(struct sockaddr *); +struct sctp_inpcb *sctp_pcb_findep(struct sockaddr *, int); #if defined(__FreeBSD__) && __FreeBSD_version >= 500000 int sctp_inpcb_bind(struct socket *, struct sockaddr *, struct thread *); @@ -388,7 +398,7 @@ struct sctp_inpcb **, struct sctp_nets **, uint32_t vtag); struct sctp_tcb *sctp_findassociation_addr_sa(struct sockaddr *, - struct sockaddr *, struct sctp_inpcb **, struct sctp_nets **); + struct sockaddr *, struct sctp_inpcb **, struct sctp_nets **, int); #ifdef SCTP_TCP_MODEL_SUPPORT void sctp_move_pcb_and_assoc(struct sctp_inpcb *, struct sctp_inpcb *, diff -u -r sctp.kame.20031215/netinet/sctp_usrreq.c sctp.kame/netinet/sctp_usrreq.c --- sctp.kame.20031215/netinet/sctp_usrreq.c Sat Dec 13 23:28:25 2003 +++ sctp.kame/netinet/sctp_usrreq.c Sun Dec 14 05:30:59 2003 @@ -496,7 +496,7 @@ #endif stcb = sctp_findassociation_addr_sa((struct sockaddr *)&from, (struct sockaddr *)&to, - &inp, &netp); + &inp, &netp, 1); /* find at TCP pool too. */ if (stcb != NULL && inp && (inp->sctp_socket != NULL)) { if (cmd != PRC_MSGSIZE) { int cm; @@ -536,6 +536,8 @@ { struct sockaddr_in addrs[2]; struct sctp_inpcb *inp; + struct sctp_nets *net; + struct sctp_tcb *tcb; int error, s; #if __FreeBSD_version >= 500000 @@ -549,13 +551,11 @@ if (error) return (error); -#if defined(__NetBSD__) || defined(__OpenBSD__) - s = splsoftnet(); -#else s = splnet(); -#endif - inp = sctp_pcb_findep((struct sockaddr *)&addrs[0]); - if (inp == NULL || inp->sctp_socket == NULL) { + tcb = sctp_findassociation_addr_sa(sintosa(&addrs[0]), + sintosa(&addrs[1]), + &inp, &net, 1); + if (tcb == NULL || inp == NULL || inp->sctp_socket == NULL) { error = ENOENT; goto out; } @@ -2830,7 +2830,7 @@ struct sctp_inpcb *lep; ((struct sockaddr_in *)addrs->addr)->sin_port = inp->sctp_lport; - lep = sctp_pcb_findep(addrs->addr); + lep = sctp_pcb_findep(addrs->addr, 1); /* find at TCP pool too. */ if (lep == inp) { /* already bound to it.. ok */ break; diff -u -r sctp.kame.20031215/netinet6/sctp6_usrreq.c sctp.kame/netinet6/sctp6_usrreq.c --- sctp.kame.20031215/netinet6/sctp6_usrreq.c Fri Nov 28 06:13:57 2003 +++ sctp.kame/netinet6/sctp6_usrreq.c Sun Dec 14 05:38:06 2003 @@ -539,7 +539,7 @@ #endif stcb = sctp_findassociation_addr_sa((struct sockaddr *)ip6cp->ip6c_src, (struct sockaddr *)&final, - &inp, &netp); + &inp, &netp, 1); if (stcb != NULL && inp && (inp->sctp_socket != NULL)) { if (cmd == PRC_MSGSIZE) { sctp6_notify_mbuf(inp, @@ -582,6 +582,8 @@ { struct sockaddr_in6 addrs[2]; struct sctp_inpcb *inp; + struct sctp_nets *net; + struct sctp_tcb *tcb; int error, s; #if __FreeBSD_version >= 500000 @@ -605,8 +607,10 @@ s = splnet(); #endif - inp = sctp_pcb_findep((struct sockaddr *)&addrs[0]); - if (inp == NULL || inp->sctp_socket == NULL) { + tcb = sctp_findassociation_addr_sa(sin6tosa(&addrs[0]), + sin6tosa(&addrs[1]), + &inp, &net, 1); /* find at TCP pool too. */ + if (tcb == NULL || inp == NULL || inp->sctp_socket == NULL) { error = ENOENT; goto out; } -- -- {^-^}/@kozuka.jp / KOZUKA Masahiro ------------=_1071469389-9760-0-- From kacheong.poon@sun.com Mon Dec 15 02:39:21 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA27421 for ; Mon, 15 Dec 2003 02:39:21 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVnK4-00061j-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:39:20 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AVnK3-00061a-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:39:19 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 14 Dec 2003 23:41:54 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBF7cRAt010160; Sun, 14 Dec 2003 23:38:27 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBF7c3Ks010710 for sctp-impl-filtered; Sun, 14 Dec 2003 23:38:05 -0800 (PST) Message-Id: <200312150738.hBF7c3Ks010710@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071473883-10708-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Sun, 14 Dec 2003 23:37:11 -0800 This is a multi-part message in MIME format... ------------=_1071473883-10708-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: >> I guess the point of having a socket for individual stream >> is not related to what SCTP the protocol tries to do. It >> is more about how SCTP apps want to use the API. So it >> seems to me that the dicussion should be focused on whether >> app programmers like to have such as interface. Then the >> second question is what kind of caveats this interface may >> have. Also note that the stream-socket can both send and >> receive. We should also consider the usefulness of having >> stream-socket for sending. >> > Streams are not bidirectional. So how will you send? Stream-socket is bidirectional means that it does the obvious things. A send() on it will cause the data to be sent in the particular stream id. Likewise, a receive() on it will only receive data sent to the particular stream id. >> From the app's point of view, having different sockets for >> different streams allow them to separate task handling easier. >> It has nothing to do with how SCTP assigns TSNs. The way an >> app will see this is that it just needs to setup one association >> but can have multiple data streams controlled individually. This >> is "like" opening multiple TCP connections. Now whether SCTP can >> provide identical semantics is another issue, we know that it >> cannot because of the issue rasied. But is it good enough so that >> a well bahaved app can make use of it. This is the question we >> need to ask and answer. > > This is my point: people want something like multiple TCP connections > in one entity and might think about about an association with n streams > in each direction such they can use them a bidirectional streams. > But the BIG difference is that the TCP connections have separate > buffer management and the receive buffer for the SCTP association > is shared. So my answer to this is: It is not good enough. See below. >> What you described above is really in the SCTP protocol level. >> What the stream scheduling I mentioned was about how an app could >> schedule different threads in handling send/receive of different >> streams. I think they are separate issues. > > This is really my point. When you write a multithreaded application > you may want to block threads intentionally by using semaphores or > other techniques. But the receive calls should be independent. And > this is not the case. If an app chooses to use this interface, it means that it will make sure that it will not block forever such that only data from one particular stream will be read. The key here is that will an app do the right thing? It seems that you are assuming that an app will always do the wrong thing... -- K. Poon. kacheong.poon@sun.com ------------=_1071473883-10708-0-- From kacheong.poon@sun.com Mon Dec 15 02:43:25 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA27715 for ; Mon, 15 Dec 2003 02:43:24 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVnO0-0006BA-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:43:24 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AVnNz-0006A4-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:43:23 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-3.cisco.com with ESMTP; 14 Dec 2003 23:45:58 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBF7gVAt012857; Sun, 14 Dec 2003 23:42:31 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBF7gJKs010774 for sctp-impl-filtered; Sun, 14 Dec 2003 23:42:21 -0800 (PST) Message-Id: <200312150742.hBF7gJKs010774@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071474138-10772-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Sun, 14 Dec 2003 23:41:33 -0800 This is a multi-part message in MIME format... ------------=_1071474138-10772-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: >> It seems that we are really discussing two different but >> related issues. Your point of view is from the protocol >> level. But we are talking about an app which tries to >> do the right thing. And the question is whether it is >> useful to provide such an API for those apps to do their >> jobs. >> > What kind of function calls are you thinking about, what > are the semantic and how will you map this to the protocol > layer? My point is that you will promise independent receive > calls which will result in my mapping to the protocol layer > to a deadlock situation. But possibly there is another way > map your functions to the protocol. For example, select() can now work on a per stream basis. Note that there is no new mapping of functions to the SCTP protocol. Right now, one can write a library to export interface for sending/receiving on a particular stream of an SCTP association. My question is very simple. Should the OS provide support to the above such that native system calls on socket can be used on a per stream basis. This is the simple question. -- K. Poon. kacheong.poon@sun.com ------------=_1071474138-10772-0-- From kacheong.poon@sun.com Mon Dec 15 02:56:45 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA28507 for ; Mon, 15 Dec 2003 02:56:45 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVnau-0006cw-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:56:44 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AVnau-0006ch-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 02:56:44 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBF7tOiN016447; Sun, 14 Dec 2003 23:55:25 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBF7tDKs010943 for sctp-impl-filtered; Sun, 14 Dec 2003 23:55:15 -0800 (PST) Message-Id: <200312150755.hBF7tDKs010943@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071474913-10941-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com, Caitlin Bestler X-Nails-Approved: kacheong.poon@sun.com Date: Sun, 14 Dec 2003 23:54:30 -0800 This is a multi-part message in MIME format... ------------=_1071474913-10941-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > Assuming that you can use recv_message_from_stream(stream, buffer) > to get data from a particular stream, how will you handle the problem > 'at the protocol level'? If you are not doing it right there you > will have problems at the upper layer. This is really an implementation issue. One implementation may do it at the SCTP layer such that it exports a way for the socket layer to receive data on a per stream basis. Another implementation may do it completely at the socket layer such that it separates the data from the SCTP layer into different streams and put them into the queues of different sockets. In the case Caitlin mentioned, the pre- posted buffer may go all the way down to the device driver. It may not be straightforward to implement this stream- socket. But it is an implementation issue. The question I'd like to answer is whether people see it as a useful feature. If nobody will use it, there is no need to discuss the details... -- K. Poon. kacheong.poon@sun.com ------------=_1071474913-10941-0-- From rrs@cisco.com Mon Dec 15 07:34:08 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA05380 for ; Mon, 15 Dec 2003 07:34:08 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVrvN-0003J5-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 07:34:09 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AVrvM-0003IF-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 07:34:08 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 15 Dec 2003 12:34:25 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBFCWqiN004947; Mon, 15 Dec 2003 04:32:52 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBFCWJKs014775 for sctp-impl-filtered; Mon, 15 Dec 2003 04:32:21 -0800 (PST) Message-Id: <200312151232.hBFCWJKs014775@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071491539-14773-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: "Randall Stewart (cisco)" Cc: Michael Tuexen , sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Mon, 15 Dec 2003 06:32:02 -0600 This is a multi-part message in MIME format... ------------=_1071491539-14773-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong Poon wrote: > Michael Tuexen wrote: > >>> I guess the point of having a socket for individual stream >>> is not related to what SCTP the protocol tries to do. It >>> is more about how SCTP apps want to use the API. So it >>> seems to me that the dicussion should be focused on whether >>> app programmers like to have such as interface. Then the >>> second question is what kind of caveats this interface may >>> have. Also note that the stream-socket can both send and >>> receive. We should also consider the usefulness of having >>> stream-socket for sending. >>> >> Streams are not bidirectional. So how will you send? > > > Stream-socket is bidirectional means that it does the > obvious things. A send() on it will cause the data to be > sent in the particular stream id. Likewise, a receive() > on it will only receive data sent to the particular stream > id. I think what Michael means here (correct me if I am wrong) is that it is possible for one side to have 10 streams going towards the peer and the other to have 1. So how do we handle this situation i.e. I peel off stream 8 into a socket. Now stream 8 on one side can send but NOT recieve. Stream 8 on the other side can receive but NOT send... Is this what you meant Michael? > > >>> From the app's point of view, having different sockets for >>> different streams allow them to separate task handling easier. >>> It has nothing to do with how SCTP assigns TSNs. The way an >>> app will see this is that it just needs to setup one association >>> but can have multiple data streams controlled individually. This >>> is "like" opening multiple TCP connections. Now whether SCTP can >>> provide identical semantics is another issue, we know that it >>> cannot because of the issue rasied. But is it good enough so that >>> a well bahaved app can make use of it. This is the question we >>> need to ask and answer. >> >> >> This is my point: people want something like multiple TCP connections >> in one entity and might think about about an association with n streams >> in each direction such they can use them a bidirectional streams. >> But the BIG difference is that the TCP connections have separate >> buffer management and the receive buffer for the SCTP association >> is shared. So my answer to this is: It is not good enough. > > > See below. > >>> What you described above is really in the SCTP protocol level. >>> What the stream scheduling I mentioned was about how an app could >>> schedule different threads in handling send/receive of different >>> streams. I think they are separate issues. >> >> >> This is really my point. When you write a multithreaded application >> you may want to block threads intentionally by using semaphores or >> other techniques. But the receive calls should be independent. And >> this is not the case. > > > If an app chooses to use this interface, it means that it will > make sure that it will not block forever such that only data > from one particular stream will be read. The key here is that > will an app do the right thing? It seems that you are assuming > that an app will always do the wrong thing... But do we code things assuming the app will always do the right thing? I think this is the fundamental question that is being asked.... R > > -- Randall R. Stewart ITD - Transport Technologies 815-477-2127(o) or 815-342-4222(c) ------------=_1071491539-14773-0-- From kacheong.poon@sun.com Mon Dec 15 07:53:26 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA05963 for ; Mon, 15 Dec 2003 07:53:26 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVsE2-0003fV-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 07:53:26 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AVsE2-0003db-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 07:53:26 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBFCqBrX008848; Mon, 15 Dec 2003 04:52:15 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBFCpaKs015019 for sctp-impl-filtered; Mon, 15 Dec 2003 04:51:38 -0800 (PST) Message-Id: <200312151251.hBFCpaKs015019@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071492696-15017-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" From: Kacheong Poon Cc: Michael Tuexen , sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Mon, 15 Dec 2003 04:50:43 -0800 This is a multi-part message in MIME format... ------------=_1071492696-15017-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Randall Stewart (cisco) wrote: > I think what Michael means here (correct me if I am wrong) is that > it is possible for one side to have 10 streams going towards the > peer and the other to have 1. So how do we handle this situation > i.e. I peel off stream 8 into a socket. Now stream 8 on one side > can send but NOT recieve. Stream 8 on the other side can receive > but NOT send... Shouldn't this be the same as sending/receiving on a wrong stream id? An error will be returned. And I don't think an app which wants to use stream-socket will do the above. This does not require special handling anyway. >> If an app chooses to use this interface, it means that it will >> make sure that it will not block forever such that only data >> from one particular stream will be read. The key here is that >> will an app do the right thing? It seems that you are assuming >> that an app will always do the wrong thing... > > > But do we code things assuming the app will always do the > right thing? I think this is the fundamental question that > is being asked.... I think the first question to ask is really whether people want to use this feature. If nobody wants that, we don't need to think more about. That was my original question. But getting back to the above, a similar situation can be said on mutex. An app can cause dealock itself by using mutex incorrectly. All OSes still provide mutex because of its usefulness. So we need to answer the question of the usefulness of the stream-socket first. -- K. Poon. kacheong.poon@sun.com ------------=_1071492696-15017-0-- From Michael.Tuexen@micmac.franken.de Mon Dec 15 11:47:23 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16241 for ; Mon, 15 Dec 2003 11:47:23 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVvsS-0000Um-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:47:24 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AVvsS-0000UF-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:47:24 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-2.cisco.com with ESMTP; 15 Dec 2003 08:49:42 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBFGkSAt004162; Mon, 15 Dec 2003 08:46:28 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBFGkBKs018097 for sctp-impl-filtered; Mon, 15 Dec 2003 08:46:13 -0800 (PST) Message-Id: <200312151646.hBFGkBKs018097@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071506770-18095-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 15 Dec 2003 17:05:03 +0100 This is a multi-part message in MIME format... ------------=_1071506770-18095-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, see my comments below. Best regards Michael On Dec 15, 2003, at 8:37 AM, Kacheong Poon wrote: > Michael Tuexen wrote: > >>> I guess the point of having a socket for individual stream >>> is not related to what SCTP the protocol tries to do. It >>> is more about how SCTP apps want to use the API. So it >>> seems to me that the dicussion should be focused on whether >>> app programmers like to have such as interface. Then the >>> second question is what kind of caveats this interface may >>> have. Also note that the stream-socket can both send and >>> receive. We should also consider the usefulness of having >>> stream-socket for sending. >>> >> Streams are not bidirectional. So how will you send? > > Stream-socket is bidirectional means that it does the > obvious things. A send() on it will cause the data to be > sent in the particular stream id. Likewise, a receive() > on it will only receive data sent to the particular stream > id. > Assume the you have 1 stream from A to B and 10 from B to A. Now you get a socket for receiving data on stream 5 at A. On what stream do you send data? >>> From the app's point of view, having different sockets for >>> different streams allow them to separate task handling easier. >>> It has nothing to do with how SCTP assigns TSNs. The way an >>> app will see this is that it just needs to setup one association >>> but can have multiple data streams controlled individually. This >>> is "like" opening multiple TCP connections. Now whether SCTP can >>> provide identical semantics is another issue, we know that it >>> cannot because of the issue rasied. But is it good enough so that >>> a well bahaved app can make use of it. This is the question we >>> need to ask and answer. >> This is my point: people want something like multiple TCP connections >> in one entity and might think about about an association with n >> streams >> in each direction such they can use them a bidirectional streams. >> But the BIG difference is that the TCP connections have separate >> buffer management and the receive buffer for the SCTP association >> is shared. So my answer to this is: It is not good enough. > > See below. > >>> What you described above is really in the SCTP protocol level. >>> What the stream scheduling I mentioned was about how an app could >>> schedule different threads in handling send/receive of different >>> streams. I think they are separate issues. >> This is really my point. When you write a multithreaded application >> you may want to block threads intentionally by using semaphores or >> other techniques. But the receive calls should be independent. And >> this is not the case. > > If an app chooses to use this interface, it means that it will > make sure that it will not block forever such that only data > from one particular stream will be read. The key here is that > will an app do the right thing? It seems that you are assuming > that an app will always do the wrong thing... > My point is not that the app does the wrong thing, but possibly it depends on the scheduling of the threads. Something which is out of scope of the transport... If you can make sure that the app will receive data on ALL streams without delay, you have no problem. But how can you do this? > > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071506770-18095-0-- From Michael.Tuexen@micmac.franken.de Mon Dec 15 11:56:06 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16522 for ; Mon, 15 Dec 2003 11:56:05 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVw0t-0000fi-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:56:07 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AVw0s-0000fS-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:56:07 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 15 Dec 2003 16:56:22 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBFGsxiN023277; Mon, 15 Dec 2003 08:55:00 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBFGshKs018210 for sctp-impl-filtered; Mon, 15 Dec 2003 08:54:45 -0800 (PST) Message-Id: <200312151654.hBFGshKs018210@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071507282-18208-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 15 Dec 2003 17:14:00 +0100 This is a multi-part message in MIME format... ------------=_1071507282-18208-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, let us see if people want this feature in the API. In the SIGTRAN environment it is not need (I think). I do know about the need of a per stream flow control but that is a different topic. The problems there can also being solved by using an appropriate scheduling function on the sender side. Best regards Michael On Dec 15, 2003, at 1:50 PM, Kacheong Poon wrote: > Randall Stewart (cisco) wrote: > >> I think what Michael means here (correct me if I am wrong) is that >> it is possible for one side to have 10 streams going towards the >> peer and the other to have 1. So how do we handle this situation >> i.e. I peel off stream 8 into a socket. Now stream 8 on one side >> can send but NOT recieve. Stream 8 on the other side can receive >> but NOT send... > > Shouldn't this be the same as sending/receiving on a wrong > stream id? An error will be returned. And I don't think > an app which wants to use stream-socket will do the above. > This does not require special handling anyway. > >>> If an app chooses to use this interface, it means that it will >>> make sure that it will not block forever such that only data >>> from one particular stream will be read. The key here is that >>> will an app do the right thing? It seems that you are assuming >>> that an app will always do the wrong thing... >> But do we code things assuming the app will always do the >> right thing? I think this is the fundamental question that >> is being asked.... > > I think the first question to ask is really whether people > want to use this feature. If nobody wants that, we don't > need to think more about. That was my original question. > > But getting back to the above, a similar situation can be > said on mutex. An app can cause dealock itself by using > mutex incorrectly. All OSes still provide mutex because > of its usefulness. So we need to answer the question of > the usefulness of the stream-socket first. > > > -- > > K. Poon. > kacheong.poon@sun.com > > > ------------=_1071507282-18208-0-- From Michael.Tuexen@micmac.franken.de Mon Dec 15 11:59:55 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16604 for ; Mon, 15 Dec 2003 11:59:55 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AVw4b-0000jC-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:59:57 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AVw4a-0000iN-00 for sctp-impl-archive@ietf.org; Mon, 15 Dec 2003 11:59:56 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 15 Dec 2003 17:00:11 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBFGx4iN025958; Mon, 15 Dec 2003 08:59:04 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBFGwrKs018266 for sctp-impl-filtered; Mon, 15 Dec 2003 08:58:55 -0800 (PST) Message-Id: <200312151658.hBFGwrKs018266@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071507533-18264-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Michael Tuexen Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 15 Dec 2003 17:18:00 +0100 This is a multi-part message in MIME format... ------------=_1071507533-18264-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong, see my comments below. Best regards Michael On Dec 15, 2003, at 1:50 PM, Kacheong Poon wrote: > Randall Stewart (cisco) wrote: > >> I think what Michael means here (correct me if I am wrong) is that >> it is possible for one side to have 10 streams going towards the >> peer and the other to have 1. So how do we handle this situation >> i.e. I peel off stream 8 into a socket. Now stream 8 on one side >> can send but NOT recieve. Stream 8 on the other side can receive >> but NOT send... > > Shouldn't this be the same as sending/receiving on a wrong > stream id? An error will be returned. And I don't think > an app which wants to use stream-socket will do the above. > This does not require special handling anyway. > This means that such a socket is not bidirectional, but unidirectional. >>> If an app chooses to use this interface, it means that it will >>> make sure that it will not block forever such that only data >>> from one particular stream will be read. The key here is that >>> will an app do the right thing? It seems that you are assuming >>> that an app will always do the wrong thing... >> But do we code things assuming the app will always do the >> right thing? I think this is the fundamental question that >> is being asked.... > > I think the first question to ask is really whether people > want to use this feature. If nobody wants that, we don't > need to think more about. That was my original question. > > But getting back to the above, a similar situation can be > said on mutex. An app can cause dealock itself by using > mutex incorrectly. All OSes still provide mutex because > of its usefulness. So we need to answer the question of > the usefulness of the stream-socket first. Mutexes are for dealing with race conditions and so on, so handling them incorrectly results in deadlocks. That is OK. stream-sockets are not for that, but to (mis)use the stream ID as a demulitplexer for an ULP. So having a deadlock is not within the scope of stream IDs, I think. > > > -- > > K. Poon. > kacheong.poon@sun.com > > ------------=_1071507533-18264-0-- From kacheong.poon@sun.com Tue Dec 16 02:56:36 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA03510 for ; Tue, 16 Dec 2003 02:56:36 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWA4J-0004EG-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 02:56:36 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWA4J-0004E9-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 02:56:35 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWA4I-0004Dt-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 02:56:34 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBG7tUAt007203; Mon, 15 Dec 2003 23:55:30 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBG7sbKs029874 for sctp-impl-filtered; Mon, 15 Dec 2003 23:54:39 -0800 (PST) Message-Id: <200312160754.hBG7sbKs029874@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071561277-29872-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Mon, 15 Dec 2003 23:53:16 -0800 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071561277-29872-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > Assume the you have 1 stream from A to B and 10 from B to A. > Now you get a socket for receiving data on stream 5 at A. On > what stream do you send data? There is only one stream from A to B, so use that stream. It is really a decision made by the app on how to arrange the use of streams, and thus stream-socket... > My point is not that the app does the wrong thing, but possibly it > depends on the scheduling of the threads. Something which is out of scope > of the transport... It seems that system scheduling has very little to do with the issue, unless the app itself tries to use specific OS scheduling service to arrange different scheduling for itself. The worst case scenario you've given is that a stream-socket will continue to read from a specific stream, while other streams are "stuck" therefore casuing a SCTP protocol issue as the TSN does not move ahead. Normally, this has nothing to do with system scheduling as the scheduler should do the right thing. The only problem is that the app deliberately does not read. This is outside the control of the system. If the app itself tries to manipulate the scheduling, it'd better do the right thing. > If you can make sure that the app will receive data on ALL streams without > delay, you have no problem. But how can you do this? Only the app itself can make sure that it continues to read. The system cannot force an app to read. But this is fine and has nothing to do with the interface. The interface just makes sure that an app can receive from all streams, whether the app chooses to read it is not the interface's problem. The stream-socket does not prevent an app from receiving from all streams. -- K. Poon. kacheong.poon@sun.com ------------=_1071561277-29872-0-- From kacheong.poon@sun.com Tue Dec 16 03:13:15 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA04000 for ; Tue, 16 Dec 2003 03:13:15 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWAKR-0004hb-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 03:13:15 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWAKP-0004hP-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 03:13:14 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AWAKP-0004gE-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 03:13:13 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBG8C6iN005441; Tue, 16 Dec 2003 00:12:06 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBG8BoKs000101 for sctp-impl-filtered; Tue, 16 Dec 2003 00:11:52 -0800 (PST) Message-Id: <200312160811.hBG8BoKs000101@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071562310-99-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Kacheong Poon Cc: sctp-impl@external.cisco.com, "Randall Stewart (cisco)" X-Nails-Approved: kacheong.poon@sun.com Date: Tue, 16 Dec 2003 00:10:58 -0800 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071562310-99-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: >> Shouldn't this be the same as sending/receiving on a wrong >> stream id? An error will be returned. And I don't think >> an app which wants to use stream-socket will do the above. >> This does not require special handling anyway. >> > This means that such a socket is not bidirectional, but unidirectional. Why? If an app picks stream number 1 to send one specific kind of data, both sides of an association can then use a stream-socket on stream number 1 to send and receive these messages. Why isn't this bi-directional? The "unidirectional" scenario above is similar to TCP's half closed case. One side chooses to send only and the other side chooses to receive only. But this does not make a TCP connection undirectional by nature. It is the user which makes it undirectional. >> But getting back to the above, a similar situation can be >> said on mutex. An app can cause dealock itself by using >> mutex incorrectly. All OSes still provide mutex because >> of its usefulness. So we need to answer the question of >> the usefulness of the stream-socket first. > > Mutexes are for dealing with race conditions and so on, so > handling them incorrectly results in deadlocks. That is OK. > stream-sockets are not for that, but to (mis)use the stream ID > as a demulitplexer for an ULP. So having a deadlock is not > within the scope of stream IDs, I think. I guess I did not pick a good example. The point I tried to make was that any inappropriate use of a routine provided by a system can cause problems. And could you elaborate the point about misusing the stream id as a demultiplexer? Do you mean to say that an SCTP app should never use stream id as it is not designed to differentiate different flows in an association? -- K. Poon. kacheong.poon@sun.com ------------=_1071562310-99-0-- From rrs@cisco.com Tue Dec 16 07:22:31 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA12902 for ; Tue, 16 Dec 2003 07:22:31 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWEDf-0007E9-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 07:22:32 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWEDf-0007E2-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 07:22:31 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWEDe-0007Ca-00 for sctp-impl-archive@ietf.org; Tue, 16 Dec 2003 07:22:30 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBGCLEAt020301; Tue, 16 Dec 2003 04:21:14 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBGCKlKs003600 for sctp-impl-filtered; Tue, 16 Dec 2003 04:20:49 -0800 (PST) Message-Id: <200312161220.hBGCKlKs003600@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071577247-3598-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: "Randall Stewart (cisco)" Cc: Michael Tuexen , sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Tue, 16 Dec 2003 06:20:24 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071577247-3598-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Kacheong: One minor correction... Kacheong Poon wrote: > Michael Tuexen wrote: > >> Assume the you have 1 stream from A to B and 10 from B to A. >> Now you get a socket for receiving data on stream 5 at A. On >> what stream do you send data? > > > There is only one stream from A to B, so use that stream. > It is really a decision made by the app on how to arrange > the use of streams, and thus stream-socket... > >> My point is not that the app does the wrong thing, but possibly it >> depends on the scheduling of the threads. Something which is out of >> scope >> of the transport... > > > It seems that system scheduling has very little to do > with the issue, unless the app itself tries to use > specific OS scheduling service to arrange different > scheduling for itself. The worst case scenario you've > given is that a stream-socket will continue to read from > a specific stream, while other streams are "stuck" > therefore casuing a SCTP protocol issue as the TSN does > not move ahead. I think you describe the above a bit off.. what will happen when a stream gets "stuck" is you would then reach a point where the stream would hold all of the rwnd. TSN's can move forward up until that point.. at which time the stack should clamp the rwnd at 0 and drop all inbound data until the "stuck" stream reads... I just want us all to be clear on what SHOULD occur :-> R > > > Normally, this has nothing to do with system scheduling > as the scheduler should do the right thing. The only > problem is that the app deliberately does not read. This > is outside the control of the system. If the app itself > tries to manipulate the scheduling, it'd better do the > right thing. > >> If you can make sure that the app will receive data on ALL streams >> without >> delay, you have no problem. But how can you do this? > > > Only the app itself can make sure that it continues to > read. The system cannot force an app to read. But this > is fine and has nothing to do with the interface. The > interface just makes sure that an app can receive from all > streams, whether the app chooses to read it is not the > interface's problem. The stream-socket does not prevent > an app from receiving from all streams. > -- Randall R. Stewart ITD - Transport Technologies 815-477-2127(o) or 815-342-4222(c) ------------=_1071577247-3598-0-- From xtang@qnx.com Wed Dec 17 01:30:33 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA26897 for ; Wed, 17 Dec 2003 01:30:33 -0500 (EST) From: xtang@qnx.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWVCa-0004ti-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 01:30:32 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWVCZ-0004tb-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 01:30:31 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWVCY-0004st-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 01:30:31 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 16 Dec 2003 22:33:07 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBH6TSZ0024145; Tue, 16 Dec 2003 22:29:28 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBH6SrKs017572 for sctp-impl-filtered; Tue, 16 Dec 2003 22:28:55 -0800 (PST) Message-Id: <200312170628.hBH6SrKs017572@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071642532-17570-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" , "Kacheong Poon" Cc: "Michael Tuexen" , X-Nails-Approved: xtang@qnx.com Date: Wed, 17 Dec 2003 05:43:10 -0000 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.3 required=5.0 tests=NO_REAL_NAME autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071642532-17570-0 Content-Type: Top Content-Disposition: inline Mime-Version: 1.0 X-Mailer: MIME-tools 5.411 (Entity 5.404) Content-Transfer-Encoding: base64 CgoKIlJhbmRhbGwgU3Rld2FydCAoY2lzY28pIiA8cnJzQGNpc2NvLmNvbT4g c2FpZDoKCj4gSSB0aGluayB5b3UgZGVzY3JpYmUgdGhlIGFib3ZlIGEgYml0 IG9mZi4uIHdoYXQgd2lsbCBoYXBwZW4KPiB3aGVuIGEgc3RyZWFtIGdldHMg InN0dWNrIiBpcyB5b3Ugd291bGQgdGhlbiByZWFjaCBhIHBvaW50Cj4gd2hl cmUgdGhlIHN0cmVhbSB3b3VsZCBob2xkIGFsbCBvZiB0aGUgcnduZC4gVFNO J3MgY2FuIG1vdmUKPiBmb3J3YXJkIHVwIHVudGlsIHRoYXQgcG9pbnQuLiBh dCB3aGljaCB0aW1lIHRoZSBzdGFjayBzaG91bGQKPiBjbGFtcCB0aGUgcndu ZCBhdCAwIGFuZCBkcm9wIGFsbCBpbmJvdW5kIGRhdGEgdW50aWwKPiB0aGUg InN0dWNrIiBzdHJlYW0gcmVhZHMuLi4KPiAKPiBJIGp1c3Qgd2FudCB1cyBh bGwgdG8gYmUgY2xlYXIgb24gd2hhdCBTSE9VTEQgb2NjdXIgOi0+CgpHcmVh dCEgVGhhdCB3YXMgd2hhdCBteSB1bmRlcnN0YW5kaW5nIGlzLiBTbyB0aGUg d29yc3QgY2FzZQppcywgaWYgdXNlciBzdG9wcGVkIHJlY2VpdmluZyBmcm9t IHN0cmVhbSAxLCB0aGVuIGRhdGFzIGZvcgpzdHJlYW0gMSB3aWxsIHN0YXJ0 ICJlYXRpbmciIHJ3bmQsIHVudGlsIHNvbWUgcG9pbnQgdGhhdAp0b3RhbCBy d25kIGFyZSBhbGwgdXNlZCB1cCBmb3Igc3RyZWFtIDEncyBkYXRhLCB0aGVu IG90aGVyCnN0cmVhbXMgZ2V0ICJibG9ja2VkIi4KCkJ1dCBhZ2FpbiwgSSdk IGFyZ3UgdGhhdCdzIHRoZSBhcHBsaWNhdGlvbidzIHByb2JsZW0uIExvb2tz CnRvIG1lIGl0IGlzIHRoZSBzYW1lIHNhdHVhdGlvbiB0aGF0IGFuIGFwcGxp Y2FpdG9uIGp1c3QKc3RvcCBkbyBBTlkgcmVjZWl2ZSwgZXZlbnR1YWxseSBh bGwgc3RyZWFtIGRhdGEgd2lsbCAiYmxvY2tlZCIuCk5vID8KCk9yLCBpZiB3 ZSBzdGFydGluZyBzdXBwb3J0IHN0cmVhbSBzcGVjaWZpYyByZWNlaXZlLCBj YW4gd2UKcHV0IGEgY2FwIG9uIGhvdyBtdWNoIHJ3bmQgYSBzdHJlYW0gaXMg YWxsb3dlZCB0byB0YWtlIG92ZXI/CgotWGlhb2RhbiBUYW5nClFTU0wgCg== ------------=_1071642532-17570-0-- From lehmann@ulticom.com Wed Dec 17 09:52:03 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA09802 for ; Wed, 17 Dec 2003 09:52:02 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWd1w-0005ac-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 09:52:04 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWd1t-0005aB-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 09:52:03 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWd1s-0005YG-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 09:52:01 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-2.cisco.com with ESMTP; 17 Dec 2003 06:54:40 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBHEp0At029047; Wed, 17 Dec 2003 06:51:01 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBHEnxKs024357 for sctp-impl-filtered; Wed, 17 Dec 2003 06:50:01 -0800 (PST) Message-Id: <200312171450.hBHEnxKs024357@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071672599-24349-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path.Max.Retrans - inactive when reaches or exceeds? List-Id: sctp-impl To: SCTP Implementors From: David Lehmann X-Nails-Approved: lehmann@ulticom.com Date: Wed, 17 Dec 2003 09:38:25 -0500 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071672599-24349-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Randall Stewart (cisco) wrote: > David Lehmann wrote: > >> Sections 8.2 and 8.3 seem to contradict each other. >> 8.2 says to mark the destination inactive when the error >> count _exceeds_ Path.Max.Retrans and 8.3 says >> to do it when it _reaches_ Path.Max.Retrans. >> ... >> Which is correct? >> > We will need to fix this in the next I-G issuance... as far > as which is correct... I don't think it really makes that much difference. > > Reaches makes more sense to me.. but I don't think it matters.. maybe > others could give their opinion.. So what was the final decision? -- David Lehmann Ulticom, Inc. AOL/Yahoo IM: davidULCM 1020 Briggs Road 1-856-787-2729 Mt. Laurel, NJ 08054 USA ------------=_1071672599-24349-0-- From rrs@cisco.com Wed Dec 17 10:07:55 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA11433 for ; Wed, 17 Dec 2003 10:07:55 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWdHI-0006wL-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 10:07:56 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWdHE-0006vV-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 10:07:56 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWdHD-0006qi-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 10:07:51 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-3.cisco.com with ESMTP; 17 Dec 2003 07:10:52 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBHF6rZ0007239; Wed, 17 Dec 2003 07:06:54 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBHF6NKs024560 for sctp-impl-filtered; Wed, 17 Dec 2003 07:06:25 -0800 (PST) Message-Id: <200312171506.hBHF6NKs024560@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071673583-24558-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path.Max.Retrans - inactive when reaches or exceeds? List-Id: sctp-impl To: David Lehmann From: "Randall Stewart (cisco)" Cc: SCTP Implementors X-Nails-Approved: rrs@cisco.com Date: Wed, 17 Dec 2003 09:05:50 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071673583-24558-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary David Lehmann wrote: > Randall Stewart (cisco) wrote: > >> David Lehmann wrote: >> >>> Sections 8.2 and 8.3 seem to contradict each other. >>> 8.2 says to mark the destination inactive when the error >>> count _exceeds_ Path.Max.Retrans and 8.3 says >>> to do it when it _reaches_ Path.Max.Retrans. >>> ... >>> Which is correct? >>> >> We will need to fix this in the next I-G issuance... as far >> as which is correct... I don't think it really makes that much >> difference. >> >> Reaches makes more sense to me.. but I don't think it matters.. maybe >> others could give their opinion.. > > > So what was the final decision? > I don't think we had many opinions.. but I would say the limited consensus was for exceeds... Speak up those that disagree :-> R -- Randall R. Stewart ITD - Transport Technologies 815-477-2127(o) or 815-342-4222(c) ------------=_1071673583-24558-0-- From kacheong.poon@sun.com Wed Dec 17 11:59:09 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA20266 for ; Wed, 17 Dec 2003 11:59:08 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWf0w-0006sn-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 11:59:10 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWf0v-0006sa-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 11:59:10 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWf0v-0006r4-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 11:59:09 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBHGwHZ0018418; Wed, 17 Dec 2003 08:58:17 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBHGvmKs026114 for sctp-impl-filtered; Wed, 17 Dec 2003 08:57:50 -0800 (PST) Message-Id: <200312171657.hBHGvmKs026114@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071680268-26106-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" From: Kacheong Poon Cc: Michael Tuexen , sctp-impl@external.cisco.com X-Nails-Approved: kacheong.poon@sun.com Date: Wed, 17 Dec 2003 08:56:05 -0800 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071680268-26106-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Randall Stewart (cisco) wrote: >> It seems that system scheduling has very little to do >> with the issue, unless the app itself tries to use >> specific OS scheduling service to arrange different >> scheduling for itself. The worst case scenario you've >> given is that a stream-socket will continue to read from >> a specific stream, while other streams are "stuck" >> therefore casuing a SCTP protocol issue as the TSN does >> not move ahead. > > > > I think you describe the above a bit off.. what will happen > when a stream gets "stuck" is you would then reach a point > where the stream would hold all of the rwnd. TSN's can move > forward up until that point.. at which time the stack should > clamp the rwnd at 0 and drop all inbound data until > the "stuck" stream reads... Thanks! I switched the role of who is "stuck" (-: But the point I tried to make is still valid, it has little to do with the scheduling of the system. It has to do with the app itself. -- K. Poon. kacheong.poon@sun.com ------------=_1071680268-26106-0-- From cait@asomi.com Wed Dec 17 17:58:17 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA10709 for ; Wed, 17 Dec 2003 17:58:17 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWkcU-0006pg-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 17:58:18 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWkcT-0006pY-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 17:58:18 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWkcT-0006of-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 17:58:17 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBHMvGBN014435; Wed, 17 Dec 2003 14:57:17 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBHMv0Ks000798 for sctp-impl-filtered; Wed, 17 Dec 2003 14:57:02 -0800 (PST) Message-Id: <200312172257.hBHMv0Ks000798@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071701820-796-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Kacheong Poon From: Caitlin Bestler Cc: Michael Tuexen , sctp-impl@external.cisco.com, "Randall Stewart (cisco)" X-Nails-Approved: cait@asomi.com Date: Wed, 17 Dec 2003 13:38:18 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.7 required=5.0 tests=DATE_IN_PAST_03_06 autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071701820-796-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 16, 2003, at 2:10 AM, Kacheong Poon wrote: > Michael Tuexen wrote: > >>> Shouldn't this be the same as sending/receiving on a wrong >>> stream id? An error will be returned. And I don't think >>> an app which wants to use stream-socket will do the above. >>> This does not require special handling anyway. >>> >> This means that such a socket is not bidirectional, but >> unidirectional. > > Why? If an app picks stream number 1 to send one specific > kind of data, both sides of an association can then use a > stream-socket on stream number 1 to send and receive > these messages. Why isn't this bi-directional? The > "unidirectional" scenario above is similar to TCP's half > closed case. One side chooses to send only and the other > side chooses to receive only. But this does not make a TCP > connection undirectional by nature. It is the user which > makes it undirectional. There are two separate issues here: a) specifying receive buffers in a stream dependent fashion b) the order in which user messages are retrieved Yes, avoiding rwnd exhaustion is ultimately the responsibility of the application. After all, failure to issue *any* read will result in the same problem. Thread specific routines merely make it easier for the application to forget to have the necessary read issued. It also makes such a condition much less *obvious*. The most likely error would be when a thread thought that a given thread was idle, and hence did not issue a read. An border condition or an error might come up where a message was actually sent, resulting in pinning the rwnd down. For that reason, I believe an interface that allows the SCTP layer to deliver the completed message with the lowest TSN is highly desirable and preferable to an interface that lets the application cherry-pick messages in arbitrary order. I can think of two ways to do this: a) pass a "buffer set" when reading, essentially allowing zero or more stream-specific buffers followed by one default buffer. The return would indicate which, if any, message had been delivered. b) Requiring one socket to remain the "default" socket for the association, where streams that had not been "peeled off" would be delivered. When an application believed a stream to be idle, it would "close" that stream's socket, thereby returning control of that stream (or streams if bi-directional) to the association's default socket. ------------=_1071701820-796-0-- From bidulock@openss7.org Wed Dec 17 19:11:39 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA15281 for ; Wed, 17 Dec 2003 19:11:39 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWllU-0001n5-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:11:40 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWllS-0001mr-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:11:40 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWllS-0001mE-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:11:38 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-3.cisco.com with ESMTP; 17 Dec 2003 16:14:43 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBI0AZZ0025705; Wed, 17 Dec 2003 16:10:35 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBI0AJKs001768 for sctp-impl-filtered; Wed, 17 Dec 2003 16:10:19 -0800 (PST) Message-Id: <200312180010.hBI0AJKs001768@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071706203-1766-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: Path.Max.Retrans - inactive when reaches or exceeds? List-Id: sctp-impl To: "Randall Stewart (cisco)" From: "Brian F. G. Bidulock" Cc: David Lehmann , SCTP Implementors X-Nails-Approved: bidulock@openss7.org Date: Wed, 17 Dec 2003 17:06:42 -0700 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071706203-1766-0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: binary Randall, Yes, exceeds. --brian On Wed, 17 Dec 2003, Randall Stewart (cisco) wrote: > David Lehmann wrote: > > > Randall Stewart (cisco) wrote: > > > >> David Lehmann wrote: > >> > >>> Sections 8.2 and 8.3 seem to contradict each other. > >>> 8.2 says to mark the destination inactive when the error > >>> count _exceeds_ Path.Max.Retrans and 8.3 says > >>> to do it when it _reaches_ Path.Max.Retrans. > >>> ... > >>> Which is correct? > >>> > >> We will need to fix this in the next I-G issuance... as far > >> as which is correct... I don't think it really makes that much > >> difference. > >> > >> Reaches makes more sense to me.. but I don't think it matters.. maybe > >> others could give their opinion.. > > > > > > So what was the final decision? > > > I don't think we had many opinions.. but I would say > the limited consensus was for exceeds... > > Speak up those that disagree :-> > > R > > -- > Randall R. Stewart > ITD - Transport Technologies > 815-477-2127(o) or 815-342-4222(c) > > -- Brian F. G. Bidulock ¦ The reasonable man adapts himself to the ¦ bidulock@openss7.org ¦ world; the unreasonable one persists in ¦ http://www.openss7.org/ ¦ trying to adapt the world to himself. ¦ ¦ Therefore all progress depends on the ¦ ¦ unreasonable man. -- George Bernard Shaw ¦ ------------=_1071706203-1766-0-- From heron@cs.cmu.edu Wed Dec 17 19:21:07 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA15681 for ; Wed, 17 Dec 2003 19:21:07 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWluf-00028R-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:21:09 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWlue-00028K-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:21:08 -0500 Received: from sj-iport-4.cisco.com ([171.68.10.86]) by ietf-mx with esmtp (Exim 4.12) id 1AWlud-000274-00 for sctp-impl-archive@ietf.org; Wed, 17 Dec 2003 19:21:07 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBI0JmrX013035; Wed, 17 Dec 2003 16:19:48 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBI0JaKs001894 for sctp-impl-filtered; Wed, 17 Dec 2003 16:19:38 -0800 (PST) Message-Id: <200312180019.hBI0JaKs001894@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071706775-1892-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Gabriel Kaelin X-Nails-Approved: heron@cs.cmu.edu Date: Wed, 17 Dec 2003 19:17:39 -0500 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071706775-1892-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Michael Tuexen wrote: > let us see if people want this feature in the API. I'm not an implementer of SCTP, but I'm currently studying the applicability of SCTP to the Coda file system. To be able to peel off streams sounds interesting from a performance point of view, because Coda clients and servers run in user space: There is a lot of meta data being transported, but also files. Files don't need the application level attention during transfer, therefore it would be nice if everything could happen in kernel space, e.g. by using sendfile(). However, this requires a dedicated socket, so to peel one stream temporarily off would really be nice. In this case I can't see a risk of getting stuck in rwnd issues. Well, this is a selfish point of view, but maybe you agree, that this is a good application of this stream peeling off. Besides, the whole discussion about giving more responsibility to the application layer is the fundamental question about mechanism vs. policy, isn't it? Me personally, I'm for mechanism, it usually has a longer life than policy. ;-) There is another point, I was just thinking about. I want to use the stream index to demultiplex and I was just confused because Michael wrote about (mis)using the stream ID. Anyway, because there can be several associations, I actually need to demultiplex based on association and stream. Now, how can this be done in an efficient manner? The best way I could think of, was to use a hash table with the association ID or a separate ID that is transferred in user data. It would be really efficient, if I could save a pointer in an association that would point me directly to a corresponding data structure. Or is there already any mechanism that supports demultiplexing from associations? Do you instead have to peel off associations (which does not really solve the problem entirely, though)? Gabriel ------------=_1071706775-1892-0-- From rrs@cisco.com Thu Dec 18 07:39:00 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA17979 for ; Thu, 18 Dec 2003 07:39:00 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AWxQi-0004eK-00 for sctp-impl-archive@ietf.org; Thu, 18 Dec 2003 07:39:00 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AWxQg-0004e3-00 for sctp-impl-archive@ietf.org; Thu, 18 Dec 2003 07:38:59 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AWxQg-0004dZ-00 for sctp-impl-archive@ietf.org; Thu, 18 Dec 2003 07:38:58 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBICbmAt029668; Thu, 18 Dec 2003 04:37:48 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBICbLKs011607 for sctp-impl-filtered; Thu, 18 Dec 2003 04:37:22 -0800 (PST) Message-Id: <200312181237.hBICbLKs011607@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071751040-11605-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Gabriel Kaelin From: "Randall Stewart (cisco)" Cc: sctp-impl@external.cisco.com X-Nails-Approved: rrs@cisco.com Date: Thu, 18 Dec 2003 06:36:49 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071751040-11605-0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Gabriel: Comments below... Gabriel Kaelin wrote: > Michael Tuexen wrote: > >> let us see if people want this feature in the API. > > > I'm not an implementer of SCTP, but I'm currently studying the > applicability of SCTP to the Coda file system. To be able to peel off > streams sounds interesting from a performance point of view, because > Coda clients and servers run in user space: > > There is a lot of meta data being transported, but also files. Files > don't need the application level attention during transfer, therefore > it would be nice if everything could happen in kernel space, e.g. by > using sendfile(). However, this requires a dedicated socket, so to > peel one stream temporarily off would really be nice. > > In this case I can't see a risk of getting stuck in rwnd issues. Well, > this is a selfish point of view, but maybe you agree, that this is a > good application of this stream peeling off. > Besides, the whole discussion about giving more responsibility to the > application layer is the fundamental question about mechanism vs. > policy, isn't it? Me personally, I'm for mechanism, it usually has a > longer life than policy. ;-) I agree... the mechanism sounds nice.. a lot of code I am thinking... but of course I still have not thought it out yet :-0 > > > > There is another point, I was just thinking about. I want to use the > stream index to demultiplex and I was just confused because Michael > wrote about (mis)using the stream ID. > Anyway, because there can be several associations, I actually need to > demultiplex based on association and stream. Now, how can this be done > in an efficient manner? The best way I could think of, was to use a > hash table with the association ID or a separate ID that is > transferred in user data. It would be really efficient, if I could > save a pointer in an association that would point me directly to a > corresponding data structure. > Or is there already any mechanism that supports demultiplexing from > associations? Do you instead have to peel off associations (which does > not really solve the problem entirely, though)? I don't know of any mechanism other than the association ID for demux'ing multiple associations from one socket.. it was what the assoc-id field was all about i.e. a unique system identifier for a association. You could easily put that with the stream number into a hash table I would think.. Another thing one could do is mis-use the PPID field in every send. It's a 32 bit integer that is passed opaquely through the data transfer... I guess you could have the sender place an ID in this field to represent something you could use for an index.... don't know... it actually is a mis-use of the feild though. It is supposed to be a way for sniffers and such to recognize what type of protocol is being ran over the wire... R > > > Gabriel > > -- Randall R. Stewart ITD - Transport Technologies 815-477-2127(o) or 815-342-4222(c) ------------=_1071751040-11605-0-- From cait@asomi.com Fri Dec 19 12:35:28 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17194 for ; Fri, 19 Dec 2003 12:35:28 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AXOXC-000211-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:35:30 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AXOXA-00020u-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:35:29 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AXOXA-00020B-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:35:28 -0500 Received: from sj-core-1.cisco.com (171.71.177.237) by sj-iport-5.cisco.com with ESMTP; 19 Dec 2003 17:35:40 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-1.cisco.com (8.12.9/8.12.6) with ESMTP id hBJHY0C6020260; Fri, 19 Dec 2003 09:34:01 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBJHX8Ks004216 for sctp-impl-filtered; Fri, 19 Dec 2003 09:33:10 -0800 (PST) Message-Id: <200312191733.hBJHX8Ks004216@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071855188-4214-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Gabriel Kaelin From: Caitlin Bestler Cc: sctp-impl@external.cisco.com X-Nails-Approved: cait@asomi.com Date: Fri, 19 Dec 2003 11:31:18 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.3 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071855188-4214-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 17, 2003, at 6:17 PM, Gabriel Kaelin wrote: > Michael Tuexen wrote: >> let us see if people want this feature in the API. > > I'm not an implementer of SCTP, but I'm currently studying the > applicability of SCTP > to the Coda file system. To be able to peel off streams sounds > interesting from a > performance point of view, because Coda clients and servers run in > user space: > > There is a lot of meta data being transported, but also files. Files > don't need the > application level attention during transfer, therefore it would be > nice if everything > could happen in kernel space, e.g. by using sendfile(). However, this > requires a dedicated > socket, so to peel one stream temporarily off would really be nice. > > In this case I can't see a risk of getting stuck in rwnd issues. Well, > this is a selfish > point of view, but maybe you agree, that this is a good application of > this stream peeling off. > Besides, the whole discussion about giving more responsibility to the > application layer is > the fundamental question about mechanism vs. policy, isn't it? Me > personally, I'm for mechanism, > it usually has a longer life than policy. ;-) > Generally, sharing an association across process boundaries is nearly as unwise as sharing the space for a general purpose heap across process boundaries. If all of the applications behave you do indeed have some savings, but that is a very big if. But, if the association belongs to the "file system", and what you want to do is target data chunks by stream directly into user specified buffers (for zero copy), then I would see per-stream buffers and SCTP as being useful. The key is that once ordered messages are involved, relying on *multiple* distinct user processes to dequeue them is at the minimum very risky. Even dealing only with unordered chunks, the DDP mapping to SCTP was envisioning a single association per user process. That can still be quite useful. If the file system client is an email or web daemon it is quite likely to have multiple concurrent file transfers pending for a variety of clients. I'm not certain if there is enough of a benefit to do this with just SCTP, as opposed to using RDMA over SCTP. But there is definitely a benefit to any client that has multiple pending file transfers to using SCTP with per-stream buffering. Being forced to use intermediate buffers could even make the solution less efficient that the use of multiple TCP connections. ------------=_1071855188-4214-0-- From cait@asomi.com Fri Dec 19 12:40:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA17363 for ; Fri, 19 Dec 2003 12:40:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AXObk-00028I-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:40:12 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AXObj-00028B-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:40:11 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AXObj-00027c-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 12:40:11 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 19 Dec 2003 17:40:21 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBJHd3lx027307; Fri, 19 Dec 2003 09:39:04 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBJHcmKs004291 for sctp-impl-filtered; Fri, 19 Dec 2003 09:38:50 -0800 (PST) Message-Id: <200312191738.hBJHcmKs004291@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071855528-4289-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: SCTP Implementors From: Caitlin Bestler X-Nails-Approved: cait@asomi.com Date: Fri, 19 Dec 2003 11:36:58 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.2 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071855528-4289-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary A general comment about possible stream-specific de-multiplexing local interfaces: any such interface should probably allow ordered and unordered messages to be handled differently. Anticipating that a sequence of ordered messages on one stream should be received into a single buffer, or matching sequence of buffers is a natural fit for many applications -- especially large file transfers where use of a single very large message is inappropriate for buffering and/or flow control reasons. But you wouldn't want an unordered message to be plopped down into that buffer sequence arbitrarily. An application that mixes ordered and unordered messages on the same stream is probably using the unordered messages for alerts or exceptions. ------------=_1071855528-4289-0-- From heron@cs.cmu.edu Fri Dec 19 18:03:54 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA01775 for ; Fri, 19 Dec 2003 18:03:54 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AXTf1-000701-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 18:03:55 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AXTf0-0006zu-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 18:03:54 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AXTf0-0006yd-00 for sctp-impl-archive@ietf.org; Fri, 19 Dec 2003 18:03:54 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-2.cisco.com with ESMTP; 19 Dec 2003 15:07:02 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBJN31VM014620; Fri, 19 Dec 2003 15:03:01 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBJN2cKs008411 for sctp-impl-filtered; Fri, 19 Dec 2003 15:02:40 -0800 (PST) Message-Id: <200312192302.hBJN2cKs008411@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1071874957-8409-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: sctp-impl@external.cisco.com From: Gabriel Kaelin X-Nails-Approved: heron@cs.cmu.edu Date: Fri, 19 Dec 2003 18:00:52 -0500 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1071874957-8409-0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary I realized that I forgot to send it to the mailing list as well... ----- I put my comments below... Caitlin Bestler wrote: > On Dec 17, 2003, at 6:17 PM, Gabriel Kaelin wrote: > >> Michael Tuexen wrote: >> >>> let us see if people want this feature in the API. >> >> >> I'm not an implementer of SCTP, but I'm currently studying the applicability of SCTP >> to the Coda file system. To be able to peel off streams sounds interesting from a >> performance point of view, because Coda clients and servers run in user space: >> >> There is a lot of meta data being transported, but also files. Files don't need the >> application level attention during transfer, therefore it would be nice if everything >> could happen in kernel space, e.g. by using sendfile(). However, this requires a dedicated >> socket, so to peel one stream temporarily off would really be nice. > > Generally, sharing an association across process boundaries is nearly as unwise as sharing > the space for a general purpose heap across process boundaries. If all of the applications > behave you do indeed have some savings, but that is a very big if. In the general case, I agree with you. In Coda, the association would be governed only by Coda's RPC implementation, which allows concurrent RPCs in one single process. The file transfers, which are considered as RPC side effects, happen under tight control of RPC2 as well. The idea would be, to use for example a kernel thread dedicated to feeding or reading from the peeled off socket. > But, if the association belongs to the "file system", and what you want to do is target > data chunks by stream directly into user specified buffers (for zero copy), then I would > see per-stream buffers and SCTP as being useful. The key is that once ordered messages > are involved, relying on *multiple* distinct user processes to dequeue them is at > the minimum very risky. I can imagine that per-stream buffers could allow me to move large data transfers mostly into kernel space, especially if I can provide user specified per-stream send buffers as well. Unordered packets could still take the standard path for delivery to the application. > > Even dealing only with unordered chunks, the DDP mapping to SCTP was envisioning a single > association per user process. That can still be quite useful. If the file system client > is an email or web daemon it is quite likely to have multiple concurrent file transfers > pending for a variety of clients. I'm not certain if there is enough of a benefit to > do this with just SCTP, as opposed to using RDMA over SCTP. But there is definitely > a benefit to any client that has multiple pending file transfers to using SCTP with > per-stream buffering. Being forced to use intermediate buffers could even make the > solution less efficient that the use of multiple TCP connections. ------------=_1071874957-8409-0-- From cait@asomi.com Tue Dec 23 12:42:55 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28462 for ; Tue, 23 Dec 2003 12:42:55 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYqYb-0006cN-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 12:42:57 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqWX-0006aC-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 12:40:50 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AYqWX-0006Zv-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 12:40:49 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-3.cisco.com with ESMTP; 23 Dec 2003 09:45:01 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBNHdbJ6018307; Tue, 23 Dec 2003 09:39:38 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNHcuKs019315 for sctp-impl-filtered; Tue, 23 Dec 2003 09:38:58 -0800 (PST) Message-Id: <200312231738.hBNHcuKs019315@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072201136-19313-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Michael Tuexen From: Caitlin Bestler Cc: sctp-impl@external.cisco.com, Gabriel Kaelin X-Nails-Approved: cait@asomi.com Date: Tue, 23 Dec 2003 11:37:11 -0600 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.1 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072201136-19313-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary On Dec 22, 2003, at 9:51 AM, Michael Tuexen wrote: > Dear all, > > when you provide per stream buffer, then you must make sure that > you have buffer for all streams. If only one steam runs out of buffer > than you have a problem. > > Best regards > Michael I agree. Any API providing per-stream buffering should make it difficult or even impossible to provide buffers for *some* of the streams while making other streams wait for buffers to be provided later. Not meeting that criteria leaves the application vulnerable to more subtle errors, and slightly increases the complexity of the API layer code that must determine if a message is deliverable in that the availability of a target buffer must be checked in a stream/ordered-vs-unordered fashion. If there is *always* a buffer for *any* message, or no buffers at all, the logic can be slightly simpler for both client and API implementation. ------------=_1072201136-19313-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 23 13:14:04 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29249 for ; Tue, 23 Dec 2003 13:14:04 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYr2k-0007T1-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:14:06 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqyx-0007Mf-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:10:11 -0500 Received: from sj-iport-2-in.cisco.com ([171.71.176.71] helo=sj-iport-2.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AYqv8-0007Hg-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:06:15 -0500 Received: from sj-core-5.cisco.com (171.71.177.238) by sj-iport-2.cisco.com with ESMTP; 23 Dec 2003 10:10:07 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBNI5NJ6009610; Tue, 23 Dec 2003 10:05:24 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNI5IKs019684 for sctp-impl-filtered; Tue, 23 Dec 2003 10:05:20 -0800 (PST) Message-Id: <200312231805.hBNI5IKs019684@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072202717-19679-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: From: Michael Tuexen Cc: "Kacheong Poon" , , "Randall Stewart (cisco)" X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 22 Dec 2003 17:14:00 +0100 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.4 required=5.0 tests=DATE_IN_PAST_12_24 autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072202717-19679-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary See my comments below. Best regards Michael On Dec 17, 2003, at 6:43 AM, wrote: > > > > "Randall Stewart (cisco)" said: > >> I think you describe the above a bit off.. what will happen >> when a stream gets "stuck" is you would then reach a point >> where the stream would hold all of the rwnd. TSN's can move >> forward up until that point.. at which time the stack should >> clamp the rwnd at 0 and drop all inbound data until >> the "stuck" stream reads... >> >> I just want us all to be clear on what SHOULD occur :-> > > Great! That was what my understanding is. So the worst case > is, if user stopped receiving from stream 1, then datas for > stream 1 will start "eating" rwnd, until some point that > total rwnd are all used up for stream 1's data, then other > streams get "blocked". > > But again, I'd argu that's the application's problem. Looks > to me it is the same satuation that an applicaiton just > stop do ANY receive, eventually all stream data will "blocked". > No ? > The point is who does the scheduling. Asume that I'm using different threads for different streams than the thread scheduling comes into the game. Something the application writer has not under control... So one has to think about what happens if the message processing time on one stream is 10 times the one on the other stream. How will the streams be scheduled? > Or, if we starting support stream specific receive, can we > put a cap on how much rwnd a stream is allowed to take over? > This is talking about oer stream flow control. Having separate buffers AND a per stream flow control works. But this is not part of the protocol and we do not want to change it. > -Xiaodan Tang > QSSL > ------------=_1072202717-19679-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 23 13:14:05 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29267 for ; Tue, 23 Dec 2003 13:14:05 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYr2l-0007TE-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:14:07 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqyy-0007Ms-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:10:12 -0500 Received: from sj-iport-1-in.cisco.com ([171.71.176.70] helo=sj-iport-1.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AYqv9-0007Hh-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:06:15 -0500 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-5.cisco.com (8.12.9/8.12.6) with ESMTP id hBNI5UJ6009740; Tue, 23 Dec 2003 10:05:31 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNI5CKs019673 for sctp-impl-filtered; Tue, 23 Dec 2003 10:05:14 -0800 (PST) Message-Id: <200312231805.hBNI5CKs019673@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072202712-19663-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Gabriel Kaelin X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 22 Dec 2003 16:51:57 +0100 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.2 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072202712-19663-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Dear all, when you provide per stream buffer, then you must make sure that you have buffer for all streams. If only one steam runs out of buffer than you have a problem. Best regards Michael On Dec 19, 2003, at 6:31 PM, Caitlin Bestler wrote: > > On Dec 17, 2003, at 6:17 PM, Gabriel Kaelin wrote: > >> Michael Tuexen wrote: >>> let us see if people want this feature in the API. >> >> I'm not an implementer of SCTP, but I'm currently studying the >> applicability of SCTP >> to the Coda file system. To be able to peel off streams sounds >> interesting from a >> performance point of view, because Coda clients and servers run in >> user space: >> >> There is a lot of meta data being transported, but also files. Files >> don't need the >> application level attention during transfer, therefore it would be >> nice if everything >> could happen in kernel space, e.g. by using sendfile(). However, this >> requires a dedicated >> socket, so to peel one stream temporarily off would really be nice. >> >> In this case I can't see a risk of getting stuck in rwnd issues. >> Well, this is a selfish >> point of view, but maybe you agree, that this is a good application >> of this stream peeling off. >> Besides, the whole discussion about giving more responsibility to the >> application layer is >> the fundamental question about mechanism vs. policy, isn't it? Me >> personally, I'm for mechanism, >> it usually has a longer life than policy. ;-) >> > > Generally, sharing an association across process boundaries is nearly > as unwise as sharing > the space for a general purpose heap across process boundaries. If all > of the applications > behave you do indeed have some savings, but that is a very big if. > > But, if the association belongs to the "file system", and what you > want to do is target > data chunks by stream directly into user specified buffers (for zero > copy), then I would > see per-stream buffers and SCTP as being useful. The key is that once > ordered messages > are involved, relying on *multiple* distinct user processes to dequeue > them is at > the minimum very risky. > > Even dealing only with unordered chunks, the DDP mapping to SCTP was > envisioning a single > association per user process. That can still be quite useful. If the > file system client > is an email or web daemon it is quite likely to have multiple > concurrent file transfers > pending for a variety of clients. I'm not certain if there is enough > of a benefit to > do this with just SCTP, as opposed to using RDMA over SCTP. But there > is definitely > a benefit to any client that has multiple pending file transfers to > using SCTP with > per-stream buffering. Being forced to use intermediate buffers could > even make the > solution less efficient that the use of multiple TCP connections. > ------------=_1072202712-19663-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 23 13:14:06 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29284 for ; Tue, 23 Dec 2003 13:14:06 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYr2n-0007TP-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:14:09 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqz0-0007NK-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:10:15 -0500 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 1AYqvB-0007Hj-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:06:17 -0500 Received: from sj-core-2.cisco.com (171.71.177.254) by sj-iport-3.cisco.com with ESMTP; 23 Dec 2003 10:10:30 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-2.cisco.com (8.12.9/8.12.6) with ESMTP id hBNI5RVM012601; Tue, 23 Dec 2003 10:05:28 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNI5CKs019674 for sctp-impl-filtered; Tue, 23 Dec 2003 10:05:14 -0800 (PST) Message-Id: <200312231805.hBNI5CKs019674@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072202712-19666-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Caitlin Bestler From: Michael Tuexen Cc: SCTP Implementors X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 22 Dec 2003 16:53:43 +0100 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.3 required=5.0 tests=AWL,DATE_IN_PAST_12_24 autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072202712-19666-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary So you are suggesting 2 buffers per stream? Best regards Michael On Dec 19, 2003, at 6:36 PM, Caitlin Bestler wrote: > A general comment about possible stream-specific de-multiplexing local > interfaces: > any such interface should probably allow ordered and unordered > messages to be handled > differently. Anticipating that a sequence of ordered messages on one > stream should > be received into a single buffer, or matching sequence of buffers is a > natural fit > for many applications -- especially large file transfers where use of > a single very > large message is inappropriate for buffering and/or flow control > reasons. > > But you wouldn't want an unordered message to be plopped down into > that buffer > sequence arbitrarily. An application that mixes ordered and unordered > messages > on the same stream is probably using the unordered messages for alerts > or exceptions. > ------------=_1072202712-19666-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 23 13:14:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29304 for ; Tue, 23 Dec 2003 13:14:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYr2q-0007U3-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:14:12 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqz7-0007Of-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:10:22 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AYqvR-0007Hl-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:06:33 -0500 Received: from sj-core-3.cisco.com (171.68.223.137) by sj-iport-5.cisco.com with ESMTP; 23 Dec 2003 18:07:35 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id hBNI5VQw011308; Tue, 23 Dec 2003 10:05:34 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNI5CKs019675 for sctp-impl-filtered; Tue, 23 Dec 2003 10:05:14 -0800 (PST) Message-Id: <200312231805.hBNI5CKs019675@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072202712-19669-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: Gabriel Kaelin From: Michael Tuexen Cc: sctp-impl@external.cisco.com X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 22 Dec 2003 17:07:47 +0100 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.1 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072202712-19669-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Dear all, see my comments below. Best regards Michael On Dec 18, 2003, at 1:17 AM, Gabriel Kaelin wrote: > Michael Tuexen wrote: >> let us see if people want this feature in the API. > > I'm not an implementer of SCTP, but I'm currently studying the > applicability of SCTP to the Coda file system. To be able to peel off > streams sounds interesting from a performance point of view, because > Coda clients and servers run in user space: > > There is a lot of meta data being transported, but also files. Files > don't need the application level attention during transfer, therefore > it would be nice if everything could happen in kernel space, e.g. by > using sendfile(). However, this requires a dedicated socket, so to > peel one stream temporarily off would really be nice. > So you want the complete file to be transfered without the user space involved and only when the file is completely copied to get it one user land interaction? So this would go fast in the kernel. But your user land client/server may not be able to handle the other traffic then, because it is not scheduled. So the rwnd will close. It is not decoupled. You would need a per stream flow control or separate streams. Another problem I see is the following. At the moment you get one user message per receive call. I'm assuming that this semantic will stay the same. This would mean that you have a lot of context switches unless you use the fragmentation feature of SCTP which will also bring in the HOL problem, what you do not want. Sou you would also need a receive_multiple_messages call... > In this case I can't see a risk of getting stuck in rwnd issues. Well, > this is a selfish point of view, but maybe you agree, that this is a > good application of this stream peeling off. > Besides, the whole discussion about giving more responsibility to the > application layer is the fundamental question about mechanism vs. > policy, isn't it? Me personally, I'm for mechanism, it usually has a > longer life than policy. ;-) > > > There is another point, I was just thinking about. I want to use the > stream index to demultiplex and I was just confused because Michael > wrote about (mis)using the stream ID. In my perspective, the SID is only used to ensure in-sequence delivery, and this is done in a way that the sender choses a SID only to minimize the HOL. There should be no need at the receiving end to look at the SID. You can demultiplex several upper layer protocols by using the PPID. But if you have to demultiplex inside one layer you should use your own field in your layer. This makes your protocol work even if the SCTP negotiates only one stream per direction. > Anyway, because there can be several associations, I actually need to > demultiplex based on association and stream. Now, how can this be done > in an efficient manner? The best way I could think of, was to use a > hash table with the association ID or a separate ID that is > transferred in user data. It would be really efficient, if I could > save a pointer in an association that would point me directly to a > corresponding data structure. > Or is there already any mechanism that supports demultiplexing from > associations? Do you instead have to peel off associations (which does > not really solve the problem entirely, though)? > > Gabriel > ------------=_1072202712-19669-0-- From Michael.Tuexen@micmac.franken.de Tue Dec 23 13:14:10 2003 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA29308 for ; Tue, 23 Dec 2003 13:14:10 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYr2r-0007U8-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:14:13 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYqz8-0007On-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:10:23 -0500 Received: from sj-iport-5.cisco.com ([171.68.10.87]) by ietf-mx with esmtp (Exim 4.12) id 1AYqvS-0007Hl-00 for sctp-impl-archive@ietf.org; Tue, 23 Dec 2003 13:06:34 -0500 Received: from sj-core-4.cisco.com (171.68.223.138) by sj-iport-5.cisco.com with ESMTP; 23 Dec 2003 18:07:37 +0000 Received: from nailgun.cisco.com (nailgun.cisco.com [171.69.11.147]) by sj-core-4.cisco.com (8.12.6/8.12.6) with ESMTP id hBNI5Slx019909; Tue, 23 Dec 2003 10:05:29 -0800 (PST) Received: from nailscatch (localhost [127.0.0.1]) by nailgun.cisco.com (8.12.2/8.12.2) with ESMTP id hBNI54Ks019656 for sctp-impl-filtered; Tue, 23 Dec 2003 10:05:06 -0800 (PST) Message-Id: <200312231805.hBNI54Ks019656@nailgun.cisco.com> Content-Type: multipart/alternative; boundary="----------=_1072202704-19654-0" Content-Transfer-Encoding: binary Mime-Version: 1.0 X-Mailer: NAILS X-Brought-To-You-BY: djohnsen, stharms, Enterprise Messaging, and the letter "E" Subject: Re: sctp_peeloff() List-Id: sctp-impl To: "Randall Stewart (cisco)" From: Michael Tuexen Cc: sctp-impl@external.cisco.com, Gabriel Kaelin X-Nails-Approved: Michael.Tuexen@micmac.franken.de Date: Mon, 22 Dec 2003 16:49:13 +0100 X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.1 required=5.0 tests=AWL autolearn=no version=2.60 This is a multi-part message in MIME format... ------------=_1072202704-19654-0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Disposition: inline Content-Transfer-Encoding: binary Hi all, see my comments below. Best regards Michael On Dec 18, 2003, at 1:36 PM, Randall Stewart (cisco) wrote: > Gabriel: > > Comments below... > > Gabriel Kaelin wrote: > >> Michael Tuexen wrote: >> >>> let us see if people want this feature in the API. >> >> >> I'm not an implementer of SCTP, but I'm currently studying the >> applicability of SCTP to the Coda file system. To be able to peel off >> streams sounds interesting from a performance point of view, because >> Coda clients and servers run in user space: >> >> There is a lot of meta data being transported, but also files. Files >> don't need the application level attention during transfer, therefore >> it would be nice if everything could happen in kernel space, e.g. by >> using sendfile(). However, this requires a dedicated socket, so to >> peel one stream temporarily off would really be nice. >> >> In this case I can't see a risk of getting stuck in rwnd issues. >> Well, this is a selfish point of view, but maybe you agree, that this >> is a good application of this stream peeling off. >> Besides, the whole discussion about giving more responsibility to the >> application layer is the fundamental question about mechanism vs. >> policy, isn't it? Me personally, I'm for mechanism, it usually has a >> longer life than policy. ;-) > > > I agree... the mechanism sounds nice.. a lot of code I am thinking... > but of > course I still have not thought it out yet :-0 > >> >> >> >> There is another point, I was just thinking about. I want to use the >> stream index to demultiplex and I was just confused because Michael >> wrote about (mis)using the stream ID. >> Anyway, because there can be several associations, I actually need to >> demultiplex based on association and stream. Now, how can this be >> done in an efficient manner? The best way I could think of, was to >> use a hash table with the association ID or a separate ID that is >> transferred in user data. It would be really efficient, if I could >> save a pointer in an association that would point me directly to a >> corresponding data structure. >> Or is there already any mechanism that supports demultiplexing from >> associations? Do you instead have to peel off associations (which >> does not really solve the problem entirely, though)? > > I don't know of any mechanism other than the association ID for > demux'ing multiple > associations from one socket.. it was what the assoc-id field was all > about i.e. a > unique system identifier for a association. You could easily put that > with the stream > number into a hash table I would think.. > > Another thing one could do is mis-use the PPID field in every send. > It's a 32 bit > integer that is passed opaquely through the data transfer... I guess > you could > have the sender place an ID in this field to represent something you > could > use for an index.... don't know... it actually is a mis-use of the > feild though. It > is supposed to be a way for sniffers and such to recognize what type of > protocol is being ran over the wire... > For multiplexing multiple protocols in one association this is not something I would call 'misuse'. This could be done in SIGTRAN and we is it that way in RSerPool to multiplex the ULP with ASAP. And yes, ethereal uses this field to detect the ULP of SCTP and this is a mechanism independent from the port numbers. Whoever that invented, it is very helpful for ethereal and other packet analyzers. > R > >> >> >> Gabriel >> >> > > > -- > Randall R. Stewart > ITD - Transport Technologies > 815-477-2127(o) or 815-342-4222(c) > > ------------=_1072202704-19654-0--