From nobody Thu May 1 08:18:12 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A47931A6F74 for ; Thu, 1 May 2014 08:18:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=unavailable Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oRj6yzJ-MVqN for ; Thu, 1 May 2014 08:18:07 -0700 (PDT) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) by ietfa.amsl.com (Postfix) with ESMTP id 88A701A08E9 for ; Thu, 1 May 2014 08:18:07 -0700 (PDT) Received: by mail-ig0-f181.google.com with SMTP id h18so604497igc.2 for ; Thu, 01 May 2014 08:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ybJDw+Nc38no8RFVLnrHBpqObEc7ybLf9CZueAAuppw=; b=MTXu3oDzW9B7e1qp2xLnFLYp99HUH0qjx+vH3cGwtTXrOW7i21x/v8VtML3wV9ZfSl qgj3CDE+N6Rjh1A+GWE76mD3yo3sBX68n/ZlCyYOFJmEuGuGkxmmDmgEw04KXy8nPRtp p2XJCAhyjMZVgYnMOWe64gXq95KNmNtXXJYNTgTAY79DNU6lZHfj3gY+3UZDbgtXODom tq64oN5naU3f2/wP16kx6K5uPNSIqUKUNHPVXfK+ptlX0Y+HBOVw5VfKcYa9nYAHaqBm Ms7cfZngtDftX7tLndPGGX9xo5+3tiSob1lwYkAQueIiOV3HgSHHga7yek19i0N0m1Hi pt9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ybJDw+Nc38no8RFVLnrHBpqObEc7ybLf9CZueAAuppw=; b=diVx3FLkHDWK/6xwLicX2QT6CADreBWZ03TphlWUJCBL7iJXu6peve/HF5Xmqs9O47 dWtBIRjAbTn3w0lsZOY+zCykYEDfQUQWYyJ6KDTjB9M9cm6RFLIaSG+D1myVoDgYnl/G QD5PDf4J7oA6GJtbc6KJVTUQOvfo0qoY9oZpBUvUXx/ohIlM26KE+ixWkGpBPu/P9EpC nI93FeAotLwy3HuYAalXhYvn86wfUbxxa6+B3n8du2m4Jt5P4MCmbGpPBJmUUily9eas XNe01LfGE006zJYPK1ICchKDq8FJLR26wzWlxmkFHHFMevY5qyX0a2TT+Y+FMEEzlcPT 0Q1A== X-Gm-Message-State: ALoCoQk22RhTzCDK8oJ2OmkHw1MWaLUyjHIsXLJG2Nq7vUcV1Dv20H0ItXASbWksv90xfnYXNqUn MIME-Version: 1.0 X-Received: by 10.42.62.196 with SMTP id z4mr10416428ich.49.1398957484670; Thu, 01 May 2014 08:18:04 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Thu, 1 May 2014 08:18:04 -0700 (PDT) In-Reply-To: <663894c8432a40f6ada92247831f7a2d@AM3PR03MB612.eurprd03.prod.outlook.com> References: <663894c8432a40f6ada92247831f7a2d@AM3PR03MB612.eurprd03.prod.outlook.com> Date: Thu, 1 May 2014 08:18:04 -0700 Message-ID: From: Tom Herbert To: Alexander Vainshtein Content-Type: multipart/alternative; boundary=90e6ba614a76ee1f9f04f8582b3e Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/BfYlMpYTsuUgjRwYo57fNVHgPus Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , "Larry Kreeger \(kreeger\)" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] [nvo3] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 15:18:10 -0000 --90e6ba614a76ee1f9f04f8582b3e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, May 1, 2014 at 1:10 AM, Alexander Vainshtein < Alexander.Vainshtein@ecitele.com> wrote: > Tom, Larry and all, > > I do not want to argue for or against UDP checksums in VXVLAN . > > Just want to point that ability of HW to check/generate the IPv4 checksum= s > does not, IMHO and FWIW, that the same HW can easily do UDP checksums. > > > > AFAIK, IPv4 forwarding HW in most cases has access to a relatively small > part at the beginning of the packet. This is enough to deal with the IPv= 4 > header but is not enough to compute UDP checksums =E2=80=93 because these= include > the entire packet payload. > > > Thanks Sasha, this is a good point. Similar concerns have been mentioned about the difficulty of switches to process packets with IP options. The solutions there were to split into a fast and slow data path based on the presence of options, or some implementations probably just dropped packets with options. In either case protocol conformance is maintained and there was no need to change IP protocol. I'd suggest a similar approach can be done where zero checksum value in a UDP packet takes an expedited data path= . Also, this discussion really seems to be specific to switch HW. I think you'd be hard pressed to find a NIC that does not provide L4 checksum offload. Retrofitting those NICs to provide checksum for encapsulation (really a matter of supporting >1 csums in a packet) should mostly be an API issue and even then most of the complexity in on TX side not RX. > This does not mean that HW-assisted computation of UDP ckechsums is not > possible =E2=80=93 only that it would be rthogonal to HW-assisted computa= tion of > IPv4 header checksums. > > My 2c, > > Sasha > > Email: Alexander.Vainshtein@ecitele.com > > Mobile: 054-9266302 > > > > *From:* nvo3 [mailto:nvo3-bounces@ietf.org] *On Behalf Of *Tom Herbert > *Sent:* Thursday, May 01, 2014 4:48 AM > *To:* Larry Kreeger (kreeger) > *Cc:* tofoo@ietf.org; nvo3@ietf.org; mallik_mahalingam@yahoo.com; > ddutt.ietf@hobbesdutt.com > *Subject:* Re: [nvo3] [Tofoo] VXLAN (UDP tunnel protocols) and non-zero > checksums > > > > > > > > On Wed, Apr 30, 2014 at 5:13 PM, Larry Kreeger (kreeger) < > kreeger@cisco.com> wrote: > > Hi Tom, > > I'll give you my perspective on why I feel the behavior described in the > VXLAN draft is a good thing. First, it is my assumption that some > implementations (e.g. many hardware implementations), will not implement > checksum generation, nor checksum validation. I believe this is an > implementation reality. If implementations are required to check a > non-zero checksum, but can't actually do it, what alternatives do they > have? > > > > Barring that the implementations you're referring to only implement IPv6, > these implementations must be already be doing checksum validation for IP= v4 > header-- so the checksum logic must have been implemented and I'm not sur= e > the argument that this HW can't calculate the UDP csum would have much > merit. A specific example implementation would be nice here, for instance > in the case that the stack decapsulates the checksum can always be done i= n > SW if verification is not provided by the device. > > > > Also, as I pointed out, the UDP checksum is *not* useless in VXLAN. In an > L3 routed network there is nothing that protects the vni from corruption > expect possibly the UDP checksum. Without any additional security > mechanisms is VXLAN header (like a cookie), the only way I could deploy > VXLAN is with checksums enabled. So in my opinion, the draft should be > encouraging use of UDP checksums instead of discouraging them. > > > > Thanks, > > Tom > > > > > > One is to drop all packets with a non-zero checksum (because one > might be invalid and even one invalid one slipping through would be > unacceptable). Another alternative is to accept all checksum values. Th= e > second option greatly enhances interoperability with implementations that > choose to generate a checksum and implementations that cannot validate th= e > checksum. It allows a mixed environment where "better" implementations > (that can validate) can interoperate with "inferior" implementations that > are unable to validate the checksum. > > > > > > BTW, VXLAN is not the only tunneling protocol to specify this behavior. > LISP (RFC 6830), specifies exactly the same behavior. > > - Larry > > > On 4/30/14 12:01 PM, "Tom Herbert" wrote: > > >Hi, > > > >I noticed that the VXLAN draft allows an implementation to potentially > >ignore a non-zero invalid UDP checksum. > > > >From: http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-09 > > > >"When a decapsulating endpoint receives a packet with a non-zero > >checksum it MAY choose to verify the checksum" > > > >However, from RFC 1122: > > > >"If a UDP datagram is received with a checksum that is non-zero and > >invalid, UDP MUST silently discard the datagram." > > > >It doesn't seem like 1122 allows checksum verification to be optional, > >so these would seem to be a conflict. Presumably, ignoring the RX csum > >is included for performance but since the sender can already send zero > >checksums in UDP (they are optional in IPv4 and allowed for IPv6 > >tunnels in RFC 6935) I'm not sure this is necessary. Besides that, the > >UDP checksum is potentially the only thing that protection of the vni > >against corruption end to end so allowing a receiver to ignore a bad > >checksum seems very risky. > > > >As a comparison, RFC 3931 (L2TP) has the following wording: > > > >"Thus, UDP checksums MAY be disabled in order to reduce the associated > >packet processing burden at the L2TP endpoints." > > > >This is somewhat ambiguous, but seems like the correct interpretation > >should be that zero checksums may be sent with L2TP/UDP, but on > >receive non-zero checksums should still be validated. > > > >Are these interpretations correct? Is there there a need to clarify > >the requirement for UDP tunnel protocols and checksums? > > > >Thanks, > >Tom > > > > >_______________________________________________ > >Tofoo mailing list > >Tofoo@ietf.org > >https://www.ietf.org/mailman/listinfo/tofoo > > > --90e6ba614a76ee1f9f04f8582b3e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On T= hu, May 1, 2014 at 1:10 AM, Alexander Vainshtein <Alexander= .Vainshtein@ecitele.com> wrote:

Tom, Larry and all,

I do not want to argue for or against UDP ch= ecksums in VXVLAN .

Just want to point that ability of HW to che= ck/generate the IPv4 checksums does not, IMHO and FWIW, that the same HW ca= n easily do UDP checksums.

=C2=A0

AFAIK, IPv4 forwarding HW in most cases has = access to a =C2=A0relatively small part at the beginning of the packet. Thi= s is enough to deal with=C2=A0 the IPv4 header but is not enough to compute UDP checksums =E2=80=93 because these include= the entire packet payload.

=C2=A0

Thanks Sasha, this is a good point. Similar concerns have been me= ntioned about the difficulty of switches to process packets with IP options= . The solutions there were to split into a fast and slow data path based on= the presence of options, or some implementations probably just dropped pac= kets with options. =C2=A0 In either case protocol conformance is maintained= and there was no need to change IP protocol. I'd suggest a similar app= roach can be done where zero checksum value in a UDP packet takes an expedi= ted data path.

Also, this discussion really seems to be specific to sw= itch HW. I think you'd be hard pressed to find a NIC that does not prov= ide L4 checksum offload. Retrofitting those NICs to provide checksum for en= capsulation (really a matter of supporting >1 csums in a packet) should = mostly be an API issue and even then most of the complexity in on TX side n= ot RX.
=C2=A0

This does not mean that HW-assisted computat= ion of UDP ckechsums is not possible =E2=80=93 only that it would be rthogo= nal to HW-assisted computation of IPv4 header checksums.

My 2c,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Sasha

Email: Alexander.Vainshtein@ecitele.com

Mobile: 054-9266302

=C2=A0

From: nvo3 [mailto:nvo3-bounces@ietf.org] On Behalf Of Tom Herbert
Sent: Thursday, May 01, 2014 4:48 AM
To: Larry Kreeger (kreeger)
Cc: tofoo@ietf.o= rg; nvo3@ietf.org; mallik_= mahalingam@yahoo.com; ddutt.ietf@hobbesdutt.com
Subject: Re: [nvo3] [Tofoo] VXLAN (UDP tunnel protocols) and non-zer= o checksums

=C2=A0

=C2=A0

=C2=A0

On Wed, Apr 30, 2014 at 5:13 PM, Larry Kreeger (kree= ger) <kreeger@cis= co.com> wrote:

Hi Tom,

I'll give you my perspective on why I feel the behavior described in th= e
VXLAN draft is a good thing. =C2=A0First, it is my assumption that some
implementations (e.g. many hardware implementations), will not implement checksum generation, nor checksum validation. =C2=A0I believe this is an implementation reality. =C2=A0If implementations are required to check a non-zero checksum, but can't actually do it, what alternatives do they<= br> have?

=C2=A0

Barring that the implementations you're referrin= g to only implement IPv6, these implementations must be already be doing ch= ecksum validation for IPv4 header-- so the checksum logic must have been im= plemented and I'm not sure the argument that this HW can't calculate the UDP csum would have much merit. A spe= cific example implementation would be nice here, for instance in the case t= hat the stack decapsulates the checksum can always be done in SW if verific= ation is not provided by the device.

=C2=A0

Also, as I pointed out, the UDP checksum is *not* us= eless in VXLAN. In an L3 routed network there is nothing that protects the = vni from corruption expect possibly the UDP checksum. Without any additiona= l security mechanisms is VXLAN header (like a cookie), the only way I could deploy VXLAN is with checksums enabl= ed. So in my opinion, the draft should be encouraging use of UDP checksums = instead of discouraging them.

=C2=A0

Thanks,

Tom

=C2=A0

=C2=A0

One is to drop all packets with a non-zero checksum = (because one
might be invalid and even one invalid one slipping through would be
unacceptable). =C2=A0Another alternative is to accept all checksum values. = =C2=A0The
second option greatly enhances interoperability with implementations that choose to generate a checksum and implementations that cannot validate the<= br> checksum. =C2=A0It allows a mixed environment where "better" impl= ementations
(that can validate) can interoperate with "inferior" implementati= ons that
are unable to validate the checksum.

=C2=A0

=C2=A0

BTW, VXLAN is not the only tunneling protocol to spe= cify this behavior.
LISP (RFC 6830), specifies exactly the same behavior.=C2=A0

=C2=A0-= Larry


On 4/30/14 12:01 PM, "Tom Herbert" <therbert@google.com> wrote:

>Hi,
>
>I noticed that the VXLAN draft allows an implementation to potentially<= br> >ignore a non-zero invalid UDP checksum.
>
>From: http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-09
>
>"When a decapsulating endpoint receives a packet with a non-zero >checksum it MAY choose to verify the checksum"
>
>However, from RFC 1122:
>
>"If a UDP datagram is received with a checksum that is non-zero an= d
>invalid, UDP MUST silently discard the datagram."
>
>It doesn't seem like 1122 allows checksum verification to be option= al,
>so these would seem to be a conflict. Presumably, ignoring the RX csum<= br> >is included for performance but since the sender can already send zero<= br> >checksums in UDP (they are optional in IPv4 and allowed for IPv6
>tunnels in RFC 6935) I'm not sure this is necessary. Besides that, = the
>UDP checksum is potentially the only thing that protection of the vni >against corruption end to end so allowing a receiver to ignore a bad >checksum seems very risky.
>
>As a comparison, RFC 3931 (L2TP) has the following wording:
>
>"Thus, UDP checksums MAY be disabled in order to reduce the associ= ated
>packet processing burden at the L2TP endpoints."
>
>This is somewhat ambiguous, but seems like the correct interpretation >should be that zero checksums may be sent with L2TP/UDP, but on
>receive non-zero checksums should still be validated.
>
>Are these interpretations correct? Is there there a need to clarify
>the requirement for UDP tunnel protocols and checksums?
>
>Thanks,
>Tom
>

>___________________= ____________________________
>Tofoo mailing list
>Tofoo@ietf.org<= br> >https://www.ietf.org/mailman/listinfo/tofoo

=C2=A0


--90e6ba614a76ee1f9f04f8582b3e-- From nobody Thu May 1 13:11:15 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6EB361A6FD1; Thu, 1 May 2014 13:11:11 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.151 X-Spam-Level: X-Spam-Status: No, score=-10.151 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QP4HMWMv7ilD; Thu, 1 May 2014 13:11:02 -0700 (PDT) Received: from alln-iport-8.cisco.com (alln-iport-8.cisco.com [173.37.142.95]) by ietfa.amsl.com (Postfix) with ESMTP id 4B2821A700D; Thu, 1 May 2014 13:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=21395; q=dns/txt; s=iport; t=1398975060; x=1400184660; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=f36ylRAJYDtaqG5Eqa170OZdFxz87/Y+kD0CZaJBVL0=; b=AWRWbLq5pomXscU45qZaQbAlna8LQA1/iEcL9WVBQAGitMZghszCRkFf VHH0CQiGpLtf2ZTGIbwh5OYXBBMcuQk27JzqjBa62XQGk3YDJ5uknqqe2 Q+E6eNenFHV9aXPArNUifUqGM0KvWT7q+9kHVx+A+9NhL5tyGX0I+wCUr o=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhgFAC6pYlOtJV2Z/2dsb2JhbABagkJET1etAY5ZgUIBhz2BFBZ0giUBAQEEAQEBZAcLEAIBCA4DAwECKAcnCxMBCQgCBA4FiC0DEQ3JZBeMO4IGDQQHhDkEhFkCAZBdggqBbIE8i1eFW4MzgWskHA X-IronPort-AV: E=Sophos; i="4.97,966,1389744000"; d="scan'208,217"; a="40358871" Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by alln-iport-8.cisco.com with ESMTP; 01 May 2014 20:10:59 +0000 Received: from xhc-rcd-x15.cisco.com (xhc-rcd-x15.cisco.com [173.37.183.89]) by rcdn-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id s41KAxv1030187 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 1 May 2014 20:10:59 GMT Received: from xmb-rcd-x01.cisco.com ([169.254.1.229]) by xhc-rcd-x15.cisco.com ([173.37.183.89]) with mapi id 14.03.0123.003; Thu, 1 May 2014 15:10:59 -0500 From: "Larry Kreeger (kreeger)" To: Tom Herbert Thread-Topic: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums Thread-Index: AQHPZKbjNYPI8b/6TUi7fhWXO3OBYJsquXGAgACP5oD//4xngIAAi68AgACmqQA= Date: Thu, 1 May 2014 20:10:58 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.1.140326 x-originating-ip: [10.21.85.178] Content-Type: multipart/alternative; boundary="_000_CF87F70EF437Ckreegerciscocom_" MIME-Version: 1.0 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/3bI-XErIqGqCwGyZXacf0IuCoOs Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:11:12 -0000 --_000_CF87F70EF437Ckreegerciscocom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable From: Tom Herbert > Date: Wednesday, April 30, 2014 8:14 PM To: Larry Kreeger > Cc: "nvo3@ietf.org" >, "tofoo@ietf.org" >, "mallik_mahalingam@yahoo.com" >, Dines= h Dutt > Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums On Wed, Apr 30, 2014 at 6:54 PM, Larry Kreeger (kreeger) > wrote: See inline, marked with LK>. - Larry From: Tom Herbert > Date: Wednesday, April 30, 2014 6:48 PM To: Larry Kreeger > Cc: "nvo3@ietf.org" >, "tofoo@ietf.org" >, "mallik_mahalingam@yahoo.com" >, Dines= h Dutt > Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums On Wed, Apr 30, 2014 at 5:13 PM, Larry Kreeger (kreeger) > wrote: Hi Tom, I'll give you my perspective on why I feel the behavior described in the VXLAN draft is a good thing. First, it is my assumption that some implementations (e.g. many hardware implementations), will not implement checksum generation, nor checksum validation. I believe this is an implementation reality. If implementations are required to check a non-zero checksum, but can't actually do it, what alternatives do they have? Barring that the implementations you're referring to only implement IPv6, t= hese implementations must be already be doing checksum validation for IPv4 = header-- so the checksum logic must have been implemented and I'm not sure = the argument that this HW can't calculate the UDP csum would have much meri= t. A specific example implementation would be nice here, for instance in th= e case that the stack decapsulates the checksum can always be done in SW if= verification is not provided by the device. LK> I am referring to switching ASIC implementations. There is no software= nor traditional UDP stack involved. Are these implementations also ignoring IPv4 header checksum then? LK2> Yep. The text in the VXLAN draft applies to both IPv4 and IPv6. In f= act, IPv6 was only added to the draft in a later version and all original i= mplementations supported only IPv4 encapsulation (I'm not aware of any IPv6= implementations, but maybe there are some out there). Also, as I pointed out, the UDP checksum is *not* useless in VXLAN. In an L= 3 routed network there is nothing that protects the vni from corruption exp= ect possibly the UDP checksum. Without any additional security mechanisms i= s VXLAN header (like a cookie), the only way I could deploy VXLAN is with c= hecksums enabled. So in my opinion, the draft should be encouraging use of = UDP checksums instead of discouraging them. LK> Neither I nor the VXLAN draft claims that UDP checksums are useless, ho= wever I think it is an exaggeration to say "nothing" protects the VNI from = corruption when the packet is being carried over Ethernet with a robust CRC= 32 protecting it. Assuming we are using Ethernet (I don't believe this can be a requirement e= ither) this only provides hop to hop protection, not end to end. I don't ha= ve a completely error free network and checksum errors while low, are non-z= ero. LK2> I suppose it depends on the routers and switches in use, but a well de= signed one should not flip any payload bits while routing/switching (e.g. u= se internal CRCs across its backplane). Sending a packet to the wrong VM is fundamentally bad and could be quite co= stly, so for me UDP checksums would be the minimum requirement. This would= need to apply to the case where the protocol is switched also. LK2> It is much easier for software on hosts to do this (and maybe NICs). = You might be hard pressed to find a hardware gateway that does it in a cost= effective manner. In any case, the requirements and constraints for when it's acceptable to i= gnore non-zero checksums for UDP should be specified in a more general way = if is being allowed. Maybe an addendum to RFC 6935? Tom Thanks, Tom One is to drop all packets with a non-zero checksum (because one might be invalid and even one invalid one slipping through would be unacceptable). Another alternative is to accept all checksum values. The second option greatly enhances interoperability with implementations that choose to generate a checksum and implementations that cannot validate the checksum. It allows a mixed environment where "better" implementations (that can validate) can interoperate with "inferior" implementations that are unable to validate the checksum. BTW, VXLAN is not the only tunneling protocol to specify this behavior. LISP (RFC 6830), specifies exactly the same behavior. - Larry On 4/30/14 12:01 PM, "Tom Herbert" > wrote: >Hi, > >I noticed that the VXLAN draft allows an implementation to potentially >ignore a non-zero invalid UDP checksum. > >From: http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-09 > >"When a decapsulating endpoint receives a packet with a non-zero >checksum it MAY choose to verify the checksum" > >However, from RFC 1122: > >"If a UDP datagram is received with a checksum that is non-zero and >invalid, UDP MUST silently discard the datagram." > >It doesn't seem like 1122 allows checksum verification to be optional, >so these would seem to be a conflict. Presumably, ignoring the RX csum >is included for performance but since the sender can already send zero >checksums in UDP (they are optional in IPv4 and allowed for IPv6 >tunnels in RFC 6935) I'm not sure this is necessary. Besides that, the >UDP checksum is potentially the only thing that protection of the vni >against corruption end to end so allowing a receiver to ignore a bad >checksum seems very risky. > >As a comparison, RFC 3931 (L2TP) has the following wording: > >"Thus, UDP checksums MAY be disabled in order to reduce the associated >packet processing burden at the L2TP endpoints." > >This is somewhat ambiguous, but seems like the correct interpretation >should be that zero checksums may be sent with L2TP/UDP, but on >receive non-zero checksums should still be validated. > >Are these interpretations correct? Is there there a need to clarify >the requirement for UDP tunnel protocols and checksums? > >Thanks, >Tom > >_______________________________________________ >Tofoo mailing list >Tofoo@ietf.org >https://www.ietf.org/mailman/listinfo/tofoo --_000_CF87F70EF437Ckreegerciscocom_ Content-Type: text/html; charset="us-ascii" Content-ID: <9A3FBB207B2E9A43A69CAF92241FB051@emea.cisco.com> Content-Transfer-Encoding: quoted-printable


From: Tom Herbert <therbert@google.com>
Date: Wednesday, April 30, 2014 8:1= 4 PM
To: Larry Kreeger <kreeger@cisco.com>
Cc: "nvo3@ietf.org" <nvo3@i= etf.org>, "tofoo@ietf.org= " <tofoo@ietf.org>, "= mallik_mahalingam@yahoo.com<= /a>" <mallik_mahalingam@yahoo= .com>, Dinesh Dutt <= ddutt.ietf@hobbesdutt.com>
Subject: Re: [Tofoo] VXLAN (UDP tun= nel protocols) and non-zero checksums




On Wed, Apr 30, 2014 at 6:54 PM, Larry Kreeger (= kreeger) <kreeger@cisco.com> wrote:
See inline, marked with LK>. - Larry

From: Tom Herbert <therbert@google.com>
Date: Wednesday, April 30, 2014 6:4= 8 PM
To: Larry Kreeger <kreeger@cisco.com>
Cc: "nvo3@ietf.org" <nvo3@ietf.org>, "tofoo@ietf.org" <tofoo@ietf.org>, "mal= lik_mahalingam@yahoo.com" <mallik_mahalingam@yahoo.com>, Dinesh Du= tt <ddutt= .ietf@hobbesdutt.com>
Subject: Re: [Tofoo] VXLAN (UDP tun= nel protocols) and non-zero checksums




On Wed, Apr 30, 2014 at 5:13 PM, Larry Kreeger (= kreeger) <kreeger@cisco.com> wrote:
Hi Tom,

I'll give you my perspective on why I feel the behavior described in the VXLAN draft is a good thing.  First, it is my assumption that some
implementations (e.g. many hardware implementations), will not implement checksum generation, nor checksum validation.  I believe this is an implementation reality.  If implementations are required to check a non-zero checksum, but can't actually do it, what alternatives do they
have?

Barring that the implementations you're referring to only implement IP= v6, these implementations must be already be doing checksum validation for = IPv4 header-- so the checksum logic must have been implemented and I'm not = sure the argument that this HW can't calculate the UDP csum would have much merit. A specific example implement= ation would be nice here, for instance in the case that the stack decapsula= tes the checksum can always be done in SW if verification is not provided b= y the device.

LK> I am referring to switching ASIC implementations.  There i= s no software nor traditional UDP stack involved.

Are these implementations also ignoring IPv4 header checksum then?

LK2> Yep.  The text in the VXLAN draft applies to both IPv4 an= d IPv6.  In fact, IPv6 was only added to the draft in a later version = and all original implementations supported only IPv4 encapsulation (I'm not= aware of any IPv6 implementations, but maybe there are some out there).
 
Also, as I pointed out, the UDP checksum is *not* useless in VXLAN. In= an L3 routed network there is nothing that protects the vni from corruptio= n expect possibly the UDP checksum. Without any additional security mechani= sms is VXLAN header (like a cookie), the only way I could deploy VXLAN is with checksums enabled. So in my opin= ion, the draft should be encouraging use of UDP checksums instead of discou= raging them.

LK> Neither I nor the VXLAN draft claims that UDP checksums are use= less, however I think it is an exaggeration to say "nothing" prot= ects the VNI from corruption when the packet is being carried over Ethernet= with a robust CRC32 protecting it.

Assuming we are using Ethernet (I don't believe this can be a requirem= ent either) this only provides hop to hop protection, not end to end. I don= 't have a completely error free network and checksum errors while low, are = non-zero.

LK2> I suppose it depends on the routers and switches in use, but a= well designed one should not flip any payload bits while routing/switching= (e.g. use internal CRCs across its backplane).

Sending a packet to the wrong VM is fundamentally bad and could be qui= te costly,  so for me UDP checksums would be the minimum requirement. = This would need to apply to the case where the protocol is switched also.

LK2> It is much easier for software on hosts to do this (and maybe = NICs).  You might be hard pressed to find a hardware gateway that does= it in a cost effective manner.

In any case, the requirements and constraints for when it's acceptable= to ignore non-zero checksums for UDP should be specified in a more general= way if is being allowed.  Maybe an addendum to RFC 6935?

Tom

Thanks,
Tom

 
One is to drop all packets with a non-zero checksum (because one
might be invalid and even one invalid one slipping through would be
unacceptable).  Another alternative is to accept all checksum values. =  The
second option greatly enhances interoperability with implementations that choose to generate a checksum and implementations that cannot validate the<= br> checksum.  It allows a mixed environment where "better" impl= ementations
(that can validate) can interoperate with "inferior" implementati= ons that
are unable to validate the checksum.
 
 
BTW, VXLAN is not the only tunneling protocol to specify this behavior.
LISP (RFC 6830), specifies exactly the same behavior. 
 - Larry

On 4/30/14 12:01 PM, "Tom Herbert" <therbert@google.com> wrote:

>Hi,
>
>I noticed that the VXLAN draft allows an implementation to potentially<= br> >ignore a non-zero invalid UDP checksum.
>
>From: http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-09
>
>"When a decapsulating endpoint receives a packet with a non-zero >checksum it MAY choose to verify the checksum"
>
>However, from RFC 1122:
>
>"If a UDP datagram is received with a checksum that is non-zero an= d
>invalid, UDP MUST silently discard the datagram."
>
>It doesn't seem like 1122 allows checksum verification to be optional,<= br> >so these would seem to be a conflict. Presumably, ignoring the RX csum<= br> >is included for performance but since the sender can already send zero<= br> >checksums in UDP (they are optional in IPv4 and allowed for IPv6
>tunnels in RFC 6935) I'm not sure this is necessary. Besides that, the<= br> >UDP checksum is potentially the only thing that protection of the vni >against corruption end to end so allowing a receiver to ignore a bad >checksum seems very risky.
>
>As a comparison, RFC 3931 (L2TP) has the following wording:
>
>"Thus, UDP checksums MAY be disabled in order to reduce the associ= ated
>packet processing burden at the L2TP endpoints."
>
>This is somewhat ambiguous, but seems like the correct interpretation >should be that zero checksums may be sent with L2TP/UDP, but on
>receive non-zero checksums should still be validated.
>
>Are these interpretations correct? Is there there a need to clarify
>the requirement for UDP tunnel protocols and checksums?
>
>Thanks,
>Tom
>
>_______________________________________________
>Tofoo mailing list
>Tofoo@ietf.org<= br> >https://www.ietf.org/mailman/listinfo/tofoo



 --_000_CF87F70EF437Ckreegerciscocom_-- From nobody Thu May 1 13:21:32 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0E4091A098B; Thu, 1 May 2014 13:21:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aJjvKYB-S-CE; Thu, 1 May 2014 13:21:30 -0700 (PDT) Received: from vapor.isi.edu (vapor.isi.edu [128.9.64.64]) by ietfa.amsl.com (Postfix) with ESMTP id 21B981A0974; Thu, 1 May 2014 13:21:30 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by vapor.isi.edu (8.13.8/8.13.8) with ESMTP id s41KKrHD010434 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 1 May 2014 13:20:53 -0700 (PDT) Message-ID: <5362ACA5.1030102@isi.edu> Date: Thu, 01 May 2014 13:20:53 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: sarikaya@ieee.org, Tom Herbert References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/nn9JKbJYQ3KzvpzL7xEEAMhBd-4 Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , mallik_mahalingam@yahoo.com, ddutt.ietf@hobbesdutt.com Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:21:31 -0000 On 4/30/2014 2:23 PM, Behcet Sarikaya wrote: > Here is what VXLAN says on tunneled traffic: > > Tunneled traffic over the IP network can be secured with traditional > security mechanisms like IPsec that authenticate and optionally > encrypt VXLAN traffic. This will, of course, need to be coupled with > an authentication infrastructure for authorized endpoints to obtain > and distribute credentials. > > Based on this, UDP checksum text seems to be consistent, no? No; the UDP checksum is not for authetication. It is an error check. The only party that can decide to make the UDP checksum optional when using IPv4 is the source - by inserting zero. It's not the receiver's choice to ignore that checksum if it's not zero. That's where this doc breaks the current standards. Joe From nobody Thu May 1 13:30:24 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 21D411A6F62; Thu, 1 May 2014 13:30:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.749 X-Spam-Level: X-Spam-Status: No, score=-1.749 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GubdFXRD00Hr; Thu, 1 May 2014 13:30:21 -0700 (PDT) Received: from mail-lb0-x22c.google.com (mail-lb0-x22c.google.com [IPv6:2a00:1450:4010:c04::22c]) by ietfa.amsl.com (Postfix) with ESMTP id AA76F1A0974; Thu, 1 May 2014 13:30:20 -0700 (PDT) Received: by mail-lb0-f172.google.com with SMTP id p9so2476097lbv.17 for ; Thu, 01 May 2014 13:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=C2fokqfoMgm2zwBD/y9sHBi86kyCubGJg/Y+LhD+Lzc=; b=K8q+FaMr//bWSZbdZd/Ldzh/+R5W4PHo08kHQqYuXUEcASQjwzoDehKXl2V/jTUOpm gepeqPMkA5BKxAesKBLxRTCjea6DyqL1sYI44ZGOLEWOTgFsaXdC4Cp9nO22pxIvDB4F ciCzQ4BfUSxmhMJeN/zZSIwM/qvGwfoOIimRCi2Je1qA1hPlHeispd2jo+amqPj8a/5V 6sp4mwE33lxBeFNfNAiokIt1H/ADcPytuxGmI2Hj94rvleb28uNigXec3NdSsOFFnmJb jCbPXlaAzmbq3c9rEKYHqnrHOQOrQN361dP+pqPd/jqiKqe1cMNu4VNxcSDGfk4r18BN DXLw== MIME-Version: 1.0 X-Received: by 10.112.35.202 with SMTP id k10mr8541373lbj.14.1398976218072; Thu, 01 May 2014 13:30:18 -0700 (PDT) Received: by 10.114.70.165 with HTTP; Thu, 1 May 2014 13:30:18 -0700 (PDT) In-Reply-To: <5362ACA5.1030102@isi.edu> References: <5362ACA5.1030102@isi.edu> Date: Thu, 1 May 2014 15:30:18 -0500 Message-ID: From: Behcet Sarikaya To: Joe Touch Content-Type: multipart/alternative; boundary=001a11c36bb2870cb204f85c8868 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/VJgeIocZZ6fd_QEs1g4AeoNunXo Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , ddutt.ietf@hobbesdutt.com, mallik_mahalingam@yahoo.com, Tom Herbert Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: sarikaya@ieee.org List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:30:22 -0000 --001a11c36bb2870cb204f85c8868 Content-Type: text/plain; charset=UTF-8 On Thu, May 1, 2014 at 3:20 PM, Joe Touch wrote: > > > On 4/30/2014 2:23 PM, Behcet Sarikaya wrote: > >> Here is what VXLAN says on tunneled traffic: >> >> Tunneled traffic over the IP network can be secured with traditional >> security mechanisms like IPsec that authenticate and optionally >> encrypt VXLAN traffic. This will, of course, need to be coupled with >> an authentication infrastructure for authorized endpoints to obtain >> and distribute credentials. >> >> Based on this, UDP checksum text seems to be consistent, no? >> > > No; the UDP checksum is not for authetication. It is an error check. > > The only party that can decide to make the UDP checksum optional when > using IPv4 is the source - by inserting zero. > > It's not the receiver's choice to ignore that checksum if it's not zero. > That's where this doc breaks the current standards. > > The important point in the above text that I quoted was encryption being optional not about authentication. So checksum would be zero if the payload is encrypted and non-zero if it is not not and both cases are possible. Behcet > Joe > --001a11c36bb2870cb204f85c8868 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, May 1, 2014 at 3:20 PM, Joe Touch <touch@isi.edu> wrote:


On 4/30/2014 2:23 PM, Behcet Sarikaya wrote:
Here is what VXLAN says on tunneled traffic:

Tunneled traffic over the IP network can be secured with traditional
=C2=A0 =C2=A0 security mechanisms like IPsec that authenticate and optional= ly
=C2=A0 =C2=A0 encrypt VXLAN traffic. This will, of course, need to be coupl= ed with
=C2=A0 =C2=A0 an authentication infrastructure for authorized endpoints to = obtain
=C2=A0 =C2=A0 and distribute credentials.

Based on this, UDP checksum text seems to be consistent, no?

No; the UDP checksum is not for authetication. It is an error check.

The only party that can decide to make the UDP checksum optional when using= IPv4 is the source - by inserting zero.

It's not the receiver's choice to ignore that checksum if it's = not zero. That's where this doc breaks the current standards.


The important point in t= he above text that I quoted was encryption being optional not about authent= ication.
So checksum would be zero if the payload is encrypte= d and non-zero if it is not not and both cases are possible.

Behcet
Joe

--001a11c36bb2870cb204f85c8868-- From nobody Thu May 1 13:32:41 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A6FF71A09C8 for ; Thu, 1 May 2014 13:32:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.749 X-Spam-Level: X-Spam-Status: No, score=-1.749 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CyG27P3Ml5eY for ; Thu, 1 May 2014 13:32:39 -0700 (PDT) Received: from mail-lb0-x22a.google.com (mail-lb0-x22a.google.com [IPv6:2a00:1450:4010:c04::22a]) by ietfa.amsl.com (Postfix) with ESMTP id A9E4B1A0976 for ; Thu, 1 May 2014 13:32:38 -0700 (PDT) Received: by mail-lb0-f170.google.com with SMTP id 10so2467255lbg.15 for ; Thu, 01 May 2014 13:32:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; bh=ZMkajkIqV2fjyK7ica7uO1QYRA1VwEacOGSFUfdX1jY=; b=MOm4/E5rBYt2AojQRzJdOoL1r+qKJUaLWXmcfDAH74vnVqPWlJQuStB4K7CHnVzErh svafpiYucRP5MpbC92ULJBQYAgI4q/uz1OStpp6Wn4QVgbLwbJyiwAx4toNnmq3BK8JC ewIC1B7Hd1PnizUWP04dzoMOrjrjvRh3vLD0kp0h1geOPuq2pmJXq39qfiXjO7Gydc2V T+ZbgKVJlW7RkFT7z7U0cD2lJ5DS2oANtT3oBP40nel9G+zAmGPnkDXPWYvfFi+JW832 Jj5npTe1QevwHykwb1zIqdAcFgeI8kUalljU2he9MvTzPE11JDqZ1wujfcwBBtfcy0KZ 5w1g== MIME-Version: 1.0 X-Received: by 10.112.160.168 with SMTP id xl8mr8642234lbb.3.1398976355882; Thu, 01 May 2014 13:32:35 -0700 (PDT) Received: by 10.114.70.165 with HTTP; Thu, 1 May 2014 13:32:35 -0700 (PDT) Date: Thu, 1 May 2014 15:32:35 -0500 Message-ID: From: Behcet Sarikaya To: Martin Stiemerling , "tofoo@ietf.org" Content-Type: multipart/alternative; boundary=001a11c3888cbddb2304f85c90cd Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/Qy_4Pil0WKQd7a_Rga-CLBbPcpc Subject: [Tofoo] VXLAN solutions X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: sarikaya@ieee.org List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:32:39 -0000 --001a11c3888cbddb2304f85c90cd Content-Type: text/plain; charset=UTF-8 Hi Martin, all, I have a question, can VXLAN solutions be discussed in this list? Or is it out of scope? Regards, Behcet --001a11c3888cbddb2304f85c90cd Content-Type: text/html; charset=UTF-8
Hi Martin, all,

I have a question, can VXLAN solutions be discussed in this list? Or is it out of scope?

Regards,

Behcet
--001a11c3888cbddb2304f85c90cd-- From nobody Thu May 1 13:34:49 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 09CC71A6F62; Thu, 1 May 2014 13:34:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NmXy6GE9Gbzj; Thu, 1 May 2014 13:34:47 -0700 (PDT) Received: from vapor.isi.edu (vapor.isi.edu [128.9.64.64]) by ietfa.amsl.com (Postfix) with ESMTP id 1912E1A0976; Thu, 1 May 2014 13:34:47 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by vapor.isi.edu (8.13.8/8.13.8) with ESMTP id s41KY3JP013901 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 1 May 2014 13:34:03 -0700 (PDT) Message-ID: <5362AFBB.6080008@isi.edu> Date: Thu, 01 May 2014 13:34:03 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: sarikaya@ieee.org References: <5362ACA5.1030102@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/spxj1Hm2bSSLBiJcpc0JTfEqUU0 Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , ddutt.ietf@hobbesdutt.com, mallik_mahalingam@yahoo.com, Tom Herbert Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:34:48 -0000 On 5/1/2014 1:30 PM, Behcet Sarikaya wrote: > > > > On Thu, May 1, 2014 at 3:20 PM, Joe Touch > wrote: > > > > On 4/30/2014 2:23 PM, Behcet Sarikaya wrote: > > Here is what VXLAN says on tunneled traffic: > > Tunneled traffic over the IP network can be secured with traditional > security mechanisms like IPsec that authenticate and optionally > encrypt VXLAN traffic. This will, of course, need to be > coupled with > an authentication infrastructure for authorized endpoints > to obtain > and distribute credentials. > > Based on this, UDP checksum text seems to be consistent, no? > > > No; the UDP checksum is not for authetication. It is an error check. > > The only party that can decide to make the UDP checksum optional > when using IPv4 is the source - by inserting zero. > > It's not the receiver's choice to ignore that checksum if it's not > zero. That's where this doc breaks the current standards. > > The important point in the above text that I quoted was encryption being > optional not about authentication. > So checksum would be zero if the payload is encrypted and non-zero if it > is not not and both cases are possible. Receiver processing is simple: - if the checksum is zero, ignore - if the checksum is NOT zero, it MUST match No other part of the packet needs to be examined. If the *sender* wants to have the receiver ignore the checksum, it inserts zero. If not, the receiver MUST process and validate it. Joe From nobody Thu May 1 13:38:30 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 027BE1A6FA6; Thu, 1 May 2014 13:38:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.749 X-Spam-Level: X-Spam-Status: No, score=-1.749 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5NZedy3VFMLV; Thu, 1 May 2014 13:38:27 -0700 (PDT) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) by ietfa.amsl.com (Postfix) with ESMTP id 881CF1A0976; Thu, 1 May 2014 13:38:26 -0700 (PDT) Received: by mail-la0-f51.google.com with SMTP id gl10so2443639lab.24 for ; Thu, 01 May 2014 13:38:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=sxRp+N7iVp/j63G0Uv0dmn/89yiAVba2b6w+9/NFP80=; b=z/1MskcOQF4/Rs+9C0y8RqoSr2Zmbjy0kx3UT1A8V6bhj+m0XSzLqAlzKQbF0LKNQt z5TTZ3GPNCtJ7UJ7KER1zx6CFPEnGFGnLllSL6DxLtumJ58UM/yv+rmh9iwdNvgc16ng AqdBrkwuVhHtJwZsMl5tx6NXnxLjgTpom1OIZTMtJRWdWYnYs9GMVZ+b/kW9c2j6u2Ia YyGGlyGLAxFtsUoJ+VFdyOyJ9WpXMMVbvaV0Ye0zOIMfLp6CUyNihb4BQJ0YDRAYSzfO mPO0MzvQfVytBMgnhrKv5muZjDQ+avDoQCjSIDugjStI6XdVDxWodPD9N0ll9HvPCPx2 EygA== MIME-Version: 1.0 X-Received: by 10.112.61.199 with SMTP id s7mr8648577lbr.25.1398976703853; Thu, 01 May 2014 13:38:23 -0700 (PDT) Received: by 10.114.70.165 with HTTP; Thu, 1 May 2014 13:38:23 -0700 (PDT) In-Reply-To: <5362AFBB.6080008@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> Date: Thu, 1 May 2014 15:38:23 -0500 Message-ID: From: Behcet Sarikaya To: Joe Touch Content-Type: multipart/alternative; boundary=001a1133d17a7b7b2404f85ca5e9 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/5x1lsfwAcO8QzGwgXeoSMELTn8o Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , ddutt.ietf@hobbesdutt.com, mallik_mahalingam@yahoo.com, Tom Herbert Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: sarikaya@ieee.org List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 20:38:28 -0000 --001a1133d17a7b7b2404f85ca5e9 Content-Type: text/plain; charset=UTF-8 On Thu, May 1, 2014 at 3:34 PM, Joe Touch wrote: > > > On 5/1/2014 1:30 PM, Behcet Sarikaya wrote: > >> >> >> >> On Thu, May 1, 2014 at 3:20 PM, Joe Touch > > wrote: >> >> >> >> On 4/30/2014 2:23 PM, Behcet Sarikaya wrote: >> >> Here is what VXLAN says on tunneled traffic: >> >> Tunneled traffic over the IP network can be secured with >> traditional >> security mechanisms like IPsec that authenticate and >> optionally >> encrypt VXLAN traffic. This will, of course, need to be >> coupled with >> an authentication infrastructure for authorized endpoints >> to obtain >> and distribute credentials. >> >> Based on this, UDP checksum text seems to be consistent, no? >> >> >> No; the UDP checksum is not for authetication. It is an error check. >> >> The only party that can decide to make the UDP checksum optional >> when using IPv4 is the source - by inserting zero. >> >> It's not the receiver's choice to ignore that checksum if it's not >> zero. That's where this doc breaks the current standards. >> >> The important point in the above text that I quoted was encryption being >> optional not about authentication. >> So checksum would be zero if the payload is encrypted and non-zero if it >> is not not and both cases are possible. >> > > Receiver processing is simple: > > - if the checksum is zero, ignore > > - if the checksum is NOT zero, it MUST match > > No other part of the packet needs to be examined. If the *sender* wants to > have the receiver ignore the checksum, it inserts zero. If not, the > receiver MUST process and validate it. > > Sure. I think we are in agreement. Behcet > Joe > --001a1133d17a7b7b2404f85ca5e9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, May 1, 2014 at 3:34 PM, Joe Touch <touch@isi.edu> wrote:


On 5/1/2014 1:30 PM, Behcet Sarikaya wrote:



On Thu, May 1, 2014 at 3:20 PM, Joe Touch <touch@isi.edu
<mailto:touch@isi.edu= >> wrote:



=C2=A0 =C2=A0 On 4/30/2014 2:23 PM, Behcet Sarikaya wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Here is what VXLAN says on tunneled traffic:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Tunneled traffic over the IP network can be sec= ured with traditional
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0security mechanisms like IP= sec that authenticate and optionally
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0encrypt VXLAN traffic. This= will, of course, need to be
=C2=A0 =C2=A0 =C2=A0 =C2=A0 coupled with
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0an authentication infrastru= cture for authorized endpoints
=C2=A0 =C2=A0 =C2=A0 =C2=A0 to obtain
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and distribute credentials.=

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Based on this, UDP checksum text seems to be co= nsistent, no?


=C2=A0 =C2=A0 No; the UDP checksum is not for authetication. It is an error= check.

=C2=A0 =C2=A0 The only party that can decide to make the UDP checksum optio= nal
=C2=A0 =C2=A0 when using IPv4 is the source - by inserting zero.

=C2=A0 =C2=A0 It's not the receiver's choice to ignore that checksu= m if it's not
=C2=A0 =C2=A0 zero. That's where this doc breaks the current standards.=

The important point in the above text that I quoted was encryption being optional not about authentication.
So checksum would be zero if the payload is encrypted and non-zero if it is not not and both cases are possible.

Receiver processing is simple:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 - if the checksum is zero, ignore

=C2=A0 =C2=A0 =C2=A0 =C2=A0 - if the checksum is NOT zero, it MUST match
No other part of the packet needs to be examined. If the *sender* wants to = have the receiver ignore the checksum, it inserts zero. If not, the receive= r MUST process and validate it.


Sure. I think we are in = agreement.

Behcet
Joe

--001a1133d17a7b7b2404f85ca5e9-- From nobody Thu May 1 14:01:03 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1F2B21A7032 for ; Thu, 1 May 2014 14:01:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FkvXyMvOajDX for ; Thu, 1 May 2014 14:00:59 -0700 (PDT) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) by ietfa.amsl.com (Postfix) with ESMTP id 901061A0971 for ; Thu, 1 May 2014 14:00:59 -0700 (PDT) Received: by mail-ie0-f179.google.com with SMTP id lx4so4125895iec.38 for ; Thu, 01 May 2014 14:00:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Mev6aRfGmzP6qIdAXdjQk8l5UMKMyhWLGc1C+JsNkLg=; b=juAN037mNkOFJgWn6YMKq7F7E23EFXz/jQLdt1mWmQKQ9kmGPc24t5d2qLrIU2iz7Y CX+GbH+EEjYjUh4TMmhO7I90ljk5zbuHh4ecJhrCKz4uX+QmPNo/c11CrcUudjUvE13J PRmaeOeqJ0Ajtdg3mdPb23H8WVHVtpW6DodvLQR7I8xbMLZISVA5aCojW3uVydIqn1J9 Qf9HB/1JEnAOdR1/ty79mBYqz4z1sklwbJVt3u8SKOyLWHH+SaZEm1iEEz7Lffug1hb2 E4unIR8CxTnaCPy5gtGC9zLT80GpacW7ZTbEyxDo98EReFhYPSo0nZbyPLDIKcY5QtL+ pM5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Mev6aRfGmzP6qIdAXdjQk8l5UMKMyhWLGc1C+JsNkLg=; b=Uae4WrLtosnJ26v83CRIn+Sa1AgB8MSqLyYJKuuxD1N54ueQbx1E+hpm/CftLjjzKy KmAiaFsdnyAg/JWbdLz3roL4Ipg5g5ZxDzDmLCMIXhZypLrYw3Hf7YlzIBwPZi2H+JTm +KF76zju6WA5xwN8UN/RaXZQFhEcbeAlR+OEnTjxFGyO9IBJUN9/RrCm5bAGjnRc2PhW IqI33QxjyNhe0ERg7lD6gfFcbXk/zNcQL6uDRkU4K/4CYw/DRbxntq14Q9Yc9P/pF8nR rC52+wrAl19IuAZqtmMq2QWoQlb6LRjvx497DwEPO9WRWi4YRgj0dCUnpWHTaRBc9kqZ H1Zw== X-Gm-Message-State: ALoCoQlBJe0EP59uBWCDc+naNhWzHIpD5/GL6oxKh9hmLsWCuPI/obeXkeBORDsLBxiF2uCe5HPm MIME-Version: 1.0 X-Received: by 10.50.143.34 with SMTP id sb2mr5942196igb.48.1398978049737; Thu, 01 May 2014 14:00:49 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Thu, 1 May 2014 14:00:49 -0700 (PDT) In-Reply-To: <5362AFBB.6080008@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> Date: Thu, 1 May 2014 14:00:49 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: multipart/alternative; boundary=001a1135effeb4238904f85cf50f Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/4rSXyMO4Sm9qO-n3NC20INk6qJA Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 21:01:01 -0000 --001a1135effeb4238904f85cf50f Content-Type: text/plain; charset=UTF-8 On Thu, May 1, 2014 at 1:34 PM, Joe Touch wrote: > > > On 5/1/2014 1:30 PM, Behcet Sarikaya wrote: > >> >> >> >> On Thu, May 1, 2014 at 3:20 PM, Joe Touch > > wrote: >> >> >> >> On 4/30/2014 2:23 PM, Behcet Sarikaya wrote: >> >> Here is what VXLAN says on tunneled traffic: >> >> Tunneled traffic over the IP network can be secured with >> traditional >> security mechanisms like IPsec that authenticate and >> optionally >> encrypt VXLAN traffic. This will, of course, need to be >> coupled with >> an authentication infrastructure for authorized endpoints >> to obtain >> and distribute credentials. >> >> Based on this, UDP checksum text seems to be consistent, no? >> >> >> No; the UDP checksum is not for authetication. It is an error check. >> >> The only party that can decide to make the UDP checksum optional >> when using IPv4 is the source - by inserting zero. >> >> It's not the receiver's choice to ignore that checksum if it's not >> zero. That's where this doc breaks the current standards. >> >> The important point in the above text that I quoted was encryption being >> optional not about authentication. >> So checksum would be zero if the payload is encrypted and non-zero if it >> is not not and both cases are possible. >> > > Receiver processing is simple: > > - if the checksum is zero, ignore > > - if the checksum is NOT zero, it MUST match > > This is true with caveat that an implementation MAY ignore a zero checksum, not MUST ignore a zero checksum. This is specified in RFC1122. VXLAN draft also breaks this by making accepting packets with zero checksums a MUST. From a practical perspective, if the UDP checksum were the only thing I had to protect the vni I would want the capability to drop packets received with zero checksums in my deployment. Tom No other part of the packet needs to be examined. If the *sender* wants to > have the receiver ignore the checksum, it inserts zero. If not, the > receiver MUST process and validate it. > > Joe > --001a1135effeb4238904f85cf50f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, May 1, 2014 at 1:34 PM, Joe Touch <touch@isi.edu> wrote:


On 5/1/2014 1:30 PM, Behcet Sarikaya wrote:



On Thu, May 1, 2014 at 3:20 PM, Joe Touch <touch@isi.edu
<mailto:touch@isi.edu= >> wrote:



=C2=A0 =C2=A0 On 4/30/2014 2:23 PM, Behcet Sarikaya wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Here is what VXLAN says on tunneled traffic:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Tunneled traffic over the IP network can be sec= ured with traditional
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0security mechanisms like IP= sec that authenticate and optionally
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0encrypt VXLAN traffic. This= will, of course, need to be
=C2=A0 =C2=A0 =C2=A0 =C2=A0 coupled with
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0an authentication infrastru= cture for authorized endpoints
=C2=A0 =C2=A0 =C2=A0 =C2=A0 to obtain
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and distribute credentials.=

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Based on this, UDP checksum text seems to be co= nsistent, no?


=C2=A0 =C2=A0 No; the UDP checksum is not for authetication. It is an error= check.

=C2=A0 =C2=A0 The only party that can decide to make the UDP checksum optio= nal
=C2=A0 =C2=A0 when using IPv4 is the source - by inserting zero.

=C2=A0 =C2=A0 It's not the receiver's choice to ignore that checksu= m if it's not
=C2=A0 =C2=A0 zero. That's where this doc breaks the current standards.=

The important point in the above text that I quoted was encryption being optional not about authentication.
So checksum would be zero if the payload is encrypted and non-zero if it is not not and both cases are possible.

Receiver processing is simple:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 - if the checksum is zero, ignore

=C2=A0 =C2=A0 =C2=A0 =C2=A0 - if the checksum is NOT zero, it MUST match
This is true with caveat that an implementation MAY i= gnore a zero checksum, not MUST ignore a zero checksum. This is specified i= n RFC1122. =C2=A0VXLAN draft also breaks this by making accepting packets w= ith zero checksums a MUST. From a practical perspective, if the UDP checksu= m were the only thing I had to protect the vni I would want the capability = to drop packets received with zero checksums in my deployment.

Tom

No other part of the packet needs to be examined. If the *sender* wants to = have the receiver ignore the checksum, it inserts zero. If not, the receive= r MUST process and validate it.

Joe

--001a1135effeb4238904f85cf50f-- From nobody Thu May 1 14:09:51 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 672141A0986; Thu, 1 May 2014 14:09:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9y1mcWr_ex1L; Thu, 1 May 2014 14:09:45 -0700 (PDT) Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by ietfa.amsl.com (Postfix) with ESMTP id 4FC551A0971; Thu, 1 May 2014 14:09:45 -0700 (PDT) Received: from [128.9.184.226] ([128.9.184.226]) (authenticated bits=0) by boreas.isi.edu (8.13.8/8.13.8) with ESMTP id s41L8oAv011076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 1 May 2014 14:08:51 -0700 (PDT) Message-ID: <5362B7E4.8060809@isi.edu> Date: Thu, 01 May 2014 14:08:52 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/G9mtf2OXXMA2QGoEZz9xnjLw92s Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 May 2014 21:09:46 -0000 On 5/1/2014 2:00 PM, Tom Herbert wrote: ... > Receiver processing is simple: > > - if the checksum is zero, ignore > > - if the checksum is NOT zero, it MUST match > > This is true with caveat that an implementation MAY ignore a zero > checksum, not MUST ignore a zero checksum. This is specified in RFC1122. It says that the app MAY optionally be able to configure the system to pass zero-checksum UDP packets up or discard them. I.e., a *configuration* MAY decide to accept or drop zero-checksum UDP packets; the UDP layer isn't in control of that by itself. > VXLAN draft also breaks this by making accepting packets with zero > checksums a MUST. That's not "breaking" anything; VXLAN is - to UDP - an application. Joe From nobody Fri May 2 03:15:49 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 659871A09D4; Fri, 2 May 2014 03:15:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.151 X-Spam-Level: X-Spam-Status: No, score=-10.151 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X3MnVm7Tlxo8; Fri, 2 May 2014 03:15:47 -0700 (PDT) Received: from aer-iport-3.cisco.com (aer-iport-3.cisco.com [173.38.203.53]) by ietfa.amsl.com (Postfix) with ESMTP id AC4C81A6EDC; Fri, 2 May 2014 03:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3300; q=dns/txt; s=iport; t=1399025745; x=1400235345; h=message-id:date:from:reply-to:mime-version:to:cc:subject: references:in-reply-to; bh=2zKEz1Ori9kk/1ibQUFbt9OB5H3CGwZx3Lo6DruTybI=; b=i1ZB9coCd1Di0Olf9oxFkqlgBrCB3l6swm6+54HvbtozLX+AydiL6HMM 769iBoKw/GrZKkI+3/yy5h8RQ1ijI4U5OlL34MClnBb5BPZvTSQxKLibw rws33NKy+6FFkAyfERgTfnGoFebmtHWieZMfzJfG4kNn5VNfe9+M41+T/ 0=; X-IronPort-AV: E=Sophos; i="4.97,971,1389744000"; d="scan'208,217"; a="30413504" Received: from aer-iport-nat.cisco.com (HELO aer-core-3.cisco.com) ([173.38.203.22]) by aer-iport-3.cisco.com with ESMTP; 02 May 2014 10:15:43 +0000 Received: from cisco.com (mrwint.cisco.com [64.103.70.36]) by aer-core-3.cisco.com (8.14.5/8.14.5) with ESMTP id s42AFhbt004512 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 2 May 2014 10:15:43 GMT Received: from STBRYANT-M-R010.CISCO.COM (localhost [127.0.0.1]) by cisco.com (8.14.4+Sun/8.8.8) with ESMTP id s42AFgpa026606; Fri, 2 May 2014 11:15:42 +0100 (BST) Message-ID: <53637052.9050405@cisco.com> Date: Fri, 02 May 2014 11:15:46 +0100 From: Stewart Bryant User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert , "Larry Kreeger (kreeger)" References: In-Reply-To: Content-Type: multipart/alternative; boundary="------------000907050002010403080905" Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/_PV5i9qMYSqtSyWIRWalAqdgLAw Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: stbryant@cisco.com List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 10:15:48 -0000 This is a multi-part message in MIME format. --------------000907050002010403080905 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 01/05/2014 04:14, Tom Herbert wrote: > > > Assuming we are using Ethernet (I don't believe this can be a > requirement either) this only provides hop to hop protection, not end > to end. I don't have a completely error free network and checksum > errors while low, are non-zero. Interesting, for years I have been looking for someone to put forward some statistics on the error rate in a modern network, to get some hard data on the UDP c/s issue. What error rate are you seeing? Also can you help me understand what class of switch/router you are seeing this on: h/w forwarding, s/w forwarding, host basted s/w forwarding... If you know whether the packet memory has any form of error detection/protection that would also be interesting. Thanks Stewart --------------000907050002010403080905 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
On 01/05/2014 04:14, Tom Herbert wrote:


Assuming we are using Ethernet (I don't believe this can be a requirement either) this only provides hop to hop protection, not end to end. I don't have a completely error free network and checksum errors while low, are non-zero.

Interesting, for years I have been looking for someone to put forward
some statistics on the error rate in a modern network, to get some
hard data on the UDP c/s issue. What error rate are you seeing?

Also can you help me understand what class of switch/router you are
seeing this on: h/w forwarding, s/w forwarding, host basted s/w forwarding... If you know whether the packet memory has any form of error detection/protection that would also be interesting.

Thanks

Stewart






--------------000907050002010403080905-- From nobody Fri May 2 09:11:05 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3C7A21A6F54 for ; Fri, 2 May 2014 09:11:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=unavailable Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y6Pps6EpQ_-Q for ; Fri, 2 May 2014 09:11:01 -0700 (PDT) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 78E2F1A08DA for ; Fri, 2 May 2014 09:11:01 -0700 (PDT) Received: by mail-ie0-f171.google.com with SMTP id to1so5287435ieb.2 for ; Fri, 02 May 2014 09:10:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P2CvGYEP3oM6LdtxSQ6T3ReAUn+gYkT3VpCzF87IJxs=; b=bEm/Kcu2y5qAsdd7o4mFYu8q8sYRU8cE1/t1xNyTq7UM44okpG0Tp3rZjehk2ltp78 /Rv7m4r7F9nByi/xCyXNotPIhDMWDXt6jRM3Nrk3bV4AWwTuK9vhMo1Pw+sLq+DNTTau 7wpgJ5m4pPxGJXhsNd0q9etYgR+vdk4Tj11MWPS7lLIFHx9v8KTz0zKbDPuXbcbQverb lYkcrYmTOKq2SoTAtCr20nPnPQZw2DRd5fYPdMLOGACL3ArLaDZL9knQFGQyeLZFAmds 4OVRxIAnKPPMs2Xp3s2vAweoRleo0M+K5NJz3v/1x+tj9im1QEHg2RhgMwK/Qjkm51nY JwZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=P2CvGYEP3oM6LdtxSQ6T3ReAUn+gYkT3VpCzF87IJxs=; b=gWEXMVLaqdWXY/R5RRbMD8BDLTQrJIUrfQK26QycqbYJ5BVuMj8rI0Jcor4PuPUVTf DjfU5+J1xEZEAcdwwiEkMA/kOKyufOKQpZv21QrD6vZ1RXzT0JlvQylRWuF9Wc+FfOw9 QS6pA48Evn+MPvRiQhuNu7XJF4h8nCh7AHjLDv+F20aARvyjTLVTD5TXf8YuQ3OaR5au VGwzatat3xgil8v9O5xNkz6qGCkDlhJXahXsSXQrf9lMu4ScmDJGP2Cbj7Th4qCP+qAs bnpYLaQZKKBdxLwXl/+3kEaR6FAWaSMkeAF8CRIMd6tjlFjhhTCetu9/yloJhezBZOHW fWbA== X-Gm-Message-State: ALoCoQkUAmNN4lQSzJ67utuoQ0k+dMoJWKR070lckgVZ9/vD+j0Y3TDjI9B1fsoQBSoTbM/7DHMl MIME-Version: 1.0 X-Received: by 10.50.25.67 with SMTP id a3mr5550502igg.28.1399047059088; Fri, 02 May 2014 09:10:59 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Fri, 2 May 2014 09:10:58 -0700 (PDT) In-Reply-To: <53637052.9050405@cisco.com> References: <53637052.9050405@cisco.com> Date: Fri, 2 May 2014 09:10:58 -0700 Message-ID: From: Tom Herbert To: stbryant@cisco.com Content-Type: multipart/alternative; boundary=047d7bd74a2cfb49cb04f86d065b Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/dVkUYVkTnjuE1UphqWV_K8TW1Bk Cc: "tofoo@ietf.org" , "nvo3@ietf.org" , "Larry Kreeger \(kreeger\)" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 16:11:03 -0000 --047d7bd74a2cfb49cb04f86d065b Content-Type: text/plain; charset=UTF-8 On Fri, May 2, 2014 at 3:15 AM, Stewart Bryant wrote: > On 01/05/2014 04:14, Tom Herbert wrote: > > > > Assuming we are using Ethernet (I don't believe this can be a > requirement either) this only provides hop to hop protection, not end to > end. I don't have a completely error free network and checksum errors while > low, are non-zero. > > > Interesting, for years I have been looking for someone to put forward > some statistics on the error rate in a modern network, to get some > hard data on the UDP c/s issue. What error rate are you seeing? > > We see well less than 100 errors per day throughout out network. This is internal traffic only, rates on from Internet are much higher as one would expect. I haven't done all the math, but this may be in line with nominal error rate in the network and the probability of an undetected error getting though MAC CRC. My bigger concern is the effects when hardware (or software) fails. Most hardware failures are fairly noticeable and don't corrupt packets, and the usual effects are packet loss or increased latency. We have seen cases though, where hardware fail by corrupting packet data and/or not doing validation correctly. In the worse case, we have seen undetected data corruption make it all the way to the application. Failures like this are still very rare, however when they occur they may be difficult to debug and even identifying the failure may take a long time. To mitigate the risks of undetected data corruption in the network, a separate end to end CRC is commonly performed by the applications over sensitive data. This is a component of an RPC layer for instance. I think we'd like to see the probability of an undetected data corruption to be no more than 2^-64 (I believe the vni in VXLAN qualifies as sensitive data). Note that neither checksum nor CRC provide for authentication which is emerging as another requirement for intra data center traffic-- and encryption as a requirement won't be far behind. Also can you help me understand what class of switch/router you are > seeing this on: h/w forwarding, s/w forwarding, host basted s/w > forwarding... If you know whether the packet memory has any form of error > detection/protection that would also be interesting. > We haven't looked into it at that level, any of those would be suspect including the NIC and host software. The normal errors seem fairly random and since there being detected at the RX host it wouldn't be obvious how to determine where in the network the corruption actually occurred. There is probably both devices that have memory protection and legacy ones that don't. Tom > > Thanks > > Stewart > > > > > > > --047d7bd74a2cfb49cb04f86d065b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On F= ri, May 2, 2014 at 3:15 AM, Stewart Bryant <stbryant@cisco.com> wrote:
=20 =20 =20
On 01/05/2014 04:14, Tom Herbert wrote:


Assuming we are using Ethernet (I don't believe this can be a requirement either) this only provides hop to hop protection, not end to end. I don't have a completely error free network and checksum errors while low, are non-zero.

Interesting, for years I have been looking for someone to put forward
some statistics on the error rate in a modern network, to get some
hard data on the UDP c/s issue. What error rate are you seeing?

We see well less than 100 errors per day th= roughout out network. This is internal traffic only, rates on from Internet= are much higher as one would expect. I haven't done all the math, but = this may be in line with nominal error rate in the network and the probabil= ity of an undetected error getting though MAC CRC.

My bigger concern is the effects when hardware (or soft= ware) fails. Most hardware failures are fairly noticeable and don't cor= rupt packets, and the usual effects are packet loss or increased latency. W= e have seen cases though, where hardware fail by corrupting packet data and= /or not doing validation correctly. In the worse case, we have seen undetec= ted data corruption make it all the way to the application. Failures like t= his are still very rare, however when they occur they may be difficult to d= ebug and even identifying the failure may take a long time.

To mitigate the risks of undetected data corruption in = the network, a separate end to end CRC is commonly performed by the applica= tions over sensitive data. This is a component of an RPC layer for instance= . I think we'd like to see the probability of an undetected data corrup= tion to be no more than 2^-64 (I believe the vni in VXLAN qualifies as sens= itive data).

Note that neither checksum nor CRC provide for authenti= cation which is emerging as another requirement for intra data center traff= ic-- and encryption as a requirement won't be far behind.=C2=A0

Also can you help me understand what class of switch/router you are
seeing this on: h/w forwarding, s/w forwarding, host basted s/w forwarding... If you know whether the packet memory has any form of error detection/protection that would also be interesting.

We haven't looked into it at that level, any = of those would be suspect including the NIC and host software. The normal e= rrors seem fairly random and since there being detected at the RX host it w= ouldn't be obvious how to determine where in the network the corruption= actually occurred.

There is probably both devices that have memory p= rotection and legacy ones that don't.

Tom

=C2=A0
=C2=A0
=
Thanks

Stewart







--047d7bd74a2cfb49cb04f86d065b-- From nobody Fri May 2 14:34:34 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C68D21A6F38 for ; Fri, 2 May 2014 14:34:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=unavailable Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ml-S2k5T8oUY for ; Fri, 2 May 2014 14:34:28 -0700 (PDT) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) by ietfa.amsl.com (Postfix) with ESMTP id E4C3C1A0A7B for ; Fri, 2 May 2014 14:34:27 -0700 (PDT) Received: by mail-ie0-f174.google.com with SMTP id ar20so5779504iec.33 for ; Fri, 02 May 2014 14:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ANL0RxizWgNRn3mid8P57Pb81oLIDJCnrS/Dhf4txSY=; b=knNXXr3B/jEjy7VkJTfDm6eb0bh32to6Ya/W5dqpl1n5qIzNtpDhhHzyiIl6uHoFOQ 88xd8UBFjlfrNfNv12I6UOxM3yKKuXED/suL51yN5/RRvh46A/wFC69wWwBvqu/1ARtp lWpYfx+P1QVTaP0ImWWODqDz2FsCYFDRrULmm4OClI8JLAAm4A1A5lp88orJxAkF6+JX fwL6df2laSqHFKBpGjmq7qiKARt0dNBLzPCNkLiiyaL/ik4HffGQRybFFNDY/1dxultM g/Oi1TNFJMagHBDe6j3sImEEgyez1vn4GRPbsc0EFqvMgXWadd0Do456CcDiol42zkq3 6+xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ANL0RxizWgNRn3mid8P57Pb81oLIDJCnrS/Dhf4txSY=; b=CS2zjLq74CvB3gsjyCDuNxbtiDr+PpcHBh9GLJ1kZSRrwRHM0SKCP0zbxYmaqoYOFm vGQ1qKJRRA0E14MpvCWideAfjdL2Mv4ssQPYpzVZZwmFffJ4B4z9YwJwdEq9TGVb/T6F JazwG1l+eCpEqxf/PE/sR59gNq2Hwrvh4rKk+8VuG/JKSEp786UJ6jHdiWavs/oaDFiD AmMOmXT3SZUo5ggFKVYs70zjq8B/EUfXVSTytPWMUy62BJktuvrM2q7X2HZiBDgV4HLX m0gpzfmvjQza2ffc2IE5b0mLtKmcsbJ/evBcp4vpk9v+7oYSUkOEQeSbOA40wAt5fY2x 9zCw== X-Gm-Message-State: ALoCoQl1rFpQ9zsbE23gr5cyhs41dIPNgAGNNz5ArAQyNFx9xuoa1fT/Z2ERNOOmbl7jQ7nL6E3p MIME-Version: 1.0 X-Received: by 10.43.59.82 with SMTP id wn18mr19561981icb.6.1399066465372; Fri, 02 May 2014 14:34:25 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Fri, 2 May 2014 14:34:25 -0700 (PDT) In-Reply-To: <5362B7E4.8060809@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> Date: Fri, 2 May 2014 14:34:25 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: multipart/alternative; boundary=bcaec51961f1afac9904f8718b4f Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/-ZzzBptmm37dQomh0XZ5bwtplvU Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 21:34:30 -0000 --bcaec51961f1afac9904f8718b4f Content-Type: text/plain; charset=UTF-8 On Thu, May 1, 2014 at 2:08 PM, Joe Touch wrote: > > > On 5/1/2014 2:00 PM, Tom Herbert wrote: > ... > > Receiver processing is simple: >> >> - if the checksum is zero, ignore >> >> - if the checksum is NOT zero, it MUST match >> >> This is true with caveat that an implementation MAY ignore a zero >> checksum, not MUST ignore a zero checksum. This is specified in RFC1122. >> > > It says that the app MAY optionally be able to configure the system to > pass zero-checksum UDP packets up or discard them. > > I.e., a *configuration* MAY decide to accept or drop zero-checksum UDP > packets; the UDP layer isn't in control of that by itself. > > > VXLAN draft also breaks this by making accepting packets with zero >> checksums a MUST. >> > > That's not "breaking" anything; VXLAN is - to UDP - an application. > > Perhaps not, but making it a MUST in the encapsulation protocol definition to always accept UDP packets with zero checksums seems a bit austere. For instance, if I know within my deployment that all senders are configured to use checksums, then receiving a packet with zero checksum should be considered an invalid packet. Tom Joe > --bcaec51961f1afac9904f8718b4f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, May 1, 2014 at 2:08 PM, Joe Touch <touch@isi.edu> wrote:


On 5/1/2014 2:00 PM, Tom Herbert wrote:
...

=C2=A0 =C2=A0 Receiver processing is simple:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0- if the checksum is zero, = ignore

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0- if the checksum is NOT ze= ro, it MUST match

This is true with caveat that an implementation MAY ignore a zero
checksum, not MUST ignore a zero checksum. This is specified in RFC1122.

It says that the app MAY optionally be able to configure the system to pass= zero-checksum UDP packets up or discard them.

I.e., a *configuration* MAY decide to accept or drop zero-checksum UDP pack= ets; the UDP layer isn't in control of that by itself.
<= br>
=C2=A0 VXLAN draft also breaks this by making accepting packets with zero checksums a MUST.

That's not "breaking" anything; VXLAN is - to UDP - an applic= ation.

Perhaps not, but making it a MUST in th= e encapsulation protocol definition to always accept UDP packets with zero = checksums seems a bit austere. For instance, if I know within my deployment= that all senders are configured to use checksums, then receiving a packet = with zero checksum should be considered an invalid packet.

Tom

= Joe

--bcaec51961f1afac9904f8718b4f-- From nobody Fri May 2 15:26:12 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 801141A0957; Fri, 2 May 2014 15:26:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EKJU97U_t1ix; Fri, 2 May 2014 15:26:03 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) by ietfa.amsl.com (Postfix) with ESMTP id 2F1081A6FE3; Fri, 2 May 2014 15:26:02 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s42MPA0Y028420 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 May 2014 15:25:10 -0700 (PDT) Message-ID: <53641B46.5000200@isi.edu> Date: Fri, 02 May 2014 15:25:10 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/Oa-WgjCEVuR8-XJM4kZGU5LxiqY Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 22:26:05 -0000 On 5/2/2014 2:34 PM, Tom Herbert wrote: ... > VXLAN draft also breaks this by making accepting packets with > zero checksums a MUST. > > That's not "breaking" anything; VXLAN is - to UDP - an application. > > Perhaps not, but making it a MUST in the encapsulation protocol > definition to always accept UDP packets with zero checksums seems a bit > austere. For instance, if I know within my deployment that all senders > are configured to use checksums, then receiving a packet with zero > checksum should be considered an invalid packet. Sorry if I'm missing something here - this started with the following, from draft-mahalingam-dutt-dcops-vxlan-09, which focuses on the non-zero checksum case: "When a decapsulating endpoint receives a packet with a non-zero checksum it MAY choose to verify the checksum" The point is that this is NOT an option. If the UDP checksum is nonzero, it MUST be validated. (this isn't open to interpretation; if you want to have VXLAN accept non-zero checksums that are invalid, then you are not compliant with what few standards the Internet has) --- When the UDP checksum is zero, RFC1122 says that: - an app MAY optionally be able to configure a system to accept zero-checksum UDP packets up or discard them It would be a bad idea for VXLAN to assume that this configuration is available. However, that leaves only two cases: 1) if accept/discard of zero-checksum IS available, then VXLAN would configure it to accept 2) if accept/discard of zero-checksum is NOT available, then UDP *MUST* pass the zero-checksum packets to VXLAN. It's up to VXLAN to decide how to handle them - and it's OK to require VXLAN to MUST accept them (i.e., to treat them the same way as a valid non-zero checksum) When you have a stronger validation mechanism, I see no reason to drop packets with zero checksums for that reason alone - however, VXLAN really needs to say that *it* will do that drop if necessary, not to assume that UDP can be configured to do so. Joe From nobody Fri May 2 15:34:15 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CBBA81A6FDE for ; Fri, 2 May 2014 15:34:13 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.03 X-Spam-Level: X-Spam-Status: No, score=-2.03 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PKZcThf69M3Z for ; Fri, 2 May 2014 15:34:12 -0700 (PDT) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) by ietfa.amsl.com (Postfix) with ESMTP id DEF9E1A0957 for ; Fri, 2 May 2014 15:34:11 -0700 (PDT) Received: by mail-ig0-f182.google.com with SMTP id uy17so925384igb.3 for ; Fri, 02 May 2014 15:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1Mla9aodA+3AvstUHbKyml78ldvof/p+sMYRVZX/z/Y=; b=HrgJUb8TddU1Jouo+zJbO0rHxVsViHSAFmRvP8UGpS+TLbo1sTejFYUuqqtRaybcLP mnMefdzSXLubMogYrSpfKHiLgTmNWG3omhTAcgVr8tlNZ+x1XnS9NvQ7mUWZ1Lw1TskU WozP7NchbJFhRpVe9ifYg9J9fwGhtsRN1wdnL3YB65zJKE39SORXxFzmj6F6mZ/o9zeP gDPRLIkwbGw1oHJcOM/Z+mX9CFQDTpgT3/luAcw9vjcJh2mbS2A30Rsb8oq9DbWmaAFo m2U9gD2vtHKBo5uJfyNKad/QhfaE6EPljLkLyAVvROV66/29AHrK2L2Rb5rp69tXmc4k faWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=1Mla9aodA+3AvstUHbKyml78ldvof/p+sMYRVZX/z/Y=; b=G2t/GpYWYJH1PPRmEiexKwJrMpoR23io8ID3c7HWfw60wQTkW1DxGg+5d0ZhsTyw4b 0lD+V3pGJpasjrWE8DI6Duj3zFjZqzt4Enl2Y5LbCWewWKRC8u8X9tVeETcymZdfNyDa onNrUtEO0XumTrQI8P4gBbOZ0ye49VnUPAOwDysGv14NxMyx+TOhU/ALvCsif6y/pwPV D7En6fwzGeKU99gRRGYGaXG0DmBzvK4JhK27hz23OA0cl5Uge2iNGzEn6Dq6MK68vdjQ d9FGszgsySRsna9wajyTpsS/1zEZTOCbdTElwdDWlSKI/ZXAAXgLWoeyIOaIivMrnUr2 FwWw== X-Gm-Message-State: ALoCoQlVylys3dyVZyc7Dc7jIFuw8TM7sa7rjNixZTGBIYvirawbpMu6WgtZRD3t1sDR00wVLLts MIME-Version: 1.0 X-Received: by 10.50.25.201 with SMTP id e9mr8119604igg.28.1399070049367; Fri, 02 May 2014 15:34:09 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Fri, 2 May 2014 15:34:09 -0700 (PDT) In-Reply-To: <53641B46.5000200@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> Date: Fri, 2 May 2014 15:34:09 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: text/plain; charset=UTF-8 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/tESU0khchGIj8P0F7i8SoH7l_AY Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 22:34:13 -0000 On Fri, May 2, 2014 at 3:25 PM, Joe Touch wrote: > > > > On 5/2/2014 2:34 PM, Tom Herbert wrote: > ... >> >> VXLAN draft also breaks this by making accepting packets with >> >> zero checksums a MUST. >> >> That's not "breaking" anything; VXLAN is - to UDP - an application. >> >> Perhaps not, but making it a MUST in the encapsulation protocol >> definition to always accept UDP packets with zero checksums seems a bit >> austere. For instance, if I know within my deployment that all senders >> are configured to use checksums, then receiving a packet with zero >> checksum should be considered an invalid packet. > > > Sorry if I'm missing something here - this started with the following, from draft-mahalingam-dutt-dcops-vxlan-09, which focuses on the non-zero checksum case: > > > "When a decapsulating endpoint receives a packet with a non-zero > checksum it MAY choose to verify the checksum" > > The point is that this is NOT an option. If the UDP checksum is nonzero, it MUST be validated. > > (this isn't open to interpretation; if you want to have VXLAN accept non-zero checksums that are invalid, then you are not compliant with what few standards the Internet has) > I am referring to: "When a packet is received with a UDP checksum of zero, it MUST be accepted for decapsulation." > > --- > > When the UDP checksum is zero, RFC1122 says that: > > - an app MAY optionally be able to configure a system > to accept zero-checksum UDP packets up or discard them > > It would be a bad idea for VXLAN to assume that this configuration is available. However, that leaves only two cases: > > 1) if accept/discard of zero-checksum IS available, then > VXLAN would configure it to accept > > 2) if accept/discard of zero-checksum is NOT available, > then UDP *MUST* pass the zero-checksum packets to VXLAN. > It's up to VXLAN to decide how to handle them - and it's > OK to require VXLAN to MUST accept them (i.e., to treat > them the same way as a valid non-zero checksum) > > When you have a stronger validation mechanism, I see no reason to drop packets with zero checksums for that reason alone - however, VXLAN really needs to say that *it* will do that drop if necessary, not to assume that UDP can be configured to do so. > > Joe > > > > > > From nobody Fri May 2 15:45:55 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8D0761A09C3; Fri, 2 May 2014 15:45:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ta080TWruD-x; Fri, 2 May 2014 15:45:53 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) by ietfa.amsl.com (Postfix) with ESMTP id 643391A0957; Fri, 2 May 2014 15:45:53 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s42MjT4S005175 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 May 2014 15:45:29 -0700 (PDT) Message-ID: <53642009.6000805@isi.edu> Date: Fri, 02 May 2014 15:45:29 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/_WvrSZ46vFf_dpfg4eL1RnmzTF8 Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 22:45:54 -0000 On 5/2/2014 3:34 PM, Tom Herbert wrote: > On Fri, May 2, 2014 at 3:25 PM, Joe Touch wrote: >> >> >> >> On 5/2/2014 2:34 PM, Tom Herbert wrote: >> ... >>> >>> VXLAN draft also breaks this by making accepting packets with >>> >>> zero checksums a MUST. >>> >>> That's not "breaking" anything; VXLAN is - to UDP - an application. >>> >>> Perhaps not, but making it a MUST in the encapsulation protocol >>> definition to always accept UDP packets with zero checksums seems a bit >>> austere. For instance, if I know within my deployment that all senders >>> are configured to use checksums, then receiving a packet with zero >>> checksum should be considered an invalid packet. >> >> >> Sorry if I'm missing something here - this started with the following, from draft-mahalingam-dutt-dcops-vxlan-09, which focuses on the non-zero checksum case: >> >> >> "When a decapsulating endpoint receives a packet with a non-zero >> checksum it MAY choose to verify the checksum" >> >> The point is that this is NOT an option. If the UDP checksum is nonzero, it MUST be validated. >> >> (this isn't open to interpretation; if you want to have VXLAN accept non-zero checksums that are invalid, then you are not compliant with what few standards the Internet has) >> > I am referring to: > > "When a packet is received with a UDP checksum of zero, it MUST be > accepted for decapsulation." That is a completely valid choice w.r.t. RFCs 1122 and 768. 1122 is the most specific on these points: A host MUST implement the facility to generate and validate UDP checksums. An application MAY optionally be able to control whether a UDP checksum will be generated, but it MUST default to checksumming on. If a UDP datagram is received with a checksum that is non- zero and invalid, UDP MUST silently discard the datagram. An application MAY optionally be able to control whether UDP datagrams without checksums should be discarded or passed to the application. The "application" here is VXLAN. So you can say that VXLAN SHOULD set the UDP checksum to zero, but you cannot say that it MUST (because app-layer control is a MAY). When receiving nonzero invalid checksums, you cannot say that VLXAN will process those packets, because UDP MUST silently discard them. When receiving zero checksums, you cannot say that VXLAN MUST NOT see them, because app-layer control of discard is optional. ---- I think your objection may be that you don't think we can assume that zero-checksum packets arrive at VXLAN, because of how RFC1122 is worded. However, RFC768 permits zero-checksum packets as valid and useful to the upper layer, so I interpret RFC 1122 as saying "the app might be able to configure UDP to discard zero-checksum packets", but if it doesn't, those packets already will end up at the application. Joe From nobody Fri May 2 16:15:45 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2FD1E1A6FDE for ; Fri, 2 May 2014 16:15:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.03 X-Spam-Level: X-Spam-Status: No, score=-2.03 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UM9_Mt7c2REa for ; Fri, 2 May 2014 16:15:35 -0700 (PDT) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) by ietfa.amsl.com (Postfix) with ESMTP id 9FF881A6FD4 for ; Fri, 2 May 2014 16:15:35 -0700 (PDT) Received: by mail-ig0-f177.google.com with SMTP id l13so60093iga.10 for ; Fri, 02 May 2014 16:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vzryqxBQqYEhNy6m/3sEMl5HlIOmC2oqYHQaer421zM=; b=KfF7Qm8H+U9KrkB6ikeD5MPCjq1JiA5HQEjeVfU/6mHEt3HphOMPKde/yPawCX6FU2 KLtv3jAE3azXaJKMRmbC9LJNoZO/Zu3U7lr84+AnARefS8gnZ4+3qnCxD7m3IiuSDyaz LHdzw017gd694uONzv+Dr3cfkGmBJDCDDxSwHm61Yo8KpvIp+8dwzSR5wyLpJP4paO7Y lXm1jyyIhsTBGM6InnlSjF9QvbhuvdBnX8GWCYhZxzGC2DlFYiF76+zMilCFIopwg3e4 ER4YItWCgK6oFfTc0Nor+Wd2DEjYsnsRV/ChCckiNNiqo5A8ghyipKZ2UdB6S5wqswCp 5XyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vzryqxBQqYEhNy6m/3sEMl5HlIOmC2oqYHQaer421zM=; b=cvfZdkNZxYzEf8Grr3aMz0mPZtyMTUWIGqsNLOZAD3R0vLNrN1QIXyEWAkMw7vMu3u OzINfkMDGPYxExTXKBpml6H/knK1AfMID1J78vLcDiijJkUaZitXTaqAvW/gX4S9t0T9 tiIOcoYm3ZZK8AoUfxnKKfBAWnAhVu4dqEUQ4fZRjWUW0TdTFAr1TFInuuI0s5B323Im zheGqEM1qTsDowD2T5HFKtR+HIuI4F1pKuJPuZwh4uSd5KI59Xjhq32FswRZ5xxD/pVh zH3cwu0fgbuJmGQAVEmxqimI4w1GITHVf4WL8bNDxjOhIhzTFthnw1Vdoc7/nStiTgco z6jQ== X-Gm-Message-State: ALoCoQkLrFXriZtjptA2iyWJUTvp/DkJnrOJE+unqfDOy+zii79xHMhuy12582WVJrLBDlf9tF9R MIME-Version: 1.0 X-Received: by 10.43.59.82 with SMTP id wn18mr19896014icb.6.1399072532931; Fri, 02 May 2014 16:15:32 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Fri, 2 May 2014 16:15:32 -0700 (PDT) In-Reply-To: <53642009.6000805@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> <53642009.6000805@isi.edu> Date: Fri, 2 May 2014 16:15:32 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: text/plain; charset=UTF-8 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/BqfK3imMyP3R4FbCCMKr9w_wjYM Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 23:15:37 -0000 > I think your objection may be that you don't think we can assume that > zero-checksum packets arrive at VXLAN, because of how RFC1122 is worded. No, I expect zero checksums will be the common case. > However, RFC768 permits zero-checksum packets as valid and useful to the > upper layer, so I interpret RFC 1122 as saying "the app might be able to > configure UDP to discard zero-checksum packets", but if it doesn't, those > packets already will end up at the application. > My interpretation of the VXLAN draft precludes the possibility of an implementation being able to reject received zero-checksums even as a configuration option, or even for select senders. So if we receive a zero checksum from a sender that we know has not disabled checksums, per the draft we need to accept even though we know this is a corrupted packet. This may not break the standard, but it doesn't seem robust either. Is my interpretation correct? My core concern in all of this is still whether the vni in VXLAN in being adequately protected against corruption (this would apply to other encapsulation protocols that carry vni also). The integrity of the vni is paramount in supporting the isolation requirements of network virtualization. Thanks, Tom > Joe From nobody Fri May 2 16:59:04 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 96A771A6FDE; Fri, 2 May 2014 16:59:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2cz9yaHBh29H; Fri, 2 May 2014 16:59:00 -0700 (PDT) Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by ietfa.amsl.com (Postfix) with ESMTP id 2D01B1A6F63; Fri, 2 May 2014 16:59:00 -0700 (PDT) Received: from [128.9.184.177] ([128.9.184.177]) (authenticated bits=0) by boreas.isi.edu (8.13.8/8.13.8) with ESMTP id s42Nw4IT016849 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 May 2014 16:58:04 -0700 (PDT) Message-ID: <5364310E.5010306@isi.edu> Date: Fri, 02 May 2014 16:58:06 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> <53642009.6000805@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/18x_w5MBGJsteWajWEaAxCLj7F4 Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 23:59:02 -0000 On 5/2/2014 4:15 PM, Tom Herbert wrote: >> I think your objection may be that you don't think we can assume that >> zero-checksum packets arrive at VXLAN, because of how RFC1122 is worded. > > No, I expect zero checksums will be the common case. I meant that you don't think that this common case will be correctly handled. >> However, RFC768 permits zero-checksum packets as valid and useful to the >> upper layer, so I interpret RFC 1122 as saying "the app might be able to >> configure UDP to discard zero-checksum packets", but if it doesn't, those >> packets already will end up at the application. > > My interpretation of the VXLAN draft precludes the possibility of an > implementation being able to reject received zero-checksums even as a > configuration option, or even for select senders. Agreed - as writte, it cannot reject those packets merely because thye have a zero checksum - "When a packet is received with a UDP checksum of zero, it MUST be accepted for decapsulation. " (note that I have no position as to whether this is good or bad; VXLAN could decide to reject such packets if that's what the WG decides). > So if we receive a > zero checksum from a sender that we know has not disabled checksums, > per the draft we need to accept even though we know this is a > corrupted packet. I don't know how you can make this claim. You don't know it's a corrupted packet - esp. because it's highly unlikely that the checksum would be zero'd solely by corruption. What you know - at best - is that the source decided to send a zero-checksum packet. A sender that transmits zero checksums does so deliberately (on an ongoing basis). It does so when it believes that another layer will check for packet corruption, or when packet corruption isn't important. > This may not break the standard, but it doesn't seem > robust either. Is my interpretation correct? VXLAN can make whatever decision you want; you can decide to accept zero-sum packets for further processing, to reject them, or to decide on a per-endpoint basis given other knowledge of the endpoint. All behaviors are compliant with current standards. What you cannot do is accept a packet with a non-zero checksum that is invalid. Those should never be sent to VXLAN; they MUST be silently dropped by UDP. That's where this discussion originated. As to robustness, IMO, the best would be to send packets with UDP checksums, so you drop corrupt packets as early as possible. If you don't want to incur the cost of checking (which is often in offload engines anyway), then you'd be relying on the upper layer (VXLAN layer) to check for errors. However, I don't see that here - what I see is an ethernet header but that doesn't protect the E2E of the UDP packet, and that's not good enough. > My core concern in all of this is still whether the vni in VXLAN in > being adequately protected against corruption (this would apply to > other encapsulation protocols that carry vni also). The integrity of > the vni is paramount in supporting the isolation requirements of > network virtualization. Then what you want is to require VXLAN to MUST send non-zero UDP checksums. At that point, saying "VXLAN receiver MUST discard zero-checksums" is fine, and both statements are compliant with existing standards. Alternately, if you allow - or encourage - zero-checksum UDP, then you want to add a checksum of some kind to the VXLAN header. But again, you're talking about the zero-checksum case, not the non-zero case where this all started. Joe From nobody Fri May 2 18:47:41 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE35A1A7026 for ; Fri, 2 May 2014 18:47:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.03 X-Spam-Level: X-Spam-Status: No, score=-2.03 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=unavailable Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lg-fFgLOqAbi for ; Fri, 2 May 2014 18:47:38 -0700 (PDT) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by ietfa.amsl.com (Postfix) with ESMTP id 6800D1A6F53 for ; Fri, 2 May 2014 18:47:38 -0700 (PDT) Received: by mail-ie0-f181.google.com with SMTP id y20so5883442ier.40 for ; Fri, 02 May 2014 18:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=75oSHPLqV7nFIplY6Q805mVkf9AtBqQraw7clOgbVRo=; b=MfiM3tox7JHOCQWfOfeYbY1x6VG4TvyuLAx/j++Ml/1rouYexaZxAZECUlQtalrkNr xxrusCNYFClONnQNPoKCIAQpnIGbKQWcYBA5qSqis8TI/H5OeEy6mjtzCZqABV4JfgXP QrgxZR6avP3Y+gt1mkGGGkJESniyD1yxLUDNoQJSbiDc5LSQ5sXg8ky2AcohQcoH2MV+ JPs885yw35/qOVUwU5B/YGWLk1ThO4+UZuAlaxestTU3jfCg6uuR4W7NvHRGzvBKy7MZ kbA1bRZIehCTzp/XbT9QpJFzWG/ySX424fEpovh3HIPHVmb9SiHANw4TXmRpVwBCJCDK HiSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=75oSHPLqV7nFIplY6Q805mVkf9AtBqQraw7clOgbVRo=; b=EDJhWBA1cMaWRHfBwFToHvRLlsEvjdfMQln5taCQiNRg6osJMnFoejhyFJ6qHja/KC CzC6mD8JDxmBraMoAVZmjbaEj3MGHLGwYQC9tEEK0J5NuOGKB6ltZyeIw/UdE6rrNQAW kfASU11rBHOYk2A16Jlwt6ECG33qBR5zHtNVgXRh7RNYF8Chepxw6nXU7tWi9Ax7IFAd Ts3BgweEzCkQRwcUSEUgVGG8Wowu4gFbk94DmhF78RZovDvAdMAKfKMWM/fZPdzlabx4 9QVhMlquiNNv76qK2iE7vwqzTDLB9zFaNu4LskoF0EA0qXF/WvshgviarHpBnl/1Ops0 q0NA== X-Gm-Message-State: ALoCoQmmnpubD3QemVwMXjB8f5hVD2/waFCRC/i3YOX4KE0G0ZlIII9+F2LhZqxy8aUoHttvh//C MIME-Version: 1.0 X-Received: by 10.50.25.201 with SMTP id e9mr8746869igg.28.1399081655877; Fri, 02 May 2014 18:47:35 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Fri, 2 May 2014 18:47:35 -0700 (PDT) In-Reply-To: <5364310E.5010306@isi.edu> References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> <53642009.6000805@isi.edu> <5364310E.5010306@isi.edu> Date: Fri, 2 May 2014 18:47:35 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: text/plain; charset=UTF-8 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/_UvK--fMUezbiV0m2ddMhOwsAIY Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 May 2014 01:47:40 -0000 > I don't know how you can make this claim. > > You don't know it's a corrupted packet - esp. because it's highly unlikely > that the checksum would be zero'd solely by corruption. What you know - at > best - is that the source decided to send a zero-checksum packet. > Factually, I only know that I received a packet with checksum of zero, not that one want sent. If I have external information that says the sender has not disabled checksums then something must have gone awry. This is not just the rare case of corrupted checksum value, but unfortunately zero is the likely value if the sender is not properly setting the checksum. Since the checksum is always zeroed in the packet before computation, there are many opportunities for bugs in drivers, stacks, and HW where the checksum is not actually written correctly (especially possible in presence of TSO and checksum offload in NICs)-- in this case packets may be sent incorrectly with a checksum of zero. This condition could be difficult to detect since everything might otherwise appear okay. I would take this into consideration when contemplating use of zero checksums. > A sender that transmits zero checksums does so deliberately (on an ongoing > basis). It does so when it believes that another layer will check for packet > corruption, or when packet corruption isn't important. > > >> This may not break the standard, but it doesn't seem >> robust either. Is my interpretation correct? > > > VXLAN can make whatever decision you want; you can decide to accept zero-sum > packets for further processing, to reject them, or to decide on a > per-endpoint basis given other knowledge of the endpoint. All behaviors are > compliant with current standards. > > What you cannot do is accept a packet with a non-zero checksum that is > invalid. Those should never be sent to VXLAN; they MUST be silently dropped > by UDP. That's where this discussion originated. > > As to robustness, IMO, the best would be to send packets with UDP checksums, > so you drop corrupt packets as early as possible. If you don't want to incur > the cost of checking (which is often in offload engines anyway), then you'd > be relying on the upper layer (VXLAN layer) to check for errors. > > However, I don't see that here - what I see is an ethernet header but that > doesn't protect the E2E of the UDP packet, and that's not good enough. > > >> My core concern in all of this is still whether the vni in VXLAN in >> being adequately protected against corruption (this would apply to >> other encapsulation protocols that carry vni also). The integrity of >> the vni is paramount in supporting the isolation requirements of >> network virtualization. > > > Then what you want is to require VXLAN to MUST send non-zero UDP checksums. > At that point, saying "VXLAN receiver MUST discard zero-checksums" is fine, > and both statements are compliant with existing standards. > > Alternately, if you allow - or encourage - zero-checksum UDP, then you want > to add a checksum of some kind to the VXLAN header. > > But again, you're talking about the zero-checksum case, not the non-zero > case where this all started. > > Joe From nobody Fri May 2 21:07:40 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 21DCA1A000A; Fri, 2 May 2014 21:07:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RiV1cXZLsaUT; Fri, 2 May 2014 21:07:35 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) by ietfa.amsl.com (Postfix) with ESMTP id 1B08A1A0004; Fri, 2 May 2014 21:07:35 -0700 (PDT) Received: from [192.168.1.93] (pool-71-105-87-112.lsanca.dsl-w.verizon.net [71.105.87.112]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s43478Tf005500 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 2 May 2014 21:07:15 -0700 (PDT) Message-ID: <53646B6D.9050203@isi.edu> Date: Fri, 02 May 2014 21:07:09 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <5362ACA5.1030102@isi.edu> <5362AFBB.6080008@isi.edu> <5362B7E4.8060809@isi.edu> <53641B46.5000200@isi.edu> <53642009.6000805@isi.edu> <5364310E.5010306@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/nA9v4wd3slujmVHlGpzOc4ry4oM Cc: "tofoo@ietf.org" , sarikaya@ieee.org, "nvo3@ietf.org" , "mallik_mahalingam@yahoo.com" , "ddutt.ietf@hobbesdutt.com" Subject: Re: [Tofoo] VXLAN (UDP tunnel protocols) and non-zero checksums X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 May 2014 04:07:37 -0000 On 5/2/2014 6:47 PM, Tom Herbert wrote: >> I don't know how you can make this claim. >> >> You don't know it's a corrupted packet - esp. because it's highly unlikely >> that the checksum would be zero'd solely by corruption. What you know - at >> best - is that the source decided to send a zero-checksum packet. >> > Factually, I only know that I received a packet with checksum of zero, > not that one want sent. If I have external information that says the > sender has not disabled checksums then something must have gone awry. Absolutely. Let's assume you follow the VXLAN draft. In that case: The UDP checksum field SHOULD be transmitted as zero. When a packet is received with a UDP checksum of zero, it MUST be accepted for decapsulation. The proposed spec says that the source can either enable or disable the checksum (SHOULD doesn't mean MUST). It says nothing about tracking whether the source has disabled zero checksum. It's very clear that when you get a zero checksum, you should accept it. If you want to track things the way you're proposing, then you need to update that draft. --- However, the case you're proposing is bizarre - it would happen only if the endpoint said one thing and did another. It *could* happen from an error, but that's unlikely. So let's say it DID happen - and that the spec says to check (which it currently doesn't). You then have conflicting info - on one hand, the endpoint said it would not zero the checksum, and on the other hand it did. So what do you do? You can treat *EITHER* the packet or the info about the endpoint as incorrect - and you have *no* information that tells you which. You want to treat the packet as an error, but the info you have about the endpoint is equally suspect. IMO, this is a case that's useless to consider in the spec; at that point, you're have no reason to trust anything you know about that endpoint at all, and you might as well cut it off completely -- regardless of what it sends. > This is not just the rare case of corrupted checksum value, but > unfortunately zero is the likely value if the sender is not properly > setting the checksum. Protocol specs are not designed to address or avoid all implementation errors. Your protocol ought to care whether the checksum is zero - or not care. The current draft doesn't care. > Since the checksum is always zeroed in the > packet before computation, there are many opportunities for bugs in > drivers, stacks, and HW where the checksum is not actually written > correctly (especially possible in presence of TSO and checksum offload > in NICs)-- in this case packets may be sent incorrectly with a > checksum of zero. If you want to prevent your protocol from being susceptible to this kind of implementation error, you need to change the spec so you always drop packets when the checksum is zero. However, checksum errors aren't there to find implementation errors; they're there to find transmission or copying errors. > This condition could be difficult to detect since > everything might otherwise appear okay. I would take this into > consideration when contemplating use of zero checksums. This discussion wasn't about zero checksums, but if that's what you want to discuss, here's my view: - allow zero checksums when you don't care about checksums - if you care about checksums, don't allow zero checksums Anything else, esp. per endpoint, is madness, IMO. The use of checksums ought to depend on whether you care about the errors they find - not the failure modes that might generate zero checksums. Joe From nobody Mon May 19 20:01:46 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 605AC1A0493; Mon, 19 May 2014 20:01:45 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.152 X-Spam-Level: X-Spam-Status: No, score=-23.152 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IPE85wVe7rtk; Mon, 19 May 2014 20:01:43 -0700 (PDT) Received: from den-mipot-001.corp.ebay.com (den-mipot-001.corp.ebay.com [216.113.175.152]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A28811A0257; Mon, 19 May 2014 20:01:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400554904; x=1432090904; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=VS+e33OUYs2kVSx6wLt0Ej1BHR0ZQoY58xtj2K4X5og=; b=VubZIJT+2c1gQQ58+pzpe3+4peBuVoFjYhEpRiWNDXQxA/1AsCYwM1mZ FY5YkYBmbE+o1urZHMghiCOJwX9kNp2K8r+6rcq8qyKenl0Xp4nLpbgEK jZTFqq2gNUfXUu/hPoHyZiE1ISF4Rc2Ixoo0C3gfEm72wiYGw24dwdvpG A=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos;i="4.98,870,1392192000"; d="scan'208";a="50789047" Received: from den-vteml-004.corp.ebay.com (HELO DEN-EXMHT-003.corp.ebay.com) ([10.101.112.120]) by den-mipot-001.corp.ebay.com with ESMTP; 19 May 2014 20:01:43 -0700 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXMHT-003.corp.ebay.com ([fe80::55d3:9d86:3fc8:dbf4%14]) with mapi id 14.03.0174.001; Mon, 19 May 2014 21:01:43 -0600 From: "Zhou, Han" To: "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Thread-Topic: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOg Date: Tue, 20 May 2014 03:01:42 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> In-Reply-To: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/FLrLSggLR1r8iAVa1jdIr8tz0zg Cc: Erik Nordmark , Tom Herbert Subject: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 May 2014 03:01:45 -0000 Hi, We have updated the VXLAN-SOE draft according to earlier comments. Now it i= s fully compatible with VXLAN-GPE. And some examples are added for better u= nderstanding. A prototype is also implemented here (patch based on Open vSwitch): https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa= 050f7dd2be netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G int= erfaces: Before the change: 2.62 Gbits/sec After the change: 6.68 Gbits/sec Speedup is ~250%. The patch attracted some interests in OVS community, but since this RFC dra= ft is in very early stage so it is regarded as inappropriate by Jesse to ap= ply the change to OVS tree. The discuss mail-thread:=20 http://openvswitch.org/pipermail/discuss/2014-May/013981.html http://openvswitch.org/pipermail/discuss/2014-May/013898.html So we would like to request a review here by NVO3/TOFOO groups and VXLAN au= thors: is this VXLAN extension is worth formally put into VXLAN as a standa= rd, so that more people can benefit from it? Best regards, Han -----Original Message----- From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of inte= rnet-drafts@ietf.org Sent: Friday, May 02, 2014 8:09 PM To: i-d-announce@ietf.org Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt A New Internet-Draft is available from the on-line Internet-Drafts director= ies. Title : Segmentation Offloading Extension for VXLAN Authors : Han Zhou Chengyuan Li Filename : draft-zhou-li-vxlan-soe-01.txt Pages : 13 Date : 2014-05-02 Abstract: Segmentation offloading is nowadays common in network stack implementation and well supported by para-virtualized network device drivers for virtual machine (VM)s. This draft describes an extension to Virtual eXtensible Local Area Network (VXLAN) so that segmentation can be decoupled from physical/underlay networks and offloaded further to the remote end-point thus improving data-plane performance for VMs running on top of overlay networks. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ There's also a htmlized version available at: http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 A diff from the previous version is available at: http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe-01 Please note that it may take a couple of minutes from the time of submissio= n until the htmlized version and diff are available at tools.ietf.org. Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ _______________________________________________ I-D-Announce mailing list I-D-Announce@ietf.org https://www.ietf.org/mailman/listinfo/i-d-announce Internet-Draft directories: http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt From nobody Tue May 20 10:33:49 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE67C1A02D4; Tue, 20 May 2014 10:33:47 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2RZjWFx4o0_i; Tue, 20 May 2014 10:33:46 -0700 (PDT) Received: from vapor.isi.edu (vapor.isi.edu [128.9.64.64]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A455B1A0158; Tue, 20 May 2014 10:33:45 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by vapor.isi.edu (8.13.8/8.13.8) with ESMTP id s4KHSflE016590 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 20 May 2014 10:28:50 -0700 (PDT) Message-ID: <537B90C9.1090003@isi.edu> Date: Tue, 20 May 2014 10:28:41 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: "Zhou, Han" , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/Ej8j9itFWi_rrnqeu40DChev0ww Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 May 2014 17:33:48 -0000 Hi, all, I had a comment and a question: Comment - (from the doc) overlays do have a hard MTU limit; it is the limit of the encapsulation mechanism. E.g., without additional layers, for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max - (min IP header + UDP header)). Using additional adaptation layers, this could be larger (e.g., see SEAL). Question - the code appears to have the VXLAN layer do the fragmentation, with the OS layer implementing the rest of TCP. There are a lot of interactions, notably: - any mechanism outside of the TCP source and TCP destination that interprets the TCP header will result in a decrease in functionality i.e., the TCP connection will support only the intersection of options and features supported by the source, dest, *and* VXLAN layers (rather than being limited only by the source-dest pair) - if passed a full TCP segment, this mechanism will be incompatible with TCP security (e.g., TCP MD5, TCP-AO, and the results of the TCPCRYPT WG. I'm not quite sure from your doc whether you're re-segmenting TCP segments, or merely collecting them for aggregate transit (e.g., as is done in burst-mode Ethernet). Can you please clarify? Joe On 5/19/2014 8:01 PM, Zhou, Han wrote: > Hi, > > We have updated the VXLAN-SOE draft according to earlier comments. Now it is fully compatible with VXLAN-GPE. And some examples are added for better understanding. > > A prototype is also implemented here (patch based on Open vSwitch): > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be > > netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G interfaces: > > Before the change: 2.62 Gbits/sec > After the change: 6.68 Gbits/sec > Speedup is ~250%. > > The patch attracted some interests in OVS community, but since this RFC draft is in very early stage so it is regarded as inappropriate by Jesse to apply the change to OVS tree. > The discuss mail-thread: > http://openvswitch.org/pipermail/discuss/2014-May/013981.html > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > So we would like to request a review here by NVO3/TOFOO groups and VXLAN authors: is this VXLAN extension is worth formally put into VXLAN as a standard, so that more people can benefit from it? > > Best regards, > Han > > -----Original Message----- > From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of internet-drafts@ietf.org > Sent: Friday, May 02, 2014 8:09 PM > To: i-d-announce@ietf.org > Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > Title : Segmentation Offloading Extension for VXLAN > Authors : Han Zhou > Chengyuan Li > Filename : draft-zhou-li-vxlan-soe-01.txt > Pages : 13 > Date : 2014-05-02 > > Abstract: > Segmentation offloading is nowadays common in network stack > implementation and well supported by para-virtualized network device > drivers for virtual machine (VM)s. This draft describes an extension > to Virtual eXtensible Local Area Network (VXLAN) so that segmentation > can be decoupled from physical/underlay networks and offloaded > further to the remote end-point thus improving data-plane performance > for VMs running on top of overlay networks. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > > There's also a htmlized version available at: > http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > > A diff from the previous version is available at: > http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01 > > > Please note that it may take a couple of minutes from the time of submission > until the htmlized version and diff are available at tools.ietf.org. > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draft directories: http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > _______________________________________________ > Tofoo mailing list > Tofoo@ietf.org > https://www.ietf.org/mailman/listinfo/tofoo > From nobody Tue May 20 11:44:13 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4F6A71A0390 for ; Tue, 20 May 2014 11:44:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=unavailable Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IJXJ97t8Vtj9 for ; Tue, 20 May 2014 11:44:08 -0700 (PDT) Received: from mail-ie0-x22e.google.com (mail-ie0-x22e.google.com [IPv6:2607:f8b0:4001:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0FEF61A0731 for ; Tue, 20 May 2014 11:43:59 -0700 (PDT) Received: by mail-ie0-f174.google.com with SMTP id lx4so875419iec.5 for ; Tue, 20 May 2014 11:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BWsy3OOu/+0LKQiv0E93XhrxZ/mxphc0VFQ26skjdYs=; b=LDvw0n/rNFzvM7TeHz7JnZ2QKuIfaiMEjgjjXL8qPVJfwHw01ZV1amJae2H4eNepN/ RzjoQQ3FaSyoOF4rkTjI5ssvFZezBYdjRpcjv/OG3Pt8TRTireYRX514AIYD48GZwdOM ObaULaovxoTa2KK+EiVxQLCFDG9j8qw2hF8D6/x5lsC1i6YE2uachOs6hAYqYpvVzY8A 0TC2sZpKtCyJObIzTIYbtyg3tWfrr2vAXg5yvDVCyZ49IFw91g1XK3La0qWIAehOYqvT pEV0m6CH6VQJkoACv/oD8o1aSXb36AH3u786i+YN6xkcF19tdlI9FegOsQ0Y6Jkf7tD7 t4Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=BWsy3OOu/+0LKQiv0E93XhrxZ/mxphc0VFQ26skjdYs=; b=KocweLNVJcJP1oEevOoPH7dJbJE2wPos5WkBymZrj4nocueM+9atRbolxjpf1Juui/ mIAB9SDiuII0gNFuRn0mC9jz1z6BA4TrAzWuZaGqpZn6w81Nu27iaOtf2UZAPh/mUXVO CvGFmCJN2log7WvgWA6/A/eXOpIcDylIWNJ8Q9JTXxzfWtqWy70+qE/cBUCZ7bYTyHr4 qsrNMkoLQaEVQK7qBHEWZDNUl/LccZ2gD9Wq+a8uTBVVR6MkgpeHL9123szUdqUVK3vH NPKuvuO7JZRifu4SIM8tjYo3/miayi/xSRWchyoAtAuP/VXaVPPTkwLTBHfBnqCWzLCR F2sQ== X-Gm-Message-State: ALoCoQlDb3w/j9hWCz2CK80cpIJcApcl+FAevEG09dzCkWdI3htVqeNn88hRCCD9FKt0scPz7q18 MIME-Version: 1.0 X-Received: by 10.43.151.7 with SMTP id kq7mr12390286icc.78.1400611439068; Tue, 20 May 2014 11:43:59 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Tue, 20 May 2014 11:43:58 -0700 (PDT) In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> Date: Tue, 20 May 2014 11:43:58 -0700 Message-ID: From: Tom Herbert To: "Zhou, Han" Content-Type: multipart/alternative; boundary=001a11c2d02e4b61ac04f9d9433d Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/U1Ijpmrhy7tk25mX6gIR23g_Sn0 Cc: "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" , Erik Nordmark Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 May 2014 18:44:10 -0000 --001a11c2d02e4b61ac04f9d9433d Content-Type: text/plain; charset=UTF-8 Hi Zou, a couple of questions inline. On Mon, May 19, 2014 at 8:01 PM, Zhou, Han wrote: > Hi, > > We have updated the VXLAN-SOE draft according to earlier comments. Now it > is fully compatible with VXLAN-GPE. And some examples are added for better > understanding > A prototype is also implemented here (patch based on Open vSwitch): > > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be > > netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G > interfaces: > > Before the change: 2.62 Gbits/sec > After the change: 6.68 Gbits/sec > Speedup is ~250%. > > Can you provide some more details on this benefit? It seems like plain TSO/LRO that understands encapsulation should provide similar benefits when going between hosts. The patch attracted some interests in OVS community, but since this RFC > draft is in very early stage so it is regarded as inappropriate by Jesse to > apply the change to OVS tree. > The discuss mail-thread: > http://openvswitch.org/pipermail/discuss/2014-May/013981.html > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > So we would like to request a review here by NVO3/TOFOO groups and VXLAN > authors: is this VXLAN extension is worth formally put into VXLAN as a > standard, so that more people can benefit from it? > > Could you get the same effect by setting larger MTUs on the overlay network interface and relying in path MTU discovery when going over physical network? > Best regards, > Han > > -----Original Message----- > From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of > internet-drafts@ietf.org > Sent: Friday, May 02, 2014 8:09 PM > To: i-d-announce@ietf.org > Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > > Title : Segmentation Offloading Extension for VXLAN > Authors : Han Zhou > Chengyuan Li > Filename : draft-zhou-li-vxlan-soe-01.txt > Pages : 13 > Date : 2014-05-02 > > Abstract: > Segmentation offloading is nowadays common in network stack > implementation and well supported by para-virtualized network device > drivers for virtual machine (VM)s. This draft describes an extension > to Virtual eXtensible Local Area Network (VXLAN) so that segmentation > can be decoupled from physical/underlay networks and offloaded > further to the remote end-point thus improving data-plane performance > for VMs running on top of overlay networks. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > > There's also a htmlized version available at: > http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > > A diff from the previous version is available at: > http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01 > > > Please note that it may take a couple of minutes from the time of > submission > until the htmlized version and diff are available at tools.ietf.org. > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draftdirectories: > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > --001a11c2d02e4b61ac04f9d9433d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Zou, a couple of questions inline.


On Mon, May 19, 2014 at 8:01 = PM, Zhou, Han <hzhou8@ebay.com> wrote:
Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it i= s fully compatible with VXLAN-GPE. And some examples are added for better u= nderstanding=C2=A0
=C2=A0
A prototype is also implemented here (patch based on Open vSwitch):
https://github.com/hzhou8/openvswi= tch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be

netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G int= erfaces:

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.

Can you provide some more details on this benefit? It= seems like plain TSO/LRO that understands encapsulation should provide sim= ilar benefits when going between hosts.

The patch attracted some interests in OVS community, but since this RFC dra= ft is in very early stage so it is regarded as inappropriate by Jesse to ap= ply the change to OVS tree.
The discuss mail-thread:
http://openvswitch.org/pipermail/discuss/2014-May/013981.h= tml
http://openvswitch.org/pipermail/discuss/2014-May/013898.h= tml

So we would like to request a review here by NVO3/TOFOO groups and VXLAN au= thors: is this VXLAN extension is worth formally put into VXLAN as a standa= rd, so that more people can benefit from it?

Could you get the same effect by setting larger MTUs = on the overlay network interface and relying in path MTU discovery when goi= ng over physical network?
=C2=A0
Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of internet-drafts@ietf= .org
Sent: Friday, May 02, 2014 8:09 PM
To: i-d-announce= @ietf.org
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt


A New Internet-Draft is available from the on-line Internet-Drafts director= ies.


=C2=A0 =C2=A0 =C2=A0 =C2=A0 Title =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Segm= entation Offloading Extension for VXLAN
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Authors =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Han Zhou<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Chengyuan Li
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Filename =C2=A0 =C2=A0 =C2=A0 =C2=A0: draft-zho= u-li-vxlan-soe-01.txt
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Pages =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 13 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:= 2014-05-02

Abstract:
=C2=A0 =C2=A0Segmentation offloading is nowadays common in network stack =C2=A0 =C2=A0implementation and well supported by para-virtualized network = device
=C2=A0 =C2=A0drivers for virtual machine (VM)s. This draft describes an ext= ension
=C2=A0 =C2=A0to Virtual eXtensible Local Area Network (VXLAN) so that segme= ntation
=C2=A0 =C2=A0can be decoupled from physical/underlay networks and offloaded=
=C2=A0 =C2=A0further to the remote end-point thus improving data-plane perf= ormance
=C2=A0 =C2=A0for VMs running on top of overlay networks.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/<= br>
There's also a htmlized version available at:
http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe= -01


Please note that it may take a couple of minutes from the time of submissio= n
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp= .ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
I-D-Announce@iet= f.org
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or = ftp://ftp.ietf.org/ietf/1shadow-sites.txt

--001a11c2d02e4b61ac04f9d9433d-- From nobody Tue May 20 17:28:47 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D5F661A03CE; Tue, 20 May 2014 17:28:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.151 X-Spam-Level: X-Spam-Status: No, score=-23.151 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lMigTM9IVWu4; Tue, 20 May 2014 17:28:41 -0700 (PDT) Received: from den-mipot-002.corp.ebay.com (den-mipot-002.corp.ebay.com [216.113.175.153]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id ECFE21A03CC; Tue, 20 May 2014 17:28:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400632120; x=1432168120; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=JsCqEx51mozSJ3X9ZFV/SXg1sH2yggW4dmxFU+wnqDE=; b=JOpH6s2QQaCuKo32NqBwGk2jBv3/DzdVVGEJmPYaMpnoZXy6n0nQRwyn HBGukP0q2mDg03MdWtyQR6TfOZknHAheHzPm9thoOmEieB4rYhpvpqACN pfjqAuT7ljRKOiUkgVy6h53DsnimndUVMwiKKJmYj8mq9cu9dxjxm8fht k=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos; i="4.98,877,1392192000"; d="scan'208,217"; a="51219374" Received: from den-vteml-003.corp.ebay.com (HELO DEN-EXMHT-004.corp.ebay.com) ([10.101.112.119]) by den-mipot-002.corp.ebay.com with ESMTP; 20 May 2014 17:28:40 -0700 Received: from DEN-EXDDA-S81.corp.ebay.com (10.241.49.126) by DEN-EXMHT-004.corp.ebay.com (10.241.17.151) with Microsoft SMTP Server (TLS) id 14.3.174.1; Tue, 20 May 2014 18:28:40 -0600 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXDDA-S81.corp.ebay.com ([10.241.49.126]) with mapi id 14.03.0174.001; Tue, 20 May 2014 18:28:39 -0600 From: "Zhou, Han" To: Tom Herbert Thread-Topic: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOggAFw/gD///LaIA== Date: Wed, 21 May 2014 00:28:39 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109C36BC@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: multipart/alternative; boundary="_000_9F56174078B48B459268EFF1DAB66B1A109C36BCDENEXDDAS32corp_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/Vatp3Wsu8sxjhq9SqOXbzeEnSWU Cc: "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" , Erik Nordmark Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 00:28:44 -0000 --_000_9F56174078B48B459268EFF1DAB66B1A109C36BCDENEXDDAS32corp_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgVG9tLA0KDQpUaGFua3MgZm9yIHlvdXIgY29tbWVudHMuDQpZZXMgVFNPL0xSTyB3aXRoIFZY TEFOIHN1cHBvcnQgc2hvdWxkIHByb3ZpZGUgc2ltaWxhciBvciBldmVuIGJldHRlciBwZXJmb3Jt YW5jZSBnYWlucywgYnV0IHRoZSBtZWNoYW5pc20gcHJvcG9zZWQgYnkgdGhpcyBkcmFmdCBkZWNv dXBsZXMgb3ZlcmxheSBhbmQgdW5kZXJsYXksIGFuZCBpdCBpcyBoYXJkd2FyZSBpbmRlcGVuZGVu dC4NClNlY29uZGx5LCBoYXJkd2FyZSBvZmZsb2FkaW5nIHVzdWFsbHkgc3VwcG9ydCBUQ1Agb25s eSAoVFNPKS4gVGhlIG1lY2hhbmlzbSBoZXJlIGNhbiBoZWxwIG9uIGxhcmdlIFVEUCBwYWNrZXQg cGVyZm9ybWFuY2UsIGFsc28gdmVyaWZpZWQgYnkgdGhlIHByb3RvdHlwZS4NCkJUVywgaG93IG1h bnkgdHlwZXMgb2Ygb2ZmLXRoZS1zaGVsZiBOSUMgc3VwcG9ydCBWWExBTiBvZmZsb2FkaW5nPyBB bnkgcGVyZm9ybWFuY2UgZGF0YT8NCg0KTGlrZXdpc2UsIHNldHRpbmcgbGFyZ2UgTVRVIG9uIG92 ZXJsYXkgaW50ZXJmYWNlcyBhY2hpZXZlcyBzaW1pbGFyIHJlc3VsdCwgYnV0IHN0aWxsIHRoZSBv dmVybGF5L3VuZGVybGF5IGRlY291cGxpbmcgaXNzdWUuIEl0IGlzIHVzdWFsbHkgYWR2aXNlZCB0 aGF0IG92ZXJsYXkgTVRVIGlzIHNsaWdodGx5IHNtYWxsZXIgdGhhbiB1bmRlcmxheSB0byBhdm9p ZCBpbmVmZmljaWVudCBmcmFnbWVudGF0aW9uIGFmdGVyIGFkZGluZyB0aGUgb3V0ZXIgaGVhZGVy LCBidXQgdG8gYWNoaWV2ZSByZWFsbHkgaGlnaCBwZXJmb3JtYW5jZSBiZXR3ZWVuIFZNcywgbGFy Z2UgTVRVIGlzIHByZWZlcnJlZC4gQW5kIGNvbnNpZGVyaW5nIG92ZXJsYXkgPC0+IHBoeXNpY2Fs IGNvbm5lY3Rpb24sIHBhdGggTVRVIGRpc2NvdmVyeSBpcyBub3QgYWx3YXlzIHdvcmsuDQpUaGlz IGtpbmQgb2YgY29uZmlndXJhdGlvbiBjb21wbGV4aXR5IGFuZCBwYWluLXBvaW50IGNhbiBiZSBy ZXNvbHZlZCBzaW1wbHkgYnkgZGVjb3VwbGluZyBvdmVybGF5IGFuZCB1bmRlcmxheSBNVFUsIGFz IHN1Z2dlc3RlZCBieSB0aGlzIGRyYWZ0LiBIZXJlIGlzIGFuIGV4YW1wbGUgb2YgY29uZmlndXJh dGlvbiBjb25mdXNpb246DQpodHRwOi8vb3BlbnZzd2l0Y2gub3JnL3BpcGVybWFpbC9kaXNjdXNz LzIwMTQtTWF5LzAxMzg5OC5odG1sDQoNCklkZWFsbHksIGFsbCBOViB0dW5uZWwgcHJvdG9jb2xz IHNob3VsZCBzdXBwb3J0IHNpbWlsYXIgbWV0YWRhdGEsIHRodXMgb3ZlcmxheSBzZWdtZW50YXRp b24gY2FuIGJlIG9mZmxvYWRlZCBob3AtYnktaG9wLg0KDQpCZXN0IHJlZ2FyZHMsDQpIYW4NCg0K RnJvbTogVG9tIEhlcmJlcnQgW21haWx0bzp0aGVyYmVydEBnb29nbGUuY29tXQ0KU2VudDogV2Vk bmVzZGF5LCBNYXkgMjEsIDIwMTQgMjo0NCBBTQ0KVG86IFpob3UsIEhhbg0KQ2M6IG52bzNAaWV0 Zi5vcmc7IHRvZm9vQGlldGYub3JnOyBkcmFmdC1tYWhhbGluZ2FtLWR1dHQtZGNvcHMtdnhsYW5A dG9vbHMuaWV0Zi5vcmc7IGRyYWZ0LXpob3UtbGktdnhsYW4tc29lQHRvb2xzLmlldGYub3JnOyBF cmlrIE5vcmRtYXJrDQpTdWJqZWN0OiBSZTogRlc6IEktRCBBY3Rpb246IGRyYWZ0LXpob3UtbGkt dnhsYW4tc29lLTAxLnR4dA0KDQpIaSBab3UsIGEgY291cGxlIG9mIHF1ZXN0aW9ucyBpbmxpbmUu DQoNCk9uIE1vbiwgTWF5IDE5LCAyMDE0IGF0IDg6MDEgUE0sIFpob3UsIEhhbiA8aHpob3U4QGVi YXkuY29tPG1haWx0bzpoemhvdThAZWJheS5jb20+PiB3cm90ZToNCkhpLA0KDQpXZSBoYXZlIHVw ZGF0ZWQgdGhlIFZYTEFOLVNPRSBkcmFmdCBhY2NvcmRpbmcgdG8gZWFybGllciBjb21tZW50cy4g Tm93IGl0IGlzIGZ1bGx5IGNvbXBhdGlibGUgd2l0aCBWWExBTi1HUEUuIEFuZCBzb21lIGV4YW1w bGVzIGFyZSBhZGRlZCBmb3IgYmV0dGVyIHVuZGVyc3RhbmRpbmcNCg0KQSBwcm90b3R5cGUgaXMg YWxzbyBpbXBsZW1lbnRlZCBoZXJlIChwYXRjaCBiYXNlZCBvbiBPcGVuIHZTd2l0Y2gpOg0KaHR0 cHM6Ly9naXRodWIuY29tL2h6aG91OC9vcGVudnN3aXRjaC9jb21taXQvOWE3ZGViOGI0MzJjZTgz YTljMDlkN2Q0ZmY4NWZhMDUwZjdkZDJiZQ0KDQpuZXRwZXJmIFRDUF9TVFJFQU0gdGVzdCByZXN1 bHQgYmV0d2VlbiBhIHBhaXJzIG9mIFZNcyBvbiBob3N0cyB3aXRoIDEwRyBpbnRlcmZhY2VzOg0K DQpCZWZvcmUgdGhlIGNoYW5nZTogMi42MiBHYml0cy9zZWMNCkFmdGVyIHRoZSBjaGFuZ2U6IDYu NjggR2JpdHMvc2VjDQpTcGVlZHVwIGlzIH4yNTAlLg0KQ2FuIHlvdSBwcm92aWRlIHNvbWUgbW9y ZSBkZXRhaWxzIG9uIHRoaXMgYmVuZWZpdD8gSXQgc2VlbXMgbGlrZSBwbGFpbiBUU08vTFJPIHRo YXQgdW5kZXJzdGFuZHMgZW5jYXBzdWxhdGlvbiBzaG91bGQgcHJvdmlkZSBzaW1pbGFyIGJlbmVm aXRzIHdoZW4gZ29pbmcgYmV0d2VlbiBob3N0cy4NCg0KDQpUaGUgcGF0Y2ggYXR0cmFjdGVkIHNv bWUgaW50ZXJlc3RzIGluIE9WUyBjb21tdW5pdHksIGJ1dCBzaW5jZSB0aGlzIFJGQyBkcmFmdCBp cyBpbiB2ZXJ5IGVhcmx5IHN0YWdlIHNvIGl0IGlzIHJlZ2FyZGVkIGFzIGluYXBwcm9wcmlhdGUg YnkgSmVzc2UgdG8gYXBwbHkgdGhlIGNoYW5nZSB0byBPVlMgdHJlZS4NClRoZSBkaXNjdXNzIG1h aWwtdGhyZWFkOg0KaHR0cDovL29wZW52c3dpdGNoLm9yZy9waXBlcm1haWwvZGlzY3Vzcy8yMDE0 LU1heS8wMTM5ODEuaHRtbA0KaHR0cDovL29wZW52c3dpdGNoLm9yZy9waXBlcm1haWwvZGlzY3Vz cy8yMDE0LU1heS8wMTM4OTguaHRtbA0KDQpTbyB3ZSB3b3VsZCBsaWtlIHRvIHJlcXVlc3QgYSBy ZXZpZXcgaGVyZSBieSBOVk8zL1RPRk9PIGdyb3VwcyBhbmQgVlhMQU4gYXV0aG9yczogaXMgdGhp cyBWWExBTiBleHRlbnNpb24gaXMgd29ydGggZm9ybWFsbHkgcHV0IGludG8gVlhMQU4gYXMgYSBz dGFuZGFyZCwgc28gdGhhdCBtb3JlIHBlb3BsZSBjYW4gYmVuZWZpdCBmcm9tIGl0Pw0KQ291bGQg eW91IGdldCB0aGUgc2FtZSBlZmZlY3QgYnkgc2V0dGluZyBsYXJnZXIgTVRVcyBvbiB0aGUgb3Zl cmxheSBuZXR3b3JrIGludGVyZmFjZSBhbmQgcmVseWluZyBpbiBwYXRoIE1UVSBkaXNjb3Zlcnkg d2hlbiBnb2luZyBvdmVyIHBoeXNpY2FsIG5ldHdvcms/DQoNCkJlc3QgcmVnYXJkcywNCkhhbg0K DQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogSS1ELUFubm91bmNlIFttYWlsdG86 aS1kLWFubm91bmNlLWJvdW5jZXNAaWV0Zi5vcmc8bWFpbHRvOmktZC1hbm5vdW5jZS1ib3VuY2Vz QGlldGYub3JnPl0gT24gQmVoYWxmIE9mIGludGVybmV0LWRyYWZ0c0BpZXRmLm9yZzxtYWlsdG86 aW50ZXJuZXQtZHJhZnRzQGlldGYub3JnPg0KU2VudDogRnJpZGF5LCBNYXkgMDIsIDIwMTQgODow OSBQTQ0KVG86IGktZC1hbm5vdW5jZUBpZXRmLm9yZzxtYWlsdG86aS1kLWFubm91bmNlQGlldGYu b3JnPg0KU3ViamVjdDogSS1EIEFjdGlvbjogZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDEudHh0 DQoNCg0KQSBOZXcgSW50ZXJuZXQtRHJhZnQgaXMgYXZhaWxhYmxlIGZyb20gdGhlIG9uLWxpbmUg SW50ZXJuZXQtRHJhZnRzIGRpcmVjdG9yaWVzLg0KDQoNCiAgICAgICAgVGl0bGUgICAgICAgICAg IDogU2VnbWVudGF0aW9uIE9mZmxvYWRpbmcgRXh0ZW5zaW9uIGZvciBWWExBTg0KICAgICAgICBB dXRob3JzICAgICAgICAgOiBIYW4gWmhvdQ0KICAgICAgICAgICAgICAgICAgICAgICAgICBDaGVu Z3l1YW4gTGkNCiAgICAgICAgRmlsZW5hbWUgICAgICAgIDogZHJhZnQtemhvdS1saS12eGxhbi1z b2UtMDEudHh0DQogICAgICAgIFBhZ2VzICAgICAgICAgICA6IDEzDQogICAgICAgIERhdGUgICAg ICAgICAgICA6IDIwMTQtMDUtMDINCg0KQWJzdHJhY3Q6DQogICBTZWdtZW50YXRpb24gb2ZmbG9h ZGluZyBpcyBub3dhZGF5cyBjb21tb24gaW4gbmV0d29yayBzdGFjaw0KICAgaW1wbGVtZW50YXRp b24gYW5kIHdlbGwgc3VwcG9ydGVkIGJ5IHBhcmEtdmlydHVhbGl6ZWQgbmV0d29yayBkZXZpY2UN CiAgIGRyaXZlcnMgZm9yIHZpcnR1YWwgbWFjaGluZSAoVk0pcy4gVGhpcyBkcmFmdCBkZXNjcmli ZXMgYW4gZXh0ZW5zaW9uDQogICB0byBWaXJ0dWFsIGVYdGVuc2libGUgTG9jYWwgQXJlYSBOZXR3 b3JrIChWWExBTikgc28gdGhhdCBzZWdtZW50YXRpb24NCiAgIGNhbiBiZSBkZWNvdXBsZWQgZnJv bSBwaHlzaWNhbC91bmRlcmxheSBuZXR3b3JrcyBhbmQgb2ZmbG9hZGVkDQogICBmdXJ0aGVyIHRv IHRoZSByZW1vdGUgZW5kLXBvaW50IHRodXMgaW1wcm92aW5nIGRhdGEtcGxhbmUgcGVyZm9ybWFu Y2UNCiAgIGZvciBWTXMgcnVubmluZyBvbiB0b3Agb2Ygb3ZlcmxheSBuZXR3b3Jrcy4NCg0KDQpU aGUgSUVURiBkYXRhdHJhY2tlciBzdGF0dXMgcGFnZSBmb3IgdGhpcyBkcmFmdCBpczoNCmh0dHBz Oi8vZGF0YXRyYWNrZXIuaWV0Zi5vcmcvZG9jL2RyYWZ0LXpob3UtbGktdnhsYW4tc29lLw0KDQpU aGVyZSdzIGFsc28gYSBodG1saXplZCB2ZXJzaW9uIGF2YWlsYWJsZSBhdDoNCmh0dHA6Ly90b29s cy5pZXRmLm9yZy9odG1sL2RyYWZ0LXpob3UtbGktdnhsYW4tc29lLTAxDQoNCkEgZGlmZiBmcm9t IHRoZSBwcmV2aW91cyB2ZXJzaW9uIGlzIGF2YWlsYWJsZSBhdDoNCmh0dHA6Ly93d3cuaWV0Zi5v cmcvcmZjZGlmZj91cmwyPWRyYWZ0LXpob3UtbGktdnhsYW4tc29lLTAxDQoNCg0KUGxlYXNlIG5v dGUgdGhhdCBpdCBtYXkgdGFrZSBhIGNvdXBsZSBvZiBtaW51dGVzIGZyb20gdGhlIHRpbWUgb2Yg c3VibWlzc2lvbg0KdW50aWwgdGhlIGh0bWxpemVkIHZlcnNpb24gYW5kIGRpZmYgYXJlIGF2YWls YWJsZSBhdCB0b29scy5pZXRmLm9yZzxodHRwOi8vdG9vbHMuaWV0Zi5vcmc+Lg0KDQpJbnRlcm5l dC1EcmFmdHMgYXJlIGFsc28gYXZhaWxhYmxlIGJ5IGFub255bW91cyBGVFAgYXQ6DQpmdHA6Ly9m dHAuaWV0Zi5vcmcvaW50ZXJuZXQtZHJhZnRzLw0KDQpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXw0KSS1ELUFubm91bmNlIG1haWxpbmcgbGlzdA0KSS1ELUFu bm91bmNlQGlldGYub3JnPG1haWx0bzpJLUQtQW5ub3VuY2VAaWV0Zi5vcmc+DQpodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ktZC1hbm5vdW5jZQ0KSW50ZXJuZXQtRHJhZnQg ZGlyZWN0b3JpZXM6IGh0dHA6Ly93d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWwNCm9yIGZ0cDovL2Z0 cC5pZXRmLm9yZy9pZXRmLzFzaGFkb3ctc2l0ZXMudHh0DQoNCg== --_000_9F56174078B48B459268EFF1DAB66B1A109C36BCDENEXDDAS32corp_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IlByb2dJZCIg Y29udGVudD0iV29yZC5Eb2N1bWVudCI+DQo8bWV0YSBuYW1lPSJHZW5lcmF0b3IiIGNvbnRlbnQ9 Ik1pY3Jvc29mdCBXb3JkIDE0Ij4NCjxtZXRhIG5hbWU9Ik9yaWdpbmF0b3IiIGNvbnRlbnQ9Ik1p Y3Jvc29mdCBXb3JkIDE0Ij4NCjxsaW5rIHJlbD0iRmlsZS1MaXN0IiBocmVmPSJjaWQ6ZmlsZWxp c3QueG1sQDAxQ0Y3NENFLkE3MzUzMjAwIj48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOk9m ZmljZURvY3VtZW50U2V0dGluZ3M+DQo8bzpBbGxvd1BORy8+DQo8L286T2ZmaWNlRG9jdW1lbnRT ZXR0aW5ncz4NCjwveG1sPjwhW2VuZGlmXS0tPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KPHc6 V29yZERvY3VtZW50Pg0KPHc6Wm9vbT4xMTA8L3c6Wm9vbT4NCjx3OlNwZWxsaW5nU3RhdGU+Q2xl YW48L3c6U3BlbGxpbmdTdGF0ZT4NCjx3OlRyYWNrTW92ZXMvPg0KPHc6VHJhY2tGb3JtYXR0aW5n Lz4NCjx3OkVudmVsb3BlVmlzLz4NCjx3OlZhbGlkYXRlQWdhaW5zdFNjaGVtYXMvPg0KPHc6U2F2 ZUlmWE1MSW52YWxpZD5mYWxzZTwvdzpTYXZlSWZYTUxJbnZhbGlkPg0KPHc6SWdub3JlTWl4ZWRD b250ZW50PmZhbHNlPC93Oklnbm9yZU1peGVkQ29udGVudD4NCjx3OkFsd2F5c1Nob3dQbGFjZWhv bGRlclRleHQ+ZmFsc2U8L3c6QWx3YXlzU2hvd1BsYWNlaG9sZGVyVGV4dD4NCjx3OkRvTm90UHJv bW90ZVFGLz4NCjx3OkxpZFRoZW1lT3RoZXI+RU4tVVM8L3c6TGlkVGhlbWVPdGhlcj4NCjx3Okxp ZFRoZW1lQXNpYW4+WkgtQ048L3c6TGlkVGhlbWVBc2lhbj4NCjx3OkxpZFRoZW1lQ29tcGxleFNj cmlwdD5YLU5PTkU8L3c6TGlkVGhlbWVDb21wbGV4U2NyaXB0Pg0KPHc6Q29tcGF0aWJpbGl0eT4N Cjx3OkRvTm90RXhwYW5kU2hpZnRSZXR1cm4vPg0KPHc6QnJlYWtXcmFwcGVkVGFibGVzLz4NCjx3 OlNwbGl0UGdCcmVha0FuZFBhcmFNYXJrLz4NCjx3OkVuYWJsZU9wZW5UeXBlS2VybmluZy8+DQo8 dzpVc2VGRUxheW91dC8+DQo8L3c6Q29tcGF0aWJpbGl0eT4NCjxtOm1hdGhQcj4NCjxtOm1hdGhG b250IG06dmFsPSJDYW1icmlhIE1hdGgiLz4NCjxtOmJya0JpbiBtOnZhbD0iYmVmb3JlIi8+DQo8 bTpicmtCaW5TdWIgbTp2YWw9IiYjNDU7LSIvPg0KPG06c21hbGxGcmFjIG06dmFsPSJvZmYiLz4N CjxtOmRpc3BEZWYvPg0KPG06bE1hcmdpbiBtOnZhbD0iMCIvPg0KPG06ck1hcmdpbiBtOnZhbD0i MCIvPg0KPG06ZGVmSmMgbTp2YWw9ImNlbnRlckdyb3VwIi8+DQo8bTp3cmFwSW5kZW50IG06dmFs PSIxNDQwIi8+DQo8bTppbnRMaW0gbTp2YWw9InN1YlN1cCIvPg0KPG06bmFyeUxpbSBtOnZhbD0i dW5kT3ZyIi8+DQo8L206bWF0aFByPjwvdzpXb3JkRG9jdW1lbnQ+DQo8L3htbD48IVtlbmRpZl0t LT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjx3OkxhdGVudFN0eWxlcyBEZWZMb2NrZWRTdGF0 ZT0iZmFsc2UiIERlZlVuaGlkZVdoZW5Vc2VkPSJ0cnVlIiBEZWZTZW1pSGlkZGVuPSJ0cnVlIiBE ZWZRRm9ybWF0PSJmYWxzZSIgRGVmUHJpb3JpdHk9Ijk5IiBMYXRlbnRTdHlsZUNvdW50PSIyNjci Pg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIwIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJOb3Jt YWwiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0idHJ1ZSIgTmFtZT0i aGVhZGluZyAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9Ijki IFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgMiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tl ZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJoZWFkaW5nIDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgUUZvcm1hdD0idHJ1 ZSIgTmFtZT0iaGVhZGluZyA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjkiIFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgNSIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJoZWFk aW5nIDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgUUZv cm1hdD0idHJ1ZSIgTmFtZT0iaGVhZGluZyA3Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjkiIFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgOCIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBO YW1lPSJoZWFkaW5nIDkiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iMzkiIE5hbWU9InRvYyAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjM5IiBOYW1lPSJ0b2MgMiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2Ui IFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iMzkiIE5hbWU9InRvYyA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2Vk PSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBOYW1lPSJ0b2MgNSIvPg0KPHc6THNkRXhjZXB0aW9uIExv Y2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDYiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzkiIE5hbWU9InRvYyA3Ii8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBOYW1lPSJ0b2MgOCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDkiLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzUiIFFGb3JtYXQ9InRydWUi IE5hbWU9ImNhcHRpb24iLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iMTAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9 InRydWUiIE5hbWU9IlRpdGxlIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjEiIE5hbWU9IkRlZmF1bHQgUGFyYWdyYXBoIEZvbnQiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMTEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVX aGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9IlN1YnRpdGxlIi8+DQo8dzpMc2RF eGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjIyIiBTZW1pSGlkZGVuPSJmYWxzZSIg VW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJTdHJvbmciLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMjAiIFNlbWlIaWRkZW49ImZh bHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9IkVtcGhhc2lz Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjU5IiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJUYWJsZSBHcmlkIi8+DQo8 dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJQbGFjZWhvbGRlciBUZXh0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3Jt YXQ9InRydWUiIE5hbWU9Ik5vIFNwYWNpbmciLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFs c2UiIE5hbWU9IkxpZ2h0IFNoYWRpbmciLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkxpZ2h0IExpc3QiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNjIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 IkxpZ2h0IEdyaWQiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBTaGFkaW5nIDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjQiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBTaGFkaW5nIDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjUiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBMaXN0IDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjYi IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBM aXN0IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjciIFNl bWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlk IDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjgiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDIi Lz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjkiIFNlbWlIaWRk ZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkRhcmsgTGlzdCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCIvPg0KPHc6THNkRXhj ZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgR3JpZCIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlk ZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgU2hhZGluZyBBY2NlbnQgMSIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgTGlzdCBBY2NlbnQgMSIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MiIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgR3JpZCBBY2NlbnQgMSIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MyIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRpbmcgMSBBY2Nl bnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NCIgU2Vt aUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRp bmcgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIExpc3QgMSBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iUmV2aXNpb24iLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzQiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9Ikxpc3QgUGFyYWdyYXBoIi8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjI5IiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJRdW90ZSIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzMCIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0idHJ1ZSIgTmFtZT0iSW50ZW5z ZSBRdW90ZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NiIg U2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIExp c3QgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIEdyaWQgMSBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFBy aW9yaXR5PSI2OCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFt ZT0iTWVkaXVtIEdyaWQgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFs c2UiIFByaW9yaXR5PSI2OSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxz ZSIgTmFtZT0iTWVkaXVtIEdyaWQgMyBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tl ZD0iZmFsc2UiIFByaW9yaXR5PSI3MCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2Vk PSJmYWxzZSIgTmFtZT0iRGFyayBMaXN0IEFjY2VudCAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9j a2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVz ZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBTaGFkaW5nIEFjY2VudCAxIi8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcyIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5o aWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBMaXN0IEFjY2VudCAxIi8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjczIiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBHcmlkIEFjY2VudCAxIi8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYwIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBTaGFkaW5nIEFjY2Vu dCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYxIiBTZW1p SGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBMaXN0IEFj Y2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYyIiBT ZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBHcmlk IEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYz IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0g U2hhZGluZyAxIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjY0IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJNZWRpdW0gU2hhZGluZyAyIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjY1IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAxIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9j a2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY2IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVz ZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAyIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRp b24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY3IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRl V2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAxIEFjY2VudCAyIi8+DQo8dzpMc2RF eGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY4IiBTZW1pSGlkZGVuPSJmYWxzZSIg VW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAyIEFjY2VudCAyIi8+DQo8 dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY5IiBTZW1pSGlkZGVuPSJm YWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAzIEFjY2VudCAy Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcwIiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJEYXJrIExpc3QgQWNjZW50 IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzEiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIFNoYWRp bmcgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NzIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9y ZnVsIExpc3QgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNzMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 IkNvbG9yZnVsIEdyaWQgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkxpZ2h0IFNoYWRpbmcgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9 ImZhbHNlIiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0i ZmFsc2UiIE5hbWU9IkxpZ2h0IExpc3QgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2Nr ZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNl ZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IEdyaWQgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDEgQWNjZW50IDMiLz4NCjx3OkxzZEV4 Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjQiIFNlbWlIaWRkZW49ImZhbHNlIiBV bmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDIgQWNjZW50IDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjUiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBMaXN0IDEgQWNjZW50 IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjYiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBMaXN0IDIg QWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjci IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBH cmlkIDEgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iNjgiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1l ZGl1bSBHcmlkIDIgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQ cmlvcml0eT0iNjkiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5h bWU9Ik1lZGl1bSBHcmlkIDMgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFs c2UiIE5hbWU9IkRhcmsgTGlzdCBBY2NlbnQgMyIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0i ZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJm YWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyBBY2NlbnQgMyIvPg0KPHc6THNkRXhjZXB0aW9u IExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdo ZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCBBY2NlbnQgMyIvPg0KPHc6THNkRXhj ZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgR3JpZCBBY2NlbnQgMyIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MCIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgU2hhZGluZyBBY2NlbnQgNCIv Pg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MSIgU2VtaUhpZGRl bj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgTGlzdCBBY2NlbnQg NCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MiIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgR3JpZCBBY2Nl bnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MyIgU2Vt aUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRp bmcgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIFNoYWRpbmcgMiBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2Ui IFByaW9yaXR5PSI2NSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIg TmFtZT0iTWVkaXVtIExpc3QgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0i ZmFsc2UiIFByaW9yaXR5PSI2NiIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJm YWxzZSIgTmFtZT0iTWVkaXVtIExpc3QgMiBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExv Y2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5V c2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2OCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlk ZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMiBBY2NlbnQgNCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2OSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMyBBY2NlbnQgNCIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MCIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iRGFyayBMaXN0IEFjY2VudCA0Ii8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcxIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBTaGFkaW5nIEFj Y2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcyIiBT ZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBM aXN0IEFjY2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9 IjczIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xv cmZ1bCBHcmlkIEFjY2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjYwIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJMaWdodCBTaGFkaW5nIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxz ZSIgUHJpb3JpdHk9IjYxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNl IiBOYW1lPSJMaWdodCBMaXN0IEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjYyIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBOYW1lPSJMaWdodCBHcmlkIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2Vk PSJmYWxzZSIgUHJpb3JpdHk9IjYzIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9 ImZhbHNlIiBOYW1lPSJNZWRpdW0gU2hhZGluZyAxIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRp b24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY0IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRl V2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gU2hhZGluZyAyIEFjY2VudCA1Ii8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY1IiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAxIEFjY2VudCA1Ii8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY2IiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAyIEFjY2Vu dCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY3IiBTZW1p SGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAx IEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY4 IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0g R3JpZCAyIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3Jp dHk9IjY5IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJN ZWRpdW0gR3JpZCAzIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIg UHJpb3JpdHk9IjcwIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBO YW1lPSJEYXJrIExpc3QgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNzEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkNvbG9yZnVsIFNoYWRpbmcgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2Nr ZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNl ZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIExpc3QgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVX aGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIEdyaWQgQWNjZW50IDUiLz4NCjx3OkxzZEV4 Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBV bmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IFNoYWRpbmcgQWNjZW50IDYiLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZh bHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IExpc3QgQWNjZW50IDYiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjIiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IEdyaWQgQWNjZW50IDYi Lz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjMiIFNlbWlIaWRk ZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDEg QWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjQi IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBT aGFkaW5nIDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNjUiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 Ik1lZGl1bSBMaXN0IDEgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjYiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9Ik1lZGl1bSBMaXN0IDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9 ImZhbHNlIiBQcmlvcml0eT0iNjciIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0i ZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDEgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjgiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2Vw dGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjkiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhp ZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDMgQWNjZW50IDYiLz4NCjx3Okxz ZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49ImZhbHNl IiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkRhcmsgTGlzdCBBY2NlbnQgNiIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyBBY2NlbnQg NiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCBB Y2NlbnQgNiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIg U2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwg R3JpZCBBY2NlbnQgNiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSIxOSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0i dHJ1ZSIgTmFtZT0iU3VidGxlIEVtcGhhc2lzIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjIxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJJbnRlbnNlIEVtcGhhc2lzIi8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjMxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5o aWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJTdWJ0bGUgUmVmZXJlbmNl Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjMyIiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJJ bnRlbnNlIFJlZmVyZW5jZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9y aXR5PSIzMyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1h dD0idHJ1ZSIgTmFtZT0iQm9vayBUaXRsZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFs c2UiIFByaW9yaXR5PSIzNyIgTmFtZT0iQmlibGlvZ3JhcGh5Ii8+DQo8dzpMc2RFeGNlcHRpb24g TG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJUT0MgSGVh ZGluZyIvPg0KPC93OkxhdGVudFN0eWxlcz4NCjwveG1sPjwhW2VuZGlmXS0tPjxzdHlsZT48IS0t DQovKiBGb250IERlZmluaXRpb25zICovDQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OuWui+S9 kzsNCglwYW5vc2UtMToyIDEgNiAwIDMgMSAxIDEgMSAxOw0KCW1zby1mb250LWFsdDpTaW1TdW47 DQoJbXNvLWZvbnQtY2hhcnNldDoxMzQ7DQoJbXNvLWdlbmVyaWMtZm9udC1mYW1pbHk6YXV0bzsN Cgltc28tZm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6MyA2ODA0NjAy ODggMjIgMCAyNjIxNDUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OuWui+S9kzsNCglw YW5vc2UtMToyIDEgNiAwIDMgMSAxIDEgMSAxOw0KCW1zby1mb250LWFsdDpTaW1TdW47DQoJbXNv LWZvbnQtY2hhcnNldDoxMzQ7DQoJbXNvLWdlbmVyaWMtZm9udC1mYW1pbHk6YXV0bzsNCgltc28t Zm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6MyA2ODA0NjAyODggMjIg MCAyNjIxNDUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3Nl LTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7DQoJbXNvLWZvbnQtYWx0OiJUaW1lcyBOZXcgUm9tYW4i Ow0KCW1zby1mb250LWNoYXJzZXQ6MDsNCgltc28tZ2VuZXJpYy1mb250LWZhbWlseTpzd2lzczsN Cgltc28tZm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6LTUzNjg3MDE0 NSAxMDczNzg2MTExIDEgMCA0MTUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OlRhaG9t YTsNCglwYW5vc2UtMToyIDExIDYgNCAzIDUgNCA0IDIgNDsNCgltc28tZm9udC1hbHQ6IlRpbWVz IE5ldyBSb21hbiI7DQoJbXNvLWZvbnQtY2hhcnNldDowOw0KCW1zby1nZW5lcmljLWZvbnQtZmFt aWx5OnN3aXNzOw0KCW1zby1mb250LXBpdGNoOnZhcmlhYmxlOw0KCW1zby1mb250LXNpZ25hdHVy ZTotNTIwMDgxNjY1IC0xMDczNzE3MTU3IDQxIDAgNjYwNDcgMDt9DQpAZm9udC1mYWNlDQoJe2Zv bnQtZmFtaWx5OiJcQOWui+S9kyI7DQoJcGFub3NlLTE6MiAxIDYgMCAzIDEgMSAxIDEgMTsNCglt c28tZm9udC1hbHQ6Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/PzsNCgltc28tZm9udC1j aGFyc2V0OjEzNDsNCgltc28tZ2VuZXJpYy1mb250LWZhbWlseTphdXRvOw0KCW1zby1mb250LXBp dGNoOnZhcmlhYmxlOw0KCW1zby1mb250LXNpZ25hdHVyZTozIDY4MDQ2MDI4OCAyMiAwIDI2MjE0 NSAwO30NCi8qIFN0eWxlIERlZmluaXRpb25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFs LCBkaXYuTXNvTm9ybWFsDQoJe21zby1zdHlsZS11bmhpZGU6bm87DQoJbXNvLXN0eWxlLXFmb3Jt YXQ6eWVzOw0KCW1zby1zdHlsZS1wYXJlbnQ6IiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90 dG9tOi4wMDAxcHQ7DQoJbXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZTox Mi4wcHQ7DQoJZm9udC1mYW1pbHk65a6L5L2TOw0KCW1zby1iaWRpLWZvbnQtZmFtaWx5OuWui+S9 kzt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsN Cgljb2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7DQoJdGV4dC11bmRlcmxp bmU6c2luZ2xlO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0KCXttc28t c3R5bGUtbm9zaG93OnllczsNCgltc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxl Ow0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7DQoJdGV4dC11bmRlcmxpbmU6c2luZ2xlO30N CnAuTXNvQWNldGF0ZSwgbGkuTXNvQWNldGF0ZSwgZGl2Lk1zb0FjZXRhdGUNCgl7bXNvLXN0eWxl LW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJC YWxsb29uIFRleHQgQ2hhciI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7 DQoJbXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZTo4LjBwdDsNCglmb250 LWZhbWlseTrlrovkvZM7DQoJbXNvLWJpZGktZm9udC1mYW1pbHk65a6L5L2TO30NCnNwYW4uRW1h aWxTdHlsZTE3DQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCW1zby1zdHlsZS1u b3Nob3c6eWVzOw0KCW1zby1zdHlsZS11bmhpZGU6bm87DQoJbXNvLWFuc2ktZm9udC1zaXplOjEw LjVwdDsNCgltc28tYmlkaS1mb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp Iiwic2Fucy1zZXJpZiI7DQoJbXNvLWFzY2lpLWZvbnQtZmFtaWx5OkNhbGlicmk7DQoJbXNvLWZh cmVhc3QtZm9udC1mYW1pbHk65a6L5L2TOw0KCW1zby1oYW5zaS1mb250LWZhbWlseTpDYWxpYnJp Ow0KCW1zby1iaWRpLWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iOw0KCWNvbG9yOiMxRjQ5 N0Q7fQ0Kc3Bhbi5CYWxsb29uVGV4dENoYXINCgl7bXNvLXN0eWxlLW5hbWU6IkJhbGxvb24gVGV4 dCBDaGFyIjsNCgltc28tc3R5bGUtbm9zaG93OnllczsNCgltc28tc3R5bGUtcHJpb3JpdHk6OTk7 DQoJbXNvLXN0eWxlLXVuaGlkZTpubzsNCgltc28tc3R5bGUtbG9ja2VkOnllczsNCgltc28tc3R5 bGUtbGluazoiQmFsbG9vbiBUZXh0IjsNCgltc28tYW5zaS1mb250LXNpemU6OC4wcHQ7DQoJbXNv LWJpZGktZm9udC1zaXplOjguMHB0Ow0KCWZvbnQtZmFtaWx5OuWui+S9kzsNCgltc28tYXNjaWkt Zm9udC1mYW1pbHk65a6L5L2TOw0KCW1zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzsNCglt c28taGFuc2ktZm9udC1mYW1pbHk65a6L5L2TOw0KCW1zby1iaWRpLWZvbnQtZmFtaWx5OuWui+S9 kzsNCgltc28tZm9udC1rZXJuaW5nOjBwdDt9DQouTXNvQ2hwRGVmYXVsdA0KCXttc28tc3R5bGUt dHlwZTpleHBvcnQtb25seTsNCgltc28tZGVmYXVsdC1wcm9wczp5ZXM7DQoJbXNvLWJpZGktZm9u dC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiI7fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3NpemU6 NjEyLjBwdCA3OTIuMHB0Ow0KCW1hcmdpbjo3Mi4wcHQgOTAuMHB0IDcyLjBwdCA5MC4wcHQ7DQoJ bXNvLWhlYWRlci1tYXJnaW46MzYuMHB0Ow0KCW1zby1mb290ZXItbWFyZ2luOjM2LjBwdDsNCglt c28tcGFwZXItc291cmNlOjA7fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldvcmRTZWN0aW9u MTt9DQotLT48L3N0eWxlPjwhLS1baWYgZ3RlIG1zbyAxMF0+PHN0eWxlPi8qIFN0eWxlIERlZmlu aXRpb25zICovDQp0YWJsZS5Nc29Ob3JtYWxUYWJsZQ0KCXttc28tc3R5bGUtbmFtZToiVGFibGUg Tm9ybWFsIjsNCgltc28tdHN0eWxlLXJvd2JhbmQtc2l6ZTowOw0KCW1zby10c3R5bGUtY29sYmFu ZC1zaXplOjA7DQoJbXNvLXN0eWxlLW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5 Ow0KCW1zby1zdHlsZS1wYXJlbnQ6IiI7DQoJbXNvLXBhZGRpbmctYWx0OjBjbSA1LjRwdCAwY20g NS40cHQ7DQoJbXNvLXBhcmEtbWFyZ2luOjBjbTsNCgltc28tcGFyYS1tYXJnaW4tYm90dG9tOi4w MDAxcHQ7DQoJbXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZToxMC41cHQ7 DQoJbXNvLWJpZGktZm9udC1zaXplOjExLjBwdDsNCglmb250LWZhbWlseToiQ2FsaWJyaSIsInNh bnMtc2VyaWYiOw0KCW1zby1hc2NpaS1mb250LWZhbWlseTpDYWxpYnJpOw0KCW1zby1oYW5zaS1m b250LWZhbWlseTpDYWxpYnJpOw0KCW1zby1mb250LWtlcm5pbmc6MS4wcHQ7fQ0KPC9zdHlsZT48 IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpl eHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0 ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6 ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0t Pg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iWkgtQ04iIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUi IHN0eWxlPSJ0YWItaW50ZXJ2YWw6MjEuMHB0Ij4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9uMSI+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6 ZToxMC41cHQ7bXNvLWJpZGktZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxp YnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtZm9udC1mYW1pbHk6 5a6L5L2TO21zby1iaWRpLWZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oztj b2xvcjojMUY0OTdEIj5IaSBUb20sPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1i aWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv dDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlk aS1mb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+ PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g bGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQtc2l6ZTox MS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1 b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZhbWlseTom cXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+VGhhbmtzIGZvciB5b3Vy IGNvbW1lbnRzLg0KPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQt c2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNl cmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZh bWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+WWVzIFRTTy9M Uk8gd2l0aCBWWExBTiBzdXBwb3J0IHNob3VsZCBwcm92aWRlIHNpbWlsYXINCiBvciBldmVuIGJl dHRlciBwZXJmb3JtYW5jZSBnYWlucywgYnV0IHRoZSBtZWNoYW5pc20gcHJvcG9zZWQgYnkgdGhp cyBkcmFmdCBkZWNvdXBsZXMgb3ZlcmxheSBhbmQgdW5kZXJsYXksIGFuZCBpdCBpcyBoYXJkd2Fy ZSBpbmRlcGVuZGVudC4NCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1m b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fu cy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9u dC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPlNlY29u ZGx5LCBoYXJkd2FyZSBvZmZsb2FkaW5nIHVzdWFsbHkgc3VwcG9ydCBUQ1Agb25seQ0KIChUU08p LiBUaGUgbWVjaGFuaXNtIGhlcmUgY2FuIGhlbHAgb24gbGFyZ2UgVURQIHBhY2tldCBwZXJmb3Jt YW5jZSwgYWxzbyB2ZXJpZmllZCBieSB0aGUgcHJvdG90eXBlLjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1z aXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0Nh bGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZhbWls eTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7 O2NvbG9yOiMxRjQ5N0QiPkJUVywgaG93IG1hbnkgdHlwZXMgb2Ygb2ZmLXRoZS1zaGVsZiBOSUMg c3VwcG9ydCBWWExBTg0KIG9mZmxvYWRpbmc/IEFueSBwZXJmb3JtYW5jZSBkYXRhPzxvOnA+PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBz dHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFz dC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3 IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXpl OjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGli cmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZhbWlseTrl rovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7O2Nv bG9yOiMxRjQ5N0QiPkxpa2V3aXNlLCBzZXR0aW5nIGxhcmdlIE1UVSBvbiBvdmVybGF5IGludGVy ZmFjZXMgYWNoaWV2ZXMNCiBzaW1pbGFyIHJlc3VsdCwgYnV0IHN0aWxsIHRoZSBvdmVybGF5L3Vu ZGVybGF5IGRlY291cGxpbmcgaXNzdWUuIEl0IGlzIHVzdWFsbHkgYWR2aXNlZCB0aGF0IG92ZXJs YXkgTVRVIGlzIHNsaWdodGx5IHNtYWxsZXIgdGhhbiB1bmRlcmxheSB0byBhdm9pZCBpbmVmZmlj aWVudCBmcmFnbWVudGF0aW9uIGFmdGVyIGFkZGluZyB0aGUgb3V0ZXIgaGVhZGVyLCBidXQgdG8g YWNoaWV2ZSByZWFsbHkgaGlnaCBwZXJmb3JtYW5jZSBiZXR3ZWVuIFZNcywNCiBsYXJnZSBNVFUg aXMgcHJlZmVycmVkLiBBbmQgY29uc2lkZXJpbmcgb3ZlcmxheSAmbHQ7LSZndDsgcGh5c2ljYWwg Y29ubmVjdGlvbiwgcGF0aCBNVFUgZGlzY292ZXJ5IGlzIG5vdCBhbHdheXMgd29yay48bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIg c3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7bXNvLWJpZGktZm9udC1zaXplOjExLjBwdDtmb250LWZh bWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVh c3QtZm9udC1mYW1pbHk65a6L5L2TO21zby1iaWRpLWZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5l dyBSb21hbiZxdW90Oztjb2xvcjojMUY0OTdEIj5UaGlzIGtpbmQgb2YgY29uZmlndXJhdGlvbiBj b21wbGV4aXR5IGFuZCBwYWluLXBvaW50DQogY2FuIGJlIHJlc29sdmVkIHNpbXBseSBieSBkZWNv dXBsaW5nIG92ZXJsYXkgYW5kIHVuZGVybGF5IE1UVSwgYXMgc3VnZ2VzdGVkIGJ5IHRoaXMgZHJh ZnQuIEhlcmUgaXMgYW4gZXhhbXBsZSBvZiBjb25maWd1cmF0aW9uIGNvbmZ1c2lvbjo8bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIg c3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7bXNvLWJpZGktZm9udC1zaXplOjExLjBwdDtmb250LWZh bWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVh c3QtZm9udC1mYW1pbHk65a6L5L2TO21zby1iaWRpLWZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5l dyBSb21hbiZxdW90Oztjb2xvcjojMUY0OTdEIj48YSBocmVmPSJodHRwOi8vb3BlbnZzd2l0Y2gu b3JnL3BpcGVybWFpbC9kaXNjdXNzLzIwMTQtTWF5LzAxMzg5OC5odG1sIj5odHRwOi8vb3BlbnZz d2l0Y2gub3JnL3BpcGVybWFpbC9kaXNjdXNzLzIwMTQtTWF5LzAxMzg5OC5odG1sPC9hPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQt ZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFy ZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMg TmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1z aXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0Nh bGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZhbWls eTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7 O2NvbG9yOiMxRjQ5N0QiPklkZWFsbHksIGFsbCBOViB0dW5uZWwgcHJvdG9jb2xzIHNob3VsZCBz dXBwb3J0IHNpbWlsYXINCiBtZXRhZGF0YSwgdGh1cyBvdmVybGF5IHNlZ21lbnRhdGlvbiBjYW4g YmUgb2ZmbG9hZGVkIGhvcC1ieS1ob3AuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21z by1iaWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90Oywm cXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28t YmlkaS1mb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3 RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQtc2l6 ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlm JnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZhbWls eTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+QmVzdCByZWdhcmRz LDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9 IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0 O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztt c28tZmFyZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7 VGltZXMgTmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPkhhbjxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9u dC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZh bWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXYgc3R5 bGU9ImJvcmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkIGJsdWUgMS41cHQ7cGFkZGluZzowY20g MGNtIDBjbSA0LjBwdCI+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRv cDpzb2xpZCAjQjVDNERGIDEuMHB0O3BhZGRpbmc6My4wcHQgMGNtIDBjbSAwY20iPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PGI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAu MHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7 Ij5Gcm9tOjwvc3Bhbj48L2I+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAu MHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7 Ij4gVG9tIEhlcmJlcnQgW21haWx0bzp0aGVyYmVydEBnb29nbGUuY29tXQ0KPGJyPg0KPGI+U2Vu dDo8L2I+IFdlZG5lc2RheSwgTWF5IDIxLCAyMDE0IDI6NDQgQU08YnI+DQo8Yj5Ubzo8L2I+IFpo b3UsIEhhbjxicj4NCjxiPkNjOjwvYj4gbnZvM0BpZXRmLm9yZzsgdG9mb29AaWV0Zi5vcmc7IGRy YWZ0LW1haGFsaW5nYW0tZHV0dC1kY29wcy12eGxhbkB0b29scy5pZXRmLm9yZzsgZHJhZnQtemhv dS1saS12eGxhbi1zb2VAdG9vbHMuaWV0Zi5vcmc7IEVyaWsgTm9yZG1hcms8YnI+DQo8Yj5TdWJq ZWN0OjwvYj4gUmU6IEZXOiBJLUQgQWN0aW9uOiBkcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMS50 eHQ8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+SGkgWm91LCBhIGNv dXBsZSBvZiBxdWVzdGlvbnMgaW5saW5lLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij48 c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdj4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5PbiBNb24sIE1heSAxOSwgMjAx NCBhdCA4OjAxIFBNLCBaaG91LCBIYW4gJmx0OzxhIGhyZWY9Im1haWx0bzpoemhvdThAZWJheS5j b20iIHRhcmdldD0iX2JsYW5rIj5oemhvdThAZWJheS5jb208L2E+Jmd0OyB3cm90ZTo8bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+ SGksPGJyPg0KPGJyPg0KV2UgaGF2ZSB1cGRhdGVkIHRoZSBWWExBTi1TT0UgZHJhZnQgYWNjb3Jk aW5nIHRvIGVhcmxpZXIgY29tbWVudHMuIE5vdyBpdCBpcyBmdWxseSBjb21wYXRpYmxlIHdpdGgg VlhMQU4tR1BFLiBBbmQgc29tZSBleGFtcGxlcyBhcmUgYWRkZWQgZm9yIGJldHRlciB1bmRlcnN0 YW5kaW5nJm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9ImJv cmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkICNDQ0NDQ0MgMS4wcHQ7bXNvLWJvcmRlci1sZWZ0 LWFsdDpzb2xpZCAjQ0NDQ0NDIC43NXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2lu LWxlZnQ6NC44cHQ7bWFyZ2luLXJpZ2h0OjBjbSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBsYW5nPSJFTi1VUyI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9ibG9ja3F1b3Rl Pg0KPGJsb2NrcXVvdGUgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkICNDQ0ND Q0MgMS4wcHQ7bXNvLWJvcmRlci1sZWZ0LWFsdDpzb2xpZCAjQ0NDQ0NDIC43NXB0O3BhZGRpbmc6 MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXJpZ2h0OjBjbSI+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPjxzcGFuIGxh bmc9IkVOLVVTIj5BIHByb3RvdHlwZSBpcyBhbHNvIGltcGxlbWVudGVkIGhlcmUgKHBhdGNoIGJh c2VkIG9uIE9wZW4gdlN3aXRjaCk6PGJyPg0KPGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL2h6 aG91OC9vcGVudnN3aXRjaC9jb21taXQvOWE3ZGViOGI0MzJjZTgzYTljMDlkN2Q0ZmY4NWZhMDUw ZjdkZDJiZSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vZ2l0aHViLmNvbS9oemhvdTgvb3BlbnZz d2l0Y2gvY29tbWl0LzlhN2RlYjhiNDMyY2U4M2E5YzA5ZDdkNGZmODVmYTA1MGY3ZGQyYmU8L2E+ PGJyPg0KPGJyPg0KbmV0cGVyZiBUQ1BfU1RSRUFNIHRlc3QgcmVzdWx0IGJldHdlZW4gYSBwYWly cyBvZiBWTXMgb24gaG9zdHMgd2l0aCAxMEcgaW50ZXJmYWNlczo8YnI+DQo8YnI+DQpCZWZvcmUg dGhlIGNoYW5nZTogMi42MiBHYml0cy9zZWM8YnI+DQpBZnRlciB0aGUgY2hhbmdlOiA2LjY4IEdi aXRzL3NlYzxicj4NClNwZWVkdXAgaXMgfjI1MCUuPG86cD48L286cD48L3NwYW4+PC9wPg0KPC9i bG9ja3F1b3RlPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT Ij5DYW4geW91IHByb3ZpZGUgc29tZSBtb3JlIGRldGFpbHMgb24gdGhpcyBiZW5lZml0PyBJdCBz ZWVtcyBsaWtlIHBsYWluIFRTTy9MUk8gdGhhdCB1bmRlcnN0YW5kcyBlbmNhcHN1bGF0aW9uIHNo b3VsZCBwcm92aWRlIHNpbWlsYXIgYmVuZWZpdHMgd2hlbiBnb2luZyBiZXR3ZWVuIGhvc3RzLjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286 cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0 eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1p bHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0 LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcg Um9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0K PC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQg I0NDQ0NDQyAxLjBwdDttc28tYm9yZGVyLWxlZnQtYWx0OnNvbGlkICNDQ0NDQ0MgLjc1cHQ7cGFk ZGluZzowY20gMGNtIDBjbSA2LjBwdDttYXJnaW4tbGVmdDo0LjhwdDttYXJnaW4tcmlnaHQ6MGNt Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEyLjBwdCI+PHNw YW4gbGFuZz0iRU4tVVMiPlRoZSBwYXRjaCBhdHRyYWN0ZWQgc29tZSBpbnRlcmVzdHMgaW4gT1ZT IGNvbW11bml0eSwgYnV0IHNpbmNlIHRoaXMgUkZDIGRyYWZ0IGlzIGluIHZlcnkgZWFybHkgc3Rh Z2Ugc28gaXQgaXMgcmVnYXJkZWQgYXMgaW5hcHByb3ByaWF0ZSBieSBKZXNzZSB0byBhcHBseSB0 aGUgY2hhbmdlIHRvIE9WUyB0cmVlLjxicj4NClRoZSBkaXNjdXNzIG1haWwtdGhyZWFkOjxicj4N CjxhIGhyZWY9Imh0dHA6Ly9vcGVudnN3aXRjaC5vcmcvcGlwZXJtYWlsL2Rpc2N1c3MvMjAxNC1N YXkvMDEzOTgxLmh0bWwiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vb3BlbnZzd2l0Y2gub3JnL3Bp cGVybWFpbC9kaXNjdXNzLzIwMTQtTWF5LzAxMzk4MS5odG1sPC9hPjxicj4NCjxhIGhyZWY9Imh0 dHA6Ly9vcGVudnN3aXRjaC5vcmcvcGlwZXJtYWlsL2Rpc2N1c3MvMjAxNC1NYXkvMDEzODk4Lmh0 bWwiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vb3BlbnZzd2l0Y2gub3JnL3BpcGVybWFpbC9kaXNj dXNzLzIwMTQtTWF5LzAxMzg5OC5odG1sPC9hPjxicj4NCjxicj4NClNvIHdlIHdvdWxkIGxpa2Ug dG8gcmVxdWVzdCBhIHJldmlldyBoZXJlIGJ5IE5WTzMvVE9GT08gZ3JvdXBzIGFuZCBWWExBTiBh dXRob3JzOiBpcyB0aGlzIFZYTEFOIGV4dGVuc2lvbiBpcyB3b3J0aCBmb3JtYWxseSBwdXQgaW50 byBWWExBTiBhcyBhIHN0YW5kYXJkLCBzbyB0aGF0IG1vcmUgcGVvcGxlIGNhbiBiZW5lZml0IGZy b20gaXQ/PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9ibG9ja3F1b3RlPg0KPGRpdj4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5Db3VsZCB5b3UgZ2V0IHRoZSBzYW1l IGVmZmVjdCBieSBzZXR0aW5nIGxhcmdlciBNVFVzIG9uIHRoZSBvdmVybGF5IG5ldHdvcmsgaW50 ZXJmYWNlIGFuZCByZWx5aW5nIGluIHBhdGggTVRVIGRpc2NvdmVyeSB3aGVuIGdvaW5nIG92ZXIg cGh5c2ljYWwgbmV0d29yaz88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7PG86cD48L286cD48 L3NwYW4+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVy LWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDttc28tYm9yZGVyLWxlZnQtYWx0OnNvbGlkICNDQ0ND Q0MgLjc1cHQ7cGFkZGluZzowY20gMGNtIDBjbSA2LjBwdDttYXJnaW4tbGVmdDo0LjhwdDttYXJn aW4tcmlnaHQ6MGNtIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tb3V0bGluZS1s ZXZlbDoxIj48c3BhbiBsYW5nPSJFTi1VUyI+QmVzdCByZWdhcmRzLDxicj4NCkhhbjxicj4NCjxi cj4NCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tPGJyPg0KRnJvbTogSS1ELUFubm91bmNlIFtt YWlsdG86PGEgaHJlZj0ibWFpbHRvOmktZC1hbm5vdW5jZS1ib3VuY2VzQGlldGYub3JnIiB0YXJn ZXQ9Il9ibGFuayI+aS1kLWFubm91bmNlLWJvdW5jZXNAaWV0Zi5vcmc8L2E+XSBPbiBCZWhhbGYg T2YNCjxhIGhyZWY9Im1haWx0bzppbnRlcm5ldC1kcmFmdHNAaWV0Zi5vcmciIHRhcmdldD0iX2Js YW5rIj5pbnRlcm5ldC1kcmFmdHNAaWV0Zi5vcmc8L2E+PGJyPg0KU2VudDogRnJpZGF5LCBNYXkg MDIsIDIwMTQgODowOSBQTTxicj4NClRvOiA8YSBocmVmPSJtYWlsdG86aS1kLWFubm91bmNlQGll dGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+aS1kLWFubm91bmNlQGlldGYub3JnPC9hPjxicj4NClN1 YmplY3Q6IEktRCBBY3Rpb246IGRyYWZ0LXpob3UtbGktdnhsYW4tc29lLTAxLnR4dDxicj4NCjxi cj4NCjxicj4NCkEgTmV3IEludGVybmV0LURyYWZ0IGlzIGF2YWlsYWJsZSBmcm9tIHRoZSBvbi1s aW5lIEludGVybmV0LURyYWZ0cyBkaXJlY3Rvcmllcy48YnI+DQo8YnI+DQo8YnI+DQombmJzcDsg Jm5ic3A7ICZuYnNwOyAmbmJzcDsgVGl0bGUgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu YnNwOyA6IFNlZ21lbnRhdGlvbiBPZmZsb2FkaW5nIEV4dGVuc2lvbiBmb3IgVlhMQU48YnI+DQom bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgQXV0aG9ycyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm bmJzcDsgOiBIYW4gWmhvdTxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBD aGVuZ3l1YW4gTGk8YnI+DQombmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgRmlsZW5hbWUgJm5i c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7OiBkcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMS50eHQ8 YnI+DQombmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgUGFnZXMgJm5ic3A7ICZuYnNwOyAmbmJz cDsgJm5ic3A7ICZuYnNwOyA6IDEzPGJyPg0KJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IERh dGUgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDs6IDIwMTQtMDUtMDI8 YnI+DQo8YnI+DQpBYnN0cmFjdDo8YnI+DQombmJzcDsgJm5ic3A7U2VnbWVudGF0aW9uIG9mZmxv YWRpbmcgaXMgbm93YWRheXMgY29tbW9uIGluIG5ldHdvcmsgc3RhY2s8YnI+DQombmJzcDsgJm5i c3A7aW1wbGVtZW50YXRpb24gYW5kIHdlbGwgc3VwcG9ydGVkIGJ5IHBhcmEtdmlydHVhbGl6ZWQg bmV0d29yayBkZXZpY2U8YnI+DQombmJzcDsgJm5ic3A7ZHJpdmVycyBmb3IgdmlydHVhbCBtYWNo aW5lIChWTSlzLiBUaGlzIGRyYWZ0IGRlc2NyaWJlcyBhbiBleHRlbnNpb248YnI+DQombmJzcDsg Jm5ic3A7dG8gVmlydHVhbCBlWHRlbnNpYmxlIExvY2FsIEFyZWEgTmV0d29yayAoVlhMQU4pIHNv IHRoYXQgc2VnbWVudGF0aW9uPGJyPg0KJm5ic3A7ICZuYnNwO2NhbiBiZSBkZWNvdXBsZWQgZnJv bSBwaHlzaWNhbC91bmRlcmxheSBuZXR3b3JrcyBhbmQgb2ZmbG9hZGVkPGJyPg0KJm5ic3A7ICZu YnNwO2Z1cnRoZXIgdG8gdGhlIHJlbW90ZSBlbmQtcG9pbnQgdGh1cyBpbXByb3ZpbmcgZGF0YS1w bGFuZSBwZXJmb3JtYW5jZTxicj4NCiZuYnNwOyAmbmJzcDtmb3IgVk1zIHJ1bm5pbmcgb24gdG9w IG9mIG92ZXJsYXkgbmV0d29ya3MuPGJyPg0KPGJyPg0KPGJyPg0KVGhlIElFVEYgZGF0YXRyYWNr ZXIgc3RhdHVzIHBhZ2UgZm9yIHRoaXMgZHJhZnQgaXM6PGJyPg0KPGEgaHJlZj0iaHR0cHM6Ly9k YXRhdHJhY2tlci5pZXRmLm9yZy9kb2MvZHJhZnQtemhvdS1saS12eGxhbi1zb2UvIiB0YXJnZXQ9 Il9ibGFuayI+aHR0cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9kb2MvZHJhZnQtemhvdS1saS12 eGxhbi1zb2UvPC9hPjxicj4NCjxicj4NClRoZXJlJ3MgYWxzbyBhIGh0bWxpemVkIHZlcnNpb24g YXZhaWxhYmxlIGF0Ojxicj4NCjxhIGhyZWY9Imh0dHA6Ly90b29scy5pZXRmLm9yZy9odG1sL2Ry YWZ0LXpob3UtbGktdnhsYW4tc29lLTAxIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Rvb2xzLmll dGYub3JnL2h0bWwvZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDE8L2E+PGJyPg0KPGJyPg0KQSBk aWZmIGZyb20gdGhlIHByZXZpb3VzIHZlcnNpb24gaXMgYXZhaWxhYmxlIGF0Ojxicj4NCjxhIGhy ZWY9Imh0dHA6Ly93d3cuaWV0Zi5vcmcvcmZjZGlmZj91cmwyPWRyYWZ0LXpob3UtbGktdnhsYW4t c29lLTAxIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3d3dy5pZXRmLm9yZy9yZmNkaWZmP3VybDI9 ZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDE8L2E+PGJyPg0KPGJyPg0KPGJyPg0KUGxlYXNlIG5v dGUgdGhhdCBpdCBtYXkgdGFrZSBhIGNvdXBsZSBvZiBtaW51dGVzIGZyb20gdGhlIHRpbWUgb2Yg c3VibWlzc2lvbjxicj4NCnVudGlsIHRoZSBodG1saXplZCB2ZXJzaW9uIGFuZCBkaWZmIGFyZSBh dmFpbGFibGUgYXQgPGEgaHJlZj0iaHR0cDovL3Rvb2xzLmlldGYub3JnIiB0YXJnZXQ9Il9ibGFu ayI+DQp0b29scy5pZXRmLm9yZzwvYT4uPGJyPg0KPGJyPg0KSW50ZXJuZXQtRHJhZnRzIGFyZSBh bHNvIGF2YWlsYWJsZSBieSBhbm9ueW1vdXMgRlRQIGF0Ojxicj4NCjxhIGhyZWY9ImZ0cDovL2Z0 cC5pZXRmLm9yZy9pbnRlcm5ldC1kcmFmdHMvIiB0YXJnZXQ9Il9ibGFuayI+ZnRwOi8vZnRwLmll dGYub3JnL2ludGVybmV0LWRyYWZ0cy88L2E+PGJyPg0KPGJyPg0KX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+DQpJLUQtQW5ub3VuY2UgbWFpbGluZyBs aXN0PGJyPg0KPGEgaHJlZj0ibWFpbHRvOkktRC1Bbm5vdW5jZUBpZXRmLm9yZyIgdGFyZ2V0PSJf YmxhbmsiPkktRC1Bbm5vdW5jZUBpZXRmLm9yZzwvYT48YnI+DQo8YSBocmVmPSJodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2ktZC1hbm5vdW5jZUludGVybmV0LURyYWZ0IiB0 YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9pLWQt YW5ub3VuY2U8YnI+DQpJbnRlcm5ldC1EcmFmdDwvYT4gZGlyZWN0b3JpZXM6IDxhIGhyZWY9Imh0 dHA6Ly93d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWwiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHA6Ly93 d3cuaWV0Zi5vcmcvc2hhZG93Lmh0bWw8L2E+PGJyPg0Kb3IgPGEgaHJlZj0iZnRwOi8vZnRwLmll dGYub3JnL2lldGYvMXNoYWRvdy1zaXRlcy50eHQiIHRhcmdldD0iX2JsYW5rIj5mdHA6Ly9mdHAu aWV0Zi5vcmcvaWV0Zi8xc2hhZG93LXNpdGVzLnR4dDwvYT48bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8L2Jsb2NrcXVvdGU+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9 IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9k aXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --_000_9F56174078B48B459268EFF1DAB66B1A109C36BCDENEXDDAS32corp_-- From nobody Tue May 20 18:01:38 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1F4361A03DE; Tue, 20 May 2014 18:01:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.152 X-Spam-Level: X-Spam-Status: No, score=-23.152 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tz1lgJVBFuFr; Tue, 20 May 2014 18:01:30 -0700 (PDT) Received: from den-mipot-002.corp.ebay.com (den-mipot-002.corp.ebay.com [216.113.175.153]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 515141A03CF; Tue, 20 May 2014 18:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400634090; x=1432170090; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=kwrUuxza/XSRX74jt1j5PIkB6zzUoSEjvfs9zmFwf08=; b=jxBekBw0yXCJ2mN8QPxwbFdlSBSQPParRajMGnrnA5zGlC7J+zjJrq4m 1nLEX+g7IIoa/HBh+jxvv1y5QDJnPiN49OzZ3g+tmYXprhKgSt7HfLiQG nxcyvRGHcWKrULUrYNgBY7s0beYtSuMZpRxzcoOn1Q/s/oZ/g9bQ5gRgz 8=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos;i="4.98,877,1392192000"; d="scan'208";a="51223184" Received: from den-vteml-001.corp.ebay.com (HELO DEN-EXMHT-006.corp.ebay.com) ([10.101.112.212]) by den-mipot-002.corp.ebay.com with ESMTP; 20 May 2014 18:01:29 -0700 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXMHT-006.corp.ebay.com ([fe80::5c45:283f:1e47:5cdf%17]) with mapi id 14.03.0174.001; Tue, 20 May 2014 19:01:29 -0600 From: "Zhou, Han" To: Joe Touch , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Thread-Topic: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOggAFb9YCAABDEMA== Date: Wed, 21 May 2014 01:01:27 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> In-Reply-To: <537B90C9.1090003@isi.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter: Scanned den1 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/PwHFazACW2DzW0FUSY_0qvJbe_w Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 01:01:33 -0000 Hi Joe, Thanks for your comment.=20 Yes you are right that "length" fields in packet headers can be regarded as= hard limit of MTU, but here in the draft we were referring interface MTU. = We will make it more precise in next versions. For your question, it seems there are misunderstandings. It is always Guest= OS (VM) handles the TCP, but segmentation is offloaded to host. Let me exp= lain the code change: - TX side: -- before the change: TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to h= ost, MSS carried by GSO metadata in skbuff. VXLAN layer add encapsulation, and overlay TCP segmentation is carried ri= ght before sending to host IP layer, which is the idea of GSO - segment at = the latest point. Host do IP fragmentation only if overlay segment + outer header exceed ph= ysical interface MTU. (e.g. when both guest MTU and host MTU are configured= to 1500) -- after the change: TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to h= ost, MSS carried by GSO metadata in skbuff. VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. So ove= rlay TCP segmentation is skipped. Host do IP fragmentation according to physical interface MTU. -RX side: -- before the change: Host do IP reassembly if necessary. (e.g. when both guest MTU and host MT= U are configured to 1500) Overlay TCP segments are decapsulated and delivered all the way to guest,= and guest OS do TCP handling. (high cost here) -- after the change: Host do IP reassembly, and large packets decapsulated and delivered to gu= est, and guest OS do TCP handling. (cost reduced here because of reduced nu= mber of packets) I hope this clarifies. Best regards, Han > -----Original Message----- > From: Joe Touch [mailto:touch@isi.edu] > Sent: Wednesday, May 21, 2014 1:29 AM > To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org > Cc: Erik Nordmark; Tom Herbert > Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt >=20 > Hi, all, >=20 > I had a comment and a question: >=20 > Comment - (from the doc) overlays do have a hard MTU limit; it is the > limit of the encapsulation mechanism. E.g., without additional layers, > for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max - > (min IP header + UDP header)). Using additional adaptation layers, this > could be larger (e.g., see SEAL). >=20 > Question - the code appears to have the VXLAN layer do the > fragmentation, with the OS layer implementing the rest of TCP. There are > a lot of interactions, notably: >=20 > - any mechanism outside of the TCP source and TCP destination > that interprets the TCP header will result in a decrease in > functionality > i.e., the TCP connection will support only the > intersection of options and features supported > by the source, dest, *and* VXLAN layers >=20 > (rather than being limited only by the > source-dest pair) >=20 > - if passed a full TCP segment, this mechanism will be > incompatible with TCP security (e.g., TCP MD5, TCP-AO, and > the results of the TCPCRYPT WG. >=20 > I'm not quite sure from your doc whether you're re-segmenting TCP > segments, or merely collecting them for aggregate transit (e.g., as is > done in burst-mode Ethernet). >=20 > Can you please clarify? >=20 > Joe >=20 >=20 > On 5/19/2014 8:01 PM, Zhou, Han wrote: > > Hi, > > > > We have updated the VXLAN-SOE draft according to earlier comments. Now = it > is fully compatible with VXLAN-GPE. And some examples are added for bette= r > understanding. > > > > A prototype is also implemented here (patch based on Open vSwitch): > > > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d > 4ff85fa050f7dd2be > > > > netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G > interfaces: > > > > Before the change: 2.62 Gbits/sec > > After the change: 6.68 Gbits/sec > > Speedup is ~250%. > > > > The patch attracted some interests in OVS community, but since this RFC= draft > is in very early stage so it is regarded as inappropriate by Jesse to app= ly the > change to OVS tree. > > The discuss mail-thread: > > http://openvswitch.org/pipermail/discuss/2014-May/013981.html > > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > > > So we would like to request a review here by NVO3/TOFOO groups and VXLA= N > authors: is this VXLAN extension is worth formally put into VXLAN as a st= andard, > so that more people can benefit from it? > > > > Best regards, > > Han > > > > -----Original Message----- > > From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of > internet-drafts@ietf.org > > Sent: Friday, May 02, 2014 8:09 PM > > To: i-d-announce@ietf.org > > Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > > > > A New Internet-Draft is available from the on-line Internet-Drafts dire= ctories. > > > > > > Title : Segmentation Offloading Extension for VXLAN > > Authors : Han Zhou > > Chengyuan Li > > Filename : draft-zhou-li-vxlan-soe-01.txt > > Pages : 13 > > Date : 2014-05-02 > > > > Abstract: > > Segmentation offloading is nowadays common in network stack > > implementation and well supported by para-virtualized network devic= e > > drivers for virtual machine (VM)s. This draft describes an extensio= n > > to Virtual eXtensible Local Area Network (VXLAN) so that segmentati= on > > can be decoupled from physical/underlay networks and offloaded > > further to the remote end-point thus improving data-plane performan= ce > > for VMs running on top of overlay networks. > > > > > > The IETF datatracker status page for this draft is: > > https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > > > > There's also a htmlized version available at: > > http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > > > > A diff from the previous version is available at: > > http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe-01 > > > > > > Please note that it may take a couple of minutes from the time of submi= ssion > > until the htmlized version and diff are available at tools.ietf.org. > > > > Internet-Drafts are also available by anonymous FTP at: > > ftp://ftp.ietf.org/internet-drafts/ > > > > _______________________________________________ > > I-D-Announce mailing list > > I-D-Announce@ietf.org > > https://www.ietf.org/mailman/listinfo/i-d-announce > > Internet-Draft directories: http://www.ietf.org/shadow.html > > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > > _______________________________________________ > > Tofoo mailing list > > Tofoo@ietf.org > > https://www.ietf.org/mailman/listinfo/tofoo > > From nobody Tue May 20 18:14:50 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3527F1A03FB; Tue, 20 May 2014 18:14:48 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dHNKhtDv3eb0; Tue, 20 May 2014 18:14:45 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C7B671A03F6; Tue, 20 May 2014 18:14:45 -0700 (PDT) Received: from [128.9.160.81] (nib.isi.edu [128.9.160.81]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s4L1ESod022812 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 20 May 2014 18:14:29 -0700 (PDT) Message-ID: <537BFDF4.9030300@isi.edu> Date: Tue, 20 May 2014 18:14:28 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: "Zhou, Han" , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/o91j_1L5KDzq3IrEYVoNg7iz2rE Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 01:14:48 -0000 Hi, Han, This helps, but doesn't quite address my concern. TCP offloading is fine when the OS hands off user data, and the offload engine creates the entire segment. The situation you're describing seems to be a hybrid, where the guest OS makes a TCP segment, and then something lower (in the VM system) parses that segment to create one or more segments on the wire. If that happens, it will be incompatible with a number of existing TCP options, and will also cause side-effects with ACK clocking, timeouts, and more than a few other TCP features. Although I appreciate the goal of efficiency and speed, there is a severe compatibility issue that doesn't seem to be addressed. We can discuss this off-list if useful, FWIW. Joe On 5/20/2014 6:01 PM, Zhou, Han wrote: > Hi Joe, > > Thanks for your comment. > > Yes you are right that "length" fields in packet headers can be regarded as hard limit of MTU, but here in the draft we were referring interface MTU. We will make it more precise in next versions. > > For your question, it seems there are misunderstandings. It is always Guest OS (VM) handles the TCP, but segmentation is offloaded to host. Let me explain the code change: > - TX side: > -- before the change: > TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to host, MSS carried by GSO metadata in skbuff. > VXLAN layer add encapsulation, and overlay TCP segmentation is carried right before sending to host IP layer, which is the idea of GSO - segment at the latest point. > Host do IP fragmentation only if overlay segment + outer header exceed physical interface MTU. (e.g. when both guest MTU and host MTU are configured to 1500) > -- after the change: > TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to host, MSS carried by GSO metadata in skbuff. > VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. So overlay TCP segmentation is skipped. > Host do IP fragmentation according to physical interface MTU. > > -RX side: > -- before the change: > Host do IP reassembly if necessary. (e.g. when both guest MTU and host MTU are configured to 1500) > Overlay TCP segments are decapsulated and delivered all the way to guest, and guest OS do TCP handling. (high cost here) > -- after the change: > Host do IP reassembly, and large packets decapsulated and delivered to guest, and guest OS do TCP handling. (cost reduced here because of reduced number of packets) > > I hope this clarifies. > > Best regards, > Han > >> -----Original Message----- >> From: Joe Touch [mailto:touch@isi.edu] >> Sent: Wednesday, May 21, 2014 1:29 AM >> To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; >> draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; >> draft-zhou-li-vxlan-soe@tools.ietf.org >> Cc: Erik Nordmark; Tom Herbert >> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt >> >> Hi, all, >> >> I had a comment and a question: >> >> Comment - (from the doc) overlays do have a hard MTU limit; it is the >> limit of the encapsulation mechanism. E.g., without additional layers, >> for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max - >> (min IP header + UDP header)). Using additional adaptation layers, this >> could be larger (e.g., see SEAL). >> >> Question - the code appears to have the VXLAN layer do the >> fragmentation, with the OS layer implementing the rest of TCP. There are >> a lot of interactions, notably: >> >> - any mechanism outside of the TCP source and TCP destination >> that interprets the TCP header will result in a decrease in >> functionality >> i.e., the TCP connection will support only the >> intersection of options and features supported >> by the source, dest, *and* VXLAN layers >> >> (rather than being limited only by the >> source-dest pair) >> >> - if passed a full TCP segment, this mechanism will be >> incompatible with TCP security (e.g., TCP MD5, TCP-AO, and >> the results of the TCPCRYPT WG. >> >> I'm not quite sure from your doc whether you're re-segmenting TCP >> segments, or merely collecting them for aggregate transit (e.g., as is >> done in burst-mode Ethernet). >> >> Can you please clarify? >> >> Joe >> >> >> On 5/19/2014 8:01 PM, Zhou, Han wrote: >>> Hi, >>> >>> We have updated the VXLAN-SOE draft according to earlier comments. Now it >> is fully compatible with VXLAN-GPE. And some examples are added for better >> understanding. >>> >>> A prototype is also implemented here (patch based on Open vSwitch): >>> >> https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d >> 4ff85fa050f7dd2be >>> >>> netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G >> interfaces: >>> >>> Before the change: 2.62 Gbits/sec >>> After the change: 6.68 Gbits/sec >>> Speedup is ~250%. >>> >>> The patch attracted some interests in OVS community, but since this RFC draft >> is in very early stage so it is regarded as inappropriate by Jesse to apply the >> change to OVS tree. >>> The discuss mail-thread: >>> http://openvswitch.org/pipermail/discuss/2014-May/013981.html >>> http://openvswitch.org/pipermail/discuss/2014-May/013898.html >>> >>> So we would like to request a review here by NVO3/TOFOO groups and VXLAN >> authors: is this VXLAN extension is worth formally put into VXLAN as a standard, >> so that more people can benefit from it? >>> >>> Best regards, >>> Han >>> >>> -----Original Message----- >>> From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of >> internet-drafts@ietf.org >>> Sent: Friday, May 02, 2014 8:09 PM >>> To: i-d-announce@ietf.org >>> Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt >>> >>> >>> A New Internet-Draft is available from the on-line Internet-Drafts directories. >>> >>> >>> Title : Segmentation Offloading Extension for VXLAN >>> Authors : Han Zhou >>> Chengyuan Li >>> Filename : draft-zhou-li-vxlan-soe-01.txt >>> Pages : 13 >>> Date : 2014-05-02 >>> >>> Abstract: >>> Segmentation offloading is nowadays common in network stack >>> implementation and well supported by para-virtualized network device >>> drivers for virtual machine (VM)s. This draft describes an extension >>> to Virtual eXtensible Local Area Network (VXLAN) so that segmentation >>> can be decoupled from physical/underlay networks and offloaded >>> further to the remote end-point thus improving data-plane performance >>> for VMs running on top of overlay networks. >>> >>> >>> The IETF datatracker status page for this draft is: >>> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ >>> >>> There's also a htmlized version available at: >>> http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 >>> >>> A diff from the previous version is available at: >>> http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01 >>> >>> >>> Please note that it may take a couple of minutes from the time of submission >>> until the htmlized version and diff are available at tools.ietf.org. >>> >>> Internet-Drafts are also available by anonymous FTP at: >>> ftp://ftp.ietf.org/internet-drafts/ >>> >>> _______________________________________________ >>> I-D-Announce mailing list >>> I-D-Announce@ietf.org >>> https://www.ietf.org/mailman/listinfo/i-d-announce >>> Internet-Draft directories: http://www.ietf.org/shadow.html >>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt >>> >>> _______________________________________________ >>> Tofoo mailing list >>> Tofoo@ietf.org >>> https://www.ietf.org/mailman/listinfo/tofoo >>> From nobody Tue May 20 19:17:32 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E2FE1A042D; Tue, 20 May 2014 19:17:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.152 X-Spam-Level: X-Spam-Status: No, score=-23.152 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mYFctbMV-VSt; Tue, 20 May 2014 19:17:26 -0700 (PDT) Received: from den-mipot-001.corp.ebay.com (den-mipot-001.corp.ebay.com [216.113.175.152]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5096D1A040F; Tue, 20 May 2014 19:17:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400638646; x=1432174646; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=D+gC249J+PZjdSWJxQVpMdD9P+JY4wr0ygta6oE2EYo=; b=vQWnJPa7AM5hlAUxmtqPqQcZ73JahKp0ZPAReU0n5wtflyeU4YBE2B55 9QBwADkuIe3ipJY8rNC1VZiufaf0E3Du4CqNczG2jirx5Cl2aQDlAbr3n 3hKMmd/wD7iYqXwVi8IZHqKJIqr/9XH97nMjFTVWd2IksvE3tXMDHItR+ o=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos;i="4.98,877,1392192000"; d="scan'208";a="50940089" Received: from den-vteml-001.corp.ebay.com (HELO DEN-EXMHT-002.corp.ebay.com) ([10.101.112.212]) by den-mipot-001.corp.ebay.com with ESMTP; 20 May 2014 19:17:25 -0700 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXMHT-002.corp.ebay.com ([fe80::cbe:ffa5:17f0:a24a%14]) with mapi id 14.03.0174.001; Tue, 20 May 2014 20:17:25 -0600 From: "Zhou, Han" To: Joe Touch , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Thread-Topic: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOggAFb9YCAABDEMIAAcWAA//+gOXA= Date: Wed, 21 May 2014 02:17:24 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109C5845@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> In-Reply-To: <537BFDF4.9030300@isi.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter: Scanned den1 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/JLMThYWGgZCMwdo5mqCM3SJSe9A Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 02:17:28 -0000 Hi Joe, This is an interesting topic. > TCP offloading is fine when the OS hands off user data, and the offload > engine creates the entire segment. Existing TSO/GSO mechanisms deliver full (large) TCP segment to "offload en= gine", which then create smaller segments according to physical MTU, and re= calculates checksums. This is the case even without overlay considered. So I suppose the problem = you pointed out is not related to my change, but a general limitation for T= SO/GSO, right? For my understanding the TCP implementation should decide whether to use of= floading or not according to the feature/options required by a TCP connecti= on. If the option required (such as MD5) is not supported by offloading, th= e TCP stack should do the segmentation by itself instead of utilizing offlo= ading. In fact, the proposal in this draft should be able to alleviate the limitat= ion for TCP connections between VMs behind same gateways, because in this c= ase there is no real TCP segmentation performed by "offload engine".=20 Let me know if you have more concerns, or maybe an example of how an option= is broken by TSO/GSO, then we can check what's the current solution in ker= nel. Best regards, Han > -----Original Message----- > From: Joe Touch [mailto:touch@isi.edu] > Sent: Wednesday, May 21, 2014 9:14 AM > To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org > Cc: Erik Nordmark; Tom Herbert > Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt >=20 > Hi, Han, >=20 > This helps, but doesn't quite address my concern. >=20 > TCP offloading is fine when the OS hands off user data, and the offload > engine creates the entire segment. >=20 > The situation you're describing seems to be a hybrid, where the guest OS > makes a TCP segment, and then something lower (in the VM system) parses > that segment to create one or more segments on the wire. >=20 > If that happens, it will be incompatible with a number of existing TCP > options, and will also cause side-effects with ACK clocking, timeouts, > and more than a few other TCP features. >=20 > Although I appreciate the goal of efficiency and speed, there is a > severe compatibility issue that doesn't seem to be addressed. >=20 > We can discuss this off-list if useful, FWIW. >=20 > Joe >=20 > On 5/20/2014 6:01 PM, Zhou, Han wrote: > > Hi Joe, > > > > Thanks for your comment. > > > > Yes you are right that "length" fields in packet headers can be regarde= d as hard > limit of MTU, but here in the draft we were referring interface MTU. We w= ill > make it more precise in next versions. > > > > For your question, it seems there are misunderstandings. It is always G= uest OS > (VM) handles the TCP, but segmentation is offloaded to host. Let me expla= in the > code change: > > - TX side: > > -- before the change: > > TCP segmentation offloaded by TSO of Guest OS virt-driver from guest= to > host, MSS carried by GSO metadata in skbuff. > > VXLAN layer add encapsulation, and overlay TCP segmentation is carri= ed > right before sending to host IP layer, which is the idea of GSO - segment= at the > latest point. > > Host do IP fragmentation only if overlay segment + outer header exce= ed > physical interface MTU. (e.g. when both guest MTU and host MTU are config= ured > to 1500) > > -- after the change: > > TCP segmentation offloaded by TSO of Guest OS virt-driver from guest= to > host, MSS carried by GSO metadata in skbuff. > > VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. S= o > overlay TCP segmentation is skipped. > > Host do IP fragmentation according to physical interface MTU. > > > > -RX side: > > -- before the change: > > Host do IP reassembly if necessary. (e.g. when both guest MTU and ho= st > MTU are configured to 1500) > > Overlay TCP segments are decapsulated and delivered all the way to g= uest, > and guest OS do TCP handling. (high cost here) > > -- after the change: > > Host do IP reassembly, and large packets decapsulated and delivered = to > guest, and guest OS do TCP handling. (cost reduced here because of reduce= d > number of packets) > > > > I hope this clarifies. > > > > Best regards, > > Han > > > >> -----Original Message----- > >> From: Joe Touch [mailto:touch@isi.edu] > >> Sent: Wednesday, May 21, 2014 1:29 AM > >> To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; > >> draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > >> draft-zhou-li-vxlan-soe@tools.ietf.org > >> Cc: Erik Nordmark; Tom Herbert > >> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt > >> > >> Hi, all, > >> > >> I had a comment and a question: > >> > >> Comment - (from the doc) overlays do have a hard MTU limit; it is the > >> limit of the encapsulation mechanism. E.g., without additional layers, > >> for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max - > >> (min IP header + UDP header)). Using additional adaptation layers, thi= s > >> could be larger (e.g., see SEAL). > >> > >> Question - the code appears to have the VXLAN layer do the > >> fragmentation, with the OS layer implementing the rest of TCP. There a= re > >> a lot of interactions, notably: > >> > >> - any mechanism outside of the TCP source and TCP destination > >> that interprets the TCP header will result in a decrease in > >> functionality > >> i.e., the TCP connection will support only the > >> intersection of options and features supported > >> by the source, dest, *and* VXLAN layers > >> > >> (rather than being limited only by the > >> source-dest pair) > >> > >> - if passed a full TCP segment, this mechanism will be > >> incompatible with TCP security (e.g., TCP MD5, TCP-AO, and > >> the results of the TCPCRYPT WG. > >> > >> I'm not quite sure from your doc whether you're re-segmenting TCP > >> segments, or merely collecting them for aggregate transit (e.g., as is > >> done in burst-mode Ethernet). > >> > >> Can you please clarify? > >> > >> Joe > >> > >> > >> On 5/19/2014 8:01 PM, Zhou, Han wrote: > >>> Hi, > >>> > >>> We have updated the VXLAN-SOE draft according to earlier comments. No= w > it > >> is fully compatible with VXLAN-GPE. And some examples are added for be= tter > >> understanding. > >>> > >>> A prototype is also implemented here (patch based on Open vSwitch): > >>> > >> > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d > >> 4ff85fa050f7dd2be > >>> > >>> netperf TCP_STREAM test result between a pairs of VMs on hosts with 1= 0G > >> interfaces: > >>> > >>> Before the change: 2.62 Gbits/sec > >>> After the change: 6.68 Gbits/sec > >>> Speedup is ~250%. > >>> > >>> The patch attracted some interests in OVS community, but since this R= FC > draft > >> is in very early stage so it is regarded as inappropriate by Jesse to = apply the > >> change to OVS tree. > >>> The discuss mail-thread: > >>> http://openvswitch.org/pipermail/discuss/2014-May/013981.html > >>> http://openvswitch.org/pipermail/discuss/2014-May/013898.html > >>> > >>> So we would like to request a review here by NVO3/TOFOO groups and > VXLAN > >> authors: is this VXLAN extension is worth formally put into VXLAN as a > standard, > >> so that more people can benefit from it? > >>> > >>> Best regards, > >>> Han > >>> > >>> -----Original Message----- > >>> From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf O= f > >> internet-drafts@ietf.org > >>> Sent: Friday, May 02, 2014 8:09 PM > >>> To: i-d-announce@ietf.org > >>> Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > >>> > >>> > >>> A New Internet-Draft is available from the on-line Internet-Drafts > directories. > >>> > >>> > >>> Title : Segmentation Offloading Extension for > VXLAN > >>> Authors : Han Zhou > >>> Chengyuan Li > >>> Filename : draft-zhou-li-vxlan-soe-01.txt > >>> Pages : 13 > >>> Date : 2014-05-02 > >>> > >>> Abstract: > >>> Segmentation offloading is nowadays common in network stack > >>> implementation and well supported by para-virtualized network > device > >>> drivers for virtual machine (VM)s. This draft describes an exten= sion > >>> to Virtual eXtensible Local Area Network (VXLAN) so that > segmentation > >>> can be decoupled from physical/underlay networks and offloaded > >>> further to the remote end-point thus improving data-plane > performance > >>> for VMs running on top of overlay networks. > >>> > >>> > >>> The IETF datatracker status page for this draft is: > >>> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > >>> > >>> There's also a htmlized version available at: > >>> http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > >>> > >>> A diff from the previous version is available at: > >>> http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe-01 > >>> > >>> > >>> Please note that it may take a couple of minutes from the time of sub= mission > >>> until the htmlized version and diff are available at tools.ietf.org. > >>> > >>> Internet-Drafts are also available by anonymous FTP at: > >>> ftp://ftp.ietf.org/internet-drafts/ > >>> > >>> _______________________________________________ > >>> I-D-Announce mailing list > >>> I-D-Announce@ietf.org > >>> https://www.ietf.org/mailman/listinfo/i-d-announce > >>> Internet-Draft directories: http://www.ietf.org/shadow.html > >>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > >>> > >>> _______________________________________________ > >>> Tofoo mailing list > >>> Tofoo@ietf.org > >>> https://www.ietf.org/mailman/listinfo/tofoo > >>> From nobody Tue May 20 19:42:36 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0DC3F1A043F for ; Tue, 20 May 2014 19:42:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wUjqO6-LoeqA for ; Tue, 20 May 2014 19:42:30 -0700 (PDT) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 126031A0433 for ; Tue, 20 May 2014 19:42:29 -0700 (PDT) Received: by mail-ie0-f182.google.com with SMTP id as1so1413153iec.13 for ; Tue, 20 May 2014 19:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dfqzi112raD0FculCE/sZefuUX7xxsz4uMwKr44rqMg=; b=YFkWzDzTQDxfN6g5s+6Pb8N6/ndzk/kG9YTrvzra1yAdp9qCxyzoePMOVupFVqLSjY ZbW/YsCcfFWk/HCWRzGOXvl55RksUpLZuckgIgew+El1K8rs47WY1BFTlU12pezc1/qd DBUbm0IPqWPJjUqu9DYbkJ83vN5patRdWRD/LId1oTHFEOo6xN3qwtAXtoVK78HagCxI WpCFPbRxTRVLOrL89TsKXJnO0rVFJzSg6gm7q95t7hrPwG7/2kFHlRJ+bn1eQkPlWIvc QqaDSjvBDwuYWB2OejOMijgBSiMFYT4qLax3tQDm5gGrgD9QEQthD4APBQ1EBzCZr9QB IW5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=dfqzi112raD0FculCE/sZefuUX7xxsz4uMwKr44rqMg=; b=Zamj1FXODIIbmCuPdNuJzo7BY0ZwqFaMIxXwNnjkKOFETMWV27jVjfVfoW1hJfn+/y n/dnjRFJSm8pLv4BKJhCA2gpiKEIxSUSGbIqxQEanXof/SXaIPzCjqP/PIrJna378UWx 5xMTerXRtS5/bZhiAIRdLhcBpSVk5hqny7mwMewHI+z/b1s6m0OLTcbGPX8ML5tNvppd 9ts/z2O4opoL3u4fj44qijEzwDGPQRr1KTOXHYrX7Pq2jGLhH4rIxZxsRH00wxr20f9Y yHN/9ausQOTUegUlGYH5BeO41W+o9soZZGCPfVKcYIfrVS2Bz5ASUd/EtNGdbeYMA/eS eyLw== X-Gm-Message-State: ALoCoQn1e81voWOsQ2Jp4t3sYp+5SIqytTjkp3whklm+BiB4P1+C0ANy0zn+w3Hl/L9KamdGOth9 MIME-Version: 1.0 X-Received: by 10.42.119.138 with SMTP id b10mr46735456icr.31.1400640149039; Tue, 20 May 2014 19:42:29 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Tue, 20 May 2014 19:42:28 -0700 (PDT) In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C36BC@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <9F56174078B48B459268EFF1DAB66B1A109C36BC@DEN-EXDDA-S32.corp.ebay.com> Date: Tue, 20 May 2014 19:42:28 -0700 Message-ID: From: Tom Herbert To: "Zhou, Han" Content-Type: multipart/alternative; boundary=90e6ba5bc1b78ac96004f9dff26e Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/gciX9CSIvEuf4W-07j6-r0L9VhA Cc: "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" , Erik Nordmark Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 02:42:33 -0000 --90e6ba5bc1b78ac96004f9dff26e Content-Type: text/plain; charset=UTF-8 On Tue, May 20, 2014 at 5:28 PM, Zhou, Han wrote: > Hi Tom, > > > > Thanks for your comments. > > Yes TSO/LRO with VXLAN support should provide similar or even better > performance gains, but the mechanism proposed by this draft decouples > overlay and underlay, and it is hardware independent. > > Secondly, hardware offloading usually support TCP only (TSO). The > mechanism here can help on large UDP packet performance, also verified by > the prototype. > BTW, how many types of off-the-shelf NIC support VXLAN offloading? Any > performance data? > > > HW is not a requirement for this offloading. There has been a lot of work recently to get software variants (GSO and GRO) working in Linux with tunnels. These do show 2-3x performance improvements. HW support would be mostly an incremental improvement to that, or becomes interesting for OS bypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and LRO so that it's possible to plumb use of large packets from the guest all the way to the host driver. It should also be possible to plumb two VMs on the same host to communicate without segmentation, i.e. output from TSO on one VM becomes input for another. The advantage that I see in your draft is that it allows an intermediate device to perform segmentation/reassembly instead of fragmentation. Likewise, setting large MTU on overlay interfaces achieves similar result, > but still the overlay/underlay decoupling issue. It is usually advised that > overlay MTU is slightly smaller than underlay to avoid inefficient > fragmentation after adding the outer header, but to achieve really high > performance between VMs, large MTU is preferred. > Depends on the performance dimension to be optimized. Larger MTUs could increase latency of high priority small packets for instance (HOL blocking), or UDP based application might try to use MTU to decide how large it should send it's packets to avoid fragmentation. And considering overlay <-> physical connection, path MTU discovery is not > always work. > This kind of configuration complexity and pain-point can be resolved > simply by decoupling overlay and underlay MTU, as suggested by this draft. > Here is an example of configuration confusion: > > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > > > Ideally, all NV tunnel protocols should support similar metadata, thus > overlay segmentation can be offloaded hop-by-hop. > I think this would be applicable to about all tunnel protocols. Then you would also want to do reassembly at each hop? Sounds expensive. Once you segment, I think you'd only only want to reassemble at the end host. One important thing to keep in mind, and a hard lesson in real deployment :-), segmentation offload is opportunistic and very dependent on the conditions of the traffic. It's value can be fleeting in real workloads. For instance imagine a host with a lot of active high throughout connections (like a video serving) which hits a hiccup causing all congestions windows to shrink. If the system is not provisioned for this event it can be very hard to recover (a lot more CPU is required to achieve same throughput). This differentiates a larger MTU from relying on segmentation, the former offers more predictable CPU savings. Tom > > Best regards, > > Han > > > > *From:* Tom Herbert [mailto:therbert@google.com] > *Sent:* Wednesday, May 21, 2014 2:44 AM > *To:* Zhou, Han > *Cc:* nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark > *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > > Hi Zou, a couple of questions inline. > > > > On Mon, May 19, 2014 at 8:01 PM, Zhou, Han wrote: > > Hi, > > We have updated the VXLAN-SOE draft according to earlier comments. Now it > is fully compatible with VXLAN-GPE. And some examples are added for better > understanding > > > > A prototype is also implemented here (patch based on Open vSwitch): > > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be > > netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G > interfaces: > > Before the change: 2.62 Gbits/sec > After the change: 6.68 Gbits/sec > Speedup is ~250%. > > Can you provide some more details on this benefit? It seems like plain > TSO/LRO that understands encapsulation should provide similar benefits when > going between hosts. > > > > > > The patch attracted some interests in OVS community, but since this RFC > draft is in very early stage so it is regarded as inappropriate by Jesse to > apply the change to OVS tree. > The discuss mail-thread: > http://openvswitch.org/pipermail/discuss/2014-May/013981.html > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > So we would like to request a review here by NVO3/TOFOO groups and VXLAN > authors: is this VXLAN extension is worth formally put into VXLAN as a > standard, so that more people can benefit from it? > > Could you get the same effect by setting larger MTUs on the overlay > network interface and relying in path MTU discovery when going over > physical network? > > > > Best regards, > Han > > -----Original Message----- > From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of > internet-drafts@ietf.org > Sent: Friday, May 02, 2014 8:09 PM > To: i-d-announce@ietf.org > Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > > Title : Segmentation Offloading Extension for VXLAN > Authors : Han Zhou > Chengyuan Li > Filename : draft-zhou-li-vxlan-soe-01.txt > Pages : 13 > Date : 2014-05-02 > > Abstract: > Segmentation offloading is nowadays common in network stack > implementation and well supported by para-virtualized network device > drivers for virtual machine (VM)s. This draft describes an extension > to Virtual eXtensible Local Area Network (VXLAN) so that segmentation > can be decoupled from physical/underlay networks and offloaded > further to the remote end-point thus improving data-plane performance > for VMs running on top of overlay networks. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > > There's also a htmlized version available at: > http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > > A diff from the previous version is available at: > http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01 > > > Please note that it may take a couple of minutes from the time of > submission > until the htmlized version and diff are available at tools.ietf.org. > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draftdirectories: > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > --90e6ba5bc1b78ac96004f9dff26e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Tue, May 20, 2014 at 5:28 PM, Zhou, Han <hzhou8@ebay.com><= /span> wrote:

Hi Tom,=

=C2= =A0

Thanks for= your comments.

Yes TSO/LR= O with VXLAN support should provide similar or even better performance gains, but the mechanism proposed by this draft= decouples overlay and underlay, and it is hardware independent.

Secondly, = hardware offloading usually support TCP only (TSO). The mechanism here can help on large UDP packet performance, also v= erified by the prototype.=C2=A0=C2=A0

BTW, how m= any types of off-the-shelf NIC support VXLAN offloading? Any performance data?

=C2= =A0

HW is not a requirement for thi= s offloading. There has been a lot of work recently to get software variant= s (GSO and GRO) working in Linux with tunnels. These do show 2-3x performan= ce improvements. HW support would be mostly an incremental improvement to t= hat, or becomes interesting for OS bypass like in SR-IOV. GSO/GRO can be pr= esented to the guest as TSO and LRO so that it's possible to plumb use = of large packets from the guest all the way to the host driver. It should a= lso be possible to plumb two VMs on the same host to communicate without se= gmentation, i.e. output from TSO on one VM becomes input for another.=C2=A0=

The advantage that I see in your draft is that it allow= s an intermediate device to perform segmentation/reassembly instead of frag= mentation.

Likewise, = setting large MTU on overlay interfaces achieves similar result, but still the overlay/underlay decoupling issue. It is usu= ally advised that overlay MTU is slightly smaller than underlay to avoid in= efficient fragmentation after adding the outer header, but to achieve reall= y high performance between VMs, large MTU is preferred.


Depends on the performance dimension to be optimized. Larger MTUs cou= ld increase latency of high priority small packets for instance (HOL blocki= ng), or UDP based application might try to use MTU to decide how large it s= hould send it's packets to avoid fragmentation.

And considering overlay <-> physical connection, pat= h MTU discovery is not always work.=C2=A0

This kind = of configuration complexity and pain-point can be resolved simply by decoupling overlay and underlay MTU, as suggeste= d by this draft. Here is an example of configuration confusion:

http://openvswitch.org/pipermail/discuss/2014-May/013898.html=

=C2= =A0

Ideally, a= ll NV tunnel protocols should support similar metadata, thus overlay segmentation can be offloaded hop-by-hop.


I think this would be applica= ble to about all tunnel protocols. Then you would also want to do reassembl= y at each hop? Sounds expensive. Once you segment, I think you'd only o= nly want to reassemble at the end host.

One important thing to keep in mind, and a hard lesson = in real deployment :-), segmentation offload is opportunistic and very depe= ndent on the conditions of the traffic. It's value can be fleeting in r= eal workloads. For instance imagine a host with a lot of active high throug= hout connections (like a video serving) which hits a hiccup causing all con= gestions windows to shrink. If the system is not provisioned for this event= it can be very hard to recover (a lot more CPU is required to achieve same= throughput). This differentiates a larger MTU from relying on segmentation= , the former offers more predictable CPU savings.

Tom

=

=C2= =A0

Best regar= ds,

Han=

=C2= =A0

From: Tom Herbert [mailto:therbert@google.com]
Sent: Wednesday, May 21, 2014 2:44 AM
To: Zhou, Han
Cc: nvo3@ietf.org= ; tofoo@ietf.org; draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; draft-= zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

=C2=A0

Hi Zou, a couple of questions i= nline.

= =C2=A0

On Mon, May 19, 2014 at 8:01 PM= , Zhou, Han <hzhou8= @ebay.com> wrote:

Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it i= s fully compatible with VXLAN-GPE. And some examples are added for better u= nderstanding=C2=A0

=C2=A0

= A prototype is also implemented here (patch based on Open vSwitch):
https://github.com/hzhou8/openvswi= tch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be

netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G int= erfaces:

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.

Can you provide some more detai= ls on this benefit? It seems like plain TSO/LRO that understands encapsulat= ion should provide similar benefits when going between hosts.=

= =C2=A0

=C2= =A0

= The patch attracted some interests in OVS community, but since this RFC dra= ft is in very early stage so it is regarded as inappropriate by Jesse to ap= ply the change to OVS tree.
The discuss mail-thread:
http://openvswitch.org/pipermail/discuss/2014-May/013981.h= tml
http://openvswitch.org/pipermail/discuss/2014-May/013898.h= tml

So we would like to request a review here by NVO3/TOFOO groups and VXLAN au= thors: is this VXLAN extension is worth formally put into VXLAN as a standa= rd, so that more people can benefit from it?

Could you get the same effect b= y setting larger MTUs on the overlay network interface and relying in path = MTU discovery when going over physical network?

=C2=A0

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of internet-draf= ts@ietf.org
Sent: Friday, May 02, 2014 8:09 PM
To: i-d-announce= @ietf.org
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt


A New Internet-Draft is available from the on-line Internet-Drafts director= ies.


=C2=A0 =C2=A0 =C2=A0 =C2=A0 Title =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Segm= entation Offloading Extension for VXLAN
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Authors =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Han Zhou<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Chengyuan Li
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Filename =C2=A0 =C2=A0 =C2=A0 =C2=A0: draft-zho= u-li-vxlan-soe-01.txt
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Pages =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 13 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:= 2014-05-02

Abstract:
=C2=A0 =C2=A0Segmentation offloading is nowadays common in network stack =C2=A0 =C2=A0implementation and well supported by para-virtualized network = device
=C2=A0 =C2=A0drivers for virtual machine (VM)s. This draft describes an ext= ension
=C2=A0 =C2=A0to Virtual eXtensible Local Area Network (VXLAN) so that segme= ntation
=C2=A0 =C2=A0can be decoupled from physical/underlay networks and offloaded=
=C2=A0 =C2=A0further to the remote end-point thus improving data-plane perf= ormance
=C2=A0 =C2=A0for VMs running on top of overlay networks.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/<= br>
There's also a htmlized version available at:
http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe= -01


Please note that it may take a couple of minutes from the time of submissio= n
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp= .ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
I-D-Announce@iet= f.org
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or = ftp://ftp.ietf.org/ietf/1shadow-sites.txt

=C2=A0


--90e6ba5bc1b78ac96004f9dff26e-- From nobody Tue May 20 20:07:26 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C6BBD1A044B; Tue, 20 May 2014 20:07:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.151 X-Spam-Level: X-Spam-Status: No, score=-23.151 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d3p5Z20Vfl6M; Tue, 20 May 2014 20:07:14 -0700 (PDT) Received: from den-mipot-002.corp.ebay.com (den-mipot-002.corp.ebay.com [216.113.175.153]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8EACC1A0448; Tue, 20 May 2014 20:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400641633; x=1432177633; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=LdpWCYvGk1ZowCL51xymh5/ddSRUneUVA/YBSbLn1Mg=; b=iLWQZV0tYxVx/s0LbjS3TJSnfKP/CnYLy51e4wmbCsAQLeVjfVWVXnDB loucV1Ley6lk5ZQCld6OjWUPyhAr7pU9YfJLbcR4jVy6pPkJxNQffjSGh kSpGn1sP2w0rFemaxFrTCLBKXghDmxWArySHdK7WiPokePMe6so02pd5C g=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos; i="4.98,878,1392192000"; d="scan'208,217"; a="51233978" Received: from den-vteml-003.corp.ebay.com (HELO DEN-EXMHT-001.corp.ebay.com) ([10.101.112.119]) by den-mipot-002.corp.ebay.com with ESMTP; 20 May 2014 20:07:13 -0700 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXMHT-001.corp.ebay.com ([fe80::345e:2420:7d3d:208d%13]) with mapi id 14.03.0174.001; Tue, 20 May 2014 21:07:12 -0600 From: "Zhou, Han" To: Tom Herbert Thread-Topic: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOggAFw/gD///LaIIAAktcA//+euYA= Date: Wed, 21 May 2014 03:07:11 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109C7F16@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <9F56174078B48B459268EFF1DAB66B1A109C36BC@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: multipart/alternative; boundary="_000_9F56174078B48B459268EFF1DAB66B1A109C7F16DENEXDDAS32corp_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/umTtthvwrVMNUegjnC_VMUZawgw Cc: "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" , Erik Nordmark Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 03:07:19 -0000 --_000_9F56174078B48B459268EFF1DAB66B1A109C7F16DENEXDDAS32corp_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgVG9tLA0KDQo+IFRoZXJlIGhhcyBiZWVuIGEgbG90IG9mIHdvcmsgcmVjZW50bHkgdG8gZ2V0 IHNvZnR3YXJlIHZhcmlhbnRzIChHU08gYW5kIEdSTykgd29ya2luZyBpbiBMaW51eCB3aXRoIHR1 bm5lbHMuDQpBcmUgeW91IHJlZmVycmluZyBVRFAgR1JPIGZvciBWWExBTj8gSSBhbSBhd2FyZSBv ZiB0aGlzIGltcGxlbWVudGF0aW9uIGluIGtlcm5lbCAzLjE0LiBUaGUgcGVyZm9ybWFuY2UgZ2Fp biBzaG91bGQgYmUgc2ltaWxhciwgYnV0IHN0aWxsIHJlcXVpcmVzIGNhcmVmdWwgY29uZmlndXJh dGlvbiBvZiBWTSBNVFUgdG8gYXZvaWQgSVAgZnJhZ21lbnRhdGlvbiBvbiBwaHlzaWNhbCBNVFUs IHJhdGhlciB0aGFuIGRlY291cGxpbmcgZnJvbSB1bmRlcmxheS4gSSB0aGluayBlYWNoIG9mIHRo ZXNlIHNvbHV0aW9ucyBoYXMgaXRzIG93biBhZHZhbnRhZ2VzLg0KDQo+IEkgdGhpbmsgdGhpcyB3 b3VsZCBiZSBhcHBsaWNhYmxlIHRvIGFib3V0IGFsbCB0dW5uZWwgcHJvdG9jb2xzLiBUaGVuIHlv dSB3b3VsZCBhbHNvIHdhbnQgdG8gZG8gcmVhc3NlbWJseSBhdCBlYWNoIGhvcD8gU291bmRzIGV4 cGVuc2l2ZS4gT25jZSB5b3Ugc2VnbWVudCwgSSB0aGluayB5b3UnZCBvbmx5IG9ubHkgd2FudCB0 byByZWFzc2VtYmxlIGF0IHRoZSBlbmQgaG9zdC4NClRoZXJlIGlzIG5vIHNlZ21lbnRhdGlvbi9y ZWFzc2VtYmx5IG5lZWRlZCBhdCBlYWNoIGhvcCwgYnV0IGluc3RlYWQg4oCcb2ZmbG9hZOKAnSBo b3AgYnkgaG9wLCBqdXN0IHNhdmUgdGhlIG1ldGFkYXRhIHRvIHRoZSBuZXh0IGhvcCBoZWFkZXJz LiBBbmQgZmluYWxseSBzZWdtZW50YXRpb24gc2hvdWxkIGJlIGF2b2lkZWQgaWYgdGhlIGRlc3Rp bmF0aW9uIGlzIGEgVk0gb24gYSBoeXBlcnZpc29yLg0KVGhpcyBzY2VuYXJpbyBpcyBsaWtlIGlu IEV4YW1wbGUgMiAoc2VjdGlvbiA1LjIpLCBqdXN0IHJlcGxhY2UgdGhlIFZYTEFOLVNPRSBiZXR3 ZWVuIHRoZSAyIGdhdGV3YXlzIGJ5IGFueSBvdGhlciBUdW5uZWwgcHJvdG9jb2wgdGhhdCBjYW4g c3VwcG9ydCBzaW1pbGFyIG9mZmxvYWRpbmcgZmVhdHVyZSwgc3VjaCBhcyBTVFQuDQoNCkJlc3Qg cmVnYXJkcywNCkhhbg0KDQpGcm9tOiBUb20gSGVyYmVydCBbbWFpbHRvOnRoZXJiZXJ0QGdvb2ds ZS5jb21dDQpTZW50OiBXZWRuZXNkYXksIE1heSAyMSwgMjAxNCAxMDo0MiBBTQ0KVG86IFpob3Us IEhhbg0KQ2M6IG52bzNAaWV0Zi5vcmc7IHRvZm9vQGlldGYub3JnOyBkcmFmdC1tYWhhbGluZ2Ft LWR1dHQtZGNvcHMtdnhsYW5AdG9vbHMuaWV0Zi5vcmc7IGRyYWZ0LXpob3UtbGktdnhsYW4tc29l QHRvb2xzLmlldGYub3JnOyBFcmlrIE5vcmRtYXJrDQpTdWJqZWN0OiBSZTogRlc6IEktRCBBY3Rp b246IGRyYWZ0LXpob3UtbGktdnhsYW4tc29lLTAxLnR4dA0KDQoNCg0KT24gVHVlLCBNYXkgMjAs IDIwMTQgYXQgNToyOCBQTSwgWmhvdSwgSGFuIDxoemhvdThAZWJheS5jb208bWFpbHRvOmh6aG91 OEBlYmF5LmNvbT4+IHdyb3RlOg0KSGkgVG9tLA0KDQpUaGFua3MgZm9yIHlvdXIgY29tbWVudHMu DQpZZXMgVFNPL0xSTyB3aXRoIFZYTEFOIHN1cHBvcnQgc2hvdWxkIHByb3ZpZGUgc2ltaWxhciBv ciBldmVuIGJldHRlciBwZXJmb3JtYW5jZSBnYWlucywgYnV0IHRoZSBtZWNoYW5pc20gcHJvcG9z ZWQgYnkgdGhpcyBkcmFmdCBkZWNvdXBsZXMgb3ZlcmxheSBhbmQgdW5kZXJsYXksIGFuZCBpdCBp cyBoYXJkd2FyZSBpbmRlcGVuZGVudC4NClNlY29uZGx5LCBoYXJkd2FyZSBvZmZsb2FkaW5nIHVz dWFsbHkgc3VwcG9ydCBUQ1Agb25seSAoVFNPKS4gVGhlIG1lY2hhbmlzbSBoZXJlIGNhbiBoZWxw IG9uIGxhcmdlIFVEUCBwYWNrZXQgcGVyZm9ybWFuY2UsIGFsc28gdmVyaWZpZWQgYnkgdGhlIHBy b3RvdHlwZS4NCkJUVywgaG93IG1hbnkgdHlwZXMgb2Ygb2ZmLXRoZS1zaGVsZiBOSUMgc3VwcG9y dCBWWExBTiBvZmZsb2FkaW5nPyBBbnkgcGVyZm9ybWFuY2UgZGF0YT8NCg0KSFcgaXMgbm90IGEg cmVxdWlyZW1lbnQgZm9yIHRoaXMgb2ZmbG9hZGluZy4gVGhlcmUgaGFzIGJlZW4gYSBsb3Qgb2Yg d29yayByZWNlbnRseSB0byBnZXQgc29mdHdhcmUgdmFyaWFudHMgKEdTTyBhbmQgR1JPKSB3b3Jr aW5nIGluIExpbnV4IHdpdGggdHVubmVscy4gVGhlc2UgZG8gc2hvdyAyLTN4IHBlcmZvcm1hbmNl IGltcHJvdmVtZW50cy4gSFcgc3VwcG9ydCB3b3VsZCBiZSBtb3N0bHkgYW4gaW5jcmVtZW50YWwg aW1wcm92ZW1lbnQgdG8gdGhhdCwgb3IgYmVjb21lcyBpbnRlcmVzdGluZyBmb3IgT1MgYnlwYXNz IGxpa2UgaW4gU1ItSU9WLiBHU08vR1JPIGNhbiBiZSBwcmVzZW50ZWQgdG8gdGhlIGd1ZXN0IGFz IFRTTyBhbmQgTFJPIHNvIHRoYXQgaXQncyBwb3NzaWJsZSB0byBwbHVtYiB1c2Ugb2YgbGFyZ2Ug cGFja2V0cyBmcm9tIHRoZSBndWVzdCBhbGwgdGhlIHdheSB0byB0aGUgaG9zdCBkcml2ZXIuIEl0 IHNob3VsZCBhbHNvIGJlIHBvc3NpYmxlIHRvIHBsdW1iIHR3byBWTXMgb24gdGhlIHNhbWUgaG9z dCB0byBjb21tdW5pY2F0ZSB3aXRob3V0IHNlZ21lbnRhdGlvbiwgaS5lLiBvdXRwdXQgZnJvbSBU U08gb24gb25lIFZNIGJlY29tZXMgaW5wdXQgZm9yIGFub3RoZXIuDQoNClRoZSBhZHZhbnRhZ2Ug dGhhdCBJIHNlZSBpbiB5b3VyIGRyYWZ0IGlzIHRoYXQgaXQgYWxsb3dzIGFuIGludGVybWVkaWF0 ZSBkZXZpY2UgdG8gcGVyZm9ybSBzZWdtZW50YXRpb24vcmVhc3NlbWJseSBpbnN0ZWFkIG9mIGZy YWdtZW50YXRpb24uDQoNCkxpa2V3aXNlLCBzZXR0aW5nIGxhcmdlIE1UVSBvbiBvdmVybGF5IGlu dGVyZmFjZXMgYWNoaWV2ZXMgc2ltaWxhciByZXN1bHQsIGJ1dCBzdGlsbCB0aGUgb3ZlcmxheS91 bmRlcmxheSBkZWNvdXBsaW5nIGlzc3VlLiBJdCBpcyB1c3VhbGx5IGFkdmlzZWQgdGhhdCBvdmVy bGF5IE1UVSBpcyBzbGlnaHRseSBzbWFsbGVyIHRoYW4gdW5kZXJsYXkgdG8gYXZvaWQgaW5lZmZp Y2llbnQgZnJhZ21lbnRhdGlvbiBhZnRlciBhZGRpbmcgdGhlIG91dGVyIGhlYWRlciwgYnV0IHRv IGFjaGlldmUgcmVhbGx5IGhpZ2ggcGVyZm9ybWFuY2UgYmV0d2VlbiBWTXMsIGxhcmdlIE1UVSBp cyBwcmVmZXJyZWQuDQoNCkRlcGVuZHMgb24gdGhlIHBlcmZvcm1hbmNlIGRpbWVuc2lvbiB0byBi ZSBvcHRpbWl6ZWQuIExhcmdlciBNVFVzIGNvdWxkIGluY3JlYXNlIGxhdGVuY3kgb2YgaGlnaCBw cmlvcml0eSBzbWFsbCBwYWNrZXRzIGZvciBpbnN0YW5jZSAoSE9MIGJsb2NraW5nKSwgb3IgVURQ IGJhc2VkIGFwcGxpY2F0aW9uIG1pZ2h0IHRyeSB0byB1c2UgTVRVIHRvIGRlY2lkZSBob3cgbGFy Z2UgaXQgc2hvdWxkIHNlbmQgaXQncyBwYWNrZXRzIHRvIGF2b2lkIGZyYWdtZW50YXRpb24uDQoN CkFuZCBjb25zaWRlcmluZyBvdmVybGF5IDwtPiBwaHlzaWNhbCBjb25uZWN0aW9uLCBwYXRoIE1U VSBkaXNjb3ZlcnkgaXMgbm90IGFsd2F5cyB3b3JrLg0KVGhpcyBraW5kIG9mIGNvbmZpZ3VyYXRp b24gY29tcGxleGl0eSBhbmQgcGFpbi1wb2ludCBjYW4gYmUgcmVzb2x2ZWQgc2ltcGx5IGJ5IGRl Y291cGxpbmcgb3ZlcmxheSBhbmQgdW5kZXJsYXkgTVRVLCBhcyBzdWdnZXN0ZWQgYnkgdGhpcyBk cmFmdC4gSGVyZSBpcyBhbiBleGFtcGxlIG9mIGNvbmZpZ3VyYXRpb24gY29uZnVzaW9uOg0KaHR0 cDovL29wZW52c3dpdGNoLm9yZy9waXBlcm1haWwvZGlzY3Vzcy8yMDE0LU1heS8wMTM4OTguaHRt bA0KDQpJZGVhbGx5LCBhbGwgTlYgdHVubmVsIHByb3RvY29scyBzaG91bGQgc3VwcG9ydCBzaW1p bGFyIG1ldGFkYXRhLCB0aHVzIG92ZXJsYXkgc2VnbWVudGF0aW9uIGNhbiBiZSBvZmZsb2FkZWQg aG9wLWJ5LWhvcC4NCg0KSSB0aGluayB0aGlzIHdvdWxkIGJlIGFwcGxpY2FibGUgdG8gYWJvdXQg YWxsIHR1bm5lbCBwcm90b2NvbHMuIFRoZW4geW91IHdvdWxkIGFsc28gd2FudCB0byBkbyByZWFz c2VtYmx5IGF0IGVhY2ggaG9wPyBTb3VuZHMgZXhwZW5zaXZlLiBPbmNlIHlvdSBzZWdtZW50LCBJ IHRoaW5rIHlvdSdkIG9ubHkgb25seSB3YW50IHRvIHJlYXNzZW1ibGUgYXQgdGhlIGVuZCBob3N0 Lg0KDQpPbmUgaW1wb3J0YW50IHRoaW5nIHRvIGtlZXAgaW4gbWluZCwgYW5kIGEgaGFyZCBsZXNz b24gaW4gcmVhbCBkZXBsb3ltZW50IDotKSwgc2VnbWVudGF0aW9uIG9mZmxvYWQgaXMgb3Bwb3J0 dW5pc3RpYyBhbmQgdmVyeSBkZXBlbmRlbnQgb24gdGhlIGNvbmRpdGlvbnMgb2YgdGhlIHRyYWZm aWMuIEl0J3MgdmFsdWUgY2FuIGJlIGZsZWV0aW5nIGluIHJlYWwgd29ya2xvYWRzLiBGb3IgaW5z dGFuY2UgaW1hZ2luZSBhIGhvc3Qgd2l0aCBhIGxvdCBvZiBhY3RpdmUgaGlnaCB0aHJvdWdob3V0 IGNvbm5lY3Rpb25zIChsaWtlIGEgdmlkZW8gc2VydmluZykgd2hpY2ggaGl0cyBhIGhpY2N1cCBj YXVzaW5nIGFsbCBjb25nZXN0aW9ucyB3aW5kb3dzIHRvIHNocmluay4gSWYgdGhlIHN5c3RlbSBp cyBub3QgcHJvdmlzaW9uZWQgZm9yIHRoaXMgZXZlbnQgaXQgY2FuIGJlIHZlcnkgaGFyZCB0byBy ZWNvdmVyIChhIGxvdCBtb3JlIENQVSBpcyByZXF1aXJlZCB0byBhY2hpZXZlIHNhbWUgdGhyb3Vn aHB1dCkuIFRoaXMgZGlmZmVyZW50aWF0ZXMgYSBsYXJnZXIgTVRVIGZyb20gcmVseWluZyBvbiBz ZWdtZW50YXRpb24sIHRoZSBmb3JtZXIgb2ZmZXJzIG1vcmUgcHJlZGljdGFibGUgQ1BVIHNhdmlu Z3MuDQoNClRvbQ0KDQoNCkJlc3QgcmVnYXJkcywNCkhhbg0KDQpGcm9tOiBUb20gSGVyYmVydCBb bWFpbHRvOnRoZXJiZXJ0QGdvb2dsZS5jb208bWFpbHRvOnRoZXJiZXJ0QGdvb2dsZS5jb20+XQ0K U2VudDogV2VkbmVzZGF5LCBNYXkgMjEsIDIwMTQgMjo0NCBBTQ0KVG86IFpob3UsIEhhbg0KQ2M6 IG52bzNAaWV0Zi5vcmc8bWFpbHRvOm52bzNAaWV0Zi5vcmc+OyB0b2Zvb0BpZXRmLm9yZzxtYWls dG86dG9mb29AaWV0Zi5vcmc+OyBkcmFmdC1tYWhhbGluZ2FtLWR1dHQtZGNvcHMtdnhsYW5AdG9v bHMuaWV0Zi5vcmc8bWFpbHRvOmRyYWZ0LW1haGFsaW5nYW0tZHV0dC1kY29wcy12eGxhbkB0b29s cy5pZXRmLm9yZz47IGRyYWZ0LXpob3UtbGktdnhsYW4tc29lQHRvb2xzLmlldGYub3JnPG1haWx0 bzpkcmFmdC16aG91LWxpLXZ4bGFuLXNvZUB0b29scy5pZXRmLm9yZz47IEVyaWsgTm9yZG1hcmsN ClN1YmplY3Q6IFJlOiBGVzogSS1EIEFjdGlvbjogZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDEu dHh0DQoNCkhpIFpvdSwgYSBjb3VwbGUgb2YgcXVlc3Rpb25zIGlubGluZS4NCg0KT24gTW9uLCBN YXkgMTksIDIwMTQgYXQgODowMSBQTSwgWmhvdSwgSGFuIDxoemhvdThAZWJheS5jb208bWFpbHRv Omh6aG91OEBlYmF5LmNvbT4+IHdyb3RlOg0KSGksDQoNCldlIGhhdmUgdXBkYXRlZCB0aGUgVlhM QU4tU09FIGRyYWZ0IGFjY29yZGluZyB0byBlYXJsaWVyIGNvbW1lbnRzLiBOb3cgaXQgaXMgZnVs bHkgY29tcGF0aWJsZSB3aXRoIFZYTEFOLUdQRS4gQW5kIHNvbWUgZXhhbXBsZXMgYXJlIGFkZGVk IGZvciBiZXR0ZXIgdW5kZXJzdGFuZGluZw0KDQpBIHByb3RvdHlwZSBpcyBhbHNvIGltcGxlbWVu dGVkIGhlcmUgKHBhdGNoIGJhc2VkIG9uIE9wZW4gdlN3aXRjaCk6DQpodHRwczovL2dpdGh1Yi5j b20vaHpob3U4L29wZW52c3dpdGNoL2NvbW1pdC85YTdkZWI4YjQzMmNlODNhOWMwOWQ3ZDRmZjg1 ZmEwNTBmN2RkMmJlDQoNCm5ldHBlcmYgVENQX1NUUkVBTSB0ZXN0IHJlc3VsdCBiZXR3ZWVuIGEg cGFpcnMgb2YgVk1zIG9uIGhvc3RzIHdpdGggMTBHIGludGVyZmFjZXM6DQoNCkJlZm9yZSB0aGUg Y2hhbmdlOiAyLjYyIEdiaXRzL3NlYw0KQWZ0ZXIgdGhlIGNoYW5nZTogNi42OCBHYml0cy9zZWMN ClNwZWVkdXAgaXMgfjI1MCUuDQpDYW4geW91IHByb3ZpZGUgc29tZSBtb3JlIGRldGFpbHMgb24g dGhpcyBiZW5lZml0PyBJdCBzZWVtcyBsaWtlIHBsYWluIFRTTy9MUk8gdGhhdCB1bmRlcnN0YW5k cyBlbmNhcHN1bGF0aW9uIHNob3VsZCBwcm92aWRlIHNpbWlsYXIgYmVuZWZpdHMgd2hlbiBnb2lu ZyBiZXR3ZWVuIGhvc3RzLg0KDQoNClRoZSBwYXRjaCBhdHRyYWN0ZWQgc29tZSBpbnRlcmVzdHMg aW4gT1ZTIGNvbW11bml0eSwgYnV0IHNpbmNlIHRoaXMgUkZDIGRyYWZ0IGlzIGluIHZlcnkgZWFy bHkgc3RhZ2Ugc28gaXQgaXMgcmVnYXJkZWQgYXMgaW5hcHByb3ByaWF0ZSBieSBKZXNzZSB0byBh cHBseSB0aGUgY2hhbmdlIHRvIE9WUyB0cmVlLg0KVGhlIGRpc2N1c3MgbWFpbC10aHJlYWQ6DQpo dHRwOi8vb3BlbnZzd2l0Y2gub3JnL3BpcGVybWFpbC9kaXNjdXNzLzIwMTQtTWF5LzAxMzk4MS5o dG1sDQpodHRwOi8vb3BlbnZzd2l0Y2gub3JnL3BpcGVybWFpbC9kaXNjdXNzLzIwMTQtTWF5LzAx Mzg5OC5odG1sDQoNClNvIHdlIHdvdWxkIGxpa2UgdG8gcmVxdWVzdCBhIHJldmlldyBoZXJlIGJ5 IE5WTzMvVE9GT08gZ3JvdXBzIGFuZCBWWExBTiBhdXRob3JzOiBpcyB0aGlzIFZYTEFOIGV4dGVu c2lvbiBpcyB3b3J0aCBmb3JtYWxseSBwdXQgaW50byBWWExBTiBhcyBhIHN0YW5kYXJkLCBzbyB0 aGF0IG1vcmUgcGVvcGxlIGNhbiBiZW5lZml0IGZyb20gaXQ/DQpDb3VsZCB5b3UgZ2V0IHRoZSBz YW1lIGVmZmVjdCBieSBzZXR0aW5nIGxhcmdlciBNVFVzIG9uIHRoZSBvdmVybGF5IG5ldHdvcmsg aW50ZXJmYWNlIGFuZCByZWx5aW5nIGluIHBhdGggTVRVIGRpc2NvdmVyeSB3aGVuIGdvaW5nIG92 ZXIgcGh5c2ljYWwgbmV0d29yaz8NCg0KQmVzdCByZWdhcmRzLA0KSGFuDQoNCi0tLS0tT3JpZ2lu YWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBJLUQtQW5ub3VuY2UgW21haWx0bzppLWQtYW5ub3VuY2Ut Ym91bmNlc0BpZXRmLm9yZzxtYWlsdG86aS1kLWFubm91bmNlLWJvdW5jZXNAaWV0Zi5vcmc+XSBP biBCZWhhbGYgT2YgaW50ZXJuZXQtZHJhZnRzQGlldGYub3JnPG1haWx0bzppbnRlcm5ldC1kcmFm dHNAaWV0Zi5vcmc+DQpTZW50OiBGcmlkYXksIE1heSAwMiwgMjAxNCA4OjA5IFBNDQpUbzogaS1k LWFubm91bmNlQGlldGYub3JnPG1haWx0bzppLWQtYW5ub3VuY2VAaWV0Zi5vcmc+DQpTdWJqZWN0 OiBJLUQgQWN0aW9uOiBkcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMS50eHQNCg0KDQpBIE5ldyBJ bnRlcm5ldC1EcmFmdCBpcyBhdmFpbGFibGUgZnJvbSB0aGUgb24tbGluZSBJbnRlcm5ldC1EcmFm dHMgZGlyZWN0b3JpZXMuDQoNCg0KICAgICAgICBUaXRsZSAgICAgICAgICAgOiBTZWdtZW50YXRp b24gT2ZmbG9hZGluZyBFeHRlbnNpb24gZm9yIFZYTEFODQogICAgICAgIEF1dGhvcnMgICAgICAg ICA6IEhhbiBaaG91DQogICAgICAgICAgICAgICAgICAgICAgICAgIENoZW5neXVhbiBMaQ0KICAg ICAgICBGaWxlbmFtZSAgICAgICAgOiBkcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMS50eHQNCiAg ICAgICAgUGFnZXMgICAgICAgICAgIDogMTMNCiAgICAgICAgRGF0ZSAgICAgICAgICAgIDogMjAx NC0wNS0wMg0KDQpBYnN0cmFjdDoNCiAgIFNlZ21lbnRhdGlvbiBvZmZsb2FkaW5nIGlzIG5vd2Fk YXlzIGNvbW1vbiBpbiBuZXR3b3JrIHN0YWNrDQogICBpbXBsZW1lbnRhdGlvbiBhbmQgd2VsbCBz dXBwb3J0ZWQgYnkgcGFyYS12aXJ0dWFsaXplZCBuZXR3b3JrIGRldmljZQ0KICAgZHJpdmVycyBm b3IgdmlydHVhbCBtYWNoaW5lIChWTSlzLiBUaGlzIGRyYWZ0IGRlc2NyaWJlcyBhbiBleHRlbnNp b24NCiAgIHRvIFZpcnR1YWwgZVh0ZW5zaWJsZSBMb2NhbCBBcmVhIE5ldHdvcmsgKFZYTEFOKSBz byB0aGF0IHNlZ21lbnRhdGlvbg0KICAgY2FuIGJlIGRlY291cGxlZCBmcm9tIHBoeXNpY2FsL3Vu ZGVybGF5IG5ldHdvcmtzIGFuZCBvZmZsb2FkZWQNCiAgIGZ1cnRoZXIgdG8gdGhlIHJlbW90ZSBl bmQtcG9pbnQgdGh1cyBpbXByb3ZpbmcgZGF0YS1wbGFuZSBwZXJmb3JtYW5jZQ0KICAgZm9yIFZN cyBydW5uaW5nIG9uIHRvcCBvZiBvdmVybGF5IG5ldHdvcmtzLg0KDQoNClRoZSBJRVRGIGRhdGF0 cmFja2VyIHN0YXR1cyBwYWdlIGZvciB0aGlzIGRyYWZ0IGlzOg0KaHR0cHM6Ly9kYXRhdHJhY2tl ci5pZXRmLm9yZy9kb2MvZHJhZnQtemhvdS1saS12eGxhbi1zb2UvDQoNClRoZXJlJ3MgYWxzbyBh IGh0bWxpemVkIHZlcnNpb24gYXZhaWxhYmxlIGF0Og0KaHR0cDovL3Rvb2xzLmlldGYub3JnL2h0 bWwvZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDENCg0KQSBkaWZmIGZyb20gdGhlIHByZXZpb3Vz IHZlcnNpb24gaXMgYXZhaWxhYmxlIGF0Og0KaHR0cDovL3d3dy5pZXRmLm9yZy9yZmNkaWZmP3Vy bDI9ZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDENCg0KDQpQbGVhc2Ugbm90ZSB0aGF0IGl0IG1h eSB0YWtlIGEgY291cGxlIG9mIG1pbnV0ZXMgZnJvbSB0aGUgdGltZSBvZiBzdWJtaXNzaW9uDQp1 bnRpbCB0aGUgaHRtbGl6ZWQgdmVyc2lvbiBhbmQgZGlmZiBhcmUgYXZhaWxhYmxlIGF0IHRvb2xz LmlldGYub3JnPGh0dHA6Ly90b29scy5pZXRmLm9yZz4uDQoNCkludGVybmV0LURyYWZ0cyBhcmUg YWxzbyBhdmFpbGFibGUgYnkgYW5vbnltb3VzIEZUUCBhdDoNCmZ0cDovL2Z0cC5pZXRmLm9yZy9p bnRlcm5ldC1kcmFmdHMvDQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fDQpJLUQtQW5ub3VuY2UgbWFpbGluZyBsaXN0DQpJLUQtQW5ub3VuY2VAaWV0Zi5v cmc8bWFpbHRvOkktRC1Bbm5vdW5jZUBpZXRmLm9yZz4NCmh0dHBzOi8vd3d3LmlldGYub3JnL21h aWxtYW4vbGlzdGluZm8vaS1kLWFubm91bmNlDQpJbnRlcm5ldC1EcmFmdCBkaXJlY3Rvcmllczog aHR0cDovL3d3dy5pZXRmLm9yZy9zaGFkb3cuaHRtbA0Kb3IgZnRwOi8vZnRwLmlldGYub3JnL2ll dGYvMXNoYWRvdy1zaXRlcy50eHQNCg0KDQo= --_000_9F56174078B48B459268EFF1DAB66B1A109C7F16DENEXDDAS32corp_ Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IlByb2dJZCIg Y29udGVudD0iV29yZC5Eb2N1bWVudCI+DQo8bWV0YSBuYW1lPSJHZW5lcmF0b3IiIGNvbnRlbnQ9 Ik1pY3Jvc29mdCBXb3JkIDE0Ij4NCjxtZXRhIG5hbWU9Ik9yaWdpbmF0b3IiIGNvbnRlbnQ9Ik1p Y3Jvc29mdCBXb3JkIDE0Ij4NCjxsaW5rIHJlbD0iRmlsZS1MaXN0IiBocmVmPSJjaWQ6ZmlsZWxp c3QueG1sQDAxQ0Y3NEU0LkNERjA4MDUwIj48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOk9m ZmljZURvY3VtZW50U2V0dGluZ3M+DQo8bzpBbGxvd1BORy8+DQo8L286T2ZmaWNlRG9jdW1lbnRT ZXR0aW5ncz4NCjwveG1sPjwhW2VuZGlmXS0tPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPg0KPHc6 V29yZERvY3VtZW50Pg0KPHc6Wm9vbT4xMTA8L3c6Wm9vbT4NCjx3OlNwZWxsaW5nU3RhdGU+Q2xl YW48L3c6U3BlbGxpbmdTdGF0ZT4NCjx3OlRyYWNrTW92ZXMvPg0KPHc6VHJhY2tGb3JtYXR0aW5n Lz4NCjx3OkVudmVsb3BlVmlzLz4NCjx3OlZhbGlkYXRlQWdhaW5zdFNjaGVtYXMvPg0KPHc6U2F2 ZUlmWE1MSW52YWxpZD5mYWxzZTwvdzpTYXZlSWZYTUxJbnZhbGlkPg0KPHc6SWdub3JlTWl4ZWRD b250ZW50PmZhbHNlPC93Oklnbm9yZU1peGVkQ29udGVudD4NCjx3OkFsd2F5c1Nob3dQbGFjZWhv bGRlclRleHQ+ZmFsc2U8L3c6QWx3YXlzU2hvd1BsYWNlaG9sZGVyVGV4dD4NCjx3OkRvTm90UHJv bW90ZVFGLz4NCjx3OkxpZFRoZW1lT3RoZXI+RU4tVVM8L3c6TGlkVGhlbWVPdGhlcj4NCjx3Okxp ZFRoZW1lQXNpYW4+WkgtQ048L3c6TGlkVGhlbWVBc2lhbj4NCjx3OkxpZFRoZW1lQ29tcGxleFNj cmlwdD5YLU5PTkU8L3c6TGlkVGhlbWVDb21wbGV4U2NyaXB0Pg0KPHc6Q29tcGF0aWJpbGl0eT4N Cjx3OkRvTm90RXhwYW5kU2hpZnRSZXR1cm4vPg0KPHc6QnJlYWtXcmFwcGVkVGFibGVzLz4NCjx3 OlNwbGl0UGdCcmVha0FuZFBhcmFNYXJrLz4NCjx3OkVuYWJsZU9wZW5UeXBlS2VybmluZy8+DQo8 dzpVc2VGRUxheW91dC8+DQo8L3c6Q29tcGF0aWJpbGl0eT4NCjxtOm1hdGhQcj4NCjxtOm1hdGhG b250IG06dmFsPSJDYW1icmlhIE1hdGgiLz4NCjxtOmJya0JpbiBtOnZhbD0iYmVmb3JlIi8+DQo8 bTpicmtCaW5TdWIgbTp2YWw9IiYjNDU7LSIvPg0KPG06c21hbGxGcmFjIG06dmFsPSJvZmYiLz4N CjxtOmRpc3BEZWYvPg0KPG06bE1hcmdpbiBtOnZhbD0iMCIvPg0KPG06ck1hcmdpbiBtOnZhbD0i MCIvPg0KPG06ZGVmSmMgbTp2YWw9ImNlbnRlckdyb3VwIi8+DQo8bTp3cmFwSW5kZW50IG06dmFs PSIxNDQwIi8+DQo8bTppbnRMaW0gbTp2YWw9InN1YlN1cCIvPg0KPG06bmFyeUxpbSBtOnZhbD0i dW5kT3ZyIi8+DQo8L206bWF0aFByPjwvdzpXb3JkRG9jdW1lbnQ+DQo8L3htbD48IVtlbmRpZl0t LT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjx3OkxhdGVudFN0eWxlcyBEZWZMb2NrZWRTdGF0 ZT0iZmFsc2UiIERlZlVuaGlkZVdoZW5Vc2VkPSJ0cnVlIiBEZWZTZW1pSGlkZGVuPSJ0cnVlIiBE ZWZRRm9ybWF0PSJmYWxzZSIgRGVmUHJpb3JpdHk9Ijk5IiBMYXRlbnRTdHlsZUNvdW50PSIyNjci Pg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIwIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJOb3Jt YWwiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0idHJ1ZSIgTmFtZT0i aGVhZGluZyAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9Ijki IFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgMiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tl ZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJoZWFkaW5nIDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgUUZvcm1hdD0idHJ1 ZSIgTmFtZT0iaGVhZGluZyA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjkiIFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgNSIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJoZWFk aW5nIDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iOSIgUUZv cm1hdD0idHJ1ZSIgTmFtZT0iaGVhZGluZyA3Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjkiIFFGb3JtYXQ9InRydWUiIE5hbWU9ImhlYWRpbmcgOCIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI5IiBRRm9ybWF0PSJ0cnVlIiBO YW1lPSJoZWFkaW5nIDkiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iMzkiIE5hbWU9InRvYyAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjM5IiBOYW1lPSJ0b2MgMiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2Ui IFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iMzkiIE5hbWU9InRvYyA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2Vk PSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBOYW1lPSJ0b2MgNSIvPg0KPHc6THNkRXhjZXB0aW9uIExv Y2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDYiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzkiIE5hbWU9InRvYyA3Ii8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBOYW1lPSJ0b2MgOCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzOSIgTmFtZT0idG9jIDkiLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzUiIFFGb3JtYXQ9InRydWUi IE5hbWU9ImNhcHRpb24iLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iMTAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9 InRydWUiIE5hbWU9IlRpdGxlIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjEiIE5hbWU9IkRlZmF1bHQgUGFyYWdyYXBoIEZvbnQiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMTEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVX aGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9IlN1YnRpdGxlIi8+DQo8dzpMc2RF eGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjIyIiBTZW1pSGlkZGVuPSJmYWxzZSIg VW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJTdHJvbmciLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMjAiIFNlbWlIaWRkZW49ImZh bHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9IkVtcGhhc2lz Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjU5IiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJUYWJsZSBHcmlkIi8+DQo8 dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJQbGFjZWhvbGRlciBUZXh0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIFFGb3Jt YXQ9InRydWUiIE5hbWU9Ik5vIFNwYWNpbmciLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFs c2UiIE5hbWU9IkxpZ2h0IFNoYWRpbmciLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkxpZ2h0IExpc3QiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNjIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 IkxpZ2h0IEdyaWQiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBTaGFkaW5nIDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjQiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBTaGFkaW5nIDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NjUiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1 bSBMaXN0IDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjYi IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBM aXN0IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjciIFNl bWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlk IDEiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjgiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDIi Lz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjkiIFNlbWlIaWRk ZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkRhcmsgTGlzdCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCIvPg0KPHc6THNkRXhj ZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgR3JpZCIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlk ZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgU2hhZGluZyBBY2NlbnQgMSIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgTGlzdCBBY2NlbnQgMSIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MiIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgR3JpZCBBY2NlbnQgMSIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MyIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRpbmcgMSBBY2Nl bnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NCIgU2Vt aUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRp bmcgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIExpc3QgMSBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iUmV2aXNpb24iLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iMzQiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIFFGb3JtYXQ9InRydWUiIE5hbWU9Ikxpc3QgUGFyYWdyYXBoIi8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjI5IiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJRdW90ZSIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSIzMCIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0idHJ1ZSIgTmFtZT0iSW50ZW5z ZSBRdW90ZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NiIg U2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIExp c3QgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIEdyaWQgMSBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFBy aW9yaXR5PSI2OCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFt ZT0iTWVkaXVtIEdyaWQgMiBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFs c2UiIFByaW9yaXR5PSI2OSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxz ZSIgTmFtZT0iTWVkaXVtIEdyaWQgMyBBY2NlbnQgMSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tl ZD0iZmFsc2UiIFByaW9yaXR5PSI3MCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2Vk PSJmYWxzZSIgTmFtZT0iRGFyayBMaXN0IEFjY2VudCAxIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9j a2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVz ZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBTaGFkaW5nIEFjY2VudCAxIi8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcyIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5o aWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBMaXN0IEFjY2VudCAxIi8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjczIiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBHcmlkIEFjY2VudCAxIi8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYwIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBTaGFkaW5nIEFjY2Vu dCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYxIiBTZW1p SGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBMaXN0IEFj Y2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYyIiBT ZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJMaWdodCBHcmlk IEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjYz IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0g U2hhZGluZyAxIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjY0IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJNZWRpdW0gU2hhZGluZyAyIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjY1IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAxIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9j a2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY2IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVz ZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAyIEFjY2VudCAyIi8+DQo8dzpMc2RFeGNlcHRp b24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY3IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRl V2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAxIEFjY2VudCAyIi8+DQo8dzpMc2RF eGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY4IiBTZW1pSGlkZGVuPSJmYWxzZSIg VW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAyIEFjY2VudCAyIi8+DQo8 dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY5IiBTZW1pSGlkZGVuPSJm YWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAzIEFjY2VudCAy Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcwIiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJEYXJrIExpc3QgQWNjZW50 IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzEiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIFNoYWRp bmcgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0i NzIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9y ZnVsIExpc3QgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNzMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 IkNvbG9yZnVsIEdyaWQgQWNjZW50IDIiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkxpZ2h0IFNoYWRpbmcgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9 ImZhbHNlIiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0i ZmFsc2UiIE5hbWU9IkxpZ2h0IExpc3QgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2Nr ZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNl ZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IEdyaWQgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDEgQWNjZW50IDMiLz4NCjx3OkxzZEV4 Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjQiIFNlbWlIaWRkZW49ImZhbHNlIiBV bmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDIgQWNjZW50IDMiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjUiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBMaXN0IDEgQWNjZW50 IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjYiIFNlbWlI aWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBMaXN0IDIg QWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjci IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBH cmlkIDEgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0 eT0iNjgiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1l ZGl1bSBHcmlkIDIgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQ cmlvcml0eT0iNjkiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5h bWU9Ik1lZGl1bSBHcmlkIDMgQWNjZW50IDMiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZh bHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFs c2UiIE5hbWU9IkRhcmsgTGlzdCBBY2NlbnQgMyIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0i ZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJm YWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyBBY2NlbnQgMyIvPg0KPHc6THNkRXhjZXB0aW9u IExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdo ZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCBBY2NlbnQgMyIvPg0KPHc6THNkRXhj ZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVu aGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgR3JpZCBBY2NlbnQgMyIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MCIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgU2hhZGluZyBBY2NlbnQgNCIv Pg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MSIgU2VtaUhpZGRl bj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgTGlzdCBBY2NlbnQg NCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MiIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTGlnaHQgR3JpZCBBY2Nl bnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2MyIgU2Vt aUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIFNoYWRp bmcgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSI2NCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVk aXVtIFNoYWRpbmcgMiBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2Ui IFByaW9yaXR5PSI2NSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIg TmFtZT0iTWVkaXVtIExpc3QgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0i ZmFsc2UiIFByaW9yaXR5PSI2NiIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJm YWxzZSIgTmFtZT0iTWVkaXVtIExpc3QgMiBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0aW9uIExv Y2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2NyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5V c2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMSBBY2NlbnQgNCIvPg0KPHc6THNkRXhjZXB0 aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2OCIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlk ZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMiBBY2NlbnQgNCIvPg0KPHc6THNk RXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI2OSIgU2VtaUhpZGRlbj0iZmFsc2Ui IFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iTWVkaXVtIEdyaWQgMyBBY2NlbnQgNCIvPg0K PHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MCIgU2VtaUhpZGRlbj0i ZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iRGFyayBMaXN0IEFjY2VudCA0Ii8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcxIiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBTaGFkaW5nIEFj Y2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjcyIiBT ZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xvcmZ1bCBM aXN0IEFjY2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9 IjczIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJDb2xv cmZ1bCBHcmlkIEFjY2VudCA0Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJp b3JpdHk9IjYwIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1l PSJMaWdodCBTaGFkaW5nIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxz ZSIgUHJpb3JpdHk9IjYxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNl IiBOYW1lPSJMaWdodCBMaXN0IEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjYyIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBOYW1lPSJMaWdodCBHcmlkIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2Vk PSJmYWxzZSIgUHJpb3JpdHk9IjYzIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9 ImZhbHNlIiBOYW1lPSJNZWRpdW0gU2hhZGluZyAxIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRp b24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY0IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRl V2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gU2hhZGluZyAyIEFjY2VudCA1Ii8+DQo8dzpM c2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY1IiBTZW1pSGlkZGVuPSJmYWxz ZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAxIEFjY2VudCA1Ii8+ DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY2IiBTZW1pSGlkZGVu PSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gTGlzdCAyIEFjY2Vu dCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY3IiBTZW1p SGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0gR3JpZCAx IEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjY4 IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJNZWRpdW0g R3JpZCAyIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3Jp dHk9IjY5IiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBOYW1lPSJN ZWRpdW0gR3JpZCAzIEFjY2VudCA1Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIg UHJpb3JpdHk9IjcwIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBO YW1lPSJEYXJrIExpc3QgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNzEiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9IkNvbG9yZnVsIFNoYWRpbmcgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2Nr ZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzIiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNl ZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIExpc3QgQWNjZW50IDUiLz4NCjx3OkxzZEV4Y2VwdGlv biBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzMiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVX aGVuVXNlZD0iZmFsc2UiIE5hbWU9IkNvbG9yZnVsIEdyaWQgQWNjZW50IDUiLz4NCjx3OkxzZEV4 Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjAiIFNlbWlIaWRkZW49ImZhbHNlIiBV bmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IFNoYWRpbmcgQWNjZW50IDYiLz4NCjx3 OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjEiIFNlbWlIaWRkZW49ImZh bHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IExpc3QgQWNjZW50IDYiLz4N Cjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjIiIFNlbWlIaWRkZW49 ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkxpZ2h0IEdyaWQgQWNjZW50IDYi Lz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjMiIFNlbWlIaWRk ZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBTaGFkaW5nIDEg QWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjQi IFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBT aGFkaW5nIDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlv cml0eT0iNjUiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9 Ik1lZGl1bSBMaXN0IDEgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNl IiBQcmlvcml0eT0iNjYiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0iZmFsc2Ui IE5hbWU9Ik1lZGl1bSBMaXN0IDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBMb2NrZWQ9 ImZhbHNlIiBQcmlvcml0eT0iNjciIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVuVXNlZD0i ZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDEgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2VwdGlvbiBM b2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjgiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhpZGVXaGVu VXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDIgQWNjZW50IDYiLz4NCjx3OkxzZEV4Y2Vw dGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNjkiIFNlbWlIaWRkZW49ImZhbHNlIiBVbmhp ZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9Ik1lZGl1bSBHcmlkIDMgQWNjZW50IDYiLz4NCjx3Okxz ZEV4Y2VwdGlvbiBMb2NrZWQ9ImZhbHNlIiBQcmlvcml0eT0iNzAiIFNlbWlIaWRkZW49ImZhbHNl IiBVbmhpZGVXaGVuVXNlZD0iZmFsc2UiIE5hbWU9IkRhcmsgTGlzdCBBY2NlbnQgNiIvPg0KPHc6 THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MSIgU2VtaUhpZGRlbj0iZmFs c2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgU2hhZGluZyBBY2NlbnQg NiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MiIgU2VtaUhp ZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwgTGlzdCBB Y2NlbnQgNiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5PSI3MyIg U2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgTmFtZT0iQ29sb3JmdWwg R3JpZCBBY2NlbnQgNiIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9yaXR5 PSIxOSIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1hdD0i dHJ1ZSIgTmFtZT0iU3VidGxlIEVtcGhhc2lzIi8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJm YWxzZSIgUHJpb3JpdHk9IjIxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZh bHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJJbnRlbnNlIEVtcGhhc2lzIi8+DQo8dzpMc2RFeGNl cHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjMxIiBTZW1pSGlkZGVuPSJmYWxzZSIgVW5o aWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJTdWJ0bGUgUmVmZXJlbmNl Ii8+DQo8dzpMc2RFeGNlcHRpb24gTG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjMyIiBTZW1pSGlk ZGVuPSJmYWxzZSIgVW5oaWRlV2hlblVzZWQ9ImZhbHNlIiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJJ bnRlbnNlIFJlZmVyZW5jZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFsc2UiIFByaW9y aXR5PSIzMyIgU2VtaUhpZGRlbj0iZmFsc2UiIFVuaGlkZVdoZW5Vc2VkPSJmYWxzZSIgUUZvcm1h dD0idHJ1ZSIgTmFtZT0iQm9vayBUaXRsZSIvPg0KPHc6THNkRXhjZXB0aW9uIExvY2tlZD0iZmFs c2UiIFByaW9yaXR5PSIzNyIgTmFtZT0iQmlibGlvZ3JhcGh5Ii8+DQo8dzpMc2RFeGNlcHRpb24g TG9ja2VkPSJmYWxzZSIgUHJpb3JpdHk9IjM5IiBRRm9ybWF0PSJ0cnVlIiBOYW1lPSJUT0MgSGVh ZGluZyIvPg0KPC93OkxhdGVudFN0eWxlcz4NCjwveG1sPjwhW2VuZGlmXS0tPjxzdHlsZT48IS0t DQovKiBGb250IERlZmluaXRpb25zICovDQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OuWui+S9 kzsNCglwYW5vc2UtMToyIDEgNiAwIDMgMSAxIDEgMSAxOw0KCW1zby1mb250LWFsdDpTaW1TdW47 DQoJbXNvLWZvbnQtY2hhcnNldDoxMzQ7DQoJbXNvLWdlbmVyaWMtZm9udC1mYW1pbHk6YXV0bzsN Cgltc28tZm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6MyA2ODA0NjAy ODggMjIgMCAyNjIxNDUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OuWui+S9kzsNCglw YW5vc2UtMToyIDEgNiAwIDMgMSAxIDEgMSAxOw0KCW1zby1mb250LWFsdDpTaW1TdW47DQoJbXNv LWZvbnQtY2hhcnNldDoxMzQ7DQoJbXNvLWdlbmVyaWMtZm9udC1mYW1pbHk6YXV0bzsNCgltc28t Zm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6MyA2ODA0NjAyODggMjIg MCAyNjIxNDUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3Nl LTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7DQoJbXNvLWZvbnQtYWx0OiJUaW1lcyBOZXcgUm9tYW4i Ow0KCW1zby1mb250LWNoYXJzZXQ6MDsNCgltc28tZ2VuZXJpYy1mb250LWZhbWlseTpzd2lzczsN Cgltc28tZm9udC1waXRjaDp2YXJpYWJsZTsNCgltc28tZm9udC1zaWduYXR1cmU6LTUzNjg3MDE0 NSAxMDczNzg2MTExIDEgMCA0MTUgMDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OlRhaG9t YTsNCglwYW5vc2UtMToyIDExIDYgNCAzIDUgNCA0IDIgNDsNCgltc28tZm9udC1hbHQ6IlRpbWVz IE5ldyBSb21hbiI7DQoJbXNvLWZvbnQtY2hhcnNldDowOw0KCW1zby1nZW5lcmljLWZvbnQtZmFt aWx5OnN3aXNzOw0KCW1zby1mb250LXBpdGNoOnZhcmlhYmxlOw0KCW1zby1mb250LXNpZ25hdHVy ZTotNTIwMDgxNjY1IC0xMDczNzE3MTU3IDQxIDAgNjYwNDcgMDt9DQpAZm9udC1mYWNlDQoJe2Zv bnQtZmFtaWx5OiJcQOWui+S9kyI7DQoJcGFub3NlLTE6MiAxIDYgMCAzIDEgMSAxIDEgMTsNCglt c28tZm9udC1hbHQ6Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/Pz8/PzsNCgltc28tZm9udC1j aGFyc2V0OjEzNDsNCgltc28tZ2VuZXJpYy1mb250LWZhbWlseTphdXRvOw0KCW1zby1mb250LXBp dGNoOnZhcmlhYmxlOw0KCW1zby1mb250LXNpZ25hdHVyZTozIDY4MDQ2MDI4OCAyMiAwIDI2MjE0 NSAwO30NCi8qIFN0eWxlIERlZmluaXRpb25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFs LCBkaXYuTXNvTm9ybWFsDQoJe21zby1zdHlsZS11bmhpZGU6bm87DQoJbXNvLXN0eWxlLXFmb3Jt YXQ6eWVzOw0KCW1zby1zdHlsZS1wYXJlbnQ6IiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90 dG9tOi4wMDAxcHQ7DQoJbXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZTox Mi4wcHQ7DQoJZm9udC1mYW1pbHk65a6L5L2TOw0KCW1zby1iaWRpLWZvbnQtZmFtaWx5OuWui+S9 kzt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1ub3Nob3c6eWVzOw0K CW1zby1zdHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1 bmRlcmxpbmU7DQoJdGV4dC11bmRlcmxpbmU6c2luZ2xlO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29I eXBlcmxpbmtGb2xsb3dlZA0KCXttc28tc3R5bGUtbm9zaG93OnllczsNCgltc28tc3R5bGUtcHJp b3JpdHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7DQoJ dGV4dC11bmRlcmxpbmU6c2luZ2xlO30NCnAuTXNvQWNldGF0ZSwgbGkuTXNvQWNldGF0ZSwgZGl2 Lk1zb0FjZXRhdGUNCgl7bXNvLXN0eWxlLW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxlLXByaW9yaXR5 Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJCYWxsb29uIFRleHQgQ2hhciI7DQoJbWFyZ2luOjBjbTsN CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJbXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0K CWZvbnQtc2l6ZTo4LjBwdDsNCglmb250LWZhbWlseTrlrovkvZM7DQoJbXNvLWJpZGktZm9udC1m YW1pbHk65a6L5L2TO30NCnAuTXNvTGlzdFBhcmFncmFwaCwgbGkuTXNvTGlzdFBhcmFncmFwaCwg ZGl2Lk1zb0xpc3RQYXJhZ3JhcGgNCgl7bXNvLXN0eWxlLXByaW9yaXR5OjM0Ow0KCW1zby1zdHls ZS11bmhpZGU6bm87DQoJbXNvLXN0eWxlLXFmb3JtYXQ6eWVzOw0KCW1hcmdpbjowY207DQoJbWFy Z2luLWJvdHRvbTouMDAwMXB0Ow0KCXRleHQtaW5kZW50OjIxLjBwdDsNCgltc28tY2hhci1pbmRl bnQtY291bnQ6Mi4wOw0KCW1zby1wYWdpbmF0aW9uOndpZG93LW9ycGhhbjsNCglmb250LXNpemU6 MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OuWui+S9kzsNCgltc28tYmlkaS1mb250LWZhbWlseTrlrovk vZM7fQ0Kc3Bhbi5FbWFpbFN0eWxlMTcNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7 DQoJbXNvLXN0eWxlLW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxlLXVuaGlkZTpubzsNCgltc28tYW5z aS1mb250LXNpemU6MTAuNXB0Ow0KCW1zby1iaWRpLWZvbnQtc2l6ZToxMS4wcHQ7DQoJZm9udC1m YW1pbHk6IkNhbGlicmkiLCJzYW5zLXNlcmlmIjsNCgltc28tYXNjaWktZm9udC1mYW1pbHk6Q2Fs aWJyaTsNCgltc28tZmFyZWFzdC1mb250LWZhbWlseTrlrovkvZM7DQoJbXNvLWhhbnNpLWZvbnQt ZmFtaWx5OkNhbGlicmk7DQoJbXNvLWJpZGktZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiI7 DQoJY29sb3I6IzFGNDk3RDt9DQpzcGFuLkJhbGxvb25UZXh0Q2hhcg0KCXttc28tc3R5bGUtbmFt ZToiQmFsbG9vbiBUZXh0IENoYXIiOw0KCW1zby1zdHlsZS1ub3Nob3c6eWVzOw0KCW1zby1zdHls ZS1wcmlvcml0eTo5OTsNCgltc28tc3R5bGUtdW5oaWRlOm5vOw0KCW1zby1zdHlsZS1sb2NrZWQ6 eWVzOw0KCW1zby1zdHlsZS1saW5rOiJCYWxsb29uIFRleHQiOw0KCW1zby1hbnNpLWZvbnQtc2l6 ZTo4LjBwdDsNCgltc28tYmlkaS1mb250LXNpemU6OC4wcHQ7DQoJZm9udC1mYW1pbHk65a6L5L2T Ow0KCW1zby1hc2NpaS1mb250LWZhbWlseTrlrovkvZM7DQoJbXNvLWZhcmVhc3QtZm9udC1mYW1p bHk65a6L5L2TOw0KCW1zby1oYW5zaS1mb250LWZhbWlseTrlrovkvZM7DQoJbXNvLWJpZGktZm9u dC1mYW1pbHk65a6L5L2TOw0KCW1zby1mb250LWtlcm5pbmc6MHB0O30NCnNwYW4uU3BlbGxFDQoJ e21zby1zdHlsZS1uYW1lOiIiOw0KCW1zby1zcGwtZTp5ZXM7fQ0KLk1zb0NocERlZmF1bHQNCgl7 bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJbXNvLWRlZmF1bHQtcHJvcHM6eWVzOw0KCWZv bnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7DQoJbXNvLWJpZGktZm9udC1mYW1pbHk6 IlRpbWVzIE5ldyBSb21hbiI7fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3NpemU6NjEyLjBwdCA3 OTIuMHB0Ow0KCW1hcmdpbjo3Mi4wcHQgOTAuMHB0IDcyLjBwdCA5MC4wcHQ7DQoJbXNvLWhlYWRl ci1tYXJnaW46MzYuMHB0Ow0KCW1zby1mb290ZXItbWFyZ2luOjM2LjBwdDsNCgltc28tcGFwZXIt c291cmNlOjA7fQ0KZGl2LldvcmRTZWN0aW9uMQ0KCXtwYWdlOldvcmRTZWN0aW9uMTt9DQotLT48 L3N0eWxlPjwhLS1baWYgZ3RlIG1zbyAxMF0+PHN0eWxlPi8qIFN0eWxlIERlZmluaXRpb25zICov DQp0YWJsZS5Nc29Ob3JtYWxUYWJsZQ0KCXttc28tc3R5bGUtbmFtZToiVGFibGUgTm9ybWFsIjsN Cgltc28tdHN0eWxlLXJvd2JhbmQtc2l6ZTowOw0KCW1zby10c3R5bGUtY29sYmFuZC1zaXplOjA7 DQoJbXNvLXN0eWxlLW5vc2hvdzp5ZXM7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1z dHlsZS1wYXJlbnQ6IiI7DQoJbXNvLXBhZGRpbmctYWx0OjBjbSA1LjRwdCAwY20gNS40cHQ7DQoJ bXNvLXBhcmEtbWFyZ2luOjBjbTsNCgltc28tcGFyYS1tYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJ bXNvLXBhZ2luYXRpb246d2lkb3ctb3JwaGFuOw0KCWZvbnQtc2l6ZToxMC41cHQ7DQoJbXNvLWJp ZGktZm9udC1zaXplOjExLjBwdDsNCglmb250LWZhbWlseToiQ2FsaWJyaSIsInNhbnMtc2VyaWYi Ow0KCW1zby1hc2NpaS1mb250LWZhbWlseTpDYWxpYnJpOw0KCW1zby1oYW5zaS1mb250LWZhbWls eTpDYWxpYnJpOw0KCW1zby1mb250LWtlcm5pbmc6MS4wcHQ7fQ0KPC9zdHlsZT48IVtlbmRpZl0t LT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQi IHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+ PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6ZXh0PSJlZGl0 IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPg0KPC9oZWFk Pg0KPGJvZHkgbGFuZz0iWkgtQ04iIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiIHN0eWxlPSJ0 YWItaW50ZXJ2YWw6MjEuMHB0Ij4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9uMSI+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7 bXNvLWJpZGktZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7 LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtZm9udC1mYW1pbHk65a6L5L2TO21z by1iaWRpLWZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oztjb2xvcjojMUY0 OTdEIj5IaSBUb20sPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQt c2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNl cmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZh bWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4t VVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9u dC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1m YXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZhbWlseTomcXVvdDtUaW1l cyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+Jmd0Ozwvc3Bhbj48c3BhbiBsYW5nPSJF Ti1VUyI+IFRoZXJlIGhhcyBiZWVuIGEgbG90IG9mDQogd29yayByZWNlbnRseSB0byBnZXQgc29m dHdhcmUgdmFyaWFudHMgKEdTTyBhbmQgR1JPKSB3b3JraW5nIGluIExpbnV4IHdpdGggdHVubmVs cy48L3NwYW4+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1i aWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv dDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlk aS1mb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+ PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0i RU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1iaWRpLWZvbnQtc2l6ZToxMS4wcHQ7 Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21z by1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlkaS1mb250LWZhbWlseTomcXVvdDtU aW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+QXJlIHlvdSByZWZlcnJpbmcgVURQ IEdSTyBmb3IgVlhMQU4/IEkgYW0gYXdhcmUgb2YNCiB0aGlzIGltcGxlbWVudGF0aW9uIGluIGtl cm5lbCAzLjE0LiBUaGUgcGVyZm9ybWFuY2UgZ2FpbiBzaG91bGQgYmUgc2ltaWxhciwgYnV0IHN0 aWxsIHJlcXVpcmVzIGNhcmVmdWwgY29uZmlndXJhdGlvbiBvZiBWTSBNVFUgdG8gYXZvaWQgSVAg ZnJhZ21lbnRhdGlvbiBvbiBwaHlzaWNhbCBNVFUsIHJhdGhlciB0aGFuIGRlY291cGxpbmcgZnJv bSB1bmRlcmxheS4gSSB0aGluayBlYWNoIG9mIHRoZXNlIHNvbHV0aW9ucyBoYXMgaXRzIG93biBh ZHZhbnRhZ2VzLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNp emU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJp ZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1p bHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7 PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQt ZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFy ZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMg TmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPiZndDsNCjwvc3Bhbj48c3BhbiBsYW5nPSJF Ti1VUyI+SSB0aGluayB0aGlzIHdvdWxkIGJlIGFwcGxpY2FibGUgdG8gYWJvdXQgYWxsIHR1bm5l bCBwcm90b2NvbHMuIFRoZW4geW91IHdvdWxkIGFsc28gd2FudCB0byBkbyByZWFzc2VtYmx5IGF0 IGVhY2ggaG9wPyBTb3VuZHMgZXhwZW5zaXZlLiBPbmNlIHlvdSBzZWdtZW50LCBJIHRoaW5rIHlv dSdkIG9ubHkNCjxzcGFuIGNsYXNzPSJTcGVsbEUiPm9ubHk8L3NwYW4+IHdhbnQgdG8gcmVhc3Nl bWJsZSBhdCB0aGUgZW5kIGhvc3QuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z b05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O21zby1i aWRpLWZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv dDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWZvbnQtZmFtaWx5OuWui+S9kzttc28tYmlk aS1mb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDs7Y29sb3I6IzFGNDk3RCI+ VGhlcmUgaXMgbm8gc2VnbWVudGF0aW9uL3JlYXNzZW1ibHkgbmVlZGVkIGF0IGVhY2gNCiBob3As IGJ1dCBpbnN0ZWFkIOKAnG9mZmxvYWTigJ0gaG9wIGJ5IGhvcCwganVzdCBzYXZlIHRoZSBtZXRh ZGF0YSB0byB0aGUgbmV4dCBob3AgaGVhZGVycy4gQW5kIGZpbmFsbHkgc2VnbWVudGF0aW9uIHNo b3VsZCBiZSBhdm9pZGVkIGlmIHRoZSBkZXN0aW5hdGlvbiBpcyBhIFZNIG9uIGEgaHlwZXJ2aXNv ci48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5n PSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7bXNvLWJpZGktZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7 bXNvLWZhcmVhc3QtZm9udC1mYW1pbHk65a6L5L2TO21zby1iaWRpLWZvbnQtZmFtaWx5OiZxdW90 O1RpbWVzIE5ldyBSb21hbiZxdW90Oztjb2xvcjojMUY0OTdEIj5UaGlzIHNjZW5hcmlvIGlzIGxp a2UgaW4gRXhhbXBsZSAyIChzZWN0aW9uIDUuMiksIGp1c3QNCiByZXBsYWNlIHRoZSBWWExBTi1T T0UgYmV0d2VlbiB0aGUgMiBnYXRld2F5cyBieSBhbnkgb3RoZXIgVHVubmVsIHByb3RvY29sIHRo YXQgY2FuIHN1cHBvcnQgc2ltaWxhciBvZmZsb2FkaW5nIGZlYXR1cmUsIHN1Y2ggYXMgU1RULjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2Zv bnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28t ZmFyZWFzdC1mb250LWZhbWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGlt ZXMgTmV3IFJvbWFuJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9u dC1zaXplOjEwLjVwdDttc28tYmlkaS1mb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1mb250LWZh bWlseTrlrovkvZM7bXNvLWJpZGktZm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7O2NvbG9yOiMxRjQ5N0QiPkJlc3QgcmVnYXJkcyw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZTox MC41cHQ7bXNvLWJpZGktZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJp JnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtZm9udC1mYW1pbHk65a6L 5L2TO21zby1iaWRpLWZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oztjb2xv cjojMUY0OTdEIj5IYW48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7bXNvLWJpZGktZm9u dC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMt c2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtZm9udC1mYW1pbHk65a6L5L2TO21zby1iaWRpLWZvbnQt ZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oztjb2xvcjojMUY0OTdEIj48bzpwPiZu YnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVm dDpzb2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQiPg0KPGRpdj4NCjxk aXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29saWQgI0I1QzRERiAxLjBwdDtwYWRk aW5nOjMuMHB0IDBjbSAwY20gMGNtIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIGxh bmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhv bWEmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+RnJvbTo8L3NwYW4+PC9iPjxzcGFuIGxh bmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhv bWEmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+IFRvbSBIZXJiZXJ0IFttYWlsdG86dGhl cmJlcnRAZ29vZ2xlLmNvbV0NCjxicj4NCjxiPlNlbnQ6PC9iPiBXZWRuZXNkYXksIE1heSAyMSwg MjAxNCAxMDo0MiBBTTxicj4NCjxiPlRvOjwvYj4gWmhvdSwgSGFuPGJyPg0KPGI+Q2M6PC9iPiBu dm8zQGlldGYub3JnOyB0b2Zvb0BpZXRmLm9yZzsgZHJhZnQtbWFoYWxpbmdhbS1kdXR0LWRjb3Bz LXZ4bGFuQHRvb2xzLmlldGYub3JnOyBkcmFmdC16aG91LWxpLXZ4bGFuLXNvZUB0b29scy5pZXRm Lm9yZzsgRXJpayBOb3JkbWFyazxicj4NCjxiPlN1YmplY3Q6PC9iPiBSZTogRlc6IEktRCBBY3Rp b246IGRyYWZ0LXpob3UtbGktdnhsYW4tc29lLTAxLnR4dDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjwvZGl2Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+ PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2Pg0K PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij48c3BhbiBs YW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdj4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5PbiBUdWUsIE1heSAyMCwgMjAxNCBhdCA1 OjI4IFBNLCBaaG91LCBIYW4gJmx0OzxhIGhyZWY9Im1haWx0bzpoemhvdThAZWJheS5jb20iIHRh cmdldD0iX2JsYW5rIj5oemhvdThAZWJheS5jb208L2E+Jmd0OyB3cm90ZTo8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28t bWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0byI+PHNwYW4gbGFu Zz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGli cmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj5IaSBUb20sPC9z cGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0 b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtm b250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29s b3I6IzFGNDk3RCI+Jm5ic3A7PC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0 OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls ZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90 O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+VGhhbmtzIGZvciB5b3VyIGNvbW1lbnRz Lg0KPC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdp bi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEw LjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVv dDs7Y29sb3I6IzFGNDk3RCI+WWVzIFRTTy9MUk8gd2l0aCBWWExBTiBzdXBwb3J0IHNob3VsZCBw cm92aWRlIHNpbWlsYXIgb3IgZXZlbiBiZXR0ZXIgcGVyZm9ybWFuY2UgZ2FpbnMsDQogYnV0IHRo ZSBtZWNoYW5pc20gcHJvcG9zZWQgYnkgdGhpcyBkcmFmdCBkZWNvdXBsZXMgb3ZlcmxheSBhbmQg dW5kZXJsYXksIGFuZCBpdCBpcyBoYXJkd2FyZSBpbmRlcGVuZGVudC4NCjwvc3Bhbj48c3BhbiBs YW5nPSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRv Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6 JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0Qi PlNlY29uZGx5LCBoYXJkd2FyZSBvZmZsb2FkaW5nIHVzdWFsbHkgc3VwcG9ydCBUQ1Agb25seSAo VFNPKS4gVGhlIG1lY2hhbmlzbSBoZXJlIGNhbg0KIGhlbHAgb24gbGFyZ2UgVURQIHBhY2tldCBw ZXJmb3JtYW5jZSwgYWxzbyB2ZXJpZmllZCBieSB0aGUgcHJvdG90eXBlLjwvc3Bhbj48c3BhbiBs YW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8 L2Rpdj4NCjxibG9ja3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAj Q0NDQ0NDIDEuMHB0O21zby1ib3JkZXItbGVmdC1hbHQ6c29saWQgI0NDQ0NDQyAuNzVwdDtwYWRk aW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowY20i Pg0KPGRpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10 b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7 LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+QlRXLCBob3cgbWFueSB0eXBl cyBvZiBvZmYtdGhlLXNoZWxmIE5JQyBzdXBwb3J0IFZYTEFOIG9mZmxvYWRpbmc/IEFueSBwZXJm b3JtYW5jZSBkYXRhPzwvc3Bhbj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+ PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRv O21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZv bnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5z LXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPiZuYnNwOzwvc3Bhbj48c3BhbiBsYW5nPSJFTi1V UyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+SFcgaXMgbm90 IGEgcmVxdWlyZW1lbnQgZm9yIHRoaXMgb2ZmbG9hZGluZy4gVGhlcmUgaGFzIGJlZW4gYSBsb3Qg b2Ygd29yayByZWNlbnRseSB0byBnZXQgc29mdHdhcmUgdmFyaWFudHMgKEdTTyBhbmQgR1JPKSB3 b3JraW5nIGluIExpbnV4IHdpdGggdHVubmVscy4gVGhlc2UgZG8gc2hvdyAyLTN4IHBlcmZvcm1h bmNlIGltcHJvdmVtZW50cy4gSFcgc3VwcG9ydCB3b3VsZCBiZQ0KIG1vc3RseSBhbiBpbmNyZW1l bnRhbCBpbXByb3ZlbWVudCB0byB0aGF0LCBvciBiZWNvbWVzIGludGVyZXN0aW5nIGZvciBPUyBi eXBhc3MgbGlrZSBpbiBTUi1JT1YuIEdTTy9HUk8gY2FuIGJlIHByZXNlbnRlZCB0byB0aGUgZ3Vl c3QgYXMgVFNPIGFuZCBMUk8gc28gdGhhdCBpdCdzIHBvc3NpYmxlIHRvIHBsdW1iIHVzZSBvZiBs YXJnZSBwYWNrZXRzIGZyb20gdGhlIGd1ZXN0IGFsbCB0aGUgd2F5IHRvIHRoZSBob3N0IGRyaXZl ci4gSXQgc2hvdWxkDQogYWxzbyBiZSBwb3NzaWJsZSB0byBwbHVtYiB0d28gVk1zIG9uIHRoZSBz YW1lIGhvc3QgdG8gY29tbXVuaWNhdGUgd2l0aG91dCBzZWdtZW50YXRpb24sIGkuZS4gb3V0cHV0 IGZyb20gVFNPIG9uIG9uZSBWTSBiZWNvbWVzIGlucHV0IGZvciBhbm90aGVyLiZuYnNwOzxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+VGhlIGFkdmFudGFn ZSB0aGF0IEkgc2VlIGluIHlvdXIgZHJhZnQgaXMgdGhhdCBpdCBhbGxvd3MgYW4gaW50ZXJtZWRp YXRlIGRldmljZSB0byBwZXJmb3JtIHNlZ21lbnRhdGlvbi9yZWFzc2VtYmx5IGluc3RlYWQgb2Yg ZnJhZ21lbnRhdGlvbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3Nw YW4+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxl ZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDttc28tYm9yZGVyLWxlZnQtYWx0OnNvbGlkICNDQ0NDQ0Mg Ljc1cHQ7cGFkZGluZzowY20gMGNtIDBjbSA2LjBwdDttYXJnaW4tbGVmdDo0LjhwdDttYXJnaW4t cmlnaHQ6MGNtIj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1z by1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3BhbiBs YW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2Fs aWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPkxpa2V3aXNl LCBzZXR0aW5nIGxhcmdlIE1UVSBvbiBvdmVybGF5IGludGVyZmFjZXMgYWNoaWV2ZXMgc2ltaWxh ciByZXN1bHQsIGJ1dCBzdGlsbA0KIHRoZSBvdmVybGF5L3VuZGVybGF5IGRlY291cGxpbmcgaXNz dWUuIEl0IGlzIHVzdWFsbHkgYWR2aXNlZCB0aGF0IG92ZXJsYXkgTVRVIGlzIHNsaWdodGx5IHNt YWxsZXIgdGhhbiB1bmRlcmxheSB0byBhdm9pZCBpbmVmZmljaWVudCBmcmFnbWVudGF0aW9uIGFm dGVyIGFkZGluZyB0aGUgb3V0ZXIgaGVhZGVyLCBidXQgdG8gYWNoaWV2ZSByZWFsbHkgaGlnaCBw ZXJmb3JtYW5jZSBiZXR3ZWVuIFZNcywgbGFyZ2UgTVRVIGlzIHByZWZlcnJlZC4NCjwvc3Bhbj48 c3BhbiBsYW5nPSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4N CjwvYmxvY2txdW90ZT4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkRlcGVuZHMgb24gdGhlIHBlcmZvcm1h bmNlIGRpbWVuc2lvbiB0byBiZSBvcHRpbWl6ZWQuIExhcmdlciBNVFVzIGNvdWxkIGluY3JlYXNl IGxhdGVuY3kgb2YgaGlnaCBwcmlvcml0eSBzbWFsbCBwYWNrZXRzIGZvciBpbnN0YW5jZSAoSE9M IGJsb2NraW5nKSwgb3IgVURQIGJhc2VkIGFwcGxpY2F0aW9uIG1pZ2h0IHRyeSB0byB1c2UgTVRV IHRvIGRlY2lkZSBob3cgbGFyZ2UgaXQNCiBzaG91bGQgc2VuZCBpdCdzIHBhY2tldHMgdG8gYXZv aWQgZnJhZ21lbnRhdGlvbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48 L3NwYW4+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVy LWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDttc28tYm9yZGVyLWxlZnQtYWx0OnNvbGlkICNDQ0ND Q0MgLjc1cHQ7cGFkZGluZzowY20gMGNtIDBjbSA2LjBwdDttYXJnaW4tbGVmdDo0LjhwdDttYXJn aW4tcmlnaHQ6MGNtIj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9 Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3Bh biBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7 Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPkFuZCBj b25zaWRlcmluZyBvdmVybGF5ICZsdDstJmd0OyBwaHlzaWNhbCBjb25uZWN0aW9uLCBwYXRoIE1U VSBkaXNjb3ZlcnkgaXMgbm90IGFsd2F5cyB3b3JrLjwvc3Bhbj48c3BhbiBsYW5nPSJFTi1VUyI+ Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90 ZT4NCjxibG9ja3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAjQ0ND Q0NDIDEuMHB0O21zby1ib3JkZXItbGVmdC1hbHQ6c29saWQgI0NDQ0NDQyAuNzVwdDtwYWRkaW5n OjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowY20iPg0K PGRpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3At YWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBz dHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZx dW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+VGhpcyBraW5kIG9mIGNvbmZpZ3Vy YXRpb24gY29tcGxleGl0eSBhbmQgcGFpbi1wb2ludCBjYW4gYmUgcmVzb2x2ZWQgc2ltcGx5IGJ5 IGRlY291cGxpbmcNCiBvdmVybGF5IGFuZCB1bmRlcmxheSBNVFUsIGFzIHN1Z2dlc3RlZCBieSB0 aGlzIGRyYWZ0LiBIZXJlIGlzIGFuIGV4YW1wbGUgb2YgY29uZmlndXJhdGlvbiBjb25mdXNpb246 PC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1i b3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVw dDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7 Y29sb3I6IzFGNDk3RCI+PGEgaHJlZj0iaHR0cDovL29wZW52c3dpdGNoLm9yZy9waXBlcm1haWwv ZGlzY3Vzcy8yMDE0LU1heS8wMTM4OTguaHRtbCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9vcGVu dnN3aXRjaC5vcmcvcGlwZXJtYWlsL2Rpc2N1c3MvMjAxNC1NYXkvMDEzODk4Lmh0bWw8L2E+PC9z cGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0 b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtm b250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29s b3I6IzFGNDk3RCI+Jm5ic3A7PC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0 OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls ZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90 O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+SWRlYWxseSwgYWxsIE5WIHR1bm5lbCBw cm90b2NvbHMgc2hvdWxkIHN1cHBvcnQgc2ltaWxhciBtZXRhZGF0YSwgdGh1cyBvdmVybGF5IHNl Z21lbnRhdGlvbg0KIGNhbiBiZSBvZmZsb2FkZWQgaG9wLWJ5LWhvcC48L3NwYW4+PHNwYW4gbGFu Zz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Jsb2Nr cXVvdGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxv OnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5JIHRoaW5rIHRoaXMgd291bGQgYmUgYXBwbGljYWJs ZSB0byBhYm91dCBhbGwgdHVubmVsIHByb3RvY29scy4gVGhlbiB5b3Ugd291bGQgYWxzbyB3YW50 IHRvIGRvIHJlYXNzZW1ibHkgYXQgZWFjaCBob3A/IFNvdW5kcyBleHBlbnNpdmUuIE9uY2UgeW91 IHNlZ21lbnQsIEkgdGhpbmsgeW91J2Qgb25seSBvbmx5IHdhbnQgdG8gcmVhc3NlbWJsZSBhdCB0 aGUgZW5kIGhvc3QuPG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIj5PbmUgaW1wb3J0YW50IHRoaW5nIHRvIGtlZXAgaW4gbWluZCwgYW5kIGEgaGFyZCBsZXNz b24gaW4gcmVhbCBkZXBsb3ltZW50IDotKSwgc2VnbWVudGF0aW9uIG9mZmxvYWQgaXMgb3Bwb3J0 dW5pc3RpYyBhbmQgdmVyeSBkZXBlbmRlbnQgb24gdGhlIGNvbmRpdGlvbnMgb2YgdGhlIHRyYWZm aWMuIEl0J3MgdmFsdWUgY2FuIGJlIGZsZWV0aW5nIGluIHJlYWwgd29ya2xvYWRzLiBGb3INCiBp bnN0YW5jZSBpbWFnaW5lIGEgaG9zdCB3aXRoIGEgbG90IG9mIGFjdGl2ZSBoaWdoIHRocm91Z2hv dXQgY29ubmVjdGlvbnMgKGxpa2UgYSB2aWRlbyBzZXJ2aW5nKSB3aGljaCBoaXRzIGEgaGljY3Vw IGNhdXNpbmcgYWxsIGNvbmdlc3Rpb25zIHdpbmRvd3MgdG8gc2hyaW5rLiBJZiB0aGUgc3lzdGVt IGlzIG5vdCBwcm92aXNpb25lZCBmb3IgdGhpcyBldmVudCBpdCBjYW4gYmUgdmVyeSBoYXJkIHRv IHJlY292ZXIgKGEgbG90IG1vcmUgQ1BVIGlzDQogcmVxdWlyZWQgdG8gYWNoaWV2ZSBzYW1lIHRo cm91Z2hwdXQpLiBUaGlzIGRpZmZlcmVudGlhdGVzIGEgbGFyZ2VyIE1UVSBmcm9tIHJlbHlpbmcg b24gc2VnbWVudGF0aW9uLCB0aGUgZm9ybWVyIG9mZmVycyBtb3JlIHByZWRpY3RhYmxlIENQVSBz YXZpbmdzLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+ DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+ VG9tPG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwv ZGl2Pg0KPGJsb2NrcXVvdGUgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkICND Q0NDQ0MgMS4wcHQ7bXNvLWJvcmRlci1sZWZ0LWFsdDpzb2xpZCAjQ0NDQ0NDIC43NXB0O3BhZGRp bmc6MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXJpZ2h0OjBjbSI+ DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRv cC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0byI+PHNwYW4gbGFuZz0iRU4tVVMi IHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDss JnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj4mbmJzcDs8L3NwYW4+PHNwYW4g bGFuZz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0 byI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTAuNXB0O2ZvbnQtZmFtaWx5 OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdE Ij5CZXN0IHJlZ2FyZHMsPC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1 dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0i Zm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3Nh bnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+SGFuPC9zcGFuPjxzcGFuIGxhbmc9IkVOLVVT Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNv LW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxh bmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjEwLjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxp YnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+Jm5ic3A7PC9z cGFuPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2IHN0eWxl PSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBj bSAwY20gNC4wcHQiPg0KPGRpdj4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6 c29saWQgI0I1QzRERiAxLjBwdDtwYWRkaW5nOjMuMHB0IDBjbSAwY20gMGNtIj4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJv dHRvbS1hbHQ6YXV0bzttc28tb3V0bGluZS1sZXZlbDoxIj4NCjxiPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhvbWEmcXVvdDss JnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+RnJvbTo8L3NwYW4+PC9iPjxzcGFuIGxhbmc9IkVOLVVT IiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhvbWEmcXVvdDss JnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+IFRvbSBIZXJiZXJ0IFttYWlsdG86PGEgaHJlZj0ibWFp bHRvOnRoZXJiZXJ0QGdvb2dsZS5jb20iIHRhcmdldD0iX2JsYW5rIj50aGVyYmVydEBnb29nbGUu Y29tPC9hPl0NCjxicj4NCjxiPlNlbnQ6PC9iPiBXZWRuZXNkYXksIE1heSAyMSwgMjAxNCAyOjQ0 IEFNPGJyPg0KPGI+VG86PC9iPiBaaG91LCBIYW48YnI+DQo8Yj5DYzo8L2I+IDxhIGhyZWY9Im1h aWx0bzpudm8zQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+bnZvM0BpZXRmLm9yZzwvYT47IDxh IGhyZWY9Im1haWx0bzp0b2Zvb0BpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPg0KdG9mb29AaWV0 Zi5vcmc8L2E+OyA8YSBocmVmPSJtYWlsdG86ZHJhZnQtbWFoYWxpbmdhbS1kdXR0LWRjb3BzLXZ4 bGFuQHRvb2xzLmlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+DQpkcmFmdC1tYWhhbGluZ2FtLWR1 dHQtZGNvcHMtdnhsYW5AdG9vbHMuaWV0Zi5vcmc8L2E+OyA8YSBocmVmPSJtYWlsdG86ZHJhZnQt emhvdS1saS12eGxhbi1zb2VAdG9vbHMuaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj4NCmRyYWZ0 LXpob3UtbGktdnhsYW4tc29lQHRvb2xzLmlldGYub3JnPC9hPjsgRXJpayBOb3JkbWFyazxicj4N CjxiPlN1YmplY3Q6PC9iPiBSZTogRlc6IEktRCBBY3Rpb246IGRyYWZ0LXpob3UtbGktdnhsYW4t c29lLTAxLnR4dDwvc3Bhbj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+PC9w Pg0KPC9kaXY+DQo8L2Rpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5 bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48 c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPGRpdj4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFy Z2luLWJvdHRvbS1hbHQ6YXV0byI+PHNwYW4gbGFuZz0iRU4tVVMiPkhpIFpvdSwgYSBjb3VwbGUg b2YgcXVlc3Rpb25zIGlubGluZS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2Pg0KPGRpdj4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttYXJn aW4tYm90dG9tOjEyLjBwdCI+PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOzxvOnA+PC9vOnA+PC9z cGFuPjwvcD4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10 b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVT Ij5PbiBNb24sIE1heSAxOSwgMjAxNCBhdCA4OjAxIFBNLCBaaG91LCBIYW4gJmx0OzxhIGhyZWY9 Im1haWx0bzpoemhvdThAZWJheS5jb20iIHRhcmdldD0iX2JsYW5rIj5oemhvdThAZWJheS5jb208 L2E+Jmd0OyB3cm90ZTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1 dG8iPjxzcGFuIGxhbmc9IkVOLVVTIj5IaSw8YnI+DQo8YnI+DQpXZSBoYXZlIHVwZGF0ZWQgdGhl IFZYTEFOLVNPRSBkcmFmdCBhY2NvcmRpbmcgdG8gZWFybGllciBjb21tZW50cy4gTm93IGl0IGlz IGZ1bGx5IGNvbXBhdGlibGUgd2l0aCBWWExBTi1HUEUuIEFuZCBzb21lIGV4YW1wbGVzIGFyZSBh ZGRlZCBmb3IgYmV0dGVyIHVuZGVyc3RhbmRpbmcmbmJzcDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0ND QyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdp bi10b3A6NS4wcHQ7bWFyZ2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxwIGNs YXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2lu LWJvdHRvbS1hbHQ6YXV0byI+PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOzxvOnA+PC9vOnA+PC9z cGFuPjwvcD4NCjwvYmxvY2txdW90ZT4NCjxibG9ja3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTti b3JkZXItbGVmdDpzb2xpZCAjQ0NDQ0NDIDEuMHB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNi4wcHQ7 bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXRvcDo1LjBwdDttYXJnaW4tcmlnaHQ6MGNtO21hcmdp bi1ib3R0b206NS4wcHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4t dG9wLWFsdDphdXRvO21hcmdpbi1ib3R0b206MTIuMHB0Ij48c3BhbiBsYW5nPSJFTi1VUyI+QSBw cm90b3R5cGUgaXMgYWxzbyBpbXBsZW1lbnRlZCBoZXJlIChwYXRjaCBiYXNlZCBvbiBPcGVuIHZT d2l0Y2gpOjxicj4NCjxhIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNvbS9oemhvdTgvb3BlbnZzd2l0 Y2gvY29tbWl0LzlhN2RlYjhiNDMyY2U4M2E5YzA5ZDdkNGZmODVmYTA1MGY3ZGQyYmUiIHRhcmdl dD0iX2JsYW5rIj5odHRwczovL2dpdGh1Yi5jb20vaHpob3U4L29wZW52c3dpdGNoL2NvbW1pdC85 YTdkZWI4YjQzMmNlODNhOWMwOWQ3ZDRmZjg1ZmEwNTBmN2RkMmJlPC9hPjxicj4NCjxicj4NCm5l dHBlcmYgVENQX1NUUkVBTSB0ZXN0IHJlc3VsdCBiZXR3ZWVuIGEgcGFpcnMgb2YgVk1zIG9uIGhv c3RzIHdpdGggMTBHIGludGVyZmFjZXM6PGJyPg0KPGJyPg0KQmVmb3JlIHRoZSBjaGFuZ2U6IDIu NjIgR2JpdHMvc2VjPGJyPg0KQWZ0ZXIgdGhlIGNoYW5nZTogNi42OCBHYml0cy9zZWM8YnI+DQpT cGVlZHVwIGlzIH4yNTAlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvYmxvY2txdW90ZT4NCjxk aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibXNvLW1hcmdpbi10b3AtYWx0OmF1dG87 bXNvLW1hcmdpbi1ib3R0b20tYWx0OmF1dG8iPjxzcGFuIGxhbmc9IkVOLVVTIj5DYW4geW91IHBy b3ZpZGUgc29tZSBtb3JlIGRldGFpbHMgb24gdGhpcyBiZW5lZml0PyBJdCBzZWVtcyBsaWtlIHBs YWluIFRTTy9MUk8gdGhhdCB1bmRlcnN0YW5kcyBlbmNhcHN1bGF0aW9uIHNob3VsZCBwcm92aWRl IHNpbWlsYXIgYmVuZWZpdHMgd2hlbiBnb2luZyBiZXR3ZWVuDQogaG9zdHMuPG86cD48L286cD48 L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1z by1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3BhbiBs YW5nPSJFTi1VUyIgc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPiZuYnNwOzwvc3Bhbj48c3BhbiBsYW5n PSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5 bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48 c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMC41cHQ7Zm9udC1mYW1pbHk6JnF1 b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPiZu YnNwOzwvc3Bhbj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9k aXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0ND Q0NDQyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21h cmdpbi10b3A6NS4wcHQ7bWFyZ2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttYXJnaW4t Ym90dG9tOjEyLjBwdCI+PHNwYW4gbGFuZz0iRU4tVVMiPlRoZSBwYXRjaCBhdHRyYWN0ZWQgc29t ZSBpbnRlcmVzdHMgaW4gT1ZTIGNvbW11bml0eSwgYnV0IHNpbmNlIHRoaXMgUkZDIGRyYWZ0IGlz IGluIHZlcnkgZWFybHkgc3RhZ2Ugc28gaXQgaXMgcmVnYXJkZWQgYXMgaW5hcHByb3ByaWF0ZSBi eSBKZXNzZSB0byBhcHBseSB0aGUgY2hhbmdlDQogdG8gT1ZTIHRyZWUuPGJyPg0KVGhlIGRpc2N1 c3MgbWFpbC10aHJlYWQ6PGJyPg0KPGEgaHJlZj0iaHR0cDovL29wZW52c3dpdGNoLm9yZy9waXBl cm1haWwvZGlzY3Vzcy8yMDE0LU1heS8wMTM5ODEuaHRtbCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6 Ly9vcGVudnN3aXRjaC5vcmcvcGlwZXJtYWlsL2Rpc2N1c3MvMjAxNC1NYXkvMDEzOTgxLmh0bWw8 L2E+PGJyPg0KPGEgaHJlZj0iaHR0cDovL29wZW52c3dpdGNoLm9yZy9waXBlcm1haWwvZGlzY3Vz cy8yMDE0LU1heS8wMTM4OTguaHRtbCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9vcGVudnN3aXRj aC5vcmcvcGlwZXJtYWlsL2Rpc2N1c3MvMjAxNC1NYXkvMDEzODk4Lmh0bWw8L2E+PGJyPg0KPGJy Pg0KU28gd2Ugd291bGQgbGlrZSB0byByZXF1ZXN0IGEgcmV2aWV3IGhlcmUgYnkgTlZPMy9UT0ZP TyBncm91cHMgYW5kIFZYTEFOIGF1dGhvcnM6IGlzIHRoaXMgVlhMQU4gZXh0ZW5zaW9uIGlzIHdv cnRoIGZvcm1hbGx5IHB1dCBpbnRvIFZYTEFOIGFzIGEgc3RhbmRhcmQsIHNvIHRoYXQgbW9yZSBw ZW9wbGUgY2FuIGJlbmVmaXQgZnJvbSBpdD88bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L2Jsb2Nr cXVvdGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9w LWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3BhbiBsYW5nPSJFTi1VUyI+ Q291bGQgeW91IGdldCB0aGUgc2FtZSBlZmZlY3QgYnkgc2V0dGluZyBsYXJnZXIgTVRVcyBvbiB0 aGUgb3ZlcmxheSBuZXR3b3JrIGludGVyZmFjZSBhbmQgcmVseWluZyBpbiBwYXRoIE1UVSBkaXNj b3Zlcnkgd2hlbiBnb2luZyBvdmVyIHBoeXNpY2FsIG5ldHdvcms/PG86cD48L286cD48L3NwYW4+ PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1zby1tYXJn aW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFsdDphdXRvIj48c3BhbiBsYW5nPSJF Ti1VUyI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBz dHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5n OjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi10b3A6NS4wcHQ7bWFy Z2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi IHN0eWxlPSJtc28tbWFyZ2luLXRvcC1hbHQ6YXV0bzttc28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0 bzttc28tb3V0bGluZS1sZXZlbDoxIj4NCjxzcGFuIGxhbmc9IkVOLVVTIj5CZXN0IHJlZ2FyZHMs PGJyPg0KSGFuPGJyPg0KPGJyPg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS08YnI+DQpGcm9t OiBJLUQtQW5ub3VuY2UgW21haWx0bzo8YSBocmVmPSJtYWlsdG86aS1kLWFubm91bmNlLWJvdW5j ZXNAaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5pLWQtYW5ub3VuY2UtYm91bmNlc0BpZXRmLm9y ZzwvYT5dIE9uIEJlaGFsZiBPZg0KPGEgaHJlZj0ibWFpbHRvOmludGVybmV0LWRyYWZ0c0BpZXRm Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmludGVybmV0LWRyYWZ0c0BpZXRmLm9yZzwvYT48YnI+DQpT ZW50OiBGcmlkYXksIE1heSAwMiwgMjAxNCA4OjA5IFBNPGJyPg0KVG86IDxhIGhyZWY9Im1haWx0 bzppLWQtYW5ub3VuY2VAaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5pLWQtYW5ub3VuY2VAaWV0 Zi5vcmc8L2E+PGJyPg0KU3ViamVjdDogSS1EIEFjdGlvbjogZHJhZnQtemhvdS1saS12eGxhbi1z b2UtMDEudHh0PGJyPg0KPGJyPg0KPGJyPg0KQSBOZXcgSW50ZXJuZXQtRHJhZnQgaXMgYXZhaWxh YmxlIGZyb20gdGhlIG9uLWxpbmUgSW50ZXJuZXQtRHJhZnRzIGRpcmVjdG9yaWVzLjxicj4NCjxi cj4NCjxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBUaXRsZSAmbmJzcDsgJm5ic3A7 ICZuYnNwOyAmbmJzcDsgJm5ic3A7IDogU2VnbWVudGF0aW9uIE9mZmxvYWRpbmcgRXh0ZW5zaW9u IGZvciBWWExBTjxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBBdXRob3JzICZuYnNw OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyA6IEhhbiBaaG91PGJyPg0KJm5ic3A7ICZuYnNwOyAmbmJz cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw OyAmbmJzcDsgJm5ic3A7IENoZW5neXVhbiBMaTxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu YnNwOyBGaWxlbmFtZSAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDs6IGRyYWZ0LXpob3UtbGkt dnhsYW4tc29lLTAxLnR4dDxicj4NCiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyBQYWdlcyAm bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IDogMTM8YnI+DQombmJzcDsgJm5ic3A7 ICZuYnNwOyAmbmJzcDsgRGF0ZSAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu YnNwOzogMjAxNC0wNS0wMjxicj4NCjxicj4NCkFic3RyYWN0Ojxicj4NCiZuYnNwOyAmbmJzcDtT ZWdtZW50YXRpb24gb2ZmbG9hZGluZyBpcyBub3dhZGF5cyBjb21tb24gaW4gbmV0d29yayBzdGFj azxicj4NCiZuYnNwOyAmbmJzcDtpbXBsZW1lbnRhdGlvbiBhbmQgd2VsbCBzdXBwb3J0ZWQgYnkg cGFyYS12aXJ0dWFsaXplZCBuZXR3b3JrIGRldmljZTxicj4NCiZuYnNwOyAmbmJzcDtkcml2ZXJz IGZvciB2aXJ0dWFsIG1hY2hpbmUgKFZNKXMuIFRoaXMgZHJhZnQgZGVzY3JpYmVzIGFuIGV4dGVu c2lvbjxicj4NCiZuYnNwOyAmbmJzcDt0byBWaXJ0dWFsIGVYdGVuc2libGUgTG9jYWwgQXJlYSBO ZXR3b3JrIChWWExBTikgc28gdGhhdCBzZWdtZW50YXRpb248YnI+DQombmJzcDsgJm5ic3A7Y2Fu IGJlIGRlY291cGxlZCBmcm9tIHBoeXNpY2FsL3VuZGVybGF5IG5ldHdvcmtzIGFuZCBvZmZsb2Fk ZWQ8YnI+DQombmJzcDsgJm5ic3A7ZnVydGhlciB0byB0aGUgcmVtb3RlIGVuZC1wb2ludCB0aHVz IGltcHJvdmluZyBkYXRhLXBsYW5lIHBlcmZvcm1hbmNlPGJyPg0KJm5ic3A7ICZuYnNwO2ZvciBW TXMgcnVubmluZyBvbiB0b3Agb2Ygb3ZlcmxheSBuZXR3b3Jrcy48YnI+DQo8YnI+DQo8YnI+DQpU aGUgSUVURiBkYXRhdHJhY2tlciBzdGF0dXMgcGFnZSBmb3IgdGhpcyBkcmFmdCBpczo8YnI+DQo8 YSBocmVmPSJodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9kcmFmdC16aG91LWxpLXZ4 bGFuLXNvZS8iIHRhcmdldD0iX2JsYW5rIj5odHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2Rv Yy9kcmFmdC16aG91LWxpLXZ4bGFuLXNvZS88L2E+PGJyPg0KPGJyPg0KVGhlcmUncyBhbHNvIGEg aHRtbGl6ZWQgdmVyc2lvbiBhdmFpbGFibGUgYXQ6PGJyPg0KPGEgaHJlZj0iaHR0cDovL3Rvb2xz LmlldGYub3JnL2h0bWwvZHJhZnQtemhvdS1saS12eGxhbi1zb2UtMDEiIHRhcmdldD0iX2JsYW5r Ij5odHRwOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMTwv YT48YnI+DQo8YnI+DQpBIGRpZmYgZnJvbSB0aGUgcHJldmlvdXMgdmVyc2lvbiBpcyBhdmFpbGFi bGUgYXQ6PGJyPg0KPGEgaHJlZj0iaHR0cDovL3d3dy5pZXRmLm9yZy9yZmNkaWZmP3VybDI9ZHJh ZnQtemhvdS1saS12eGxhbi1zb2UtMDEiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vd3d3LmlldGYu b3JnL3JmY2RpZmY/dXJsMj1kcmFmdC16aG91LWxpLXZ4bGFuLXNvZS0wMTwvYT48YnI+DQo8YnI+ DQo8YnI+DQpQbGVhc2Ugbm90ZSB0aGF0IGl0IG1heSB0YWtlIGEgY291cGxlIG9mIG1pbnV0ZXMg ZnJvbSB0aGUgdGltZSBvZiBzdWJtaXNzaW9uPGJyPg0KdW50aWwgdGhlIGh0bWxpemVkIHZlcnNp b24gYW5kIGRpZmYgYXJlIGF2YWlsYWJsZSBhdCA8YSBocmVmPSJodHRwOi8vdG9vbHMuaWV0Zi5v cmciIHRhcmdldD0iX2JsYW5rIj4NCnRvb2xzLmlldGYub3JnPC9hPi48YnI+DQo8YnI+DQpJbnRl cm5ldC1EcmFmdHMgYXJlIGFsc28gYXZhaWxhYmxlIGJ5IGFub255bW91cyBGVFAgYXQ6PGJyPg0K PGEgaHJlZj0iZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0cy8iIHRhcmdldD0iX2Js YW5rIj5mdHA6Ly9mdHAuaWV0Zi5vcmcvaW50ZXJuZXQtZHJhZnRzLzwvYT48YnI+DQo8YnI+DQpf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4NCkktRC1B bm5vdW5jZSBtYWlsaW5nIGxpc3Q8YnI+DQo8YSBocmVmPSJtYWlsdG86SS1ELUFubm91bmNlQGll dGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+SS1ELUFubm91bmNlQGlldGYub3JnPC9hPjxicj4NCjxh IGhyZWY9Imh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vaS1kLWFubm91bmNl SW50ZXJuZXQtRHJhZnQiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL3d3dy5pZXRmLm9yZy9tYWls bWFuL2xpc3RpbmZvL2ktZC1hbm5vdW5jZTxicj4NCkludGVybmV0LURyYWZ0PC9hPiBkaXJlY3Rv cmllczogPGEgaHJlZj0iaHR0cDovL3d3dy5pZXRmLm9yZy9zaGFkb3cuaHRtbCIgdGFyZ2V0PSJf YmxhbmsiPg0KaHR0cDovL3d3dy5pZXRmLm9yZy9zaGFkb3cuaHRtbDwvYT48YnI+DQpvciA8YSBo cmVmPSJmdHA6Ly9mdHAuaWV0Zi5vcmcvaWV0Zi8xc2hhZG93LXNpdGVzLnR4dCIgdGFyZ2V0PSJf YmxhbmsiPmZ0cDovL2Z0cC5pZXRmLm9yZy9pZXRmLzFzaGFkb3ctc2l0ZXMudHh0PC9hPjxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1zby1tYXJnaW4tdG9wLWFsdDphdXRvO21zby1tYXJnaW4tYm90dG9tLWFs dDphdXRvIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0K PC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8 L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g bGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPC9kaXY+ DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --_000_9F56174078B48B459268EFF1DAB66B1A109C7F16DENEXDDAS32corp_-- From nobody Wed May 21 10:02:56 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 46FFD1A0879; Wed, 21 May 2014 10:02:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.851 X-Spam-Level: X-Spam-Status: No, score=-4.851 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sKqlW9HHP-_S; Wed, 21 May 2014 10:02:54 -0700 (PDT) Received: from vapor.isi.edu (vapor.isi.edu [128.9.64.64]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C88161A085D; Wed, 21 May 2014 10:02:53 -0700 (PDT) Received: from [128.9.160.166] (abc.isi.edu [128.9.160.166]) (authenticated bits=0) by vapor.isi.edu (8.13.8/8.13.8) with ESMTP id s4LH22ie014119 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 21 May 2014 10:02:02 -0700 (PDT) Message-ID: <537CDC0A.7030303@isi.edu> Date: Wed, 21 May 2014 10:02:02 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: "Zhou, Han" , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C5845@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C5845@DEN-EXDDA-S32.corp.ebay.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/VJKWRFnWX2sp1VhuNM0B1p2mg4w Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 17:02:55 -0000 On 5/20/2014 7:17 PM, Zhou, Han wrote: > Hi Joe, > > This is an interesting topic. > >> TCP offloading is fine when the OS hands off user data, and the offload >> engine creates the entire segment. > > Existing TSO/GSO mechanisms deliver full (large) TCP segment to > "offload engine", which then create smaller segments according to > physical MTU, and recalculates checksums. This is the case even > without overlay considered. So I suppose the problem you pointed out > is not related to my change, but a general limitation for TSO/GSO, > right? It depends on what part of TCP happens in the guest OS vs. the underlying engine. If you expose the TCP API to the guest OS (the API spec'd in RFC793), and hand "Send" call data down to the engine, that's fine. However, what I think is happening is this: - the guest OS receives the "Send" call and creates a TCP segment, including TCP header and TCP options - the guest OS hands the TCP segment to the engine - the engine parses that TCP segment to create multiple outgoing segments, typically by copying the passed segment's header and options, and recalculating the fields it thinks it needs to Simply put, that's as bad as having any middlebox re-calculating TCP segments, and is guaranteed to create problems (even if the 'typical' case doesn't trip over them). The problem is that the engine's TCP interpreter may not understand all TCP header options - when (not if) that happens, what does it do? RFC793 is clear on this - when a SYN arrives with an option that isn't understood, the receiver MUST silently ignore that option. So the engine ought to have stripped out all options it doesn't understand from the first SYN sent*. But I suspect that's not what it thinks it should do - I suspect it thinks it's OK to merely copy - or pass through - options it doesn't understand. What should happen is that the engine interface should NEVER be a TCP segment formed by the guest OS. If what you want is to offload segmentation, you ought to pass the user data and TCP header (and its options) as separate parameters. (* this is why a correctly-written engine ends up reducing TCP functionality, because a connection can support only what is supported by the endpoints AND the engine [on each end]). Any option the engine doesn't support should never be allowed on the connection. > For my understanding the TCP implementation should decide whether to > use offloading or not according to the feature/options required by a TCP > connection. If the option required (such as MD5) is not supported by > offloading, the TCP stack should do the segmentation by itself instead > of utilizing offloading. That works if unknown options are assumed NOT SUPPORTED. But I still don't quite understand why you want the segmentation happening in the VM - why not pass the MTU info to the virtual interface in the guest OS and let it handle things? > In fact, the proposal in this draft should be able to alleviate the > limitation for TCP connections between VMs behind same gateways, because > in this case there is no real TCP segmentation performed by "offload > engine". > > Let me know if you have more concerns, or maybe an example of how an > option is broken by TSO/GSO, then we can check what's the current > solution in kernel. See above - and thanks, Joe From nobody Wed May 21 11:29:36 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AD0BB1A06EB for ; Wed, 21 May 2014 11:29:34 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DTCaHR51d48v for ; Wed, 21 May 2014 11:29:31 -0700 (PDT) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A511E1A06E2 for ; Wed, 21 May 2014 11:29:31 -0700 (PDT) Received: by mail-ie0-f169.google.com with SMTP id at1so2427326iec.0 for ; Wed, 21 May 2014 11:29:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=aXt2X/XmOM/zqc/m87qrvtPZHcb2vnugE2pU+r4Yrv8=; b=FcS/rr8WMCqjGvYTHI7Bjt14k9/w1TwLxIgb3ojprgVu6we6zM2JG4BrEtWeKJBCe0 fDE0SWSUK8o0xC41cw/C7y+5xN+GbPV75sSNupGWhfUcp7bpG8i9GkS2H33eXH7mrYnm OsF1cMAq2+04mDG19v2AReBoRADyQaOUPQhs6s/qw+Q0vqQDAKpk+7FhcDYEHnnOiCYF 3NF4CBl2PA4UP9ekhSxa2KOtVRlUV5tNwRu250xVicF/aEz1Ti6Gp8mQ4+z/mx5PH+XL c0pRIb5tR4mU9hSGhAQoNPYtewc5UWU3jWDjnuaxsWauGRibkjjlRALFeWwX5jWFFyOW sj6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=aXt2X/XmOM/zqc/m87qrvtPZHcb2vnugE2pU+r4Yrv8=; b=BKnrqcyPdWkU/Aw/MlIo7RF+uN7JnB4fv6uqbmLiJ0egihHZ+oiy4mdAUaa9KuYFDk HgXqo+/GXc+n8WDS/n34d/kIOQ4KLg/DWF7JRaOC7l+JFUjrR3eB/J7p7qNNUAy50XFk MNeLkssDXSSqEib1RHcu3u/RUPrFCdtmdRntMu929fCClKOaboTOh4YJ5py1gl2bnMYK 7w90M24N/NUKry7QubTatl0HnkenOBRvskAkq1s35GCLwJnDWr07p2u5HqMAXHdZmj2O 8NKvCBVd8NietY+WsxB7rOObP5mlO0nlh2lyuFD/Cc8iHgEJDbK2yhZ7SOR1ixzbTg39 2j4Q== X-Gm-Message-State: ALoCoQmlaEDj3b1VO+R/3OrTSPu5nHvKr4DE+iT72lRNMLXZ9PVyNvBQi0J4l1EA1qEmF+1nbcFF MIME-Version: 1.0 X-Received: by 10.50.20.97 with SMTP id m1mr16001920ige.28.1400696970202; Wed, 21 May 2014 11:29:30 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Wed, 21 May 2014 11:29:30 -0700 (PDT) In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109C7F16@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <9F56174078B48B459268EFF1DAB66B1A109C36BC@DEN-EXDDA-S32.corp.ebay.com> <9F56174078B48B459268EFF1DAB66B1A109C7F16@DEN-EXDDA-S32.corp.ebay.com> Date: Wed, 21 May 2014 11:29:30 -0700 Message-ID: From: Tom Herbert To: "Zhou, Han" Content-Type: multipart/alternative; boundary=047d7bd76afe59453904f9ed2d3c Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/6oJl1OkKbub6rkLP9BdzCdLTN7I Cc: "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" , Erik Nordmark Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2014 18:29:34 -0000 --047d7bd76afe59453904f9ed2d3c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Tue, May 20, 2014 at 8:07 PM, Zhou, Han wrote: > Hi Tom, > > > > > There has been a lot of work recently to get software variants (GSO and > GRO) working in Linux with tunnels. > > Are you referring UDP GRO for VXLAN? I am aware of this implementation in > kernel 3.14. > GRO and GSO for UDP tunnels, VXLAN is one use case. > The performance gain should be similar, but still requires careful > configuration of VM MTU to avoid IP fragmentation on physical MTU, rather > than decoupling from underlay. I think each of these solutions has its ow= n > advantages. > > In the case of TCP it's really the path MTU that is relevant. We really want the MSS to be the largest value which avoids fragmentation in the path= . For UDP it is less straightforward. If the VM MTU is greater than the physical MTU or even the path MTU then fragmentation is possible. Most UDP applications really want to avoid fragmentation (much worse than doing TCP segmentation), so they need to ascertain what size to send packets-- either they choose a fixed minimum value, use the MTU, or some out of band negotiation. With the occurrence of transport protocols running over UDP (such as QUIC), this becomes important. In this case, it's really hard to meaningfully decouple physical MTU (path MTU) from those of the overlay. > > > > I think this would be applicable to about all tunnel protocols. Then > you would also want to do reassembly at each hop? Sounds expensive. Once > you segment, I think you'd only only want to reassemble at the end host. > > There is no segmentation/reassembly needed at each hop, but instead > =E2=80=9Coffload=E2=80=9D hop by hop, just save the metadata to the next = hop headers. And > finally segmentation should be avoided if the destination is a VM on a > hypervisor. > > This scenario is like in Example 2 (section 5.2), just replace the > VXLAN-SOE between the 2 gateways by any other Tunnel protocol that can > support similar offloading feature, such as STT. > > > This where your proposal confuses me, we already have a lot of offloading support of protocols without having to change protocols to support offloading. I don't know if the concept of offloading is generic enough to need intrinsic protocol support, or if it is at what protocol level this should be (for instance I could conceive of this being an option in TCP to allow mega sized segments). A example where use case SOE is usable but GSO/GRO wouldn't be might be enlightening! Thanks, Tom Best regards, > > Han > > > > *From:* Tom Herbert [mailto:therbert@google.com] > *Sent:* Wednesday, May 21, 2014 10:42 AM > > *To:* Zhou, Han > *Cc:* nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark > *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > > > > > > On Tue, May 20, 2014 at 5:28 PM, Zhou, Han wrote: > > Hi Tom, > > > > Thanks for your comments. > > Yes TSO/LRO with VXLAN support should provide similar or even better > performance gains, but the mechanism proposed by this draft decouples > overlay and underlay, and it is hardware independent. > > Secondly, hardware offloading usually support TCP only (TSO). The > mechanism here can help on large UDP packet performance, also verified by > the prototype. > > BTW, how many types of off-the-shelf NIC support VXLAN offloading? Any > performance data? > > > > HW is not a requirement for this offloading. There has been a lot of > work recently to get software variants (GSO and GRO) working in Linux wit= h > tunnels. These do show 2-3x performance improvements. HW support would be > mostly an incremental improvement to that, or becomes interesting for OS > bypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and L= RO > so that it's possible to plumb use of large packets from the guest all th= e > way to the host driver. It should also be possible to plumb two VMs on th= e > same host to communicate without segmentation, i.e. output from TSO on on= e > VM becomes input for another. > > > > The advantage that I see in your draft is that it allows an intermediate > device to perform segmentation/reassembly instead of fragmentation. > > > > Likewise, setting large MTU on overlay interfaces achieves similar > result, but still the overlay/underlay decoupling issue. It is usually > advised that overlay MTU is slightly smaller than underlay to avoid > inefficient fragmentation after adding the outer header, but to achieve > really high performance between VMs, large MTU is preferred. > > > > Depends on the performance dimension to be optimized. Larger MTUs could > increase latency of high priority small packets for instance (HOL > blocking), or UDP based application might try to use MTU to decide how > large it should send it's packets to avoid fragmentation. > > > > And considering overlay <-> physical connection, path MTU discovery is > not always work. > > This kind of configuration complexity and pain-point can be resolved > simply by decoupling overlay and underlay MTU, as suggested by this draft= . > Here is an example of configuration confusion: > > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > > > Ideally, all NV tunnel protocols should support similar metadata, thus > overlay segmentation can be offloaded hop-by-hop. > > > > I think this would be applicable to about all tunnel protocols. Then you > would also want to do reassembly at each hop? Sounds expensive. Once you > segment, I think you'd only only want to reassemble at the end host. > > > > One important thing to keep in mind, and a hard lesson in real deployment > :-), segmentation offload is opportunistic and very dependent on the > conditions of the traffic. It's value can be fleeting in real workloads. > For instance imagine a host with a lot of active high throughout > connections (like a video serving) which hits a hiccup causing all > congestions windows to shrink. If the system is not provisioned for this > event it can be very hard to recover (a lot more CPU is required to achie= ve > same throughput). This differentiates a larger MTU from relying on > segmentation, the former offers more predictable CPU savings. > > > > Tom > > > > > > Best regards, > > Han > > > > *From:* Tom Herbert [mailto:therbert@google.com] > *Sent:* Wednesday, May 21, 2014 2:44 AM > *To:* Zhou, Han > *Cc:* nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark > *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > > Hi Zou, a couple of questions inline. > > > > On Mon, May 19, 2014 at 8:01 PM, Zhou, Han wrote: > > Hi, > > We have updated the VXLAN-SOE draft according to earlier comments. Now it > is fully compatible with VXLAN-GPE. And some examples are added for bette= r > understanding > > > > A prototype is also implemented here (patch based on Open vSwitch): > > https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85= fa050f7dd2be > > netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G > interfaces: > > Before the change: 2.62 Gbits/sec > After the change: 6.68 Gbits/sec > Speedup is ~250%. > > Can you provide some more details on this benefit? It seems like plain > TSO/LRO that understands encapsulation should provide similar benefits wh= en > going between hosts. > > > > > > The patch attracted some interests in OVS community, but since this RFC > draft is in very early stage so it is regarded as inappropriate by Jesse = to > apply the change to OVS tree. > The discuss mail-thread: > http://openvswitch.org/pipermail/discuss/2014-May/013981.html > http://openvswitch.org/pipermail/discuss/2014-May/013898.html > > So we would like to request a review here by NVO3/TOFOO groups and VXLAN > authors: is this VXLAN extension is worth formally put into VXLAN as a > standard, so that more people can benefit from it? > > Could you get the same effect by setting larger MTUs on the overlay > network interface and relying in path MTU discovery when going over > physical network? > > > > Best regards, > Han > > -----Original Message----- > From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of > internet-drafts@ietf.org > Sent: Friday, May 02, 2014 8:09 PM > To: i-d-announce@ietf.org > Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > > Title : Segmentation Offloading Extension for VXLAN > Authors : Han Zhou > Chengyuan Li > Filename : draft-zhou-li-vxlan-soe-01.txt > Pages : 13 > Date : 2014-05-02 > > Abstract: > Segmentation offloading is nowadays common in network stack > implementation and well supported by para-virtualized network device > drivers for virtual machine (VM)s. This draft describes an extension > to Virtual eXtensible Local Area Network (VXLAN) so that segmentation > can be decoupled from physical/underlay networks and offloaded > further to the remote end-point thus improving data-plane performance > for VMs running on top of overlay networks. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ > > There's also a htmlized version available at: > http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 > > A diff from the previous version is available at: > http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe-01 > > > Please note that it may take a couple of minutes from the time of > submission > until the htmlized version and diff are available at tools.ietf.org. > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draftdirectories: > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > > > --047d7bd76afe59453904f9ed2d3c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Tue, May 20, 2014 at 8:07 PM, Zhou, Han <hzhou8@ebay.com><= /span> wrote:

Hi Tom,=

=C2= =A0

> There has been a lot of work recently to get software variants (GSO and GRO) working in Linux with= tunnels.

Are = you referring UDP GRO for VXLAN? I am aware of this implementation in kernel 3.14.


GRO and GSO for UDP tunnels, VXLAN is one use case.
=
=C2=A0

The performance gain should b= e similar, but still requires careful configuration of VM MTU to avoid IP f= ragmentation on physical MTU, rather than decoupling from underlay. I think= each of these solutions has its own advantages.


In the case of T= CP it's really the path MTU that is relevant. We really want the MSS to= be the largest value which avoids fragmentation in the path.

For UDP it is less straightforward. If the VM MTU is gr= eater than the physical MTU or even the path MTU then fragmentation is poss= ible. Most UDP applications really want to avoid fragmentation (much worse = than doing TCP segmentation), so they need to ascertain what size to send p= ackets-- either they choose a fixed minimum value, use the MTU, or some out= of band negotiation. With the occurrence of transport protocols running ov= er UDP (such as QUIC), this becomes important. In this case, it's reall= y hard to meaningfully decouple physical MTU (path MTU) from those of the o= verlay.

=C2=A0

> I think this would be applicable to about all t= unnel protocols. Then you would also want to do reassembly at each hop? Sou= nds expensive. Once you segment, I think you'd only only want to reassemble at the end host.<= /p>

Ther= e is no segmentation/reassembly needed at each hop, but instead =E2=80=9Coffload=E2=80=9D hop by hop, just save the metad= ata to the next hop headers. And finally segmentation should be avoided if = the destination is a VM on a hypervisor.

This scena= rio is like in Example 2 (section 5.2), just replace the VXLAN-SOE between the 2 gateways by any other Tunnel protocol = that can support similar offloading feature, such as STT.

=C2= =A0

This where your proposal confus= es me, we already have a lot of offloading support of protocols without hav= ing to change protocols to support offloading. I don't know if the conc= ept of offloading is generic enough to need intrinsic protocol support, or = if it is at what protocol level this should be (for instance I could concei= ve of this being an option in TCP to allow mega sized segments).

A example where use case SOE is usable but GSO/GRO woul= dn't be might be enlightening!

Thanks,
Tom

Best regar= ds,

Han=

=C2= =A0

From: Tom Herbert [mailto:therbert@google.com]
Sent: Wednesday, May 21, 2014 10:42 AM


To: Zhou, Han
Cc: nvo3@ietf.org= ; tofoo@ietf.org; draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; draft-= zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

=C2=A0

=C2=A0

= =C2=A0

On Tue, May 20, 2014 at 5:28 PM= , Zhou, Han <hzhou8= @ebay.com> wrote:

Hi Tom,

=C2=A0

Thanks for= your comments.

Yes TSO/LR= O with VXLAN support should provide similar or even better performance gain= s, but the mechanism proposed by this draft decouples overlay and underlay, a= nd it is hardware independent.

Secondly, = hardware offloading usually support TCP only (TSO). The mechanism here can help on large UDP packet performance, also verified by the prototype.=C2=A0=C2=A0

BTW, how m= any types of off-the-shelf NIC support VXLAN offloading? Any performance da= ta?

=C2=A0

HW is not a requirement for thi= s offloading. There has been a lot of work recently to get software variant= s (GSO and GRO) working in Linux with tunnels. These do show 2-3x performan= ce improvements. HW support would be mostly an incremental improvement to that, or becomes interesting for OS b= ypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and LRO = so that it's possible to plumb use of large packets from the guest all = the way to the host driver. It should also be possible to plumb two VMs on the same host to communicate without = segmentation, i.e. output from TSO on one VM becomes input for another.=C2= =A0

=C2=A0

The advantage that I see in you= r draft is that it allows an intermediate device to perform segmentation/re= assembly instead of fragmentation.

=C2=A0

Likewise, = setting large MTU on overlay interfaces achieves similar result, but still the overlay/underlay decoupling issue. It is usually advised that overlay = MTU is slightly smaller than underlay to avoid inefficient fragmentation af= ter adding the outer header, but to achieve really high performance between= VMs, large MTU is preferred.

=C2=A0

Depends on the performance dime= nsion to be optimized. Larger MTUs could increase latency of high priority = small packets for instance (HOL blocking), or UDP based application might t= ry to use MTU to decide how large it should send it's packets to avoid fragmentation.<= /p>

=C2=A0

And consid= ering overlay <-> physical connection, path MTU discovery is not alwa= ys work.=C2=A0

This kind = of configuration complexity and pain-point can be resolved simply by decoup= ling overlay and underlay MTU, as suggested by this draft. Here is an example o= f configuration confusion:=

http://openvswitch.org/pipermail/discuss/2014-May/013898.html

=C2=A0

Ideally, a= ll NV tunnel protocols should support similar metadata, thus overlay segmen= tation can be offloaded hop-by-hop.

=C2=A0

I think this would be applicabl= e to about all tunnel protocols. Then you would also want to do reassembly = at each hop? Sounds expensive. Once you segment, I think you'd only onl= y want to reassemble at the end host.

=C2=A0

One important thing to keep in = mind, and a hard lesson in real deployment :-), segmentation offload is opp= ortunistic and very dependent on the conditions of the traffic. It's va= lue can be fleeting in real workloads. For instance imagine a host with a lot of active high throughout connections (= like a video serving) which hits a hiccup causing all congestions windows t= o shrink. If the system is not provisioned for this event it can be very ha= rd to recover (a lot more CPU is required to achieve same throughput). This differentiates a larger MTU fro= m relying on segmentation, the former offers more predictable CPU savings.<= u>

=C2=A0

Tom

=C2=A0

=C2=A0

Best regar= ds,

Han=

=C2=A0

From: T= om Herbert [mailto:therbert@google.com]
Sent: Wednesday, May 21, 2014 2:44 AM
To: Zhou, Han
Cc: nvo3@ietf.org= ; tofoo@ietf.org; draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; draft-zhou-li-vxlan-soe@tools.ietf.org; Erik Nordmark
Subject: Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

=C2=A0

Hi Zou, a couple of questions i= nline.

= =C2=A0

On Mon, May 19, 2014 at 8:01 PM= , Zhou, Han <hzhou8= @ebay.com> wrote:

Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it i= s fully compatible with VXLAN-GPE. And some examples are added for better u= nderstanding=C2=A0

=C2=A0

= A prototype is also implemented here (patch based on Open vSwitch):
https://github.com/hzhou8/openvswi= tch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be

netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G int= erfaces:

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.

Can you provide some more detai= ls on this benefit? It seems like plain TSO/LRO that understands encapsulat= ion should provide similar benefits when going between hosts.

=C2=A0<= /span>

=C2=A0

= The patch attracted some interests in OVS community, but since this RFC dra= ft is in very early stage so it is regarded as inappropriate by Jesse to ap= ply the change to OVS tree.
The discuss mail-thread:
http://openvswitch.org/pipermail/discuss/2014-May/013981.h= tml
http://openvswitch.org/pipermail/discuss/2014-May/013898.h= tml

So we would like to request a review here by NVO3/TOFOO groups and VXLAN au= thors: is this VXLAN extension is worth formally put into VXLAN as a standa= rd, so that more people can benefit from it?

Could you get the same effect b= y setting larger MTUs on the overlay network interface and relying in path = MTU discovery when going over physical network?

=C2=A0

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of internet-draf= ts@ietf.org
Sent: Friday, May 02, 2014 8:09 PM
To: i-d-announce= @ietf.org
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt


A New Internet-Draft is available from the on-line Internet-Drafts director= ies.


=C2=A0 =C2=A0 =C2=A0 =C2=A0 Title =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Segm= entation Offloading Extension for VXLAN
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Authors =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Han Zhou<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Chengyuan Li
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Filename =C2=A0 =C2=A0 =C2=A0 =C2=A0: draft-zho= u-li-vxlan-soe-01.txt
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Pages =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 13 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:= 2014-05-02

Abstract:
=C2=A0 =C2=A0Segmentation offloading is nowadays common in network stack =C2=A0 =C2=A0implementation and well supported by para-virtualized network = device
=C2=A0 =C2=A0drivers for virtual machine (VM)s. This draft describes an ext= ension
=C2=A0 =C2=A0to Virtual eXtensible Local Area Network (VXLAN) so that segme= ntation
=C2=A0 =C2=A0can be decoupled from physical/underlay networks and offloaded=
=C2=A0 =C2=A0further to the remote end-point thus improving data-plane perf= ormance
=C2=A0 =C2=A0for VMs running on top of overlay networks.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/<= br>
There's also a htmlized version available at:
http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vxlan-soe= -01


Please note that it may take a couple of minutes from the time of submissio= n
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp= .ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
I-D-Announce@iet= f.org
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or = ftp://ftp.ietf.org/ietf/1shadow-sites.txt

=C2=A0

=C2=A0


--047d7bd76afe59453904f9ed2d3c-- From nobody Wed May 21 19:44:58 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E8CBC1A0079; Wed, 21 May 2014 19:44:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -23.152 X-Spam-Level: X-Spam-Status: No, score=-23.152 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gp4X4TPOYMwP; Wed, 21 May 2014 19:44:54 -0700 (PDT) Received: from den-mipot-001.corp.ebay.com (den-mipot-001.corp.ebay.com [216.113.175.152]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1A7291A007B; Wed, 21 May 2014 19:44:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ebay.com; i=@ebay.com; q=dns/txt; s=ebaycorp; t=1400726693; x=1432262693; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=/ZMLxm7SQYVJjczojPiU6qNHS+qt7vvdwdVe8gHTvCo=; b=mSXFM5JDxqzCMwQv6U1Fx0t2esnceDJFEJ+ohQncnwHH2mU0FvE1G7uM VObTTqhwNTYXhETu3f5XrJ2Ff/sDvOqaqOWe9PIMZrIXlJeOHKDxBQ7o9 /o6CO2idyUMmYGPq0U+wlSi0z96RO7B1xdSEA4bvqWcNDqs8bkg6X+Z0S c=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos;i="4.98,884,1392192000"; d="scan'208";a="51091600" Received: from den-vteml-001.corp.ebay.com (HELO DEN-EXMHT-004.corp.ebay.com) ([10.101.112.212]) by den-mipot-001.corp.ebay.com with ESMTP; 21 May 2014 19:44:53 -0700 Received: from DEN-EXDDA-S32.corp.ebay.com ([fe80::e420:c190:6f77:31f7]) by DEN-EXMHT-004.corp.ebay.com ([fe80::a487:c570:9abc:bb59%14]) with mapi id 14.03.0174.001; Wed, 21 May 2014 20:44:52 -0600 From: "Zhou, Han" To: Joe Touch , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Thread-Topic: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt Thread-Index: AQHPZf9ognVu1+LRHU+2rgpg3xG8G5tI3oOggAFb9YCAABDEMIAAcWAA//+gOXCAAWiGAIAANQ2A Date: Thu, 22 May 2014 02:44:52 +0000 Message-ID: <9F56174078B48B459268EFF1DAB66B1A109CC2A7@DEN-EXDDA-S32.corp.ebay.com> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C5845@DEN-EXDDA-S32.corp.ebay.com> <537CDC0A.7030303@isi.edu> In-Reply-To: <537CDC0A.7030303@isi.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.241.19.243] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter: Scanned den1 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/-osvaZhDvtqZN88tzrfmyp3hpao Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 02:44:56 -0000 Hi Joe, Thanks for your explain, and in theory I tend to agree with you. However, look at the real world implementation of offload, e.g. Linux kerne= l, GSO is implemented in the way I described: a whole TCP packet (including= headers and options) are generated by the TCP layer, and segmentation is p= erformed as late as possible, depending on the FEATURES supported by net-de= vice: 1) TSO is NOT supported by net-device then GSO software segmentation is per= formed (tcp_gso_segment()) 2) TSO is supported by NIC, then segmentation is offloaded to NIC hardware.= (in this case tcp_dump will not capture small packets but only the large p= ackets before segmentation) 3) It is in Guest OS and TSO is supported by the virtual net-device, then s= egmentation is offloaded to host OS.=20 And my change is specific for 3): without the change segmentation is perfor= med according to the MSS specified by guest OS, and with the change this s= egmentation is skipped. As you can see the compatibility problem you mentioned, if it is a real pro= blem, is not introduced by my change. Because of this, we might discuss it somewhere else, such as linux kernel c= ommunity. I don't think current Linux kernel GSO implementation addresses the problem= you mentioned (I could be wrong). Do you know any example of *correct* imp= lementation of TCP offloading? Or it could be a TODO in linux kernel. Best regards, Han > -----Original Message----- > From: Joe Touch [mailto:touch@isi.edu] > Sent: Thursday, May 22, 2014 1:02 AM > To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; > draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; > draft-zhou-li-vxlan-soe@tools.ietf.org > Cc: Erik Nordmark; Tom Herbert > Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt >=20 >=20 >=20 > On 5/20/2014 7:17 PM, Zhou, Han wrote: > > Hi Joe, > > > > This is an interesting topic. > > > >> TCP offloading is fine when the OS hands off user data, and the offloa= d > >> engine creates the entire segment. > > > > Existing TSO/GSO mechanisms deliver full (large) TCP segment to > > "offload engine", which then create smaller segments according to > > physical MTU, and recalculates checksums. This is the case even > > without overlay considered. So I suppose the problem you pointed out > > is not related to my change, but a general limitation for TSO/GSO, > > right? >=20 > It depends on what part of TCP happens in the guest OS vs. the > underlying engine. If you expose the TCP API to the guest OS (the API > spec'd in RFC793), and hand "Send" call data down to the engine, that's > fine. >=20 > However, what I think is happening is this: >=20 > - the guest OS receives the "Send" call and creates a TCP > segment, including TCP header and TCP options >=20 > - the guest OS hands the TCP segment to the engine >=20 > - the engine parses that TCP segment to create multiple > outgoing segments, typically by copying the passed segment's > header and options, and recalculating the fields it > thinks it needs to >=20 > Simply put, that's as bad as having any middlebox re-calculating TCP > segments, and is guaranteed to create problems (even if the 'typical' > case doesn't trip over them). >=20 > The problem is that the engine's TCP interpreter may not understand all > TCP header options - when (not if) that happens, what does it do? >=20 > RFC793 is clear on this - when a SYN arrives with an option that isn't > understood, the receiver MUST silently ignore that option. >=20 > So the engine ought to have stripped out all options it doesn't > understand from the first SYN sent*. But I suspect that's not what it > thinks it should do - I suspect it thinks it's OK to merely copy - or > pass through - options it doesn't understand. >=20 > What should happen is that the engine interface should NEVER be a TCP > segment formed by the guest OS. If what you want is to offload > segmentation, you ought to pass the user data and TCP header (and its > options) as separate parameters. >=20 > (* this is why a correctly-written engine ends up reducing TCP > functionality, because a connection can support only what is supported > by the endpoints AND the engine [on each end]). Any option the engine > doesn't support should never be allowed on the connection. >=20 > > For my understanding the TCP implementation should decide whether to > > use offloading or not according to the feature/options required by a TC= P > > connection. If the option required (such as MD5) is not supported by > > offloading, the TCP stack should do the segmentation by itself instead > > of utilizing offloading. >=20 > That works if unknown options are assumed NOT SUPPORTED. >=20 > But I still don't quite understand why you want the segmentation > happening in the VM - why not pass the MTU info to the virtual interface > in the guest OS and let it handle things? >=20 > > In fact, the proposal in this draft should be able to alleviate the > > limitation for TCP connections between VMs behind same gateways, becaus= e > > in this case there is no real TCP segmentation performed by "offload > > engine". > > > > Let me know if you have more concerns, or maybe an example of how an > > option is broken by TSO/GSO, then we can check what's the current > > solution in kernel. >=20 > See above - and thanks, >=20 > Joe >=20 From nobody Thu May 22 11:17:08 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 904FD1A024B; Thu, 22 May 2014 11:17:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V_QvJWK99Xwj; Thu, 22 May 2014 11:17:06 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0B5871A027E; Thu, 22 May 2014 11:17:06 -0700 (PDT) Received: from [128.9.160.81] (nib.isi.edu [128.9.160.81]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s4MIGZxM016425 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 22 May 2014 11:16:36 -0700 (PDT) Message-ID: <537E3F03.2070903@isi.edu> Date: Thu, 22 May 2014 11:16:35 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: "Zhou, Han" , "nvo3@ietf.org" , "tofoo@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C5845@DEN-EXDDA-S32.corp.ebay.com> <537CDC0A.7030303@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109CC2A7@DEN-EXDDA-S32.corp.ebay.com> In-Reply-To: <9F56174078B48B459268EFF1DAB66B1A109CC2A7@DEN-EXDDA-S32.corp.ebay.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/SLf6KU_NTqr-m7WicvAtxhiLKV4 Cc: Erik Nordmark , Tom Herbert Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 18:17:07 -0000 On 5/21/2014 7:44 PM, Zhou, Han wrote: > Hi Joe, > > Thanks for your explain, and in theory I tend to agree with you. > However, look at the real world implementation of offload, e.g. Linux kernel, GSO is implemented in the way I described: a whole TCP packet (including headers and options) are generated by the TCP layer, and segmentation is performed as late as possible, depending on the FEATURES supported by net-device: > 1) TSO is NOT supported by net-device then GSO software segmentation is performed (tcp_gso_segment()) > 2) TSO is supported by NIC, then segmentation is offloaded to NIC hardware. (in this case tcp_dump will not capture small packets but only the large packets before segmentation) > 3) It is in Guest OS and TSO is supported by the virtual net-device, then segmentation is offloaded to host OS. > > And my change is specific for 3): without the change segmentation is performed according to the MSS specified by guest OS, and with the change this segmentation is skipped. > > As you can see the compatibility problem you mentioned, if it is a real problem, is not introduced by my change. Perhaps, but dealing with guest OS/TSO interactions could be designed as a correct example, rather than replicating the failure of other implementers. > Because of this, we might discuss it somewhere else, such as linux kernel community. > I don't think current Linux kernel GSO implementation addresses the problem you mentioned (I could be wrong). Do you know any example of *correct* implementation of TCP offloading? Or it could be a TODO in linux kernel. I haven't studied TSO implementations; there are too many, and many are proprietary. Joe From nobody Thu May 22 11:43:56 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B98511A0298 for ; Thu, 22 May 2014 11:43:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jj05VvKrTa36 for ; Thu, 22 May 2014 11:43:43 -0700 (PDT) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9D8151A0276 for ; Thu, 22 May 2014 11:43:43 -0700 (PDT) Received: by mail-ie0-f181.google.com with SMTP id rp18so2345142iec.26 for ; Thu, 22 May 2014 11:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vW2a75i7A3ks6Y74TX8xDSS4Rf5zVvdqcHwyayVcF9U=; b=DqCR34lYfwT/fOmnXAcL81CpFNGM3juaTawbqXCtzyZDjUPBpzBBiyhFnn6sqiLLTV NSqG9cxsT4/pu/5ZUsSWcAF/cy1y2H2q5Dhjn8LYV/J28hfMSmSxfzKIsBPnhL8MgseH NUV6ViLtWFmE9cMVCQUJ3/+HZh2/lagWUHKXFbCBvqUomTh+mgyks6IzdbwgcgIAoYLs lVsjOOLrzwscot1gVp/qTkpWceBTptwhA4gvvHI1sMMSp3EOZAhgmyPxreMUj61i3ahm fELNOQBrkslTrQBiwTGBN81JRW92sixKhV5+ySWLjJ3SkPOLif1oJH3/3A3e8LCv1AsA f/fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vW2a75i7A3ks6Y74TX8xDSS4Rf5zVvdqcHwyayVcF9U=; b=GTRCujGRTRNWD4n1xjHZCP9YUUj7Hx5hlaG1wy3Hac5RFsvw9edsnYg5kIv7wvIZE7 II6GvnduZ9R5xxQpGJjpMhEkkPPmJH+CkJ43tmdLSharx9zPxt33pdOa4HgITIAx8M8N mMJYAxM15OkK4Jzf4MyqfZP99j3Bcsc4lnlf3ohCnaLPtH5vITPo6eivEWSRxpv7Dz5X hsPKm5P5l3sts53hHvpTks1V8kPpRkwHvzV8nU7dv+df7QWfCe76Enipbo8HwY0zsl65 B0u9qUYSdrZT13FUy/qlVaVajcSrpcHg2YfqH8bvzJUYXDOprnYBXzOb07zXzCG7kHp0 7pxw== X-Gm-Message-State: ALoCoQk4w5avRGrcqzakY0+/+evvbNJD7EGgonqrphHyi5NXfs49DQckOKBNzVbOjkggK/8BYqf4 MIME-Version: 1.0 X-Received: by 10.50.141.232 with SMTP id rr8mr679471igb.48.1400784221929; Thu, 22 May 2014 11:43:41 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Thu, 22 May 2014 11:43:41 -0700 (PDT) In-Reply-To: <537BFDF4.9030300@isi.edu> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> Date: Thu, 22 May 2014 11:43:41 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: multipart/alternative; boundary=089e0129555cf49d8204fa017dbc Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/DsM5Qbswabmhe7l8gr9BJE9Qz0w Cc: "Zhou, Han" , Erik Nordmark , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 18:43:46 -0000 --089e0129555cf49d8204fa017dbc Content-Type: text/plain; charset=UTF-8 On Tue, May 20, 2014 at 6:14 PM, Joe Touch wrote: > Hi, Han, > > This helps, but doesn't quite address my concern. > > TCP offloading is fine when the OS hands off user data, and the offload > engine creates the entire segment. > > This would essentially be TOE. > The situation you're describing seems to be a hybrid, where the guest OS > makes a TCP segment, and then something lower (in the VM system) parses > that segment to create one or more segments on the wire. > > This how Linux and probably about every NIC implements TCP segmentation offload. A large TCP segment is broken up into MSS sized segments (MSS is provided in ancillary data). The process replicates the IP/TCP headers per packet, adjusts the length, sequence numbers, and computes checksum for each segment. There's no special handling of any options, it is assumed they can be replicated in each segment. If that happens, it will be incompatible with a number of existing TCP > options, and will also cause side-effects with ACK clocking, timeouts, and > more than a few other TCP features. > Although I appreciate the goal of efficiency and speed, there is a severe > compatibility issue that doesn't seem to be addressed. > > TCP segmentation offload is already widely deployed. Since this draft would be using the same mechanism it's unlikely to create new compatibility issues except for the fact that the segmentation might be deferred to an off host entity which is a new concept that would probably have side effects. Tom We can discuss this off-list if useful, FWIW. > > Joe > > > On 5/20/2014 6:01 PM, Zhou, Han wrote: > >> Hi Joe, >> >> Thanks for your comment. >> >> Yes you are right that "length" fields in packet headers can be regarded >> as hard limit of MTU, but here in the draft we were referring interface >> MTU. We will make it more precise in next versions. >> >> For your question, it seems there are misunderstandings. It is always >> Guest OS (VM) handles the TCP, but segmentation is offloaded to host. Let >> me explain the code change: >> - TX side: >> -- before the change: >> TCP segmentation offloaded by TSO of Guest OS virt-driver from guest >> to host, MSS carried by GSO metadata in skbuff. >> VXLAN layer add encapsulation, and overlay TCP segmentation is carried >> right before sending to host IP layer, which is the idea of GSO - segment >> at the latest point. >> Host do IP fragmentation only if overlay segment + outer header exceed >> physical interface MTU. (e.g. when both guest MTU and host MTU are >> configured to 1500) >> -- after the change: >> TCP segmentation offloaded by TSO of Guest OS virt-driver from guest >> to host, MSS carried by GSO metadata in skbuff. >> VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. So >> overlay TCP segmentation is skipped. >> Host do IP fragmentation according to physical interface MTU. >> >> -RX side: >> -- before the change: >> Host do IP reassembly if necessary. (e.g. when both guest MTU and host >> MTU are configured to 1500) >> Overlay TCP segments are decapsulated and delivered all the way to >> guest, and guest OS do TCP handling. (high cost here) >> -- after the change: >> Host do IP reassembly, and large packets decapsulated and delivered to >> guest, and guest OS do TCP handling. (cost reduced here because of reduced >> number of packets) >> >> I hope this clarifies. >> >> Best regards, >> Han >> >> -----Original Message----- >>> From: Joe Touch [mailto:touch@isi.edu] >>> Sent: Wednesday, May 21, 2014 1:29 AM >>> To: Zhou, Han; nvo3@ietf.org; tofoo@ietf.org; >>> draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; >>> draft-zhou-li-vxlan-soe@tools.ietf.org >>> Cc: Erik Nordmark; Tom Herbert >>> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt >>> >>> Hi, all, >>> >>> I had a comment and a question: >>> >>> Comment - (from the doc) overlays do have a hard MTU limit; it is the >>> limit of the encapsulation mechanism. E.g., without additional layers, >>> for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max - >>> (min IP header + UDP header)). Using additional adaptation layers, this >>> could be larger (e.g., see SEAL). >>> >>> Question - the code appears to have the VXLAN layer do the >>> fragmentation, with the OS layer implementing the rest of TCP. There are >>> a lot of interactions, notably: >>> >>> - any mechanism outside of the TCP source and TCP destination >>> that interprets the TCP header will result in a decrease in >>> functionality >>> i.e., the TCP connection will support only the >>> intersection of options and features supported >>> by the source, dest, *and* VXLAN layers >>> >>> (rather than being limited only by the >>> source-dest pair) >>> >>> - if passed a full TCP segment, this mechanism will be >>> incompatible with TCP security (e.g., TCP MD5, TCP-AO, and >>> the results of the TCPCRYPT WG. >>> >>> I'm not quite sure from your doc whether you're re-segmenting TCP >>> segments, or merely collecting them for aggregate transit (e.g., as is >>> done in burst-mode Ethernet). >>> >>> Can you please clarify? >>> >>> Joe >>> >>> >>> On 5/19/2014 8:01 PM, Zhou, Han wrote: >>> >>>> Hi, >>>> >>>> We have updated the VXLAN-SOE draft according to earlier comments. Now >>>> it >>>> >>> is fully compatible with VXLAN-GPE. And some examples are added for >>> better >>> understanding. >>> >>>> >>>> A prototype is also implemented here (patch based on Open vSwitch): >>>> >>>> https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d >>> 4ff85fa050f7dd2be >>> >>>> >>>> netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G >>>> >>> interfaces: >>> >>>> >>>> Before the change: 2.62 Gbits/sec >>>> After the change: 6.68 Gbits/sec >>>> Speedup is ~250%. >>>> >>>> The patch attracted some interests in OVS community, but since this RFC >>>> draft >>>> >>> is in very early stage so it is regarded as inappropriate by Jesse to >>> apply the >>> change to OVS tree. >>> >>>> The discuss mail-thread: >>>> http://openvswitch.org/pipermail/discuss/2014-May/013981.html >>>> http://openvswitch.org/pipermail/discuss/2014-May/013898.html >>>> >>>> So we would like to request a review here by NVO3/TOFOO groups and VXLAN >>>> >>> authors: is this VXLAN extension is worth formally put into VXLAN as a >>> standard, >>> so that more people can benefit from it? >>> >>>> >>>> Best regards, >>>> Han >>>> >>>> -----Original Message----- >>>> From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of >>>> >>> internet-drafts@ietf.org >>> >>>> Sent: Friday, May 02, 2014 8:09 PM >>>> To: i-d-announce@ietf.org >>>> Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt >>>> >>>> >>>> A New Internet-Draft is available from the on-line Internet-Drafts >>>> directories. >>>> >>>> >>>> Title : Segmentation Offloading Extension for VXLAN >>>> Authors : Han Zhou >>>> Chengyuan Li >>>> Filename : draft-zhou-li-vxlan-soe-01.txt >>>> Pages : 13 >>>> Date : 2014-05-02 >>>> >>>> Abstract: >>>> Segmentation offloading is nowadays common in network stack >>>> implementation and well supported by para-virtualized network >>>> device >>>> drivers for virtual machine (VM)s. This draft describes an >>>> extension >>>> to Virtual eXtensible Local Area Network (VXLAN) so that >>>> segmentation >>>> can be decoupled from physical/underlay networks and offloaded >>>> further to the remote end-point thus improving data-plane >>>> performance >>>> for VMs running on top of overlay networks. >>>> >>>> >>>> The IETF datatracker status page for this draft is: >>>> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/ >>>> >>>> There's also a htmlized version available at: >>>> http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01 >>>> >>>> A diff from the previous version is available at: >>>> http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01 >>>> >>>> >>>> Please note that it may take a couple of minutes from the time of >>>> submission >>>> until the htmlized version and diff are available at tools.ietf.org. >>>> >>>> Internet-Drafts are also available by anonymous FTP at: >>>> ftp://ftp.ietf.org/internet-drafts/ >>>> >>>> _______________________________________________ >>>> I-D-Announce mailing list >>>> I-D-Announce@ietf.org >>>> https://www.ietf.org/mailman/listinfo/i-d-announce >>>> Internet-Draft directories: http://www.ietf.org/shadow.html >>>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt >>>> >>>> _______________________________________________ >>>> Tofoo mailing list >>>> Tofoo@ietf.org >>>> https://www.ietf.org/mailman/listinfo/tofoo >>>> >>>> --089e0129555cf49d8204fa017dbc Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Tue, May 20, 2014 at 6:14 PM, Joe Touch <touch@isi.edu> wrote:
Hi, Han,

This helps, but doesn't quite address my concern.

TCP offloading is fine when the OS hands off user data, and the offload eng= ine creates the entire segment.

This would essentially be TOE.
=C2=A0
=
The situation you're describing seems to be a hybrid, where the guest O= S makes a TCP segment, and then something lower (in the VM system) parses t= hat segment to create one or more segments on the wire.

This how Linux and probably about every NIC implement= s TCP segmentation offload. A large TCP segment is broken up into MSS sized= segments (MSS is provided in ancillary data). The process replicates the I= P/TCP headers per packet, adjusts the length, sequence numbers, and compute= s checksum for each segment. =C2=A0There's no special handling of any o= ptions, it is assumed they can be replicated in each segment.


If that happens, it will be incompatible with a number of existing TCP opti= ons, and will also cause side-effects with ACK clocking, timeouts, and more= than a few other TCP features.=C2=A0
=C2=A0
Although I appreciate the goal of efficiency and speed, there is a severe c= ompatibility issue that doesn't seem to be addressed.

TCP segmentation offload is already widely deployed. = Since this draft would be using the same mechanism it's unlikely to cre= ate new compatibility issues except for the fact that the segmentation migh= t be deferred to an off host entity which is a new concept that would proba= bly have side effects.

Tom

We can discuss this off-list if useful, FWIW.

Joe


On 5/20/2014 6:01 PM, Zhou, Han wrote:
Hi Joe,

Thanks for your comment.

Yes you are right that "length" fields in packet headers can be r= egarded as hard limit of MTU, but here in the draft we were referring inter= face MTU. We will make it more precise in next versions.

For your question, it seems there are misunderstandings. It is always Guest= OS (VM) handles the TCP, but segmentation is offloaded to host. Let me exp= lain the code change:
- TX side:
-- before the change:
=C2=A0 =C2=A0TCP segmentation offloaded by TSO of Guest OS virt-driver from= guest to host, MSS carried by GSO metadata in skbuff.
=C2=A0 =C2=A0VXLAN layer add encapsulation, and overlay TCP segmentation is= carried right before sending to host IP layer, which is the idea of GSO - = segment at the latest point.
=C2=A0 =C2=A0Host do IP fragmentation only if overlay segment + outer heade= r exceed physical interface MTU. (e.g. when both guest MTU and host MTU are= configured to 1500)
-- after the change:
=C2=A0 =C2=A0TCP segmentation offloaded by TSO of Guest OS virt-driver from= guest to host, MSS carried by GSO metadata in skbuff.
=C2=A0 =C2=A0VXLAN layer removes GSO metadata and store it in VXLAN-SOE hea= der. So overlay TCP segmentation is skipped.
=C2=A0 =C2=A0Host do IP fragmentation according to physical interface MTU.<= br>
-RX side:
-- before the change:
=C2=A0 =C2=A0Host do IP reassembly if necessary. (e.g. when both guest MTU = and host MTU are configured to 1500)
=C2=A0 =C2=A0Overlay TCP segments are decapsulated and delivered all the wa= y to guest, and guest OS do TCP handling. (high cost here)
-- after the change:
=C2=A0 =C2=A0Host do IP reassembly, and large packets decapsulated and deli= vered to guest, and guest OS do TCP handling. (cost reduced here because of= reduced number of packets)

I hope this clarifies.

Best regards,
Han

-----Original Message-----
From: Joe Touch [mailto:= touch@isi.edu]
Sent: Wednesday, May 21, 2014 1:29 AM
To: Zhou, Han; nvo3@ietf= .org; tofoo@ietf.or= g;
draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org; draft-zhou-li-vxlan-soe@tools.ietf.org
Cc: Erik Nordmark; Tom Herbert
Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

Hi, all,

I had a comment and a question:

Comment - (from the doc) overlays do have a hard MTU limit; it is the
limit of the encapsulation mechanism. E.g., without additional layers,
for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max -
(min IP header + UDP header)). Using additional adaptation layers, this
could be larger (e.g., see SEAL).

Question - the code appears to have the VXLAN layer do the
fragmentation, with the OS layer implementing the rest of TCP. There are a lot of interactions, notably:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 - any mechanism outside of the TCP source and T= CP destination
=C2=A0 =C2=A0 =C2=A0 =C2=A0 that interprets the TCP header will result in a= decrease in
=C2=A0 =C2=A0 =C2=A0 =C2=A0 functionality
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 i.e., the TCP conne= ction will support only the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 intersection of opt= ions and features supported
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 by the source, dest= , *and* VXLAN layers

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (rather than being = limited only by the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 source-dest pair)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 - if passed a full TCP segment, this mechanism = will be
=C2=A0 =C2=A0 =C2=A0 =C2=A0 incompatible with TCP security (e.g., TCP MD5, = TCP-AO, and
=C2=A0 =C2=A0 =C2=A0 =C2=A0 the results of the TCPCRYPT WG.

I'm not quite sure from your doc whether you're re-segmenting TCP segments, or merely collecting them for aggregate transit (e.g., as is
done in burst-mode Ethernet).

Can you please clarify?

Joe


On 5/19/2014 8:01 PM, Zhou, Han wrote:
Hi,

We have updated the VXLAN-SOE draft according to earlier comments. Now it
is fully compatible with VXLAN-GPE. And some examples are added for better<= br> understanding.

A prototype is also implemented here (patch based on Open vSwitch):

https://github.com/hzhou8/openvswitch/commit= /9a7deb8b432ce83a9c09d7d
4ff85fa050f7dd2be

netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G
interfaces:

Before the change: 2.62 Gbits/sec
After the change: 6.68 Gbits/sec
Speedup is ~250%.

The patch attracted some interests in OVS community, but since this RFC dra= ft
is in very early stage so it is regarded as inappropriate by Jesse to apply= the
change to OVS tree.
The discuss mail-thread:
http://openvswitch.org/pipermail/discuss/2014-May/<= u>013981.html
http://openvswitch.org/pipermail/discuss/2014-May/<= u>013898.html

So we would like to request a review here by NVO3/TOFOO groups and VXLAN
authors: is this VXLAN extension is worth formally put into VXLAN as a stan= dard,
so that more people can benefit from it?

Best regards,
Han

-----Original Message-----
From: I-D-Announce [mailto:i-d-announce-bounces@ietf.org] On Behalf Of
internet-draf= ts@ietf.org
Sent: Friday, May 02, 2014 8:09 PM
To: i-d-announce= @ietf.org
Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt


A New Internet-Draft is available from the on-line Internet-Drafts director= ies.


=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Title =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= : Segmentation Offloading Extension for VXLAN
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Authors =C2=A0 =C2=A0 =C2=A0 =C2=A0 : Ha= n Zhou
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Chengyuan Li
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Filename =C2=A0 =C2=A0 =C2=A0 =C2=A0: draft-zho= u-li-vxlan-soe-01.txt
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Pages =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 13 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0:= 2014-05-02

Abstract:
=C2=A0 =C2=A0 =C2=A0Segmentation offloading is nowadays common in network s= tack
=C2=A0 =C2=A0 =C2=A0implementation and well supported by para-virtualized n= etwork device
=C2=A0 =C2=A0 =C2=A0drivers for virtual machine (VM)s. This draft describes= an extension
=C2=A0 =C2=A0 =C2=A0to Virtual eXtensible Local Area Network (VXLAN) so tha= t segmentation
=C2=A0 =C2=A0 =C2=A0can be decoupled from physical/underlay networks and of= floaded
=C2=A0 =C2=A0 =C2=A0further to the remote end-point thus improving data-pla= ne performance
=C2=A0 =C2=A0 =C2=A0for VMs running on top of overlay networks.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-so= e/

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01
A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=3Ddraft-zhou-li-vx= lan-soe-01


Please note that it may take a couple of minutes from the time of submissio= n
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp= .ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
I-D-Announce@iet= f.org
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or = ftp://ftp.ietf.org/ietf/1shadow-sites.txt

_______________________________________________
Tofoo mailing list
Tofoo@ietf.org
h= ttps://www.ietf.org/mailman/listinfo/tofoo


--089e0129555cf49d8204fa017dbc-- From nobody Thu May 22 11:50:42 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1594C1A02DD; Thu, 22 May 2014 11:50:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z9kp8IOC1JSz; Thu, 22 May 2014 11:50:37 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 45E481A02B2; Thu, 22 May 2014 11:50:37 -0700 (PDT) Received: from [128.9.160.81] (nib.isi.edu [128.9.160.81]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s4MIo4Wl027758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 22 May 2014 11:50:05 -0700 (PDT) Message-ID: <537E46DC.9000303@isi.edu> Date: Thu, 22 May 2014 11:50:04 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/y56sKCok8t_M_rkg8co0CwWhrqQ Cc: "Zhou, Han" , Erik Nordmark , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 18:50:39 -0000 On 5/22/2014 11:43 AM, Tom Herbert wrote: > > > > On Tue, May 20, 2014 at 6:14 PM, Joe Touch > wrote: > > Hi, Han, > > This helps, but doesn't quite address my concern. > > TCP offloading is fine when the OS hands off user data, and the > offload engine creates the entire segment. > > This would essentially be TOE. > > The situation you're describing seems to be a hybrid, where the > guest OS makes a TCP segment, and then something lower (in the VM > system) parses that segment to create one or more segments on the wire. > > This how Linux and probably about every NIC implements TCP segmentation > offload. A large TCP segment is broken up into MSS sized segments (MSS > is provided in ancillary data). The process replicates the IP/TCP > headers per packet, adjusts the length, sequence numbers, and computes > checksum for each segment. There's no special handling of any options, > it is assumed they can be replicated in each segment. That's an incorrect assumption. > If that happens, it will be incompatible with a number of existing > TCP options, and will also cause side-effects with ACK clocking, > timeouts, and more than a few other TCP features. > > > Although I appreciate the goal of efficiency and speed, there is a > severe compatibility issue that doesn't seem to be addressed. > > TCP segmentation offload is already widely deployed. Since this draft > would be using the same mechanism it's unlikely to create new > compatibility issues except for the fact that the segmentation might be > deferred to an off host entity which is a new concept that would > probably have side effects. IMO, this draft should not rely on or further promote a mechanism with known problems. Joe From nobody Thu May 22 14:02:59 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DD3561A0283 for ; Thu, 22 May 2014 14:02:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.029 X-Spam-Level: X-Spam-Status: No, score=-2.029 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RP_MATCHES_RCVD=-0.651, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3NmJREGX4eid for ; Thu, 22 May 2014 14:02:50 -0700 (PDT) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 399BC1A0307 for ; Thu, 22 May 2014 14:02:50 -0700 (PDT) Received: by mail-ie0-f182.google.com with SMTP id as1so4204726iec.27 for ; Thu, 22 May 2014 14:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wJi2QMS5McXPP+/gqP8sqtm1NOLDR0LgXSKQ6uuyAOI=; b=OoaN7XWb/QpzddAezY55s0zNOqFnM0Rw0TRQEo6UGlAgAwjTJxZi26WVMlvWCQtcGE 5DaWg5kdY0JlO7Zu7UglUkS3NuMw+p83WMaiwvmJAHpJKlQP78L8D+xX0/NAxd8j5OO/ 4EvJQpWtY/1DXHU3os4uCWVuftv+nU9Y4y/IMIBTRYM57lg9ADKdbizhMmC2EctTs81r esbsQ/NlLGSNrEgvTaVOW36UefZuKWgB+idhY3Dqb09TXyGwqU6zH9t72z7mvpNHzEMD TnZRWHJhTd+nLGrz/6SmwyK2KnuPWP2O608IVPsqbl8qxlAt1uIf4oa/wXyNGxeDHWJt eKVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=wJi2QMS5McXPP+/gqP8sqtm1NOLDR0LgXSKQ6uuyAOI=; b=Cz5dF12IOkAlibsooKXVNOMFWliWP7olOE0EZSrUrnON721pNGBkhv+82wd4i/7Zp2 l8yr0z4YxBAFRgGescZcOqIUBY8YvEZdI9v4EjO3JNwP/ilf516OWRFVJuCAvByEFz9q 0Xp+1B1z7q+Sgww3pXFhvzoCe2C2cf+DtmjAyR18qqszbBbqVsg9t6o6IUoD/zdrYfQ0 Yi+IKEVKDsLn5QK3hUl4wQlJsHpURjzDo1iyRdoDf/z4bvTFJ6MkwnIS9Mv34PRfJYmu Kw/Zlw1pE1N3I+QY6sxi/345nlWq1DTcX/y0Hm4kuuSCUZzEFXUQWOV48i2/vmkbrsYP gZCA== X-Gm-Message-State: ALoCoQlmaEGpd/g8gB1Ij4hynKzR/PHAU304D5rvixOq1bqrZiMxCHd41cU9lwT0fY1iHAt/oXkv MIME-Version: 1.0 X-Received: by 10.43.160.69 with SMTP id mb5mr162552icc.49.1400792568534; Thu, 22 May 2014 14:02:48 -0700 (PDT) Received: by 10.64.148.98 with HTTP; Thu, 22 May 2014 14:02:48 -0700 (PDT) In-Reply-To: <537E46DC.9000303@isi.edu> References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> <537E46DC.9000303@isi.edu> Date: Thu, 22 May 2014 14:02:48 -0700 Message-ID: From: Tom Herbert To: Joe Touch Content-Type: multipart/alternative; boundary=001a11c2d62c73b79504fa036fd8 Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/sPFA4RDJ9F6t09-ZUzUQ88Kr9Qw Cc: "Zhou, Han" , Erik Nordmark , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 21:02:54 -0000 --001a11c2d62c73b79504fa036fd8 Content-Type: text/plain; charset=UTF-8 On Thu, May 22, 2014 at 11:50 AM, Joe Touch wrote: > > > On 5/22/2014 11:43 AM, Tom Herbert wrote: > >> >> >> >> On Tue, May 20, 2014 at 6:14 PM, Joe Touch > > wrote: >> >> Hi, Han, >> >> This helps, but doesn't quite address my concern. >> >> TCP offloading is fine when the OS hands off user data, and the >> offload engine creates the entire segment. >> >> This would essentially be TOE. >> >> The situation you're describing seems to be a hybrid, where the >> guest OS makes a TCP segment, and then something lower (in the VM >> system) parses that segment to create one or more segments on the >> wire. >> >> This how Linux and probably about every NIC implements TCP segmentation >> offload. A large TCP segment is broken up into MSS sized segments (MSS >> is provided in ancillary data). The process replicates the IP/TCP >> headers per packet, adjusts the length, sequence numbers, and computes >> checksum for each segment. There's no special handling of any options, >> it is assumed they can be replicated in each segment. >> > > That's an incorrect assumption. > > > If there are options that can't be replicated then the stack won't use the offload mechanism for those cases. Of course in that situation the performance benefits would be lost, so we really like new options and fields in protocols to be able to be replicated across a few segments :-). New instances of per packet checksums, lengths, sequence numbers, etc. make segmentation offload harder. > If that happens, it will be incompatible with a number of existing >> TCP options, and will also cause side-effects with ACK clocking, >> timeouts, and more than a few other TCP features. >> >> >> Although I appreciate the goal of efficiency and speed, there is a >> severe compatibility issue that doesn't seem to be addressed. >> >> TCP segmentation offload is already widely deployed. Since this draft >> would be using the same mechanism it's unlikely to create new >> compatibility issues except for the fact that the segmentation might be >> deferred to an off host entity which is a new concept that would >> probably have side effects. >> > > IMO, this draft should not rely on or further promote a mechanism with > known problems. > > The mechanism is not inherently problematic, it is still up to the implementation to ensure correctness. What is different with SOE is that the offload functionality becomes visible in the protocol and creates cross implementation dependencies to implement correctly. Tom > Joe > --001a11c2d62c73b79504fa036fd8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable



On Thu, May 22, 2014 at 11:50 AM, Joe Touch <<= a href=3D"mailto:touch@isi.edu" target=3D"_blank">touch@isi.edu> wrote:


On 5/22/2014 11:43 AM, Tom Herbert wrote:



On Tue, May 20, 2014 at 6:14 PM, Joe Touch <touch@isi.edu
<mailto:touch@isi.edu= >> wrote:

=C2=A0 =C2=A0 Hi, Han,

=C2=A0 =C2=A0 This helps, but doesn't quite address my concern.

=C2=A0 =C2=A0 TCP offloading is fine when the OS hands off user data, and t= he
=C2=A0 =C2=A0 offload engine creates the entire segment.

This would essentially be TOE.

=C2=A0 =C2=A0 The situation you're describing seems to be a hybrid, whe= re the
=C2=A0 =C2=A0 guest OS makes a TCP segment, and then something lower (in th= e VM
=C2=A0 =C2=A0 system) parses that segment to create one or more segments on= the wire.

This how Linux and probably about every NIC implements TCP segmentation
offload. A large TCP segment is broken up into MSS sized segments (MSS
is provided in ancillary data). The process replicates the IP/TCP
headers per packet, adjusts the length, sequence numbers, and computes
checksum for each segment. =C2=A0There's no special handling of any opt= ions,
it is assumed they can be replicated in each segment.

That's an incorrect assumption.


If there are options that can't be replicat= ed then the stack won't use the offload mechanism for those cases. Of c= ourse in that situation the performance benefits would be lost, so we reall= y like new options and fields in protocols to be able to be replicated acro= ss a few segments :-). New instances of per packet checksums, lengths, sequ= ence numbers, etc. make segmentation offload harder.
=C2=A0
=C2=A0 =C2=A0 If that happens, it will be incompatible with a number of exi= sting
=C2=A0 =C2=A0 TCP options, and will also cause side-effects with ACK clocki= ng,
=C2=A0 =C2=A0 timeouts, and more than a few other TCP features.


=C2=A0 =C2=A0 Although I appreciate the goal of efficiency and speed, there= is a
=C2=A0 =C2=A0 severe compatibility issue that doesn't seem to be addres= sed.

TCP segmentation offload is already widely deployed. Since this draft
would be using the same mechanism it's unlikely to create new
compatibility issues except for the fact that the segmentation might be
deferred to an off host entity which is a new concept that would
probably have side effects.

IMO, this draft should not rely on or further promote a mechanism with know= n problems.

The mechanism is not inherently problem= atic, it is still up to the implementation to ensure correctness. What is d= ifferent with SOE is that the offload functionality becomes visible in the = protocol and creates cross implementation dependencies to implement correct= ly.

Tom
=C2=A0
Joe

--001a11c2d62c73b79504fa036fd8-- From nobody Thu May 22 14:11:17 2014 Return-Path: X-Original-To: tofoo@ietfa.amsl.com Delivered-To: tofoo@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D77F1A0349; Thu, 22 May 2014 14:11:09 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.651] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 31EHjNCcUpMk; Thu, 22 May 2014 14:11:08 -0700 (PDT) Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4C7D61A034D; Thu, 22 May 2014 14:11:08 -0700 (PDT) Received: from [128.9.160.81] (nib.isi.edu [128.9.160.81]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id s4MLAl0c014697 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 22 May 2014 14:10:47 -0700 (PDT) Message-ID: <537E67D6.1060405@isi.edu> Date: Thu, 22 May 2014 14:10:46 -0700 From: Joe Touch User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: Tom Herbert References: <20140502120923.9835.17537.idtracker@ietfa.amsl.com> <9F56174078B48B459268EFF1DAB66B1A109C2DD3@DEN-EXDDA-S32.corp.ebay.com> <537B90C9.1090003@isi.edu> <9F56174078B48B459268EFF1DAB66B1A109C36FC@DEN-EXDDA-S32.corp.ebay.com> <537BFDF4.9030300@isi.edu> <537E46DC.9000303@isi.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ISI-4-43-8-MailScanner: Found to be clean X-MailScanner-From: touch@isi.edu Archived-At: http://mailarchive.ietf.org/arch/msg/tofoo/obAenCwNgDEahgvXoe6s2Q4dOh4 Cc: "Zhou, Han" , Erik Nordmark , "tofoo@ietf.org" , "nvo3@ietf.org" , "draft-mahalingam-dutt-dcops-vxlan@tools.ietf.org" , "draft-zhou-li-vxlan-soe@tools.ietf.org" Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt X-BeenThere: tofoo@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Discussion list for Tunneling over Foo \(with\)in IP networks \(TOFOO\)." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 May 2014 21:11:09 -0000 On 5/22/2014 2:02 PM, Tom Herbert wrote: ... > If there are options that can't be replicated then the stack won't use > the offload mechanism for those cases. Of course in that situation the > performance benefits would be lost, so we really like new options and > fields in protocols to be able to be replicated across a few segments > :-). New instances of per packet checksums, lengths, sequence numbers, > etc. make segmentation offload harder. Segmentation should never need to happen in an offload engine unless that's where headers are generated. It's never appropriate to merely repeat TCP options - consider that this affects timestamps, ACKs, etc. > If that happens, it will be incompatible with a number of > existing > TCP options, and will also cause side-effects with ACK > clocking, > timeouts, and more than a few other TCP features. > > > Although I appreciate the goal of efficiency and speed, > there is a > severe compatibility issue that doesn't seem to be addressed. > > TCP segmentation offload is already widely deployed. Since this > draft > would be using the same mechanism it's unlikely to create new > compatibility issues except for the fact that the segmentation > might be > deferred to an off host entity which is a new concept that would > probably have side effects. > > > IMO, this draft should not rely on or further promote a mechanism > with known problems. > > The mechanism is not inherently problematic,it is still up to the > implementation to ensure correctness. What is different with SOE is that > the offload functionality becomes visible in the protocol and creates > cross implementation dependencies to implement correctly. I don't agree about the lack of inherent problems. Joe