From shtsuchi@cisco.com Tue Feb 7 21:17:01 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 16ED611E8088; Tue, 7 Feb 2012 21:17:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.999 X-Spam-Level: X-Spam-Status: No, score=-6.999 tagged_above=-999 required=5 tests=[AWL=-1.600, BAYES_00=-2.599, J_CHICKENPOX_42=0.6, J_CHICKENPOX_56=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3+f2K4F1kBXv; Tue, 7 Feb 2012 21:17:00 -0800 (PST) Received: from mtv-iport-2.cisco.com (mtv-iport-2.cisco.com [173.36.130.13]) by ietfa.amsl.com (Postfix) with ESMTP id 8FB7D11E8094; Tue, 7 Feb 2012 21:16:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=shtsuchi@cisco.com; l=1797; q=dns/txt; s=iport; t=1328678219; x=1329887819; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to:content-transfer-encoding; bh=abd6fC2wBSptmpobZToYYQA1bZx5fKVb+ZOmTdWu/S0=; b=JPRKz51u4ABVO/US323MfbHX6Hvme/vAIQkb5nV+ddF0BDR6pVgv3Luc 4YKOEQ/Ra32d53KluzihFh2bsFBnFESZKzJzh8VJNYJAZf8RTHueVkOA7 L1v6NpVtM8fYwwxsgO3I5cQkP62SFYyAVUQYbZx1RtRje3LlNa0o22UA3 I=; X-IronPort-AV: E=Sophos;i="4.73,382,1325462400"; d="scan'208";a="29241710" Received: from mtv-core-1.cisco.com ([171.68.58.6]) by mtv-iport-2.cisco.com with ESMTP; 08 Feb 2012 05:16:59 +0000 Received: from [64.104.9.218] (dhcp-shinjuku-64-104-9-218.cisco.com [64.104.9.218]) by mtv-core-1.cisco.com (8.14.3/8.14.3) with ESMTP id q185Gwn1010353; Wed, 8 Feb 2012 05:16:58 GMT Message-ID: <4F320549.20100@cisco.com> Date: Wed, 08 Feb 2012 14:16:57 +0900 From: Shishio Tsuchiya User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: armd@ietf.org, v6ops@ietf.org References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com> In-Reply-To: <20111114092126.3494.19513.idtracker@ietfa.amsl.com> X-Forwarded-Message-Id: <20111114092126.3494.19513.idtracker@ietfa.amsl.com> Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Cc: softwires@ietf.org, draft-sakura-6rd-datacenter@tools.ietf.org Subject: [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 05:17:01 -0000 v6ops and ARMD Sakura internet are providing IPv6 service without scalability impact to core network equipment to customer using 6rd. We thought this is interesting idea,so published the draft. http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 I think both of mailing list participants are also interesting in this technique. If you have any comment and question of the draft,please let me know. Regards, -Shishio -------- Original Message -------- Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt Date: Mon, 14 Nov 2011 17:21:26 +0800 From: Reply-To: To: A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : IPv6 Rapid Deployment (6rd) in a Large Data Center Author(s) : Shishio Tsuchiya Mark Townsley Shuichi Ohkubo Filename : draft-sakura-6rd-datacenter-03.txt Pages : 15 Date : 2011-11-14 IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid deployment of IPv6 by an access service provider which has difficulty deploying native IPv6. This document describes how 6rd can be used to deliver IPv6 within a Large Data Center. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ This Internet-Draft can be retrieved at: ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt _______________________________________________ I-D-Announce mailing list I-D-Announce@ietf.org https://www.ietf.org/mailman/listinfo/i-d-announce Internet-Draft directories: http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt From shtsuchi@cisco.com Tue Feb 7 21:39:51 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 43F1311E8089; Tue, 7 Feb 2012 21:39:51 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.408 X-Spam-Level: X-Spam-Status: No, score=-5.408 tagged_above=-999 required=5 tests=[AWL=-2.124, BAYES_00=-2.599, J_CHICKENPOX_13=0.6, J_CHICKENPOX_34=0.6, J_CHICKENPOX_42=0.6, J_CHICKENPOX_53=0.6, J_CHICKENPOX_56=0.6, RCVD_IN_DNSWL_MED=-4, SARE_MILLIONSOF=0.315] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EyVeyHlPo6av; Tue, 7 Feb 2012 21:39:50 -0800 (PST) Received: from mtv-iport-4.cisco.com (mtv-iport-4.cisco.com [173.36.130.15]) by ietfa.amsl.com (Postfix) with ESMTP id 545B621F8507; Tue, 7 Feb 2012 21:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=shtsuchi@cisco.com; l=2893; q=dns/txt; s=iport; t=1328679590; x=1329889190; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to:content-transfer-encoding; bh=+pxfYkUbPwPJDB3arVToosXMP7EeF5y37/lSTDMZAdQ=; b=Sd9zfxHDABI2lBslDD7a6Q4TtIk1V+2I+01+9DtHcJ1es8Mpo/oyogep lepDS1iQOOwPF3ckW6KRElBrWvrrjBhU8Y6/zPUwLnltIl7W1qTCfEXb2 0m2QV4jexRFsaGJx43b5r9SeJu/FV6oi2rL1+iZkR1Fl7sCqoATK+03Tq w=; X-IronPort-AV: E=Sophos;i="4.73,382,1325462400"; d="scan'208";a="29107881" Received: from mtv-core-3.cisco.com ([171.68.58.8]) by mtv-iport-4.cisco.com with ESMTP; 08 Feb 2012 05:39:49 +0000 Received: from [64.104.9.218] (dhcp-shinjuku-64-104-9-218.cisco.com [64.104.9.218]) by mtv-core-3.cisco.com (8.14.3/8.14.3) with ESMTP id q185dlZw003253; Wed, 8 Feb 2012 05:39:48 GMT Message-ID: <4F320AA3.1020803@cisco.com> Date: Wed, 08 Feb 2012 14:39:47 +0900 From: Shishio Tsuchiya User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: Yiu_Lee@Cable.Comcast.com References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com>, <4F320549.20100@cisco.com> <40137D15-A014-4C28-B531-9D949173C222@Cable.Comcast.com> In-Reply-To: <40137D15-A014-4C28-B531-9D949173C222@Cable.Comcast.com> Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Cc: softwires@ietf.org, v6ops@ietf.org, draft-sakura-6rd-datacenter@tools.ietf.org, armd@ietf.org Subject: Re: [armd] [Softwires] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 05:39:51 -0000 Yiu Thanks for comments. Yes,Free,Swisscom,Softbank and so on already deployed 6rd to the access network. But Sakura deployed IPv6 to the datacenter network by server based 6rd with variable OSs and 6rd BR. http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03#section-3 This is my interesting point,and this would be useful information. Regards, -Shishio (2012/02/08 14:23), Lee, Yiu wrote: > Hi Shishio, > > Thanks for sharing this. I am glad but not surprised to see the result because free.fr has shown to scale 6rd to millions of their users since few years ago. > > Yiu > > On Feb 8, 2012, at 0:17, "Shishio Tsuchiya" wrote: > > > v6ops and ARMD > > Sakura internet are providing IPv6 service without scalability impact to core network equipment to customer using 6rd. > > We thought this is interesting idea,so published the draft. > > > > http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 > > > > I think both of mailing list participants are also interesting in this technique. > > If you have any comment and question of the draft,please let me know. > > > > Regards, > > -Shishio > > -------- Original Message -------- > > Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt > > Date: Mon, 14 Nov 2011 17:21:26 +0800 > > From: > > Reply-To: > > To: > > > > > > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > > Title : IPv6 Rapid Deployment (6rd) in a Large Data Center > > Author(s) : Shishio Tsuchiya > > Mark Townsley > > Shuichi Ohkubo > > Filename : draft-sakura-6rd-datacenter-03.txt > > Pages : 15 > > Date : 2011-11-14 > > > > IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid > > deployment of IPv6 by an access service provider which has difficulty > > deploying native IPv6. This document describes how 6rd can be used > > to deliver IPv6 within a Large Data Center. > > > > > > A URL for this Internet-Draft is: > > http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > > > > Internet-Drafts are also available by anonymous FTP at: > > ftp://ftp.ietf.org/internet-drafts/ > > > > This Internet-Draft can be retrieved at: > > ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > > _______________________________________________ > > I-D-Announce mailing list > > I-D-Announce@ietf.org > > https://www.ietf.org/mailman/listinfo/i-d-announce > > Internet-Draft directories: http://www.ietf.org/shadow.html > > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > > > > _______________________________________________ > > Softwires mailing list > > Softwires@ietf.org > > https://www.ietf.org/mailman/listinfo/softwires > From yiu_lee@cable.comcast.com Tue Feb 7 21:23:45 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2838611E80A5; Tue, 7 Feb 2012 21:23:45 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -100.648 X-Spam-Level: X-Spam-Status: No, score=-100.648 tagged_above=-999 required=5 tests=[AWL=-1.532, BAYES_00=-2.599, HOST_EQ_MODEMCABLE=1.368, J_CHICKENPOX_13=0.6, J_CHICKENPOX_42=0.6, J_CHICKENPOX_56=0.6, SARE_MILLIONSOF=0.315, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2R-DV5h4qQ7Q; Tue, 7 Feb 2012 21:23:44 -0800 (PST) Received: from cable.comcast.com (copdcavout01.cable.comcast.com [76.96.32.253]) by ietfa.amsl.com (Postfix) with ESMTP id 1816111E8088; Tue, 7 Feb 2012 21:23:44 -0800 (PST) Received: from ([24.40.56.115]) by copdcavout01.cable.comcast.com with ESMTP id C7WM3M1.5455620; Tue, 07 Feb 2012 22:14:04 -0700 Received: from PACDCEXMB05.cable.comcast.com ([fe80::a5b0:e5c4:df1b:2367]) by PACDCEXHUB02.cable.comcast.com ([fe80::492e:3fa1:c2ad:e04e%17]) with mapi id 14.01.0355.002; Wed, 8 Feb 2012 00:23:41 -0500 From: "Lee, Yiu" To: Shishio Tsuchiya Thread-Topic: [Softwires] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt Thread-Index: AQHM5iDpvq1pF/9+ckGtbaXvkonN/5Yyd2BV Date: Wed, 8 Feb 2012 05:23:40 +0000 Message-ID: <40137D15-A014-4C28-B531-9D949173C222@Cable.Comcast.com> References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com>, <4F320549.20100@cisco.com> In-Reply-To: <4F320549.20100@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 08 Feb 2012 09:44:53 -0800 Cc: "softwires@ietf.org" , "v6ops@ietf.org" , "draft-sakura-6rd-datacenter@tools.ietf.org" , "armd@ietf.org" Subject: Re: [armd] [Softwires] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 05:23:45 -0000 Hi Shishio, Thanks for sharing this. I am glad but not surprised to see the result beca= use free.fr has shown to scale 6rd to millions of their users since few yea= rs ago. Yiu On Feb 8, 2012, at 0:17, "Shishio Tsuchiya" wrote: > v6ops and ARMD > Sakura internet are providing IPv6 service without scalability impact to = core network equipment to customer using 6rd. > We thought this is interesting idea,so published the draft. >=20 > http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 >=20 > I think both of mailing list participants are also interesting in this te= chnique. > If you have any comment and question of the draft,please let me know. >=20 > Regards, > -Shishio > -------- Original Message -------- > Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt > Date: Mon, 14 Nov 2011 17:21:26 +0800 > From: > Reply-To: > To: >=20 >=20 >=20 > A New Internet-Draft is available from the on-line Internet-Drafts direct= ories. >=20 > Title : IPv6 Rapid Deployment (6rd) in a Large Data Center > Author(s) : Shishio Tsuchiya > Mark Townsley > Shuichi Ohkubo > Filename : draft-sakura-6rd-datacenter-03.txt > Pages : 15 > Date : 2011-11-14 >=20 > IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid > deployment of IPv6 by an access service provider which has difficulty > deploying native IPv6. This document describes how 6rd can be used > to deliver IPv6 within a Large Data Center. >=20 >=20 > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt >=20 > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ >=20 > This Internet-Draft can be retrieved at: > ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draft directories: http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt >=20 >=20 > _______________________________________________ > Softwires mailing list > Softwires@ietf.org > https://www.ietf.org/mailman/listinfo/softwires From dunbar.ll@gmail.com Thu Feb 9 15:26:18 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 361E221E8062; Thu, 9 Feb 2012 15:26:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.798 X-Spam-Level: X-Spam-Status: No, score=-1.798 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_13=0.6, J_CHICKENPOX_42=0.6, J_CHICKENPOX_56=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I-JVhXOiHDpN; Thu, 9 Feb 2012 15:26:15 -0800 (PST) Received: from mail-bk0-f44.google.com (mail-bk0-f44.google.com [209.85.214.44]) by ietfa.amsl.com (Postfix) with ESMTP id 09F9E21E805E; Thu, 9 Feb 2012 15:26:14 -0800 (PST) Received: by bkuw12 with SMTP id w12so2299657bku.31 for ; Thu, 09 Feb 2012 15:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=v3RChyv2tECrWeNW6pB/62ecQRxA/q/kGGByju1608I=; b=BnUY4gqnFvtcglnuk5GUZ2L+LnmbeJkpV7dzcd0rNKThlkSntC5M0oQrc1WSo1lVda Az83wFnyOlytOkvj+2+iNpCHbvOIZxVoOky6GF3gWwhDo27iu6bexhblI7maGtaHTm41 jgpdR/CTa34pJ+mVI76cpfmyBBRln9DsadgYk= MIME-Version: 1.0 Received: by 10.204.150.69 with SMTP id x5mr1536581bkv.53.1328829974062; Thu, 09 Feb 2012 15:26:14 -0800 (PST) Received: by 10.204.62.199 with HTTP; Thu, 9 Feb 2012 15:26:14 -0800 (PST) In-Reply-To: <4F320549.20100@cisco.com> References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com> <4F320549.20100@cisco.com> Date: Thu, 9 Feb 2012 17:26:14 -0600 Message-ID: From: Linda Dunbar To: Shishio Tsuchiya Content-Type: multipart/alternative; boundary=0015175cdafa9227df04b8905671 Cc: softwires@ietf.org, v6ops@ietf.org, draft-sakura-6rd-datacenter@tools.ietf.org, armd@ietf.org Subject: Re: [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 23:26:18 -0000 --0015175cdafa9227df04b8905671 Content-Type: text/plain; charset=ISO-8859-1 Shishio, Thank you very much for sharing the v6 deployment in data center. I am not an IPv6 expert. So I have a simple question: when you say "server-based 6rd", do you mean that server does the IPv4 encapsulation? Does your data center allow hosts/servers under one IPv6 subnet (e.g. /64) to be placed under different server racks? Thanks, Linda Dunbar On Tue, Feb 7, 2012 at 11:16 PM, Shishio Tsuchiya wrote: > v6ops and ARMD > Sakura internet are providing IPv6 service without scalability impact to > core network equipment to customer using 6rd. > We thought this is interesting idea,so published the draft. > > http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 > > I think both of mailing list participants are also interesting in this > technique. > If you have any comment and question of the draft,please let me know. > > Regards, > -Shishio > -------- Original Message -------- > Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt > Date: Mon, 14 Nov 2011 17:21:26 +0800 > From: > Reply-To: > To: > > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > Title : IPv6 Rapid Deployment (6rd) in a Large Data Center > Author(s) : Shishio Tsuchiya > Mark Townsley > Shuichi Ohkubo > Filename : draft-sakura-6rd-datacenter-03.txt > Pages : 15 > Date : 2011-11-14 > > IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid > deployment of IPv6 by an access service provider which has difficulty > deploying native IPv6. This document describes how 6rd can be used > to deliver IPv6 within a Large Data Center. > > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > This Internet-Draft can be retrieved at: > ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draft directories: http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > --0015175cdafa9227df04b8905671 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Shishio,

Thank you very much for sharing the v6 deployment in data = center.

I am not an IPv6 expert. So I have a simple question: when = you say "server-based 6rd", do you mean that server does the IPv4= encapsulation?


Does your data center allow hosts/servers under one IPv6 subnet (e.= g. /64) to be placed under different server racks?

Thanks, Linda Dun= bar





On Tue, Feb 7, 2012 a= t 11:16 PM, Shishio Tsuchiya <shtsuchi@cisco.com> wrote:
v6ops and ARMD
Sakura internet are providing IPv6 service without scalability impact to co= re network equipment to customer using 6rd.
We thought this is interesting idea,so published the draft.

http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03<= br>
I think both of mailing list participants are also interesting in this tech= nique.
If you have any comment and question of the draft,please let me know.

Regards,
-Shishio
-------- Original Message --------
Subject: =A0 =A0 =A0 =A0I-D Action: draft-sakura-6rd-datacenter-03.txt
Date: =A0 Mon, 14 Nov 2011 17:21:26 +0800
From: =A0 <internet-drafts@ietf.org>
Reply-To: =A0 =A0 =A0 <internet-drafts@ietf.org>
To: =A0 =A0 <= i-d-announce@ietf.org>



A New Internet-Draft is available from the on-line Internet-Drafts director= ies.

Title : IPv6 Rapid Deployment (6rd) in a Large Data Center
Author(s) : Shishio Tsuchiya
Mark Townsley
Shuichi Ohkubo
Filename : draft-sakura-6rd-datacenter-03.txt
Pages : 15
Date : 2011-11-14

IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid
deployment of IPv6 by an access service provider which has difficulty
deploying native IPv6. This document describes how 6rd can be used
to deliver IPv6 within a Large Data Center.


A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-sakura-= 6rd-datacenter-03.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp= .ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-sakura-6r= d-datacenter-03.txt
_______________________________________________
I-D-Announce mailing list
I-D-Announce@iet= f.org
https://www.ietf.org/mailman/listinfo/i-d-announce Internet-Draft directories: http://www.ietf.org/shadow.html
or = ftp://ftp.ietf.org/ietf/1shadow-sites.txt


_______________________________________________
armd mailing list
armd@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/armd

--0015175cdafa9227df04b8905671-- From shtsuchi@cisco.com Thu Feb 9 20:44:37 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5EFEF21F85DB; Thu, 9 Feb 2012 20:44:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.135 X-Spam-Level: X-Spam-Status: No, score=-2.135 tagged_above=-999 required=5 tests=[AWL=-4.336, BAYES_00=-2.599, J_CHICKENPOX_13=0.6, J_CHICKENPOX_26=0.6, J_CHICKENPOX_42=0.6, J_CHICKENPOX_54=0.6, J_CHICKENPOX_55=0.6, J_CHICKENPOX_56=0.6, J_CHICKENPOX_64=0.6, J_CHICKENPOX_84=0.6] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9TECnOlY6X4y; Thu, 9 Feb 2012 20:44:36 -0800 (PST) Received: from bgl-iport-1.cisco.com (bgl-iport-1.cisco.com [72.163.197.25]) by ietfa.amsl.com (Postfix) with ESMTP id 51BDD21F85D8; Thu, 9 Feb 2012 20:44:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=shtsuchi@cisco.com; l=4214; q=dns/txt; s=iport; t=1328849073; x=1330058673; h=message-id:date:from:mime-version:to:cc:subject: references:in-reply-to:content-transfer-encoding; bh=9Q7NnMHNjXBGTghYArNDL/ysh93IonMuhYwPRjMb2CI=; b=gyLg8esOZNjj4eszpun97L2pwrJUFO8iz4eBNMFa7dJOg/b1fItWyY9L JolTPmX3cKxLbBnGa09/fzawzjy4HYYIw65/rJ7HJ79XpEQ2LKRQWUh85 +QHeBsCDjmPdFynvCxf4PbJcQcanyjwSSwYxyfSQ/jBmnRcdAPnRTx/CH g=; X-IronPort-AV: E=Sophos;i="4.73,394,1325462400"; d="scan'208";a="5237742" Received: from vla196-nat.cisco.com (HELO bgl-core-1.cisco.com) ([72.163.197.24]) by bgl-iport-1.cisco.com with ESMTP; 10 Feb 2012 04:44:30 +0000 Received: from [64.104.53.150] (dhcp-tmt-wirelessdata-64-104-53-150.cisco.com [64.104.53.150]) by bgl-core-1.cisco.com (8.14.3/8.14.3) with ESMTP id q1A4iT3b019125; Fri, 10 Feb 2012 04:44:29 GMT Message-ID: <4F34A0AD.4040300@cisco.com> Date: Fri, 10 Feb 2012 13:44:29 +0900 From: Shishio Tsuchiya User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0) Gecko/20120129 Thunderbird/10.0 MIME-Version: 1.0 To: dunbar.ll@gmail.com References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com><4F320549.20100@cisco.com> In-Reply-To: Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Cc: softwires@ietf.org, v6ops@ietf.org, draft-sakura-6rd-datacenter@tools.ietf.org, armd@ietf.org Subject: Re: [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 04:44:37 -0000 Linda Thanks for comments. (2012/02/10 8:26), Linda Dunbar wrote: > Shishio, > > Thank you very much for sharing the v6 deployment in data center. > > I am not an IPv6 expert. So I have a simple question: when you say "server-based 6rd", do you mean that server does the IPv4 encapsulation? Yes. 6rd is IPv6 over IPv4 tunnel technology. So backbone operator does not need to care to IPv6 on the core. Additionally 6rd has stateless feature. Once the operator has placed/configured 6rd BR on the exit point to the IPv6 internet,does not need additional operation to add 6rd CE(server based 6rd). And if the server would like to go to the IPv6 internet,then IPv4 packets will through 6rd BR and de-capsulate. If the server would like to communicate other 6rd server within the data center,then IPv4 packets will reach 6rd server directly and de-capsulate. > > > Does your data center allow hosts/servers under one IPv6 subnet (e.g. /64) to be placed under different server racks? In Sakura internet 6rd cases,each of 6rd server has one 6rd delegate prefix(/64). http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03#section-4.1 The 6rd delegate prefix calculates from Sakura's IPv6 prefix(/32) and IPv4 address. Sakura has the scalability problem which is same as ARMD draft. http://tools.ietf.org/html/draft-ietf-armd-problem-statement-00 They also had to update switches to resolve security issue(Rogue RA). http://tools.ietf.org/html/rfc6104 They selected "server-based 6rd" as the solution to avoid scalability and security problem. Regards, -Shishio > > Thanks, Linda Dunbar > > > > > > On Tue, Feb 7, 2012 at 11:16 PM, Shishio Tsuchiya > wrote: > > v6ops and ARMD > Sakura internet are providing IPv6 service without scalability impact to core network equipment to customer using 6rd. > We thought this is interesting idea,so published the draft. > > http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 > > I think both of mailing list participants are also interesting in this technique. > If you have any comment and question of the draft,please let me know. > > Regards, > -Shishio > -------- Original Message -------- > Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt > Date: Mon, 14 Nov 2011 17:21:26 +0800 > From: > > Reply-To: > > To: > > > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > Title : IPv6 Rapid Deployment (6rd) in a Large Data Center > Author(s) : Shishio Tsuchiya > Mark Townsley > Shuichi Ohkubo > Filename : draft-sakura-6rd-datacenter-03.txt > Pages : 15 > Date : 2011-11-14 > > IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid > deployment of IPv6 by an access service provider which has difficulty > deploying native IPv6. This document describes how 6rd can be used > to deliver IPv6 within a Large Data Center. > > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > This Internet-Draft can be retrieved at: > ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt > _______________________________________________ > I-D-Announce mailing list > I-D-Announce@ietf.org > https://www.ietf.org/mailman/listinfo/i-d-announce > Internet-Draft directories: http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > > From Tina.Tsou.Zouting@huawei.com Thu Feb 9 18:13:36 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 83BF711E8075; Thu, 9 Feb 2012 18:13:36 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.564 X-Spam-Level: X-Spam-Status: No, score=-5.564 tagged_above=-999 required=5 tests=[AWL=-0.766, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_13=0.6, J_CHICKENPOX_42=0.6, J_CHICKENPOX_56=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kj-Kl-OykHk2; Thu, 9 Feb 2012 18:13:35 -0800 (PST) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [119.145.14.66]) by ietfa.amsl.com (Postfix) with ESMTP id E335311E8076; Thu, 9 Feb 2012 18:13:33 -0800 (PST) Received: from huawei.com (szxga03-in [172.24.2.9]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LZ5004XFNEIU7@szxga03-in.huawei.com>; Fri, 10 Feb 2012 10:11:06 +0800 (CST) Received: from szxrg02-dlp.huawei.com ([172.24.2.119]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0LZ500L5CNEFPJ@szxga03-in.huawei.com>; Fri, 10 Feb 2012 10:11:06 +0800 (CST) Received: from szxeml213-edg.china.huawei.com ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.1.9-GA) with ESMTP id AGZ93690; Fri, 10 Feb 2012 10:11:05 +0800 Received: from SZXEML413-HUB.china.huawei.com (10.82.67.152) by szxeml213-edg.china.huawei.com (172.24.2.30) with Microsoft SMTP Server (TLS) id 14.1.323.3; Fri, 10 Feb 2012 10:10:22 +0800 Received: from SZXEML526-MBS.china.huawei.com ([169.254.7.225]) by szxeml413-hub.china.huawei.com ([10.82.67.152]) with mapi id 14.01.0323.003; Fri, 10 Feb 2012 10:11:15 +0800 Date: Fri, 10 Feb 2012 02:10:47 +0000 From: Tina TSOU In-reply-to: X-Originating-IP: [10.212.244.142] To: Linda Dunbar , Shishio Tsuchiya Message-id: MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_jV0Jsk21GpYdhx328yWtcQ)" Content-language: en-US Accept-Language: en-US, zh-CN Thread-topic: [v6ops] [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt Thread-index: AQHM55OqKuMa1bvJ8kCzdvdIpYIP2ZY1Yw1A X-MS-Has-Attach: X-MS-TNEF-Correlator: X-CFilter-Loop: Reflected References: <20111114092126.3494.19513.idtracker@ietfa.amsl.com> <4F320549.20100@cisco.com> X-Mailman-Approved-At: Fri, 10 Feb 2012 06:59:24 -0800 Cc: "softwires@ietf.org" , "v6ops@ietf.org" , "armd@ietf.org" , "draft-sakura-6rd-datacenter@tools.ietf.org" Subject: Re: [armd] [v6ops] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 02:13:36 -0000 --Boundary_(ID_jV0Jsk21GpYdhx328yWtcQ) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Tina From: v6ops-bounces@ietf.org [mailto:v6ops-bounces@ietf.org] On Behalf Of Linda Dunbar Sent: Thursday, February 09, 2012 3:26 PM To: Shishio Tsuchiya Cc: softwires@ietf.org; v6ops@ietf.org; draft-sakura-6rd-datacenter@tools.ietf.org; armd@ietf.org Subject: Re: [v6ops] [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt Shishio, Thank you very much for sharing the v6 deployment in data center. I am not an IPv6 expert. So I have a simple question: when you say "server-based 6rd", do you mean that server does the IPv4 encapsulation? Does your data center allow hosts/servers under one IPv6 subnet (e.g. /64) to be placed under different server racks? [Tina] In my case, yes. Thanks, Linda Dunbar On Tue, Feb 7, 2012 at 11:16 PM, Shishio Tsuchiya > wrote: v6ops and ARMD Sakura internet are providing IPv6 service without scalability impact to core network equipment to customer using 6rd. We thought this is interesting idea,so published the draft. http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03 I think both of mailing list participants are also interesting in this technique. If you have any comment and question of the draft,please let me know. Regards, -Shishio -------- Original Message -------- Subject: I-D Action: draft-sakura-6rd-datacenter-03.txt Date: Mon, 14 Nov 2011 17:21:26 +0800 From: > Reply-To: > To: > A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : IPv6 Rapid Deployment (6rd) in a Large Data Center Author(s) : Shishio Tsuchiya Mark Townsley Shuichi Ohkubo Filename : draft-sakura-6rd-datacenter-03.txt Pages : 15 Date : 2011-11-14 IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid deployment of IPv6 by an access service provider which has difficulty deploying native IPv6. This document describes how 6rd can be used to deliver IPv6 within a Large Data Center. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ This Internet-Draft can be retrieved at: ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt _______________________________________________ I-D-Announce mailing list I-D-Announce@ietf.org https://www.ietf.org/mailman/listinfo/i-d-announce Internet-Draft directories: http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_jV0Jsk21GpYdhx328yWtcQ) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

 

 

Tina

 

From: v6ops-bounces@ietf.org [mailto:v6ops-bounces@ietf.org] On Behalf Of Linda Dunbar
Sent: Thursday, February 09, 2012 3:26 PM
To: Shishio Tsuchiya
Cc: softwires@ietf.org; v6ops@ietf.org; draft-sakura-6rd-datacenter@tools.ietf.org; armd@ietf.org
Subject: Re: [v6ops] [armd] Fwd: I-D Action: draft-sakura-6rd-datacenter-03.txt

 

Shishio,

Thank you very much for sharing the v6 deployment in data center.

I am not an IPv6 expert. So I have a simple question: when you say "server-based 6rd", do you mean that server does the IPv4 encapsulation?


Does your data center allow hosts/servers under one IPv6 subnet (e.g. /64) to be placed under different server racks?

[Tina] In my case, yes.



Thanks, Linda Dunbar




On Tue, Feb 7, 2012 at 11:16 PM, Shishio Tsuchiya <shtsuchi@cisco.com> wrote:

v6ops and ARMD
Sakura internet are providing IPv6 service without scalability impact to core network equipment to customer using 6rd.
We thought this is interesting idea,so published the draft.

http://tools.ietf.org/html/draft-sakura-6rd-datacenter-03

I think both of mailing list participants are also interesting in this technique.
If you have any comment and question of the draft,please let me know.

Regards,
-Shishio
-------- Original Message --------
Subject:        I-D Action: draft-sakura-6rd-datacenter-03.txt
Date:   Mon, 14 Nov 2011 17:21:26 +0800
From:   <internet-drafts@ietf.org>
Reply-To:       <internet-drafts@ietf.org>
To:     <i-d-announce@ietf.org>



A New Internet-Draft is available from the on-line Internet-Drafts directories.

Title : IPv6 Rapid Deployment (6rd) in a Large Data Center
Author(s) : Shishio Tsuchiya
Mark Townsley
Shuichi Ohkubo
Filename : draft-sakura-6rd-datacenter-03.txt
Pages : 15
Date : 2011-11-14

IPv6 Rapid Deployment (6rd) as defined in RFC 5969 focuses on rapid
deployment of IPv6 by an access service provider which has difficulty
deploying native IPv6. This document describes how 6rd can be used
to deliver IPv6 within a Large Data Center.


A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-sakura-6rd-datacenter-03.txt
_______________________________________________
I-D-Announce mailing list
I-D-Announce@ietf.org
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt


_______________________________________________
armd mailing list
armd@ietf.org
https://www.ietf.org/mailman/listinfo/armd

 

--Boundary_(ID_jV0Jsk21GpYdhx328yWtcQ)-- From linda.dunbar@huawei.com Fri Feb 10 13:31:18 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8017E21F8759 for ; Fri, 10 Feb 2012 13:31:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.515 X-Spam-Level: X-Spam-Status: No, score=-2.515 tagged_above=-999 required=5 tests=[AWL=0.083, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id okuhz8TXtsvM for ; Fri, 10 Feb 2012 13:31:17 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 0499521F8748 for ; Fri, 10 Feb 2012 13:31:16 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml201-edg.china.huawei.com) ([172.18.9.243]) by dfwrg02-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADC37980; Fri, 10 Feb 2012 16:31:15 -0500 (EST) Received: from DFWEML404-HUB.china.huawei.com (10.193.5.203) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.323.3; Fri, 10 Feb 2012 13:28:18 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml404-hub.china.huawei.com ([10.193.5.203]) with mapi id 14.01.0323.003; Fri, 10 Feb 2012 13:28:21 -0800 From: Linda Dunbar To: "armd@ietf.org" Thread-Topic: address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cg== Date: Fri, 10 Feb 2012 21:28:20 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E1CE52@dfweml505-mbx> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: GDna HZEZ Ht6C H1ZE H7Dt QHDV W37t sOXY uVma yWmf 0gKz 2CzP 4wJS 5OB7 7jIT 7/7U; 5; YQByAG0AZABAAGkAZQB0AGYALgBvAHIAZwA7AGIAcwBjAGgAbABpAGUAcwBAAGMAaQBzAGMAbwAuAGMAbwBtADsAaQBnAG8AcgBAAHkAYQBoAG8AbwAtAGkAbgBjAC4AYwBvAG0AOwBtAHUAcgBhAHIAaQBzAEAAbQBpAGMAcgBvAHMAbwBmAHQALgBjAG8AbQA7AG4AYQByAHQAZQBuAEAAdQBzAC4AaQBiAG0ALgBjAG8AbQA=; Sosha1_v1; 7; {A85D8EAF-165D-4891-9488-CEC9DE45BDBD}; bABpAG4AZABhAC4AZAB1AG4AYgBhAHIAQABoAHUAYQB3AGUAaQAuAGMAbwBtAA==; Fri, 10 Feb 2012 21:28:08 GMT; YQBkAGQAcgBlAHMAcwAgAHIAZQBzAG8AbAB1AHQAaQBvAG4AIAByAGUAcQB1AGkAcgBlAG0AZQBuAHQAIABmAHIAbwBtACAAaABvAHMAdABzACAAdABvACAAbwB2AGUAcgBsAGEAeQAgAGUAZABnAGUAIABuAG8AZABlAHMALgAgAEEAbgB5ACAAbwBwAGkAbgBpAG8AbgA/AA== x-cr-puzzleid: {A85D8EAF-165D-4891-9488-CEC9DE45BDBD} x-originating-ip: [10.192.11.97] Content-Type: multipart/alternative; boundary="_000_4A95BA014132FF49AE685FAB4B9F17F632E1CE52dfweml505mbx_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , Igor Gashinsky Subject: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Feb 2012 21:31:18 -0000 --_000_4A95BA014132FF49AE685FAB4B9F17F632E1CE52dfweml505mbx_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable We all believe that the current trend for Data Center network is going towa= rds overlay network, be it TRILL, MAC-in-MAC (SPB), NV03, etc. In Overlay n= etwork, each Overlay Border Node (OBN) has to map the destination address o= f a data packet to an egress Overlay Border Node. Very much like ARP/ND whi= ch is to resolve physical addresses (i.e. MAC Addresses) from target IP add= resses. Here are some distinct characteristics of hosts in Data Center: 1. Even though the communication between DC hosts to external peers may be= small in volume comparing with the east-west (intra-data center) traffic, = majority of hosts in data center still do communicate with external peers. 2. Traffic entering/exiting the Data Center Overlay could be via multiple = Overlay Border nodes. 3. The placement of VMs/hosts to server racks is orchestrated by Managemen= t Entities. So Overlay Border Nodes can, at least in theory, get the hosts = <-> OverlayBorderNode mapping information from directory server(s). There c= ould be multiple directory servers serving a very large data center. >From the very high level view, the directory providing mapping from hosts t= o Overlay Border Node seems very straight forward. However, if you look dee= per, following issues come out: - When hosts can enter or exit the overlay network via multiple OBNs, is = it necessary for each OBN to be aware of which egress OBN can reach the tar= get? Or, is it necessary for each OBN to advertise its connectivity status = of all the attached hosts to all other OBNs? - When migration is orchestrated by management entity (or entities), ther= e could be multiple instances of directory servers o is it necessary for each directory server to have full information? Wha= t if some directory servers only have partial information (for very large D= C environment)? o Is it necessary for each OBN to report connectivity status of all the a= ttached hosts to directory servers? o Should resolution requests be load balanced among them? - A data frame arriving at OBN normally might have an VID in the data fra= me, the egress OBN might have a different VID value for the same virtual n= etwork instance. Original VID loses its meaning when traversing to Egress o= verlay border. Proper Ingress VID -> overlay network instance ID - >to Egre= ss VID resolution (or mapping) is needed. - Etc. It was suggested that ARMD WG should have a requirement draft on Address Re= solution from data center host addresses to Overlay Border Node addresses s= o that the industry can have a common requirement for various types of over= lay for data center environment. A draft on this topic could potentially trigger more input on various const= raints or requirement for hosts <-> OBN mapping in data center environment. Any opinion on this? Linda Dunbar --_000_4A95BA014132FF49AE685FAB4B9F17F632E1CE52dfweml505mbx_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We all believe that the current trend for Data Ce= nter network is going towards overlay network, be it TRILL, MAC-in-MAC (SPB= ), NV03, etc. In Overlay network, each Overlay Border Node (OBN) has to map= the destination address of a data packet to an egress Overlay Border Node. Very much like ARP/ND which is to= resolve physical addresses (i.e. MAC Addresses) from target IP addresses.<= o:p>

 

Here are some distinct characteristics of hosts i= n Data Center:

1.  Even though the communication between DC hosts to e= xternal peers may be small in volume comparing with the east-west (intra-da= ta center) traffic,  majority of hosts in data center still do communi= cate with external peers.

 

2.  Traffic entering/exiting the Data Center Overlay co= uld be via multiple Overlay Border nodes.

 

3.  The placement of VMs/hosts to server racks is orche= strated by Management Entities. So Overlay Border Nodes can, at least in th= eory, get the hosts <-> OverlayBorderNode mapping information from di= rectory server(s). There could be multiple directory servers serving a very large data center.

 

From the very high level view, the directory prov= iding mapping from hosts to Overlay Border Node seems very straight forward= . However, if you look deeper, following issues come out:

 

-  = ; When hosts can= enter or exit the overlay network via multiple OBNs, is it necessary for e= ach OBN to be aware of which egress OBN can reach the target? Or, is it nec= essary for each OBN to advertise its connectivity status of all the attached hosts to all other OBNs?

 

-  = ; When migration= is orchestrated by management entity (or entities), there could be multipl= e instances of directory servers  

o   is it necessar= y for each directory server to have full information? What if some director= y servers only have partial information (for very large DC environment)?

o   Is it necessar= y for each OBN to report connectivity status of all the attached hosts to d= irectory servers?

o   Should resolut= ion requests be load balanced among them?

 

-  = ; A data frame a= rriving at OBN normally might have an VID in the data frame, the egress OBN= might have a different VID value for the  same virtual network instan= ce. Original VID loses its meaning when traversing to Egress overlay border. Proper Ingress VID -> overlay netw= ork instance ID - >to Egress VID  resolution (or mapping) is needed= .

 

-  = ; Etc.

 

It was suggested that ARMD WG should have a requi= rement draft on Address Resolution from data center host addresses to Overl= ay Border Node addresses so that the industry can have a common requirement= for various types of overlay for data center environment.

 

A draft on this topic could potentially trigger m= ore input on various constraints or requirement for hosts <-> OBN map= ping in data center environment.

 

Any opinion on this?

 

Linda Dunbar

 

--_000_4A95BA014132FF49AE685FAB4B9F17F632E1CE52dfweml505mbx_-- From ghanwani@gmail.com Fri Feb 10 18:12:29 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 120FF21F8495 for ; Fri, 10 Feb 2012 18:12:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.177 X-Spam-Level: X-Spam-Status: No, score=-4.177 tagged_above=-999 required=5 tests=[AWL=-1.200, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c0GVehQ7TSMG for ; Fri, 10 Feb 2012 18:12:28 -0800 (PST) Received: from mail-qw0-f51.google.com (mail-qw0-f51.google.com [209.85.216.51]) by ietfa.amsl.com (Postfix) with ESMTP id ECFC721F8491 for ; Fri, 10 Feb 2012 18:12:27 -0800 (PST) Received: by qan41 with SMTP id 41so2103472qan.10 for ; Fri, 10 Feb 2012 18:12:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=scPK7mkIH9YLc8d5evr4GOUELoNEOW+15eYKuWHu9iw=; b=A7SfOp3OY3FJaULNJWhxuNCvDMyEBT7bBzi6WS3yNYS7LT8TfI6TpHAd2ED4Rsc2bk mpkavpn6ynuCiAbYjYVj/43KBzDwK9Y1KYlO1mcIjzDGALGr3kiM94s2F9Y5WlVT4yDz nGLyZQOWxxRaGrpCavIVu6kVfLBuPEpqcylaY= MIME-Version: 1.0 Received: by 10.229.135.193 with SMTP id o1mr5319635qct.74.1328926346280; Fri, 10 Feb 2012 18:12:26 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Fri, 10 Feb 2012 18:12:25 -0800 (PST) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F632E1CE52@dfweml505-mbx> References: <4A95BA014132FF49AE685FAB4B9F17F632E1CE52@dfweml505-mbx> Date: Fri, 10 Feb 2012 18:12:25 -0800 X-Google-Sender-Auth: 3FONNKOGEPeD0yXf0oSt88mfnXU Message-ID: From: Anoop Ghanwani To: Linda Dunbar Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Feb 2012 02:12:29 -0000 I think this is a very interesting problem. If BOTH the MAC-to-IP binding, and the MAC-to-tunnel-end-point binding can be resolved using a directory service, it would eliminate the need for IP multicast support from the netw= ork, provided there are no multicast applications. If both cannot be done and ARP/ND must still be used to find the MAC-to-IP binding, then I don't see much value in a directory service to get only the MAC-to-tunnel-end-point mapping since that can be inferred from the ARP traffic itself (as long as a MAC address sits behind only a single tunnel end point). Anoop On Fri, Feb 10, 2012 at 1:28 PM, Linda Dunbar wro= te: > We all believe that the current trend for Data Center network is going > towards overlay network, be it TRILL, MAC-in-MAC (SPB), NV03, etc. In > Overlay network, each Overlay Border Node (OBN) has to map the destinatio= n > address of a data packet to an egress Overlay Border Node. Very much like > ARP/ND which is to resolve physical addresses (i.e. MAC Addresses) from > target IP addresses. > > > > Here are some distinct characteristics of hosts in Data Center: > > 1.=A0 Even though the communication between DC hosts to external peers ma= y be > small in volume comparing with the east-west (intra-data center) traffic, > majority of hosts in data center still do communicate with external peers= . > > > > 2.=A0 Traffic entering/exiting the Data Center Overlay could be via multi= ple > Overlay Border nodes. > > > > 3.=A0 The placement of VMs/hosts to server racks is orchestrated by Manag= ement > Entities. So Overlay Border Nodes can, at least in theory, get the hosts = <-> > OverlayBorderNode mapping information from directory server(s). There cou= ld > be multiple directory servers serving a very large data center. > > > > From the very high level view, the directory providing mapping from hosts= to > Overlay Border Node seems very straight forward. However, if you look > deeper, following issues come out: > > > > -=A0=A0 When hosts can enter or exit the overlay network via multiple OBN= s, is > it necessary for each OBN to be aware of which egress OBN can reach the > target? Or, is it necessary for each OBN to advertise its connectivity > status of all the attached hosts to all other OBNs? > > > > -=A0=A0 When migration is orchestrated by management entity (or entities)= , there > could be multiple instances of directory servers > > o=A0=A0 is it necessary for each directory server to have full informatio= n? What > if some directory servers only have partial information (for very large D= C > environment)? > > o=A0=A0 Is it necessary for each OBN to report connectivity status of all= the > attached hosts to directory servers? > > o=A0=A0 Should resolution requests be load balanced among them? > > > > -=A0=A0 A data frame arriving at OBN normally might have an VID in the da= ta > frame, the egress OBN might have a different VID value for the =A0same vi= rtual > network instance. Original VID loses its meaning when traversing to Egres= s > overlay border. Proper Ingress VID -> overlay network instance ID - >to > Egress VID =A0resolution (or mapping) is needed. > > > > -=A0=A0 Etc. > > > > It was suggested that ARMD WG should have a requirement draft on Address > Resolution from data center host addresses to Overlay Border Node address= es > so that the industry can have a common requirement for various types of > overlay for data center environment. > > > > A draft on this topic could potentially trigger more input on various > constraints or requirement for hosts <-> OBN mapping in data center > environment. > > > > Any opinion on this? > > > > Linda Dunbar > > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > From mmcbride7@gmail.com Mon Feb 13 16:01:47 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 41BB821F867A for ; Mon, 13 Feb 2012 16:01:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.867 X-Spam-Level: X-Spam-Status: No, score=-2.867 tagged_above=-999 required=5 tests=[AWL=-0.268, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4dPl5a3mEPwK for ; Mon, 13 Feb 2012 16:01:46 -0800 (PST) Received: from mail-lpp01m020-f172.google.com (mail-lpp01m020-f172.google.com [209.85.217.172]) by ietfa.amsl.com (Postfix) with ESMTP id EE86C21F8675 for ; Mon, 13 Feb 2012 16:01:45 -0800 (PST) Received: by lbbgk8 with SMTP id gk8so3237209lbb.31 for ; Mon, 13 Feb 2012 16:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Sb76doSZbeZ/6g+G4JODnxCdj+/wZS4uU1tGnZBQ38A=; b=H3VNBV8xul79zqPq5a4Kyahl2STn0dkD6PZeV+rye2jydDIC8wrl7E2GNeLVznZum2 YctNJUo0Y0z8k9xkWalE7XPQxASCn7Vfdo+uajZ06MOZYJEoU602LbgOuXN8rIUe8qf5 flAnVJAX9FVkcRBDzk6iUNz3PND6SE3p5CBHw= MIME-Version: 1.0 Received: by 10.112.28.169 with SMTP id c9mr6316591lbh.42.1329177704924; Mon, 13 Feb 2012 16:01:44 -0800 (PST) Received: by 10.112.45.99 with HTTP; Mon, 13 Feb 2012 16:01:44 -0800 (PST) In-Reply-To: References: <4A95BA014132FF49AE685FAB4B9F17F632E1CE52@dfweml505-mbx> Date: Mon, 13 Feb 2012 16:01:44 -0800 Message-ID: From: Mike McBride To: Anoop Ghanwani Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 00:01:47 -0000 Anoop, On Fri, Feb 10, 2012 at 6:12 PM, Anoop Ghanwani wro= te: > I think this is a very interesting problem. =A0If BOTH the MAC-to-IP bind= ing, > and the MAC-to-tunnel-end-point binding can be resolved using a directory > service, it would eliminate the need for IP multicast support from the ne= twork, > provided there are no multicast applications. A directory could eliminate any reliance upon IP Multicast for MAC to tunnel end point binding but would not eliminate the need for IP Multicast in the network. You would still want the dynamic nature of joining and pruning along with the efficient data delivery across the overlay network. Good solutions will take mcast applications into proper consideration. > If both cannot be done and ARP/ND must still be used to find the MAC-to-I= P > binding, then I don't see much value in a directory service to get only t= he > MAC-to-tunnel-end-point mapping since that can be inferred from the ARP > traffic itself (as long as a MAC address sits behind only a single > tunnel end point). You are probably right but, either way, a directory approach is probably worth further exploration. mike > Anoop > > On Fri, Feb 10, 2012 at 1:28 PM, Linda Dunbar w= rote: >> We all believe that the current trend for Data Center network is going >> towards overlay network, be it TRILL, MAC-in-MAC (SPB), NV03, etc. In >> Overlay network, each Overlay Border Node (OBN) has to map the destinati= on >> address of a data packet to an egress Overlay Border Node. Very much lik= e >> ARP/ND which is to resolve physical addresses (i.e. MAC Addresses) from >> target IP addresses. >> >> >> >> Here are some distinct characteristics of hosts in Data Center: >> >> 1.=A0 Even though the communication between DC hosts to external peers m= ay be >> small in volume comparing with the east-west (intra-data center) traffic= , >> majority of hosts in data center still do communicate with external peer= s. >> >> >> >> 2.=A0 Traffic entering/exiting the Data Center Overlay could be via mult= iple >> Overlay Border nodes. >> >> >> >> 3.=A0 The placement of VMs/hosts to server racks is orchestrated by Mana= gement >> Entities. So Overlay Border Nodes can, at least in theory, get the hosts= <-> >> OverlayBorderNode mapping information from directory server(s). There co= uld >> be multiple directory servers serving a very large data center. >> >> >> >> From the very high level view, the directory providing mapping from host= s to >> Overlay Border Node seems very straight forward. However, if you look >> deeper, following issues come out: >> >> >> >> -=A0=A0 When hosts can enter or exit the overlay network via multiple OB= Ns, is >> it necessary for each OBN to be aware of which egress OBN can reach the >> target? Or, is it necessary for each OBN to advertise its connectivity >> status of all the attached hosts to all other OBNs? >> >> >> >> -=A0=A0 When migration is orchestrated by management entity (or entities= ), there >> could be multiple instances of directory servers >> >> o=A0=A0 is it necessary for each directory server to have full informati= on? What >> if some directory servers only have partial information (for very large = DC >> environment)? >> >> o=A0=A0 Is it necessary for each OBN to report connectivity status of al= l the >> attached hosts to directory servers? >> >> o=A0=A0 Should resolution requests be load balanced among them? >> >> >> >> -=A0=A0 A data frame arriving at OBN normally might have an VID in the d= ata >> frame, the egress OBN might have a different VID value for the =A0same v= irtual >> network instance. Original VID loses its meaning when traversing to Egre= ss >> overlay border. Proper Ingress VID -> overlay network instance ID - >to >> Egress VID =A0resolution (or mapping) is needed. >> >> >> >> -=A0=A0 Etc. >> >> >> >> It was suggested that ARMD WG should have a requirement draft on Address >> Resolution from data center host addresses to Overlay Border Node addres= ses >> so that the industry can have a common requirement for various types of >> overlay for data center environment. >> >> >> >> A draft on this topic could potentially trigger more input on various >> constraints or requirement for hosts <-> OBN mapping in data center >> environment. >> >> >> >> Any opinion on this? >> >> >> >> Linda Dunbar >> >> >> >> >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd >> > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From ghanwani@gmail.com Tue Feb 14 08:23:52 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4A2DF21F8681 for ; Tue, 14 Feb 2012 08:23:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.577 X-Spam-Level: X-Spam-Status: No, score=-3.577 tagged_above=-999 required=5 tests=[AWL=-0.600, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JnkajrvaG38f for ; Tue, 14 Feb 2012 08:23:51 -0800 (PST) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id A93D121F8557 for ; Tue, 14 Feb 2012 08:23:48 -0800 (PST) Received: by qcsq5 with SMTP id q5so94658qcs.31 for ; Tue, 14 Feb 2012 08:23:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=t0z7otEChRieWU8v0PXq2gFB6j5aAvLtTUVTwsoCR0o=; b=BQFNGptAnpKoJDcbEedHjSDTE41XNEUahiSqMiGWpR9nSN0g5QVKdGwAVV3vo4sX1e JdoambP7g8NBfJVw3DRFFZF+5pObGYbFNXtTfS9egrVS/t4I915RRwySXDPkTDgWzqxi QSQJPAjOVGwK5IzuZ1xIITGi17JR2UdekXLyI= MIME-Version: 1.0 Received: by 10.229.136.77 with SMTP id q13mr12552784qct.154.1329236628181; Tue, 14 Feb 2012 08:23:48 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Tue, 14 Feb 2012 08:23:47 -0800 (PST) In-Reply-To: References: <4A95BA014132FF49AE685FAB4B9F17F632E1CE52@dfweml505-mbx> Date: Tue, 14 Feb 2012 08:23:47 -0800 X-Google-Sender-Auth: AL0uJoNcPcXzdEf9vx5P8K_x3K4 Message-ID: From: Anoop Ghanwani To: Mike McBride Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 16:23:52 -0000 On Mon, Feb 13, 2012 at 4:01 PM, Mike McBride wrote: > On Fri, Feb 10, 2012 at 6:12 PM, Anoop Ghanwani w= rote: >> I think this is a very interesting problem. =A0If BOTH the MAC-to-IP bin= ding, >> and the MAC-to-tunnel-end-point binding can be resolved using a director= y >> service, it would eliminate the need for IP multicast support from the n= etwork, >> provided there are no multicast applications. > > A directory could eliminate any reliance upon IP Multicast for MAC to > tunnel end point binding but would not eliminate the need for IP > Multicast in the network. You would still want the dynamic nature of > joining and pruning along with the efficient data delivery across the > overlay network. Good solutions will take mcast applications into > proper consideration. I agree that we would still need IP multicast in the network to support multicast applications. My point was that a directory-based mechanism would eliminate the need for IP multicast if the only reason it was need was for the purpose of getting ARP traffic through the network. That said, there are unicast applications (such as Microsoft's Network Load Balancing) that require multicast at L2, and because they require multicast at L2, such a network would still need to support IP multicast. So really, the directory-based approach would be helpful only in a narrow set of circumstances -- no multicast applications including things like NLB. Anoop From Peter.AshwoodSmith@huawei.com Tue Feb 14 10:31:45 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 19A8921E809C for ; Tue, 14 Feb 2012 10:31:45 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.549 X-Spam-Level: X-Spam-Status: No, score=-2.549 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6zqGpy-qQoGQ for ; Tue, 14 Feb 2012 10:31:44 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 33FD021E8058 for ; Tue, 14 Feb 2012 10:31:32 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml201-edg.china.huawei.com) ([172.18.9.243]) by dfwrg01-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADP16476; Tue, 14 Feb 2012 13:31:31 -0500 (EST) Received: from DFWEML405-HUB.china.huawei.com (10.193.5.102) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.323.3; Tue, 14 Feb 2012 10:29:11 -0800 Received: from DFWEML503-MBX.china.huawei.com ([10.124.31.29]) by dfweml405-hub.china.huawei.com ([10.193.5.102]) with mapi id 14.01.0323.003; Wed, 15 Feb 2012 02:29:13 +0800 From: AshwoodsmithPeter To: Anoop Ghanwani , Mike McBride Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cgAasSGAAHDIVwAAIkw2gAAU7kJQ Date: Tue, 14 Feb 2012 18:29:13 +0000 Message-ID: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.193.60.75] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 18:31:45 -0000 >That said, there are unicast applications (such as Microsoft's Network >Load Balancing) that require multicast at L2, and because they >require multicast at L2, such a network would still need to support >IP multicast. >So really, the directory-based approach would be helpful >only in a narrow set of circumstances -- no multicast applications >including things like NLB. Anoop, even if you have a good multicast in the underlay network and its be= ing used by things like MS NLB there is still enormous value in eliminating= the multicast methods that are used to establish the tripple relationships via some form of database etc. Peter From ghanwani@gmail.com Tue Feb 14 10:55:30 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 08B2521F8780 for ; Tue, 14 Feb 2012 10:55:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.577 X-Spam-Level: X-Spam-Status: No, score=-3.577 tagged_above=-999 required=5 tests=[AWL=-0.600, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5GqmPgpEdzm5 for ; Tue, 14 Feb 2012 10:55:29 -0800 (PST) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 99DD621F8778 for ; Tue, 14 Feb 2012 10:55:26 -0800 (PST) Received: by qcsq5 with SMTP id q5so199649qcs.31 for ; Tue, 14 Feb 2012 10:55:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=eBbmalO9iC+C8VDXwtkEvA8wrdKvzNa/ivBl6jsJ+8o=; b=Zu0yHTwgXEFj5Ko2/7A9j+TpMT2lOM2H+P3X9szg9MCKJNzeP0RLR2Nyfj1/QlL2GE CQilxUEWDdZmQBa0ffVNh71xlIvaZYW/++EvZYWLVZgz6I8hk1+hYwXw1vch9pFL4N6X KL0/aZMz7VXrAj0vVTztRGkCB+2u0npD0JWe8= MIME-Version: 1.0 Received: by 10.229.76.89 with SMTP id b25mr12860630qck.54.1329245726110; Tue, 14 Feb 2012 10:55:26 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Tue, 14 Feb 2012 10:55:26 -0800 (PST) In-Reply-To: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> Date: Tue, 14 Feb 2012 10:55:26 -0800 X-Google-Sender-Auth: b3_PrvJ90IrfO9wjWbjywpXGKDU Message-ID: From: Anoop Ghanwani To: AshwoodsmithPeter Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 18:55:30 -0000 On Tue, Feb 14, 2012 at 10:29 AM, AshwoodsmithPeter wrote: >>That said, there are unicast applications (such as Microsoft's Network >>Load Balancing) that require multicast at L2, and because they >>require multicast at L2, such a network would still need to support >>IP multicast. > >>So really, the directory-based approach would be helpful >>only in a narrow set of circumstances -- no multicast applications >>including things like NLB. > > Anoop, even if you have a good multicast in the underlay network and its = being used by things like MS NLB there is still enormous value in eliminati= ng the multicast methods that are used to establish the tripple relationships via some form of database etc. That's correct, Peter. As I said in my first response, we have to solve both problems -- ipaddr->cmac and cmac->bmac/tunnel. If we solve only one, then the value proposition is significantly diluted. As an added benefit, if we solve this problem and there is no other need for L2 multicast in the network (NLB, directed broadcast), then the overlay does not need multicast support from the underlying network. Requiring IP multicast (especially PIM bidir) from the network is really a big deal, especially at the kind of scale that the VXLAN/NVGRE proposals need it. Anoop From david.i.allan@ericsson.com Tue Feb 14 11:05:48 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6BC8A21E80A7 for ; Tue, 14 Feb 2012 11:05:48 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.32 X-Spam-Level: X-Spam-Status: No, score=-6.32 tagged_above=-999 required=5 tests=[AWL=0.279, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z++zrkEEr7eW for ; Tue, 14 Feb 2012 11:05:43 -0800 (PST) Received: from imr4.ericy.com (imr4.ericy.com [198.24.6.9]) by ietfa.amsl.com (Postfix) with ESMTP id 0854D21E8087 for ; Tue, 14 Feb 2012 11:05:42 -0800 (PST) Received: from eusaamw0712.eamcs.ericsson.se ([147.117.20.181]) by imr4.ericy.com (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id q1EJ5abX020281; Tue, 14 Feb 2012 13:05:37 -0600 Received: from EUSAACMS0703.eamcs.ericsson.se ([169.254.1.142]) by eusaamw0712.eamcs.ericsson.se ([147.117.20.181]) with mapi; Tue, 14 Feb 2012 14:05:31 -0500 From: David Allan I To: Anoop Ghanwani , AshwoodsmithPeter Date: Tue, 14 Feb 2012 14:05:30 -0500 Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczrSkIVFf3wgkz1TkmILegWRm7rvwAADlbw Message-ID: <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 19:05:48 -0000 I have to admit I am a bit confused by the direction this is going.=20 There seems to be a view that multicast is bad, yet it is a useful tool to = permit a significant amount of functionality to be delegated to end systems= , and is a part of useful address aggregation in the form of the subnet. IM= O virtualized broadcast domains are good things, not something to be avoide= d, and they permit a trusted resource community to do a lot of self-organiz= ing.=20 Perhaps you should be thinking about improving multicast, not evicerating e= verything else. And as a general design principle, I believe that applies t= o most areas where the tail is wagging the dog in these deliberations. My 2 cents Dave -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Ano= op Ghanwani Sent: Tuesday, February 14, 2012 10:55 AM To: AshwoodsmithPeter Cc: Thomas Narten; armd@ietf.org; Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay ed= ge nodes. Any opinion? On Tue, Feb 14, 2012 at 10:29 AM, AshwoodsmithPeter wrote: >>That said, there are unicast applications (such as Microsoft's Network=20 >>Load Balancing) that require multicast at L2, and because they require=20 >>multicast at L2, such a network would still need to support IP=20 >>multicast. > >>So really, the directory-based approach would be helpful only in a=20 >>narrow set of circumstances -- no multicast applications including=20 >>things like NLB. > > Anoop, even if you have a good multicast in the underlay network and its = being used by things like MS NLB there is still enormous value in eliminati= ng the multicast methods that are used to establish the tripple relationships via some form of database etc. That's correct, Peter. As I said in my first response, we have to solve bo= th problems -- ipaddr->cmac and cmac->bmac/tunnel. If we solve only one, then the value proposition is significantly diluted. = As an added benefit, if we solve this problem and there is no other need f= or L2 multicast in the network (NLB, directed broadcast), then the overlay = does not need multicast support from the underlying network. Requiring IP = multicast (especially PIM bidir) from the network is really a big deal, esp= ecially at the kind of scale that the VXLAN/NVGRE proposals need it. Anoop _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From ghanwani@gmail.com Tue Feb 14 11:36:02 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2099821E8023 for ; Tue, 14 Feb 2012 11:36:02 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.457 X-Spam-Level: X-Spam-Status: No, score=-3.457 tagged_above=-999 required=5 tests=[AWL=-0.480, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bTNHOqIYv8M1 for ; Tue, 14 Feb 2012 11:36:01 -0800 (PST) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 7212821E801B for ; Tue, 14 Feb 2012 11:36:01 -0800 (PST) Received: by qafi29 with SMTP id i29so2471685qaf.10 for ; Tue, 14 Feb 2012 11:36:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=NFSimr/YZavjtnTnkT+AIuNQ6oQ6qBcDu6hnizH2ScU=; b=UDhk1Gy8lFpobIzlXM4Vmm5xF3awReUD37uz1EOcQn6O3cOgukETwWmc1+FMMD5Cwe xKH4thy46uCHVEoJPwbcjlknzO8xcUKpMHY8CcxjMSN7MjhpkedYoK+T4MMZVjnEhl78 xmQECT59HJWI6O+Du0aFBWQtoQVnkfp9SM7uo= MIME-Version: 1.0 Received: by 10.229.135.10 with SMTP id l10mr13141379qct.14.1329248160993; Tue, 14 Feb 2012 11:36:00 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Tue, 14 Feb 2012 11:36:00 -0800 (PST) In-Reply-To: <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Date: Tue, 14 Feb 2012 11:36:00 -0800 X-Google-Sender-Auth: YWYtQFkAO18alOLctIqmFsZirio Message-ID: From: Anoop Ghanwani To: David Allan I Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 19:36:02 -0000 On Tue, Feb 14, 2012 at 11:05 AM, David Allan I wrote: > I have to admit I am a bit confused by the direction this is going. > > There seems to be a view that multicast is bad, yet it is a useful tool t= o permit a significant amount of functionality to be delegated to end syste= ms, and is a part of useful address aggregation in the form of the subnet. = IMO virtualized broadcast domains are good things, not something to be avoi= ded, and they permit a trusted resource community to do a lot of self-organ= izing. > > Perhaps you should be thinking about improving multicast, not evicerating= everything else. And as a general design principle, I believe that applies= to most areas where the tail is wagging the dog in these deliberations. Theoretically speaking, it is easy to brush this off and say "let's just im= prove multicast", and if we do nothing then that is pretty much what will happen. However, as a practical matter, there is a lot of hardware out there that has significant problems with multicast. This means we delay the deploymen= t of such solutions until the hardware catches up (if it ever does), or let s= mart people who don't care about standards figure out ways around it that are non-interoperable, and worry about about the standardization aspect later. Anoop From david.i.allan@ericsson.com Tue Feb 14 11:59:04 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A504B21F8663 for ; Tue, 14 Feb 2012 11:59:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.339 X-Spam-Level: X-Spam-Status: No, score=-6.339 tagged_above=-999 required=5 tests=[AWL=0.260, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y-tJZd-HP27u for ; Tue, 14 Feb 2012 11:59:03 -0800 (PST) Received: from imr4.ericy.com (imr4.ericy.com [198.24.6.9]) by ietfa.amsl.com (Postfix) with ESMTP id AEF7921F8637 for ; Tue, 14 Feb 2012 11:59:03 -0800 (PST) Received: from eusaamw0706.eamcs.ericsson.se ([147.117.20.31]) by imr4.ericy.com (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id q1EJwwum000980; Tue, 14 Feb 2012 13:59:00 -0600 Received: from EUSAACMS0703.eamcs.ericsson.se ([169.254.1.142]) by eusaamw0706.eamcs.ericsson.se ([147.117.20.31]) with mapi; Tue, 14 Feb 2012 14:58:55 -0500 From: David Allan I To: Anoop Ghanwani Date: Tue, 14 Feb 2012 14:58:54 -0500 Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczrT+idram2Vye5Shep+1EpD+DruQAApkHA Message-ID: <60C093A41B5E45409A19D42CF7786DFD522A9BE2DC@EUSAACMS0703.eamcs.ericsson.se> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 19:59:04 -0000 Hi Anoop: Eeesh, no offence but you are advocating overlaying complexity on complexit= y vs. cutting to the heart of the problem.=20 "No-hardware left behind" IMO is social engineering, not technology. Cheers Dave=20 -----Original Message----- From: ghanwani@gmail.com [mailto:ghanwani@gmail.com] On Behalf Of Anoop Gha= nwani Sent: Tuesday, February 14, 2012 11:36 AM To: David Allan I Cc: AshwoodsmithPeter; Thomas Narten; armd@ietf.org; Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay ed= ge nodes. Any opinion? On Tue, Feb 14, 2012 at 11:05 AM, David Allan I wrote: > I have to admit I am a bit confused by the direction this is going. > > There seems to be a view that multicast is bad, yet it is a useful tool t= o permit a significant amount of functionality to be delegated to end syste= ms, and is a part of useful address aggregation in the form of the subnet. = IMO virtualized broadcast domains are good things, not something to be avoi= ded, and they permit a trusted resource community to do a lot of self-organ= izing. > > Perhaps you should be thinking about improving multicast, not evicerating= everything else. And as a general design principle, I believe that applies= to most areas where the tail is wagging the dog in these deliberations. Theoretically speaking, it is easy to brush this off and say "let's just im= prove multicast", and if we do nothing then that is pretty much what will h= appen. However, as a practical matter, there is a lot of hardware out there that h= as significant problems with multicast. This means we delay the deployment= of such solutions until the hardware catches up (if it ever does), or let = smart people who don't care about standards figure out ways around it that = are non-interoperable, and worry about about the standardization aspect lat= er. Anoop From mmcbride7@gmail.com Tue Feb 14 12:14:49 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C122D21E80D3 for ; Tue, 14 Feb 2012 12:14:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.84 X-Spam-Level: X-Spam-Status: No, score=-2.84 tagged_above=-999 required=5 tests=[AWL=-0.241, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nNBO0cmN1DQe for ; Tue, 14 Feb 2012 12:14:48 -0800 (PST) Received: from mail-lpp01m020-f172.google.com (mail-lpp01m020-f172.google.com [209.85.217.172]) by ietfa.amsl.com (Postfix) with ESMTP id 5681221E8106 for ; Tue, 14 Feb 2012 12:14:48 -0800 (PST) Received: by lbbgk8 with SMTP id gk8so144430lbb.31 for ; Tue, 14 Feb 2012 12:14:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=exLyegcWe/TJ6jbRF5eYWEvcO11cEpWxhlGg0a6vBrE=; b=PmU7i2lUftYlCvZxdjchNnfGMAW7mBjG7kTeS4J6t6kiwaCZ4oIDaqkLb971qWDWxX klZ/eh8tyz4MzAhPWG2v9c7EFCPXMcbqULHDfDjgxp/NBJlqE+KSGsUtib2F8+JOEI8A nCYaOv8s5SWngvA0+YzqppX1bd7W3ru91qFe8= MIME-Version: 1.0 Received: by 10.112.86.67 with SMTP id n3mr7762878lbz.29.1329250484219; Tue, 14 Feb 2012 12:14:44 -0800 (PST) Received: by 10.112.45.99 with HTTP; Tue, 14 Feb 2012 12:14:44 -0800 (PST) In-Reply-To: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Date: Tue, 14 Feb 2012 12:14:44 -0800 Message-ID: From: Mike McBride To: Anoop Ghanwani Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 20:14:50 -0000 And what protocol improvements are needed for multicast with overlays? The multicast community is ready to help. Multicast makes us happy. Or are you solely talking about vendor implementation, s/w release, hw support problems? AFAIK multicast works fairly well with VXLAN, TRILL, L2VPN... What's missing? mike On Tue, Feb 14, 2012 at 11:36 AM, Anoop Ghanwani wr= ote: > On Tue, Feb 14, 2012 at 11:05 AM, David Allan I > wrote: >> I have to admit I am a bit confused by the direction this is going. >> >> There seems to be a view that multicast is bad, yet it is a useful tool = to permit a significant amount of functionality to be delegated to end syst= ems, and is a part of useful address aggregation in the form of the subnet.= IMO virtualized broadcast domains are good things, not something to be avo= ided, and they permit a trusted resource community to do a lot of self-orga= nizing. >> >> Perhaps you should be thinking about improving multicast, not eviceratin= g everything else. And as a general design principle, I believe that applie= s to most areas where the tail is wagging the dog in these deliberations. > > Theoretically speaking, it is easy to brush this off and say "let's just = improve > multicast", and if we do nothing then that is pretty much what will happe= n. > However, as a practical matter, there is a lot of hardware out there that > has significant problems with multicast. =A0This means we delay the deplo= yment > of such solutions until the hardware catches up (if it ever does), or let= smart > people who don't care about standards figure out ways around it that are > non-interoperable, and worry about about the standardization aspect later= . > > Anoop > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From ghanwani@gmail.com Tue Feb 14 12:32:59 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F0D4D21F8623 for ; Tue, 14 Feb 2012 12:32:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.377 X-Spam-Level: X-Spam-Status: No, score=-3.377 tagged_above=-999 required=5 tests=[AWL=-0.400, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z+FJMrbYtJiU for ; Tue, 14 Feb 2012 12:32:59 -0800 (PST) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 1A41921F8622 for ; Tue, 14 Feb 2012 12:32:59 -0800 (PST) Received: by qafi29 with SMTP id i29so2505811qaf.10 for ; Tue, 14 Feb 2012 12:32:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=OrQPhRSTvJUXQ7D+VzVM5e5ZeTXoZ7cS44R40CAwSVw=; b=Gv/xnrl+AreFFCXzAOauwjv/GZGEarpDGuTSRmnqo5745p9WL90z4lv4tbYYUCyw5a s/dCemYnHt6mQveTsTtyJOVLPPj0B8+55IXy5V/zgVxNWygqifNC8iBByxsQwbIZ6lz0 hn0kawsHS7c3t3knOXXiSCP1noJaplL1wjh4E= MIME-Version: 1.0 Received: by 10.229.76.132 with SMTP id c4mr12989139qck.134.1329251578586; Tue, 14 Feb 2012 12:32:58 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Tue, 14 Feb 2012 12:32:58 -0800 (PST) In-Reply-To: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Date: Tue, 14 Feb 2012 12:32:58 -0800 X-Google-Sender-Auth: gdEHTh81wgENfooE0YR-pWV9INk Message-ID: From: Anoop Ghanwani To: Mike McBride Content-Type: text/plain; charset=ISO-8859-1 Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 20:33:00 -0000 On Tue, Feb 14, 2012 at 12:14 PM, Mike McBride wrote: > And what protocol improvements are needed for multicast with overlays? > The multicast community is ready to help. Multicast makes us happy. Or > are you solely talking about vendor implementation, s/w release, hw > support problems? AFAIK multicast works fairly well with VXLAN, TRILL, > L2VPN... What's missing? The whole problem of sending L2 multicast over a campus or data center backbone, in any sort of significant way, is a new one enabled for the first time by overlays. There are interesting challenges when pushing large amounts of multicast traffic through a network, and have thus far been dealt with using purpose-built networks. While the overlay proposals have been careful not to impose new protocol requirements, they have not addressed the issues of performance and scalability, nor the large-scale availability of these protocols. Anoop From mmcbride7@gmail.com Tue Feb 14 13:16:30 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 34C8F21F8652 for ; Tue, 14 Feb 2012 13:16:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.318 X-Spam-Level: X-Spam-Status: No, score=-3.318 tagged_above=-999 required=5 tests=[AWL=0.281, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rxMyxowj3aAr for ; Tue, 14 Feb 2012 13:16:29 -0800 (PST) Received: from mail-lpp01m010-f44.google.com (mail-lpp01m010-f44.google.com [209.85.215.44]) by ietfa.amsl.com (Postfix) with ESMTP id 443E221F8617 for ; Tue, 14 Feb 2012 13:16:29 -0800 (PST) Received: by lahl5 with SMTP id l5so441035lah.31 for ; Tue, 14 Feb 2012 13:16:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=9FV+6VcKYOObSPY8beBzuwZVKRuOuI4WnS5L1pi5okg=; b=Jt1sCWvzsXlKqFbL6/Q+Gel96NIk+1A5T2gCHCAvosfMwMv21SNw45If58lkMZN7La 3Ii429VeO4uItfnpTffgEOQzA8Im1qCxMCtUCGq1qlpPensV/OXzPx04omP/NiTnY3rm Ny5l24/fRC/2a5/hMszTyAOhtlVgRqIs4wFkI= MIME-Version: 1.0 Received: by 10.112.28.169 with SMTP id c9mr7841629lbh.42.1329254188223; Tue, 14 Feb 2012 13:16:28 -0800 (PST) Received: by 10.112.45.99 with HTTP; Tue, 14 Feb 2012 13:16:28 -0800 (PST) In-Reply-To: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Date: Tue, 14 Feb 2012 13:16:28 -0800 Message-ID: From: Mike McBride To: Anoop Ghanwani Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 21:16:30 -0000 Anoop, On Tue, Feb 14, 2012 at 12:32 PM, Anoop Ghanwani wr= ote: > On Tue, Feb 14, 2012 at 12:14 PM, Mike McBride wrot= e: >> And what protocol improvements are needed for multicast with overlays? >> The multicast community is ready to help. Multicast makes us happy. Or >> are you solely talking about vendor implementation, s/w release, hw >> support problems? AFAIK multicast works fairly well with VXLAN, TRILL, >> L2VPN... What's missing? > > The whole problem of sending L2 multicast over a campus or data center > backbone, in any sort of significant way, is a new one enabled for the fi= rst > time by overlays. =A0There are interesting challenges when pushing large > amounts of multicast traffic through a network, and =A0have thus far been= dealt > with using purpose-built networks. =A0While the overlay proposals have > been careful not to impose new protocol requirements, they have not > addressed the issues of performance and scalability, nor the large-scale > availability of these protocols. There are interesting challenges with unicast and multicast in big flat DC networks. That's why we are here. One of the constant themes we hear with multicast is the FUD that it doesn't scale. Especially from vendors who are pushing another solution or who have a crappy multicast implementation. Meanwhile multicast is being deployed in large scale networks without scaling issue. Large L2 overlay networks? I don't know. Would be good to find out from the community about performance and scalability of multicast in the DC. mike From ghanwani@gmail.com Tue Feb 14 13:53:22 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 652A221E801D for ; Tue, 14 Feb 2012 13:53:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.32 X-Spam-Level: X-Spam-Status: No, score=-3.32 tagged_above=-999 required=5 tests=[AWL=-0.343, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JRKp6EgiN+gx for ; Tue, 14 Feb 2012 13:53:21 -0800 (PST) Received: from mail-qy0-f172.google.com (mail-qy0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 7CDBA21E8011 for ; Tue, 14 Feb 2012 13:53:21 -0800 (PST) Received: by qcsq5 with SMTP id q5so306886qcs.31 for ; Tue, 14 Feb 2012 13:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=f4g08YO9YDoKBYlFWSyp8Y+MS4u4rW2yCGWGn6VhYlk=; b=XE6aUGk1X3OdnAIA/thIDvNdfBz/obtwqS7ZkrOKglKCrSNLzUrPABhG3CIDN4K3UK 3vKACbC4ouseOMMqb2kGng86+0OmB1F804Fzz6Ahbm17iEyCA4wzY7x4kQJYooxEWOy6 WLBSzhTJ0YTU7O+76NFaVCTeECXM+8e/le+pY= MIME-Version: 1.0 Received: by 10.229.77.72 with SMTP id f8mr13316279qck.34.1329256401023; Tue, 14 Feb 2012 13:53:21 -0800 (PST) Sender: ghanwani@gmail.com Received: by 10.229.155.206 with HTTP; Tue, 14 Feb 2012 13:53:20 -0800 (PST) In-Reply-To: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Date: Tue, 14 Feb 2012 13:53:20 -0800 X-Google-Sender-Auth: s0yGiJBjN_IAbAxzvF9UBGhRXts Message-ID: From: Anoop Ghanwani To: Mike McBride Content-Type: text/plain; charset=ISO-8859-1 Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 21:53:22 -0000 On Tue, Feb 14, 2012 at 1:16 PM, Mike McBride wrote: > There are interesting challenges with unicast and multicast in big > flat DC networks. That's why we are here. One of the constant themes > we hear with multicast is the FUD that it doesn't scale. Especially > from vendors who are pushing another solution or who have a crappy > multicast implementation. Meanwhile multicast is being deployed in > large scale networks without scaling issue. Like I said earlier...we have large-scale multicast today, but those are purpose-built networks, and at least today, they are no where near the kinds of scale where people want to go with overlay technology (in terms of size of network and traffic patterns). > Large L2 overlay networks? > I don't know. Would be good to find out from the community about > performance and scalability of multicast in the DC. Perhaps it's best we wait until the problems are little more obvious. Or may be they're just non-existent and I get proven wrong. Either way, it looks like the correct thing for now is to do nothing. Anoop From Peter.AshwoodSmith@huawei.com Tue Feb 14 14:18:53 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8E46521E800E for ; Tue, 14 Feb 2012 14:18:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.556 X-Spam-Level: X-Spam-Status: No, score=-2.556 tagged_above=-999 required=5 tests=[AWL=0.043, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q0+pm5ynQiAN for ; Tue, 14 Feb 2012 14:18:52 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 6C84C21F84EC for ; Tue, 14 Feb 2012 14:18:52 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml201-edg.china.huawei.com) ([172.18.9.243]) by dfwrg01-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADP30506; Tue, 14 Feb 2012 17:18:52 -0500 (EST) Received: from DFWEML407-HUB.china.huawei.com (10.193.5.132) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.323.3; Tue, 14 Feb 2012 14:15:29 -0800 Received: from DFWEML503-MBX.china.huawei.com ([10.124.31.29]) by dfweml407-hub.china.huawei.com ([10.193.5.132]) with mapi id 14.01.0323.003; Wed, 15 Feb 2012 06:15:30 +0800 From: AshwoodsmithPeter To: Anoop Ghanwani , Mike McBride Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AQHM60o2jmo4HB2iRmy2e6GgqFO6cpY9RsQA//78TQCAAArTAIAABRgAgAAMJwCAARaGAP//enGw Date: Tue, 14 Feb 2012 22:15:29 +0000 Message-ID: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.193.60.75] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 22:18:53 -0000 Irrespective of what is right or wrong etc. technically, I think it would b= e very useful to have a taxonomy of the different applications that are usi= ng multicast / broadcast for things other than ARP. The complexity of the underlay depends a lot on what happens with the use o= f multicast by the overlay. For example. Assume Microsoft load balancing (for better or worse). If I re= member correctly, and sombody correct me if I'm wrong, it generates multica= st frames from otherwise unicast packets at the router so that multiple hos= ts all get copies and then only one responds for a given flow... or somethi= ng like that. So if we are using something like a VXLAN overlay we need to map the L2 mul= ticast in the overlay to L3 multicast in the underlay or do head end replic= ation in the overlay and get a wack load of duplicate frames on the first l= ink from the router to the core "switch" (really it's a router). Not good. = The solution presumably is to run potentially thousands of PIM-SM or someth= ing messages to generate/maintain the required multicast state in the IP un= derlay. Anyway the behavior of the upper layer with respect to broadcast / multicas= t affects the choice of head end, (*,G) or (S,G) replication in the underla= y, which affects the opex and capex of the entire solution, so I think its = an important question to get a handle on earlier rather than later. Peter -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Ano= op Ghanwani Sent: Tuesday, February 14, 2012 4:53 PM To: Mike McBride Cc: Thomas Narten; armd@ietf.org; Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay ed= ge nodes. Any opinion? On Tue, Feb 14, 2012 at 1:16 PM, Mike McBride wrote: > There are interesting challenges with unicast and multicast in big > flat DC networks. That's why we are here. One of the constant themes > we hear with multicast is the FUD that it doesn't scale. Especially > from vendors who are pushing another solution or who have a crappy > multicast implementation. Meanwhile multicast is being deployed in > large scale networks without scaling issue. Like I said earlier...we have large-scale multicast today, but those are purpose-built networks, and at least today, they are no where near the kinds of scale where people want to go with overlay technology (in terms of size of network and traffic patterns). > Large L2 overlay networks? > I don't know. Would be good to find out from the community about > performance and scalability of multicast in the DC. Perhaps it's best we wait until the problems are little more obvious. Or may be they're just non-existent and I get proven wrong. Either way, it looks like the correct thing for now is to do nothing. Anoop _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From mksmith@adhost.com Tue Feb 14 14:22:57 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1D33721E800E for ; Tue, 14 Feb 2012 14:22:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id InUNDzhJCPRm for ; Tue, 14 Feb 2012 14:22:52 -0800 (PST) Received: from mail-in07.adhost.com (mail-in07.adhost.com [216.211.128.137]) by ietfa.amsl.com (Postfix) with ESMTP id D25F521F8562 for ; Tue, 14 Feb 2012 14:22:51 -0800 (PST) Received: from AD-EXH02.adhost.lan (unknown [10.142.0.21]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail-in07.adhost.com (Postfix) with ESMTPS id 9A7848ADF49; Tue, 14 Feb 2012 14:22:50 -0800 (PST) (envelope-from mksmith@adhost.com) Received: from AD-EXH02.adhost.lan ([fe80::1c5b:7ead:8ba3:6108]) by AD-EXH02.adhost.lan ([fe80::1c5b:7ead:8ba3:6108%11]) with mapi id 14.01.0255.000; Tue, 14 Feb 2012 14:22:50 -0800 From: "Michael K. Smith - Adhost" To: AshwoodsmithPeter , Anoop Ghanwani , Mike McBride Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cgAasSGAAJJPaQAAIkw2gAAEYXaAAADqZQAAAFoBAAABELEAAAFaTgAAAKMEAAABhOwAAAFJnQAAAMYJgAAQkC3Q Date: Tue, 14 Feb 2012 22:22:49 +0000 Message-ID: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> In-Reply-To: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.142.1.52] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2012 22:22:57 -0000 > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > AshwoodsmithPeter > Sent: Tuesday, February 14, 2012 2:15 PM > To: Anoop Ghanwani; Mike McBride > Cc: Thomas Narten; Igor Gashinsky; armd@ietf.org > Subject: Re: [armd] address resolution requirement from hosts to overlay > edge nodes. Any opinion? >=20 >=20 > Irrespective of what is right or wrong etc. technically, I think it would= be very > useful to have a taxonomy of the different applications that are using > multicast / broadcast for things other than ARP. >=20 > The complexity of the underlay depends a lot on what happens with the use > of multicast by the overlay. >=20 > For example. Assume Microsoft load balancing (for better or worse). If I > remember correctly, and sombody correct me if I'm wrong, it generates > multicast frames from otherwise unicast packets at the router so that > multiple hosts all get copies and then only one responds for a given flow= ... or > something like that. >=20 > So if we are using something like a VXLAN overlay we need to map the L2 > multicast in the overlay to L3 multicast in the underlay or do head end > replication in the overlay and get a wack load of duplicate frames on the= first > link from the router to the core "switch" (really it's a router). Not goo= d. The > solution presumably is to run potentially thousands of PIM-SM or somethin= g > messages to generate/maintain the required multicast state in the IP > underlay. >=20 > Anyway the behavior of the upper layer with respect to broadcast / multic= ast > affects the choice of head end, (*,G) or (S,G) replication in the underla= y, > which affects the opex and capex of the entire solution, so I think its a= n > important question to get a handle on earlier rather than later. >=20 Microsoft is configurable for IGMP Multicast and Unicast. There are also t= he other LB protocols such as CARP, VRRP, GLBP, HSRP, etc. =20 Mike From joelja@bogus.com Tue Feb 14 21:36:55 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A861B21F8693 for ; Tue, 14 Feb 2012 21:36:55 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.739 X-Spam-Level: X-Spam-Status: No, score=-102.739 tagged_above=-999 required=5 tests=[AWL=-0.139, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l9N8yWptiqky for ; Tue, 14 Feb 2012 21:36:54 -0800 (PST) Received: from nagasaki.bogus.com (nagasaki.bogus.com [IPv6:2001:418:1::81]) by ietfa.amsl.com (Postfix) with ESMTP id 8DCB721F8683 for ; Tue, 14 Feb 2012 21:36:53 -0800 (PST) Received: from Joels-MacBook-Pro.local (c-98-234-216-143.hsd1.ca.comcast.net [98.234.216.143]) (authenticated bits=0) by nagasaki.bogus.com (8.14.4/8.14.4) with ESMTP id q1F5aQsV014770 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Wed, 15 Feb 2012 05:36:27 GMT (envelope-from joelja@bogus.com) Message-ID: <4F3B4459.2080907@bogus.com> Date: Tue, 14 Feb 2012 21:36:25 -0800 From: Joel jaeggli User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: AshwoodsmithPeter References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> In-Reply-To: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (nagasaki.bogus.com [147.28.0.81]); Wed, 15 Feb 2012 05:36:28 +0000 (UTC) Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 05:36:55 -0000 On 2/14/12 14:15 , AshwoodsmithPeter wrote: > > Irrespective of what is right or wrong etc. technically, I think it > would be very useful to have a taxonomy of the different applications > that are using multicast / broadcast for things other than ARP. ganglia is a common datacenter multicast user vertica clusters can use broadcast for communication and span hundres of nodes, however unwise that approach may be. > The complexity of the underlay depends a lot on what happens with the > use of multicast by the overlay. > > For example. Assume Microsoft load balancing (for better or worse). > If I remember correctly, and sombody correct me if I'm wrong, it > generates multicast frames from otherwise unicast packets at the > router so that multiple hosts all get copies and then only one > responds for a given flow... or something like that. > > So if we are using something like a VXLAN overlay we need to map the > L2 multicast in the overlay to L3 multicast in the underlay or do > head end replication in the overlay and get a wack load of duplicate > frames on the first link from the router to the core "switch" (really > it's a router). Not good. The solution presumably is to run > potentially thousands of PIM-SM or something messages to > generate/maintain the required multicast state in the IP underlay. > > Anyway the behavior of the upper layer with respect to broadcast / > multicast affects the choice of head end, (*,G) or (S,G) replication > in the underlay, which affects the opex and capex of the entire > solution, so I think its an important question to get a handle on > earlier rather than later. > > Peter > > -----Original Message----- From: armd-bounces@ietf.org > [mailto:armd-bounces@ietf.org] On Behalf Of Anoop Ghanwani Sent: > Tuesday, February 14, 2012 4:53 PM To: Mike McBride Cc: Thomas > Narten; armd@ietf.org; Igor Gashinsky Subject: Re: [armd] address > resolution requirement from hosts to overlay edge nodes. Any > opinion? > > On Tue, Feb 14, 2012 at 1:16 PM, Mike McBride > wrote: > >> There are interesting challenges with unicast and multicast in big >> flat DC networks. That's why we are here. One of the constant >> themes we hear with multicast is the FUD that it doesn't scale. >> Especially from vendors who are pushing another solution or who >> have a crappy multicast implementation. Meanwhile multicast is >> being deployed in large scale networks without scaling issue. > > Like I said earlier...we have large-scale multicast today, but those > are purpose-built networks, and at least today, they are no where > near the kinds of scale where people want to go with overlay > technology (in terms of size of network and traffic patterns). > >> Large L2 overlay networks? I don't know. Would be good to find out >> from the community about performance and scalability of multicast >> in the DC. > > Perhaps it's best we wait until the problems are little more > obvious. Or may be they're just non-existent and I get proven wrong. > Either way, it looks like the correct thing for now is to do > nothing. > > Anoop _______________________________________________ armd mailing > list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > From narten@us.ibm.com Wed Feb 15 06:09:17 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 756B921F8671 for ; Wed, 15 Feb 2012 06:09:17 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -109.208 X-Spam-Level: X-Spam-Status: No, score=-109.208 tagged_above=-999 required=5 tests=[AWL=1.391, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HtbhGO6-hP7K for ; Wed, 15 Feb 2012 06:09:16 -0800 (PST) Received: from e36.co.us.ibm.com (e36.co.us.ibm.com [32.97.110.154]) by ietfa.amsl.com (Postfix) with ESMTP id A3FB621F852E for ; Wed, 15 Feb 2012 06:09:15 -0800 (PST) Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 15 Feb 2012 07:09:13 -0700 Received: from d01dlp03.pok.ibm.com (9.56.224.17) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 15 Feb 2012 07:08:16 -0700 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 2E078C9005C for ; Wed, 15 Feb 2012 09:08:15 -0500 (EST) Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q1FE8DSV259592 for ; Wed, 15 Feb 2012 09:08:13 -0500 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q1FE8JSQ021040 for ; Wed, 15 Feb 2012 07:08:19 -0700 Received: from cichlid.raleigh.ibm.com (sig-9-65-205-222.mts.ibm.com [9.65.205.222]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q1FE8Igl020910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 15 Feb 2012 07:08:19 -0700 Received: from cichlid.raleigh.ibm.com (localhost [127.0.0.1]) by cichlid.raleigh.ibm.com (8.14.5/8.12.5) with ESMTP id q1FE7cWW022379; Wed, 15 Feb 2012 09:07:38 -0500 Message-Id: <201202151407.q1FE7cWW022379@cichlid.raleigh.ibm.com> To: Mike McBride In-reply-to: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> Comments: In-reply-to Mike McBride message dated "Tue, 14 Feb 2012 13:16:28 -0800." Date: Wed, 15 Feb 2012 09:07:38 -0500 From: Thomas Narten X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12021514-3352-0000-0000-000002A4E81B Cc: "armd@ietf.org" Subject: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 14:09:17 -0000 Mike McBride writes: > Large L2 overlay networks? I don't know. Would be good to find out > from the community about performance and scalability of multicast in > the DC. My impression is that many data centers do not enable IP multicast on their routers. That means you can use link-local multicast (which works fine within one IP subnet and doesn't really have scaling issues). But if you want multicast that goes beyond one link (and IP subnet), which is presumably necessary for an overlay like VXLAN/NVGRE, that is where you have problems. The question is not even whether L3 multicast scales. It's whether the DC operater is willing to enable such multicast. Thomas From aldrin.isaac@gmail.com Wed Feb 15 06:39:46 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9859521E8018 for ; Wed, 15 Feb 2012 06:39:46 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.295 X-Spam-Level: X-Spam-Status: No, score=-3.295 tagged_above=-999 required=5 tests=[AWL=0.304, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ewSwkjTzbw2Y for ; Wed, 15 Feb 2012 06:39:45 -0800 (PST) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id 924C421F8778 for ; Wed, 15 Feb 2012 06:39:45 -0800 (PST) Received: by qafi29 with SMTP id i29so3068846qaf.10 for ; Wed, 15 Feb 2012 06:39:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=1W0jOxR6d1Q8P/lLL4ouBhhlr/jzKQ+C4akfIread/I=; b=JrjiN3UGL+5912GEiNWBM8k0uVuG+HHzlNLngzL+7/nz1WMFnDFVMsProOYI+xIC+T FA2OEe2gtxEwbzFoxAVqHQj3XlA8gVCoeBrlrPgFTW1QfZpect5PbTVn83Y5T9r7kNTg 52wc/wrufHvHY37GcF6UeonwZyflvF798QbRQ= Received: by 10.229.107.33 with SMTP id z33mr15551466qco.7.1329316785069; Wed, 15 Feb 2012 06:39:45 -0800 (PST) Received: from mymac.home (ool-44c1c730.dyn.optonline.net. [68.193.199.48]) by mx.google.com with ESMTPS id hi8sm10986763qab.3.2012.02.15.06.39.41 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 15 Feb 2012 06:39:42 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Aldrin Isaac In-Reply-To: <201202151407.q1FE7cWW022379@cichlid.raleigh.ibm.com> Date: Wed, 15 Feb 2012 09:39:40 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> <201202151407.q1FE7cWW022379@cichlid.raleigh.ibm.com> To: Thomas Narten X-Mailer: Apple Mail (2.1257) Cc: "armd@ietf.org" Subject: Re: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 14:39:46 -0000 Any DC with applications performing heavy pub-sub are more than likely = using multicast. I am aware of many companies for whom multicast is = critical and enabled on their DC routers. On Feb 15, 2012, at 9:07 AM, Thomas Narten wrote: > Mike McBride writes: >=20 >> Large L2 overlay networks? I don't know. Would be good to find out >> from the community about performance and scalability of multicast in >> the DC. >=20 > My impression is that many data centers do not enable IP multicast on > their routers. That means you can use link-local multicast (which > works fine within one IP subnet and doesn't really have scaling > issues). But if you want multicast that goes beyond one link (and IP > subnet), which is presumably necessary for an overlay like > VXLAN/NVGRE, that is where you have problems. >=20 > The question is not even whether L3 multicast scales. It's whether the > DC operater is willing to enable such multicast. >=20 > Thomas >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From Peter.AshwoodSmith@huawei.com Wed Feb 15 06:46:26 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4C7B421E8021 for ; Wed, 15 Feb 2012 06:46:26 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.562 X-Spam-Level: X-Spam-Status: No, score=-2.562 tagged_above=-999 required=5 tests=[AWL=0.038, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oVnNK2-gfQGt for ; Wed, 15 Feb 2012 06:46:22 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id D16FC21E801C for ; Wed, 15 Feb 2012 06:46:15 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml202-edg.china.huawei.com) ([172.18.9.243]) by dfwrg02-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADH82379; Wed, 15 Feb 2012 09:46:14 -0500 (EST) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 15 Feb 2012 06:42:32 -0800 Received: from DFWEML503-MBX.china.huawei.com ([10.124.31.29]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0323.003; Wed, 15 Feb 2012 06:42:28 -0800 From: AshwoodsmithPeter To: Joel jaeggli Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AQHM60o2jmo4HB2iRmy2e6GgqFO6cpY9RsQA//78TQCAAArTAIAABRgAgAAMJwCAARaGAP//enGw///6uIAAIwixUA== Date: Wed, 15 Feb 2012 14:42:27 +0000 Message-ID: <7AE6A4247B044C4ABE0A5B6BF427F8E291E46D@dfweml503-mbx> In-Reply-To: <4F3B4459.2080907@bogus.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.193.60.101] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , "armd@ietf.org" , Igor Gashinsky Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 14:46:26 -0000 Thanks Joel, Something else to consider here is just what the multicast is being used fo= r. Obviously one use is true multicast data, and if that's the case the underl= ay needs to be efficient (S,G) (*,G) or such when any serious volume of tra= ffic is involved. However another use is multicast as a way to find/signal other participants= in your application. I do not know how the two applications 'ganglia' and = 'vertica' behave below, but if they and others fit in this category perhaps= a better solution is to give them an API a bit closer to what they are try= ing to do eg: register to be member in a group, find members in some group,= signal members in some group, rather than a lower level 'multicast this me= ssage to this group'... because there are lighter ways to do these membersh= ip/signalling functions than brute force multicast. Peter -----Original Message----- From: Joel jaeggli [mailto:joelja@bogus.com]=20 Sent: Wednesday, February 15, 2012 12:36 AM To: AshwoodsmithPeter Cc: Anoop Ghanwani; Mike McBride; Thomas Narten; Igor Gashinsky; armd@ietf.= org Subject: Re: [armd] address resolution requirement from hosts to overlay ed= ge nodes. Any opinion? On 2/14/12 14:15 , AshwoodsmithPeter wrote: >=20 > Irrespective of what is right or wrong etc. technically, I think it > would be very useful to have a taxonomy of the different applications > that are using multicast / broadcast for things other than ARP. ganglia is a common datacenter multicast user vertica clusters can use broadcast for communication and span hundres of nodes, however unwise that approach may be. > The complexity of the underlay depends a lot on what happens with the > use of multicast by the overlay. >=20 > For example. Assume Microsoft load balancing (for better or worse). > If I remember correctly, and sombody correct me if I'm wrong, it > generates multicast frames from otherwise unicast packets at the > router so that multiple hosts all get copies and then only one > responds for a given flow... or something like that. >=20 > So if we are using something like a VXLAN overlay we need to map the > L2 multicast in the overlay to L3 multicast in the underlay or do > head end replication in the overlay and get a wack load of duplicate > frames on the first link from the router to the core "switch" (really > it's a router). Not good. The solution presumably is to run > potentially thousands of PIM-SM or something messages to > generate/maintain the required multicast state in the IP underlay. >=20 > Anyway the behavior of the upper layer with respect to broadcast / > multicast affects the choice of head end, (*,G) or (S,G) replication > in the underlay, which affects the opex and capex of the entire > solution, so I think its an important question to get a handle on > earlier rather than later. >=20 > Peter >=20 > -----Original Message----- From: armd-bounces@ietf.org > [mailto:armd-bounces@ietf.org] On Behalf Of Anoop Ghanwani Sent: > Tuesday, February 14, 2012 4:53 PM To: Mike McBride Cc: Thomas > Narten; armd@ietf.org; Igor Gashinsky Subject: Re: [armd] address > resolution requirement from hosts to overlay edge nodes. Any > opinion? >=20 > On Tue, Feb 14, 2012 at 1:16 PM, Mike McBride > wrote: >=20 >> There are interesting challenges with unicast and multicast in big=20 >> flat DC networks. That's why we are here. One of the constant >> themes we hear with multicast is the FUD that it doesn't scale. >> Especially from vendors who are pushing another solution or who >> have a crappy multicast implementation. Meanwhile multicast is >> being deployed in large scale networks without scaling issue. >=20 > Like I said earlier...we have large-scale multicast today, but those > are purpose-built networks, and at least today, they are no where > near the kinds of scale where people want to go with overlay=20 > technology (in terms of size of network and traffic patterns). >=20 >> Large L2 overlay networks? I don't know. Would be good to find out >> from the community about performance and scalability of multicast >> in the DC. >=20 > Perhaps it's best we wait until the problems are little more > obvious. Or may be they're just non-existent and I get proven wrong. > Either way, it looks like the correct thing for now is to do > nothing. >=20 > Anoop _______________________________________________ armd mailing > list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd=20 > _______________________________________________ armd mailing list=20 > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd >=20 From linda.dunbar@huawei.com Wed Feb 15 06:56:10 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 947C421E804C for ; Wed, 15 Feb 2012 06:56:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.52 X-Spam-Level: X-Spam-Status: No, score=-2.52 tagged_above=-999 required=5 tests=[AWL=0.079, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A8GB0bWtly4X for ; Wed, 15 Feb 2012 06:56:00 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id EA64D21E8018 for ; Wed, 15 Feb 2012 06:55:59 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml201-edg.china.huawei.com) ([172.18.9.243]) by dfwrg01-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADP85515; Wed, 15 Feb 2012 09:55:59 -0500 (EST) Received: from DFWEML404-HUB.china.huawei.com (10.193.5.203) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 15 Feb 2012 06:53:25 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml404-hub.china.huawei.com ([10.193.5.203]) with mapi id 14.01.0323.003; Wed, 15 Feb 2012 06:53:12 -0800 From: Linda Dunbar To: Aldrin Isaac , Thomas Narten Thread-Topic: [armd] Multicast in the data center [was Re: address resolution Thread-Index: AQHM6/GRUZL5R//tE0esOFvTEvRsbw== Date: Wed, 15 Feb 2012 14:53:11 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E1DF6C@dfweml505-mbx> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> <201202151407.q1FE7cWW022379@cichlid.raleigh.ibm.com> <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> In-Reply-To: <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.47.157.151] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "armd@ietf.org" Subject: Re: [armd] Multicast in the data center [was Re: address resolution X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 14:56:10 -0000 Aldrin,=20 Is "pub-sub" same as "Publish-subscribe" related applications?=20 Are all the "subscribers" in the same subnet as the "publisher"?=20 If yes, it is almost like many multicast groups being created (in real time= ?), with subscribers subscribing to a subset of the multicast groups, isn't= it?=20 If they are in different subnets, is it like L3 multicast? Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Aldrin Isaac > Sent: Wednesday, February 15, 2012 8:40 AM > To: Thomas Narten > Cc: armd@ietf.org > Subject: Re: [armd] Multicast in the data center [was Re: address > resolution requirement from hosts to overlay edge nodes. Any opinion?] >=20 > Any DC with applications performing heavy pub-sub are more than likely > using multicast. I am aware of many companies for whom multicast is > critical and enabled on their DC routers. >=20 > On Feb 15, 2012, at 9:07 AM, Thomas Narten wrote: >=20 > > Mike McBride writes: > > > >> Large L2 overlay networks? I don't know. Would be good to find out > >> from the community about performance and scalability of multicast in > >> the DC. > > > > My impression is that many data centers do not enable IP multicast on > > their routers. That means you can use link-local multicast (which > > works fine within one IP subnet and doesn't really have scaling > > issues). But if you want multicast that goes beyond one link (and IP > > subnet), which is presumably necessary for an overlay like > > VXLAN/NVGRE, that is where you have problems. > > > > The question is not even whether L3 multicast scales. It's whether > the > > DC operater is willing to enable such multicast. > > > > Thomas > > > > _______________________________________________ > > armd mailing list > > armd@ietf.org > > https://www.ietf.org/mailman/listinfo/armd >=20 > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd From Peter.AshwoodSmith@huawei.com Wed Feb 15 07:38:28 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4287A21F85C5 for ; Wed, 15 Feb 2012 07:38:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.566 X-Spam-Level: X-Spam-Status: No, score=-2.566 tagged_above=-999 required=5 tests=[AWL=0.033, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id szh6Fhdm8yVv for ; Wed, 15 Feb 2012 07:38:27 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 8947C21F85C3 for ; Wed, 15 Feb 2012 07:38:27 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml202-edg.china.huawei.com) ([172.18.9.243]) by dfwrg01-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADP88062; Wed, 15 Feb 2012 10:38:27 -0500 (EST) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 15 Feb 2012 07:35:16 -0800 Received: from DFWEML503-MBX.china.huawei.com ([10.124.31.29]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0323.003; Wed, 15 Feb 2012 07:35:11 -0800 From: AshwoodsmithPeter To: Aldrin Isaac , Thomas Narten Thread-Topic: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] Thread-Index: AQHM6+tz2aQ3CA7uUUiJzv2UoHOYiJY+jZEA//+DSwA= Date: Wed, 15 Feb 2012 15:35:10 +0000 Message-ID: <7AE6A4247B044C4ABE0A5B6BF427F8E291E4DB@dfweml503-mbx> In-Reply-To: <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.193.60.101] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "armd@ietf.org" Subject: Re: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 15:38:28 -0000 > My impression is that many data centers do not enable IP multicast on > their routers. That means you can use link-local multicast (which > works fine within one IP subnet and doesn't really have scaling > issues). But if you want multicast that goes beyond one link (and IP > subnet), which is presumably necessary for an overlay like > VXLAN/NVGRE, that is where you have problems. Agreed. If you extrapolate a bit and imagine VXLAN with 1000's of logical g= roups.. then that maps either to head end replication in the hypervisor, or= to IGMP from the hypervisor and PIM between the TOR and CORE 'switches' an= d the gateway router. > The question is not even whether L3 multicast scales. It's whether the > DC operater is willing to enable such multicast. No choice if you want to run a Layer 2 Emulation over IP and you care about= multicast banwdidh effeciency. This highlights one of the major differences between the L2 over IP and L2 = over L2 solutions, i.e. signalling v.s. computation for the multicast state= . This is why I think an understanding of the true uses of that multicast is = important because if 90% of the applications are only finding each other an= d signalling each other using multicast there are better ways but if they a= re truly multicasting data in large volumes .. well we need extremely good = underlay multicast scale. Peter From david.i.allan@ericsson.com Wed Feb 15 07:45:17 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 781BD21F8669 for ; Wed, 15 Feb 2012 07:45:17 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.355 X-Spam-Level: X-Spam-Status: No, score=-6.355 tagged_above=-999 required=5 tests=[AWL=0.244, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YIEGS3ABaLP5 for ; Wed, 15 Feb 2012 07:45:13 -0800 (PST) Received: from imr4.ericy.com (imr4.ericy.com [198.24.6.9]) by ietfa.amsl.com (Postfix) with ESMTP id 53CB421F8623 for ; Wed, 15 Feb 2012 07:45:13 -0800 (PST) Received: from eusaamw0712.eamcs.ericsson.se ([147.117.20.181]) by imr4.ericy.com (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id q1FFj3GX023890; Wed, 15 Feb 2012 09:45:08 -0600 Received: from EUSAACMS0703.eamcs.ericsson.se ([169.254.1.142]) by eusaamw0712.eamcs.ericsson.se ([147.117.20.181]) with mapi; Wed, 15 Feb 2012 10:45:06 -0500 From: David Allan I To: AshwoodsmithPeter , Aldrin Isaac , Thomas Narten Date: Wed, 15 Feb 2012 10:45:04 -0500 Thread-Topic: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] Thread-Index: AQHM6+tz2aQ3CA7uUUiJzv2UoHOYiJY+jZEA//+DSwCAAAdAcA== Message-ID: <60C093A41B5E45409A19D42CF7786DFD522A9BE827@EUSAACMS0703.eamcs.ericsson.se> References: <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> <7AE6A4247B044C4ABE0A5B6BF427F8E291E4DB@dfweml503-mbx> In-Reply-To: <7AE6A4247B044C4ABE0A5B6BF427F8E291E4DB@dfweml503-mbx> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "armd@ietf.org" Subject: Re: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?] X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 15:45:17 -0000 We do need to be clear between protocols that leverage layer 2 broadcast an= d multicast vs. those that genuinely need layer 3 and the multicast group s= pans multiple subnets. I suspect we are really discussing the former, the latter looks to me like = complexity well beyond what a pool of VMs supporting an application need to= autodiscover and self organize. That an overlay may use IP multicast at the server layer to virtualize laye= r 2 broadcast/multicast is a separate issue. Cheers Dave -----Original Message----- From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Ash= woodsmithPeter Sent: Wednesday, February 15, 2012 7:35 AM To: Aldrin Isaac; Thomas Narten Cc: armd@ietf.org Subject: Re: [armd] Multicast in the data center [was Re: address resolutio= n requirement from hosts to overlay edge nodes. Any opinion?] > My impression is that many data centers do not enable IP multicast on=20 > their routers. That means you can use link-local multicast (which=20 > works fine within one IP subnet and doesn't really have scaling=20 > issues). But if you want multicast that goes beyond one link (and IP=20 > subnet), which is presumably necessary for an overlay like=20 > VXLAN/NVGRE, that is where you have problems. Agreed. If you extrapolate a bit and imagine VXLAN with 1000's of logical g= roups.. then that maps either to head end replication in the hypervisor, or= to IGMP from the hypervisor and PIM between the TOR and CORE 'switches' an= d the gateway router. > The question is not even whether L3 multicast scales. It's whether the=20 > DC operater is willing to enable such multicast. No choice if you want to run a Layer 2 Emulation over IP and you care about= multicast banwdidh effeciency. This highlights one of the major differences between the L2 over IP and L2 = over L2 solutions, i.e. signalling v.s. computation for the multicast state= . This is why I think an understanding of the true uses of that multicast is = important because if 90% of the applications are only finding each other an= d signalling each other using multicast there are better ways but if they a= re truly multicasting data in large volumes .. well we need extremely good = underlay multicast scale. Peter _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd From aldrin.isaac@gmail.com Wed Feb 15 13:03:18 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 96C5021F8570 for ; Wed, 15 Feb 2012 13:03:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.317 X-Spam-Level: X-Spam-Status: No, score=-3.317 tagged_above=-999 required=5 tests=[AWL=0.282, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yhjeYokzpZwg for ; Wed, 15 Feb 2012 13:03:13 -0800 (PST) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by ietfa.amsl.com (Postfix) with ESMTP id DB82121F8572 for ; Wed, 15 Feb 2012 13:03:12 -0800 (PST) Received: by qafi29 with SMTP id i29so3534311qaf.10 for ; Wed, 15 Feb 2012 13:03:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=FreiXXu74sXWI7oLjEkMnrdCj3mmH4vlNZTeKP/qjVk=; b=fqHFQzhZoEZT1kUwN7d3dYboDMWXz0VmDZaHg/DLBae/uPRJphrqijjo+3p4kUBYSZ t4zGLblVDyzHG7YKSUSCRPvESS7hNZgMTyizHezz/FhFgllDSYKI3WAZ4gxnTgSsQOz4 Jhyro9EE+QeqJq0jocg0GZ7IQeC9j+SNSQCO4= Received: by 10.229.136.19 with SMTP id p19mr16322302qct.133.1329339792425; Wed, 15 Feb 2012 13:03:12 -0800 (PST) Received: from mymac.home (ool-44c1c730.dyn.optonline.net. [68.193.199.48]) by mx.google.com with ESMTPS id bd19sm12967243qab.17.2012.02.15.13.03.10 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 15 Feb 2012 13:03:11 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: text/plain; charset=us-ascii From: Aldrin Isaac In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F632E1DF6C@dfweml505-mbx> Date: Wed, 15 Feb 2012 16:03:09 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <3307C61A-5DA1-4A43-B807-095C3BBDB700@gmail.com> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> <201202151407.q1FE7cWW022379@cichlid.raleigh.ibm.com> <4D9F477D-70C5-44B7-8146-992579B0D543@gmail.com> <4A95BA014132FF49AE685FAB4B9F17F632E1DF6C@dfweml505-mbx> To: Linda Dunbar X-Mailer: Apple Mail (2.1257) Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] Multicast in the data center [was Re: address resolution X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 21:03:18 -0000 Hi Linda, Yes, pub-sub is the same as publish-subscribe. The publishers and = subscribers are both in the same subnet and across subnets, even on = opposite sides of of a WAN. In fact many of the applications both = publish and subscribe simultaneously. Some applications use it for = heart-beat and as bi-directional application control channels, while = others use it to send fat (50Mbps to 1Gbps+) one-way data flows to many = downstream recipients, and everything in between. Multicast is = generally organized into small, medium and large flow groups to manage = state. Small flows are generally ad-hoc (come and go at will) and = forced to share a limited set of small flow groups. Medium flow groups = are shared by communities of interest. Large flows get their own = multicast groups to avoid excessive bandwidth use. A PGM layer is used = for reliable multicast. Others will have different use-cases and = operating models. Bottom line -- multicast IS being used extensively in = the DC, but not by run-of-the-mill applications. -- aldrin On Feb 15, 2012, at 9:53 AM, Linda Dunbar wrote: > Aldrin,=20 >=20 > Is "pub-sub" same as "Publish-subscribe" related applications?=20 > Are all the "subscribers" in the same subnet as the "publisher"?=20 > If yes, it is almost like many multicast groups being created (in real = time?), with subscribers subscribing to a subset of the multicast = groups, isn't it?=20 > If they are in different subnets, is it like L3 multicast? >=20 > Linda >=20 >> -----Original Message----- >> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf = Of >> Aldrin Isaac >> Sent: Wednesday, February 15, 2012 8:40 AM >> To: Thomas Narten >> Cc: armd@ietf.org >> Subject: Re: [armd] Multicast in the data center [was Re: address >> resolution requirement from hosts to overlay edge nodes. Any = opinion?] >>=20 >> Any DC with applications performing heavy pub-sub are more than = likely >> using multicast. I am aware of many companies for whom multicast is >> critical and enabled on their DC routers. >>=20 >> On Feb 15, 2012, at 9:07 AM, Thomas Narten wrote: >>=20 >>> Mike McBride writes: >>>=20 >>>> Large L2 overlay networks? I don't know. Would be good to find out >>>> from the community about performance and scalability of multicast = in >>>> the DC. >>>=20 >>> My impression is that many data centers do not enable IP multicast = on >>> their routers. That means you can use link-local multicast (which >>> works fine within one IP subnet and doesn't really have scaling >>> issues). But if you want multicast that goes beyond one link (and IP >>> subnet), which is presumably necessary for an overlay like >>> VXLAN/NVGRE, that is where you have problems. >>>=20 >>> The question is not even whether L3 multicast scales. It's whether >> the >>> DC operater is willing to enable such multicast. >>>=20 >>> Thomas >>>=20 >>> _______________________________________________ >>> armd mailing list >>> armd@ietf.org >>> https://www.ietf.org/mailman/listinfo/armd >>=20 >> _______________________________________________ >> armd mailing list >> armd@ietf.org >> https://www.ietf.org/mailman/listinfo/armd From igor@yahoo-inc.com Wed Feb 15 14:08:54 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4C6C321E80A2 for ; Wed, 15 Feb 2012 14:08:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.599 X-Spam-Level: X-Spam-Status: No, score=-18.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_DEF_WHITELIST=-15] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kVc2x8tWvLG7 for ; Wed, 15 Feb 2012 14:08:50 -0800 (PST) Received: from mrout2.yahoo.com (mrout2.yahoo.com [216.145.54.172]) by ietfa.amsl.com (Postfix) with ESMTP id 1348B21E8032 for ; Wed, 15 Feb 2012 14:08:49 -0800 (PST) Received: from netops1.corp.bf1.yahoo.com (netops1.corp.bf1.yahoo.com [98.139.254.110]) by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q1FM8VLK011533; Wed, 15 Feb 2012 14:08:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1329343711; bh=DpmZOM9EWe4B88hr+7sTdkmeY4NUjgoPAbKj7cwoG5Q=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type:Content-ID; b=fPBh346CeGQkXvYGqVDNSqa99iuacoOKd8PUlbZvQti+uR7z826vKw9EPFUxqjc13 puddmAMMDUrEZcoW7E7QEuAXU8GG5Pq7yJpFTrB/4hgMUVQAwWk3BHtfv6Y8BZU6o6 6MHb0y8rdM+lE4KgAJ+zxytiNFKExuE21h7mKu/c= Date: Wed, 15 Feb 2012 14:08:30 -0800 (PST) From: Igor Gashinsky X-X-Sender: igor@netops1.corp.bf1.yahoo.com To: Mike McBride In-Reply-To: Message-ID: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> User-Agent: Alpine 2.00 (LRH 1167 2008-08-23) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-1956483330-726143141-1329260620=:7083" Content-ID: X-Mailman-Approved-At: Wed, 15 Feb 2012 14:11:20 -0800 Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 22:08:54 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. ---1956483330-726143141-1329260620=:7083 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-15 Content-Transfer-Encoding: 8BIT Content-ID: On Tue, 14 Feb 2012, Mike McBride wrote: :: Anoop, :: :: On Tue, Feb 14, 2012 at 12:32 PM, Anoop Ghanwani wrote: :: > The whole problem of sending L2 multicast over a campus or data center :: > backbone, in any sort of significant way, is a new one enabled for the first :: > time by overlays.  There are interesting challenges when pushing large :: > amounts of multicast traffic through a network, and  have thus far been dealt :: > with using purpose-built networks.  While the overlay proposals have :: > been careful not to impose new protocol requirements, they have not :: > addressed the issues of performance and scalability, nor the large-scale :: > availability of these protocols. :: :: There are interesting challenges with unicast and multicast in big :: flat DC networks. That's why we are here. One of the constant themes :: we hear with multicast is the FUD that it doesn't scale. Especially :: from vendors who are pushing another solution or who have a crappy :: multicast implementation. Meanwhile multicast is being deployed in :: large scale networks without scaling issue. Large L2 overlay networks? :: I don't know. Would be good to find out from the community about :: performance and scalability of multicast in the DC. I've so far stayed pretty quiet on this, but, on this, I have to strongly disagree. It is not FUD that multicast doesn't scale well inside large datacenters -- it is a simple fact, and I speak as an operator of what several of my vendors called the 2nd largest multicast deployment they have ever seen, with many 10's of thousands (S,G) entries. Sorry, but many of us who operate really large scale datacenters with a lot of variability in the communication flows have removed all *application* use of multicast from the network due to scalability and stability reasons (ie by the time an (S,G) entry is programmed into hardware, we might be 3k+ packets deep into the flow, or the flow is over already, or what happens when you need to scale to 20k+ (S,G)'s?). Obviously, network protocols are still able to use multicast, but due to the very low volumes of flows and (S,G)s that is still acceptable. Yes, we could have pushed the multicast scaling capabilities a bit higher by use of BIDIR, and yes, if the network hardware/software was built better it could have scaled a bit further, but, that's not the world we live in, and that would have cost significant troubleshooting capability, and a lot of extra complexity which we chose to instead solve via unicast, more bandwidth, and better application software. So, whatever overlay protocols are proposed, I would suggest that counting on there being multicast underneath to transport you is not a very good idea... Thanks, -igor ---1956483330-726143141-1329260620=:7083-- From dino@cisco.com Wed Feb 15 14:20:25 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0E42921F86D0 for ; Wed, 15 Feb 2012 14:20:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -8.377 X-Spam-Level: X-Spam-Status: No, score=-8.377 tagged_above=-999 required=5 tests=[AWL=2.222, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7MlKy3cJ6Qc5 for ; Wed, 15 Feb 2012 14:20:20 -0800 (PST) Received: from mtv-iport-2.cisco.com (mtv-iport-2.cisco.com [173.36.130.13]) by ietfa.amsl.com (Postfix) with ESMTP id B584421F86CB for ; Wed, 15 Feb 2012 14:20:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=dino@cisco.com; l=1039; q=dns/txt; s=iport; t=1329344420; x=1330554020; h=subject:mime-version:from:in-reply-to:date:cc: content-transfer-encoding:message-id:references:to; bh=vPEPJBwlwQ9A3jFOjydxpfEOdfjnHhMS9kBjXjynvTw=; b=CZ5sz1JPgSRtNZTxryqCL4rOadptorm8dMb9JUOxV+Ho/o07AemrLVlI 7FTvk+qsHijeEZpYkk6UQx5nKfP/Oz+59tf74WCSWtQS8HzMwJ2lYJscU 6YzdMzEYSmj7R2qxYBKj5KbHhF5aokMGaaJcalbfTQodxFyBNBw1KhihO A=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EAIAuPE+rRDoG/2dsb2JhbABCsGWBB4FyAQEBAwESASc/BQsLRlcGNYddmxUBnlaLaQIEAwQDFAoQBgYJDAIKAg0BEAODVQICAQQCBAEFgnZjBIhNjGmTCQ X-IronPort-AV: E=Sophos;i="4.73,425,1325462400"; d="scan'208";a="30644998" Received: from mtv-core-1.cisco.com ([171.68.58.6]) by mtv-iport-2.cisco.com with ESMTP; 15 Feb 2012 22:20:20 +0000 Received: from sjc-vpn6-975.cisco.com (sjc-vpn6-975.cisco.com [10.21.123.207]) by mtv-core-1.cisco.com (8.14.3/8.14.3) with ESMTP id q1FMKKaw015980; Wed, 15 Feb 2012 22:20:20 GMT Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Dino Farinacci In-Reply-To: Date: Wed, 15 Feb 2012 14:20:20 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <5EC573B7-3DF4-42AC-A3C5-BEA3C2AB8A1D@cisco.com> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> To: Igor Gashinsky X-Mailer: Apple Mail (2.1251.1) Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 22:20:25 -0000 > I've so far stayed pretty quiet on this, but, on this, I have to = strongly=20 > disagree. It is not FUD that multicast doesn't scale well inside large=20= There are others who probably wish to not divulge their proprietary = scalign numbers who would disagree with you. > datacenters -- it is a simple fact, and I speak as an operator of what=20= > several of my vendors called the 2nd largest multicast deployment they=20= > have ever seen, with many 10's of thousands (S,G) entries.=20 It depends what you are comparing 10,000 to. Comparing to unicast = numbers would not being comparing apples with apples. Remember the = granularity of a multicast route is much finer than a unciast route = because it wants to conserve bandwidith and build good distribution = trees. It is a simple bandwidth versus state tradeoff and 10,000 is pretty = large. Many think a data center with 10,000 hosts are large too. I know you can = one-mag-up us Igor, but 10,000 is a large number for enterprise sites. Dino From igor@yahoo-inc.com Thu Feb 16 23:16:29 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9469E21F8812 for ; Thu, 16 Feb 2012 23:16:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.599 X-Spam-Level: X-Spam-Status: No, score=-18.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_DEF_WHITELIST=-15] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bs2jmHhISBkC for ; Thu, 16 Feb 2012 23:16:25 -0800 (PST) Received: from mrout2.yahoo.com (mrout2.yahoo.com [216.145.54.172]) by ietfa.amsl.com (Postfix) with ESMTP id 72B3621F8800 for ; Thu, 16 Feb 2012 23:16:25 -0800 (PST) Received: from netops1.corp.bf1.yahoo.com (netops1.corp.bf1.yahoo.com [98.139.254.110]) by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q1H7G5I4081386; Thu, 16 Feb 2012 23:16:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1329462965; bh=dxXfAcfKE4g47xopaUXAfxk4cpufvygmRpC4lyrp1kk=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=ja1A+86u1G/DiQu0GqfdpezOiMAgNdgs7cXucKsthCL1tt5itdKaqz1pJdgYdTNF/ Yi+xKaft6Y2DgoO/gRT0156TN+gGp7K+38z5EBukctaSbMnAkn2IrUcBnHdpgMVoZx I+hi/c9/tZ9J1oC1TFELvag6Zhf1XvmXHjAQ4zPA= Date: Thu, 16 Feb 2012 23:16:05 -0800 (PST) From: Igor Gashinsky X-X-Sender: igor@netops1.corp.bf1.yahoo.com To: Dino Farinacci In-Reply-To: <5EC573B7-3DF4-42AC-A3C5-BEA3C2AB8A1D@cisco.com> Message-ID: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E1A5@dfweml503-mbx> <60C093A41B5E45409A19D42CF7786DFD522A9BE1F1@EUSAACMS0703.eamcs.ericsson.se> <5EC573B7-3DF4-42AC-A3C5-BEA3C2AB8A1D@cisco.com> User-Agent: Alpine 2.00 (LRH 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Mailman-Approved-At: Fri, 17 Feb 2012 07:00:02 -0800 Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2012 07:16:29 -0000 On Wed, 15 Feb 2012, Dino Farinacci wrote: :: > I've so far stayed pretty quiet on this, but, on this, I have to strongly :: > disagree. It is not FUD that multicast doesn't scale well inside large :: :: There are others who probably wish to not divulge their proprietary :: scalign numbers who would disagree with you. Would anybody be willing to speak about orders of magnitude here? We had to do a *lot* of nasty, ugly, hacky-ish things to get mcast to scale to the 10's of Ks (S,G) scale, but, when we looked at what it would take to make the next order-of-magnitude jump to 100k+, that's when the cost/complexity tradeoff simply made it not practical -- I'd love to hear if anybody is actually doing 100k+ (S,G) of random-ish traffic profile. :: > datacenters -- it is a simple fact, and I speak as an operator of what :: > several of my vendors called the 2nd largest multicast deployment they :: > have ever seen, with many 10's of thousands (S,G) entries. :: :: It depends what you are comparing 10,000 to. Comparing to unicast :: numbers would not being comparing apples with apples. Remember the :: granularity of a multicast route is much finer than a unciast route :: because it wants to conserve bandwidith and build good distribution :: trees. :: :: It is a simple bandwidth versus state tradeoff and 10,000 is pretty large. You are absolutely right.. but the tradeoff is mostly dependant on the *degree of replication* for all those mcast routes vs the state that they cost. For my network, the *average* degree of replication is actually quite small (on the order of 2-3 hosts per group, taking up a tiny fraction of the bandwidth), so trading that amount of bandwidth for state was a no brainer. However, if you have very few groups with a very high replication factor (say, 1000 hosts), then you would arrive at a very different conclusion, but, then you likely don't have a multicast scaling problem, since then you likely only have a few hundred (S,G) entries.. So, do people have some other datapoints on degree of replication, amount of bw saved by mcast vs state on large datacenter (ie not-designed-for-video) multicast networks? :: Many think a data center with 10,000 hosts are large too. I know you :: can one-mag-up us Igor, but 10,000 is a large number for enterprise :: sites. I guess that brings us to the crux of the question -- ARMD = Address Resolution for *Massive* numbers of hosts in the Data center (from the charter), so, what is the order of magnitude for massive? :) To me, (and perhaps i'm in a very small minority here?), massive implies cloud scale datacenters, so, 10-20k physical hosts *per cluster*, and somewhere around 400-500k VM's is the absolute *minimum* for what I would concider to be massive (and, really, I think 100k physical, 2M+ logical is what i believe a realistic aiming point these days should be). If that's the case, then we are looking at about 2-order-of-magnitude higher then what most enterprises do.. so, what problem space do we want to solve? Thanks, -igor PS as you can probably tell, Dino and I have had this discussion before, quite a few times :) --------------------+----------------------+------------------ Igor Gashinsky | Network Architecture | Yahoo! Inc. igor@yahoo-inc.com | cell 917.807.2213 | Do You... Yahoo? --------------------+----------------------+------------------ From internet-drafts@ietf.org Tue Feb 21 11:11:06 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 453BA21F88A1; Tue, 21 Feb 2012 11:11:05 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.578 X-Spam-Level: X-Spam-Status: No, score=-102.578 tagged_above=-999 required=5 tests=[AWL=0.021, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jb-S1sNTm5Cp; Tue, 21 Feb 2012 11:11:00 -0800 (PST) Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9954E21F889C; Tue, 21 Feb 2012 11:10:59 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: internet-drafts@ietf.org To: i-d-announce@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 3.64p2 Message-ID: <20120221191058.18671.76493.idtracker@ietfa.amsl.com> Date: Tue, 21 Feb 2012 11:10:58 -0800 Cc: armd@ietf.org Subject: [armd] I-D Action: draft-ietf-armd-problem-statement-01.txt X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Feb 2012 19:11:09 -0000 A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the Address Resolution for Massive number= s of hosts in the Data center Working Group of the IETF. Title : Problem Statement for ARMD Author(s) : Thomas Narten Manish Karir Ian Foo Filename : draft-ietf-armd-problem-statement-01.txt Pages : 16 Date : 2012-02-21 This document examines issues related to the massive scaling of data centers. Our initial scope is relatively narrow. Specifically, we focus on address resolution (ARP and ND) within the data center. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-armd-problem-statement-01.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ This Internet-Draft can be retrieved at: ftp://ftp.ietf.org/internet-drafts/draft-ietf-armd-problem-statement-01.txt From linda.dunbar@huawei.com Wed Feb 22 14:52:49 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2614121F85F6 for ; Wed, 22 Feb 2012 14:52:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.614 X-Spam-Level: X-Spam-Status: No, score=-2.614 tagged_above=-999 required=5 tests=[AWL=-0.015, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2uDcqSe8nBid for ; Wed, 22 Feb 2012 14:52:48 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 7D2B821F85F0 for ; Wed, 22 Feb 2012 14:52:48 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml201-edg.china.huawei.com) ([172.18.9.243]) by dfwrg02-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADM78585; Wed, 22 Feb 2012 17:52:47 -0500 (EST) Received: from DFWEML403-HUB.china.huawei.com (10.193.5.151) by dfweml201-edg.china.huawei.com (172.18.9.107) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 22 Feb 2012 14:50:11 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml403-hub.china.huawei.com ([10.193.5.151]) with mapi id 14.01.0323.003; Wed, 22 Feb 2012 14:50:08 -0800 From: Linda Dunbar To: "Michael K. Smith - Adhost" , AshwoodsmithPeter , Anoop Ghanwani , Mike McBride Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cgAasSGAAJJPaQAAIkw2gAAEYXaAAADqZQAAAFoBAAABELEAAAFaTgAAAKMEAAABhOwAAAFJnQAAAMYJgAAQkC3QAXInkiA= Date: Wed, 22 Feb 2012 22:49:28 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E2223D@dfweml505-mbx> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.192.11.97] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , Igor Gashinsky , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 22:52:49 -0000 Michael,=20 > -----Original Message----- > Microsoft is configurable for IGMP Multicast and Unicast. There are > also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc. >=20 Why do you consider VRRP as LB protocols? Hosts are not even aware of if VR= RP is used or not, correct? Linda=20 >=20 From igor@yahoo-inc.com Wed Feb 22 15:19:26 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E907621F84DC for ; Wed, 22 Feb 2012 15:19:26 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.599 X-Spam-Level: X-Spam-Status: No, score=-18.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_DEF_WHITELIST=-15] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LLulvl3HoyAU for ; Wed, 22 Feb 2012 15:19:26 -0800 (PST) Received: from mrout1.yahoo.com (mrout1.yahoo.com [216.145.54.171]) by ietfa.amsl.com (Postfix) with ESMTP id 46EB521F84DF for ; Wed, 22 Feb 2012 15:19:26 -0800 (PST) Received: from netops1.corp.bf1.yahoo.com (netops1.corp.bf1.yahoo.com [98.139.254.110]) by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q1MN8mJG081111; Wed, 22 Feb 2012 15:08:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1329952133; bh=yzG676lHbDuiDQ70xp0xfGHoDAJmFp7f6pzrO1hywW0=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=qTT3yZPUK7EoThRDQQwdfEt6tcO8KiIiNU+i4404K0enm1IC5IV/CLnMyXX2P8VJu bnYDnfNuTXGhEH2rUqx2zzIlz3p5EcXsbyZjo+qT/SM9ObLq0gKZTuUGNhZtAnholh wcxA94xEiRbzzk4Ngvz/goSfLMGYhwNTekWub3tU= Date: Wed, 22 Feb 2012 15:08:48 -0800 (PST) From: Igor Gashinsky X-X-Sender: igor@netops1.corp.bf1.yahoo.com To: Linda Dunbar In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F632E2223D@dfweml505-mbx> Message-ID: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> <4A95BA014132FF49AE685FAB4B9F17F632E2223D@dfweml505-mbx> User-Agent: Alpine 2.00 (LRH 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:19:27 -0000 On Wed, 22 Feb 2012, Linda Dunbar wrote: :: Michael, :: :: > -----Original Message----- :: > Microsoft is configurable for IGMP Multicast and Unicast. There are :: > also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc. :: > :: :: Why do you consider VRRP as LB protocols? Hosts are not even aware of if VRRP is used or not, correct? I actually can easily see VRRP being termed an LB protocol -- while it's true that VRRP and HSRP are first-hop redundancy protocols, by striping VRRP/HSRP groups along multiple switches, you can manually achieve load-ballancing between multiple gateways, in fact, quite a few people do that very thing (and have done it for over a decade). -igor --------------------+----------------------+------------------ Igor Gashinsky | Network Architecture | Yahoo! Inc. igor@yahoo-inc.com | cell 917.807.2213 | Do You... Yahoo? --------------------+----------------------+------------------ From linda.dunbar@huawei.com Wed Feb 22 15:21:22 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C5A1221E801D for ; Wed, 22 Feb 2012 15:21:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.613 X-Spam-Level: X-Spam-Status: No, score=-2.613 tagged_above=-999 required=5 tests=[AWL=-0.014, BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TVH3BAFqscqA for ; Wed, 22 Feb 2012 15:21:22 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 220B721F84E0 for ; Wed, 22 Feb 2012 15:21:22 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml202-edg.china.huawei.com) ([172.18.9.243]) by dfwrg02-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADM79848; Wed, 22 Feb 2012 18:21:21 -0500 (EST) Received: from DFWEML405-HUB.china.huawei.com (10.193.5.102) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 22 Feb 2012 15:18:19 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml405-hub.china.huawei.com ([10.193.5.102]) with mapi id 14.01.0323.003; Thu, 23 Feb 2012 07:18:08 +0800 From: Linda Dunbar To: Igor Gashinsky Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cgAasSGAAJJPaQAAIkw2gAAEYXaAAADqZQAAAFoBAAABELEAAAFaTgAAAKMEAAABhOwAAAFJnQAAAMYJgAAQkC3QAXInkiAAEXnFAAAQjDpg Date: Wed, 22 Feb 2012 23:17:27 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E22305@dfweml505-mbx> References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> <4A95BA014132FF49AE685FAB4B9F17F632E2223D@dfweml505-mbx> In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.192.11.97] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:21:23 -0000 In VRRP environment unicast address is used by hosts, not the multicast add= ress, correct?=20 So there is no multicast address in the Layer 2 domain as some other LB des= cribed in the email list (e.g. Microsoft LB)?=20 Linda=20 > -----Original Message----- > From: Igor Gashinsky [mailto:igor@yahoo-inc.com] > Sent: Wednesday, February 22, 2012 5:09 PM > To: Linda Dunbar > Cc: Michael K. Smith - Adhost; AshwoodsmithPeter; Anoop Ghanwani; Mike > McBride; Thomas Narten; armd@ietf.org > Subject: RE: [armd] address resolution requirement from hosts to > overlay edge nodes. Any opinion? >=20 > On Wed, 22 Feb 2012, Linda Dunbar wrote: >=20 > :: Michael, > :: > :: > -----Original Message----- > :: > Microsoft is configurable for IGMP Multicast and Unicast. There > are > :: > also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc. > :: > > :: > :: Why do you consider VRRP as LB protocols? Hosts are not even aware > of if VRRP is used or not, correct? >=20 > I actually can easily see VRRP being termed an LB protocol -- while > it's > true that VRRP and HSRP are first-hop redundancy protocols, by striping > VRRP/HSRP groups along multiple switches, you can manually achieve > load-ballancing between multiple gateways, in fact, quite a few people > do > that very thing (and have done it for over a decade). >=20 > -igor >=20 > --------------------+----------------------+------------------ > Igor Gashinsky | Network Architecture | Yahoo! Inc. > igor@yahoo-inc.com | cell 917.807.2213 | Do You... Yahoo? > --------------------+----------------------+------------------ From mksmith@adhost.com Wed Feb 22 15:34:59 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1A5C921F850C for ; Wed, 22 Feb 2012 15:34:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aEVk5E-hY2Wv for ; Wed, 22 Feb 2012 15:34:58 -0800 (PST) Received: from mail-in06.adhost.com (mail-in06.adhost.com [216.211.128.136]) by ietfa.amsl.com (Postfix) with ESMTP id E188521F84F9 for ; Wed, 22 Feb 2012 15:34:53 -0800 (PST) Received: from AD-EXH02.adhost.lan (unknown [10.142.0.21]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail-in06.adhost.com (Postfix) with ESMTPS id D6AAAD5CAB6; Wed, 22 Feb 2012 15:34:52 -0800 (PST) (envelope-from mksmith@adhost.com) Received: from AD-EXH02.adhost.lan ([fe80::1c5b:7ead:8ba3:6108]) by AD-EXH02.adhost.lan ([fe80::1c5b:7ead:8ba3:6108%11]) with mapi id 14.01.0255.000; Wed, 22 Feb 2012 15:34:52 -0800 From: "Michael K. Smith - Adhost" To: Linda Dunbar , Igor Gashinsky Thread-Topic: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? Thread-Index: AczoOuIfjmo4HB2iRmy2e6GgqFO6cgAasSGAAJJPaQAAIkw2gAAEYXaAAADqZQAAAFoBAAABELEAAAFaTgAAAKMEAAABhOwAAAFJnQAAAMYJgAAQkC3QAXInkiAAEXnFAAAQjDpgACB4xeA= Date: Wed, 22 Feb 2012 23:34:51 +0000 Message-ID: References: <7AE6A4247B044C4ABE0A5B6BF427F8E291E2C8@dfweml503-mbx> <4A95BA014132FF49AE685FAB4B9F17F632E2223D@dfweml505-mbx> <4A95BA014132FF49AE685FAB4B9F17F632E22305@dfweml505-mbx> In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F632E22305@dfweml505-mbx> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.142.1.52] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Thomas Narten , "armd@ietf.org" Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:34:59 -0000 > -----Original Message----- > From: Linda Dunbar [mailto:linda.dunbar@huawei.com] > Sent: Wednesday, February 22, 2012 3:17 PM > To: Igor Gashinsky > Cc: Michael K. Smith - Adhost; AshwoodsmithPeter; Anoop Ghanwani; Mike > McBride; Thomas Narten; armd@ietf.org > Subject: RE: [armd] address resolution requirement from hosts to overlay > edge nodes. Any opinion? >=20 > In VRRP environment unicast address is used by hosts, not the multicast > address, correct? > So there is no multicast address in the Layer 2 domain as some other LB > described in the email list (e.g. Microsoft LB)? >=20 > Linda >=20 VRRP and similar use multicast to communicate to one another (heartbeat). = The communication between the host and the default gateway is unicast. Wit= h that said, the multicast heartbeat can be very chatty when you have thous= ands of VRRP pairs with sub-second heartbeat calls back and forth. Mike From linda.dunbar@huawei.com Wed Feb 22 15:47:52 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EEE6E11E8079 for ; Wed, 22 Feb 2012 15:47:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.612 X-Spam-Level: X-Spam-Status: No, score=-2.612 tagged_above=-999 required=5 tests=[AWL=-0.014, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ppy6xalSZGQe for ; Wed, 22 Feb 2012 15:47:52 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 359FA11E8074 for ; Wed, 22 Feb 2012 15:47:52 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml202-edg.china.huawei.com) ([172.18.9.243]) by dfwrg01-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADU84453; Wed, 22 Feb 2012 18:47:51 -0500 (EST) Received: from DFWEML404-HUB.china.huawei.com (10.193.5.203) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 22 Feb 2012 15:44:19 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml404-hub.china.huawei.com ([10.193.5.203]) with mapi id 14.01.0323.003; Wed, 22 Feb 2012 15:43:35 -0800 From: Linda Dunbar To: "armd@ietf.org" Thread-Topic: Are there any issues with Address Resolution for Multicast (besides ARP/ND) in Data Centers? Thread-Index: Aczxu4rV915h6o4sR9CxX78Q9DTggAAABYCA Date: Wed, 22 Feb 2012 23:43:35 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E22386@dfweml505-mbx> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: CWI= Aq87 DD+I DTOx DjEn DtHV Dtsq E0Tj Fv7x GVAg I+CU J5tf J55B KDo3 KEpa KIHM; 1; YQByAG0AZABAAGkAZQB0AGYALgBvAHIAZwA=; Sosha1_v1; 7; {259568BF-D387-4C7D-BA16-1EB7A00B6FCA}; bABpAG4AZABhAC4AZAB1AG4AYgBhAHIAQABoAHUAYQB3AGUAaQAuAGMAbwBtAA==; Wed, 22 Feb 2012 23:44:13 GMT; QQByAGUAIAB0AGgAZQByAGUAIABhAG4AeQAgAGkAcwBzAHUAZQBzACAAdwBpAHQAaAAgAEEAZABkAHIAZQBzAHMAIABSAGUAcwBvAGwAdQB0AGkAbwBuACAAZgBvAHIAIABNAHUAbAB0AGkAYwBhAHMAdAAgACgAYgBlAHMAaQBkAGUAcwAgAEEAUgBQAC8ATgBEACkAIABpAG4AIABEAGEAdABhACAAQwBlAG4AdABlAHIAcwA/AA== x-cr-puzzleid: {259568BF-D387-4C7D-BA16-1EB7A00B6FCA} x-originating-ip: [10.192.11.97] Content-Type: multipart/alternative; boundary="_000_4A95BA014132FF49AE685FAB4B9F17F632E22386dfweml505mbx_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Subject: [armd] Are there any issues with Address Resolution for Multicast (besides ARP/ND) in Data Centers? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Feb 2012 23:47:53 -0000 --_000_4A95BA014132FF49AE685FAB4B9F17F632E22386dfweml505mbx_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Here are some (non-ARP/ND) multicast applications which people stated on th= e armd@ietf.org that are used in data centers: Joel Jaeggli: ganglia is a common datacenter multicast user vertica cluster= s can use broadcast for communication and span hundreds of nodes, however u= nwise that approach may be. Aldrin Isaac : Any DC with applications performing heavy pub-sub are more = than likely using multicast. I am aware of many companies for whom multica= st is critical and enabled on their DC routers. Michael Smith: Microsoft is configurable for IGMP Multicast and Unicast. T= here are also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc. There are also other people stating that multicast has to be disabled in ma= ssive sized data centers simply because the multicast improvement is limite= d, which is not worth the effort in massive sized data centers. In the context of address resolution, does multicast address resolution hav= e different issues than uni-cast address resolution? Any comments? Linda --_000_4A95BA014132FF49AE685FAB4B9F17F632E22386dfweml505mbx_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

Here are some (non-ARP/ND) multicast applications wh= ich people stated on the armd@ietf.org that are used in data ce= nters:

 

Joel Jaeggli: ganglia is a common da=
tacenter multicast user vertica clusters can use broadcast for communicatio=
n and span hundreds of nodes, however unwise that approach may be.
 
Aldrin Isaac :  Any DC with app=
lications performing heavy pub-sub are more than likely using multicast.&nb=
sp; I am aware of many companies for whom multicast is critical and enabled=
 on their DC routers.

 

Michael Smith: Mic= rosoft is configurable for IGMP Multicast and Unicast.  There are also= the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc.

 

There are also other people stating that multicast h= as to be disabled in massive sized data centers simply because the multicas= t improvement is limited, which is not worth the effort in massive sized da= ta centers.

 

In the context of address resolution, does multicast= address resolution have different issues than uni-cast address resolution?

 

Any comments?

 

 

 

 

Linda

 

 

--_000_4A95BA014132FF49AE685FAB4B9F17F632E22386dfweml505mbx_-- From pfrejborg@gmail.com Wed Feb 22 23:51:53 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 37B7D21F84A6 for ; Wed, 22 Feb 2012 23:51:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lJiN3nrtrzCJ for ; Wed, 22 Feb 2012 23:51:52 -0800 (PST) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by ietfa.amsl.com (Postfix) with ESMTP id 6A0F021F84A5 for ; Wed, 22 Feb 2012 23:51:52 -0800 (PST) Received: by wibhm9 with SMTP id hm9so671719wib.31 for ; Wed, 22 Feb 2012 23:51:51 -0800 (PST) Received-SPF: pass (google.com: domain of pfrejborg@gmail.com designates 10.180.74.177 as permitted sender) client-ip=10.180.74.177; Authentication-Results: mr.google.com; spf=pass (google.com: domain of pfrejborg@gmail.com designates 10.180.74.177 as permitted sender) smtp.mail=pfrejborg@gmail.com; dkim=pass header.i=pfrejborg@gmail.com Received: from mr.google.com ([10.180.74.177]) by 10.180.74.177 with SMTP id u17mr160227wiv.13.1329983511697 (num_hops = 1); Wed, 22 Feb 2012 23:51:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Yr571u3bAhQ3LKf1Mloy9VDMuGDSyQHpydHfE1SDtXU=; b=BF1uz4gCMcj5CVEdX9YejzH9MPyPSjNmi5NjCLjtXPmkUNGz3U+wHl0b79Y7fGGDz5 u8lj4+YhJkYmdvLxeJuskImuK3LvdAaD8NQGwa+6LVjOEq4pFvSnv/ZyQG+1OcqtqCDW iIjqWj3/FQP/nV+Ls03JKEsdZdpeUnHKf70+U= MIME-Version: 1.0 Received: by 10.180.74.177 with SMTP id u17mr129202wiv.13.1329983511637; Wed, 22 Feb 2012 23:51:51 -0800 (PST) Received: by 10.227.168.12 with HTTP; Wed, 22 Feb 2012 23:51:51 -0800 (PST) In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F632E22386@dfweml505-mbx> References: <4A95BA014132FF49AE685FAB4B9F17F632E22386@dfweml505-mbx> Date: Thu, 23 Feb 2012 09:51:51 +0200 Message-ID: From: Patrick Frejborg To: Linda Dunbar Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "armd@ietf.org" Subject: Re: [armd] Are there any issues with Address Resolution for Multicast (besides ARP/ND) in Data Centers? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2012 07:51:53 -0000 Linda, if you do a search with the words "cluster, multicast" and "NIC teaming, multicast" you'll find a bunch of solutions that make use of link local multicast. These shouldn't be processed by the router, neither should they traverse over the L3 boundary but will be seen by the router connected to the L2 domains. Patrick On Thu, Feb 23, 2012 at 1:43 AM, Linda Dunbar wro= te: > > > Here are some (non-ARP/ND) multicast applications which people stated on = the > armd@ietf.org that are used in data centers: > > > > Joel Jaeggli: ganglia is a common datacenter multicast user vertica clust= ers > can use broadcast for communication and span hundreds of nodes, however > unwise that approach may be. > > > > Aldrin Isaac :=A0 Any DC with applications performing heavy pub-sub are m= ore > than likely using multicast.=A0 I am aware of many companies for whom > multicast is critical and enabled on their DC routers. > > > > Michael Smith: Microsoft is configurable for IGMP Multicast and Unicast. > There are also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc= . > > > > There are also other people stating that multicast has to be disabled in > massive sized data centers simply because the multicast improvement is > limited, which is not worth the effort in massive sized data centers. > > > > In the context of address resolution, does multicast address resolution h= ave > different issues than uni-cast address resolution? > > > > Any comments? > > > > > > > > > > Linda > > > > > > > _______________________________________________ > armd mailing list > armd@ietf.org > https://www.ietf.org/mailman/listinfo/armd > From gaberger@cisco.com Fri Feb 24 14:22:44 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 742E421F872F for ; Fri, 24 Feb 2012 14:22:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.202 X-Spam-Level: X-Spam-Status: No, score=-9.202 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kKiuwHMJM5QR for ; Fri, 24 Feb 2012 14:22:42 -0800 (PST) Received: from rcdn-iport-5.cisco.com (rcdn-iport-5.cisco.com [173.37.86.76]) by ietfa.amsl.com (Postfix) with ESMTP id 88AFC21F86D9 for ; Fri, 24 Feb 2012 14:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=47047; q=dns/txt; s=iport; t=1330122161; x=1331331761; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version; bh=6MvpmrBYa44uxNJG3J3ZFOJC28oBZu97mRSNchR4WGc=; b=HPL+FnDHs7osQwOCWBUuGdNfiPV+JsKQwfnn4LowsSuwqWmc5XLtceY+ ZEaucBUb0aigpxALjnNlH5AGPpAnINefYV+6bJv4xfxoK3OLo9bGA5rrm x9AEdUZqEvxdZxc6lhyvQEs25r9k+XLi774miOZcChCzCGu/kcHjRclEM 4=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqcAAGYNSE+tJXHA/2dsb2JhbABDglFCniqIWAGIHG+BB4FzAQEBAwEBAQEPAQcTEDEEBwUHBwgRBAEBASABAgQoBh8JCAYBDQUJEgeHXwULoDIBlmyJDoNuDwEBAQIBAgICAQUDAw5BEAEKhGsCAQUJBAwCGjcBBwUGAQQDCAoIAwEDgy0EiE+MbIVdhTuHdYE+ X-IronPort-AV: E=Sophos;i="4.73,478,1325462400"; d="scan'208,217";a="61673360" Received: from rcdn-core2-5.cisco.com ([173.37.113.192]) by rcdn-iport-5.cisco.com with ESMTP; 24 Feb 2012 22:22:40 +0000 Received: from [10.82.243.3] (rtp-vpn2-771.cisco.com [10.82.243.3]) by rcdn-core2-5.cisco.com (8.14.3/8.14.3) with ESMTP id q1OMMcgk004485; Fri, 24 Feb 2012 22:22:39 GMT User-Agent: Microsoft-MacOutlook/14.12.0.110505 Date: Fri, 24 Feb 2012 17:22:37 -0500 From: Gary Berger To: Linda Dunbar , Vishwas Manral , Murari Sridharan Message-ID: Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? In-Reply-To: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3412948960_27688259" Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Feb 2012 22:22:44 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3412948960_27688259 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit > Protocols have been developed for IP mobility between Home Gateway and Remote > Gateway. The question is if we want similar protocols among TORs or among > vSwitches. Handset mobility is random, but VM migration is planned. I wouldn't put such a restriction, vm migration may not be planned.. They both degenerate into a multi-homing problem. > > Linda > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Vishwas Manral > Sent: Thursday, September 22, 2011 9:38 PM > To: Murari Sridharan > Cc: armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > > Murari think IP mobility. :) > > > > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: > > Do you have a scenario in mind? > > > From: Vishwas Manral > Sent: 9/22/2011 4:55 PM > > > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > Hi Murari, > > > > Yes that is what I mean. > > > > Thanks, > > Vishwas > > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: > > You mean not an Ethernet frame but some IP payload? > > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:49 PM > To: Murari Sridharan > Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > > > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > > > Murari, > > > > What I am saying is the inner header should be allowed to be L3. > > > > From the diagram you have that does not seem to be the case. Am I missing it > totally? > > > > Thanks, > > Vishwas > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: > > Vishwas, Thanks for the feedback we will definitely consider adding that. I am > not sure what you mean by doing L3 instead of L2. We allow any arbitrary > virtual topology including L3. > > Thanks > > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:19 PM > > > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > > > Hi Simha, > > > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device for > hashing and using lesser number of bytes). > > > > I would think you should add it to your draft immediately. With tunneling you > consolidate the addresses visible to the core and by providing a hash > mechanism, you are providing some level of randomness. > > > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 > alone. I guess it would be the same comment for the VXLAN proposal too. > > > > Thanks, > > Vishwas > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: > > The draft mentions exactly this as one use of the reserved 8 bits in Section > 4. An NVGRE endpoint could use the 8 bits to further distribute flows > belonging to a particular TNI and the switches use all 32 bits to get entropy. > One step further would be for the switches to get full entropy from the inner > Ethernet frame. I take it that your comment would be to make it explicit in > the draft. Right? > > One > such example could be to use the upper 8 bits of the Key field to > add flow based entropy and tag all the packets from a flow with an entropy > label. > > Simha > > From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] > Sent: Thursday, September 22, 2011 4:04 PM > To: Narasimhan Venkataramaiah > Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world? > > > > Hi Simha, > > > > The main (Standards Track) change in your draft is the addition of TNI. > > > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes in > switches to do hashing to use TNI). > > > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths? > > > > Thanks, > > Vishwas > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from the > enterprise point of view. Are you referring to a scenario where the VMs subnet > is split between the cloud and the enterprise? Otherwise I don't see the > implication on virtualization as its no different than getting the traffic > routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar > [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise > its external facing hosts' IP addresses to external world? > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with external > peers, the host either uses public IP address which is visible to external > peers or uses private IP address which is translated to public address at the > Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data center" > advertise this host address to external peers? Or will all external peers go > through enterprise's gateway to reach this host which is no longer residing in > the enterprise site? > > Thanks, Linda > >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> > Murari Sridharan >> > Sent: Saturday, September 17, 2011 3:02 PM >> > To: david.black@emc.com; armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > FYI, here is a talk that I gave last week in relation to the nvgre >> > draft below. >> > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T >> > >> > Thanks >> > Murari >> > >> > -----Original Message----- >> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of >> > david.black@emc.com >> > Sent: Friday, September 16, 2011 6:14 AM >> > To: armd@ietf.org >> > Subject: Re: [armd] soliciting typical network designs for ARMD >> > >> > And two more drafts on this topic: >> > >> > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt >> > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt >> > >> > The edge switches could be the software switches in hypervisors. >> > >> > Thanks, >> > --David >> > >> > >>> > > -----Original Message----- >>> > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf >>> > > Of Warren Kumari >>> > > Sent: Wednesday, August 31, 2011 3:16 PM >>> > > To: Vishwas Manral >>> > > Cc: armd@ietf.org >>> > > Subject: Re: [armd] soliciting typical network designs for ARMD >>> > > >>> > > >>> > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: >>> > > >>>> > > > Hi Linda/ Anoop, >>>> > > > >>>> > > > Here is the example of the design I was talking about, as defined >> > by google. >>> > > >>> > > Just a clarification -- s/as defined by google/as described by >> > someone >>> > > who happens to work for google/ >>> > > >>> > > W >>> > > >>>> > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani >> > wrote: >>>> > > > >>>>>>>> > > > >>>> >>>> > > > (though I think if there was a standard way to map Multicast MAC to >>>> > > > Multicast IP, they could >>> > > probably use such a standard mechanisms). >>>>>>>> > > > >>>> >>>> > > > >>>> > > > They can do that, but then this imposes requirements on the >>>> > > > equipment to be able to do multicast forwarding, and even if does, >>>> > > > because of pruning requirements the number of groups would be very >>>> > > > large. The average data center switch probably won't handle that >>>> > > > many groups. >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Anoop, >>>> > > > >>>> > > > From what I know they do not use Multicast GRE (I hear the extra 4 >>>> > > > bytes in the GRE header is a >>> > > proprietery extension). >>>> > > > >>>> > > > I think a directory based mechanism is what is used (though I think >>>> > > > if there was a standard way to >>> > > map Multicast MAC to Multicast IP, they could probably use such a >> > standard mechanisms). >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani >> > wrote: >>>> > > > Hi Vishwas, >>>> > > > >>>> > > > How do they get multicast through the network in that case? >>>> > > > Are they planning to use multicast GRE, or just use directory based >>>> > > > lookups and not worry about multicast applications for now? >>>> > > > >>>> > > > Anoop >>>> > > > >>>> > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > >>>> > > > The data packets can be tunnelled at the ToR over say a GRE packet >>>> > > > and the core is a Layer-3 core >>> > > (except for the downstream ports). So we could have encapsulation/ >>> > > decapsulation of L2 over GRE at the ToR. >>>> > > > >>>> > > > The very same thing can be done at the hypervisor layer too, in >>>> > > > which case the entire DC network >>> > > would look like a Layer-3 flat network including the ToR to server >>> > > link and the hypervisor would do the tunneling. >>>> > > > >>>> > > > I am not sure if you got the points above or not. I know cloud OS >>>> > > > companies that provide the service >>> > > and have big announced customers. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar >> > wrote: >>>> > > > Vishwas, >>>> > > > >>>> > > > In my mind the bullet 1) in the list refers to ToR switches >>>> > > > downstream ports (facing servers) >>> > > running Layer 2 and ToR uplinks ports run IP Layer 3. >>>> > > > >>>> > > > Have you seen data center networks with ToR switches downstream >>>> > > > ports (i.e. facing servers) enabling >>> > > IP routing, even though the physical links are Ethernet? >>>> > > > If yes, we should definitely include it in the ARMD draft. >>>> > > > >>>> > > > Thanks, >>>> > > > Linda >>>> > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral >> > wrote: >>>> > > > Hi Linda, >>>> > > > I am unsure what you mean by this, but: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), We can also >>>> > > > have a heirarchical network, with the core totally Layer-3 (and >>>> > > > having seperate >>> > > routing), from the hosts still in a large Layer-3 subnet. Another >>> > > aspect could be to have a totally >>> > > Layer-3 network. >>>> > > > >>>> > > > The difference between them is the link between the servers and the >> > ToR. >>>> > > > >>>> > > > Thanks, >>>> > > > Vishwas >>>> > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar >> > wrote: >>>> > > > During the 81st IETF ARMD WG discussion, it was suggested that it >> > is >>>> > > > necessary to document typical >>> > > data center network designs so that address resolution scaling issues >>> > > can be properly described. Many data center operators have expressed >> > that they can't openly reveal their detailed network designs. >>> > > Therefore, we only want to document anonymous designs without too >> > much >>> > > detail. During the journey of establishing ARMD, we have come across >> > the following typical data center network designs: >>>> > > > * layer 3 all the way to TOR (Top of Rack switches), >>>> > > > * large layer 2 with hundreds (or thousands) of ToRs being >>>> > > > interconnected by Layer 2. This >>> > > design will have thousands of hosts under the L2/L3 boundary router >>> > > (s) >>>> > > > * CLOS design with thousands of switches. This design will have >>>> > > > thousands of hosts under the >>> > > L2/L3 boundary router(s) >>>> > > > We have heard that each of the designs above has its own problems. >>>> > > > ARMD problem statements might >>> > > need to document DC problems under each typical design. >>>> > > > Please send feedback to us (either to the armd email list or to >> > the >>>> > > > ARMD chair Benson & Linda) to >>> > > indicate if we have missed any typical Data Center network designs. >>>> > > > >>>> > > > Your contribution can greatly accelerate the progress of ARMD WG. >>>> > > > >>>> > > > Thank you very much. >>>> > > > >>>> > > > Linda & Benson >>>> > > > > > > > > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --B_3412948960_27688259 Content-type: text/html; charset="US-ASCII" Content-transfer-encoding: quoted-printable


Protocols = have been developed for IP mobility between Home Gateway and Remote Gateway.= The question is if we want similar protocols among TORs or among vSwitches.= Handset mobility is random, but VM migration is planned.  

=

I wouldn't put suc= h a restriction, vm migration may not be planned.. They both degenerate into= a multi-homing problem.

 

Linda

<= p class=3D"MsoNormal"> 

From: armd-bounces@iet= f.org [mailto:armd-bounces@ietf.o= rg] On Behalf Of Vishwas Manral
Sent: Thursday, September 22, = 2011 9:38 PM
To: Murari Sridharan
Cc: armd@ietf.org
Subject: Re: [armd] how does "draft= -sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP = addresses to external world?

 

Murari think IP mobili= ty. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridha= ran <muraris@microsoft.com>= wrote:

Do you have a scenario in mi= nd?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasi= mhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualiz= ation-nvgre-00" advertise its external facing hosts' IP addresses to externa= l world?

Hi Murari,=

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <= ;muraris@microsoft.co= m> wrote:

You mean not an Ethernet frame but some IP payload?

 

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari = Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; arm= d@ietf.org


Su= bject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adv= ertise its external facing hosts' IP addresses to external world?=

 

Murari,

 

What I am saying is= the inner header should be allowed to be L3.

 

From the = diagram you have that does not seem to be the case. Am I missing it totally?=

 

Thanks,

Vishwas

On Thu, Sep = 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Vishwas, Thanks for t= he feedback we will definitely consider adding that. I am not sure what you = mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

<= p class=3D"MsoNormal" style=3D"mso-margin-top-alt:auto;mso-margin-bottom-alt:aut= o"> 

=

Thanks

 

From: Vishwas M= anral [mailto:vishwa= s.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venk= ataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; arm= d@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtu= alization-nvgre-00" advertise its external facing hosts' IP addresses to ext= ernal world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE s= olution (besides ofcourse that TNI needs to be parsed in the intermediate de= vice for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tunneling&= nbsp;you consolidate the addresses visible to the core and by prov= iding a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) = over NVGRE instead of L2 alone. I guess it would be the same comment fo= r the VXLAN proposal too.

 

=

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <= narave@microsoft.com> wrote:

The draft mentions exactly this as= one use of the reserved 8 bits in Section 4. An NVGRE endpoint could use th= e 8 bits to further distribute flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take = it that your comment would be to make it explicit in the draft. Right?<= /o:p>

&nb= sp;

One

   such example could be to use the uppe= r 8 bits of the Key field to

   add flow based entropy an= d tag all the packets from a flow with an entropy label.

 

Simha

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasim= han Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; arm= d@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtu= alization-nvgre-00" advertise its external facing hosts' IP addresses to ext= ernal world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the addition of TNI= .

 

A questio= n I have is a TNI identifies a particular tenant and all flows from/to a ten= ant will be hashed to the same path (even with the changes in switches to do= hashing to use TNI).

 

Why= do you not use the last 8 bits which you have kept as reserved for providin= g the randomization for hashing flows between same to/from on different path= s?

 

Thanks,<= o:p>

Vishwas

On Sun, Se= p 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fro= m the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizatio= n as its no different than getting the traffic routed to the enterprise in t= he first case.

Simha

________________________________________
From: armd-bounces@i= etf.org [armd-bou= nces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; = david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world?

<= div>


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to externa= l peers or uses private IP address which is translated to public address at = the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers = go through enterprise's gateway to reach this host which is no longer residi= ng in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-boun= ces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.black@e= mc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
= > draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-boun= ces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@emc.c= om
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: armd= -bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org<= /a>
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumn= i.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ie= tf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumn= i.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ie= tf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ie= tf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

_______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/m= ailman/listinfo/armd
--B_3412948960_27688259-- From vishwas.ietf@gmail.com Mon Feb 27 16:48:49 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6245A21E8015 for ; Mon, 27 Feb 2012 16:48:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.805 X-Spam-Level: X-Spam-Status: No, score=-3.805 tagged_above=-999 required=5 tests=[AWL=-0.207, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M8hQq4kwGdqx for ; Mon, 27 Feb 2012 16:48:47 -0800 (PST) Received: from mail-tul01m020-f172.google.com (mail-tul01m020-f172.google.com [209.85.214.172]) by ietfa.amsl.com (Postfix) with ESMTP id D6B5021E800C for ; Mon, 27 Feb 2012 16:48:46 -0800 (PST) Received: by obbeh20 with SMTP id eh20so2386418obb.31 for ; Mon, 27 Feb 2012 16:48:46 -0800 (PST) Received-SPF: pass (google.com: domain of vishwas.ietf@gmail.com designates 10.60.20.9 as permitted sender) client-ip=10.60.20.9; Authentication-Results: mr.google.com; spf=pass (google.com: domain of vishwas.ietf@gmail.com designates 10.60.20.9 as permitted sender) smtp.mail=vishwas.ietf@gmail.com; dkim=pass header.i=vishwas.ietf@gmail.com Received: from mr.google.com ([10.60.20.9]) by 10.60.20.9 with SMTP id j9mr7054367oee.6.1330390126588 (num_hops = 1); Mon, 27 Feb 2012 16:48:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7scOyp/vpTNDWwjxOTHG7g6tfVDse1nmjSX3mRWtcS8=; b=YjUl063zIoNB7MeZZ9MQZWm7MWyPtr1V7nR7rwP40JmMARMtg/Z3EtB0b7KHFulH4w uofyU6lcC2eIrDoHxf3GCfDq1h674DojkukZE4idz+GFZouagNjYjWVW+u1u/xzCvhF+ aOKmLSdqFQdiiTzutCGpZsQy0Ea9a1DoCgjHc= MIME-Version: 1.0 Received: by 10.60.20.9 with SMTP id j9mr6147875oee.6.1330390126480; Mon, 27 Feb 2012 16:48:46 -0800 (PST) Received: by 10.182.165.1 with HTTP; Mon, 27 Feb 2012 16:48:46 -0800 (PST) In-Reply-To: References: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> Date: Mon, 27 Feb 2012 16:48:46 -0800 Message-ID: From: Vishwas Manral To: Gary Berger Content-Type: multipart/alternative; boundary=e89a8fb20652e6cd2b04b9fb96da Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2012 00:48:49 -0000 --e89a8fb20652e6cd2b04b9fb96da Content-Type: text/plain; charset=ISO-8859-1 Hi Gary, I guess what Linda was trying to say is that the station moves autonomously, however there is some entity moving the VM (which can in turn could move the state on the switch). An example of this could be VEPA. Thanks, Vishwas On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger wrote: > > > Protocols have been developed for IP mobility between Home Gateway and > Remote Gateway. The question is if we want similar protocols among TORs or > among vSwitches. Handset mobility is random, but VM migration is planned. > > > I wouldn't put such a restriction, vm migration may not be planned.. They > both degenerate into a multi-homing problem. > > **** > > ** ** > > Linda**** > > ** ** > > *From:* armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] > *On Behalf Of *Vishwas Manral > *Sent:* Thursday, September 22, 2011 9:38 PM > *To:* Murari Sridharan > *Cc:* armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > ** ** > > Murari think IP mobility. :)**** > > **** > > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote:**** > > Do you have a scenario in mind?**** > ------------------------------ > > *From: *Vishwas Manral > *Sent: *9/22/2011 4:55 PM**** > > > *To: *Murari Sridharan > *Cc: *Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject: *Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > Hi Murari,**** > > **** > > Yes that is what I mean.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote:**** > > You mean not an Ethernet frame but some IP payload?**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:49 PM > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org**** > > > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari,**** > > **** > > What I am saying is the inner header should be allowed to be L3. **** > > **** > > From the diagram you have that does not seem to be the case. Am I missing > it totally?**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote:**** > > Vishwas, Thanks for the feedback we will definitely consider adding that. > I am not sure what you mean by doing L3 instead of L2. We allow any > arbitrary virtual topology including L3. **** > > **** > > Thanks**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM **** > > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device > for hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by > providing a hash mechanism, you are providing some level of randomness.*** > * > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead > of L2 alone. I guess it would be the same comment for the VXLAN proposal > too.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to get > entropy. One step further would be for the switches to get full entropy > from the inner Ethernet frame. I take it that your comment would be to make > it explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.*** > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes in > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the VMs > subnet is split between the cloud and the enterprise? Otherwise I don't see > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible to > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all external > peers go through enterprise's gateway to reach this host which is no longer > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > **** > > **** > > ** ** > > ** ** > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd > > --e89a8fb20652e6cd2b04b9fb96da Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Gary,

I guess what Linda was trying to say is that the station mo= ves autonomously, however there is some entity moving the VM (which can in = turn could move the state on the switch).

An example of this could = be VEPA.

Thanks,
Vishwas

On Fri, Feb 24, 20= 12 at 2:22 PM, Gary Berger <gaberger@cisco.com> wrote:

Protocols have been developed for IP mobility between Home= Gateway and Remote Gateway. The question is if we want similar protocols a= mong TORs or among vSwitches. Handset mobility is random, but VM migration = is planned. =A0


I wouldn= 9;t put such a restriction, vm migration may not be planned.. They both deg= enerate into a multi-homing problem.

=A0

Linda=

=A0

From: a= rmd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manral
Sent: Thursday, September 22, = 2011 9:38 PM
To: Murari Sridharan
Cc: armd@ietf.org
Subject: Re: = [armd] how does "draft-sridharan-virtualization-nvgre-00" adverti= se its external facing hosts' IP addresses to external world?=

=A0

Murari think IP mobility. :)

=A0

On Th= u, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Do you have a scenario in mind?

From: Vis= hwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan<= /span>
Cc: = Narasimhan= Venkataramaiah; Linda Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualiza= tion-nvgre-00" advertise its external facing hosts' IP addresses t= o external world?

Hi Murari,

=A0

= Yes that is what I mean.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> wrote= :

You mean not an Ethernet frame but some IP payload?

=A0<= /span>

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari = Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualizati= on-nvgre-00" advertise its external facing hosts' IP addresses to = external world?

=A0

Murari,

=A0

What I am saying = is the inner header should be allowed to be L3.

=A0

From the diagram you have that does not see= m to be the case. Am I missing it totally?

=A0

Thanks,

Vishwas

=

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan &l= t;muraris@micros= oft.com> wrote:

Vishwas, Thanks for the feedback we will definitely consider adding th= at. I am not sure what you mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

=A0=

Thanks

=A0=

From: Vishwas Manral [mai= lto:vishwas.iet= f@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM =


<= b>To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Srid= haran; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridh= aran-virtualization-nvgre-00" advertise its external facing hosts'= IP addresses to external world?

=A0

Hi Simha,

=A0

I see this as t= he only difference between VXLAN and the NVGRE solution (besides ofcourse t= hat TNI needs to be parsed in the intermediate device for hashing and using= lesser number of bytes).

=A0

I would think you should add it = to your draft immediately. With tunneling=A0you=A0consolidate=A0the address= es visible to the core and by providing a hash mechanism, you are providing some level of randomness.

=A0

The other thing y= ou should look at is L3 (IPv4/ IPv6)=A0over NVGRE instead of L2 alone.=A0I = guess it would be the same comment for the VXLAN proposal too.

=A0

Thanks,

V= ishwas

On Thu, Sep 22, 2= 011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:<= /u>

The draft mentions exactly this as one use= of the reserved 8 bits in Section 4. An NVGRE endpoint could use the 8 bit= s to further distribute flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?<= /u>

=A0

One<= /u>

=A0=A0 such example could be to use the upper 8 bits of the Ke= y field to

=A0=A0 add flow based entro= py and tag all the packets from a flow with an entropy label.=

=A0

Simha

=A0

=

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasim= han Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridh= aran-virtualization-nvgre-00" advertise its external facing hosts'= IP addresses to external world?

=A0

Hi Simha,

=A0<= /u>

The main (Standards Track)= =A0change in your draft is the addition of TNI.

=A0

A question I have is a TNI identifies a particular tenant an= d all flows from/to a tenant will be hashed to the same path (even with the= changes in switches to do hashing to use TNI).

=A0=

Why do you not use the last 8 bits wh= ich you have kept as reserved for providing the randomization for hashing f= lows between same to/from on different paths?

=A0

Thanks,

V= ishwas

On Sun, Sep 18, 2= 011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:=

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtuali= zation as its no different than getting the traffic routed to the enterpris= e in the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?<= u>


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host= which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
= > draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches), We = can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list = =A0or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

<= /div>

=A0

=A0

=A0

<= p class=3D"MsoNormal">=A0

_______________________________________________ armd mailing list armd@ietf.org ht= tps://www.ietf.org/mailman/listinfo/armd

--e89a8fb20652e6cd2b04b9fb96da-- From linda.dunbar@huawei.com Tue Feb 28 13:19:57 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2BEFD11E8087 for ; Tue, 28 Feb 2012 13:19:57 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.612 X-Spam-Level: X-Spam-Status: No, score=-2.612 tagged_above=-999 required=5 tests=[AWL=-0.014, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id arLGKLRq-t-j for ; Tue, 28 Feb 2012 13:19:51 -0800 (PST) Received: from dfwrgout.huawei.com (dfwrgout.huawei.com [206.16.17.72]) by ietfa.amsl.com (Postfix) with ESMTP id 5ED5611E8076 for ; Tue, 28 Feb 2012 13:19:51 -0800 (PST) Received: from 172.18.9.243 (EHLO dfweml202-edg.china.huawei.com) ([172.18.9.243]) by dfwrg02-dlp.huawei.com (MOS 4.2.3-GA FastPath) with ESMTP id ADQ87842; Tue, 28 Feb 2012 16:19:51 -0500 (EST) Received: from DFWEML405-HUB.china.huawei.com (10.193.5.102) by dfweml202-edg.china.huawei.com (172.18.9.108) with Microsoft SMTP Server (TLS) id 14.1.323.3; Tue, 28 Feb 2012 13:17:38 -0800 Received: from DFWEML505-MBX.china.huawei.com ([10.124.31.100]) by dfweml405-hub.china.huawei.com ([10.193.5.102]) with mapi id 14.01.0323.003; Wed, 29 Feb 2012 05:17:34 +0800 From: Linda Dunbar To: Vishwas Manral , Gary Berger Thread-Topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Thread-Index: AQHM80LfFUrl8No3FUamG0nVkZHmFJZQ+NEAgAHdFUA= Date: Tue, 28 Feb 2012 21:16:32 +0000 Message-ID: <4A95BA014132FF49AE685FAB4B9F17F632E2495E@dfweml505-mbx> References: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.47.155.224] Content-Type: multipart/alternative; boundary="_000_4A95BA014132FF49AE685FAB4B9F17F632E2495Edfweml505mbx_" MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "armd@ietf.org" Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2012 21:19:57 -0000 --_000_4A95BA014132FF49AE685FAB4B9F17F632E2495Edfweml505mbx_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Gary, Can you elaborate why "vm migration may not be planned"? I thought that VM migration is orchestrated by some entity, like vMotion or= VM managers. Linda From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Monday, February 27, 2012 6:49 PM To: Gary Berger Cc: Linda Dunbar; Murari Sridharan; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"adver= tise its external facing hosts' IP addresses to external world? Hi Gary, I guess what Linda was trying to say is that the station moves autonomously= , however there is some entity moving the VM (which can in turn could move = the state on the switch). An example of this could be VEPA. Thanks, Vishwas On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger > wrote: Protocols have been developed for IP mobility between Home Gateway and Remo= te Gateway. The question is if we want similar protocols among TORs or amon= g vSwitches. Handset mobility is random, but VM migration is planned. I wouldn't put such a restriction, vm migration may not be planned.. They b= oth degenerate into a multi-homing problem. Linda From: armd-bounces@ietf.org [mailto:armd-boun= ces@ietf.org] On Behalf Of Vishwas Manral Sent: Thursday, September 22, 2011 9:38 PM To: Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote: Do you have a scenario in mind? ________________________________ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing i= t totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I= am not sure what you mean by doing L3 instead of L2. We allow any arbitrar= y virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (bes= ides ofcourse that TNI needs to be parsed in the intermediate device for ha= shing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling y= ou consolidate the addresses visible to the core and by providing a hash me= chanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of= L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah > wrote: The draft mentions exactly this as one use of the reserved 8 bits in Sectio= n 4. An NVGRE endpoint could use the 8 bits to further distribute flows bel= onging to a particular TNI and the switches use all 32 bits to get entropy.= One step further would be for the switches to get full entropy from the in= ner Ethernet frame. I take it that your comment would be to make it explici= t in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entro= py label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" adve= rtise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows fro= m/to a tenant will be hashed to the same path (even with the changes in swi= tches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for prov= iding the randomization for hashing flows between same to/from on different= paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah > wrote: The easiest from the point of view of configuration would be to route every= thing back through the enterprise - not necessarily the optimal from the en= terprise point of view. Are you referring to a scenario where the VMs subne= t is split between the cloud and the enterprise? Otherwise I don't see the = implication on virtualization as its no different than getting the traffic = routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@iet= f.org] on behalf of Linda Dunbar [linda.dunba= r@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd= @ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertis= e its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center= " advertise this host address to external peers? Or will all external peers= go through enterprise's gateway to reach this host which is no longer resi= ding in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bo= unces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-= bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > _______________________________________________ armd mailing list armd@ietf= .org https://www.ietf.org/mailman/listinfo/armd --_000_4A95BA014132FF49AE685FAB4B9F17F632E2495Edfweml505mbx_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Gary,

 <= /p>

Can you elaborate why = 220;vm migration may not be planned”?

 

I thought that VM migrati= on is orchestrated by some entity, like vMotion or VM managers.

 <= /p>

Linda

 <= /p>

From: Vishwas = Manral [mailto:vishwas.ietf@gmail.com]
Sent: Monday, February 27, 2012 6:49 PM
To: Gary Berger
Cc: Linda Dunbar; Murari Sridharan; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00"advertise its external facing hosts' IP addresses to external w= orld?

 

Hi Gary,

I guess what Linda was trying to say is that the station moves autonomously= , however there is some entity moving the VM (which can in turn could move = the state on the switch).

An example of this could be VEPA.

Thanks,
Vishwas

On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger <gaberger@cisco.com> wrote:

 

 

Protocols have been developed for IP mo= bility between Home Gateway and Remote Gateway. The question is if we want similar protocols among TORs or among vSwitches. Handset mob= ility is random, but VM migration is planned.  

 

I wouldn't put such a restriction, vm mi= gration may not be planned.. They both degenerate into a multi-homing probl= em.

 

Linda

 

From: armd-bounces@iet= f.org [mailt= o:armd-bounces@ietf.org] On Behalf Of Vishwas Manral
Sent: Thursday, September 22, 2011 9:38 PM
To: Murari Sridharan
Cc: armd@ietf.org=
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari think IP mobility. :)

 

On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Do you have a scenario in mind?

From: Vishwas Manral
Sent: 9/22/2011 4:55 PM


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.black@emc.co= m; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

Hi Murari,

 

Yes that is what I mean.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

You mean not an Eth= ernet frame but some IP payload?

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Murari,

 

What I am saying is the inner header should be allowed to be L3.

 

From the diagram you have that does not seem to be the case. Am I = missing it totally?

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com>= ; wrote:

Vishwas, Thanks for= the feedback we will definitely consider adding that. I am not sure what y= ou mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3.

 <= /o:p>

Thanks<= /o:p>

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:19 PM


To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

I see this as the only difference between VXLAN and the NVGRE solu= tion (besides ofcourse that TNI needs to be parsed in the intermediate devi= ce for hashing and using lesser number of bytes).

 

I would think you should add it to your draft immediately. With tu= nneling you consolidate the addresses visible to the core an= d by providing a hash mechanism, you are providing some level of randomness.

 

The other thing you should look at is L3 (IPv4/ IPv6) over NV= GRE instead of L2 alone. I guess it would be the same comment for the = VXLAN proposal too.

 

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com= > wrote:

The draft mentions exactly this as one use of the reserved 8 bits = in Section 4. An NVGRE endpoint could use the 8 bits to further distribute = flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be= for the switches to get full entropy from the inner Ethernet frame. I take= it that your comment would be to make it explicit in the draft. Right?

 

One

   such example could be to use the upper 8 bits of the Key fi= eld to

   add flow based entropy and tag all the packets from a flow = with an entropy label.

 

Simha

 <= /o:p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to external = world?

 

Hi Simha,

 

The main (Standards Track) change in your draft is the additi= on of TNI.

 

A question I have is a TNI identifies a particular tenant and all = flows from/to a tenant will be hashed to the same path (even with the chang= es in switches to do hashing to use TNI).

 

Why do you not use the last 8 bits which you have kept as reserved= for providing the randomization for hashing flows between same to/from on = different paths?

 

Thanks,

Vishwas

On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.co= m> wrote:

The easiest from the point of view of configuration would be to ro= ute everything back through the enterprise - not necessarily the optimal fr= om the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the clou= d and the enterprise? Otherwise I don't see the implication on virtualizati= on as its no different than getting the traffic routed to the enterprise in= the first case.

Simha

________________________________________
From: armd-bounc= es@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00"= ; advertise its external facing hosts' IP addresses to external world?=


Hi Murari,

Thank you very much for sharing the presentation.

One question:

For a host within an Enterprise site which needs to communicate with extern= al peers, the host either uses public IP address which is visible to extern= al peers or uses private IP address which is translated to public address a= t the Enterprise site's gateway.

When this host is moved to "Cloud data center", will the "Cl= oud Data center" advertise this host address to external peers? Or wil= l all external peers go through enterprise's gateway to reach this host whi= ch is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: david.bla= ck@emc.com; armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> FYI, here is a talk that I gave last week in relation to the nvgre
> draft below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message-----
> From: armd-= bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt
>
> The edge switches could be the software switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Original Message-----
> > From: = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.= org
> > Subject: Re: [armd] soliciting typical network designs for ARMD > >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote:
> >
> > > Hi Linda/ Anoop,
> > >
> > > Here is the example of the design I was talking about, as de= fined
> by google.
> >
> > Just a clarification -- s/as defined by google/as described by > someone
> > who happens to work for google/
> >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > >
> > > >>>>
> > > (though I think if there was a standard way to map Multicast= MAC to
> > > Multicast IP, they could
> > probably use such a standard mechanisms).
> > > >>>>
> > >
> > > They can do that, but then this imposes requirements on the<= br> > > > equipment to be able to do multicast forwarding, and even if= does,
> > > because of pruning requirements the number of groups would b= e very
> > > large.  The average data center switch probably won't h= andle that
> > > many groups.
> > >
> > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Anoop,
> > >
> > > From what I know they do not use Multicast GRE (I hear the e= xtra 4
> > > bytes in the GRE header is a
> > proprietery extension).
> > >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map Multicast MAC to Multicast IP, they could probably use such a=
> standard mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@a= lumni.duke.edu> wrote:
> > > Hi Vishwas,
> > >
> > > How do they get multicast through the network in that case?<= br> > > > Are they planning to use multicast GRE, or just use director= y based
> > > lookups and not worry about multicast applications for now?<= br> > > >
> > > Anoop
> > >
> > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > >
> > > The data packets can be tunnelled at the ToR over say a GRE = packet
> > > and the core is a Layer-3 core
> > (except for the downstream ports). So we could have encapsulation= /
> > decapsulation of L2 over GRE at the ToR.
> > >
> > > The very same thing can be done at the hypervisor layer too,= in
> > > which case the entire DC network
> > would look like a Layer-3 flat network including the ToR to serve= r
> > link and the hypervisor would do the tunneling.
> > >
> > > I am not sure if you got the points above or not. I know clo= ud OS
> > > companies that provide the service
> > and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > Vishwas,
> > >
> > > In my mind the bullet 1) in the list refers to ToR switches<= br> > > > downstream ports (facing servers)
> > running Layer 2 and ToR uplinks ports run IP Layer 3.
> > >
> > > Have you seen data center networks with ToR switches downstr= eam
> > > ports (i.e. facing servers) enabling
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft. > > >
> > > Thanks,
> > > Linda
> > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwa= s.ietf@gmail.com> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by this, but:
> > >   * layer 3 all the way to TOR (Top of Rack switches), = We can also
> > > have a heirarchical network, with the core totally Layer-3 (= and
> > > having seperate
> > routing), from the hosts still in a large Layer-3 subnet. Another=
> > aspect could be to have a totally
> > Layer-3 network.
> > >
> > > The difference between them is the link between the servers = and the
> ToR.
> > >
> > > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com><= br> > wrote:
> > > During the 81st IETF ARMD WG discussion, it was suggested th= at it
> is
> > > necessary to document typical
> > data center network designs so that address resolution scaling is= sues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network designs.
> > Therefore, we only want to document anonymous designs without too=
> much
> > detail. During the journey of establishing ARMD, we have come acr= oss
> the following typical data center network designs:
> > >   * layer 3 all the way to TOR (Top of Rack switches),<= br> > > >   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > design will have thousands of hosts under the L2/L3 boundary rout= er
> > (s)
> > >   * CLOS design  with thousands of switches. This = design will have
> > > thousands of hosts under the
> > L2/L3 boundary router(s)
> > > We have heard that each of the designs above has its own pro= blems.
> > > ARMD problem statements might
> > need to document DC problems under each typical design.
> > > Please send feedback to us (either to the armd email list &n= bsp;or to
> the
> > > ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accelerate the progress of ARM= D WG.
> > >
> > > Thank you very much.
> > >
> > > Linda & Benson
> > >

 

 

 

 

________________________________________= _______ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd

 

--_000_4A95BA014132FF49AE685FAB4B9F17F632E2495Edfweml505mbx_-- From mksmith@mac.com Tue Feb 28 13:31:37 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A90EA21E802C for ; Tue, 28 Feb 2012 13:31:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x9n51idRe5kk for ; Tue, 28 Feb 2012 13:31:32 -0800 (PST) Received: from nk11p99mm-asmtpout003.mac.com (nk11p03mm-asmtp993.mac.com [17.158.233.224]) by ietfa.amsl.com (Postfix) with ESMTP id 1A86721E8053 for ; Tue, 28 Feb 2012 13:31:31 -0800 (PST) MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_JaqPnYzZ/HTrxg1AyB4VGw)" Received: from adwks52 (unknown [216.211.143.124]) by nk11p03mm-asmtp993.mac.com (Oracle Communications Messaging Exchange Server 7u4-22.01 64bit (built Apr 21 2011)) with ESMTPSA id <0M0400FCOH4AQ100@nk11p03mm-asmtp993.mac.com> for armd@ietf.org; Tue, 28 Feb 2012 13:31:24 -0800 (PST) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.6.7498,1.0.260,0.0.0000 definitions=2012-02-28_08:2012-02-28, 2012-02-28, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=6.0.2-1012030000 definitions=main-1202280219 From: Michael Smith Sender: Michael Smith To: 'Linda Dunbar' , 'Vishwas Manral' , 'Gary Berger' References: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> <4A95BA014132FF49AE685FAB4B9F17F632E2495E@dfweml505-mbx> In-reply-to: <4A95BA014132FF49AE685FAB4B9F17F632E2495E@dfweml505-mbx> Date: Tue, 28 Feb 2012 13:31:22 -0800 Message-id: <151901ccf660$5175f500$f461df00$@mac.com> X-Mailer: Microsoft Outlook 14.0 Thread-index: AQFBgImgjHc9GOBNhB03d9UiBPUVDgIidqOuAoKhqSYCXK5zlpcxsESQ Content-language: en-us Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2012 21:31:37 -0000 This is a multipart message in MIME format. --Boundary_(ID_JaqPnYzZ/HTrxg1AyB4VGw) Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT VM migration can be planned or unplanned, either manually or automatically. In the event of a host failing that is configured for automatic migration, the VM's on the host will migrate to another host as configured. There is some internal intelligence that will allow for best placement of VM's, so they can move to any one of a number of available hosts. Mike From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar Sent: Tuesday, February 28, 2012 1:17 PM To: Vishwas Manral; Gary Berger Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Gary, Can you elaborate why "vm migration may not be planned"? I thought that VM migration is orchestrated by some entity, like vMotion or VM managers. Linda From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Monday, February 27, 2012 6:49 PM To: Gary Berger Cc: Linda Dunbar; Murari Sridharan; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? Hi Gary, I guess what Linda was trying to say is that the station moves autonomously, however there is some entity moving the VM (which can in turn could move the state on the switch). An example of this could be VEPA. Thanks, Vishwas On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger wrote: Protocols have been developed for IP mobility between Home Gateway and Remote Gateway. The question is if we want similar protocols among TORs or among vSwitches. Handset mobility is random, but VM migration is planned. I wouldn't put such a restriction, vm migration may not be planned.. They both degenerate into a multi-homing problem. Linda From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manral Sent: Thursday, September 22, 2011 9:38 PM To: Murari Sridharan Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Murari think IP mobility. :) On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan wrote: Do you have a scenario in mind? _____ From: Vishwas Manral Sent: 9/22/2011 4:55 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Murari, Yes that is what I mean. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan wrote: You mean not an Ethernet frame but some IP payload? From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:49 PM To: Murari Sridharan Cc: Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Murari, What I am saying is the inner header should be allowed to be L3. >From the diagram you have that does not seem to be the case. Am I missing it totally? Thanks, Vishwas On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan wrote: Vishwas, Thanks for the feedback we will definitely consider adding that. I am not sure what you mean by doing L3 instead of L2. We allow any arbitrary virtual topology including L3. Thanks From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:19 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Simha, I see this as the only difference between VXLAN and the NVGRE solution (besides ofcourse that TNI needs to be parsed in the intermediate device for hashing and using lesser number of bytes). I would think you should add it to your draft immediately. With tunneling you consolidate the addresses visible to the core and by providing a hash mechanism, you are providing some level of randomness. The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of L2 alone. I guess it would be the same comment for the VXLAN proposal too. Thanks, Vishwas On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah wrote: The draft mentions exactly this as one use of the reserved 8 bits in Section 4. An NVGRE endpoint could use the 8 bits to further distribute flows belonging to a particular TNI and the switches use all 32 bits to get entropy. One step further would be for the switches to get full entropy from the inner Ethernet frame. I take it that your comment would be to make it explicit in the draft. Right? One such example could be to use the upper 8 bits of the Key field to add flow based entropy and tag all the packets from a flow with an entropy label. Simha From: Vishwas Manral [mailto:vishwas.ietf@gmail.com] Sent: Thursday, September 22, 2011 4:04 PM To: Narasimhan Venkataramaiah Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Simha, The main (Standards Track) change in your draft is the addition of TNI. A question I have is a TNI identifies a particular tenant and all flows from/to a tenant will be hashed to the same path (even with the changes in switches to do hashing to use TNI). Why do you not use the last 8 bits which you have kept as reserved for providing the randomization for hashing flows between same to/from on different paths? Thanks, Vishwas On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah wrote: The easiest from the point of view of configuration would be to route everything back through the enterprise - not necessarily the optimal from the enterprise point of view. Are you referring to a scenario where the VMs subnet is split between the cloud and the enterprise? Otherwise I don't see the implication on virtualization as its no different than getting the traffic routed to the enterprise in the first case. Simha ________________________________________ From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com] Sent: Sunday, September 18, 2011 7:06 AM To: Murari Sridharan; david.black@emc.com; armd@ietf.org Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" advertise its external facing hosts' IP addresses to external world? Hi Murari, Thank you very much for sharing the presentation. One question: For a host within an Enterprise site which needs to communicate with external peers, the host either uses public IP address which is visible to external peers or uses private IP address which is translated to public address at the Enterprise site's gateway. When this host is moved to "Cloud data center", will the "Cloud Data center" advertise this host address to external peers? Or will all external peers go through enterprise's gateway to reach this host which is no longer residing in the enterprise site? Thanks, Linda > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > Murari Sridharan > Sent: Saturday, September 17, 2011 3:02 PM > To: david.black@emc.com; armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > FYI, here is a talk that I gave last week in relation to the nvgre > draft below. > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > Thanks > Murari > > -----Original Message----- > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > david.black@emc.com > Sent: Friday, September 16, 2011 6:14 AM > To: armd@ietf.org > Subject: Re: [armd] soliciting typical network designs for ARMD > > And two more drafts on this topic: > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > The edge switches could be the software switches in hypervisors. > > Thanks, > --David > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > Of Warren Kumari > > Sent: Wednesday, August 31, 2011 3:16 PM > > To: Vishwas Manral > > Cc: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > Hi Linda/ Anoop, > > > > > > Here is the example of the design I was talking about, as defined > by google. > > > > Just a clarification -- s/as defined by google/as described by > someone > > who happens to work for google/ > > > > W > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > wrote: > > > > > > >>>> > > > (though I think if there was a standard way to map Multicast MAC to > > > Multicast IP, they could > > probably use such a standard mechanisms). > > > >>>> > > > > > > They can do that, but then this imposes requirements on the > > > equipment to be able to do multicast forwarding, and even if does, > > > because of pruning requirements the number of groups would be very > > > large. The average data center switch probably won't handle that > > > many groups. > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > wrote: > > > Hi Anoop, > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > bytes in the GRE header is a > > proprietery extension). > > > > > > I think a directory based mechanism is what is used (though I think > > > if there was a standard way to > > map Multicast MAC to Multicast IP, they could probably use such a > standard mechanisms). > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > wrote: > > > Hi Vishwas, > > > > > > How do they get multicast through the network in that case? > > > Are they planning to use multicast GRE, or just use directory based > > > lookups and not worry about multicast applications for now? > > > > > > Anoop > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > and the core is a Layer-3 core > > (except for the downstream ports). So we could have encapsulation/ > > decapsulation of L2 over GRE at the ToR. > > > > > > The very same thing can be done at the hypervisor layer too, in > > > which case the entire DC network > > would look like a Layer-3 flat network including the ToR to server > > link and the hypervisor would do the tunneling. > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > companies that provide the service > > and have big announced customers. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > wrote: > > > Vishwas, > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > downstream ports (facing servers) > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > Have you seen data center networks with ToR switches downstream > > > ports (i.e. facing servers) enabling > > IP routing, even though the physical links are Ethernet? > > > If yes, we should definitely include it in the ARMD draft. > > > > > > Thanks, > > > Linda > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > wrote: > > > Hi Linda, > > > I am unsure what you mean by this, but: > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > have a heirarchical network, with the core totally Layer-3 (and > > > having seperate > > routing), from the hosts still in a large Layer-3 subnet. Another > > aspect could be to have a totally > > Layer-3 network. > > > > > > The difference between them is the link between the servers and the > ToR. > > > > > > Thanks, > > > Vishwas > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > wrote: > > > During the 81st IETF ARMD WG discussion, it was suggested that it > is > > > necessary to document typical > > data center network designs so that address resolution scaling issues > > can be properly described. Many data center operators have expressed > that they can't openly reveal their detailed network designs. > > Therefore, we only want to document anonymous designs without too > much > > detail. During the journey of establishing ARMD, we have come across > the following typical data center network designs: > > > * layer 3 all the way to TOR (Top of Rack switches), > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > interconnected by Layer 2. This > > design will have thousands of hosts under the L2/L3 boundary router > > (s) > > > * CLOS design with thousands of switches. This design will have > > > thousands of hosts under the > > L2/L3 boundary router(s) > > > We have heard that each of the designs above has its own problems. > > > ARMD problem statements might > > need to document DC problems under each typical design. > > > Please send feedback to us (either to the armd email list or to > the > > > ARMD chair Benson & Linda) to > > indicate if we have missed any typical Data Center network designs. > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > Thank you very much. > > > > > > Linda & Benson > > > _______________________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd --Boundary_(ID_JaqPnYzZ/HTrxg1AyB4VGw) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: quoted-printable

VM migration can be planned or = unplanned, either manually or automatically.   In the event of a host = failing that is configured for automatic migration, the VM’s on = the host will migrate to another host as configured.  There is some internal = intelligence that will allow for best placement of VM’s, so they = can move to any one of a number of available = hosts.


Mike

 

From: armd-bounces@ietf.org = [mailto:armd-bounces@ietf.org] On Behalf Of Linda = Dunbar
Sent: Tuesday, February 28, 2012 1:17 PM
To: = Vishwas Manral; Gary Berger
Cc: = armd@ietf.org
Subject: Re: [armd] how does = "draft-sridharan-virtualization-nvgre-00"advertise its = external facing hosts' IP addresses to external = world?

 

Gary,

 

Can you elaborate why “vm = migration may not be planned”?

 <= /o:p>

I thought that VM migration is orchestrated by some entity, like = vMotion or VM managers.

 

Linda

 

From:= = Vishwas Manral [mailto:vishwas.ietf@gmai= l.com]
Sent: Monday, February 27, 2012 6:49 = PM
To: Gary Berger
Cc: Linda Dunbar; Murari = Sridharan; armd@ietf.org
Subject: Re: = [armd] how does = "draft-sridharan-virtualization-nvgre-00"advertise its = external facing hosts' IP addresses to external = world?

 

Hi Gary,

I guess what Linda was = trying to say is that the station moves autonomously, however there is = some entity moving the VM (which can in turn could move the state on the = switch).

An example of this could be = VEPA.

Thanks,
Vishwas

On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger <gaberger@cisco.com> = wrote:

 <= /o:p>

 <= /o:p>

Protocols have been developed for IP mobility between Home Gateway = and Remote Gateway. The question is if we want similar protocols among = TORs or among vSwitches. Handset mobility is random, but VM migration is = planned. =  

 <= /o:p>

I wouldn't = put such a restriction, vm migration may not be planned.. They both = degenerate into a multi-homing = problem.

 

Linda

 

From:= = armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of = Vishwas Manral
Sent: Thursday, September 22, 2011 9:38 = PM
To: Murari Sridharan
Cc: armd@ietf.org
Subject: Re: [armd] how = does "draft-sridharan-virtualization-nvgre-00" advertise its = external facing hosts' IP addresses to external = world?

 <= /o:p>

Murari = think IP mobility. :)

 <= /o:p>

On Thu, Sep = 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com> = wrote:

Do you = have a scenario in mind?

From: = Vishwas = Manral
Sent: = 9/22/2011 = 4:55 PM


To: = Murari = Sridharan
Cc: = Narasimhan = Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org
Subject: = Re: [armd] = how does "draft-sridharan-virtualization-nvgre-00" advertise = its external facing hosts' IP addresses to external = world?

Hi = Murari,

 <= /o:p>

Yes that is = what I mean.

 <= /o:p>

Thanks,=

Vishwas=

On Thu, Sep = 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> = wrote:

You mean not an Ethernet frame = but some IP payload?

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, = September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: = Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; armd@ietf.org


Subje= ct: Re: [armd] how does = "draft-sridharan-virtualization-nvgre-00" advertise its = external facing hosts' IP addresses to external = world?

 <= /o:p>

Murari,=

 <= /o:p>

What I am = saying is the inner header should be allowed to be L3. =

 <= /o:p>

From the = diagram you have that does not seem to be the case. Am I missing it = totally?

 <= /o:p>

Thanks,=

Vishwas=

On Thu, Sep = 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> = wrote:

Vishwas, Thanks for the = feedback we will definitely consider adding that. I am not sure what you = mean by doing L3 instead of L2. We allow any arbitrary virtual topology = including L3.

 

Thanks

 

=

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, = September 22, 2011 4:19 PM


To: Narasimhan = Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how = does "draft-sridharan-virtualization-nvgre-00" advertise its = external facing hosts' IP addresses to external = world?

 <= /o:p>

Hi = Simha,

 <= /o:p>

I see this = as the only difference between VXLAN and the NVGRE solution (besides = ofcourse that TNI needs to be parsed in the intermediate device for = hashing and using lesser number of bytes).

 <= /o:p>

I would = think you should add it to your draft immediately. With = tunneling you consolidate the addresses visible to the = core and by providing a hash mechanism, you are providing some level of = randomness.

 <= /o:p>

The other = thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead of = L2 alone. I guess it would be the same comment for the VXLAN = proposal too.

 <= /o:p>

Thanks,=

Vishwas=

On Thu, Sep = 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com> = wrote:

The draft = mentions exactly this as one use of the reserved 8 bits in Section 4. An = NVGRE endpoint could use the 8 bits to further distribute flows = belonging to a particular TNI and the switches use all 32 bits to get = entropy. One step further would be for the switches to get full entropy = from the inner Ethernet frame. I take it that your comment would be to = make it explicit in the draft. Right?

 

One

   such = example could be to use the upper 8 bits of the Key field = to

   add = flow based entropy and tag all the packets from a flow with an entropy = label.

 

Simha

 

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, = September 22, 2011 4:04 PM
To: Narasimhan = Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how = does "draft-sridharan-virtualization-nvgre-00" advertise its = external facing hosts' IP addresses to external = world?

 <= /o:p>

Hi = Simha,

 <= /o:p>

The main = (Standards Track) change in your draft is the addition of = TNI.

 <= /o:p>

A question = I have is a TNI identifies a particular tenant and all flows from/to a = tenant will be hashed to the same path (even with the changes in = switches to do hashing to use TNI).

 <= /o:p>

Why do you = not use the last 8 bits which you have kept as reserved for providing = the randomization for hashing flows between same to/from on different = paths?

 <= /o:p>

Thanks,=

Vishwas=

On Sun, Sep = 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:

The easiest from the point of view of configuration would be to = route everything back through the enterprise - not necessarily the = optimal from the enterprise point of view. Are you referring to a = scenario where the VMs subnet is split between the cloud and the = enterprise? Otherwise I don't see the implication on virtualization as = its no different than getting the traffic routed to the enterprise in = the first = case.

Simha

________________________________________
Fro= m: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda Dunbar = [linda.dunbar@huawei.com]
Sent: Sunday, = September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: [armd] how does = "draft-sridharan-virtualization-nvgre-00" advertise its = external facing hosts' IP addresses to external = world?


Hi = Murari,

Thank you very much for sharing the = presentation.

One question:

For a host within an = Enterprise site which needs to communicate with external peers, the host = either uses public IP address which is visible to external peers or uses = private IP address which is translated to public address at the = Enterprise site's gateway.

When this host is moved to "Cloud = data center", will the "Cloud Data center" advertise this = host address to external peers? Or will all external peers go through = enterprise's gateway to reach this host which is no longer residing in = the enterprise site?

Thanks, Linda

> -----Original = Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> Murari = Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM
> To: = david.black@emc.com; armd@ietf.org
> Subject: Re: [armd] = soliciting typical network designs for ARMD
>
> FYI, here is = a talk that I gave last week in relation to the nvgre
> draft = below.
> http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442= T
>
> Thanks
> Murari
>
> = -----Original Message-----
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@emc.com
> Sent: Friday, = September 16, 2011 6:14 AM
> To: armd@ietf.org
> Subject: Re: [armd] = soliciting typical network designs for ARMD
>
> And two more = drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxla= n-00.txt
> http://www.ietf.org/id/draft-sridharan-virtualization-n= vgre-00.txt
>
> The edge switches could be the software = switches in hypervisors.
>
> Thanks,
> = --David
>
>
> > -----Original Message-----
> = > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf
> > Of = Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:16 = PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: [armd] = soliciting typical network designs for ARMD
> >
> = >
> > On Aug 11, 2011, at 11:40 PM, Vishwas Manral = wrote:
> >
> > > Hi Linda/ Anoop,
> > = >
> > > Here is the example of the design I was talking = about, as defined
> by google.
> >
> > Just a = clarification -- s/as defined by google/as described by
> = someone
> > who happens to work for google/
> = >
> > W
> >
> > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobilit= y-00.txt
> > >
> > > Thanks,
> > = > Vishwas
> > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop = Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > = >
> > > >>>>
> > > (though I = think if there was a standard way to map Multicast MAC to
> > = > Multicast IP, they could
> > probably use such a standard = mechanisms).
> > > >>>>
> > = >
> > > They can do that, but then this imposes = requirements on the
> > > equipment to be able to do = multicast forwarding, and even if does,
> > > because of = pruning requirements the number of groups would be very
> > = > large.  The average data center switch probably won't handle = that
> > > many groups.
> > >
> > > = On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > = > Hi Anoop,
> > >
> > > From what I know they = do not use Multicast GRE (I hear the extra 4
> > > bytes in = the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is = used (though I think
> > > if there was a standard way = to
> > map Multicast MAC to Multicast IP, they could probably = use such a
> standard mechanisms).
> > >
> > = > Thanks,
> > > Vishwas
> > > On Tue, Aug 9, = 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu> wrote:
> > > = Hi Vishwas,
> > >
> > > How do they get = multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > = > lookups and not worry about multicast applications for now?
> = > >
> > > Anoop
> > >
> > > On = Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > = > Hi Linda,
> > >
> > > The data packets can = be tunnelled at the ToR over say a GRE packet
> > > and the = core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over = GRE at the ToR.
> > >
> > > The very same thing = can be done at the hypervisor layer too, in
> > > which case = the entire DC network
> > would look like a Layer-3 flat = network including the ToR to server
> > link and the hypervisor = would do the tunneling.
> > >
> > > I am not = sure if you got the points above or not. I know cloud OS
> > = > companies that provide the service
> > and have big = announced customers.
> > >
> > > Thanks,
> = > > Vishwas
> > > On Tue, Aug 9, 2011 at 11:51 AM, = Linda Dunbar <dunbar.ll@gmail.com>
> wrote:
> = > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > = downstream ports (facing servers)
> > running Layer 2 and ToR = uplinks ports run IP Layer 3.
> > >
> > > Have = you seen data center networks with ToR switches downstream
> > = > ports (i.e. facing servers) enabling
> > IP routing, even = though the physical links are Ethernet?
> > > If yes, we = should definitely include it in the ARMD draft.
> > = >
> > > Thanks,
> > > Linda
> > > = On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> wrote:
> > = > Hi Linda,
> > > I am unsure what you mean by this, = but:
> > >   * layer 3 all the way to TOR (Top of Rack = switches), We can also
> > > have a heirarchical network, = with the core totally Layer-3 (and
> > > having = seperate
> > routing), from the hosts still in a large Layer-3 = subnet. Another
> > aspect could be to have a totally
> = > Layer-3 network.
> > >
> > > The difference = between them is the link between the servers and the
> = ToR.
> > >
> > > Thanks,
> > > = Vishwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar = <dunbar.ll@gmail.com>
> wrote:
> = > > During the 81st IETF ARMD WG discussion, it was suggested that = it
> is
> > > necessary to document typical
> = > data center network designs so that address resolution scaling = issues
> > can be properly described. Many data center = operators have expressed
> that they can't openly reveal their = detailed network designs.
> > Therefore, we only want to = document anonymous designs without too
> much
> > detail. = During the journey of establishing ARMD, we have come across
> the = following typical data center network designs:
> > >   = * layer 3 all the way to TOR (Top of Rack switches),
> > > =   * large layer 2 with hundreds (or thousands) of ToRs = being
> > > interconnected by Layer 2. This
> > = design will have thousands of hosts under the L2/L3 boundary = router
> > (s)
> > >   * CLOS design =  with thousands of switches. This design will have
> > = > thousands of hosts under the
> > L2/L3 boundary = router(s)
> > > We have heard that each of the designs above = has its own problems.
> > > ARMD problem statements = might
> > need to document DC problems under each typical = design.
> > > Please send feedback to us (either to the armd = email list  or to
> the
> > > ARMD chair Benson = & Linda) to
> > indicate if we have missed any typical Data = Center network designs.
> > >
> > > Your = contribution can greatly accelerate the progress of ARMD WG.
> = > >
> > > Thank you very much.
> > = >
> > > Linda & Benson
> > = >

 <= /o:p>

 <= /o:p>

 <= /o:p>

 <= /o:p>

____________= ___________________________________ armd mailing list armd@ietf.org https://www.ietf.org/mailman/listinfo/armd =

 

= --Boundary_(ID_JaqPnYzZ/HTrxg1AyB4VGw)-- From vishwas.ietf@gmail.com Wed Feb 29 15:31:52 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8E4E621F86D7 for ; Wed, 29 Feb 2012 15:31:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.786 X-Spam-Level: X-Spam-Status: No, score=-3.786 tagged_above=-999 required=5 tests=[AWL=-0.188, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8q4OjFbHaLq1 for ; Wed, 29 Feb 2012 15:31:49 -0800 (PST) Received: from mail-tul01m020-f172.google.com (mail-tul01m020-f172.google.com [209.85.214.172]) by ietfa.amsl.com (Postfix) with ESMTP id 99BD421F86B8 for ; Wed, 29 Feb 2012 15:31:49 -0800 (PST) Received: by obbeh20 with SMTP id eh20so5536707obb.31 for ; Wed, 29 Feb 2012 15:31:49 -0800 (PST) Received-SPF: pass (google.com: domain of vishwas.ietf@gmail.com designates 10.182.232.106 as permitted sender) client-ip=10.182.232.106; Authentication-Results: mr.google.com; spf=pass (google.com: domain of vishwas.ietf@gmail.com designates 10.182.232.106 as permitted sender) smtp.mail=vishwas.ietf@gmail.com; dkim=pass header.i=vishwas.ietf@gmail.com Received: from mr.google.com ([10.182.232.106]) by 10.182.232.106 with SMTP id tn10mr1064764obc.6.1330558309298 (num_hops = 1); Wed, 29 Feb 2012 15:31:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wRbd6ZeyGQn9RyE4Vi4Iq7lZsgTtge310L+J3bJaTOY=; b=iP7+rW0s+SpYTzeNIQA0yVehfTgdSkKB+0nyF5CkFOrpyNaqO78G0xDrfpHkSTkWrl hlPW0zqzP7zlWx9TRIErNPtmBDqJrsSM4+9yY6F0UrSZAGGK6a9jsgfF0EkwpB6L62DU 3r3JqzoM4QKX+HzppTvzfIzk5aMnL+VAwoj9E= MIME-Version: 1.0 Received: by 10.182.232.106 with SMTP id tn10mr900825obc.6.1330558309161; Wed, 29 Feb 2012 15:31:49 -0800 (PST) Received: by 10.182.165.1 with HTTP; Wed, 29 Feb 2012 15:31:48 -0800 (PST) In-Reply-To: <151901ccf660$5175f500$f461df00$@mac.com> References: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> <4A95BA014132FF49AE685FAB4B9F17F632E2495E@dfweml505-mbx> <151901ccf660$5175f500$f461df00$@mac.com> Date: Wed, 29 Feb 2012 15:31:48 -0800 Message-ID: From: Vishwas Manral To: Michael Smith Content-Type: multipart/alternative; boundary=f46d044473675ed9b504ba22bf17 Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Feb 2012 23:31:52 -0000 --f46d044473675ed9b504ba22bf17 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Michael, Please forgive my ignorance here. I guess what you are saying is that the trigger could be some asynchronous event not initiated by the operator. However isn't the management entity (not manual intervention) involved in actually moving the VM and any storage/ state associated with it? If it is the management entity knows of the movement. Thanks, Vishwas On Tue, Feb 28, 2012 at 1:31 PM, Michael Smith wrote: > VM migration can be planned or unplanned, either manually or > automatically. In the event of a host failing that is configured for > automatic migration, the VM=92s on the host will migrate to another host = as > configured. There is some internal intelligence that will allow for best > placement of VM=92s, so they can move to any one of a number of available > hosts.**** > > > Mike**** > > ** ** > > *From:* armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] *On Behalf > Of *Linda Dunbar > > *Sent:* Tuesday, February 28, 2012 1:17 PM > *To:* Vishwas Manral; Gary Berger > *Cc:* armd@ietf.org > *Subject:* Re: [armd] how does > "draft-sridharan-virtualization-nvgre-00"advertise its external facing > hosts' IP addresses to external world?**** > > ** ** > > Gary, **** > > ** ** > > Can you elaborate why =93vm migration may not be planned=94?**** > > ** ** > > I thought that VM migration is orchestrated by some entity, like vMotion > or VM managers. **** > > ** ** > > Linda **** > > ** ** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Monday, February 27, 2012 6:49 PM > *To:* Gary Berger > *Cc:* Linda Dunbar; Murari Sridharan; armd@ietf.org > *Subject:* Re: [armd] how does > "draft-sridharan-virtualization-nvgre-00"advertise its external facing > hosts' IP addresses to external world?**** > > ** ** > > Hi Gary, > > I guess what Linda was trying to say is that the station moves > autonomously, however there is some entity moving the VM (which can in tu= rn > could move the state on the switch). > > An example of this could be VEPA. > > Thanks, > Vishwas**** > > On Fri, Feb 24, 2012 at 2:22 PM, Gary Berger wrote:*= * > ** > > ** ** > > ** ** > > Protocols have been developed for IP mobility between Home Gateway and > Remote Gateway. The question is if we want similar protocols among TORs o= r > among vSwitches. Handset mobility is random, but VM migration is planned. > **** > > ** ** > > I wouldn't put such a restriction, vm migration may not be planned.. They > both degenerate into a multi-homing problem.**** > > **** > > Linda**** > > **** > > *From:* armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] > *On Behalf Of *Vishwas Manral > *Sent:* Thursday, September 22, 2011 9:38 PM > *To:* Murari Sridharan > *Cc:* armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari think IP mobility. :)**** > > **** > > On Thu, Sep 22, 2011 at 5:00 PM, Murari Sridharan > wrote:**** > > Do you have a scenario in mind?**** > ------------------------------ > > *From: *Vishwas Manral > *Sent: *9/22/2011 4:55 PM**** > > > *To: *Murari Sridharan > *Cc: *Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org > *Subject: *Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > Hi Murari,**** > > **** > > Yes that is what I mean.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan > wrote:**** > > You mean not an Ethernet frame but some IP payload?**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:49 PM > *To:* Murari Sridharan > *Cc:* Narasimhan Venkataramaiah; Linda Dunbar; david.black@emc.com; > armd@ietf.org**** > > > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Murari,**** > > **** > > What I am saying is the inner header should be allowed to be L3. **** > > **** > > From the diagram you have that does not seem to be the case. Am I missing > it totally?**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan > wrote:**** > > Vishwas, Thanks for the feedback we will definitely consider adding that. > I am not sure what you mean by doing L3 instead of L2. We allow any > arbitrary virtual topology including L3. **** > > **** > > Thanks**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:19 PM **** > > > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > I see this as the only difference between VXLAN and the NVGRE solution > (besides ofcourse that TNI needs to be parsed in the intermediate device > for hashing and using lesser number of bytes).**** > > **** > > I would think you should add it to your draft immediately. With > tunneling you consolidate the addresses visible to the core and by > providing a hash mechanism, you are providing some level of randomness.**= * > * > > **** > > The other thing you should look at is L3 (IPv4/ IPv6) over NVGRE instead > of L2 alone. I guess it would be the same comment for the VXLAN proposal > too.**** > > **** > > Thanks,**** > > Vishwas**** > > On Thu, Sep 22, 2011 at 4:11 PM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The draft mentions exactly this as one use of the reserved 8 bits in > Section 4. An NVGRE endpoint could use the 8 bits to further distribute > flows belonging to a particular TNI and the switches use all 32 bits to g= et > entropy. One step further would be for the switches to get full entropy > from the inner Ethernet frame. I take it that your comment would be to ma= ke > it explicit in the draft. Right?**** > > **** > > One**** > > such example could be to use the upper 8 bits of the Key field to**** > > add flow based entropy and tag all the packets from a flow with an > entropy label.**** > > **** > > Simha**** > > **** > > *From:* Vishwas Manral [mailto:vishwas.ietf@gmail.com] > *Sent:* Thursday, September 22, 2011 4:04 PM > *To:* Narasimhan Venkataramaiah > *Cc:* Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org > *Subject:* Re: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > **** > > Hi Simha,**** > > **** > > The main (Standards Track) change in your draft is the addition of TNI.**= * > * > > **** > > A question I have is a TNI identifies a particular tenant and all flows > from/to a tenant will be hashed to the same path (even with the changes i= n > switches to do hashing to use TNI).**** > > **** > > Why do you not use the last 8 bits which you have kept as reserved for > providing the randomization for hashing flows between same to/from on > different paths?**** > > **** > > Thanks,**** > > Vishwas**** > > On Sun, Sep 18, 2011 at 11:01 AM, Narasimhan Venkataramaiah < > narave@microsoft.com> wrote:**** > > The easiest from the point of view of configuration would be to route > everything back through the enterprise - not necessarily the optimal from > the enterprise point of view. Are you referring to a scenario where the V= Ms > subnet is split between the cloud and the enterprise? Otherwise I don't s= ee > the implication on virtualization as its no different than getting the > traffic routed to the enterprise in the first case. > > Simha > > ________________________________________ > From: armd-bounces@ietf.org [armd-bounces@ietf.org] on behalf of Linda > Dunbar [linda.dunbar@huawei.com] > Sent: Sunday, September 18, 2011 7:06 AM > To: Murari Sridharan; david.black@emc.com; armd@ietf.org > Subject: [armd] how does "draft-sridharan-virtualization-nvgre-00" > advertise its external facing hosts' IP addresses to external world?**** > > > Hi Murari, > > Thank you very much for sharing the presentation. > > One question: > > For a host within an Enterprise site which needs to communicate with > external peers, the host either uses public IP address which is visible t= o > external peers or uses private IP address which is translated to public > address at the Enterprise site's gateway. > > When this host is moved to "Cloud data center", will the "Cloud Data > center" advertise this host address to external peers? Or will all extern= al > peers go through enterprise's gateway to reach this host which is no long= er > residing in the enterprise site? > > Thanks, Linda > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > Murari Sridharan > > Sent: Saturday, September 17, 2011 3:02 PM > > To: david.black@emc.com; armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > FYI, here is a talk that I gave last week in relation to the nvgre > > draft below. > > http://channel9.msdn.com/Events/BUILD/BUILD2011/SAC-442T > > > > Thanks > > Murari > > > > -----Original Message----- > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of > > david.black@emc.com > > Sent: Friday, September 16, 2011 6:14 AM > > To: armd@ietf.org > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > And two more drafts on this topic: > > > > http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt > > http://www.ietf.org/id/draft-sridharan-virtualization-nvgre-00.txt > > > > The edge switches could be the software switches in hypervisors. > > > > Thanks, > > --David > > > > > > > -----Original Message----- > > > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf > > > Of Warren Kumari > > > Sent: Wednesday, August 31, 2011 3:16 PM > > > To: Vishwas Manral > > > Cc: armd@ietf.org > > > Subject: Re: [armd] soliciting typical network designs for ARMD > > > > > > > > > On Aug 11, 2011, at 11:40 PM, Vishwas Manral wrote: > > > > > > > Hi Linda/ Anoop, > > > > > > > > Here is the example of the design I was talking about, as defined > > by google. > > > > > > Just a clarification -- s/as defined by google/as described by > > someone > > > who happens to work for google/ > > > > > > W > > > > > > > http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility-00.txt > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani > > wrote: > > > > > > > > >>>> > > > > (though I think if there was a standard way to map Multicast MAC to > > > > Multicast IP, they could > > > probably use such a standard mechanisms). > > > > >>>> > > > > > > > > They can do that, but then this imposes requirements on the > > > > equipment to be able to do multicast forwarding, and even if does, > > > > because of pruning requirements the number of groups would be very > > > > large. The average data center switch probably won't handle that > > > > many groups. > > > > > > > > On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral > > wrote: > > > > Hi Anoop, > > > > > > > > From what I know they do not use Multicast GRE (I hear the extra 4 > > > > bytes in the GRE header is a > > > proprietery extension). > > > > > > > > I think a directory based mechanism is what is used (though I think > > > > if there was a standard way to > > > map Multicast MAC to Multicast IP, they could probably use such a > > standard mechanisms). > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani > > wrote: > > > > Hi Vishwas, > > > > > > > > How do they get multicast through the network in that case? > > > > Are they planning to use multicast GRE, or just use directory based > > > > lookups and not worry about multicast applications for now? > > > > > > > > Anoop > > > > > > > > On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > > > > > The data packets can be tunnelled at the ToR over say a GRE packet > > > > and the core is a Layer-3 core > > > (except for the downstream ports). So we could have encapsulation/ > > > decapsulation of L2 over GRE at the ToR. > > > > > > > > The very same thing can be done at the hypervisor layer too, in > > > > which case the entire DC network > > > would look like a Layer-3 flat network including the ToR to server > > > link and the hypervisor would do the tunneling. > > > > > > > > I am not sure if you got the points above or not. I know cloud OS > > > > companies that provide the service > > > and have big announced customers. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar > > wrote: > > > > Vishwas, > > > > > > > > In my mind the bullet 1) in the list refers to ToR switches > > > > downstream ports (facing servers) > > > running Layer 2 and ToR uplinks ports run IP Layer 3. > > > > > > > > Have you seen data center networks with ToR switches downstream > > > > ports (i.e. facing servers) enabling > > > IP routing, even though the physical links are Ethernet? > > > > If yes, we should definitely include it in the ARMD draft. > > > > > > > > Thanks, > > > > Linda > > > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral > > wrote: > > > > Hi Linda, > > > > I am unsure what you mean by this, but: > > > > * layer 3 all the way to TOR (Top of Rack switches), We can also > > > > have a heirarchical network, with the core totally Layer-3 (and > > > > having seperate > > > routing), from the hosts still in a large Layer-3 subnet. Another > > > aspect could be to have a totally > > > Layer-3 network. > > > > > > > > The difference between them is the link between the servers and the > > ToR. > > > > > > > > Thanks, > > > > Vishwas > > > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar > > wrote: > > > > During the 81st IETF ARMD WG discussion, it was suggested that it > > is > > > > necessary to document typical > > > data center network designs so that address resolution scaling issues > > > can be properly described. Many data center operators have expressed > > that they can't openly reveal their detailed network designs. > > > Therefore, we only want to document anonymous designs without too > > much > > > detail. During the journey of establishing ARMD, we have come across > > the following typical data center network designs: > > > > * layer 3 all the way to TOR (Top of Rack switches), > > > > * large layer 2 with hundreds (or thousands) of ToRs being > > > > interconnected by Layer 2. This > > > design will have thousands of hosts under the L2/L3 boundary router > > > (s) > > > > * CLOS design with thousands of switches. This design will have > > > > thousands of hosts under the > > > L2/L3 boundary router(s) > > > > We have heard that each of the designs above has its own problems. > > > > ARMD problem statements might > > > need to document DC problems under each typical design. > > > > Please send feedback to us (either to the armd email list or to > > the > > > > ARMD chair Benson & Linda) to > > > indicate if we have missed any typical Data Center network designs. > > > > > > > > Your contribution can greatly accelerate the progress of ARMD WG. > > > > > > > > Thank you very much. > > > > > > > > Linda & Benson > > > >**** > > **** > > **** > > **** > > **** > > _______________________________________________ armd mailing list > armd@ietf.org https://www.ietf.org/mailman/listinfo/armd **** > > ** ** > --f46d044473675ed9b504ba22bf17 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi Michael,

Please forgive my ignorance here. I guess what you are s= aying is that the trigger could be some asynchronous event not initiated by= the operator.

However isn't the management entity (not manual i= ntervention) involved in actually moving the VM and any storage/ state asso= ciated with it? If it is the management entity knows of the movement.

Thanks,
Vishwas

On Tue, Feb 28, 20= 12 at 1:31 PM, Michael Smith <mksmith@mac.com> wrote:

VM migration can be planned or unplanned, ei= ther manually or automatically. =A0=A0In the event of a host f= ailing that is configured for automatic migration, the VM=92s on the host w= ill migrate to another host as configured.=A0 There is some in= ternal intelligence that will allow for best placement of VM=92s, so they c= an move to any one of a number of available hosts.


Mike

=A0=

From:= arm= d-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Linda Dunbar<= /span>


Sent: Tuesday, February 28, 2012 1:17 PM
To: Vishwas Manral; Gary Berger
Cc:= armd@ietf.orgSubject: Re: [armd] how does "draft-sridharan-virtualization-= nvgre-00"advertise its external facing hosts' IP addresses to exte= rnal world?

<= /u>=A0

Gary,

=A0<= /p>

Can you elaborate why = =93vm migration may not be planned=94?

=A0

I thought that VM migration is orches= trated by some entity, like vMotion or VM managers.

=A0<= /p>

Linda

=A0<= /p>

From: Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Monday, February 27, 2012 6:49 PM
To: Gary BergerCc: Linda Dunbar; Murari Sridharan; armd@ietf.org
Subject: Re: [armd] how doe= s "draft-sridharan-virtualization-nvgre-00"advertise its external= facing hosts' IP addresses to external world?

=A0

Hi Gary,

I guess what Linda was t= rying to say is that the station moves autonomously, however there is some = entity moving the VM (which can in turn could move the state on the switch)= .

An example of this could be VEPA.

Thanks,
Vishwas

On Fri, Feb 24, 2012 at 2:22 PM, Gary Ber= ger <gaberger@ci= sco.com> wrote:

=A0

<= /u>=A0

Protocols have been developed for IP mobility between Home Gatewa= y and Remote Gateway. The question is if we want similar protocols among TO= Rs or among vSwitches. Handset mobility is random, but VM migration is plan= ned. =A0

<= /u>=A0

I wouldn't put such a restriction, vm migration may not be planned.. T= hey both degenerate into a multi-homing problem.

=A0

Linda

=A0=

From:= arm= d-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of Vishwas Manra= l
Sent: Thursday, September 22, 2011 9:38 PM
To: Murari Srid= haran
Cc: armd= @ietf.org
Subject: Re: [armd] how does "draft-sridharan-= virtualization-nvgre-00" advertise its external facing hosts' IP a= ddresses to external world?

=A0

Murari think IP mobility. :)

=A0

On Th= u, Sep 22, 2011 at 5:00 PM, Murari Sridharan <muraris@microsoft.com> wrote:

Do you have a scenario i= n mind?

From: Vishwas Man= ral
Sent: 9/22/2011 4:55 PM=


To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org
Subject: Re: [armd] how does "dr= aft-sridharan-virtualization-nvgre-00" advertise its external facing h= osts' IP addresses to external world?

Hi Murari,

=A0

= Yes that is what I mean.

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:50 PM, Murari Sridharan <muraris@microsoft.com> wrote= :

You mean not an Ethernet frame but some IP payload?

=A0

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
<= b>Sent: Thursday, September 22, 2011 4:49 PM
To: Murari Sridharan
Cc: Narasimhan Venkataramaiah; Linda = Dunbar; david.blac= k@emc.com; armd@ietf= .org


Subject: Re: [armd] how does &q= uot;draft-sridharan-virtualization-nvgre-00" advertise its external fa= cing hosts' IP addresses to external world?

=A0

Murari,

=A0

What I am saying is the inner= header should be allowed to be L3.

=A0

From the diagram you have that does not seem to be the case.= Am I missing it totally?

=A0

Thanks,

Vishwas

On Thu, Sep 22, 2011 at 4:43 PM, Murari Sridharan <muraris@microsoft.com> wrote= :

Vishwas, Thanks for the feedback we will definitely consider adding th= at. I am not sure what you mean by doing L3 instead of L2. We allow any arb= itrary virtual topology including L3.

=A0

Thanks

=A0

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
<= b>Sent: Thursday, September 22, 2011 4:19 PM

=A0

On Thu, Sep 22, 2= 011 at 4:11 PM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:<= /u>

The draft mentions exactly this as one use= of the reserved 8 bits in Section 4. An NVGRE endpoint could use the 8 bit= s to further distribute flows belonging to a particular TNI and the switche= s use all 32 bits to get entropy. One step further would be for the switche= s to get full entropy from the inner Ethernet frame. I take it that your co= mment would be to make it explicit in the draft. Right?

=A0

One=

=A0=A0 such example could be to use the upper 8 bits of th= e Key field to

=A0=A0 add flow b= ased entropy and tag all the packets from a flow with an entropy label.

=A0

Simha

=A0

=

From:= Vishwas Manral [mailto:vishwas.ietf@gmail.com]
Sent: Thursday, September 22, 2011 4:04 PM
To: Narasimhan = Venkataramaiah
Cc: Linda Dunbar; Murari Sridharan; david.black@emc.com; armd@ietf.org
Subject: Re: [armd] how does "draft-sridharan-virtualization-nv= gre-00" advertise its external facing hosts' IP addresses to exter= nal world?

=A0<= /u>

Hi Simha,

=A0

The m= ain (Standards Track)=A0change in your draft is the addition of TNI.=

=A0

A question I have is a TNI identifies a particular tenant an= d all flows from/to a tenant will be hashed to the same path (even with the= changes in switches to do hashing to use TNI).

=A0

Why do you not use the last 8 bits which you have kept as re= served for providing the randomization for hashing flows between same to/fr= om on different paths?

=A0

Thanks,

V= ishwas

On Sun, Sep 18, 2= 011 at 11:01 AM, Narasimhan Venkataramaiah <narave@microsoft.com> wrote:=

The easiest from the point of view of configuration = would be to route everything back through the enterprise - not necessarily = the optimal from the enterprise point of view. Are you referring to a scena= rio where the VMs subnet is split between the cloud and the enterprise? Oth= erwise I don't see the implication on virtualization as its no differen= t than getting the traffic routed to the enterprise in the first case.

Simha

________________________________________
From: armd-bounces@ietf.org [armd-bounces= @ietf.org] on behalf of Linda Dunbar [linda.dunbar@huawei.com]
Sent: Sunday, September 18, 2011 7:06 AM
To: Murari Sridharan; david.black@emc.com; = armd@ietf.org
Sub= ject: [armd] how does "draft-sridharan-virtualization-nvgre-00" a= dvertise its external facing hosts' IP addresses to external world?<= /u>


Hi Murari,

Thank you very much = for sharing the presentation.

One question:

For a host within= an Enterprise site which needs to communicate with external peers, the hos= t either uses public IP address which is visible to external peers or uses = private IP address which is translated to public address at the Enterprise = site's gateway.

When this host is moved to "Cloud data center", will the &quo= t;Cloud Data center" advertise this host address to external peers? Or= will all external peers go through enterprise's gateway to reach this = host which is no longer residing in the enterprise site?

Thanks, Linda

> -----Original Message-----
> From: armd-bounces@ietf.or= g [mailto:ar= md-bounces@ietf.org] On Behalf Of
> Murari Sridharan
> Sent: Saturday, September 17, 2011 3:02 PM> To: david.bl= ack@emc.com; armd@ie= tf.org
> Subject: Re: [armd] soliciting typical network designs for ARMD
>= ;
> FYI, here is a talk that I gave last week in relation to the nvgr= e
> draft below.
> http://channel9.msdn.com/Events/= BUILD/BUILD2011/SAC-442T
>
> Thanks
> Murari
>
> -----Original Message---= --
> From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf Of
> david.black@e= mc.com
> Sent: Friday, September 16, 2011 6:14 AM
> To: armd@ietf.org
> S= ubject: Re: [armd] soliciting typical network designs for ARMD
>
> And two more drafts on this topic:
>
> http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-00.txt=
> http://www.ietf.org/id/draft-sridharan-virtualiz= ation-nvgre-00.txt
>
> The edge switches could be the softw= are switches in hypervisors.
>
> Thanks,
> --David
>
>
> > -----Orig= inal Message-----
> > From: armd-bounces@ietf.org [mailto:armd-bounces@ietf.org] On Behalf=
> > Of Warren Kumari
> > Sent: Wednesday, August 31, 2011 3:= 16 PM
> > To: Vishwas Manral
> > Cc: armd@ietf.org
> > Subject: Re: = [armd] soliciting typical network designs for ARMD
> >
> >
> > On Aug 11, 2011, at 11:40 PM, Vishwas M= anral wrote:
> >
> > > Hi Linda/ Anoop,
> > &= gt;
> > > Here is the example of the design I was talking about= , as defined
> by google.
> >
> > Just a clarification -- s/as defi= ned by google/as described by
> someone
> > who happens to w= ork for google/
> >
> > W
> >
> > > = http://www.ietf.org/id/draft-wkumari-dcops-l3-vmmobility= -00.txt
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:50 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > >
> > > >>>>
> > > (though= I think if there was a standard way to map Multicast MAC to
> > &= gt; Multicast IP, they could
> > probably use such a standard mech= anisms).
> > > >>>>
> > >
> > > They ca= n do that, but then this imposes requirements on the
> > > equi= pment to be able to do multicast forwarding, and even if does,
> >= > because of pruning requirements the number of groups would be very > > > large. =A0The average data center switch probably won't = handle that
> > > many groups.
> > >
> > &= gt; On Tue, Aug 9, 2011 at 2:41 PM, Vishwas Manral
> <vishwas.ietf@gmail.com&= gt; wrote:
> > > Hi Anoop,
> > >
> > > From what I kn= ow they do not use Multicast GRE (I hear the extra 4
> > > byte= s in the GRE header is a
> > proprietery extension).
> > = >
> > > I think a directory based mechanism is what is used (though = I think
> > > if there was a standard way to
> > map M= ulticast MAC to Multicast IP, they could probably use such a
> standa= rd mechanisms).
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 2:03 PM, Anoop Ghanwani
> <anoop@alumni.duke.edu<= /a>> wrote:
> > > Hi Vishwas,
> > >
> > > How do they = get multicast through the network in that case?
> > > Are they = planning to use multicast GRE, or just use directory based
> > >= ; lookups and not worry about multicast applications for now?
> > >
> > > Anoop
> > >
> > > = On Tue, Aug 9, 2011 at 1:27 PM, Vishwas Manral
> <vishwas.ietf@gmail.com> = wrote:
> > > Hi Linda,
> > >
> > > The data packe= ts can be tunnelled at the ToR over say a GRE packet
> > > and = the core is a Layer-3 core
> > (except for the downstream ports). = So we could have encapsulation/
> > decapsulation of L2 over GRE at the ToR.
> > >
>= ; > > The very same thing can be done at the hypervisor layer too, in=
> > > which case the entire DC network
> > would look= like a Layer-3 flat network including the ToR to server
> > link and the hypervisor would do the tunneling.
> > >=
> > > I am not sure if you got the points above or not. I know= cloud OS
> > > companies that provide the service
> >= and have big announced customers.
> > >
> > > Thanks,
> > > Vishwas
> = > > On Tue, Aug 9, 2011 at 11:51 AM, Linda Dunbar <dunbar.ll@gmail.com>
&g= t; wrote:
> > > Vishwas,
> > >
> > > In my mind the = bullet 1) in the list refers to ToR switches
> > > downstream p= orts (facing servers)
> > running Layer 2 and ToR uplinks ports ru= n IP Layer 3.
> > >
> > > Have you seen data center networks with To= R switches downstream
> > > ports (i.e. facing servers) enablin= g
> > IP routing, even though the physical links are Ethernet?
> > > If yes, we should definitely include it in the ARMD draft.> > >
> > > Thanks,
> > > Linda
> = > > On Tue, Aug 9, 2011 at 12:58 PM, Vishwas Manral
> <vishwas.ietf@gmail.c= om> wrote:
> > > Hi Linda,
> > > I am unsure what you mean by thi= s, but:
> > > =A0 * layer 3 all the way to TOR (Top of Rack swi= tches), We can also
> > > have a heirarchical network, with the= core totally Layer-3 (and
> > > having seperate
> > routing), from the hosts still = in a large Layer-3 subnet. Another
> > aspect could be to have a t= otally
> > Layer-3 network.
> > >
> > > Th= e difference between them is the link between the servers and the
> ToR.
> > >
> > > Thanks,
> > > Vis= hwas
> > > On Tue, Aug 9, 2011 at 10:22 AM, Linda Dunbar <dunbar.ll@gmail.com<= /a>>
> wrote:
> > > During the 81st IETF ARMD WG discussion, it w= as suggested that it
> is
> > > necessary to document typ= ical
> > data center network designs so that address resolution sc= aling issues
> > can be properly described. Many data center operators have expres= sed
> that they can't openly reveal their detailed network design= s.
> > Therefore, we only want to document anonymous designs witho= ut too
> much
> > detail. During the journey of establishing ARMD, we = have come across
> the following typical data center network designs:=
> > > =A0 * layer 3 all the way to TOR (Top of Rack switches),=
> > > =A0 * large layer 2 with hundreds (or thousands) of ToRs bei= ng
> > > interconnected by Layer 2. This
> > design wi= ll have thousands of hosts under the L2/L3 boundary router
> > (s)=
> > > =A0 * CLOS design =A0with thousands of switches. This design= will have
> > > thousands of hosts under the
> > L2/L= 3 boundary router(s)
> > > We have heard that each of the desig= ns above has its own problems.
> > > ARMD problem statements might
> > need to document = DC problems under each typical design.
> > > Please send feedba= ck to us (either to the armd email list =A0or to
> the
> > &= gt; ARMD chair Benson & Linda) to
> > indicate if we have missed any typical Data Center network design= s.
> > >
> > > Your contribution can greatly accele= rate the progress of ARMD WG.
> > >
> > > Thank you= very much.
> > >
> > > Linda & Benson
> > >

=A0

=A0

= =A0

= =A0

_______________________________________________ armd maili= ng list armd@ietf.org = https://www.ietf.org/mailman/listinfo/armd

=A0

=

--f46d044473675ed9b504ba22bf17-- From gaberger@cisco.com Wed Feb 29 16:24:11 2012 Return-Path: X-Original-To: armd@ietfa.amsl.com Delivered-To: armd@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 08D9421E8019 for ; Wed, 29 Feb 2012 16:24:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -8.531 X-Spam-Level: X-Spam-Status: No, score=-8.531 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, RCVD_NUMERIC_HELO=2.067] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9HTUfEoYgnjz for ; Wed, 29 Feb 2012 16:24:08 -0800 (PST) Received: from rcdn-iport-6.cisco.com (rcdn-iport-6.cisco.com [173.37.86.77]) by ietfa.amsl.com (Postfix) with ESMTP id 6C7AD21F86B1 for ; Wed, 29 Feb 2012 16:24:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=gaberger@cisco.com; l=70732; q=dns/txt; s=iport; t=1330561447; x=1331771047; h=subject:references:content-transfer-encoding:from: in-reply-to:message-id:date:to:cc:mime-version; bh=slTte7DRIgUOyT7uu5dt9gc2+duNw1IWaNFbqGMnl7Q=; b=CyxdAcfRD4L5UNeVvLHeU5gC3RVtoL8cVpDvxjEZZHS8NzxH8Oz8sFoc ID5/UjUG/VDhS/hCbRwgPF9i+aBHPf4w0EzTnj3Wh8y0rwR0navlP2/fh vyZOKebi5VTwBYt5Kg3lmrzmOLB8ixNEn8pGj3iG+gO7IH9mVWnLCEv2F g=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: At0AAIvATk+tJXG8/2dsb2JhbABEgj+CaJx3iFoBh311AoEHgXoBAQEDAQEBAQ8BBwEISwQHBQcEAgEIEQQBAQECAhwBAgQDAgICHwYfCQgBAQQTCRIHh2IFC5pPAYxlkjGBL4docIJ1BQEFAQIDAgECAQQBAwEBAQUGAQQCAjmEcw0BEVEEAQUDBQEGAQMCBAEDAQkIAwGCFjNjBIhPjHCLGod1gTcH X-IronPort-AV: E=Sophos;i="4.73,506,1325462400"; d="scan'208,217";a="62830355" Received: from rcdn-core2-1.cisco.com ([173.37.113.188]) by rcdn-iport-6.cisco.com with ESMTP; 01 Mar 2012 00:24:06 +0000 Received: from xbh-rcd-301.cisco.com (xbh-rcd-301.cisco.com [72.163.63.8]) by rcdn-core2-1.cisco.com (8.14.3/8.14.3) with ESMTP id q210O5R1014041; Thu, 1 Mar 2012 00:24:05 GMT Received: from xmb-rcd-201.cisco.com ([72.163.62.208]) by xbh-rcd-301.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 29 Feb 2012 18:24:05 -0600 Received: from 72.163.62.205 ([72.163.62.205]) by XMB-RCD-201.cisco.com ([72.163.62.208]) with Microsoft Exchange Server HTTP-DAV ; Thu, 1 Mar 2012 00:24:04 +0000 References: <4A95BA014132FF49AE685FAB4B9F17F610CAB74A@dfweml503-mbx.china.huawei.com> <4A95BA014132FF49AE685FAB4B9F17F632E2495E@dfweml505-mbx> <151901ccf660$5175f500$f461df00$@mac.com> Content-Transfer-Encoding: 7bit From: "Gary Berger (gaberger)" thread-topic: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? thread-index: Acz3QZvlMWnR2vknRPOdfzcKYfc0hQ== Content-Type: multipart/alternative; boundary="Apple-Mail-7F5EEDB1-8D21-4608-A0A5-2F15B1B5D303"; charset="utf-8" In-Reply-To: Message-ID: <4159FD35-265B-4A76-A858-D6AE0884290A@cisco.com> Date: Wed, 29 Feb 2012 19:24:00 -0500 To: "Vishwas Manral" MIME-Version: 1.0 (1.0) X-OriginalArrivalTime: 01 Mar 2012 00:24:05.0167 (UTC) FILETIME=[9C277BF0:01CCF741] Cc: armd@ietf.org Subject: Re: [armd] how does "draft-sridharan-virtualization-nvgre-00"advertise its external facing hosts' IP addresses to external world? X-BeenThere: armd@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "Discussion of issues associated with large amount of virtual machines being introduced in data centers and virtual hosts introduced by Cloud Computing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2012 00:24:11 -0000 --Apple-Mail-7F5EEDB1-8D21-4608-A0A5-2F15B1B5D303 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset=utf-8 VGhlIGtleXdvcmQgaGVyZSB3YXMgInBsYW5uZWQiLiBJZiB0aGVyZSBpcyBhIGNhdGFzdHJvcGhp YyBmYWlsdXJlIHN1Y2ggYXMgcG93ZXIgbG9zcyBpdCBtYXkgYmUgYW5vdGhlciBlbnRpdHkgcmVz cG9uc2libGUgZm9yIHJlY292ZXJpbmcgc2VydmljZXMgaS5lIGhlYXJ0YmVhdA0KDQpZb3UgY2Fu IGFzc3VtZSB1bmRlciBjZXJ0YWluIHNpdHVhdGlvbnMgeW91ciBpbiBhIGNyaXRpY2FsIGZhaWx1 cmUgYW5kIGFyZSB3aWxsaW5nIHRvIGxvc2UgYXBwbGljYXRpb24gc3RhdGUuIE9mIGNvdXJzZSBh cyBwb2ludGVkIG91dCB5b3Ugd291bGQgbmVlZCB5b3VyIGFwcGxpY2F0aW9uIGRhdGEgcmVwbGlj YXRlZCBhcyB3ZWxsLiANCg0KVGhlIHJlcXVpcmVtZW50cyBoZXJlIHdvdWxkIGJlIGJhc2VkIG9u IGEgcmlnaWQgc2V0IG9mIGNvbnN0cmFpbnRzIHN1Y2ggYXMgaGFyZGNvZGVkIGFkZHJlc3NlcyBv ciBzcGVjaWZpYyBiaW5kaW5ncyB0aGF0IG5lZWQgdG8gYmUgbWFpbnRhaW5lZCB3aGljaCBpcyBu b3QgaWRlYWwsIGJ1dCB0aGVyZSBpcyBhIGxvbmctdGFpbCBvZiBwb29ybHkgZGVzaWduZWQgYXBw cyBvdXQgdGhlcmUuIA0KDQpJIHRoaW5rIHdlIGFyZSBqdXN0IHNpbXBseSBzYXlpbmcgdGhhdCBt dWx0aS1ob21pbmcgc2hvdWxkIGJlIGEgbmF0aXZlIGFyY2hpdGVjdHVyYWwgZWxlbWVudCByZWdh cmRsZXNzIHRvIHdoaWNoIG5ldHdvcmsgeW91IGF0dGFjaCB0by4gDQoNCkcNCg0KDQoNClNlbnQg ZnJvbSBteSBpUGhvbmUNCg0KT24gRmViIDI5LCAyMDEyLCBhdCA2OjMxIFBNLCAiVmlzaHdhcyBN YW5yYWwiIDx2aXNod2FzLmlldGZAZ21haWwuY29tPiB3cm90ZToNCg0KPiBIaSBNaWNoYWVsLA0K PiANCj4gUGxlYXNlIGZvcmdpdmUgbXkgaWdub3JhbmNlIGhlcmUuIEkgZ3Vlc3Mgd2hhdCB5b3Ug YXJlIHNheWluZyBpcyB0aGF0IHRoZSB0cmlnZ2VyIGNvdWxkIGJlIHNvbWUgYXN5bmNocm9ub3Vz IGV2ZW50IG5vdCBpbml0aWF0ZWQgYnkgdGhlIG9wZXJhdG9yLg0KPiANCj4gSG93ZXZlciBpc24n dCB0aGUgbWFuYWdlbWVudCBlbnRpdHkgKG5vdCBtYW51YWwgaW50ZXJ2ZW50aW9uKSBpbnZvbHZl ZCBpbiBhY3R1YWxseSBtb3ZpbmcgdGhlIFZNIGFuZCBhbnkgc3RvcmFnZS8gc3RhdGUgYXNzb2Np YXRlZCB3aXRoIGl0PyBJZiBpdCBpcyB0aGUgbWFuYWdlbWVudCBlbnRpdHkga25vd3Mgb2YgdGhl IG1vdmVtZW50Lg0KPiANCj4gVGhhbmtzLA0KPiBWaXNod2FzDQo+IA0KPiBPbiBUdWUsIEZlYiAy OCwgMjAxMiBhdCAxOjMxIFBNLCBNaWNoYWVsIFNtaXRoIDxta3NtaXRoQG1hYy5jb20+IHdyb3Rl Og0KPiBWTSBtaWdyYXRpb24gY2FuIGJlIHBsYW5uZWQgb3IgdW5wbGFubmVkLCBlaXRoZXIgbWFu dWFsbHkgb3IgYXV0b21hdGljYWxseS4gICBJbiB0aGUgZXZlbnQgb2YgYSBob3N0IGZhaWxpbmcg dGhhdCBpcyBjb25maWd1cmVkIGZvciBhdXRvbWF0aWMgbWlncmF0aW9uLCB0aGUgVk3igJlzIG9u IHRoZSBob3N0IHdpbGwgbWlncmF0ZSB0byBhbm90aGVyIGhvc3QgYXMgY29uZmlndXJlZC4gIFRo ZXJlIGlzIHNvbWUgaW50ZXJuYWwgaW50ZWxsaWdlbmNlIHRoYXQgd2lsbCBhbGxvdyBmb3IgYmVz dCBwbGFjZW1lbnQgb2YgVk3igJlzLCBzbyB0aGV5IGNhbiBtb3ZlIHRvIGFueSBvbmUgb2YgYSBu dW1iZXIgb2YgYXZhaWxhYmxlIGhvc3RzLg0KPiANCj4gDQo+IE1pa2UNCj4gDQo+ICANCj4gDQo+ IEZyb206IGFybWQtYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOmFybWQtYm91bmNlc0BpZXRmLm9y Z10gT24gQmVoYWxmIE9mIExpbmRhIER1bmJhcg0KPiANCj4gDQo+IFNlbnQ6IFR1ZXNkYXksIEZl YnJ1YXJ5IDI4LCAyMDEyIDE6MTcgUE0NCj4gVG86IFZpc2h3YXMgTWFucmFsOyBHYXJ5IEJlcmdl cg0KPiBDYzogYXJtZEBpZXRmLm9yZw0KPiBTdWJqZWN0OiBSZTogW2FybWRdIGhvdyBkb2VzICJk cmFmdC1zcmlkaGFyYW4tdmlydHVhbGl6YXRpb24tbnZncmUtMDAiYWR2ZXJ0aXNlIGl0cyBleHRl cm5hbCBmYWNpbmcgaG9zdHMnIElQIGFkZHJlc3NlcyB0byBleHRlcm5hbCB3b3JsZD8NCj4gIA0K PiANCj4gR2FyeSwNCj4gDQo+ICANCj4gDQo+IENhbiB5b3UgZWxhYm9yYXRlIHdoeSDigJx2bSBt aWdyYXRpb24gbWF5IG5vdCBiZSBwbGFubmVk4oCdPw0KPiANCj4gIA0KPiANCj4gSSB0aG91Z2h0 IHRoYXQgVk0gbWlncmF0aW9uIGlzIG9yY2hlc3RyYXRlZCBieSBzb21lIGVudGl0eSwgbGlrZSB2 TW90aW9uIG9yIFZNIG1hbmFnZXJzLg0KPiANCj4gIA0KPiANCj4gTGluZGENCj4gDQo+ICANCj4g DQo+IEZyb206IFZpc2h3YXMgTWFucmFsIFttYWlsdG86dmlzaHdhcy5pZXRmQGdtYWlsLmNvbV0g DQo+IFNlbnQ6IE1vbmRheSwgRmVicnVhcnkgMjcsIDIwMTIgNjo0OSBQTQ0KPiBUbzogR2FyeSBC ZXJnZXINCj4gQ2M6IExpbmRhIER1bmJhcjsgTXVyYXJpIFNyaWRoYXJhbjsgYXJtZEBpZXRmLm9y Zw0KPiBTdWJqZWN0OiBSZTogW2FybWRdIGhvdyBkb2VzICJkcmFmdC1zcmlkaGFyYW4tdmlydHVh bGl6YXRpb24tbnZncmUtMDAiYWR2ZXJ0aXNlIGl0cyBleHRlcm5hbCBmYWNpbmcgaG9zdHMnIElQ IGFkZHJlc3NlcyB0byBleHRlcm5hbCB3b3JsZD8NCj4gDQo+ICANCj4gDQo+IEhpIEdhcnksDQo+ IA0KPiBJIGd1ZXNzIHdoYXQgTGluZGEgd2FzIHRyeWluZyB0byBzYXkgaXMgdGhhdCB0aGUgc3Rh dGlvbiBtb3ZlcyBhdXRvbm9tb3VzbHksIGhvd2V2ZXIgdGhlcmUgaXMgc29tZSBlbnRpdHkgbW92 aW5nIHRoZSBWTSAod2hpY2ggY2FuIGluIHR1cm4gY291bGQgbW92ZSB0aGUgc3RhdGUgb24gdGhl IHN3aXRjaCkuIA0KPiANCj4gQW4gZXhhbXBsZSBvZiB0aGlzIGNvdWxkIGJlIFZFUEEuDQo+IA0K PiBUaGFua3MsDQo+IFZpc2h3YXMNCj4gDQo+IE9uIEZyaSwgRmViIDI0LCAyMDEyIGF0IDI6MjIg UE0sIEdhcnkgQmVyZ2VyIDxnYWJlcmdlckBjaXNjby5jb20+IHdyb3RlOg0KPiANCj4gIA0KPiAN Cj4gIA0KPiANCj4gUHJvdG9jb2xzIGhhdmUgYmVlbiBkZXZlbG9wZWQgZm9yIElQIG1vYmlsaXR5 IGJldHdlZW4gSG9tZSBHYXRld2F5IGFuZCBSZW1vdGUgR2F0ZXdheS4gVGhlIHF1ZXN0aW9uIGlz IGlmIHdlIHdhbnQgc2ltaWxhciBwcm90b2NvbHMgYW1vbmcgVE9ScyBvciBhbW9uZyB2U3dpdGNo ZXMuIEhhbmRzZXQgbW9iaWxpdHkgaXMgcmFuZG9tLCBidXQgVk0gbWlncmF0aW9uIGlzIHBsYW5u ZWQuICANCj4gDQo+ICANCj4gDQo+IEkgd291bGRuJ3QgcHV0IHN1Y2ggYSByZXN0cmljdGlvbiwg dm0gbWlncmF0aW9uIG1heSBub3QgYmUgcGxhbm5lZC4uIFRoZXkgYm90aCBkZWdlbmVyYXRlIGlu dG8gYSBtdWx0aS1ob21pbmcgcHJvYmxlbS4NCj4gDQo+ICANCj4gDQo+IExpbmRhDQo+IA0KPiAg DQo+IA0KPiBGcm9tOiBhcm1kLWJvdW5jZXNAaWV0Zi5vcmcgW21haWx0bzphcm1kLWJvdW5jZXNA aWV0Zi5vcmddIE9uIEJlaGFsZiBPZiBWaXNod2FzIE1hbnJhbA0KPiBTZW50OiBUaHVyc2RheSwg U2VwdGVtYmVyIDIyLCAyMDExIDk6MzggUE0NCj4gVG86IE11cmFyaSBTcmlkaGFyYW4NCj4gQ2M6 IGFybWRAaWV0Zi5vcmcNCj4gU3ViamVjdDogUmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3Jp ZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZh Y2luZyBob3N0cycgSVAgYWRkcmVzc2VzIHRvIGV4dGVybmFsIHdvcmxkPw0KPiANCj4gIA0KPiAN Cj4gTXVyYXJpIHRoaW5rIElQIG1vYmlsaXR5LiA6KQ0KPiANCj4gIA0KPiANCj4gT24gVGh1LCBT ZXAgMjIsIDIwMTEgYXQgNTowMCBQTSwgTXVyYXJpIFNyaWRoYXJhbiA8bXVyYXJpc0BtaWNyb3Nv ZnQuY29tPiB3cm90ZToNCj4gDQo+IERvIHlvdSBoYXZlIGEgc2NlbmFyaW8gaW4gbWluZD8NCj4g DQo+IEZyb206IFZpc2h3YXMgTWFucmFsDQo+IFNlbnQ6IDkvMjIvMjAxMSA0OjU1IFBNDQo+IA0K PiANCj4gVG86IE11cmFyaSBTcmlkaGFyYW4NCj4gQ2M6IE5hcmFzaW1oYW4gVmVua2F0YXJhbWFp YWg7IExpbmRhIER1bmJhcjsgZGF2aWQuYmxhY2tAZW1jLmNvbTsgYXJtZEBpZXRmLm9yZw0KPiBT dWJqZWN0OiBSZTogW2FybWRdIGhvdyBkb2VzICJkcmFmdC1zcmlkaGFyYW4tdmlydHVhbGl6YXRp b24tbnZncmUtMDAiIGFkdmVydGlzZSBpdHMgZXh0ZXJuYWwgZmFjaW5nIGhvc3RzJyBJUCBhZGRy ZXNzZXMgdG8gZXh0ZXJuYWwgd29ybGQ/DQo+IA0KPiBIaSBNdXJhcmksDQo+IA0KPiAgDQo+IA0K PiBZZXMgdGhhdCBpcyB3aGF0IEkgbWVhbi4NCj4gDQo+ICANCj4gDQo+IFRoYW5rcywNCj4gDQo+ IFZpc2h3YXMNCj4gDQo+IE9uIFRodSwgU2VwIDIyLCAyMDExIGF0IDQ6NTAgUE0sIE11cmFyaSBT cmlkaGFyYW4gPG11cmFyaXNAbWljcm9zb2Z0LmNvbT4gd3JvdGU6DQo+IA0KPiBZb3UgbWVhbiBu b3QgYW4gRXRoZXJuZXQgZnJhbWUgYnV0IHNvbWUgSVAgcGF5bG9hZD8NCj4gDQo+ICANCj4gDQo+ IEZyb206IFZpc2h3YXMgTWFucmFsIFttYWlsdG86dmlzaHdhcy5pZXRmQGdtYWlsLmNvbV0gDQo+ IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjIsIDIwMTEgNDo0OSBQTQ0KPiBUbzogTXVyYXJp IFNyaWRoYXJhbg0KPiBDYzogTmFyYXNpbWhhbiBWZW5rYXRhcmFtYWlhaDsgTGluZGEgRHVuYmFy OyBkYXZpZC5ibGFja0BlbWMuY29tOyBhcm1kQGlldGYub3JnDQo+IA0KPiANCj4gU3ViamVjdDog UmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3Jl LTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZhY2luZyBob3N0cycgSVAgYWRkcmVzc2VzIHRv IGV4dGVybmFsIHdvcmxkPw0KPiANCj4gIA0KPiANCj4gTXVyYXJpLA0KPiANCj4gIA0KPiANCj4g V2hhdCBJIGFtIHNheWluZyBpcyB0aGUgaW5uZXIgaGVhZGVyIHNob3VsZCBiZSBhbGxvd2VkIHRv IGJlIEwzLg0KPiANCj4gIA0KPiANCj4gRnJvbSB0aGUgZGlhZ3JhbSB5b3UgaGF2ZSB0aGF0IGRv ZXMgbm90IHNlZW0gdG8gYmUgdGhlIGNhc2UuIEFtIEkgbWlzc2luZyBpdCB0b3RhbGx5Pw0KPiAN Cj4gIA0KPiANCj4gVGhhbmtzLA0KPiANCj4gVmlzaHdhcw0KPiANCj4gT24gVGh1LCBTZXAgMjIs IDIwMTEgYXQgNDo0MyBQTSwgTXVyYXJpIFNyaWRoYXJhbiA8bXVyYXJpc0BtaWNyb3NvZnQuY29t PiB3cm90ZToNCj4gDQo+IFZpc2h3YXMsIFRoYW5rcyBmb3IgdGhlIGZlZWRiYWNrIHdlIHdpbGwg ZGVmaW5pdGVseSBjb25zaWRlciBhZGRpbmcgdGhhdC4gSSBhbSBub3Qgc3VyZSB3aGF0IHlvdSBt ZWFuIGJ5IGRvaW5nIEwzIGluc3RlYWQgb2YgTDIuIFdlIGFsbG93IGFueSBhcmJpdHJhcnkgdmly dHVhbCB0b3BvbG9neSBpbmNsdWRpbmcgTDMuDQo+IA0KPiAgDQo+IA0KPiBUaGFua3MNCj4gDQo+ ICANCj4gDQo+IEZyb206IFZpc2h3YXMgTWFucmFsIFttYWlsdG86dmlzaHdhcy5pZXRmQGdtYWls LmNvbV0gDQo+IFNlbnQ6IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjIsIDIwMTEgNDoxOSBQTQ0KPiAN Cj4gDQo+IFRvOiBOYXJhc2ltaGFuIFZlbmthdGFyYW1haWFoDQo+IENjOiBMaW5kYSBEdW5iYXI7 IE11cmFyaSBTcmlkaGFyYW47IGRhdmlkLmJsYWNrQGVtYy5jb207IGFybWRAaWV0Zi5vcmcNCj4g U3ViamVjdDogUmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxpemF0 aW9uLW52Z3JlLTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZhY2luZyBob3N0cycgSVAgYWRk cmVzc2VzIHRvIGV4dGVybmFsIHdvcmxkPw0KPiANCj4gIA0KPiANCj4gSGkgU2ltaGEsDQo+IA0K PiAgDQo+IA0KPiBJIHNlZSB0aGlzIGFzIHRoZSBvbmx5IGRpZmZlcmVuY2UgYmV0d2VlbiBWWExB TiBhbmQgdGhlIE5WR1JFIHNvbHV0aW9uIChiZXNpZGVzIG9mY291cnNlIHRoYXQgVE5JIG5lZWRz IHRvIGJlIHBhcnNlZCBpbiB0aGUgaW50ZXJtZWRpYXRlIGRldmljZSBmb3IgaGFzaGluZyBhbmQg dXNpbmcgbGVzc2VyIG51bWJlciBvZiBieXRlcykuDQo+IA0KPiAgDQo+IA0KPiBJIHdvdWxkIHRo aW5rIHlvdSBzaG91bGQgYWRkIGl0IHRvIHlvdXIgZHJhZnQgaW1tZWRpYXRlbHkuIFdpdGggdHVu bmVsaW5nIHlvdSBjb25zb2xpZGF0ZSB0aGUgYWRkcmVzc2VzIHZpc2libGUgdG8gdGhlIGNvcmUg YW5kIGJ5IHByb3ZpZGluZyBhIGhhc2ggbWVjaGFuaXNtLCB5b3UgYXJlIHByb3ZpZGluZyBzb21l IGxldmVsIG9mIHJhbmRvbW5lc3MuDQo+IA0KPiAgDQo+IA0KPiBUaGUgb3RoZXIgdGhpbmcgeW91 IHNob3VsZCBsb29rIGF0IGlzIEwzIChJUHY0LyBJUHY2KSBvdmVyIE5WR1JFIGluc3RlYWQgb2Yg TDIgYWxvbmUuIEkgZ3Vlc3MgaXQgd291bGQgYmUgdGhlIHNhbWUgY29tbWVudCBmb3IgdGhlIFZY TEFOIHByb3Bvc2FsIHRvby4NCj4gDQo+ICANCj4gDQo+IFRoYW5rcywNCj4gDQo+IFZpc2h3YXMN Cj4gDQo+IE9uIFRodSwgU2VwIDIyLCAyMDExIGF0IDQ6MTEgUE0sIE5hcmFzaW1oYW4gVmVua2F0 YXJhbWFpYWggPG5hcmF2ZUBtaWNyb3NvZnQuY29tPiB3cm90ZToNCj4gDQo+IFRoZSBkcmFmdCBt ZW50aW9ucyBleGFjdGx5IHRoaXMgYXMgb25lIHVzZSBvZiB0aGUgcmVzZXJ2ZWQgOCBiaXRzIGlu IFNlY3Rpb24gNC4gQW4gTlZHUkUgZW5kcG9pbnQgY291bGQgdXNlIHRoZSA4IGJpdHMgdG8gZnVy dGhlciBkaXN0cmlidXRlIGZsb3dzIGJlbG9uZ2luZyB0byBhIHBhcnRpY3VsYXIgVE5JIGFuZCB0 aGUgc3dpdGNoZXMgdXNlIGFsbCAzMiBiaXRzIHRvIGdldCBlbnRyb3B5LiBPbmUgc3RlcCBmdXJ0 aGVyIHdvdWxkIGJlIGZvciB0aGUgc3dpdGNoZXMgdG8gZ2V0IGZ1bGwgZW50cm9weSBmcm9tIHRo ZSBpbm5lciBFdGhlcm5ldCBmcmFtZS4gSSB0YWtlIGl0IHRoYXQgeW91ciBjb21tZW50IHdvdWxk IGJlIHRvIG1ha2UgaXQgZXhwbGljaXQgaW4gdGhlIGRyYWZ0LiBSaWdodD8NCj4gDQo+ICANCj4g DQo+IE9uZQ0KPiANCj4gICAgc3VjaCBleGFtcGxlIGNvdWxkIGJlIHRvIHVzZSB0aGUgdXBwZXIg OCBiaXRzIG9mIHRoZSBLZXkgZmllbGQgdG8NCj4gDQo+ICAgIGFkZCBmbG93IGJhc2VkIGVudHJv cHkgYW5kIHRhZyBhbGwgdGhlIHBhY2tldHMgZnJvbSBhIGZsb3cgd2l0aCBhbiBlbnRyb3B5IGxh YmVsLg0KPiANCj4gIA0KPiANCj4gU2ltaGENCj4gDQo+ICANCj4gDQo+IEZyb206IFZpc2h3YXMg TWFucmFsIFttYWlsdG86dmlzaHdhcy5pZXRmQGdtYWlsLmNvbV0gDQo+IFNlbnQ6IFRodXJzZGF5 LCBTZXB0ZW1iZXIgMjIsIDIwMTEgNDowNCBQTQ0KPiBUbzogTmFyYXNpbWhhbiBWZW5rYXRhcmFt YWlhaA0KPiBDYzogTGluZGEgRHVuYmFyOyBNdXJhcmkgU3JpZGhhcmFuOyBkYXZpZC5ibGFja0Bl bWMuY29tOyBhcm1kQGlldGYub3JnDQo+IFN1YmplY3Q6IFJlOiBbYXJtZF0gaG93IGRvZXMgImRy YWZ0LXNyaWRoYXJhbi12aXJ0dWFsaXphdGlvbi1udmdyZS0wMCIgYWR2ZXJ0aXNlIGl0cyBleHRl cm5hbCBmYWNpbmcgaG9zdHMnIElQIGFkZHJlc3NlcyB0byBleHRlcm5hbCB3b3JsZD8NCj4gDQo+ ICANCj4gDQo+IEhpIFNpbWhhLA0KPiANCj4gIA0KPiANCj4gVGhlIG1haW4gKFN0YW5kYXJkcyBU cmFjaykgY2hhbmdlIGluIHlvdXIgZHJhZnQgaXMgdGhlIGFkZGl0aW9uIG9mIFROSS4NCj4gDQo+ ICANCj4gDQo+IEEgcXVlc3Rpb24gSSBoYXZlIGlzIGEgVE5JIGlkZW50aWZpZXMgYSBwYXJ0aWN1 bGFyIHRlbmFudCBhbmQgYWxsIGZsb3dzIGZyb20vdG8gYSB0ZW5hbnQgd2lsbCBiZSBoYXNoZWQg dG8gdGhlIHNhbWUgcGF0aCAoZXZlbiB3aXRoIHRoZSBjaGFuZ2VzIGluIHN3aXRjaGVzIHRvIGRv IGhhc2hpbmcgdG8gdXNlIFROSSkuDQo+IA0KPiAgDQo+IA0KPiBXaHkgZG8geW91IG5vdCB1c2Ug dGhlIGxhc3QgOCBiaXRzIHdoaWNoIHlvdSBoYXZlIGtlcHQgYXMgcmVzZXJ2ZWQgZm9yIHByb3Zp ZGluZyB0aGUgcmFuZG9taXphdGlvbiBmb3IgaGFzaGluZyBmbG93cyBiZXR3ZWVuIHNhbWUgdG8v ZnJvbSBvbiBkaWZmZXJlbnQgcGF0aHM/DQo+IA0KPiAgDQo+IA0KPiBUaGFua3MsDQo+IA0KPiBW aXNod2FzDQo+IA0KPiBPbiBTdW4sIFNlcCAxOCwgMjAxMSBhdCAxMTowMSBBTSwgTmFyYXNpbWhh biBWZW5rYXRhcmFtYWlhaCA8bmFyYXZlQG1pY3Jvc29mdC5jb20+IHdyb3RlOg0KPiANCj4gVGhl IGVhc2llc3QgZnJvbSB0aGUgcG9pbnQgb2YgdmlldyBvZiBjb25maWd1cmF0aW9uIHdvdWxkIGJl IHRvIHJvdXRlIGV2ZXJ5dGhpbmcgYmFjayB0aHJvdWdoIHRoZSBlbnRlcnByaXNlIC0gbm90IG5l Y2Vzc2FyaWx5IHRoZSBvcHRpbWFsIGZyb20gdGhlIGVudGVycHJpc2UgcG9pbnQgb2Ygdmlldy4g QXJlIHlvdSByZWZlcnJpbmcgdG8gYSBzY2VuYXJpbyB3aGVyZSB0aGUgVk1zIHN1Ym5ldCBpcyBz cGxpdCBiZXR3ZWVuIHRoZSBjbG91ZCBhbmQgdGhlIGVudGVycHJpc2U/IE90aGVyd2lzZSBJIGRv bid0IHNlZSB0aGUgaW1wbGljYXRpb24gb24gdmlydHVhbGl6YXRpb24gYXMgaXRzIG5vIGRpZmZl cmVudCB0aGFuIGdldHRpbmcgdGhlIHRyYWZmaWMgcm91dGVkIHRvIHRoZSBlbnRlcnByaXNlIGlu IHRoZSBmaXJzdCBjYXNlLg0KPiANCj4gU2ltaGENCj4gDQo+IF9fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX18NCj4gRnJvbTogYXJtZC1ib3VuY2VzQGlldGYub3JnIFthcm1k LWJvdW5jZXNAaWV0Zi5vcmddIG9uIGJlaGFsZiBvZiBMaW5kYSBEdW5iYXIgW2xpbmRhLmR1bmJh ckBodWF3ZWkuY29tXQ0KPiBTZW50OiBTdW5kYXksIFNlcHRlbWJlciAxOCwgMjAxMSA3OjA2IEFN DQo+IFRvOiBNdXJhcmkgU3JpZGhhcmFuOyBkYXZpZC5ibGFja0BlbWMuY29tOyBhcm1kQGlldGYu b3JnDQo+IFN1YmplY3Q6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxp emF0aW9uLW52Z3JlLTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZhY2luZyBob3N0cycgSVAg YWRkcmVzc2VzIHRvIGV4dGVybmFsIHdvcmxkPw0KPiANCj4gDQo+IEhpIE11cmFyaSwNCj4gDQo+ IFRoYW5rIHlvdSB2ZXJ5IG11Y2ggZm9yIHNoYXJpbmcgdGhlIHByZXNlbnRhdGlvbi4NCj4gDQo+ IE9uZSBxdWVzdGlvbjoNCj4gDQo+IEZvciBhIGhvc3Qgd2l0aGluIGFuIEVudGVycHJpc2Ugc2l0 ZSB3aGljaCBuZWVkcyB0byBjb21tdW5pY2F0ZSB3aXRoIGV4dGVybmFsIHBlZXJzLCB0aGUgaG9z dCBlaXRoZXIgdXNlcyBwdWJsaWMgSVAgYWRkcmVzcyB3aGljaCBpcyB2aXNpYmxlIHRvIGV4dGVy bmFsIHBlZXJzIG9yIHVzZXMgcHJpdmF0ZSBJUCBhZGRyZXNzIHdoaWNoIGlzIHRyYW5zbGF0ZWQg dG8gcHVibGljIGFkZHJlc3MgYXQgdGhlIEVudGVycHJpc2Ugc2l0ZSdzIGdhdGV3YXkuDQo+IA0K PiBXaGVuIHRoaXMgaG9zdCBpcyBtb3ZlZCB0byAiQ2xvdWQgZGF0YSBjZW50ZXIiLCB3aWxsIHRo ZSAiQ2xvdWQgRGF0YSBjZW50ZXIiIGFkdmVydGlzZSB0aGlzIGhvc3QgYWRkcmVzcyB0byBleHRl cm5hbCBwZWVycz8gT3Igd2lsbCBhbGwgZXh0ZXJuYWwgcGVlcnMgZ28gdGhyb3VnaCBlbnRlcnBy aXNlJ3MgZ2F0ZXdheSB0byByZWFjaCB0aGlzIGhvc3Qgd2hpY2ggaXMgbm8gbG9uZ2VyIHJlc2lk aW5nIGluIHRoZSBlbnRlcnByaXNlIHNpdGU/DQo+IA0KPiBUaGFua3MsIExpbmRhDQo+IA0KPiA+ IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gRnJvbTogYXJtZC1ib3VuY2VzQGlldGYu b3JnIFttYWlsdG86YXJtZC1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YNCj4gPiBNdXJh cmkgU3JpZGhhcmFuDQo+ID4gU2VudDogU2F0dXJkYXksIFNlcHRlbWJlciAxNywgMjAxMSAzOjAy IFBNDQo+ID4gVG86IGRhdmlkLmJsYWNrQGVtYy5jb207IGFybWRAaWV0Zi5vcmcNCj4gPiBTdWJq ZWN0OiBSZTogW2FybWRdIHNvbGljaXRpbmcgdHlwaWNhbCBuZXR3b3JrIGRlc2lnbnMgZm9yIEFS TUQNCj4gPg0KPiA+IEZZSSwgaGVyZSBpcyBhIHRhbGsgdGhhdCBJIGdhdmUgbGFzdCB3ZWVrIGlu IHJlbGF0aW9uIHRvIHRoZSBudmdyZQ0KPiA+IGRyYWZ0IGJlbG93Lg0KPiA+IGh0dHA6Ly9jaGFu bmVsOS5tc2RuLmNvbS9FdmVudHMvQlVJTEQvQlVJTEQyMDExL1NBQy00NDJUDQo+ID4NCj4gPiBU aGFua3MNCj4gPiBNdXJhcmkNCj4gPg0KPiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ ID4gRnJvbTogYXJtZC1ib3VuY2VzQGlldGYub3JnIFttYWlsdG86YXJtZC1ib3VuY2VzQGlldGYu b3JnXSBPbiBCZWhhbGYgT2YNCj4gPiBkYXZpZC5ibGFja0BlbWMuY29tDQo+ID4gU2VudDogRnJp ZGF5LCBTZXB0ZW1iZXIgMTYsIDIwMTEgNjoxNCBBTQ0KPiA+IFRvOiBhcm1kQGlldGYub3JnDQo+ ID4gU3ViamVjdDogUmU6IFthcm1kXSBzb2xpY2l0aW5nIHR5cGljYWwgbmV0d29yayBkZXNpZ25z IGZvciBBUk1EDQo+ID4NCj4gPiBBbmQgdHdvIG1vcmUgZHJhZnRzIG9uIHRoaXMgdG9waWM6DQo+ ID4NCj4gPiBodHRwOi8vd3d3LmlldGYub3JnL2lkL2RyYWZ0LW1haGFsaW5nYW0tZHV0dC1kY29w cy12eGxhbi0wMC50eHQNCj4gPiBodHRwOi8vd3d3LmlldGYub3JnL2lkL2RyYWZ0LXNyaWRoYXJh bi12aXJ0dWFsaXphdGlvbi1udmdyZS0wMC50eHQNCj4gPg0KPiA+IFRoZSBlZGdlIHN3aXRjaGVz IGNvdWxkIGJlIHRoZSBzb2Z0d2FyZSBzd2l0Y2hlcyBpbiBoeXBlcnZpc29ycy4NCj4gPg0KPiA+ IFRoYW5rcywNCj4gPiAtLURhdmlkDQo+ID4NCj4gPg0KPiA+ID4gLS0tLS1PcmlnaW5hbCBNZXNz YWdlLS0tLS0NCj4gPiA+IEZyb206IGFybWQtYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOmFybWQt Ym91bmNlc0BpZXRmLm9yZ10gT24gQmVoYWxmDQo+ID4gPiBPZiBXYXJyZW4gS3VtYXJpDQo+ID4g PiBTZW50OiBXZWRuZXNkYXksIEF1Z3VzdCAzMSwgMjAxMSAzOjE2IFBNDQo+ID4gPiBUbzogVmlz aHdhcyBNYW5yYWwNCj4gPiA+IENjOiBhcm1kQGlldGYub3JnDQo+ID4gPiBTdWJqZWN0OiBSZTog W2FybWRdIHNvbGljaXRpbmcgdHlwaWNhbCBuZXR3b3JrIGRlc2lnbnMgZm9yIEFSTUQNCj4gPiA+ DQo+ID4gPg0KPiA+ID4gT24gQXVnIDExLCAyMDExLCBhdCAxMTo0MCBQTSwgVmlzaHdhcyBNYW5y YWwgd3JvdGU6DQo+ID4gPg0KPiA+ID4gPiBIaSBMaW5kYS8gQW5vb3AsDQo+ID4gPiA+DQo+ID4g PiA+IEhlcmUgaXMgdGhlIGV4YW1wbGUgb2YgdGhlIGRlc2lnbiBJIHdhcyB0YWxraW5nIGFib3V0 LCBhcyBkZWZpbmVkDQo+ID4gYnkgZ29vZ2xlLg0KPiA+ID4NCj4gPiA+IEp1c3QgYSBjbGFyaWZp Y2F0aW9uIC0tIHMvYXMgZGVmaW5lZCBieSBnb29nbGUvYXMgZGVzY3JpYmVkIGJ5DQo+ID4gc29t ZW9uZQ0KPiA+ID4gd2hvIGhhcHBlbnMgdG8gd29yayBmb3IgZ29vZ2xlLw0KPiA+ID4NCj4gPiA+ IFcNCj4gPiA+DQo+ID4gPiA+IGh0dHA6Ly93d3cuaWV0Zi5vcmcvaWQvZHJhZnQtd2t1bWFyaS1k Y29wcy1sMy12bW1vYmlsaXR5LTAwLnR4dA0KPiA+ID4gPg0KPiA+ID4gPiBUaGFua3MsDQo+ID4g PiA+IFZpc2h3YXMNCj4gPiA+ID4gT24gVHVlLCBBdWcgOSwgMjAxMSBhdCAyOjUwIFBNLCBBbm9v cCBHaGFud2FuaQ0KPiA+IDxhbm9vcEBhbHVtbmkuZHVrZS5lZHU+IHdyb3RlOg0KPiA+ID4gPg0K PiA+ID4gPiA+Pj4+DQo+ID4gPiA+ICh0aG91Z2ggSSB0aGluayBpZiB0aGVyZSB3YXMgYSBzdGFu ZGFyZCB3YXkgdG8gbWFwIE11bHRpY2FzdCBNQUMgdG8NCj4gPiA+ID4gTXVsdGljYXN0IElQLCB0 aGV5IGNvdWxkDQo+ID4gPiBwcm9iYWJseSB1c2Ugc3VjaCBhIHN0YW5kYXJkIG1lY2hhbmlzbXMp Lg0KPiA+ID4gPiA+Pj4+DQo+ID4gPiA+DQo+ID4gPiA+IFRoZXkgY2FuIGRvIHRoYXQsIGJ1dCB0 aGVuIHRoaXMgaW1wb3NlcyByZXF1aXJlbWVudHMgb24gdGhlDQo+ID4gPiA+IGVxdWlwbWVudCB0 byBiZSBhYmxlIHRvIGRvIG11bHRpY2FzdCBmb3J3YXJkaW5nLCBhbmQgZXZlbiBpZiBkb2VzLA0K PiA+ID4gPiBiZWNhdXNlIG9mIHBydW5pbmcgcmVxdWlyZW1lbnRzIHRoZSBudW1iZXIgb2YgZ3Jv dXBzIHdvdWxkIGJlIHZlcnkNCj4gPiA+ID4gbGFyZ2UuICBUaGUgYXZlcmFnZSBkYXRhIGNlbnRl ciBzd2l0Y2ggcHJvYmFibHkgd29uJ3QgaGFuZGxlIHRoYXQNCj4gPiA+ID4gbWFueSBncm91cHMu DQo+ID4gPiA+DQo+ID4gPiA+IE9uIFR1ZSwgQXVnIDksIDIwMTEgYXQgMjo0MSBQTSwgVmlzaHdh cyBNYW5yYWwNCj4gPiA8dmlzaHdhcy5pZXRmQGdtYWlsLmNvbT4gd3JvdGU6DQo+ID4gPiA+IEhp IEFub29wLA0KPiA+ID4gPg0KPiA+ID4gPiBGcm9tIHdoYXQgSSBrbm93IHRoZXkgZG8gbm90IHVz ZSBNdWx0aWNhc3QgR1JFIChJIGhlYXIgdGhlIGV4dHJhIDQNCj4gPiA+ID4gYnl0ZXMgaW4gdGhl IEdSRSBoZWFkZXIgaXMgYQ0KPiA+ID4gcHJvcHJpZXRlcnkgZXh0ZW5zaW9uKS4NCj4gPiA+ID4N Cj4gPiA+ID4gSSB0aGluayBhIGRpcmVjdG9yeSBiYXNlZCBtZWNoYW5pc20gaXMgd2hhdCBpcyB1 c2VkICh0aG91Z2ggSSB0aGluaw0KPiA+ID4gPiBpZiB0aGVyZSB3YXMgYSBzdGFuZGFyZCB3YXkg dG8NCj4gPiA+IG1hcCBNdWx0aWNhc3QgTUFDIHRvIE11bHRpY2FzdCBJUCwgdGhleSBjb3VsZCBw cm9iYWJseSB1c2Ugc3VjaCBhDQo+ID4gc3RhbmRhcmQgbWVjaGFuaXNtcykuDQo+ID4gPiA+DQo+ ID4gPiA+IFRoYW5rcywNCj4gPiA+ID4gVmlzaHdhcw0KPiA+ID4gPiBPbiBUdWUsIEF1ZyA5LCAy MDExIGF0IDI6MDMgUE0sIEFub29wIEdoYW53YW5pDQo+ID4gPGFub29wQGFsdW1uaS5kdWtlLmVk dT4gd3JvdGU6DQo+ID4gPiA+IEhpIFZpc2h3YXMsDQo+ID4gPiA+DQo+ID4gPiA+IEhvdyBkbyB0 aGV5IGdldCBtdWx0aWNhc3QgdGhyb3VnaCB0aGUgbmV0d29yayBpbiB0aGF0IGNhc2U/DQo+ID4g PiA+IEFyZSB0aGV5IHBsYW5uaW5nIHRvIHVzZSBtdWx0aWNhc3QgR1JFLCBvciBqdXN0IHVzZSBk aXJlY3RvcnkgYmFzZWQNCj4gPiA+ID4gbG9va3VwcyBhbmQgbm90IHdvcnJ5IGFib3V0IG11bHRp Y2FzdCBhcHBsaWNhdGlvbnMgZm9yIG5vdz8NCj4gPiA+ID4NCj4gPiA+ID4gQW5vb3ANCj4gPiA+ ID4NCj4gPiA+ID4gT24gVHVlLCBBdWcgOSwgMjAxMSBhdCAxOjI3IFBNLCBWaXNod2FzIE1hbnJh bA0KPiA+IDx2aXNod2FzLmlldGZAZ21haWwuY29tPiB3cm90ZToNCj4gPiA+ID4gSGkgTGluZGEs DQo+ID4gPiA+DQo+ID4gPiA+IFRoZSBkYXRhIHBhY2tldHMgY2FuIGJlIHR1bm5lbGxlZCBhdCB0 aGUgVG9SIG92ZXIgc2F5IGEgR1JFIHBhY2tldA0KPiA+ID4gPiBhbmQgdGhlIGNvcmUgaXMgYSBM YXllci0zIGNvcmUNCj4gPiA+IChleGNlcHQgZm9yIHRoZSBkb3duc3RyZWFtIHBvcnRzKS4gU28g d2UgY291bGQgaGF2ZSBlbmNhcHN1bGF0aW9uLw0KPiA+ID4gZGVjYXBzdWxhdGlvbiBvZiBMMiBv dmVyIEdSRSBhdCB0aGUgVG9SLg0KPiA+ID4gPg0KPiA+ID4gPiBUaGUgdmVyeSBzYW1lIHRoaW5n IGNhbiBiZSBkb25lIGF0IHRoZSBoeXBlcnZpc29yIGxheWVyIHRvbywgaW4NCj4gPiA+ID4gd2hp Y2ggY2FzZSB0aGUgZW50aXJlIERDIG5ldHdvcmsNCj4gPiA+IHdvdWxkIGxvb2sgbGlrZSBhIExh eWVyLTMgZmxhdCBuZXR3b3JrIGluY2x1ZGluZyB0aGUgVG9SIHRvIHNlcnZlcg0KPiA+ID4gbGlu ayBhbmQgdGhlIGh5cGVydmlzb3Igd291bGQgZG8gdGhlIHR1bm5lbGluZy4NCj4gPiA+ID4NCj4g PiA+ID4gSSBhbSBub3Qgc3VyZSBpZiB5b3UgZ290IHRoZSBwb2ludHMgYWJvdmUgb3Igbm90LiBJ IGtub3cgY2xvdWQgT1MNCj4gPiA+ID4gY29tcGFuaWVzIHRoYXQgcHJvdmlkZSB0aGUgc2Vydmlj ZQ0KPiA+ID4gYW5kIGhhdmUgYmlnIGFubm91bmNlZCBjdXN0b21lcnMuDQo+ID4gPiA+DQo+ID4g PiA+IFRoYW5rcywNCj4gPiA+ID4gVmlzaHdhcw0KPiA+ID4gPiBPbiBUdWUsIEF1ZyA5LCAyMDEx IGF0IDExOjUxIEFNLCBMaW5kYSBEdW5iYXIgPGR1bmJhci5sbEBnbWFpbC5jb20+DQo+ID4gd3Jv dGU6DQo+ID4gPiA+IFZpc2h3YXMsDQo+ID4gPiA+DQo+ID4gPiA+IEluIG15IG1pbmQgdGhlIGJ1 bGxldCAxKSBpbiB0aGUgbGlzdCByZWZlcnMgdG8gVG9SIHN3aXRjaGVzDQo+ID4gPiA+IGRvd25z dHJlYW0gcG9ydHMgKGZhY2luZyBzZXJ2ZXJzKQ0KPiA+ID4gcnVubmluZyBMYXllciAyIGFuZCBU b1IgdXBsaW5rcyBwb3J0cyBydW4gSVAgTGF5ZXIgMy4NCj4gPiA+ID4NCj4gPiA+ID4gSGF2ZSB5 b3Ugc2VlbiBkYXRhIGNlbnRlciBuZXR3b3JrcyB3aXRoIFRvUiBzd2l0Y2hlcyBkb3duc3RyZWFt DQo+ID4gPiA+IHBvcnRzIChpLmUuIGZhY2luZyBzZXJ2ZXJzKSBlbmFibGluZw0KPiA+ID4gSVAg cm91dGluZywgZXZlbiB0aG91Z2ggdGhlIHBoeXNpY2FsIGxpbmtzIGFyZSBFdGhlcm5ldD8NCj4g PiA+ID4gSWYgeWVzLCB3ZSBzaG91bGQgZGVmaW5pdGVseSBpbmNsdWRlIGl0IGluIHRoZSBBUk1E IGRyYWZ0Lg0KPiA+ID4gPg0KPiA+ID4gPiBUaGFua3MsDQo+ID4gPiA+IExpbmRhDQo+ID4gPiA+ IE9uIFR1ZSwgQXVnIDksIDIwMTEgYXQgMTI6NTggUE0sIFZpc2h3YXMgTWFucmFsDQo+ID4gPHZp c2h3YXMuaWV0ZkBnbWFpbC5jb20+IHdyb3RlOg0KPiA+ID4gPiBIaSBMaW5kYSwNCj4gPiA+ID4g SSBhbSB1bnN1cmUgd2hhdCB5b3UgbWVhbiBieSB0aGlzLCBidXQ6DQo+ID4gPiA+ICAgKiBsYXll ciAzIGFsbCB0aGUgd2F5IHRvIFRPUiAoVG9wIG9mIFJhY2sgc3dpdGNoZXMpLCBXZSBjYW4gYWxz bw0KPiA+ID4gPiBoYXZlIGEgaGVpcmFyY2hpY2FsIG5ldHdvcmssIHdpdGggdGhlIGNvcmUgdG90 YWxseSBMYXllci0zIChhbmQNCj4gPiA+ID4gaGF2aW5nIHNlcGVyYXRlDQo+ID4gPiByb3V0aW5n KSwgZnJvbSB0aGUgaG9zdHMgc3RpbGwgaW4gYSBsYXJnZSBMYXllci0zIHN1Ym5ldC4gQW5vdGhl cg0KPiA+ID4gYXNwZWN0IGNvdWxkIGJlIHRvIGhhdmUgYSB0b3RhbGx5DQo+ID4gPiBMYXllci0z IG5ldHdvcmsuDQo+ID4gPiA+DQo+ID4gPiA+IFRoZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlbSBp cyB0aGUgbGluayBiZXR3ZWVuIHRoZSBzZXJ2ZXJzIGFuZCB0aGUNCj4gPiBUb1IuDQo+ID4gPiA+ DQo+ID4gPiA+IFRoYW5rcywNCj4gPiA+ID4gVmlzaHdhcw0KPiA+ID4gPiBPbiBUdWUsIEF1ZyA5 LCAyMDExIGF0IDEwOjIyIEFNLCBMaW5kYSBEdW5iYXIgPGR1bmJhci5sbEBnbWFpbC5jb20+DQo+ ID4gd3JvdGU6DQo+ID4gPiA+IER1cmluZyB0aGUgODFzdCBJRVRGIEFSTUQgV0cgZGlzY3Vzc2lv biwgaXQgd2FzIHN1Z2dlc3RlZCB0aGF0IGl0DQo+ID4gaXMNCj4gPiA+ID4gbmVjZXNzYXJ5IHRv IGRvY3VtZW50IHR5cGljYWwNCj4gPiA+IGRhdGEgY2VudGVyIG5ldHdvcmsgZGVzaWducyBzbyB0 aGF0IGFkZHJlc3MgcmVzb2x1dGlvbiBzY2FsaW5nIGlzc3Vlcw0KPiA+ID4gY2FuIGJlIHByb3Bl cmx5IGRlc2NyaWJlZC4gTWFueSBkYXRhIGNlbnRlciBvcGVyYXRvcnMgaGF2ZSBleHByZXNzZWQN Cj4gPiB0aGF0IHRoZXkgY2FuJ3Qgb3Blbmx5IHJldmVhbCB0aGVpciBkZXRhaWxlZCBuZXR3b3Jr IGRlc2lnbnMuDQo+ID4gPiBUaGVyZWZvcmUsIHdlIG9ubHkgd2FudCB0byBkb2N1bWVudCBhbm9u eW1vdXMgZGVzaWducyB3aXRob3V0IHRvbw0KPiA+IG11Y2gNCj4gPiA+IGRldGFpbC4gRHVyaW5n IHRoZSBqb3VybmV5IG9mIGVzdGFibGlzaGluZyBBUk1ELCB3ZSBoYXZlIGNvbWUgYWNyb3NzDQo+ ID4gdGhlIGZvbGxvd2luZyB0eXBpY2FsIGRhdGEgY2VudGVyIG5ldHdvcmsgZGVzaWduczoNCj4g PiA+ID4gICAqIGxheWVyIDMgYWxsIHRoZSB3YXkgdG8gVE9SIChUb3Agb2YgUmFjayBzd2l0Y2hl cyksDQo+ID4gPiA+ICAgKiBsYXJnZSBsYXllciAyIHdpdGggaHVuZHJlZHMgKG9yIHRob3VzYW5k cykgb2YgVG9ScyBiZWluZw0KPiA+ID4gPiBpbnRlcmNvbm5lY3RlZCBieSBMYXllciAyLiBUaGlz DQo+ID4gPiBkZXNpZ24gd2lsbCBoYXZlIHRob3VzYW5kcyBvZiBob3N0cyB1bmRlciB0aGUgTDIv TDMgYm91bmRhcnkgcm91dGVyDQo+ID4gPiAocykNCj4gPiA+ID4gICAqIENMT1MgZGVzaWduICB3 aXRoIHRob3VzYW5kcyBvZiBzd2l0Y2hlcy4gVGhpcyBkZXNpZ24gd2lsbCBoYXZlDQo+ID4gPiA+ IHRob3VzYW5kcyBvZiBob3N0cyB1bmRlciB0aGUNCj4gPiA+IEwyL0wzIGJvdW5kYXJ5IHJvdXRl cihzKQ0KPiA+ID4gPiBXZSBoYXZlIGhlYXJkIHRoYXQgZWFjaCBvZiB0aGUgZGVzaWducyBhYm92 ZSBoYXMgaXRzIG93biBwcm9ibGVtcy4NCj4gPiA+ID4gQVJNRCBwcm9ibGVtIHN0YXRlbWVudHMg bWlnaHQNCj4gPiA+IG5lZWQgdG8gZG9jdW1lbnQgREMgcHJvYmxlbXMgdW5kZXIgZWFjaCB0eXBp Y2FsIGRlc2lnbi4NCj4gPiA+ID4gUGxlYXNlIHNlbmQgZmVlZGJhY2sgdG8gdXMgKGVpdGhlciB0 byB0aGUgYXJtZCBlbWFpbCBsaXN0ICBvciB0bw0KPiA+IHRoZQ0KPiA+ID4gPiBBUk1EIGNoYWly IEJlbnNvbiAmIExpbmRhKSB0bw0KPiA+ID4gaW5kaWNhdGUgaWYgd2UgaGF2ZSBtaXNzZWQgYW55 IHR5cGljYWwgRGF0YSBDZW50ZXIgbmV0d29yayBkZXNpZ25zLg0KPiA+ID4gPg0KPiA+ID4gPiBZ b3VyIGNvbnRyaWJ1dGlvbiBjYW4gZ3JlYXRseSBhY2NlbGVyYXRlIHRoZSBwcm9ncmVzcyBvZiBB Uk1EIFdHLg0KPiA+ID4gPg0KPiA+ID4gPiBUaGFuayB5b3UgdmVyeSBtdWNoLg0KPiA+ID4gPg0K PiA+ID4gPiBMaW5kYSAmIEJlbnNvbg0KPiA+ID4gPg0KPiANCj4gIA0KPiANCj4gIA0KPiANCj4g IA0KPiANCj4gIA0KPiANCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18gYXJtZCBtYWlsaW5nIGxpc3QgYXJtZEBpZXRmLm9yZyBodHRwczovL3d3dy5pZXRm Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FybWQNCj4gDQo+ICANCj4gDQo+IA0K --Apple-Mail-7F5EEDB1-8D21-4608-A0A5-2F15B1B5D303 Content-Transfer-Encoding: base64 Content-Type: text/html; charset=utf-8 PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5IGJnY29sb3I9IiNGRkZGRkYiPjxkaXY+VGhlIGtleXdv cmQgaGVyZSB3YXMgInBsYW5uZWQiLiBJZiB0aGVyZSBpcyBhIGNhdGFzdHJvcGhpYyBmYWlsdXJl IHN1Y2ggYXMgcG93ZXIgbG9zcyBpdCBtYXkgYmUgYW5vdGhlciBlbnRpdHkgcmVzcG9uc2libGUg Zm9yIHJlY292ZXJpbmcgc2VydmljZXMgaS5lIGhlYXJ0YmVhdDwvZGl2PjxkaXY+PGJyPjwvZGl2 PjxkaXY+PHNwYW4gY2xhc3M9IkFwcGxlLXN0eWxlLXNwYW4iIHN0eWxlPSItd2Via2l0LXRhcC1o aWdobGlnaHQtY29sb3I6IHJnYmEoMjYsIDI2LCAyNiwgMC4yOTI5NjkpOyAtd2Via2l0LWNvbXBv c2l0aW9uLWZpbGwtY29sb3I6IHJnYmEoMTc1LCAxOTIsIDIyNywgMC4yMzA0NjkpOyAtd2Via2l0 LWNvbXBvc2l0aW9uLWZyYW1lLWNvbG9yOiByZ2JhKDc3LCAxMjgsIDE4MCwgMC4yMzA0NjkpOyAi PllvdSBjYW4gYXNzdW1lIHVuZGVyIGNlcnRhaW4gc2l0dWF0aW9ucyB5b3VyIGluIGEgY3JpdGlj YWwgZmFpbHVyZSBhbmQgYXJlIHdpbGxpbmcgdG8gbG9zZSBhcHBsaWNhdGlvbiBzdGF0ZS4gT2Yg Y291cnNlIGFzIHBvaW50ZWQgb3V0IHlvdSB3b3VsZCBuZWVkIHlvdXIgYXBwbGljYXRpb24gZGF0 YSByZXBsaWNhdGVkIGFzIHdlbGwuJm5ic3A7PC9zcGFuPjxicj48L2Rpdj48ZGl2PjxzcGFuIGNs YXNzPSJBcHBsZS1zdHlsZS1zcGFuIiBzdHlsZT0iLXdlYmtpdC10YXAtaGlnaGxpZ2h0LWNvbG9y OiByZ2JhKDI2LCAyNiwgMjYsIDAuMjkyOTY5KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1maWxsLWNv bG9yOiByZ2JhKDE3NSwgMTkyLCAyMjcsIDAuMjMwNDY5KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1m cmFtZS1jb2xvcjogcmdiYSg3NywgMTI4LCAxODAsIDAuMjMwNDY5KTsgIj48YnI+PC9zcGFuPjwv ZGl2PjxkaXY+PHNwYW4gY2xhc3M9IkFwcGxlLXN0eWxlLXNwYW4iIHN0eWxlPSItd2Via2l0LXRh cC1oaWdobGlnaHQtY29sb3I6IHJnYmEoMjYsIDI2LCAyNiwgMC4yOTI5NjkpOyAtd2Via2l0LWNv bXBvc2l0aW9uLWZpbGwtY29sb3I6IHJnYmEoMTc1LCAxOTIsIDIyNywgMC4yMzA0NjkpOyAtd2Vi a2l0LWNvbXBvc2l0aW9uLWZyYW1lLWNvbG9yOiByZ2JhKDc3LCAxMjgsIDE4MCwgMC4yMzA0Njkp OyAiPlRoZSByZXF1aXJlbWVudHMgaGVyZSB3b3VsZCBiZSBiYXNlZCBvbiBhIHJpZ2lkIHNldCBv ZiBjb25zdHJhaW50cyBzdWNoIGFzIGhhcmRjb2RlZCBhZGRyZXNzZXMgb3Igc3BlY2lmaWMgYmlu ZGluZ3MgdGhhdCBuZWVkIHRvIGJlIG1haW50YWluZWQgd2hpY2ggaXMgbm90IGlkZWFsLCBidXQg dGhlcmUgaXMgYSBsb25nLXRhaWwgb2YgcG9vcmx5IGRlc2lnbmVkIGFwcHMgb3V0IHRoZXJlLiZu YnNwOzwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuIGNsYXNzPSJBcHBsZS1zdHlsZS1zcGFuIiBzdHls ZT0iLXdlYmtpdC10YXAtaGlnaGxpZ2h0LWNvbG9yOiByZ2JhKDI2LCAyNiwgMjYsIDAuMjkyOTY5 KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1maWxsLWNvbG9yOiByZ2JhKDE3NSwgMTkyLCAyMjcsIDAu MjMwNDY5KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1mcmFtZS1jb2xvcjogcmdiYSg3NywgMTI4LCAx ODAsIDAuMjMwNDY5KTsgIj48YnI+PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4gY2xhc3M9IkFwcGxl LXN0eWxlLXNwYW4iIHN0eWxlPSItd2Via2l0LXRhcC1oaWdobGlnaHQtY29sb3I6IHJnYmEoMjYs IDI2LCAyNiwgMC4yOTI5NjkpOyAtd2Via2l0LWNvbXBvc2l0aW9uLWZpbGwtY29sb3I6IHJnYmEo MTc1LCAxOTIsIDIyNywgMC4yMzA0NjkpOyAtd2Via2l0LWNvbXBvc2l0aW9uLWZyYW1lLWNvbG9y OiByZ2JhKDc3LCAxMjgsIDE4MCwgMC4yMzA0NjkpOyI+SSB0aGluayB3ZSBhcmUganVzdCBzaW1w bHkgc2F5aW5nIHRoYXQgbXVsdGktaG9taW5nIHNob3VsZCBiZSBhIG5hdGl2ZSBhcmNoaXRlY3R1 cmFsIGVsZW1lbnQgcmVnYXJkbGVzcyB0byB3aGljaCBuZXR3b3JrIHlvdSBhdHRhY2ggdG8uJm5i c3A7PC9zcGFuPjwvZGl2PjxkaXY+PHNwYW4gY2xhc3M9IkFwcGxlLXN0eWxlLXNwYW4iIHN0eWxl PSItd2Via2l0LXRhcC1oaWdobGlnaHQtY29sb3I6IHJnYmEoMjYsIDI2LCAyNiwgMC4yOTI5Njkp OyAtd2Via2l0LWNvbXBvc2l0aW9uLWZpbGwtY29sb3I6IHJnYmEoMTc1LCAxOTIsIDIyNywgMC4y MzA0NjkpOyAtd2Via2l0LWNvbXBvc2l0aW9uLWZyYW1lLWNvbG9yOiByZ2JhKDc3LCAxMjgsIDE4 MCwgMC4yMzA0NjkpOyI+PGJyPjwvc3Bhbj48L2Rpdj48ZGl2PjxzcGFuIGNsYXNzPSJBcHBsZS1z dHlsZS1zcGFuIiBzdHlsZT0iLXdlYmtpdC10YXAtaGlnaGxpZ2h0LWNvbG9yOiByZ2JhKDI2LCAy NiwgMjYsIDAuMjkyOTY5KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1maWxsLWNvbG9yOiByZ2JhKDE3 NSwgMTkyLCAyMjcsIDAuMjMwNDY5KTsgLXdlYmtpdC1jb21wb3NpdGlvbi1mcmFtZS1jb2xvcjog cmdiYSg3NywgMTI4LCAxODAsIDAuMjMwNDY5KTsiPkc8L3NwYW4+PC9kaXY+PGRpdj48c3BhbiBj bGFzcz0iQXBwbGUtc3R5bGUtc3BhbiIgc3R5bGU9Ii13ZWJraXQtdGFwLWhpZ2hsaWdodC1jb2xv cjogcmdiYSgyNiwgMjYsIDI2LCAwLjI5Mjk2OSk7IC13ZWJraXQtY29tcG9zaXRpb24tZmlsbC1j b2xvcjogcmdiYSgxNzUsIDE5MiwgMjI3LCAwLjIzMDQ2OSk7IC13ZWJraXQtY29tcG9zaXRpb24t ZnJhbWUtY29sb3I6IHJnYmEoNzcsIDEyOCwgMTgwLCAwLjIzMDQ2OSk7Ij48YnI+PC9zcGFuPjwv ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGJyPlNlbnQgZnJvbSBteSBpUGhvbmU8L2Rpdj48ZGl2 Pjxicj5PbiBGZWIgMjksIDIwMTIsIGF0IDY6MzEgUE0sICJWaXNod2FzIE1hbnJhbCIgJmx0Ozxh IGhyZWY9Im1haWx0bzp2aXNod2FzLmlldGZAZ21haWwuY29tIj52aXNod2FzLmlldGZAZ21haWwu Y29tPC9hPiZndDsgd3JvdGU6PGJyPjxicj48L2Rpdj48ZGl2PjwvZGl2PjxibG9ja3F1b3RlIHR5 cGU9ImNpdGUiPjxkaXY+SGkgTWljaGFlbCw8YnI+PGJyPlBsZWFzZSBmb3JnaXZlIG15IGlnbm9y YW5jZSBoZXJlLiBJIGd1ZXNzIHdoYXQgeW91IGFyZSBzYXlpbmcgaXMgdGhhdCB0aGUgdHJpZ2dl ciBjb3VsZCBiZSBzb21lIGFzeW5jaHJvbm91cyBldmVudCBub3QgaW5pdGlhdGVkIGJ5IHRoZSBv cGVyYXRvci48YnI+PGJyPkhvd2V2ZXIgaXNuJ3QgdGhlIG1hbmFnZW1lbnQgZW50aXR5IChub3Qg bWFudWFsIGludGVydmVudGlvbikgaW52b2x2ZWQgaW4gYWN0dWFsbHkgbW92aW5nIHRoZSBWTSBh bmQgYW55IHN0b3JhZ2UvIHN0YXRlIGFzc29jaWF0ZWQgd2l0aCBpdD8gSWYgaXQgaXMgdGhlIG1h bmFnZW1lbnQgZW50aXR5IGtub3dzIG9mIHRoZSBtb3ZlbWVudC48YnI+DQo8YnI+VGhhbmtzLDxi cj5WaXNod2FzPGJyPjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVHVlLCBGZWIgMjgs IDIwMTIgYXQgMTozMSBQTSwgTWljaGFlbCBTbWl0aCA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhy ZWY9Im1haWx0bzpta3NtaXRoQG1hYy5jb20iPm1rc21pdGhAbWFjLmNvbTwvYT4mZ3Q7PC9zcGFu PiB3cm90ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2lu OjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+ DQo8ZGl2IGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiIGxhbmc9IkVOLVVTIj48ZGl2PjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5 OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMWY0OTdk Ij5WTSBtaWdyYXRpb24gY2FuIGJlIHBsYW5uZWQgb3IgdW5wbGFubmVkLCBlaXRoZXIgbWFudWFs bHkgb3IgYXV0b21hdGljYWxseS4gPHNwYW4+Jm5ic3A7Jm5ic3A7PC9zcGFuPkluIHRoZSBldmVu dCBvZiBhIGhvc3QgZmFpbGluZyB0aGF0IGlzIGNvbmZpZ3VyZWQgZm9yIGF1dG9tYXRpYyBtaWdy YXRpb24sIHRoZSBWTeKAmXMgb24gdGhlIGhvc3Qgd2lsbCBtaWdyYXRlIHRvIGFub3RoZXIgaG9z dCBhcyBjb25maWd1cmVkLjxzcGFuPiZuYnNwOyA8L3NwYW4+VGhlcmUgaXMgc29tZSBpbnRlcm5h bCBpbnRlbGxpZ2VuY2UgdGhhdCB3aWxsIGFsbG93IGZvciBiZXN0IHBsYWNlbWVudCBvZiBWTeKA mXMsIHNvIHRoZXkgY2FuIG1vdmUgdG8gYW55IG9uZSBvZiBhIG51bWJlciBvZiBhdmFpbGFibGUg aG9zdHMuPHU+PC91Pjx1PjwvdT48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90 OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxZjQ5N2QiPjxicj5NaWtlPHU+PC91Pjx1 PjwvdT48L3NwYW4+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJp ZiZxdW90Oztjb2xvcjojMWY0OTdkIj48dT48L3U+Jm5ic3A7PHU+PC91Pjwvc3Bhbj48L3A+DQo8 ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRp bmc6MGluIDBpbiAwaW4gNC4wcHQiPjxkaXY+PGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVy LXRvcDpzb2xpZCAjYjVjNGRmIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7Ij5Gcm9tOjwvc3Bh bj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFo b21hJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsiPiA8YSBocmVmPSJtYWlsdG86YXJtZC1i b3VuY2VzQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGlldGYub3JnPC9h PiBbbWFpbHRvOjxhIGhyZWY9Im1haWx0bzphcm1kLWJvdW5jZXNAaWV0Zi5vcmciIHRhcmdldD0i X2JsYW5rIj5hcm1kLWJvdW5jZXNAaWV0Zi5vcmc8L2E+XSA8Yj5PbiBCZWhhbGYgT2YgPC9iPkxp bmRhIER1bmJhcjwvc3Bhbj48L3A+DQo8ZGl2IGNsYXNzPSJpbSI+PGJyPjxiPlNlbnQ6PC9iPiBU dWVzZGF5LCBGZWJydWFyeSAyOCwgMjAxMiAxOjE3IFBNPGJyPjxiPlRvOjwvYj4gVmlzaHdhcyBN YW5yYWw7IEdhcnkgQmVyZ2VyPGJyPjwvZGl2PjxkaXY+PGRpdiBjbGFzcz0iaDUiPjxiPkNjOjwv Yj4gPGEgaHJlZj0ibWFpbHRvOmFybWRAaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5hcm1kQGll dGYub3JnPC9hPjxicj48Yj5TdWJqZWN0OjwvYj4gUmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQt c3JpZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwImFkdmVydGlzZSBpdHMgZXh0ZXJuYWwg ZmFjaW5nIGhvc3RzJyBJUCBhZGRyZXNzZXMgdG8gZXh0ZXJuYWwgd29ybGQ/PHU+PC91Pjx1Pjwv dT48L2Rpdj4NCjwvZGl2PjxwPjwvcD48L2Rpdj48L2Rpdj48ZGl2PjxkaXYgY2xhc3M9Img1Ij48 cCBjbGFzcz0iTXNvTm9ybWFsIj48dT48L3U+Jm5ic3A7PHU+PC91PjwvcD48cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtD YWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFmNDk3ZCI+R2FyeSwg PHU+PC91Pjx1PjwvdT48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVv dDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxZjQ5N2QiPjx1PjwvdT4mbmJzcDs8dT48L3U+PC9z cGFuPjwvcD48cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7 Y29sb3I6IzFmNDk3ZCI+Q2FuIHlvdSBlbGFib3JhdGUgd2h5IOKAnDwvc3Bhbj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjguNXB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7 c2Fucy1zZXJpZiZxdW90OyI+dm0gbWlncmF0aW9uIG1heSBub3QgYmUgcGxhbm5lZOKAnT88dT48 L3U+PHU+PC91Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0i Zm9udC1zaXplOjguNXB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fu cy1zZXJpZiZxdW90OyI+PHU+PC91PiZuYnNwOzx1PjwvdT48L3NwYW4+PC9wPjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMWY0OTdkIj5JIHRo b3VnaHQgdGhhdCBWTSBtaWdyYXRpb24gaXMgb3JjaGVzdHJhdGVkIGJ5IHNvbWUgZW50aXR5LCBs aWtlIHZNb3Rpb24gb3IgVk0gbWFuYWdlcnMuIDx1PjwvdT48dT48L3U+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMWY0 OTdkIj48dT48L3U+Jm5ic3A7PHU+PC91Pjwvc3Bhbj48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZx dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxZjQ5N2QiPkxpbmRhIDx1PjwvdT48 dT48L3U+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1z ZXJpZiZxdW90Oztjb2xvcjojMWY0OTdkIj48dT48L3U+Jm5ic3A7PHU+PC91Pjwvc3Bhbj48L3A+ PGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgYmx1ZSAxLjVwdDtwYWRk aW5nOjBpbiAwaW4gMGluIDQuMHB0Ij4NCjxkaXY+PGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9y ZGVyLXRvcDpzb2xpZCAjYjVjNGRmIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQt ZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7Ij5Gcm9tOjwv c3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7 VGFob21hJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsiPiBWaXNod2FzIE1hbnJhbCA8YSBo cmVmPSJtYWlsdG86W21haWx0bzp2aXNod2FzLmlldGZAZ21haWwuY29tXSIgdGFyZ2V0PSJfYmxh bmsiPlttYWlsdG86dmlzaHdhcy5pZXRmQGdtYWlsLmNvbV08L2E+IDxicj4NCjxiPlNlbnQ6PC9i PiBNb25kYXksIEZlYnJ1YXJ5IDI3LCAyMDEyIDY6NDkgUE08YnI+PGI+VG86PC9iPiBHYXJ5IEJl cmdlcjxicj48Yj5DYzo8L2I+IExpbmRhIER1bmJhcjsgTXVyYXJpIFNyaWRoYXJhbjsgPGEgaHJl Zj0ibWFpbHRvOmFybWRAaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5hcm1kQGlldGYub3JnPC9h Pjxicj48Yj5TdWJqZWN0OjwvYj4gUmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFu LXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwImFkdmVydGlzZSBpdHMgZXh0ZXJuYWwgZmFjaW5nIGhv c3RzJyBJUCBhZGRyZXNzZXMgdG8gZXh0ZXJuYWwgd29ybGQ/PHU+PC91Pjx1PjwvdT48L3NwYW4+ PC9wPg0KPC9kaXY+PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PiZuYnNwOzx1Pjwv dT48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij5I aSBHYXJ5LDxicj48YnI+SSBndWVzcyB3aGF0IExpbmRhIHdhcyB0cnlpbmcgdG8gc2F5IGlzIHRo YXQgdGhlIHN0YXRpb24gbW92ZXMgYXV0b25vbW91c2x5LCBob3dldmVyIHRoZXJlIGlzIHNvbWUg ZW50aXR5IG1vdmluZyB0aGUgVk0gKHdoaWNoIGNhbiBpbiB0dXJuIGNvdWxkIG1vdmUgdGhlIHN0 YXRlIG9uIHRoZSBzd2l0Y2gpLiA8YnI+DQo8YnI+QW4gZXhhbXBsZSBvZiB0aGlzIGNvdWxkIGJl IFZFUEEuPGJyPjxicj5UaGFua3MsPGJyPlZpc2h3YXM8dT48L3U+PHU+PC91PjwvcD48ZGl2Pjxw IGNsYXNzPSJNc29Ob3JtYWwiPk9uIEZyaSwgRmViIDI0LCAyMDEyIGF0IDI6MjIgUE0sIEdhcnkg QmVyZ2VyICZsdDs8YSBocmVmPSJtYWlsdG86Z2FiZXJnZXJAY2lzY28uY29tIiB0YXJnZXQ9Il9i bGFuayI+Z2FiZXJnZXJAY2lzY28uY29tPC9hPiZndDsgd3JvdGU6PHU+PC91Pjx1PjwvdT48L3A+ DQo8ZGl2PjxkaXY+PGRpdj48ZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZTo4LjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90 O3NhbnMtc2VyaWYmcXVvdDsiPjx1PjwvdT4mbmJzcDs8dT48L3U+PC9zcGFuPjwvcD48L2Rpdj48 L2Rpdj48L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6OC41cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlm JnF1b3Q7Ij48dT48L3U+Jm5ic3A7PHU+PC91Pjwvc3Bhbj48L3A+DQo8L2Rpdj48YmxvY2txdW90 ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI2I1YzRkZiAzLjBwdDtwYWRk aW5nOjBpbiAwaW4gMGluIDMuMHB0O21hcmdpbi1sZWZ0OjMuMHB0O21hcmdpbi10b3A6NS4wcHQ7 bWFyZ2luLXJpZ2h0OjBpbjttYXJnaW4tYm90dG9tOjUuMHB0Ij48ZGl2PjxkaXY+PGRpdj48cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWls eTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFmNDk3 ZCI+UHJvdG9jb2xzIGhhdmUgYmVlbiBkZXZlbG9wZWQgZm9yIElQIG1vYmlsaXR5IGJldHdlZW4g SG9tZSBHYXRld2F5IGFuZCBSZW1vdGUgR2F0ZXdheS4gVGhlIHF1ZXN0aW9uIGlzIGlmIHdlIHdh bnQgc2ltaWxhciBwcm90b2NvbHMgYW1vbmcgVE9ScyBvciBhbW9uZyB2U3dpdGNoZXMuIEhhbmRz ZXQgbW9iaWxpdHkgaXMgcmFuZG9tLCBidXQgVk0gbWlncmF0aW9uIGlzIHBsYW5uZWQuICZuYnNw Ozwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90 ZT48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6OC41cHQ7 Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7Ij48 dT48L3U+Jm5ic3A7PHU+PC91Pjwvc3Bhbj48L3A+PC9kaXY+PC9kaXY+PGRpdj48cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjguNXB0O2ZvbnQtZmFtaWx5OiZxdW90 O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+SSB3b3VsZG4ndCBwdXQgc3Vj aCBhIHJlc3RyaWN0aW9uLCB2bSBtaWdyYXRpb24gbWF5IG5vdCBiZSBwbGFubmVkLi4gVGhleSBi b3RoIGRlZ2VuZXJhdGUgaW50byBhIG11bHRpLWhvbWluZyBwcm9ibGVtLjx1PjwvdT48dT48L3U+ PC9zcGFuPjwvcD4NCjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXIt bGVmdDpzb2xpZCAjYjVjNGRmIDMuMHB0O3BhZGRpbmc6MGluIDBpbiAwaW4gMy4wcHQ7bWFyZ2lu LWxlZnQ6My4wcHQ7bWFyZ2luLXRvcDo1LjBwdDttYXJnaW4tcmlnaHQ6MGluO21hcmdpbi1ib3R0 b206NS4wcHQiPjxkaXY+PGRpdj48ZGl2PjxkaXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1 b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFmNDk3ZCI+Jm5ic3A7PC9zcGFuPjx1 PjwvdT48dT48L3U+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNl cmlmJnF1b3Q7O2NvbG9yOiMxZjQ5N2QiPkxpbmRhPC9zcGFuPjx1PjwvdT48dT48L3U+PC9wPjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMWY0 OTdkIj4mbmJzcDs8L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6 bm9uZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGluIDBpbiAwaW4gNC4w cHQiPjxkaXY+PGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjYjVjNGRm IDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxi PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZx dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7Ij5Gcm9tOjwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3Nh bnMtc2VyaWYmcXVvdDsiPiA8YSBocmVmPSJtYWlsdG86YXJtZC1ib3VuY2VzQGlldGYub3JnIiB0 YXJnZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGlldGYub3JnPC9hPiBbPGEgaHJlZj0ibWFpbHRv OmFybWQtYm91bmNlc0BpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPm1haWx0bzphcm1kLWJvdW5j ZXNAaWV0Zi5vcmc8L2E+XSA8Yj5PbiBCZWhhbGYgT2YgPC9iPlZpc2h3YXMgTWFucmFsPGJyPg0K PGI+U2VudDo8L2I+IFRodXJzZGF5LCBTZXB0ZW1iZXIgMjIsIDIwMTEgOTozOCBQTTxicj48Yj5U bzo8L2I+IE11cmFyaSBTcmlkaGFyYW48YnI+PGI+Q2M6PC9iPiA8YSBocmVmPSJtYWlsdG86YXJt ZEBpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWRAaWV0Zi5vcmc8L2E+PGJyPjxiPlN1Ympl Y3Q6PC9iPiBSZTogW2FybWRdIGhvdyBkb2VzICJkcmFmdC1zcmlkaGFyYW4tdmlydHVhbGl6YXRp b24tbnZncmUtMDAiIGFkdmVydGlzZSBpdHMgZXh0ZXJuYWwgZmFjaW5nIGhvc3RzJyBJUCBhZGRy ZXNzZXMgdG8gZXh0ZXJuYWwgd29ybGQ/PC9zcGFuPjx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+ PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PGRpdj48 cCBjbGFzcz0iTXNvTm9ybWFsIj5NdXJhcmkgdGhpbmsgSVAgbW9iaWxpdHkuIDopPHU+PC91Pjx1 PjwvdT48L3A+PC9kaXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj4mbmJzcDs8dT48L3U+PHU+ PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIFRodSwgU2VwIDIyLCAy MDExIGF0IDU6MDAgUE0sIE11cmFyaSBTcmlkaGFyYW4gJmx0OzxhIGhyZWY9Im1haWx0bzptdXJh cmlzQG1pY3Jvc29mdC5jb20iIHRhcmdldD0iX2JsYW5rIj5tdXJhcmlzQG1pY3Jvc29mdC5jb208 L2E+Jmd0OyB3cm90ZTo8dT48L3U+PHU+PC91PjwvcD4NCjxkaXY+PGRpdj48ZGl2PjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZx dW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+RG8geW91IGhhdmUgYSBz Y2VuYXJpbyBpbiBtaW5kPzwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD48L2Rpdj48L2Rpdj48ZGl2 IGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJ0ZXh0LWFsaWduOmNlbnRlciIgYWxpZ249ImNlbnRl ciI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjguNXB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGli cmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+PGhyIHNpemU9IjMiIHdpZHRoPSIxMDAl IiBhbGlnbj0iY2VudGVyIj48L3NwYW4+PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PGI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7 LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsiPkZyb206IDwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3NhbnMt c2VyaWYmcXVvdDsiPlZpc2h3YXMgTWFucmFsPC9zcGFuPjxicj4NCjxiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5z LXNlcmlmJnF1b3Q7Ij5TZW50OiA8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAu MHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7 Ij45LzIyLzIwMTEgNDo1NSBQTTwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD4NCjxkaXY+PGRpdj48 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPjxicj48Yj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtUYWhvbWEmcXVv dDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90OyI+VG86IDwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3NhbnMt c2VyaWYmcXVvdDsiPk11cmFyaSBTcmlkaGFyYW48L3NwYW4+PGJyPg0KPGI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3Nh bnMtc2VyaWYmcXVvdDsiPkNjOiA8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAu MHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RhaG9tYSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7 Ij5OYXJhc2ltaGFuIFZlbmthdGFyYW1haWFoOyBMaW5kYSBEdW5iYXI7IDxhIGhyZWY9Im1haWx0 bzpkYXZpZC5ibGFja0BlbWMuY29tIiB0YXJnZXQ9Il9ibGFuayI+ZGF2aWQuYmxhY2tAZW1jLmNv bTwvYT47IDxhIGhyZWY9Im1haWx0bzphcm1kQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJt ZEBpZXRmLm9yZzwvYT48L3NwYW4+PGJyPg0KPGI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4w cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsi PlN1YmplY3Q6IDwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1m YW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDsiPlJlOiBbYXJt ZF0gaG93IGRvZXMgImRyYWZ0LXNyaWRoYXJhbi12aXJ0dWFsaXphdGlvbi1udmdyZS0wMCIgYWR2 ZXJ0aXNlIGl0cyBleHRlcm5hbCBmYWNpbmcgaG9zdHMnIElQIGFkZHJlc3NlcyB0byBleHRlcm5h bCB3b3JsZD88L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+DQo8ZGl2PjxkaXY+PHAgY2xhc3M9Ik1z b05vcm1hbCI+SGkgTXVyYXJpLDx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjxkaXY+PHAgY2xhc3M9 Ik1zb05vcm1hbCI+Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+PGRpdj48cCBjbGFzcz0i TXNvTm9ybWFsIj5ZZXMgdGhhdCBpcyB3aGF0IEkgbWVhbi48dT48L3U+PHU+PC91PjwvcD48L2Rp dj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPg0KPC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5UaGFua3MsPHU+PC91Pjx1PjwvdT48L3A+PC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5WaXNod2FzPHU+PC91Pjx1PjwvdT48L3A+PC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5PbiBUaHUsIFNlcCAyMiwgMjAxMSBhdCA0OjUw IFBNLCBNdXJhcmkgU3JpZGhhcmFuICZsdDs8YSBocmVmPSJtYWlsdG86bXVyYXJpc0BtaWNyb3Nv ZnQuY29tIiB0YXJnZXQ9Il9ibGFuayI+bXVyYXJpc0BtaWNyb3NvZnQuY29tPC9hPiZndDsgd3Jv dGU6PHU+PC91Pjx1PjwvdT48L3A+DQo8ZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzFmNDk3ZCI+WW91IG1lYW4gbm90IGFu IEV0aGVybmV0IGZyYW1lIGJ1dCBzb21lIElQIHBheWxvYWQ/PC9zcGFuPjx1PjwvdT48dT48L3U+ PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2Nv bG9yOiMxZjQ5N2QiPiZuYnNwOzwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij5Gcm9tOjwvc3Bhbj48 L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiBWaXNod2FzIE1hbnJhbCBbbWFpbHRv OjxhIGhyZWY9Im1haWx0bzp2aXNod2FzLmlldGZAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+ dmlzaHdhcy5pZXRmQGdtYWlsLmNvbTwvYT5dIDxicj48Yj5TZW50OjwvYj4gVGh1cnNkYXksIFNl cHRlbWJlciAyMiwgMjAxMSA0OjQ5IFBNPGJyPg0KPGI+VG86PC9iPiBNdXJhcmkgU3JpZGhhcmFu PGJyPjxiPkNjOjwvYj4gTmFyYXNpbWhhbiBWZW5rYXRhcmFtYWlhaDsgTGluZGEgRHVuYmFyOyA8 YSBocmVmPSJtYWlsdG86ZGF2aWQuYmxhY2tAZW1jLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmRhdmlk LmJsYWNrQGVtYy5jb208L2E+OyA8YSBocmVmPSJtYWlsdG86YXJtZEBpZXRmLm9yZyIgdGFyZ2V0 PSJfYmxhbmsiPmFybWRAaWV0Zi5vcmc8L2E+PC9zcGFuPjx1PjwvdT48dT48L3U+PC9wPg0KPGRp dj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj48Yj5TdWJqZWN0OjwvYj4gUmU6IFthcm1k XSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwIiBhZHZl cnRpc2UgaXRzIGV4dGVybmFsIGZhY2luZyBob3N0cycgSVAgYWRkcmVzc2VzIHRvIGV4dGVybmFs IHdvcmxkPzx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjwvZGl2Pg0KPGRpdj48ZGl2PjxwIGNsYXNz PSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPjxkaXY+PHAgY2xhc3M9Ik1zb05v cm1hbCI+TXVyYXJpLDx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05v cm1hbCI+Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+PGRpdj48cCBjbGFzcz0iTXNvTm9y bWFsIj5XaGF0IEkgYW0gc2F5aW5nIGlzIHRoZSBpbm5lciBoZWFkZXIgc2hvdWxkIGJlIGFsbG93 ZWQgdG8gYmUgTDMuIDx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+PGRpdj48cCBjbGFzcz0iTXNv Tm9ybWFsIj4mbmJzcDs8dT48L3U+PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29O b3JtYWwiPkZyb20gdGhlIGRpYWdyYW0geW91IGhhdmUgdGhhdCBkb2VzIG5vdCBzZWVtIHRvIGJl IHRoZSBjYXNlLiBBbSBJIG1pc3NpbmcgaXQgdG90YWxseT88dT48L3U+PHU+PC91PjwvcD48L2Rp dj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPg0KPC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5UaGFua3MsPHU+PC91Pjx1PjwvdT48L3A+PC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5WaXNod2FzPHU+PC91Pjx1PjwvdT48L3A+PC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5PbiBUaHUsIFNlcCAyMiwgMjAxMSBhdCA0OjQz IFBNLCBNdXJhcmkgU3JpZGhhcmFuICZsdDs8YSBocmVmPSJtYWlsdG86bXVyYXJpc0BtaWNyb3Nv ZnQuY29tIiB0YXJnZXQ9Il9ibGFuayI+bXVyYXJpc0BtaWNyb3NvZnQuY29tPC9hPiZndDsgd3Jv dGU6PHU+PC91Pjx1PjwvdT48L3A+DQo8ZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzFmNDk3ZCI+VmlzaHdhcywgVGhhbmtz IGZvciB0aGUgZmVlZGJhY2sgd2Ugd2lsbCBkZWZpbml0ZWx5IGNvbnNpZGVyIGFkZGluZyB0aGF0 LiBJIGFtIG5vdCBzdXJlIHdoYXQgeW91IG1lYW4gYnkgZG9pbmcgTDMgaW5zdGVhZCBvZiBMMi4g V2UgYWxsb3cgYW55IGFyYml0cmFyeSB2aXJ0dWFsIHRvcG9sb2d5IGluY2x1ZGluZyBMMy4gPC9z cGFuPjx1PjwvdT48dT48L3U+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQ7Y29sb3I6IzFmNDk3ZCI+Jm5ic3A7PC9zcGFuPjx1PjwvdT48dT48 L3U+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0 O2NvbG9yOiMxZjQ5N2QiPlRoYW5rczwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD48cCBjbGFzcz0i TXNvTm9ybWFsIj48YSBuYW1lPSIxMzVjNWUwMDA1NjYyMjliXzEzNWIxNzU4MWU3MThkMzFfMTMy OTM5ODAyZDc3MjBiN18xMzI5MzgiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2NvbG9y OiMxZjQ5N2QiPiZuYnNwOzwvc3Bhbj48L2E+PHU+PC91Pjx1PjwvdT48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdCI+RnJvbTo8L3NwYW4+ PC9iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij4gVmlzaHdhcyBNYW5yYWwgW21haWx0 bzo8YSBocmVmPSJtYWlsdG86dmlzaHdhcy5pZXRmQGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsi PnZpc2h3YXMuaWV0ZkBnbWFpbC5jb208L2E+XSA8YnI+PGI+U2VudDo8L2I+IFRodXJzZGF5LCBT ZXB0ZW1iZXIgMjIsIDIwMTEgNDoxOSBQTSA8L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+DQo8ZGl2 PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQi Pjxicj48Yj5Ubzo8L2I+IE5hcmFzaW1oYW4gVmVua2F0YXJhbWFpYWg8YnI+PGI+Q2M6PC9iPiBM aW5kYSBEdW5iYXI7IE11cmFyaSBTcmlkaGFyYW47IDxhIGhyZWY9Im1haWx0bzpkYXZpZC5ibGFj a0BlbWMuY29tIiB0YXJnZXQ9Il9ibGFuayI+ZGF2aWQuYmxhY2tAZW1jLmNvbTwvYT47IDxhIGhy ZWY9Im1haWx0bzphcm1kQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJtZEBpZXRmLm9yZzwv YT48YnI+DQo8Yj5TdWJqZWN0OjwvYj4gUmU6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhh cmFuLXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZhY2lu ZyBob3N0cycgSVAgYWRkcmVzc2VzIHRvIGV4dGVybmFsIHdvcmxkPzwvc3Bhbj48dT48L3U+PHU+ PC91PjwvcD48L2Rpdj48L2Rpdj48ZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+DQombmJz cDs8dT48L3U+PHU+PC91PjwvcD48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPkhpIFNpbWhhLDx1 PjwvdT48dT48L3U+PC9wPjwvZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PHU+ PC91Pjx1PjwvdT48L3A+PC9kaXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5JIHNlZSB0aGlz IGFzIHRoZSBvbmx5IGRpZmZlcmVuY2UgYmV0d2VlbiBWWExBTiBhbmQgdGhlIE5WR1JFIHNvbHV0 aW9uIChiZXNpZGVzIG9mY291cnNlIHRoYXQgVE5JIG5lZWRzIHRvIGJlIHBhcnNlZCBpbiB0aGUg aW50ZXJtZWRpYXRlIGRldmljZSBmb3IgaGFzaGluZyBhbmQgdXNpbmcgbGVzc2VyIG51bWJlciBv ZiBieXRlcykuPHU+PC91Pjx1PjwvdT48L3A+DQo8L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3Jt YWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1h bCI+SSB3b3VsZCB0aGluayB5b3Ugc2hvdWxkIGFkZCBpdCB0byB5b3VyIGRyYWZ0IGltbWVkaWF0 ZWx5LiBXaXRoIHR1bm5lbGluZyZuYnNwO3lvdSZuYnNwO2NvbnNvbGlkYXRlJm5ic3A7dGhlIGFk ZHJlc3NlcyB2aXNpYmxlIHRvIHRoZSBjb3JlIGFuZCBieSBwcm92aWRpbmcgYSBoYXNoIG1lY2hh bmlzbSwgeW91IGFyZSBwcm92aWRpbmcgc29tZSBsZXZlbCBvZiByYW5kb21uZXNzLjx1PjwvdT48 dT48L3U+PC9wPg0KPC9kaXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj4mbmJzcDs8dT48L3U+ PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPlRoZSBvdGhlciB0aGlu ZyB5b3Ugc2hvdWxkIGxvb2sgYXQgaXMgTDMgKElQdjQvIElQdjYpJm5ic3A7b3ZlciBOVkdSRSBp bnN0ZWFkIG9mIEwyIGFsb25lLiZuYnNwO0kgZ3Vlc3MgaXQgd291bGQgYmUgdGhlIHNhbWUgY29t bWVudCBmb3IgdGhlIFZYTEFOIHByb3Bvc2FsIHRvby48dT48L3U+PHU+PC91PjwvcD4NCjwvZGl2 PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+ PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5UaGFua3MsPHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+ PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5WaXNod2FzPHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+ PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj5PbiBUaHUsIFNlcCAyMiwgMjAxMSBhdCA0OjExIFBN LCBOYXJhc2ltaGFuIFZlbmthdGFyYW1haWFoICZsdDs8YSBocmVmPSJtYWlsdG86bmFyYXZlQG1p Y3Jvc29mdC5jb20iIHRhcmdldD0iX2JsYW5rIj5uYXJhdmVAbWljcm9zb2Z0LmNvbTwvYT4mZ3Q7 IHdyb3RlOjx1PjwvdT48dT48L3U+PC9wPg0KPGRpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwi PlRoZSBkcmFmdCBtZW50aW9ucyBleGFjdGx5IHRoaXMgYXMgb25lIHVzZSBvZiB0aGUgcmVzZXJ2 ZWQgOCBiaXRzIGluIFNlY3Rpb24gNC4gQW4gTlZHUkUgZW5kcG9pbnQgY291bGQgdXNlIHRoZSA4 IGJpdHMgdG8gZnVydGhlciBkaXN0cmlidXRlIGZsb3dzIGJlbG9uZ2luZyB0byBhIHBhcnRpY3Vs YXIgVE5JIGFuZCB0aGUgc3dpdGNoZXMgdXNlIGFsbCAzMiBiaXRzIHRvIGdldCBlbnRyb3B5LiBP bmUgc3RlcCBmdXJ0aGVyIHdvdWxkIGJlIGZvciB0aGUgc3dpdGNoZXMgdG8gZ2V0IGZ1bGwgZW50 cm9weSBmcm9tIHRoZSBpbm5lciBFdGhlcm5ldCBmcmFtZS4gSSB0YWtlIGl0IHRoYXQgeW91ciBj b21tZW50IHdvdWxkIGJlIHRvIG1ha2UgaXQgZXhwbGljaXQgaW4gdGhlIGRyYWZ0LiBSaWdodD88 dT48L3U+PHU+PC91PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7Ij4mbmJzcDs8 L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsiPk9u ZTwvc3Bhbj48dT48L3U+PHU+PC91PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NvdXJpZXIgTmV3JnF1b3Q7 Ij4mbmJzcDsmbmJzcDsgc3VjaCBleGFtcGxlIGNvdWxkIGJlIHRvIHVzZSB0aGUgdXBwZXIgOCBi aXRzIG9mIHRoZSBLZXkgZmllbGQgdG88L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+PHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1 b3Q7Q291cmllciBOZXcmcXVvdDsiPiZuYnNwOyZuYnNwOyBhZGQgZmxvdyBiYXNlZCBlbnRyb3B5 IGFuZCB0YWcgYWxsIHRoZSBwYWNrZXRzIGZyb20gYSBmbG93IHdpdGggYW4gZW50cm9weSBsYWJl bC48L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBz dHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtDb3VyaWVyIE5ldyZxdW90 OyI+Jm5ic3A7PC9zcGFuPjx1PjwvdT48dT48L3U+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2NvbG9yOiMxZjQ5N2QiPlNpbWhhPC9zcGFuPjx1 PjwvdT48dT48L3U+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPg0KPHNwYW4gc3R5bGU9ImZvbnQt c2l6ZToxMS4wcHQ7Y29sb3I6IzFmNDk3ZCI+Jm5ic3A7PC9zcGFuPjx1PjwvdT48dT48L3U+PC9w PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij5G cm9tOjwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiBWaXNod2FzIE1h bnJhbCBbbWFpbHRvOjxhIGhyZWY9Im1haWx0bzp2aXNod2FzLmlldGZAZ21haWwuY29tIiB0YXJn ZXQ9Il9ibGFuayI+dmlzaHdhcy5pZXRmQGdtYWlsLmNvbTwvYT5dIDxicj4NCjxiPlNlbnQ6PC9i PiBUaHVyc2RheSwgU2VwdGVtYmVyIDIyLCAyMDExIDQ6MDQgUE08YnI+PGI+VG86PC9iPiBOYXJh c2ltaGFuIFZlbmthdGFyYW1haWFoPGJyPjxiPkNjOjwvYj4gTGluZGEgRHVuYmFyOyBNdXJhcmkg U3JpZGhhcmFuOyA8YSBocmVmPSJtYWlsdG86ZGF2aWQuYmxhY2tAZW1jLmNvbSIgdGFyZ2V0PSJf YmxhbmsiPmRhdmlkLmJsYWNrQGVtYy5jb208L2E+OyA8YSBocmVmPSJtYWlsdG86YXJtZEBpZXRm Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWRAaWV0Zi5vcmc8L2E+PGJyPg0KPGI+U3ViamVjdDo8 L2I+IFJlOiBbYXJtZF0gaG93IGRvZXMgImRyYWZ0LXNyaWRoYXJhbi12aXJ0dWFsaXphdGlvbi1u dmdyZS0wMCIgYWR2ZXJ0aXNlIGl0cyBleHRlcm5hbCBmYWNpbmcgaG9zdHMnIElQIGFkZHJlc3Nl cyB0byBleHRlcm5hbCB3b3JsZD88L3NwYW4+PHU+PC91Pjx1PjwvdT48L3A+PGRpdj48ZGl2Pjxw IGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPg0KPGRpdj48cCBjbGFz cz0iTXNvTm9ybWFsIj5IaSBTaW1oYSw8dT48L3U+PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNs YXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjxkaXY+PHAgY2xh c3M9Ik1zb05vcm1hbCI+VGhlIG1haW4gKFN0YW5kYXJkcyBUcmFjaykmbmJzcDtjaGFuZ2UgaW4g eW91ciBkcmFmdCBpcyB0aGUgYWRkaXRpb24gb2YgVE5JLjx1PjwvdT48dT48L3U+PC9wPg0KPC9k aXY+PGRpdj48cCBjbGFzcz0iTXNvTm9ybWFsIj4mbmJzcDs8dT48L3U+PHU+PC91PjwvcD48L2Rp dj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPkEgcXVlc3Rpb24gSSBoYXZlIGlzIGEgVE5JIGlk ZW50aWZpZXMgYSBwYXJ0aWN1bGFyIHRlbmFudCBhbmQgYWxsIGZsb3dzIGZyb20vdG8gYSB0ZW5h bnQgd2lsbCBiZSBoYXNoZWQgdG8gdGhlIHNhbWUgcGF0aCAoZXZlbiB3aXRoIHRoZSBjaGFuZ2Vz IGluIHN3aXRjaGVzIHRvIGRvIGhhc2hpbmcgdG8gdXNlIFROSSkuPHU+PC91Pjx1PjwvdT48L3A+ DQo8L2Rpdj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9w PjwvZGl2PjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+V2h5IGRvIHlvdSBub3QgdXNlIHRoZSBs YXN0IDggYml0cyB3aGljaCB5b3UgaGF2ZSBrZXB0IGFzIHJlc2VydmVkIGZvciBwcm92aWRpbmcg dGhlIHJhbmRvbWl6YXRpb24gZm9yIGhhc2hpbmcgZmxvd3MgYmV0d2VlbiBzYW1lIHRvL2Zyb20g b24gZGlmZmVyZW50IHBhdGhzPzx1PjwvdT48dT48L3U+PC9wPg0KPC9kaXY+PGRpdj48cCBjbGFz cz0iTXNvTm9ybWFsIj4mbmJzcDs8dT48L3U+PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNz PSJNc29Ob3JtYWwiPlRoYW5rcyw8dT48L3U+PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNz PSJNc29Ob3JtYWwiPlZpc2h3YXM8dT48L3U+PHU+PC91PjwvcD48L2Rpdj48ZGl2PjxwIGNsYXNz PSJNc29Ob3JtYWwiPk9uIFN1biwgU2VwIDE4LCAyMDExIGF0IDExOjAxIEFNLCBOYXJhc2ltaGFu IFZlbmthdGFyYW1haWFoICZsdDs8YSBocmVmPSJtYWlsdG86bmFyYXZlQG1pY3Jvc29mdC5jb20i IHRhcmdldD0iX2JsYW5rIj5uYXJhdmVAbWljcm9zb2Z0LmNvbTwvYT4mZ3Q7IHdyb3RlOjx1Pjwv dT48dT48L3U+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+VGhlIGVhc2llc3QgZnJvbSB0aGUg cG9pbnQgb2YgdmlldyBvZiBjb25maWd1cmF0aW9uIHdvdWxkIGJlIHRvIHJvdXRlIGV2ZXJ5dGhp bmcgYmFjayB0aHJvdWdoIHRoZSBlbnRlcnByaXNlIC0gbm90IG5lY2Vzc2FyaWx5IHRoZSBvcHRp bWFsIGZyb20gdGhlIGVudGVycHJpc2UgcG9pbnQgb2Ygdmlldy4gQXJlIHlvdSByZWZlcnJpbmcg dG8gYSBzY2VuYXJpbyB3aGVyZSB0aGUgVk1zIHN1Ym5ldCBpcyBzcGxpdCBiZXR3ZWVuIHRoZSBj bG91ZCBhbmQgdGhlIGVudGVycHJpc2U/IE90aGVyd2lzZSBJIGRvbid0IHNlZSB0aGUgaW1wbGlj YXRpb24gb24gdmlydHVhbGl6YXRpb24gYXMgaXRzIG5vIGRpZmZlcmVudCB0aGFuIGdldHRpbmcg dGhlIHRyYWZmaWMgcm91dGVkIHRvIHRoZSBlbnRlcnByaXNlIGluIHRoZSBmaXJzdCBjYXNlLjxi cj4NCjxicj5TaW1oYTxicj48YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXzxicj5Gcm9tOiA8YSBocmVmPSJtYWlsdG86YXJtZC1ib3VuY2VzQGlldGYub3JnIiB0YXJn ZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGlldGYub3JnPC9hPiBbPGEgaHJlZj0ibWFpbHRvOmFy bWQtYm91bmNlc0BpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWQtYm91bmNlc0BpZXRmLm9y ZzwvYT5dIG9uIGJlaGFsZiBvZiBMaW5kYSBEdW5iYXIgWzxhIGhyZWY9Im1haWx0bzpsaW5kYS5k dW5iYXJAaHVhd2VpLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmxpbmRhLmR1bmJhckBodWF3ZWkuY29t PC9hPl08YnI+DQpTZW50OiBTdW5kYXksIFNlcHRlbWJlciAxOCwgMjAxMSA3OjA2IEFNPGJyPlRv OiBNdXJhcmkgU3JpZGhhcmFuOyA8YSBocmVmPSJtYWlsdG86ZGF2aWQuYmxhY2tAZW1jLmNvbSIg dGFyZ2V0PSJfYmxhbmsiPmRhdmlkLmJsYWNrQGVtYy5jb208L2E+OyA8YSBocmVmPSJtYWlsdG86 YXJtZEBpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWRAaWV0Zi5vcmc8L2E+PGJyPlN1Ympl Y3Q6IFthcm1kXSBob3cgZG9lcyAiZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3Jl LTAwIiBhZHZlcnRpc2UgaXRzIGV4dGVybmFsIGZhY2luZyBob3N0cycgSVAgYWRkcmVzc2VzIHRv IGV4dGVybmFsIHdvcmxkPzx1PjwvdT48dT48L3U+PC9wPg0KPGRpdj48ZGl2PjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxicj5IaSBNdXJhcmksPGJyPjxicj5UaGFuayB5b3UgdmVyeSBtdWNoIGZvciBz aGFyaW5nIHRoZSBwcmVzZW50YXRpb24uPGJyPjxicj5PbmUgcXVlc3Rpb246PGJyPjxicj5Gb3Ig YSBob3N0IHdpdGhpbiBhbiBFbnRlcnByaXNlIHNpdGUgd2hpY2ggbmVlZHMgdG8gY29tbXVuaWNh dGUgd2l0aCBleHRlcm5hbCBwZWVycywgdGhlIGhvc3QgZWl0aGVyIHVzZXMgcHVibGljIElQIGFk ZHJlc3Mgd2hpY2ggaXMgdmlzaWJsZSB0byBleHRlcm5hbCBwZWVycyBvciB1c2VzIHByaXZhdGUg SVAgYWRkcmVzcyB3aGljaCBpcyB0cmFuc2xhdGVkIHRvIHB1YmxpYyBhZGRyZXNzIGF0IHRoZSBF bnRlcnByaXNlIHNpdGUncyBnYXRld2F5Ljxicj4NCjxicj5XaGVuIHRoaXMgaG9zdCBpcyBtb3Zl ZCB0byAiQ2xvdWQgZGF0YSBjZW50ZXIiLCB3aWxsIHRoZSAiQ2xvdWQgRGF0YSBjZW50ZXIiIGFk dmVydGlzZSB0aGlzIGhvc3QgYWRkcmVzcyB0byBleHRlcm5hbCBwZWVycz8gT3Igd2lsbCBhbGwg ZXh0ZXJuYWwgcGVlcnMgZ28gdGhyb3VnaCBlbnRlcnByaXNlJ3MgZ2F0ZXdheSB0byByZWFjaCB0 aGlzIGhvc3Qgd2hpY2ggaXMgbm8gbG9uZ2VyIHJlc2lkaW5nIGluIHRoZSBlbnRlcnByaXNlIHNp dGU/PGJyPg0KPGJyPlRoYW5rcywgTGluZGE8YnI+PGJyPiZndDsgLS0tLS1PcmlnaW5hbCBNZXNz YWdlLS0tLS08YnI+Jmd0OyBGcm9tOiA8YSBocmVmPSJtYWlsdG86YXJtZC1ib3VuY2VzQGlldGYu b3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGlldGYub3JnPC9hPiBbbWFpbHRvOjxh IGhyZWY9Im1haWx0bzphcm1kLWJvdW5jZXNAaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5hcm1k LWJvdW5jZXNAaWV0Zi5vcmc8L2E+XSBPbiBCZWhhbGYgT2Y8YnI+DQomZ3Q7IE11cmFyaSBTcmlk aGFyYW48YnI+Jmd0OyBTZW50OiBTYXR1cmRheSwgU2VwdGVtYmVyIDE3LCAyMDExIDM6MDIgUE08 YnI+Jmd0OyBUbzogPGEgaHJlZj0ibWFpbHRvOmRhdmlkLmJsYWNrQGVtYy5jb20iIHRhcmdldD0i X2JsYW5rIj5kYXZpZC5ibGFja0BlbWMuY29tPC9hPjsgPGEgaHJlZj0ibWFpbHRvOmFybWRAaWV0 Zi5vcmciIHRhcmdldD0iX2JsYW5rIj5hcm1kQGlldGYub3JnPC9hPjxicj4NCiZndDsgU3ViamVj dDogUmU6IFthcm1kXSBzb2xpY2l0aW5nIHR5cGljYWwgbmV0d29yayBkZXNpZ25zIGZvciBBUk1E PGJyPiZndDs8YnI+Jmd0OyBGWUksIGhlcmUgaXMgYSB0YWxrIHRoYXQgSSBnYXZlIGxhc3Qgd2Vl ayBpbiByZWxhdGlvbiB0byB0aGUgbnZncmU8YnI+Jmd0OyBkcmFmdCBiZWxvdy48YnI+Jmd0OyA8 YSBocmVmPSJodHRwOi8vY2hhbm5lbDkubXNkbi5jb20vRXZlbnRzL0JVSUxEL0JVSUxEMjAxMS9T QUMtNDQyVCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9jaGFubmVsOS5tc2RuLmNvbS9FdmVudHMv QlVJTEQvQlVJTEQyMDExL1NBQy00NDJUPC9hPjxicj4NCiZndDs8YnI+Jmd0OyBUaGFua3M8YnI+ Jmd0OyBNdXJhcmk8YnI+Jmd0Ozxicj4mZ3Q7IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tPGJy PiZndDsgRnJvbTogPGEgaHJlZj0ibWFpbHRvOmFybWQtYm91bmNlc0BpZXRmLm9yZyIgdGFyZ2V0 PSJfYmxhbmsiPmFybWQtYm91bmNlc0BpZXRmLm9yZzwvYT4gW21haWx0bzo8YSBocmVmPSJtYWls dG86YXJtZC1ib3VuY2VzQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGll dGYub3JnPC9hPl0gT24gQmVoYWxmIE9mPGJyPg0KJmd0OyA8YSBocmVmPSJtYWlsdG86ZGF2aWQu YmxhY2tAZW1jLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmRhdmlkLmJsYWNrQGVtYy5jb208L2E+PGJy PiZndDsgU2VudDogRnJpZGF5LCBTZXB0ZW1iZXIgMTYsIDIwMTEgNjoxNCBBTTxicj4mZ3Q7IFRv OiA8YSBocmVmPSJtYWlsdG86YXJtZEBpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWRAaWV0 Zi5vcmc8L2E+PGJyPiZndDsgU3ViamVjdDogUmU6IFthcm1kXSBzb2xpY2l0aW5nIHR5cGljYWwg bmV0d29yayBkZXNpZ25zIGZvciBBUk1EPGJyPg0KJmd0Ozxicj4mZ3Q7IEFuZCB0d28gbW9yZSBk cmFmdHMgb24gdGhpcyB0b3BpYzo8YnI+Jmd0Ozxicj4mZ3Q7IDxhIGhyZWY9Imh0dHA6Ly93d3cu aWV0Zi5vcmcvaWQvZHJhZnQtbWFoYWxpbmdhbS1kdXR0LWRjb3BzLXZ4bGFuLTAwLnR4dCIgdGFy Z2V0PSJfYmxhbmsiPmh0dHA6Ly93d3cuaWV0Zi5vcmcvaWQvZHJhZnQtbWFoYWxpbmdhbS1kdXR0 LWRjb3BzLXZ4bGFuLTAwLnR4dDwvYT48YnI+DQomZ3Q7IDxhIGhyZWY9Imh0dHA6Ly93d3cuaWV0 Zi5vcmcvaWQvZHJhZnQtc3JpZGhhcmFuLXZpcnR1YWxpemF0aW9uLW52Z3JlLTAwLnR4dCIgdGFy Z2V0PSJfYmxhbmsiPmh0dHA6Ly93d3cuaWV0Zi5vcmcvaWQvZHJhZnQtc3JpZGhhcmFuLXZpcnR1 YWxpemF0aW9uLW52Z3JlLTAwLnR4dDwvYT48YnI+Jmd0Ozxicj4mZ3Q7IFRoZSBlZGdlIHN3aXRj aGVzIGNvdWxkIGJlIHRoZSBzb2Z0d2FyZSBzd2l0Y2hlcyBpbiBoeXBlcnZpc29ycy48YnI+DQom Z3Q7PGJyPiZndDsgVGhhbmtzLDxicj4mZ3Q7IC0tRGF2aWQ8YnI+Jmd0Ozxicj4mZ3Q7PGJyPiZn dDsgJmd0OyAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLTxicj4mZ3Q7ICZndDsgRnJvbTogPGEg aHJlZj0ibWFpbHRvOmFybWQtYm91bmNlc0BpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPmFybWQt Ym91bmNlc0BpZXRmLm9yZzwvYT4gW21haWx0bzo8YSBocmVmPSJtYWlsdG86YXJtZC1ib3VuY2Vz QGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+YXJtZC1ib3VuY2VzQGlldGYub3JnPC9hPl0gT24g QmVoYWxmPGJyPg0KJmd0OyAmZ3Q7IE9mIFdhcnJlbiBLdW1hcmk8YnI+Jmd0OyAmZ3Q7IFNlbnQ6 IFdlZG5lc2RheSwgQXVndXN0IDMxLCAyMDExIDM6MTYgUE08YnI+Jmd0OyAmZ3Q7IFRvOiBWaXNo d2FzIE1hbnJhbDxicj4mZ3Q7ICZndDsgQ2M6IDxhIGhyZWY9Im1haWx0bzphcm1kQGlldGYub3Jn IiB0YXJnZXQ9Il9ibGFuayI+YXJtZEBpZXRmLm9yZzwvYT48YnI+Jmd0OyAmZ3Q7IFN1YmplY3Q6 IFJlOiBbYXJtZF0gc29saWNpdGluZyB0eXBpY2FsIG5ldHdvcmsgZGVzaWducyBmb3IgQVJNRDxi cj4NCiZndDsgJmd0Ozxicj4mZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7IE9uIEF1ZyAxMSwgMjAxMSwg YXQgMTE6NDAgUE0sIFZpc2h3YXMgTWFucmFsIHdyb3RlOjxicj4mZ3Q7ICZndDs8YnI+Jmd0OyAm Z3Q7ICZndDsgSGkgTGluZGEvIEFub29wLDxicj4mZ3Q7ICZndDsgJmd0Ozxicj4mZ3Q7ICZndDsg Jmd0OyBIZXJlIGlzIHRoZSBleGFtcGxlIG9mIHRoZSBkZXNpZ24gSSB3YXMgdGFsa2luZyBhYm91 dCwgYXMgZGVmaW5lZDxicj4NCiZndDsgYnkgZ29vZ2xlLjxicj4mZ3Q7ICZndDs8YnI+Jmd0OyAm Z3Q7IEp1c3QgYSBjbGFyaWZpY2F0aW9uIC0tIHMvYXMgZGVmaW5lZCBieSBnb29nbGUvYXMgZGVz Y3JpYmVkIGJ5PGJyPiZndDsgc29tZW9uZTxicj4mZ3Q7ICZndDsgd2hvIGhhcHBlbnMgdG8gd29y ayBmb3IgZ29vZ2xlLzxicj4mZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7IFc8YnI+Jmd0OyAmZ3Q7PGJy PiZndDsgJmd0OyAmZ3Q7IDxhIGhyZWY9Imh0dHA6Ly93d3cuaWV0Zi5vcmcvaWQvZHJhZnQtd2t1 bWFyaS1kY29wcy1sMy12bW1vYmlsaXR5LTAwLnR4dCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly93 d3cuaWV0Zi5vcmcvaWQvZHJhZnQtd2t1bWFyaS1kY29wcy1sMy12bW1vYmlsaXR5LTAwLnR4dDwv YT48YnI+DQomZ3Q7ICZndDsgJmd0Ozxicj4mZ3Q7ICZndDsgJmd0OyBUaGFua3MsPGJyPiZndDsg Jmd0OyAmZ3Q7IFZpc2h3YXM8YnI+Jmd0OyAmZ3Q7ICZndDsgT24gVHVlLCBBdWcgOSwgMjAxMSBh dCAyOjUwIFBNLCBBbm9vcCBHaGFud2FuaTxicj4mZ3Q7ICZsdDs8YSBocmVmPSJtYWlsdG86YW5v b3BAYWx1bW5pLmR1a2UuZWR1IiB0YXJnZXQ9Il9ibGFuayI+YW5vb3BAYWx1bW5pLmR1a2UuZWR1 PC9hPiZndDsgd3JvdGU6PGJyPg0KJmd0OyAmZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7ICZndDsgJmd0 OyZndDsmZ3Q7Jmd0Ozxicj4mZ3Q7ICZndDsgJmd0OyAodGhvdWdoIEkgdGhpbmsgaWYgdGhlcmUg d2FzIGEgc3RhbmRhcmQgd2F5IHRvIG1hcCBNdWx0aWNhc3QgTUFDIHRvPGJyPiZndDsgJmd0OyAm Z3Q7IE11bHRpY2FzdCBJUCwgdGhleSBjb3VsZDxicj4mZ3Q7ICZndDsgcHJvYmFibHkgdXNlIHN1 Y2ggYSBzdGFuZGFyZCBtZWNoYW5pc21zKS48YnI+DQomZ3Q7ICZndDsgJmd0OyAmZ3Q7Jmd0OyZn dDsmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IFRoZXkgY2FuIGRvIHRo YXQsIGJ1dCB0aGVuIHRoaXMgaW1wb3NlcyByZXF1aXJlbWVudHMgb24gdGhlPGJyPiZndDsgJmd0 OyAmZ3Q7IGVxdWlwbWVudCB0byBiZSBhYmxlIHRvIGRvIG11bHRpY2FzdCBmb3J3YXJkaW5nLCBh bmQgZXZlbiBpZiBkb2VzLDxicj4mZ3Q7ICZndDsgJmd0OyBiZWNhdXNlIG9mIHBydW5pbmcgcmVx dWlyZW1lbnRzIHRoZSBudW1iZXIgb2YgZ3JvdXBzIHdvdWxkIGJlIHZlcnk8YnI+DQomZ3Q7ICZn dDsgJmd0OyBsYXJnZS4gJm5ic3A7VGhlIGF2ZXJhZ2UgZGF0YSBjZW50ZXIgc3dpdGNoIHByb2Jh Ymx5IHdvbid0IGhhbmRsZSB0aGF0PGJyPiZndDsgJmd0OyAmZ3Q7IG1hbnkgZ3JvdXBzLjxicj4m Z3Q7ICZndDsgJmd0Ozxicj4mZ3Q7ICZndDsgJmd0OyBPbiBUdWUsIEF1ZyA5LCAyMDExIGF0IDI6 NDEgUE0sIFZpc2h3YXMgTWFucmFsPGJyPiZndDsgJmx0OzxhIGhyZWY9Im1haWx0bzp2aXNod2Fz LmlldGZAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+dmlzaHdhcy5pZXRmQGdtYWlsLmNvbTwv YT4mZ3Q7IHdyb3RlOjxicj4NCiZndDsgJmd0OyAmZ3Q7IEhpIEFub29wLDxicj4mZ3Q7ICZndDsg Jmd0Ozxicj4mZ3Q7ICZndDsgJmd0OyBGcm9tIHdoYXQgSSBrbm93IHRoZXkgZG8gbm90IHVzZSBN dWx0aWNhc3QgR1JFIChJIGhlYXIgdGhlIGV4dHJhIDQ8YnI+Jmd0OyAmZ3Q7ICZndDsgYnl0ZXMg aW4gdGhlIEdSRSBoZWFkZXIgaXMgYTxicj4mZ3Q7ICZndDsgcHJvcHJpZXRlcnkgZXh0ZW5zaW9u KS48YnI+Jmd0OyAmZ3Q7ICZndDs8YnI+DQomZ3Q7ICZndDsgJmd0OyBJIHRoaW5rIGEgZGlyZWN0 b3J5IGJhc2VkIG1lY2hhbmlzbSBpcyB3aGF0IGlzIHVzZWQgKHRob3VnaCBJIHRoaW5rPGJyPiZn dDsgJmd0OyAmZ3Q7IGlmIHRoZXJlIHdhcyBhIHN0YW5kYXJkIHdheSB0bzxicj4mZ3Q7ICZndDsg bWFwIE11bHRpY2FzdCBNQUMgdG8gTXVsdGljYXN0IElQLCB0aGV5IGNvdWxkIHByb2JhYmx5IHVz ZSBzdWNoIGE8YnI+Jmd0OyBzdGFuZGFyZCBtZWNoYW5pc21zKS48YnI+DQomZ3Q7ICZndDsgJmd0 Ozxicj4mZ3Q7ICZndDsgJmd0OyBUaGFua3MsPGJyPiZndDsgJmd0OyAmZ3Q7IFZpc2h3YXM8YnI+ Jmd0OyAmZ3Q7ICZndDsgT24gVHVlLCBBdWcgOSwgMjAxMSBhdCAyOjAzIFBNLCBBbm9vcCBHaGFu d2FuaTxicj4mZ3Q7ICZsdDs8YSBocmVmPSJtYWlsdG86YW5vb3BAYWx1bW5pLmR1a2UuZWR1IiB0 YXJnZXQ9Il9ibGFuayI+YW5vb3BAYWx1bW5pLmR1a2UuZWR1PC9hPiZndDsgd3JvdGU6PGJyPg0K Jmd0OyAmZ3Q7ICZndDsgSGkgVmlzaHdhcyw8YnI+Jmd0OyAmZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7 ICZndDsgSG93IGRvIHRoZXkgZ2V0IG11bHRpY2FzdCB0aHJvdWdoIHRoZSBuZXR3b3JrIGluIHRo YXQgY2FzZT88YnI+Jmd0OyAmZ3Q7ICZndDsgQXJlIHRoZXkgcGxhbm5pbmcgdG8gdXNlIG11bHRp Y2FzdCBHUkUsIG9yIGp1c3QgdXNlIGRpcmVjdG9yeSBiYXNlZDxicj4mZ3Q7ICZndDsgJmd0OyBs b29rdXBzIGFuZCBub3Qgd29ycnkgYWJvdXQgbXVsdGljYXN0IGFwcGxpY2F0aW9ucyBmb3Igbm93 Pzxicj4NCiZndDsgJmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IEFub29wPGJyPiZndDsgJmd0 OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IE9uIFR1ZSwgQXVnIDksIDIwMTEgYXQgMToyNyBQTSwg VmlzaHdhcyBNYW5yYWw8YnI+Jmd0OyAmbHQ7PGEgaHJlZj0ibWFpbHRvOnZpc2h3YXMuaWV0ZkBn bWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj52aXNod2FzLmlldGZAZ21haWwuY29tPC9hPiZndDsg d3JvdGU6PGJyPg0KJmd0OyAmZ3Q7ICZndDsgSGkgTGluZGEsPGJyPiZndDsgJmd0OyAmZ3Q7PGJy PiZndDsgJmd0OyAmZ3Q7IFRoZSBkYXRhIHBhY2tldHMgY2FuIGJlIHR1bm5lbGxlZCBhdCB0aGUg VG9SIG92ZXIgc2F5IGEgR1JFIHBhY2tldDxicj4mZ3Q7ICZndDsgJmd0OyBhbmQgdGhlIGNvcmUg aXMgYSBMYXllci0zIGNvcmU8YnI+Jmd0OyAmZ3Q7IChleGNlcHQgZm9yIHRoZSBkb3duc3RyZWFt IHBvcnRzKS4gU28gd2UgY291bGQgaGF2ZSBlbmNhcHN1bGF0aW9uLzxicj4NCiZndDsgJmd0OyBk ZWNhcHN1bGF0aW9uIG9mIEwyIG92ZXIgR1JFIGF0IHRoZSBUb1IuPGJyPiZndDsgJmd0OyAmZ3Q7 PGJyPiZndDsgJmd0OyAmZ3Q7IFRoZSB2ZXJ5IHNhbWUgdGhpbmcgY2FuIGJlIGRvbmUgYXQgdGhl IGh5cGVydmlzb3IgbGF5ZXIgdG9vLCBpbjxicj4mZ3Q7ICZndDsgJmd0OyB3aGljaCBjYXNlIHRo ZSBlbnRpcmUgREMgbmV0d29yazxicj4mZ3Q7ICZndDsgd291bGQgbG9vayBsaWtlIGEgTGF5ZXIt MyBmbGF0IG5ldHdvcmsgaW5jbHVkaW5nIHRoZSBUb1IgdG8gc2VydmVyPGJyPg0KJmd0OyAmZ3Q7 IGxpbmsgYW5kIHRoZSBoeXBlcnZpc29yIHdvdWxkIGRvIHRoZSB0dW5uZWxpbmcuPGJyPiZndDsg Jmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IEkgYW0gbm90IHN1cmUgaWYgeW91IGdvdCB0aGUg cG9pbnRzIGFib3ZlIG9yIG5vdC4gSSBrbm93IGNsb3VkIE9TPGJyPiZndDsgJmd0OyAmZ3Q7IGNv bXBhbmllcyB0aGF0IHByb3ZpZGUgdGhlIHNlcnZpY2U8YnI+Jmd0OyAmZ3Q7IGFuZCBoYXZlIGJp ZyBhbm5vdW5jZWQgY3VzdG9tZXJzLjxicj4NCiZndDsgJmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAm Z3Q7IFRoYW5rcyw8YnI+Jmd0OyAmZ3Q7ICZndDsgVmlzaHdhczxicj4mZ3Q7ICZndDsgJmd0OyBP biBUdWUsIEF1ZyA5LCAyMDExIGF0IDExOjUxIEFNLCBMaW5kYSBEdW5iYXIgJmx0OzxhIGhyZWY9 Im1haWx0bzpkdW5iYXIubGxAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+ZHVuYmFyLmxsQGdt YWlsLmNvbTwvYT4mZ3Q7PGJyPiZndDsgd3JvdGU6PGJyPg0KJmd0OyAmZ3Q7ICZndDsgVmlzaHdh cyw8YnI+Jmd0OyAmZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7ICZndDsgSW4gbXkgbWluZCB0aGUgYnVs bGV0IDEpIGluIHRoZSBsaXN0IHJlZmVycyB0byBUb1Igc3dpdGNoZXM8YnI+Jmd0OyAmZ3Q7ICZn dDsgZG93bnN0cmVhbSBwb3J0cyAoZmFjaW5nIHNlcnZlcnMpPGJyPiZndDsgJmd0OyBydW5uaW5n IExheWVyIDIgYW5kIFRvUiB1cGxpbmtzIHBvcnRzIHJ1biBJUCBMYXllciAzLjxicj4NCiZndDsg Jmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IEhhdmUgeW91IHNlZW4gZGF0YSBjZW50ZXIgbmV0 d29ya3Mgd2l0aCBUb1Igc3dpdGNoZXMgZG93bnN0cmVhbTxicj4mZ3Q7ICZndDsgJmd0OyBwb3J0 cyAoaS5lLiBmYWNpbmcgc2VydmVycykgZW5hYmxpbmc8YnI+Jmd0OyAmZ3Q7IElQIHJvdXRpbmcs IGV2ZW4gdGhvdWdoIHRoZSBwaHlzaWNhbCBsaW5rcyBhcmUgRXRoZXJuZXQ/PGJyPg0KJmd0OyAm Z3Q7ICZndDsgSWYgeWVzLCB3ZSBzaG91bGQgZGVmaW5pdGVseSBpbmNsdWRlIGl0IGluIHRoZSBB Uk1EIGRyYWZ0Ljxicj4mZ3Q7ICZndDsgJmd0Ozxicj4mZ3Q7ICZndDsgJmd0OyBUaGFua3MsPGJy PiZndDsgJmd0OyAmZ3Q7IExpbmRhPGJyPiZndDsgJmd0OyAmZ3Q7IE9uIFR1ZSwgQXVnIDksIDIw MTEgYXQgMTI6NTggUE0sIFZpc2h3YXMgTWFucmFsPGJyPiZndDsgJmx0OzxhIGhyZWY9Im1haWx0 bzp2aXNod2FzLmlldGZAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+dmlzaHdhcy5pZXRmQGdt YWlsLmNvbTwvYT4mZ3Q7IHdyb3RlOjxicj4NCiZndDsgJmd0OyAmZ3Q7IEhpIExpbmRhLDxicj4m Z3Q7ICZndDsgJmd0OyBJIGFtIHVuc3VyZSB3aGF0IHlvdSBtZWFuIGJ5IHRoaXMsIGJ1dDo8YnI+ Jmd0OyAmZ3Q7ICZndDsgJm5ic3A7ICogbGF5ZXIgMyBhbGwgdGhlIHdheSB0byBUT1IgKFRvcCBv ZiBSYWNrIHN3aXRjaGVzKSwgV2UgY2FuIGFsc288YnI+Jmd0OyAmZ3Q7ICZndDsgaGF2ZSBhIGhl aXJhcmNoaWNhbCBuZXR3b3JrLCB3aXRoIHRoZSBjb3JlIHRvdGFsbHkgTGF5ZXItMyAoYW5kPGJy Pg0KJmd0OyAmZ3Q7ICZndDsgaGF2aW5nIHNlcGVyYXRlPGJyPiZndDsgJmd0OyByb3V0aW5nKSwg ZnJvbSB0aGUgaG9zdHMgc3RpbGwgaW4gYSBsYXJnZSBMYXllci0zIHN1Ym5ldC4gQW5vdGhlcjxi cj4mZ3Q7ICZndDsgYXNwZWN0IGNvdWxkIGJlIHRvIGhhdmUgYSB0b3RhbGx5PGJyPiZndDsgJmd0 OyBMYXllci0zIG5ldHdvcmsuPGJyPiZndDsgJmd0OyAmZ3Q7PGJyPiZndDsgJmd0OyAmZ3Q7IFRo ZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlbSBpcyB0aGUgbGluayBiZXR3ZWVuIHRoZSBzZXJ2ZXJz IGFuZCB0aGU8YnI+DQomZ3Q7IFRvUi48YnI+Jmd0OyAmZ3Q7ICZndDs8YnI+Jmd0OyAmZ3Q7ICZn dDsgVGhhbmtzLDxicj4mZ3Q7ICZndDsgJmd0OyBWaXNod2FzPGJyPiZndDsgJmd0OyAmZ3Q7IE9u IFR1ZSwgQXVnIDksIDIwMTEgYXQgMTA6MjIgQU0sIExpbmRhIER1bmJhciAmbHQ7PGEgaHJlZj0i bWFpbHRvOmR1bmJhci5sbEBnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5kdW5iYXIubGxAZ21h aWwuY29tPC9hPiZndDs8YnI+DQomZ3Q7IHdyb3RlOjxicj4mZ3Q7ICZndDsgJmd0OyBEdXJpbmcg dGhlIDgxc3QgSUVURiBBUk1EIFdHIGRpc2N1c3Npb24sIGl0IHdhcyBzdWdnZXN0ZWQgdGhhdCBp dDxicj4mZ3Q7IGlzPGJyPiZndDsgJmd0OyAmZ3Q7IG5lY2Vzc2FyeSB0byBkb2N1bWVudCB0eXBp Y2FsPGJyPiZndDsgJmd0OyBkYXRhIGNlbnRlciBuZXR3b3JrIGRlc2lnbnMgc28gdGhhdCBhZGRy ZXNzIHJlc29sdXRpb24gc2NhbGluZyBpc3N1ZXM8YnI+DQomZ3Q7ICZndDsgY2FuIGJlIHByb3Bl cmx5IGRlc2NyaWJlZC4gTWFueSBkYXRhIGNlbnRlciBvcGVyYXRvcnMgaGF2ZSBleHByZXNzZWQ8 YnI+Jmd0OyB0aGF0IHRoZXkgY2FuJ3Qgb3Blbmx5IHJldmVhbCB0aGVpciBkZXRhaWxlZCBuZXR3 b3JrIGRlc2lnbnMuPGJyPiZndDsgJmd0OyBUaGVyZWZvcmUsIHdlIG9ubHkgd2FudCB0byBkb2N1 bWVudCBhbm9ueW1vdXMgZGVzaWducyB3aXRob3V0IHRvbzxicj4NCiZndDsgbXVjaDxicj4mZ3Q7 ICZndDsgZGV0YWlsLiBEdXJpbmcgdGhlIGpvdXJuZXkgb2YgZXN0YWJsaXNoaW5nIEFSTUQsIHdl IGhhdmUgY29tZSBhY3Jvc3M8YnI+Jmd0OyB0aGUgZm9sbG93aW5nIHR5cGljYWwgZGF0YSBjZW50 ZXIgbmV0d29yayBkZXNpZ25zOjxicj4mZ3Q7ICZndDsgJmd0OyAmbmJzcDsgKiBsYXllciAzIGFs bCB0aGUgd2F5IHRvIFRPUiAoVG9wIG9mIFJhY2sgc3dpdGNoZXMpLDxicj4NCiZndDsgJmd0OyAm Z3Q7ICZuYnNwOyAqIGxhcmdlIGxheWVyIDIgd2l0aCBodW5kcmVkcyAob3IgdGhvdXNhbmRzKSBv ZiBUb1JzIGJlaW5nPGJyPiZndDsgJmd0OyAmZ3Q7IGludGVyY29ubmVjdGVkIGJ5IExheWVyIDIu IFRoaXM8YnI+Jmd0OyAmZ3Q7IGRlc2lnbiB3aWxsIGhhdmUgdGhvdXNhbmRzIG9mIGhvc3RzIHVu ZGVyIHRoZSBMMi9MMyBib3VuZGFyeSByb3V0ZXI8YnI+Jmd0OyAmZ3Q7IChzKTxicj4NCiZndDsg Jmd0OyAmZ3Q7ICZuYnNwOyAqIENMT1MgZGVzaWduICZuYnNwO3dpdGggdGhvdXNhbmRzIG9mIHN3 aXRjaGVzLiBUaGlzIGRlc2lnbiB3aWxsIGhhdmU8YnI+Jmd0OyAmZ3Q7ICZndDsgdGhvdXNhbmRz IG9mIGhvc3RzIHVuZGVyIHRoZTxicj4mZ3Q7ICZndDsgTDIvTDMgYm91bmRhcnkgcm91dGVyKHMp PGJyPiZndDsgJmd0OyAmZ3Q7IFdlIGhhdmUgaGVhcmQgdGhhdCBlYWNoIG9mIHRoZSBkZXNpZ25z IGFib3ZlIGhhcyBpdHMgb3duIHByb2JsZW1zLjxicj4NCiZndDsgJmd0OyAmZ3Q7IEFSTUQgcHJv YmxlbSBzdGF0ZW1lbnRzIG1pZ2h0PGJyPiZndDsgJmd0OyBuZWVkIHRvIGRvY3VtZW50IERDIHBy b2JsZW1zIHVuZGVyIGVhY2ggdHlwaWNhbCBkZXNpZ24uPGJyPiZndDsgJmd0OyAmZ3Q7IFBsZWFz ZSBzZW5kIGZlZWRiYWNrIHRvIHVzIChlaXRoZXIgdG8gdGhlIGFybWQgZW1haWwgbGlzdCAmbmJz cDtvciB0bzxicj4mZ3Q7IHRoZTxicj4mZ3Q7ICZndDsgJmd0OyBBUk1EIGNoYWlyIEJlbnNvbiAm YW1wOyBMaW5kYSkgdG88YnI+DQomZ3Q7ICZndDsgaW5kaWNhdGUgaWYgd2UgaGF2ZSBtaXNzZWQg YW55IHR5cGljYWwgRGF0YSBDZW50ZXIgbmV0d29yayBkZXNpZ25zLjxicj4mZ3Q7ICZndDsgJmd0 Ozxicj4mZ3Q7ICZndDsgJmd0OyBZb3VyIGNvbnRyaWJ1dGlvbiBjYW4gZ3JlYXRseSBhY2NlbGVy YXRlIHRoZSBwcm9ncmVzcyBvZiBBUk1EIFdHLjxicj4mZ3Q7ICZndDsgJmd0Ozxicj4mZ3Q7ICZn dDsgJmd0OyBUaGFuayB5b3UgdmVyeSBtdWNoLjxicj4NCiZndDsgJmd0OyAmZ3Q7PGJyPiZndDsg Jmd0OyAmZ3Q7IExpbmRhICZhbXA7IEJlbnNvbjxicj4mZ3Q7ICZndDsgJmd0Ozx1PjwvdT48dT48 L3U+PC9wPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pjxw IGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzx1PjwvdT48dT48L3U+PC9wPjwvZGl2PjwvZGl2Pjwv ZGl2PjwvZGl2PjwvZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPg0KJm5ic3A7PHU+PC91Pjx1Pjwv dT48L3A+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+ Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PHAg Y2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PHU+PC91Pjx1PjwvdT48L3A+PC9kaXY+PC9kaXY+PC9k aXY+PC9kaXY+PC9kaXY+PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZTo4LjVwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMt c2VyaWYmcXVvdDsiPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fIGFybWQgbWFpbGluZyBsaXN0IDxhIGhyZWY9Im1haWx0bzphcm1kQGlldGYub3JnIiB0YXJn ZXQ9Il9ibGFuayI+YXJtZEBpZXRmLm9yZzwvYT4gPGEgaHJlZj0iaHR0cHM6Ly93d3cuaWV0Zi5v cmcvbWFpbG1hbi9saXN0aW5mby9hcm1kIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly93d3cuaWV0 Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9hcm1kPC9hPiA8dT48L3U+PHU+PC91Pjwvc3Bhbj48L3A+ DQo8L2Jsb2NrcXVvdGU+PC9kaXY+PC9kaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PiZu YnNwOzx1PjwvdT48L3A+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9ibG9j a3F1b3RlPjwvZGl2Pjxicj4NCjwvZGl2PjwvYmxvY2txdW90ZT48L2JvZHk+PC9odG1sPg== --Apple-Mail-7F5EEDB1-8D21-4608-A0A5-2F15B1B5D303--