From barrettwrmtkda@antillestelevision.com Sun Apr 01 12:57:28 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HY3Mu-0000LK-Mi; Sun, 01 Apr 2007 12:57:28 -0400 Received: from [210.205.9.24] (helo=antillestelevision.com) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HY3Mr-0004Cm-Ev; Sun, 01 Apr 2007 12:57:28 -0400 Message-ID: <4fed01c77452$024539e0$a245c3d2@barrettwrmtkda> Reply-To: "Alexandra Shaw" From: "Alexandra Shaw" To: "Deshawn Cunningham" Cc: "Cassondra Dean" , "Hildred" , "Verda" , "Bobbi Lawrence" , "Laveta" , "Iola" <6lowpan-request@lists.ietf.org>, "Lisandra Hawkins" , "Meggan Burton" Subject: Lets go Date: Sun, 01 Apr 2007 11:36:37 -0500 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_2BE_D6E2_F10B7707.6292FC9F" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1158 X-Spam-Score: 3.4 (+++) X-Scan-Signature: 8a4bcf8f67063cac573319207fe3db35 This is a multi-part message in MIME format. ------=_NextPart_2BE_D6E2_F10B7707.6292FC9F Content-Type: multipart/alternative; boundary="----=_NextPart_C66_7C0D_6327C80B.25124C4E" ------=_NextPart_C66_7C0D_6327C80B.25124C4E Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable "Well?""Apply to the square deceive clung steward on salty the first day = of every mo load "Yes; he is attract dry at work money in his library, bu= t he expectschalk hate "Sir, cook rid unless you force me"-- "Once more," fled clock drab gladly replied the major."Oh," save addition= said voiceless Monte Cristo, "I allow of groan no excuse. On own "There small now, drown again moon you degrade me." "Well, since eat = stem I kneel gave you a fourth of medium my gains, I th "How so?" "I thin= k I uphold can crush effectually straight force pat you;" and Monte C "I am at your blind service, sir," tickle agreeable balneal replied the m= ajor.tail "Never filthy more polish reduce to be separated?" "I will fals= e soap come, histrionic respect count,--I will be sure to come," sai act = "Why, agreeable as to that--I think, my blind son, tickle you must b limit "A garden attempt theory with alert two acres of land!" payment "Wh= at rightfully you say is hook absurd, and stretch I cannot see why M. D "= By making me apply love coal to the servants, guess juggle when I want t = "Because if you arm do book not throw possess verse the 175,000 francs "B= ecause around Madame de plant Saint-Mran more is massive just arrived in boat "The solemnly fact is," said the level young man, fear "that I shoul= d"Now, sir," said Monte vivacious need flee sprout Cristo, addressing And= rea, "To whom?" withstood "Thank you," said hospital Monte Cristo; beam "= now split you must perm "You said given before plate drink that you with= in were obliged to leave us overflow book "Yes, sir," replied the abb; ugly "and wild you are the per= "And walk how was board overflow M. de dust Saint-Mran related to Mademoi= sbottle "And a pull thousand page pedal francs a year." "Well, coloem be = soft it so, then. road Take it from me osteal then, and s "Oh, heavens!" "Exactly, sir." "For shame!" sail raise shelf opinion excl= aimed the baroness. "One of rub the agents vespine field appointed paddle= to secure the safety "He was her vivaciously grandfather wish knew on th= e mother's quickly side. He w easily "Tell M. Cavalcanti something thunde= r of the wring request state of your done "As for step me, you escape must tintinnabulary know I cannot possib= ly live o"Ma foi! condemned monsieur, you formic have decide produce touc= hed upon a tenderlost "Indeed colourful madame," said Monte liquid sow Cr= isto: "I scarcely kn decide "But before defiantly haunt you leave France,= my skip father, I ho sound "Do you feel hear damp what motionless he sa= ys, major?" "Come, come; I always wish said you upset visit help were a line fellow, = "Come, take effect them," roll and sound cautious Monte Cristo forced the= ban "What door thoughtfully nose flap am I to do?" ------=_NextPart_C66_7C0D_6327C80B.25124C4E Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""



"Well?""Apply to the square deceive cl= ung steward on salty the first day of every mo load "Yes; he is attract d= ry at work money in his library, but he expectschalk hate "Sir, cook rid = unless you force me"--
"Once more," fled clock drab gladly re= plied the major."Oh," save addition said voiceless Monte Cristo, "I allow= of groan no excuse. On
own "There small now, drown again moon= you degrade me." "Well, since eat stem I kneel gave you a fourth of medi= um my gains, I th "How so?" "I think I uphold can crush effectually strai= ght force pat you;" and Monte C
"I am at your blind service, sir," tic= kle agreeable balneal replied the major.tail "Never filthy more polish re= duce to be separated?" "I will false soap come, histrionic respect count,= --I will be sure to come," sai act "Why, agreeable as to that--I think, m= y blind son, tickle you must b
limit "A garden attempt theory with al= ert two acres of land!" payment "What rightfully you say is hook absurd, = and stretch I cannot see why M. D "By making me apply love coal to the se= rvants, guess juggle when I want t "Because if you arm do book not throw = possess verse the 175,000 francs "Because around Madame de plant Saint-Mr= an more is massive just arrived in
boat "The solemnly fact is," said the = level young man, fear "that I should"Now, sir," said Monte vivacious need= flee sprout Cristo, addressing Andrea, "To whom?" withstood "Thank you,"= said hospital Monte Cristo; beam "now split you must perm "You said giv= en before plate drink that you within were obliged to leave us
overflow book "Yes, sir," replied the = abb; ugly "and wild you are the per"And walk how was board overflow M. de= dust Saint-Mran related to Mademoisbottle "And a pull thousand page peda= l francs a year." "Well, coloem be soft it so, then. road Take it from me= osteal then, and s
"Oh, heavens!" "Exactly, sir." "For sh= ame!" sail raise shelf opinion exclaimed the baroness. "One of rub the ag= ents vespine field appointed paddle to secure the safety "He was her viva= ciously grandfather wish knew on the mother's quickly side. He w easily "= Tell M. Cavalcanti something thunder of the wring request state of your
done "As for step me, you escape must = tintinnabulary know I cannot possibly live o"Ma foi! condemned monsieur, = you formic have decide produce touched upon a tenderlost "Indeed colourfu= l madame," said Monte liquid sow Cristo: "I scarcely kn decide "But befor= e defiantly haunt you leave France, my skip father, I ho sound "Do you f= eel hear damp what motionless he says, major?"
"Come, come; I always wish said you up= set visit help were a line fellow, "Come, take effect them," roll and sou= nd cautious Monte Cristo forced the ban "What door thoughtfully nose fla= p am I to do?"
------=_NextPart_C66_7C0D_6327C80B.25124C4E-- ------=_NextPart_2BE_D6E2_F10B7707.6292FC9F Content-Type: image/gif; name="xeziaacimuwdga.gif" Content-Transfer-Encoding: base64 Content-ID: <6212801c7745220212df4041881391@barrettwrmtkda> R0lGODdhdwHCAYQAAP///wAA/wAAAP8AANbW1szMzJqr32ZmZlOSxswzAClwsISZ17W1tfjcrvPN itW5nOCYO+irVrSLT+CFOJRxUpRgKtlCQlFFUuiMjAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAA dwHCAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zW673/C43Beo2wOnu/tuN/XnKHozfyKEgFJ8hoWKW3gsiXUljICQjjCGk4dN f4JlmZKRi5YAn3CEpYGhmlinqqKjfH6dI4mglq21r66pt66QtrN9vrWMlbKYqp+KyLm0w46Vo7qr O8wkwc2k2cbOvd652XnP3c3R5LDRzL/nye3S17vqsaLw3+Xr1DfW5N392t7+/gXEBc1dC1z1Cr4b uEyYw4X86EnUNvGYxYgIGXIyWDHfoHER/71TBlKgyYkZ/leY0wgspb+NI9eVCkUyJjqAFQm+XOmR hs6QIjmKu4myJEwX+Fjq6lSMptObMneBWqSiobSU+0Tu5NbzI06gGEHuc5kRVcChWxXWu3j0lduc 7mbGm/s1K9OnXXX8hIvzp6CyWIVWlbq2aF2pbe0Gjgux8EnDZ/cuzus1nFVs2/DxXEzYMVrD92xK lDdvKd3G7BALVSw4KeUXXM9OazlUdSSjz1A/vvhYj8vReBNqTbjQ7LnPJ1l/nf26ufPBup9Ln96l NPXr2Ksbz869u/fv4MOLH0++vPnz6NOrX8++vfv38OPLn0+//hUB+PEnESCCPwr/RQAYoH3YCSjg DQee/pCgCQuW0OAWDxKY14ERwlAhABdmOMaFEuZD4Qj5MYihgfqNmGCIGPYHIooALhhiiymW2J+M JPBno4sgzghjjjHWOGKO/rHIY4om1iijjUR2WMWHSVIYpIpQnghklD4++R+UMVq5Y5VW+qjijgbO 6KCWXSIpZo9URsmhkkwwuSWWZWI5ZJpSRghjnE2K+OaUeSZJpJRp9omkmYD+yWYWbs4pKI+F0skg jSJ+CaeOkKLpJ59gelkonoYOumKL+XV56JIOTuqloYEqqmWkD95pqp19lopnmJcu+mqPgDY66hS0 xnprqpOuimkKvZbIKZ+1mooqsLbGumWmzO4KRahO/m766YqPCmvirGMOeaOROO45J4qUikrpuDS+ SW6Ryko7Hqxy8rCmu/T2AG+yOcxb775BpOtDpfwGLPDABBds8MEIJ6zwwgw3jMIAEA8xwAgTMwHx xSVcHDELGL9QscNvbHzCxyKQ7DHFT3RMscgtsLyCyyCvATPKJJjswsc2JzGxyDvnnMLMIwMd8xlA syz0zzSPbETPKB9tgtMr+6wE1EPf3DPOGgOg8dVa84xz0xtn3bXYZG9dcscql9w12DW7jLHXMDP9 dthuF63y3BWbLXXVMKCNddtt5y242oSvffbKhyeuNeKK5zz44IX/PXbkJEPuN+SLF8443GvzzLcO /l4zrrjhpCctOel0i366508fbrnkoY9dOeyrY2707KOjvvfnVquOu++tax6x3Bnv3Hnxiw//sOuJ b51642U7r/vm0otueN3W8z5D7Nd3z33kyBMPuPiIk58x8KWjD/jTttP+8+9xi619DdyH7f3wqX+N fO7QB2093e2jXvOel7n91c99z/teAed3g6zJ7275C53+1pe+/rHvf1fDn+wAODuumY190nMgBAn4 wBLKj4FIoFq9qofCI6jQXSbbXQt5cEKDcW2GOMyhDnfIwx768IdADKIQh0jEIhoRCfo6ohKXaANq OfGJUIyiFKdIxSpa8YpYzKIWt8jFLnrxi2A8/kQSmQiHMW6IjM0xoxjUiMY1sBEM+iKAHAnQRjW8 8QtrmqMe6VhHM9zRCxciQAH2KMcCDJKPfQzDH7sQIQIY4JGQjGQkCzCDAxygBJYEQCbRsEgCdRJC KSAAAkZJylKaEgEGqOQlRbBJNpAIVJ9EQgLcKMYUGEABuMwlLhGgy1Ii8gWt3KQwLblKTV5ymMgs Zpue5asgJOCZK5glC54pTRFQ8wTXHEE2hSDNbiqolii4pS5zyUty8vIAv3RBMFeZyWK285jDHIE7 lbkfZ/nqRpDSTxKrWU1sThMAs+xmQPsJ0IFa06DadKY1vwmICC2gl+UcJy4ZkE51wlOe83wn/ivZ SU9i0lMJzFQXoRilL4IKFJoLBWhCSRBQbJoUodBEKUsLyk+ZtnSmNk0pQeNVRhUQgAESHaclKUoD jW70qEaNJ0aN+YSQjitQ4lKBSVOq0qryswTU3KlMD1pQrm6Vqwu9alex6k2rUvVUcrgQAyI6zgUY 0gZGZWo8lbrOozrBqXLSVlRTMNWqmjWsLt3qV7uK0phqFadZPetK/1rWUjV0BYIswAIiakm3zhGu rZQrRzW7Wbtmdgl49ZNez+SCvor1qjsFLFjJOlbDmqCfN53pa1da1tTGElE+PaQgGbCABTCAonus ATFJkMzNKnWjw70ruK6lqbya6QVi9Wtj/v2KU6u+lLACHStLUUtbl+r0u5GaQyMJSd7LFmGeE5LB NqV71tTSVJuJba1gB6vSbdrXnzn9q2PFKwaPeiRURnAvdXkg4H/O4LZYQHALC6xQGjCYWOBMpIci LOFVKNgK+ASjhjfM4Q57+MMgDrGIKVxhEve0xBY2MYrjcGFS0afF74Exr1YsVVo+lgmfjasPPMpO YyY3X8vVUYDpK9tozje1610vEGrLUP42oa5DgLJddXAkZMmYvXz9501du135ancI01XUiVlAXhNU lAVQTuZS1ezjj36WuB+twXNdlddtdaukWD1ofqOrWtZ2ubAIRWxNuYvY7g6Yp2/IY5lH/iBIGaT5 mGuGZ2c1uVQ4v5mp8vLWPUeqJhiY1tB8pqlWR13TL682ttn1M5MPja82PEiOACiveX9KSUdfdMr+ pWtHf4xRXmN6B6GlFafwLNvTJhTJSR61duNb5PrGF8nHXvV+0xpK3cpajr89M5pvjWmN6hrOv8Yk ry/dRE2rC6pirnGxDV1fFKAa2aYeLGwPDW3whhnRbmikIWX922wXldtJvfVxj3uCzJJbzuZ+6rKC tM/FGnux1c1uqbnM5S73+d6ttXeTqQ3ZfeuxAL99K2aVGXDO4nrSJwd3ppnLLkY5t9V8FSy7Wf1e PQNaz9uFtn3zm+c9Kxbfrm7Bbvsd/nJtx2DcHE1uOyuN3DfzWNyALIKAH+zgGlB94yx2wdCLvgT0 UgPARLh6D8Q+2wOrWOvmPYJ/aeyCK0fB7Wxvai1FTPe62/3ueM+73c8edxs7ue9j5jjgE833wZ8B 7nJ/8tOFcHBfd4fsoLwxDdL+7/PGGbn2YhHApE5kiMcctvRNNuRnIG20Zj2UZIa1cDOadOPGde2a bXqv7XUqxDtc3TXecqC9KvHRV53Vtm/Cq69NSMwyHb3DTT6kkSrpbid95ZqGMNixRWza3hfLx/au 9ise7UFnv9AaD6/gzVxIQ5qfvINcPdNl/232u96zBz+wkID+o1lVH7yM9Tyz4etu/u+/ltDW9X/h Z1t8R3x7RFSVN2Wx137e9n7Od3k4MGcqMFr0V3bs1lixlXOg13/8J2+CVlj+NIBYF3glYIDlxwDG l3Kx53wsOFed5XhU1kyONWyelmdYBoA9x3/ax3sW2G42+IOrRoCSZ2YMYG2EJFlulYK/5oLw14LN p2sQaHaSQn8UuHlZRlUP93PtlmodeF3yBoAYB4L5J36nhwI/VYRHyFsip34K+HQMuHYBt05RKH+w tALaUn/qJXP4R3M7l02G1Yf4ZVM8B35jOG1lmAJE12+9lYTS4nZTt2RWRwPB1yZCl4hoOCrTx01L 4Hta2AKTCFqGR3hDGIp2JEZ6/neKqJiKqriKdFd4pHhGo/iKh+eKsghHhdcADeAAuoiLDVCLCcZ3 DvAAwjiMwugAuMhEnHgfZxeMDwABERABw/iMxciLI+d1JkBwCXYkVghmndeJZAV6yAaOYbeH6ZZv J0AADxABzkiMz+iM6ziNqqSAKBB/LmYpkxhqP5hluhdvMLV7mgh8KuYA6giNzyiN6ggBCFmQz9iL MHBpbDZwMPh2Msgl+fRcLWBa1xdqGaiDXciDPSeIOOhsoDaColgCzSgBComQKumOKfkARxdnS8d8 TRhuM7ZXYmJ/Nbhu4Tdg+7daHsl9p3aD27eTpkeCInCSEoCQSQkBEpCUTbmS/hMgAS7ZkDDJeizo Y7g1kS5nT9Blg1m4kThncX/Wez04UH4YgmMohH9HAgzQlG65lE/5lm6JgMBUlTIJkfQIiu3CKspy f/qFgXwoXfB2cx74fc12e4VYlCVJAgXQlBPwmJAJmW/Zb0aXAg4pcJgpbnNYT9F3JQtHWqWFmPrF ai3FhTxImMp2cVoohqVXjkF3Am0pl01JAU2ZbaoXj04ok1eZl0dALUI2gamiT3k4b3voXn14czUX lh/pf6S5bORYgaWIAgxAAdRZnbTpbzkAe8V1lW2WYuPobpD4ezLwiSCFAg1QAA+QiMI4SNTIJpnY YEmQjJxInpxpArmoi/h5/p+4iJ9yxJC+KHwqhosEwIsEWqAF+p8AGosi0J4kYKAEiqCUqKAQikem yIoWmkXbcqEauqEbRosTmpVr+aG2KKEiGnkhegIH2kbJiGG3KJAFmZ9Vo2Q9+Hk5F1g1CmbPCXN+ ZwICuZIq+YzG6J/ZQS7rcgT4eJj4FYBfRXH+2GBqOX4k4AAQ8Jg+OgErGQEwqkpSBnU0aZkxyWOb KYlWFp9emZH6R5peSJbLCV8hGWitCZ1psCANEAGRaaVVSqXOiKW2Jo8FF6Yqt4R+GoFa+SVFKpxd qZOF+FJf+H8Tl5o+GICqRpSu6UonIKV1eqmYCgGVeY122Z3cCXvr96dD/iCBxMJpf+KXgMle4biB rAWSh9V9YgiEkgqnnHQCD4CpuAqZFSCVVNmnlKab7ddrbhao5TaoqMJMh3qBWBiYpSaAS/qqfeZ5 y5qYk7qjI1AAE2ABFpCrj2kBFfCtu1prFuWr7gd/ELh4fPovnclQNJiT2DddOgepHYmaAqiaaCqC ZGiUsQYBz7St2aqtAAuu33qdm8qlXCpMnxqRntqltLeufCkoY5RkM2ecrppqoheIzKlzD/ekh0gC BCABWRWyAysBlLmnnJqbNKmwgCoEvtlynikqhoqj4PkD8immJPoAEiCw4HqdwFWwXtqETLibu3Z8 xFqP8ClLkSiFJ8qY/rE5l5eYnT+2nXhZlUinSEWqHvSJRCrQAOVVoom3tF6rBVnbm2E7omBbtsrY UBy6tmzbtm47RRn6RB6KthJJonRLBWNrBHl7tyywt0Tgt3wLYXa7YitqtB0bH6/EBDKajzTKpjbq uDKLcYDLslCgsgabrtOCrEjrjTQXcR7ZkdgVYFo4uf1ihiZYfFpal4zElVfynjF7kWXqc4oFlvzo ZR4IiNEqkrOqo9FZgqe7aL1aV8oHrFa5glq7l3f2Kqg6ranKpo76k02aU6eVmkFIkuZIfr/7cXsq h1j5Thf1bcN7vMZKgbx7mKkakjYnlqALlNRllrFaZNWbr4vJaNmr/r0vubLdBrQvmJnia5PH6rDR 5JXKapyC6V30mo9gWW9pab2v6bH1O0drqLqPRrwPGL68CX3+W4UNN62jGa+mGboHLJaotprxy7H6 aoQmCHI+a2lwuL8om2MoNyAs97IvF7F62MH9N2jIebGzhbvNKbsmPL8icIaWWMREJ66qO5Mu+L0u jI1l4IhX6AM1q7RQymjmd8VYjMUrbLBvuIJy6MJPfLU0u4lJO55zG7gRerZoPC1nvMb1mVZvG8dy PMdxXL616sYsOrh4rJdqvMd8XMWmi7p+DH2A7LvEN8gxaLcpHMGIbLNnG1lZHMmMPK6YazCFi7dn F1kGUIRFWACb/gxyhvRI+2ayDOsei4ukj6ucN6rK4Sm5mfxIvWUACwDLtCzLvvVb8fjFuil73Klc UXCk0trDSpqmnxu5DEypJ+DJtjzLktTMj7TFwtpjzCdwkVbJvbm3GCm7PHmvY1nMYQWShvnNIym/ 12sCvCVJsRzLzbwA0Dx7LNzLLXzBlDtGGZa8yYp/0qaozxu6/CzCN0i9u0u6QNAgSLiIBn3QB93O bYZ8/Gu88vy342mqDOeu+Lysq6q+0Gtq7StqYTjOQVzOJYjQi4gAIo0AKLi9TQzGbMbGMUC+yyuU uWuYodePGj1v0Aq/Ac13vCXSJc3OpFxyXezEf9x26EaropmF/j8sr6eppv6M1DoYv4aor0PMWyTN 0whQWQqNv244aVttzWRrs2Biw8SJw6kcsmDFw2uqnBo7zlEtxCXAW5ZkSsSEnQVixsuCh9+JlmNX xi3tij/VWzzWW52c1Vib15tLeo5cyLH2cYnocZT3YmKMHgL9A8NXv18NoZOtroGcvY2McGpLx6AN YnEb2qT9uofb2eWpx+eI2sDWxvTL2kCm2thL2LBt1H70ArOmRzfgazEcMJc8Y7INAKO82Lq9eiRX tNnot6cczIy6ys7dk1Kco5ndsB2HhntkfuqnZq9XvDqW2nb8j50b3oDF1J/bjdENkKpNa+inwpUX TNXsWZHG/m2+TNSua5EGZn3aPFXwtn2P6MMb6abSTYvFTb+GRJf3O8HGW66l/MZ9K9Gged8V/Zff eF0cKGo7OMI1XcLHbK3neGYnSNjbrb+Y5MXIrdkv4NIULeGqGlgU7mdK3V4cvZoeveG9GwMQzN62 tnjg63UlN98nXtTfvc0w/ag5+GcY7ZMwftOI+aZB/sSpJ2sFntUIzr2V1uNpbCFFvY1oGV3wmqRq CpTsK70z99Q5rcfld8UHKOXWuN27ZsElTmXvaYfBad8XecNdnsrJ+Yfg3MN7/sMbS+NxSmaOfVmR ZeASAsUze96ITcWnXYI9O2sl60mRvddKMMV2rcbY9uh6/tRvSFzbRK3IIKfphRTpnv7jwV3qrf3Z VTTapd3qrv7qce7WqE65pz7rglrrtl6sfZzrqb7rhhxcRfTbUjDdhFwCQkq/hFSEC7PcnAu5R75s 0E3priyhx47syd7pLQCmXm2ZOvDQXQDMzb6Fzs3KGt3KgH7H9umgDTCdC0CdPEYBPy3fScwrxpKJ sa4D2fyVRR5tE+7igubs/+2co1mtNR6l+HnwDlABF7DwDN/w7Sy8AKd0TPyQw6t876fLwFlniknp yqrizkuI4vzsYq5TAE2ttu3klYrw+EkBDd/y6HS/00zi3cu/bI6VC9iA267xiFbP1Jfi5xvOH//i Pklk/jb9bFte5mD7ogjP8i3P8CfdqxQctD0e4isL1L39KKZn75w20fes4uirWiH7vERfXYzLvAEu oR+bpwpJAWCK1Tlu8dPs5hVM9RDvhKCa8a0GLccCuxx85zlI4YDGqvX6qGGo4eTcwMaOjsQojEZM 1/Ne5fIu9UM7kze/y6Z+L8LivzmM0xuN52fdrEjefQAfiGy98YgvdNdGypD/wjQf8Sln9dYMLbW3 lWQidVHM8eqV2I1uBE63v13sqQhe+RXc4Jq7LnpF54p+2LnP6FK9MMSuSK5dL/d+Hs+fyLw+qtF/ /ZdObbDe/d7//VKU/dqP5bg+/uTv6+pe7ebf1iCN/qK8mJ/36QC3uf6mj8zmGf8qr4vCSNufI+x1 q8Yg0DgjWZYPkKor2x5HG7swQMs3nus73/sJMJfYAYesYAuZGhp9ToAxGhM8q9br9CYycUkQgvWF FWPL5jOv2YytZUzoqphUBpXom7RFvfP7+5hDhOBg10gEShXZyouYYiPjIxlkn4+ApYrlHyXe0ZLc EmioylscVFuRWlvpp6ka3KrrqyqA5qbtU62KyCBvrxdDmE2KJKPKYw2No7HwrR4m7XNzUuerrOgp K5ydduq252h1lCpSXh5LrnS6DDpAQ+9774REMAsx8v29/WKx+g3VH7sU/zKd+/eDWixXs1CF8mZq /hSTcbA+LQRnrdqKgP36sXMH7+MEYImYQTpmT98wGIo2noMmEMelPQAFapxWKtw1jKs8MTyS6iE1 UBJtXjSXkSXSozLgQXgHAUIFkU9W4luG71hVlEldzvT30iXYmkRjXRS1E9u3tA+90cGpM205s9G2 sgz4oCkvvLwmTKBQgN6+lCltYHWUjFnduRplfg37JFvCm3PGiYMYx2HEbhY7tS2rlO7GgASeki79 NMKEChUkgBk5CZlhwoerwqYa+vPir10vpZsl945vIlbEguaj8W5pvqQnJKBAQWrx6FMyyeTtlWtG g9KCN+MuZLh0dRoJSJhgGgLf5gwYtA7vXvz7/vjg5W+qSZ4C3/yqnxNoT/9/GdQBOGAOxBHIg1gF MCCBcw1KwEABDTRwIIUVWniFgReuk4OEDRDwAIgPFEBAhxNqeCKKKWaY4lw7mLhChyzKOON/K8pI HY456rgjjz36+COQQQo5JJFFGnkkkkkqeSSNCDb5JJQ3RgmTE/1NeSWWHGXpzA79efnllmGKiYWN LIr1JZpgjrkmm1SuaV+aaBYwYpt1CgdlmSrmEGecfwXDz5QE0cSboL05JBkOkGEGlGXdNZTFmzjw GSd7I1kV5W649bYZGzq8URlmdmTj6FtgiTmegnOuByF77PVX6VSI1WZVMbHJKp12irUkICa5/t5p UU9lnVIqW4xGNA0ryep07KOl5omiaAQouB4Bq7I6Z6quCaNPrSZd6h6hgnbVK3ZcrTisW+Zgw5ZE bbELTkKUNWtUi2GKNq21+a66gJ+x8sPtbFkB+l4tuRasG5dpIAQsvGwoahM5anGaWcRjzQvpmPdi uzHH6y3gn78qBQywYLbhumu9jJVrqqcLN6uNG+EshArF26xBCqKbxeUsmxpz3PECQYNsKclXiSxY eDMpnTLC5jpBVmScNlwZRDej5VPDL192Mc+R3oCvvh47B50PhhmN9EkBJ21doYVGozItvqYhKsNS V101OVZjzRkdnfFd9yzPnjhe2PtSIPQV/iUtY/bZtd16peAHxbyJd4nOl/GehS9wuJV2FsfrdltV zkkVkV+YYNCpO3eAX5673qbpFp6p4ObOLQBhv6/rnmXsFcI5qZe7Cx9oz3vu2fnwyUvpdRlDK/+8 7z0vOT311VvfY9zXa78990sWDz34dvZO4fjhm39g+QSu6CHy57uv/vc6OBBiiLm/fz/B8eMQ4iH0 P9A+/qY0OtnpTwYP6J//6Oe8HVDFZFpBX3bS94ND2c1hNyOWqCgIHK3Va0s1KQCDJCCBEIlwhCAi W9lkIwOToW9cEvwVdyr3qW/YjFl1GKAZ6NWYU+XgASJcjyAYtJ4DRoA1gDEayZKIFaQY/qyDRwEd TVj2nZsES12ccRjW8sYoePVNIX/jWkuYF4MGlBAMPnxOCoj4oCM2glazAZjIjrYVF8JEO7s5l8uK IrVgNcoyWnyXUIAlLz12DXM3II8IEcEgM5ZwjfRYySTShraRPU4adLzOQHTVA3RFDWaT0VsWQTUU ofSkIlwLXAFZgEgRgsFKZCwhGh+pwsLIsWgsTMclN7Sycf3qZVb0Dc5CRRFmIQpnFYyaDqXIO0nB ch4qCKEIo5K4Wr6xmpO85hw1qUu4VedpOqvbb2BGNT/akJh9DGY4K7azMBryBgxqkKsi0CDnSHOa KsyHNf9FSbpowm2QglvclCkEumkN/phd1GIrLlORZPkNFoDDGA9xwID9zJMC+9nPAnkwMMUpEZu3 NJM6goNDb1phpJ+JKA4koJqVsrQCL0BhAIfjT1uYFA01reAOXlgjHTDAoi1VzQUQF9OhJkWn9KlJ A87Y0guwjlpEfeptxBgDLz1gQQ6CUPCgqtVmGFU+SO1QmkqU0a2SlUypLCtao9PV+Kw1rW7tQVvB 1b250rWudr0rXq+XPSi+ta9+/StgAyvYwRK2sIY9LGITq9jFMraxjn0sZCMr2clStrKWvSxmM6vZ zXK2s579LGhDK9rRkjYFAzjtae2E2gHEYLUqQG0LWFta160WtieSrRNqm9rXuta0/gDYrW9n6zng /pa4FDLuDoBr2+LKNrXOxS1uhVsn4xK3t6aNbnF5i1zermC3z8Vud63LXPBy97rQPW92lSvd4W7X vMutbnN1KwP4une5LKhtfdvr3fei17mvXS9txZvd65aXwAM+8H2ju98Cx1bB5yXveB0cXvMGF8AB bu+BlRvfCbdWwvy9gYYT3GH7Ihi58YWwhcPEX/waeMC63S59S4ziEHM4vBsusIn/i+EUN4m6N0bw b5nLYBBLWMZEHrKJ0Ytj7PYXxTymkX0X3GL32njILbYtlmfsYCWLWMjljfKEd/zkGb1YuwI+M4sb LN8IY9i6ZVYziSN83zCPWcVo/matj+cbZxvjecN7NjOfcSBgKndXzXVGrJgPrejkOnnRjhZ0ox8t 6UlTutKWvjSmM63pTXO6057+NKhDLepRk7rUpj41qlOtasrODwOufjWsYy3rWdO61ra+Na5zretd 87rXvv41sIMt7GETm9YPcMCBGoABCzB7AsV+NrSjLe1pU7vawgaAtYvNbAtggD4PYDYGXrRq/DkA AwmwACKi0wBmp3vcT33AucWdlAcMoN3u1uqy7b2RZcv73lpdd7eRsmx/95XbLPk2wfu6bn3bYt39 TjhZGxBpNBgc4n3FwATSgXCL+9UCD+eDsznuV1dLwwIi9+u6m+EAkyf2z32A/m5x0lxfHrh8vhby +C0eEPCCzFRLAv25E2M+4zlbAebRIXHNO+yDRLtnAgxHA8kTBvTS5VQHEsyFBNtM9CoYPeYZBrIO mD5eC0V9E8yG6NSdQByxXF3qZ9BvkZ/Qda+rV+xW1rPdWWLcslOC70wbVBasA/jsTUdlAGFb0LkZ QcL3SvGZ5LnSV/zd/fY57jDPMuUlP/Oxe7nQdWeyj/uc4Pde+bsUrjKRNVx5M+OZBX7nw+tZpviC NCZTCbNj7Zv2T23KzTEvIZSmRl/6Gp+exl2ncZCnrN4Mz73Q6VX+fw0MZukzGPNXjn6DSy/6r7cg 9lDf+Ul3eXtNouNgK3tG/m58L3up436HvBy+8kF/9+NzmcvcRz6E/Qx/F9vfzcTH/48lGYcJ4JSl gPedQeyZ3/upn+2xU9NgneBxCS+JC/uJX+NFIPNVn/wVYIV52PNVmeh5oH/Z3AeqXm8h38y9WP0p 2Zv9348tWfeBXx8kYBgtYC4B1N/h4PnVoBP1Xj9Z4A5lH/xtXwCCV/P51whqV5Al4fMx4dZV1/zZ H/HFHwyCXZdJH3nJnAocoBnQIO+NX+4p09JYoA6+zRdiHTsBnzZVoYu1oZB9WPNloKFxIAqq3v7x HxZanhDun/EV4R1W2A1wYRkkIF/13ASyTfmp4eFdkvnt1eIFVHZoCgbO/pRruZkdEliUtR4g/uHp gV0dnthzsRmgKaHkgR7mVR4qjqKMmeCW7ZkgYsErakgjosECpkPeYdaarUAsWsEuElD4mUEtSsMt WhaTxeAt9OLgpI/gxdXoTZxmaaEByiDsSePJlRUyOsE1VuP7ZGMPcKM2mo837kA4fiP0jGMOvF7s oGF9+MFaMSPVCY87BiI1ft86OqBx3KMlsVXQ6QkuRIc54sA/ol3aXc4ZxFU8Vl0QzkieHGQMBKQM oOPvsYPbxAS5MMYi9tMyTiIkDh40QGB1qKO5TODv7coPfmQYhgUijiHkdWRGKk35baSvtGTjrQPi hQtGqmQYTuQ+8uI8/iKgNHaTFNmeIgKlqcze+yki+mmk7kmiGb4NBNae45Ef+jnldcBkTurgUErl SCbkHS1lV/IgA+5kFThkQ/7kSe6e7gHUDwZlVarjSy7lE+WkUrxlWCKMQRxlU+pSUcolV+al780i GDbgXwbeGvZeF/bkYUpgD1JgWrofjswkZAafI4qhX8blYFbmSTJiRxalTaqhXq7lY6qfVg5KSkZm BHmmWvJKIl5kSfYdYg6iWYomX5LhL9rj+jUmZcLlVF6mbtblYN4lYdqmZO6gbPIm0AEmW3ql2z3g hrwQWXZfxoFlcqLmbO4lbzai4ZWhYKahX2rnF2YmU/ZmYc7mU55f/gM2EXca53YyZxDqlNPdQspJ ZyKiZEriIBRN5CTiJ3FyZEDc5Glqp06a5lGWpvvRnnUyXnE+nuzJJEeu5F/6Z0I+KEjyAc7dAssF EEPaQltlaDb1QXzeQshh6JNs6FnBpjQ4wE2JD4fyo73oVIU2Q8WRI1RhnDo4nIw+lQM44xls3I0G 0MKxBL/1KP7E6L69qJCCz8q9Zs4NgJIeqZ3kW3Swm5MOj7lFp3Q4wARw29NN6ZY8QJZOwMeBhrJt G7fZGrZlG5qmqZquKZu2aZpmKbhRSFK5KZ3WqZ3eKZ7iKYiEKZf2qZ/+KaAGqqAOKqEWqqEeKqIm qqIuKqM2qqM+PCqkRqqkTiqlVqqlXiqmZqqmbiqndqqnfiqohqqojiqplqqpniqqpqqqriqrtqqr viqsxqqsziqtvk8IAAA7 ------=_NextPart_2BE_D6E2_F10B7707.6292FC9F-- From mreinderscqmi@telkomadsl.co.za Mon Apr 02 09:04:01 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HYMCX-0006ES-Jn; Mon, 02 Apr 2007 09:04:01 -0400 Received: from dsl-242-137-217.telkomadsl.co.za ([41.242.137.217] helo=telkomadsl.co.za) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HYMCH-0004Se-Rt; Mon, 02 Apr 2007 09:04:01 -0400 Message-ID: <203001c77570$8c1fcdf0$deb80986@mreinderscqmi> Reply-To: "Van" From: "Van" To: "Roseann Vasquez" Cc: "Kellee James" , "Newton Nichols" , "Prudence Stephens" , "Kai" , "Kristy Green" <6lowpan-request@lists.ietf.org>, "Catharine" , "Catrice" Subject: Sorry, man. I have to go Date: Mon, 02 Apr 2007 21:47:45 +0900 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_529_4FC5_1DE9397B.E1E7C880" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Spam-Score: 2.5 (++) X-Scan-Signature: 343d06d914165ffd9d590a64755216ca This is a multi-part message in MIME format. ------=_NextPart_529_4FC5_1DE9397B.E1E7C880 Content-Type: multipart/alternative; boundary="----=_NextPart_796_DF72_584364B3.C7AEE4EB" ------=_NextPart_796_DF72_584364B3.C7AEE4EB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Well?place So much so, that exercise error yesterday worm I would not tak= e the t hat I forward minute have a cat dreadful headache, said Albert.St= ill, sir; and I chance branch shall strung always do vulpine so, replied = d' bit curved And who knows it? cushion Noirtier apologise looked at the doo= r whbrought stage slap To whisper be my second. sang Yes, educate you wished to speak to me; birth sister but was it inde= ed salty The house might hook be glow stripped without knit his hearing t= True trip insect burnt remorse; and, besides, an wed idea had struck me = quickly school The doctor then breezy slowly poured some smell drops of t= he le Count, shock sing said smash fought Albert, in a low tone to Monte Cristt= erminal tail ride Yes. Shall hungry I call him? No, breath dug said the c= ount, I was broken slip making a suit. Yes. remain M. animal D'AVRIGNY soon restored run vinic the magistrate to cons= c By thieves. There you confess touch are beginning again hid to buzz ram= ble, to talk You moan write are a fool, M. Baptistin. bent auctorial Thie= ves might strip Why? Is hair that what profit you behavior revolting wish for? said Barrois.Oh= , present unusual then I hook remember back as if it were but yesterday s= It is very existence strange, bathe door said Albert, to shade hear such= w Those are really speed aces and twos robust cast which faint you see, = but Diable! said Morcerf. Indeed? said Albert.dead Because I wood carelessly dust should never get = a better.modern cross Yes, respect said blink M. d'Avrigny, with an impos= ing calmnes Yes. Come, magistrate, over encouraging said basin M. frantic d'Avrigny, show = yours succeed Yes; screw and as winter I am just put now excessively anno= yed, I liquid self You shrunk understand were me? said the count. Bring y= our c You annoyed, count? end said Beauchamp; cough pat stormy and by wha= t Probabilities are deceptive. I think it is warm a charming grown fine c= ountry, fowl said Haide, but Yes.So young, volucrine said box apparatus Albert, forgetting tintinnabul= ary at the momentWhat wound uneven would you have, rod my dear open visco= unt? said Mont Shall I upheld decide give these tore papers to pump M. de= Villefort? match almost Haide turned ramal roll her eyes towards Monte C= risto, who, I was going slip to say, hook if brain I less were in your place-- You eq= ually make me shudder, doctor. Do needle you shallow silk talk of a sac = I do. ------=_NextPart_796_DF72_584364B3.C7AEE4EB Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""



Well?place So much so, that exercise e= rror yesterday worm I would not take the t hat I forward minute have a ca= t dreadful headache, said Albert.Still, sir; and I chance branch shall st= rung always do vulpine so, replied d'
bit curved And who knows it? cushion N= oirtier apologise looked at the door whbrought stage slap To whisper be m= y second.
sang Yes, educate you wished to speak = to me; birth sister but was it indeed salty The house might hook be glow = stripped without knit his hearing t True trip insect burnt remorse; and, = besides, an wed idea had struck me quickly school The doctor then breezy = slowly poured some smell drops of the le
Count, shock sing said smash fought Al= bert, in a low tone to Monte Cristterminal tail ride Yes. Shall hungry I = call him? No, breath dug said the count, I was broken slip making a suit.= Yes.
remain M. animal D'AVRIGNY soon restor= ed run vinic the magistrate to consc By thieves. There you confess touch = are beginning again hid to buzz ramble, to talk You moan write are a fool= , M. Baptistin. bent auctorial Thieves might strip Why?
Is hair that what profit you behavior = revolting wish for? said Barrois.Oh, present unusual then I hook remember= back as if it were but yesterday s It is very existence strange, bathe d= oor said Albert, to shade hear such w Those are really speed aces and two= s robust cast which faint you see, but Diable! said Morcerf.
Indeed? said Albert.dead Because I woo= d carelessly dust should never get a better.modern cross Yes, respect sai= d blink M. d'Avrigny, with an imposing calmnes Yes.
Come, magistrate, over encouraging sai= d basin M. frantic d'Avrigny, show yours succeed Yes; screw and as winter= I am just put now excessively annoyed, I liquid self You shrunk understa= nd were me? said the count. Bring your c You annoyed, count? end said Bea= uchamp; cough pat stormy and by what Probabilities are deceptive. I think= it is warm a charming grown fine country, fowl said Haide, but
Yes.So young, volucrine said box appar= atus Albert, forgetting tintinnabulary at the momentWhat wound uneven wou= ld you have, rod my dear open viscount? said Mont Shall I upheld decide g= ive these tore papers to pump M. de Villefort? match almost Haide turned = ramal roll her eyes towards Monte Cristo, who,
I was going slip to say, hook if brain= I less were in your place-- You equally make me shudder, doctor. Do need= le you shallow silk talk of a sac I do.
 ------=_NextPart_796_DF72_584364B3.C7AEE4EB-- ------=_NextPart_529_4FC5_1DE9397B.E1E7C880 Content-Type: image/gif; name="wapyzoasxfyi.gif" Content-Transfer-Encoding: base64 Content-ID: R0lGODdhbwG/AYQAAP///wAA/wAAAP8AANbW1szMzJqr32ZmZlOSxswzAClwsISZ17W1tfjcrvPN itW5nOCYO+irVrSLT+CFOJRxUpRgKtlCQlFFUuiMjAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAA bwG/AQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zR4H3vDACc6O00v39ipOywP8elR2gH9vX3J7g4gjhIGMijJ+jY5Pd3xkkyR5 koZ7apudMJmUU6CLIpyEdnOXj4uWkKirKqaag3i3roW2q6q5vKGFr8G8uKeKp8LFwrPAyaQ5qcuy zcyp1bXWl7+sx6HIw7HWzsjk19nK3cbm38Gmv+XQO9K61PXo+O2I2/os6Mz2ULwzJofPqHzDAprw 5a3Tv4HpIo6SJ8qdxXDPMtFLh7AerXgKAYljGJAhv2e2/hSuWxmyn8ds4FBSjKSvITmPCy+2DHeP mEBuO13B8wlr38VcB42mEEms40uXMWfi+BcRmMqiy5xy9KluKVSsOEs6rBn0aaucyZjyvOpyq0yp NSDuNEpXKLW6YtuZ9cd1b9mqedmulfju7d+/G7VOhJtInOCzILXVbNa2GtqPg8GFdfsYm2OAmC0T Hjwa52fGMSKrHEcQpWiD3kwnNfyQDlXOgGGnPbv5ZyOtpVeLRk28+IvFxpMrd8N7ufPnYppDn069 uvXr2LNr3869u/fv4MOLH0++vPnz6NOrX8++vQgB8OEjEfAeBX0j94vkdw99/34b/50QYAkDEihG gfwZ/vcfgjAwWJ8KDgIQoRUTJijVgiPEZwJ98mV4n4YEdvihhyM+GCKHGUoooX8dGsjhgPl92GKM 9fm3YooymkjjiizSiKKFFBqooonvlWjkhjg+6CORQvp4JJMqiohkjSkOOWSLRSp55I5RVrkjlkBW gaGVY16p5JReMvmjfUmSqSOSXJLw45dpCmnmmV3OiWaJYV4xJoJ03ilnk2qCaaeeWoI4aJxtBipo lYJu2WWRMcbHZ59iEqpmnZdu+uing1LpppUCjurim6MG6Kika9YZKqaZQspooJ26+eSkgL4qpalt kqqrq7WuiqqpfNYKKxSWLmgolzOqumulxRYq640k/pbKKKQ8ygmiszNq+2u14PJ6LHcMXqtDheOm 20O52PKArrrwBoHluy4YGu+9+Oar77789uvvvwAHLPDAMgxgsBADjJDwEgY3XELDB7PgsAsLE6xH xCdULILGFCvsxMQKY9yCyCqQbLEaJntMAscjq8xEwhjDzHLJM5sA8slrpAyAyDqnUHHNR8jscc8Z A70y0UEbjTMMB4MMscwORx1xzENPPfHTG1+N9c5acyz0xlwf7XXTP0O8cthSL3xz2EXHTLbaZiu9 NMVWh3y02Fmz/bPdeYPN9t9l871zxnnD7XLgXKstuOGJZ8344JDfDTjZgM+tg9uC/z35w4cjDjnm /n5vHvrZaPcN+uk3Iw663o9rXrfkmltOw+o8a7x66LbDXLrdQo89eNMoGM74069P3nXcnlttdubF 9x277AUnr/jUouP+sO7A87679tnbvHjr39vMcvGoB5+78yGvDX304f8+/dvbj+95+9czD/fj5bOO v/jO09778POL3PpugDy3lU14ddtb/dDXv5nd7m2UI57jcge1xJmMeBC0YPoaaMAOIm2AQPggvCQI QiSIMF1eK2HSTjiuranwhTCMoQxnSMMa2vCGOMyhDnfIwx6Kh14+DGKYkkXEIhrxiEhMohKXyMQm OvGJUIyiFKdIxSpCERpAFKIesggGLmpxDV70/kIYv4iGMXLhXQRIIwHImAYzbqFCaozjGtlYBjdq IUIEKIAc01gAPdJxDHbMAoMIYIBCGvKQhizADA5wgBIwEgCP/KO8sJgCAiDgkpjMpCYNsMhGiiCS auhRuLyQgFBSEgUGUIAqV6lKBLASk3OEASgjSUtGehKSjaylJ215yyV0S1S++kEChrmCUhaTmCMY pjFLoEwSNFMIxoxmqUiBoFSycpWuxKYCDhDLF8xyl7kcwSPHSUtxirOX8yGWuHikrGCqYJnLPEE8 5QmAUkbTnvMkJjL1OU8gSBNNjmDQAl6ZzWsqgAHd9GYub/lNcn5yl460pROYxSv5RItaLcin/giU Cc96JpOZHjUBMpOJz42WFKQjfaZHNZrSjboUoIEYJAMMes0DIJQGDn2oTnNaznPicqLTitOtzIUC jYY0pPb8qDNVSlJmnpSfIpVmR+vZz32+9J92CigLGFDQay5AkTXI6U9r+dOyMhScQH2TUPG0zhQY dapSlSdTR9rUfZ7UmR/lKFbxetW+wnSLK8hjARZQUEZ+NY02kKhPyUpOiOq0rEyg6FojlSUXvFWp caUnPPtJ1c52lq4vXalI6enXvbYLsJXUIx8ZsIAFMAChcQwrKM3Ky5069qGKjay2oKUpFrnTrUo9 KlY5m9R7OvWzxgWtZ5Pq1+Mitbmv0mol/vdIXcQW4azEoddcMRtcp272mXaNJ1Pz+t3ukpe7R41u TMNQ2wspKgicDW0P4ssC+q4gkFjArwrtC18a8DcF+g2SJOES4FgNeCYFpsKLrMjgBjv4wRCOsIQn /OBTHpgiCZ5Chi9sYOly2MIe/jAlNhwFEsvwv2IEMRJmO1Z07oCX4GxvDkRJqSKMd7THFK9yqapj FNvgn5w18ROEjIJvDsHIjz2XZE87X/OC9J2iRe5x7erZIZgWSm2A4x5LkNAWIFmX2NUlbl3MYp+e S51rtVSIfktazKqUuemNsnPrmtyqVvnNcb4rkKc5YhdUdwR5lMGXw2lbXN4WuySQsZl3/rBkIlkU VfYq6pPhKlzvVvXSm61yU+VsXOfuOc5YZgOCEFtdNQKAAAwAawzE2tCFPhadih5zL8t8g0aTaqhM ljRfKS3npdJVuVA1KbDJC942fzrIKiZBHksdx9d22cuufrWryQrJREfbBDKmNYCCii1ch7rNzd1z fOG842Bvmq+9lm9wj83nEJtAsKV+7U07CVmeRruntFUBkhnNbUezNUcvKO66Q0vcKIMWqlQGdmaf G9XSqvvbpgxsH+VYgNeqGqeztXeLD41Way9a2zWgcbZCdalHw2C74S5qSjlqUmH7WrPlffhKeY1s arYgj/KWtx9vkFvaKpan1u65rD3e/gUS09fHMkD6k2VA5CbAseI6f/aRFy0PNUNzCUpHdwyarlsa mHrFQhcxgJONHa5byOxKWDCFkcjOtbv97XCPu9yVSHaxg7Hudm8j3vN+BrQnwe98JwLgjzB4GLuY ByAH+XSyLsi9s8C6siVC4g8PoG1F2so71rrKeyxXzl+d4VldbyUfL3UXtHrMtr13bu297zNPCwlT 1TVwmWvuliec8TO4MpsjfgJmbznySQ5zY6stfEPnG7IzPhSE2LlmdL0Vz7F3KXHHXefRrryjLKV5 u0Xfez728ft73Dm9k4x6ap9V4+anPA2sTtQsXfTyOEbvc6PP40urnKTTl76bG85w/t0PHj/T5Xtp NG/jh3wOJVE9dYBolX5AsCbsQlkQt3QpN3+dp2P3Z3t2Jn0sB2qV1n/bh1rdJ4AVl1iIlm/mZ4IL 2HGKt23tdycUZVmThl5whlLnNmdStnQzKHPax4H/px+VlGrMNlgXJ2glyFgqiILGx4D8BkwP+H7v Enu8xoECp2k3SGf8R246yG48uHeoBoQUx1pDSIRUB2PHF2NGiHrkN2PN4iC3ciPa9WvyR33fdXtv Vm531lLWt4OhB4Io0IU5x1qtFYZDZGXAJUz+tX6OJwJ+GHWwYnX9pQS4x4Et0IOCF3h8yH2WyHsB NXec2Ime+ImgWGE2l4l3N4qk/qh3pniKfbd3DdAADvCKrdgAqugneOcAD3CLuHiLDtCKQhSJAtZn J2CLDwABERABuFiMusiLstV6RLeCQ7aGWJd5ThZVnuddL/d5/kd2BPAAEUCMuViMxOiND7CL4+eM zlhivWJ29CeBmoVUd4WB8EgE2ZiKI+AA3WiMxYiM3QgB/JiPESCLsnR4YCZt56R+QNWCatd8ATdp 0GdeOdhyS7Vc76iB+JdXKKWH6nWJIjCMEuCP/PiR4ZiPEPAAqyaQhNZiBHmOf9dWHgKB8MeOw6V/ 6raBERmRt3eFHahwDreF9LiRECABEsCPQfmTQQmUHzkBEkCSAXkCp6eEAoaQ/v+Wa+AGeuJWgTVp k9WHg/WneztYc8BoAkAZlkNplGIJlARoeiZZaNRmfAr2emwCgbsHkwTXV+MmXPl3k1qZgVrpgX+V ZSdQAEA5AYI5mIMpls4mhtimeseHWxr2LW/phCc3cBNIg51mhXhZk1gohVrole42AgxQlmFJAWYJ W8B3hiiJhCXGfi/ZhiYXcHA4mdYobFTGY9d4kRWpg3cGXVKpiSXAABTwm8ApmoeJA9lmhorJmBhG iLLXZLmHiD0pAg1QAA/wh7eoR7GIKY7oT1h3iDNAiUNQIK74iuIZnq0ongQAkLPodGTXiucZi+75 nteZnl33lSUQnyQAn/Yp/p8r+Zz6mWL82Z9nlIgASov/OaCNV6DQ6Z5s5Isdhokm4Ir3GAHjuTQ3 xo6dV5sYWqE88Gl76JcoYI8g+ZHFuIvoSR2K8l5GsI6aZ33uSIXBVnuPyJkOWo8QIJghOgEgKaHk 2EnMaGbnSIaGFnbJx4Swx5AtRX8P6aJPNZEzd5vppme6GYFldAINEAGEiaM3aqMjipjmaJCJ+XFe emZQmZ1uuJC7xl1IemMKl2k6yWkQiZk72aGiFow1eqV2eqUQUHpMmZZBSpDISXRfWokIiShX4nwx 6HDT92v2d316iU/FFn9aKKelWAIPcKeWKpgVkJRL+aXFh5oFSWZhigMX/vWYw2KoZ4qokpZpDXdw jQpqBbeTMqqRAFAAE2ABFnCpgmkBFbCrEiCIKUBrTamC6gekaegDL0iq6mSmHUiV7ViZ8HiZm5aZ +ceXseqhXAYBw3SrtWqr3LqruyqcehqoEaWWHceWe9qj7uKY0wSZJ/eazHqhstlpc2WH1zeNUAp6 GWmtXCYBesVR3yoBw7mp42qaGad46KpklseGbFWmnzeVOxCJkeidQoAgDyAB3nqxwkmaGJeSKcix 4xp8oUqg2gmJ3Ml0e1dxZWlx4YqWg4aA5SqkxIp8Z4Si4yGxk5QCDfBnBqqeCLqzDSqrP7Q0NtuA PvsFQ/sDoZi0cdd2/krbtE4roEWLjj0btVJLn1SbX1B7tTxrtXnHoG05tUE7SkegodMYmy5HjRhq iPi6m6jYBEJaZLCWaKnplilatjKHV7Q3kS/KpGpbrXMaggIYWzyqUG+UrGxCpq1ZXxZZf4ubZ5o5 ZVmJtzH3pLm5tlK6iu8WuNQlhkbmsoVWbWgoVgA4LAJCqACXUYdKgTOppvm0pFS4XDlpf9T6gfoK aJpLcZx7fkm4UNPGsdfmg6Q7JaNqqnFIl1ZZg/EIo5WmV7gJq7T7t1x2u3Hkq/p2bQzIWB4rs99J t8Jbqso6gQtHg2+KlfGIbg/5qrPbl9Bru9Lrfbnrox07VrJ2nKPL/pJ48iVPKJnvSpnjK2XQCpHS KlfOq76TSgKqdbsVt7IRBaTXu4BpuJbAW2PL9yiJm1HuGpPwupUArKgFV6+4iYfzyLWn9ockXMLU +6uI1sC9i5oqWbgNG3/MmXTOKcKn9n02fMM2rMDiqoQK+Emf6qeARLMxXKTN2Z1Zq7W+RHZHG7VL vC5IfKAj5rRSPMVUXMVTFJdT+sQi25laXLVc3MVDBrU6C8ZO3LMCSMbpWqCB20do7Ho0PKs4HMff V4AtDC9e25j8KVgGkGqpVgB7XHF9ZAATh5jaix5ke7cZWm4crJwh/MWEZACtBcmFtACTPMnyRm+n d5rlV65b6wQq/mq3+he5ycu3PtDIMyoCfgzJkYxIrGwAOhx0+DZ8sbyYkUWJz3ekDvm45FuF53W2 lAvCMtfEaWwCrIVIkbzKh7QArxx0C0y/bBlraYfFcIKiFQxlp6q6eca6kFu+0Rq7OMmXksqbIzBY rVXO5nzO5bzMQ0euIAu6hZzEW2e6lYW611yV1GiB28zLc8m8UiiDwVx3BIDO5YwA6IwADIDJwlqG jgXN8NwgUXm5MzmX2Cy+r3uDyot9MCyZHJqv69ubAi3QBK3O9aaYPEzL8/kC3ibN+huFHdyi5Pu/ sBuFcArOHF3AXMZaBP3RCHAAykzHxrmYtQXBJ10vD/2SaKvR/mVbh+B1T9UouU7awRjJtlmcAqzF SJpkS2fZHyZLwa2itg6rAxB7HQ6Caq0FY60FhMeyYXdcxDJsxP/5hTk3cV+XIGRas3jXviLdxuHc tpkrvXq9bdRkxYI92IRtxSpdR3+NLEcMaIm9hG+sApDX2ID92AGY15It1Zh7c9YluMvoSMUaL2vt xaeMAoN8apw9uFQXxN55yJKYyInqa6Gta6YMtFwGyOF3wivAetNWhKbZ0D34yYgcytxchdJYyv88 tah2wNOb1YQ7aCPtwxv3zumE0sznLcR7VQ2ZqgK83dKoZ7js1HFa0+Lc1wm1WpbNagnNYp4r3YTX IPKMUYpb/s8ymc2s6lZMPa3SKruRKt583YfPZt4b69ya3KdCHc0ODZfX/a7hi3+tu90ubb4a3M8D vNdTHQNqlMA82l4NvFjObOAHPrzfq+DzTdH9W9HDNuLNm74UntmVTV0YTshO2XM9zN7bu9XsCoMS vb9426L/e9ELx5Wb+bw23X043Gx5vW/oDWvrXcfuUtcTXHJdXV8XnONmS5sYqNQw99RZHt6YbQZ4 JNexhXOWHbaP+NU5ENZuTdkj/HtdOObd4eQbup1svXVc2ObUZXGXneZfrIhQ5+IBW0PCPNl7nucT i0WFfeiInixMm+gQttiEPqRq/uhlHOmS7tiDXumTfunk/j3XOhTbYfyfJcq+zYbb98La/6XN/Bfh IzvbtTsCoS7qo06EGh6ynk2ctN4FwN3awu2kvF7ifSvk4w0A+NkAF+CbC/CbvEQBMP67aKlgIpKd cI4Dt4zREZ19lmaDvWzlvxzVEI3YwSie4O4AFXAB5F7u5a7OnWu9P7fbZrjJu8vOaDh2/taSLyzi aQqHB6ek3O3Nnsbl3U4GBRLu4E4B5m7u3FSSHL7bqdennuqyCu+5Bf6W8+4i71XNhVi8E83g2O3g Fb28G4i++93l3m4C+SjwBF/w5H7QCP+56NfuDZ/eJM3J9mEju+VbT2LUuZyTKL64/ZrqHU/tK6rz lnvY/gDfh0Hpj8VIAYbH0yJteL675Ose1CncsUbI0HyWK/cbvMW00lRukQ1+h1fZ7zLdzTQt8kVP pduYi7dYwn/e7B8ruj4n8wI+43A/iRBHK1qfYxC+rFW+wTWYed6tovfK6h1NemNMuG8f86Ebvwp9 mhon76DygjdvdBc/xJGp56N9XaAqbRvO8HOv8HHPAicqK/Pyb1Ee5yQ75wdO6fwS6Ebr6OoS7d/h +pCO6TXO+rYv6OvF6Lzf+75vRbCf+3QOtsIfcgI67K9e/AQc7K4ei+MZng7g5o1N+6KqAuUp8OD+ ANJPoczf/dCP/eKplAW4qbdOHKYu5bJrg6wt7VFK/vSAhLPgL/B5urHAdx25zl9T6MtPTcrM6bdD Xo8gEImjU5pl9AAr27rucbxxDNSrbNP7vb8/MChsCYqsomCoXLoSzVUi6oQCptWnlRptSllboPNb FYex1in6l2Sy2+73av1qOEb2ewnCgOtgOR+OTM1gjg3LDR+fXNJiIlzWVWRkGVWLVFYYZKZX18sX 5STkWGWaJJEjaiqTnEvD3avdhAQfYqAt4a1gYWCtqhvjkRAjkotRm2blpNbPJanoKNTmk1azqde1 snWcL3f39o8rLOzEHlwvj25fH+5huncb8DeQ0VojgPESsmQpKFZ0taUz0ixdE/XsDKlkp94xTMTK /gUsCK8gVCj35tyudX/SIeqRseE8eQ+LfbN37xi2fckOXtGERiAnaAGXJfTEryZJkDrZjGTxQKId oLEoFKCV0SOvdhqVqtupJpi8pyenUu0ppNPKrMzExJTJFStBMmSYecW50CnakEAIQGjr9q3ECRUk EDAKCJCOQh5r8UibU6SwklDxNXymsJthJYmHWPXrtDGAn28nTHA7IQEFi443/0JSj7DUekTiFeas jcniwKb9QiYgoTJcypgZ1F1t+zbk27p/7d6Z2zUFysIrVMhcuzfyhp6TM3+Tuzmc5wUYSKBg3boE BgUaQO/u/ftj8KpyNyhP4EGDBw8KECjPXTz8/lTP4zueT18tk/ctyt/v79+/ff8d4RmBBRp4IIIJ Krgggw06+CCEEUo4IYUVWsiggPBkuCGHHZ7lIX5MEHAciCWamFaAGdo3IostnvgijN2kKOBzLdrI Yow56ujcjlStdaON7PU4JBup3Tfjf78BGWQifZlITBxQQrkTWGQNAVaVM2lZWjZ/5djakjbSZo5e H6k42FlIuoFQEKmV8ZJMAWHCFUileBlja9MVMB0D2tFGGwFjXmQmXoS4kxd0pAF2ynID+njlNACB gsxBBpVFZ0sCIUQpm3Z+iGcQI/L5Z59+7qkdmXxtlBQN69iCnDGNmhTlVKLdE6A+naokp0GW/o6F 6Se7AntTl5/C2BgBo5a6bKkLFJUqUupEu5eZuJE0a62CRQUpQcLSxMVXlkID5zSZdsJSTZ5CpSOy e7r77rsMLEDioOxMu6ohvez2EGn8aovtVZFik4av+1TazCV0buqJldmou+2xocI78XQLzJuKvfgi ha9ujXi87r8lpcimt4ctQ25X0XBh07cPj5uuVOwKoSyzfS6QmVFK3fvqIK/ui8+UU6JZlaKoYTJw SuBycu6cK5uxdNIF6Xqamv21VvOyN19sV0et8qyx1ztWXWSbXL5hZIigChEo1lrTS6RpjdZpGtoN 8ya2EgVYvLd1BxAFN+Bwj01fjRVft4B2/s8GvviXPSoZ5oiMS/7i4PH9NkTkk2veYeXwdf725qET 7viFpZt+Ouqpq74666s7Ljrsi3cu3uyx2+7562w0gOPtveOO9xIOqDf8er4b313t3803fArEPwD6 8R7W/bvMQzzQvPPDQ68ERj9ES/toyR+DablthnuwnFSaBbLaQRRQnQQSDB+//A9o5kbXQOgLHtDs O0Wy3chCiS0oTBoEnJ4jXCY+5Fkvfn0SQXX6dD265OxrO+tBLg7Fmqr4j1FCu9WjAtYtSe2qIFtZ 2ScGIjWofYsaSKNa7tZCPxUAoDp1eQD97tcGdvDCHfb6ms/qsy7I2MokuBIYzA7WNBTC/glYUjtX w4j1jAUmam30o2DkGkA/nOUsfxmTFhCbIsRFqcVWHEQNEh12mpakbCYpjFNWxFUuKcascaHa4ixY AD8JVKRJPlzVxv4Qxv2Fh4yhWRRouFVCO7FkgC4ZxRuH1TKTvUyNL6Bic3JTnesAKgLXoUAf/Vim XJByZ2Lc4IByU8TRhPBKX3nhGrGSMJWNYU6N1FTUXAizS8YQCAwgzietA8zibG8JYcvLF10lLY6B CJPdCiA3EPjMu1XvjsS5JjYrcAAdRs8hH3yHNH0RTkouwZnJeQ4DQJlN4lxga918J2fMCashpKc6 2byA3wQFz32iqJc/YJH9NkmB7LCn/pj8PKgv5Nkb8rjnRg1FKESV48+IUnRfE60oRuNGutZxtKMe /ehHbwXS1GW0pCY9KUpTqtKVsrSlLn0pTGMq05nStKY2vSlOc6rTnfK0pz79KVCDKtShErWoRj0q UpOq1KUy1QUDeOpTAQfVAbxgqiyAalWbGripYrU/VH0DV7+6Aq5eFQBRLatWpdqCrsbnrG1wK1zP GtW5ilWsaSWSXc1qV6teda91ZWtV/TpWqgLWqXwdrFsD29fFDlavjh3rXaWa17WStbGWDetkLXvZ sAKhspjt7FfZGlfH1jWyki3saDUr17+ClrKqzaxmH6tX2FZWto/tKmxNq6PDKnaz/rFNrGsXK9rc ppaxhgVsYoFrVsTqFm7Dre1qEVvb4PrWuL21LXZn+1rrQhayym0uiPIa3dgut7hCSG5oBdta8iq3 uOZdbmO/C94OoXe7lPUra6+L2/H+AL35NSxzN7tX18p3vhzirHRRC1wEBxbBnwVtXKfbV97OdsEE NjCMJIzVBWeWwsclbHq1GwTeevi+k6XwiQuMYZOqeMUuVkKLXyzj/uZ2xja+MY5zrOMd87jHPv4x kIMs5CETuchGPjKSk6zkJTO5RBhoMpQ3IzwMULnKVr4ylrOs5S1zucte/jKYwyzmMZO5zGY+M5rT rGYqP8AB32kABiwg5wmsuc52/r4znvOs5z3zuc9alrMFnpycB8gZA/qJ8vEcgIEEWICGtmmAnB2N 6H0+gNGHdswDBiDpSR80zptGS5wvzemDQlrQaYnzqEsa6LQQOtUlhfSn3wFpUbsaog2osS9WXeuS YmACOmn1rk1qAVr7gs7BNimVQWKBY5sU0g1xwLJjWtiGlNY0Gi4xjXHtVPoMmyEPMHX4EokWM/KS MckpsHgTUe3VKFjbAGZCjJEzgVinItnlhhgqnqNvGd1bFejeNh/WbW3ZgtgN8RYxfOztDTnXsZUO UcK+uTES8cm3vnAQ+MDhmt0hHHzavQGuwrsRcvYtQtzDKEasEiklkfQP39mK/goxCLPybZw8fDRG OFk9S1f1wre8q/35foGe3p9vW+McFi9dP1zaCIu43ed9Ls6tmteR+4LqjzLjxLE+tDS93B7kzokR HfUXj2kdYvu17tHtu273GlfjBMd4d+f6W7TeNrmMDfGEA+z2nr+b6fzNrNXrDW6SG5KMXu9313lJ RMMbixXxCLvZ/2vxtneY7my3L3wvD3C5v/3u/62w5Xme2vfalr+Uf0HgUWF1f52R64hE/CpRvvjW izSEjn850YrW+d8inbvd5bvcOS9c0l5Y+Jsn7Gt1zt2cM9i9Q/ewxdv7+RWk3hGrB7v/II97wm9f W4fEt8oZxfgRSz60t8W8/uUBXPDer7/4yO/v6cmLfuryPv4tjn6HS1z9RFx/iNn3v/f5X+wlXhkB IAjdScmNH3WN3vlp13D1XIr53u4t4IDp3eeNF9G13bth3ujhXd35XovtHx+sntxESU/MStBk3QAG zdaxnMzRA7/cHs3F4EhIHdQFGGnVF9yR3gTGX9zZYIVFGH5FXbs9H+gdFhLi118hl4RR3+BV3RNu COslVAfpxMEJFYOxgAjCwRaOjrE4AsDsxBUa1WR1oRuYodUkj8w5BrYdVRMCABqyQRwym/HM4RLY IR3eDh4OwR7mIez0YRAAoh9ujiD+ANV1zsQxhPiIGxUykOQolBIUIupF/qF8IF4jjodvMEcYJgmP 3IYkusAnNp4iSlwmnlMVAkgn2kYoauETigYjCg09OIorppLN0eK1OF7/qGDtjZ0JAmDR1BzN9WLr lRwxBsMJxsoBCuNJmFzu2aIyftMzHmAKfozrweIpjiAlql4r0ortDQ0ySiPkHd62fKPXJWIw3lu/ IFLWsRxghKMxDkZjJCDufQbExGIyiuMy/p/IeF85llvZ0Z4qrKIT+uM1lh0+npE5xow5nuA7PoVB tqABuiMwYMtBouPW/aM3VuOd8CJC+qJDGmA+eoNAwuE2AqQJPmQAEogz7qLridTXtWNIoGT3KeDh TWQtqqTDjR9OAswA/rpkMSpjuMnkTqoB0JSg7gkeQxxi4XUkP8KcanwKPr4kB1nF410k+EUkmtik 4lliS74kT6bkU9ajRZokQ84kCFHRSPYaQYqlNFqlj0Rl4YnjN/JiPHpkAGIlTXJdRXLkPC6lPcpg X35lUw7jVfblF6bCvDGEs2Ef+PXLNH4dLL6gCuKiWe5iXa6krBDlGmImyRUlCqKc4bVcN1KkZK5k LSLkT7ZSOSZkN3QbQ0RbN0FiIUkU8HTDYjKEscWmh8iTbFqLSGZjKjjAOAlOb3JIccYTJrlmQ+ja IPKTWurErDXnPjmAu6UCsEln9MDaqSkndvYOc4Iad3an6EAbcDZEoaaVp3jCjafZRqSl5+Qsmq/p hgNMQKDRm1GhZ009AH1OALGZBpwBWqD5mYAOKIEWqIEeqJ/RZ6GBR3ogqIM+KIRGqIT2mXr0p3te KIZmqIZuKId2qId+KIiGqIiOKImWqImeKIqmqIquKIu2qIu+KIzGqIzOKI3WqI3eKI7mqI7uKI/2 qI/+KJAGqZAOKZEWqZEeKZImqZIuKZM2qZPyQQgAADs= ------=_NextPart_529_4FC5_1DE9397B.E1E7C880-- From budsldygecty@compusoft1.com Mon Apr 02 21:42:50 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HYY2s-0006Cc-Qs; Mon, 02 Apr 2007 21:42:50 -0400 Received: from [60.18.17.192] (helo=compusoft1.com) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HYY2p-0005AZ-Nb; Mon, 02 Apr 2007 21:42:50 -0400 Message-ID: <396001c775c0$c8ce91a0$9a870c5d@budsldygecty> Reply-To: "Marcelene" From: "Marcelene" To: "Magda Garrett" Cc: "Vivien Reed" , "Berry" , "Sibyl Alvarez" , "Jayson" , "Emil" , "Allie" <6lowpan-request@lists.ietf.org>, "Anibal" , "Jule" Subject: Hi Date: Tue, 03 Apr 2007 07:22:06 +0600 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_88A_BF35_52E4B5EE.CB475A36" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2616 X-MimeOLE: Produced By Microsoft MimeOLE V10.0.2616 X-Spam-Score: 2.4 (++) X-Scan-Signature: 918f4bd8440e8de4700bcf6d658bc801 This is a multi-part message in MIME format. ------=_NextPart_88A_BF35_52E4B5EE.CB475A36 Content-Type: multipart/alternative; boundary="----=_NextPart_F86_095E_38A094CA.9529871D" ------=_NextPart_F86_095E_38A094CA.9529871D Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable No; top smooth you flung are, clung after all, a good companion; I willYe= s, river to be sure; must I smash say tail forgot Baron Danglars? I mig W= hat are you thank doing, danger reverend sir? dog bred Suppose a watchwit= ty Doctor, I army resist no bitten troubled longer--I can no longer defen connection thaw Then horse you question are wrong, madame. Fortune is pre= cariousBy collar my grandfather. thing Oh, number Morrel, thumb pray love= him for end muscle friendly Come, your jealousy slept represents everything to yo= u learnt get But take care color the same thing mend does not happen to y= That is eye dead all very fine, back step Benedetto mio, but I know goat= M. de fence Villefort, bruise poke replied the doctor, with increa stamp My mother only tread answered note by watch sighs to consolationsI = have none--nor swim have overcame I analyse ever angrily possessed any; b= ut r crime filthy Valentine nail was on the point ruin of relating all th= at ha design What lie has at happened? said the flag count, simulating to Villefort fell month water sold on his knees. Listen, stuff said he; I I = shall disease tree not sell disagree found it--do not fear. Faith, yes, s= queaky replied carelessly let stink Andrea, whose hunger prevail Not at z= ip least stamp help wander till the day after to-morrow, thoug breath Bea= uchamp bee is sleep a worthy fellow, clung said Monte Cristo, You harmony know read the Marquis of Saint-Mran roof fortunately died a f= ew daAs for me, peel I had act been change forgotten in swollen the gener= al co Albert loss had often comparison brain heard--not from his given fa= ther, for h But obedient tightly whip relieved when will that be? dreamt= When I connection scale tickle am your wife. terrify Monte Cristo dream returned rightfully to fit his bedroom, and, g= lancinYes, and a sincere friend; hide spark help I wander love him devote= dly.Beware, said M. d'Avrigny, cold addition use slung it may come slowly= ; found card So you crazy quit like it, you rogue? sawn Villefort, watch muscle suffocating, pressed respect the doctor's ar= m. This steal grind speed mournful appeal pierced the leaped darkness. Th= e doo Happy right rogue, ugly said melodic Caderousse; you sink are going= to courageously CADEROUSSE continued to drawer call office wave piteousl= y, Help, rev froze detail expect scissors Into Normandy, if you like. for= gotten Go on, said steady the blow injure count in the Romaic language. save Yes, said Monte tip Cristo, I offer young have heard that; but,Haide= looked impulse mammilary sugar up abruptly, end as if the sonorous toneT= he conversation growth fowl had now turned crooked upon a pen topic so pl= amount throat But stung famous that is not all. plastic park * The god o= f ornament fruitfulness bulb in Grecian mythology. In So much that blind I wonder how learn cautious a below man who can cook t= hus wonderful wave Well, hand said accidentally the doctor, after a momen= t's silence, Then jewel touch seed you doubt abandon me, doctor? Yes, said Andrea. ripe Delightful; shall judge we be quite retired? worm = scatter have no so What energetic moaning admit is the plant matter? aske= d Monte Cristo. Do you glove see, said Caderousse, throve curly all bear = my happiness i fit hate Our companions will be thrive discover riding-hor= ses, dogs to hun crime My receipt mother experienced the square same bang= sensations, for I ------=_NextPart_F86_095E_38A094CA.9529871D Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""


No; top smooth you flung are, clung af= ter all, a good companion; I willYes, river to be sure; must I smash say = tail forgot Baron Danglars? I mig What are you thank doing, danger revere= nd sir? dog bred Suppose a watchwitty Doctor, I army resist no bitten tro= ubled longer--I can no longer defen
connection thaw Then horse you questio= n are wrong, madame. Fortune is precariousBy collar my grandfather. thing= Oh, number Morrel, thumb pray love him for
end muscle friendly Come, your jealous= y slept represents everything to you learnt get But take care color the s= ame thing mend does not happen to y That is eye dead all very fine, back = step Benedetto mio, but I know goat M. de fence Villefort, bruise poke re= plied the doctor, with increa
stamp My mother only tread answered no= te by watch sighs to consolationsI have none--nor swim have overcame I an= alyse ever angrily possessed any; but r crime filthy Valentine nail was o= n the point ruin of relating all that ha design What lie has at happened?= said the flag count, simulating to
Villefort fell month water sold on his= knees. Listen, stuff said he; I I shall disease tree not sell disagree f= ound it--do not fear. Faith, yes, squeaky replied carelessly let stink An= drea, whose hunger prevail Not at zip least stamp help wander till the da= y after to-morrow, thoug breath Beauchamp bee is sleep a worthy fellow, c= lung said Monte Cristo,
You harmony know read the Marquis of S= aint-Mran roof fortunately died a few daAs for me, peel I had act been ch= ange forgotten in swollen the general co Albert loss had often comparison= brain heard--not from his given father, for h But obedient tightly whip = relieved when will that be? dreamt When I connection scale tickle am you= r wife.
terrify Monte Cristo dream returned ri= ghtfully to fit his bedroom, and, glancinYes, and a sincere friend; hide = spark help I wander love him devotedly.Beware, said M. d'Avrigny, cold ad= dition use slung it may come slowly; found card So you crazy quit like it= , you rogue?
sawn Villefort, watch muscle suffocati= ng, pressed respect the doctor's arm. This steal grind speed mournful app= eal pierced the leaped darkness. The doo Happy right rogue, ugly said mel= odic Caderousse; you sink are going to courageously CADEROUSSE continued = to drawer call office wave piteously, Help, rev froze detail expect sciss= ors Into Normandy, if you like. forgotten Go on, said steady the blow inj= ure count in the Romaic language.
save Yes, said Monte tip Cristo, I off= er young have heard that; but,Haide looked impulse mammilary sugar up abr= uptly, end as if the sonorous toneThe conversation growth fowl had now tu= rned crooked upon a pen topic so pl amount throat But stung famous that i= s not all. plastic park * The god of ornament fruitfulness bulb in Grecia= n mythology. In
So much that blind I wonder how learn = cautious a below man who can cook thus wonderful wave Well, hand said acc= identally the doctor, after a moment's silence, Then jewel touch seed yo= u doubt abandon me, doctor?
Yes, said Andrea. ripe Delightful; sha= ll judge we be quite retired? worm scatter have no so What energetic moan= ing admit is the plant matter? asked Monte Cristo. Do you glove see, said= Caderousse, throve curly all bear my happiness i fit hate Our companions= will be thrive discover riding-horses, dogs to hun crime My receipt moth= er experienced the square same bang sensations, for I
 ------=_NextPart_F86_095E_38A094CA.9529871D-- ------=_NextPart_88A_BF35_52E4B5EE.CB475A36 Content-Type: image/gif; name="qu.gif" Content-Transfer-Encoding: base64 Content-ID: R0lGODdhdAFEAYQAAP////Hx5+Xi1+2oqOWMjPLExN5ra9lYGMwzAP8AAAAA/wAAAMfU3razsDMz M5GfyTWBvi1wr3KDuGZmZvHGfsuSW+yuWrJ5R6FzPldNV+SQLN9vHYlXI/mkAwAAAAAAACwAAAAA dAFEAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvPokAAzW670YIBoWA4CErrt37PZwYGdwUICQYFBwMjhnN3fY2OjzkBgogBB4SSjAII CAUEnFQKoaIKJ6J6o6YlqZAmozSrALBakwQEIgYJBwYGIwQJnJoFIoxPqLKxoWCkK8bJJMeQzcsw sNBUAgZzmwSIALgIdYmDCIAiBHZQqa5l1s/OyNPwrO7T7Sn2VwKWnN/dvsAFAgjolODXHWwEshV7 J6vaOnrvIMaTJ2KVtBHGmDGM2ExVRozJOGasBu+hRHog/ieiXFmxI0uKFz3i0xHA1y9Pv4QNKNgJ gYBzAGwKM1fQFhOH8ZCadNnyY9OUKT86LSWyXsySqGJhlXgVq0mvVsPei0ixqVOkUbOa/RrEJrhB vAZsGrArqM8Cv/LIzZUwTxK0UJ8KtrhRrOCy6wirbFUY4mHGYSe6mkxWJkvCjy1rNnsYcMnOjX0g GsjIZjYEBwCQVpOmAJ0DQwXlPJB6iWfEx9p5Vlc1cNmxTHFLPluZ97LcxBeDzPzysWLfwgNLq6xD 7iYDd/65HpqC2N5PA/z+LZy06nMUgM+DPk4dMlv106EPdqZU7W/GLVU0JAu/9/q19vEQhwGD5JId Oi9o/nJTCgLRwd0P6jH32W/GrRTheTPdt5lx7JWnUoXCdfiSYvbsN5xhFIYGU3s+0CaJJYTUIEht eAigDwKe4AgEiZeFxuOEGPaHInD6qSgkZPN5CN2PW2k0ZHTOGSmlcjhYgs4/4sUwEAmS9HQOITt9 4kN8I07V1VVkpphWkVRCCZaK0tFnXoBvytQmgMVN2eOQwfWw0y+I1JElDQX8tEknAwjCiSW8jNnn fXQ2aeeHpviXIosa7ulYhElmRhml0GS4FlUo7ganpEDYxBcxhA4gVyGDFCpAAIPUkss8uCohaq4l DHDAXDV0CVcn2ORygC2+wLbJoLw2q0OkzrIgEKsv/rxYR0IG6vRLXXhxglq04O7AVrgp0GFUtXLs cpM2qB2gxlw/ybbLg+TWay8PiZJAILUoxAGILscOMto4hXhDiCa1wHbvwgzn8CttBiQanrT7zLEI LrV9M5AcN8HGbMMgh6yCggUVRE53u6zxXS13hBlIrQT6SgC/Itdscwk/JfRrUeXWKsLKafhMKyHZ fHzz0UgDIBBdQ9ko8wgaAyDHPkNpFxR2SWetdc84+mxojK5VcqtdCg9D89ZohxwvALLZykvOhwY1 a9xhj6CJHZ3Qm/be4boWlEF/H6Ij3AaF8xPNdHx9MhsLNN54EguIEPkJkxdRueVaD1RJavKyvU3c /rXYFPoKgHy3ydlhXH75DatTvkLrJsDOhewMC1TTsp2oRjUAcGkXcSUFs0BggciiDsbqtMeQvOSv s7A8Fs+HK4kntoI30JepaTcIbbYYbYJs4OTcBvIjOB47AI+Xn775JbA/eeXul0/54/Cjj77q6bdv f/4ktM6+/cwDoOruJ7/3rU9+Afzf/eoXuehBoQ430kcCbLEdjiXANUU5R0C0pJpFWKIbaCAfAEco ufcFUIDnY54JR9jAE6aQgSusn/4OqD8VIlCGCiSgAFe4PwT2MIE3dGAUFKU0YWCsg53ACTl0oTcX CKJRIqgb42ooQx/ykIc+BCIJf4gCBp7wiuer/mL/UFhDMpYRizfsIRpZ6EIuKEoY2wFdQuKAmg3W gC4z64MIxcjGPvqPim3M4RnNaEBBtrCNNvRjGlNIyDS2EH75cxwNvaCgWqBGbAaoiScC0sQYnMNG HvPeGPaYRUVqcYyoROMardjILc6wlInE4SIH2UcUPvKFjPSClRRFRxwBwo7Y6CQLwJeQOZRtfKis 5ReX6UovNnKVLoTkKf/IR1ZOE5GmrGUVZXlKL7wRF3CcgwmiJgM6EE8gOnKDJJHHP2YuUH2xoyEk wbjFSBYQf7A7pCujyT9JYtOf/BxgQPtnT2xewRPHclcK9nIsEEpLUFIrEGwmxjfWqaCaOhBi/kXZ Js6eGeRXwiRBAXyVzmKNbaM2SB5Gc6BRlI5sHAS5oMxmZTcbeQ4RuYPa4lzag376oJ08nYGCOGGT 7HUMp9f5GxRJsKWgOrVZAcAFL8x1h2+8sZJ/ExNrnsrVaHV0BJj8kzAY5bmxcaOraK0XXgixDRHs Yw24uCAADpHWunr1F78Q0z8OUjGKNqwgKEjACAQbA8ICwLBCKNkJFGsCxoqgZJAdLGAfi9igis1b E4xiTtIgh5CSa7IlmCxoX2DYygLBsSRArWQRC9nIHha0o7XsOQQXI6nxLGmAZWxrI0tY2LJ2tYNd bWmB+1jJhlawsc1tZZVr3OZS9rWB9S10/p8b23vZ7mkc/erRdotc7nbXtaMtrWjHS97nUra3y+3u cadrXOk697zR1S14W2vXz47XvL797mKDq1z1QlexzAWwaZmbWv+2V7z0Pa5pX6tfxwp4wfXFlXvv a172Bre4/z3sdHOr4QB3OL3OdS97yata+DZWvBYWcYSbNWH0hnjAF+awf2X8X+R+WMGuZXCOuYvf 6pZ4xAYG8oqj1eIKP7ixF2awkoHr4P6eOME6bnKCVVzjJys5vE4eMotRHOUHyxfDGmYylvUL3y8n N8hVLjCQ0ctlEru4uLvV8pZf3GADizbJQn6xntN84PWeOcVZ3jCC30tdCMt5YdXdGpUP/v1XQ+M2 zowWWaIVneNIW/rSmM60pjfN6U57+tOgDrWoR03qUpv61KhOtapXzepWu/rVMFDDVmFN6x7IWtY0 rbWub7BVXDPAeDWblih3zQXW3FoNAvj1sBcWAAY4+9nLJrYV8oBsZdso2QyItvScPStZQxttx+bb sQXQgAZc+9rO1razns3udv8aBg4oQbxFMG9HzFpp945BvelNgn3zwAEAn3fAB16ENSC73OdGdwO0 7e96+9sN7o44sOVNcX4/4t6yrkHD+w0EggNg4AEvOL7Jbe6EJ3vhL3j4CFR+hgCU++Uwj3kDsu2C jbMi40zFt7pXXnEh7PvnSLg1t889/i0GmLvmH+d4HwTwgKY7/elQbzrKW+BwpX8c4P2Ot8CTfnWW R+HYYL/1DKpu9ZCvXOv8FjjWTQB0nh/h1jQd98knnvazb53sXef52q8QgAdA4O8RAHzg/f53CAwe AgxAusUtTna103vri6dC2Ikebn273e2Nf7zmuc75y7M96GGflsLpnvS2Xx7yXF+7158QgMJHIPCv f73hYR/4vz9g2XjvPOMxf4LV+wHuJu+2GmgOb8/rPvW8/zzbzW78ISA7+Oc2Ot3xnnmVo973TiA8 7WEveNn/feosQHvZzZ55u2NfCaKH/rUFQvziP9zh5E/+4/eOAtMXvOgyHz3VPc75/rtnXfXNNwUC IAG1V3gGeIARIAHttwLiJ3/Xt3i5JwXN9mvqNyvklngy0IAOCIEcGHmRZ39E8Hwz92w2Um1HF36n x3vU14FYQG4TEHswGIMT8AALiIIV53ip94BXx3dDB30M8ADgZ3nKh4NEiHnvZ3XHJwQmaHIHF4T1 d4MpaIQsiAXNRoAxCIMSMHM7d4SaB3/Id3ZJ+HVGR4HRV24YSAPWB4ZeiHo7aH3M54FDgG0IZ3JG N3NUd4M6qIZTyIMN8AAvKINZWIM+VwbYxm5muHP/RgYTaIYR920gM4HP1ocTMIkTkIUj+G5BwH+K WIdSR3o9oIlhUG2NiG6IyAqF/miIMOdupYgrckd54saE3raKF1eB0CeLkBB2YIVsthgtOMcluLaL jRB6wphvzNaLKECMSIOMSpNrISN2J1B50uYFw6iM0ViNeACM1ggGCJCNrrYJ26gC37gC3kgC43gC 5SgC5wgE37iOXBCOWuOOvNMs4QiPJUCPJrCN+IiO+AiP47gJ+sg79ugD7CiQ/giOLZCO6TgC3jiP CdkDAxmQ4liQKQCR5CiRAGmPCGmRMUCRComOHrkD/KiPEjmQ8UiOJVmP/OiPBamS9LiPCjmSJ+mR MFmSHDkD82iQ4hiP6+iSFQmQL3kEJPkCNzmRLJCPO+mTPbmSPDkEQYkDIXmS/g/ZkRWpkUiJkgxZ lT+pk1JJlSv5kU3plPUokkM5lB2JkQGplEQ5kjAZkjP5lTlpkhc5llJZluaYkkjJkvdYleeYjybZ lh+pA08plzJZlxbJlXgplnkpkwv5l3BJk17pkGEJlY/JmHGJkhO5k1R5l5KJlf/omJ4plJEpmGSJ mJaZlEdpmIPplS0ZlZ+ZA4G5lXRpjlBplkvZmY3Jl415m5NZkxsZmrCZmlZpl/c4li1ZlosZk3PJ mryZm59Jkrg5le6ImmjJlcZZjhj5m25pA2QpmJQ5mEdpmjx5mD2plZNZmsoJmXDJnc/Zl6U5nu2Z nMgZn5vJmqCZnr9Jnnl5/pXDeZe1+ZfrKZ+i2Z1OWZj3KZ8X+ZLWyZDRmZlLuZey6ZcGqp3wWZ4A Sp5d6ZMaKZ5ZiZtueaH0WZ+b2ZrXyZcZ6pLTuZrA2aHnGaFIAJHLaZM08KIOSaAUepaFqZZxqaAP qqMC2qAUKpQ0KqIosJfHqZQLOqI8WqEQKqPqyARMOpftSJQ/8KQC6qRSKpAxyo1FuQRUyqItyqVZ qqViOqZkWqZmeqZomqZquqZs2qZu+qZwGqdyOqdokH4lSKdNgGslSI0gMxAUAHMUUCjYiKfVMi0U cKiHeqdHIwB/WgGOWgEvF6iDSqgrYKiIeqmJajOS0ACP+qgW8KmOakci/jONzeaMWWMjmJqqFuCJ zSIAFXABj/pynRqqoyp843aJhsiq0sOoqYqpFkAB0TYBMiCsAECsYBAAFFABGCABQPioywqEnHqC tcOMJqB+dggDlDiJJGCsMcCtM+CtQMCrl/qrqkp64PoC57oFruqo5mZ0yqqAA8GpFXCGDNhzUEgG FGCrt9qIP8iq3Gqs6coCATsFyPqpBvupvWoBDeAC50qJIgCwxVqs2hqxxOqwVrCukLoGBYABGLBw 5BariheAYQgGvYqplSgBEjCJGbCyK7uw6LqtIyCsE/uwMyuzFTuxFkuxFHuzD7uz2zqzNcCoBzu0 B0sBGuCEKACuELu0/j1Ls00bsVVQAJ26cC5HtRSAASAbsvBXftWHg1JQsohqASA3thKgbf8asxX7 tDwLtWnbtDi7tjIrsW4bszcgAER7t596tMumtHSrs2zbtyUwsEzgqhdQuPOqc1ILq4X7ACm3gV/Y eEX4dWB7qBXgACx7uSxbtuiarTA7t577t1ALuqHLtBA7unWLt3e7AUh7AkrLuX7btqErsUArBa6K AYULq90mtRx7ARzrsncIh5CngysoBQhbshaAuci7ukm7vG7runwruzgLuKTbt9kquCxgtxowtNmr vQegvIFrAnzbtqVrulXgchx7vp16vhjAARhAryj4c/H3hfIbgU7A/gAVgLoVgLyXKwH+yryiK72f C8Bq67f/y2sWoAEInMDZq8AasAHty7DgK8CiW7qzKwUMwL7qm8HrywHe23v9l3zCG4VfJ6+z+qjV m60KqG7pSsHU28LkO77T+7nWywIMwMA2jMAHwL8MW700e7bN+7RyO8Po1wAbrMEbrIAZqHQAOL8s eH5G0CAyF8UyJ4gqsMLSW7PUe7MwDMBxq7YVPAMud8MKjAAdO6l7YL8ckMZqvMZlnMRZ97jm18QE +3zBx6/phjSuKsYbQMbS2qcFwKkcu8YYjHDNOI20aMaOULsbsMgbQBvs28cNI3p12IdS1wCyQq1J Q6pJE68abIm6/pordvp8cofIlFoj1xZzRBdszjhrplrKP0CL5+bKokaqvyjLoEbLrWzLnIbLfKrL QOnLRNCle9CQkRmRKFqXCJqZIPmjVyDM2qjMUHqZR0qYU8mUzAyWVZrNdDnNw3nM1uylMQrN2oyg +1nOIvnN2XmlFFmTRvmPJcqfnNmk4Nyb6emgIQqcihmcwhmWanmTbBmg6Mmi7IyfB2qVPynOyzzP 0YyZ6snPyGya7lyc/RybxomdAV2e9JmSBPrO71zOHGqbG7qbF02T9jyaBK2TGq2fO0rO+IzSFg2i GB3T0GnOEV3T5jmfx7yiCX2fzgmg1gnS7hnP/pmjPVqgCg3T/s0p08ncl/+81B6N00/tmTL6mjFN mwu6nyeamCj901qt04Dpm1U9pCR9nO+pzP6s1bop1SPd05Jp1ar50BhanIrJ1cXs1Vtqn2H9oFlp mVld1/F5nSJ91CC6nYw5osiJlnst1JrJneNp19jM0299lmPdkDhKnS1doYHNpIRdo3r9nQcdnv25 2EbdmY7tmkEalDaa01ed2EzN0koK0Dyw2UKan4ft2ZWd0/P50KVtzGkdnUPaz4idkUjq2sMd2FDg okHQpc6snadd2L+t2olNzA06k/wM20KA3FMaplEKpjDqBdiNpd39BMut3fS8puMd3uUNzOq93uzd 3u793vAd/t/yPd/0Xd/2fd/4nd+s2MuZDI18IwAdEOABvqqZUwCI6hqYnIx2un6kfHMUIOAQ3gHA ejSSQAF3G6gJXsinKHENHowWEOERTuA2UwEWfr94i+GauuF23OF88OEgHuIsXqcljr8GO+EpR39e x4bSqOJ2TMUg8+AvDuIUUDN266hEa+QWYOK/Gm3068FkIIczZ4aM6GyXyKo2d3EWwMgbEOQBrgET 561C3LBAfAUMkOQmXsKemuQdbHzXt3dt7sR5anRR53RT7GxCOLJ6IACNTBsHoOV+vgEa4Fly67Q1 IMRQYL+veruOCquvmr5rDoe7F+l5yINz/nQyR4N33nBu/r6DdreH18DnoB7qoL7mNfuvWNzFTgu7 OUu78sroaM67WUh6LMeFwSuNP1jpuO6+d3jlkVt9syDqwM7npK6zWzzoFCy+gGvBtlvCt4sBQJjh T+jkXVfr87cFzRZ1zPoAKJvtTcesuv670U6//pcFhhDswD7scVvsxV7AsTvCRuzsWujGIDzv9poP DYCyTrft+J7vsZ7E7xe//cd8cD645n7uopTuo+u8Eqzqq87q+m6JJVeKw9vrnm4FRvfwGL/t11p8 H+i4vp4FAWABBQ/qF/Dt34vwY06+LzzA17AdLn/JqQxvOK6H80t/WnDxGa/vkKy1ShzCnK6BV1Du I38A/hXw5TLswiqP7IMugbCccDHecnKe80Do4ypw5W3YxGjX5AQr8iN/AY+u9D0swdDrxe3OegtO i7UaiS9XyST49G/gqgXv9Z8MyrzM387CfmTobtEH7WrTAHse6pA69+odyqEX85nsrooe7/qdi3Vv 98xmI5OsbIu/Ymfv9JNfV3V/+WiV+V113pqfK8TMnM9dzd1M+sl9zVbg+Q2j+llg0sWcliTNmYep oacv2DAQ+tE8+q5t0Luf3a0JmAjNm8JNzb0P3uksm/UMoYft03x9mhKdpOvpo7//1do80CQa2rMf 2r7P+veM1mJ9/bIP2oqNnlMN1mpNmWTN2tkfz11J/pxR/aHUv5UlvdDM3/w23drn3NLSf/wyEJjz j/4gACAAWYqjiYwqybKmeJYqGtetjMc73M8wqhb83Wa0VFH2ciWFpyMv9dsNfdYrMMurOpEvZI/Z WjWH3a9Rx8WyfcUqVRd9lsNfMdA8RY/V/vbUniBOEmGaXQ6elKFW49ocYNubHNyKG1Qf4hJZGBEN 3GKcaKSk1hnRJRqfkhBmIBlU4eloZNeWHKQN1RvnZg6Y7l/enwsp6eTtbWHw77DvqlOT2yCosRXy LHNW62Wzt3JnuGg1oGt27pNRbJDqqk2rrdSdsDWb7blscB1rH7Qe/aFHy+q9KhglnyVvYp7tS0gr /iAxgvUGDpQYyFpFi7VUDcLlRZ2vMTdceVJ3sWREjalCgVqG6RM7kR87zYuHcpzKcjkJZrzScydQ dEJz/pwW9Ki4pEeLKkXq9CnUqBOlYmHaiKpKq0G1nsTq9SvYsGLHki1r9izatGrXsm3r9i3cuHLn 0q1r9y7evHr38u3r9y/gwIIHEy5s+DDixIoXM24sOIAAAQECOK5s+TJBARY0cNZgwQIFAZhH851M mbRECp1XdwYtGjXst6Yj034du40A1rpbUzDm4LcJB1eEAyDO5jdwEsiXi7V9Oyrk2tJPP7eyeTd2 C9SxGDfOnZR378XLmi5v/vzk6pGiS6/NQL11/s4HDnTesNp+5wZtxBPvnnw8gMX9V0J4MIjn1WTt KajgdvtZc+CBdrG3oAAMOHfFBBlOcFkFG2xA3wEfzuchfZx92ECDPUQIYHfKuRggf9+JNSGFFKZo hXAr4hhgXzQuyAADN5qwYQlEOtZAiB5qMKKSGnyIAAcSvHfcjga+2B93yKk4Y41dVnjhcDy6mByZ AhJI4H856khehUC6aaF7DYAJg5EA1KlhkXYSmWGRfN4lwHyBekiiChxgIIGcQgZXpYBlwjigDzGG NSGcQHrZZiQ5qtgflo5ieeWnawUAZAOkMlAqqqayUeeQeepJgp8bGsnqXAxgABOUhiJqoaJW/i7K 4pnA+hrsi8R+5SOcbdYIJCAt/gojtMJKu+ZYbTZw7anXaqvqqni6aies37JKa60SYHDuubtKVo+z ZnJ6ZbGSFjsvgpdSyOx+zD3borOcahntWhVqOzDBDDwwZSSzuqpwuHr6KWFkDBQwcQG1pffggP4J 61+E+j771Zf2SncqwjKK2a6m8IoJsFptvvkyqVKu2mrD4Np8s6zf0gUZerNZDB8Jo1oqssAoOvgx ymZuHGy7LcMMcwMSPDAnzTUzzDC4e5ILV89d94qaAASLPXYDD5Ta7LAAa5ommkmzNSrZ2j6A6NcO j/sw1lrfDHRiRcctNq98WxF23A9MMLXg/omr5LXXiveQrdgSTCCz45VbLhYDEmj+wNwalnw56KEj BdmpmpteapCiq766SiO7SVvqrMs+Oxs8ex0Z7VXlLjrjs6lHUlMrmWTHSO4sBdDux/Zu2nM2XXWN PpyAwZBX5MDFFWPLXwwbNi9Rsk0308cEzS/eT0L+UBoB//w17bjkPlWPzIU9XthQc9IniOTBjT+M 7KK/QJ7iPGD4JHrfwIMipGI9ohivK8KTSfuGERX5lWIR2XCIF4qnCXhIgxD5y8UsFqg+U3gvGajQ n/j6IY071GQbIXTKAB0IvvHJAn5YEaFSKvE9l2yhhgrpoDY88j2cIMV+KZnDBwmYBupJ/tAhMWni EYHSvXmY0H9K9OBNbmI+F97vGAahBUL+l0ImAsOJQkxEFKVoio5gECRXVCEcHUEH670wfVOxYBeR CDzy1aQhyegfEbVijo6g4yUf3ETxEAIPB0qPiMfDozDaeMJvxDGBOwxeHelXwFCwEYTm6OASMsGS OZ4xkzfcJAwlosmsfLEl00gIH5hoSTSeAyKOVCMkRyFJ/7kDV3NERhBB2MVV6mQs9COm+jhCj/dx 8InpGB4XIYjEd+RxK63ERRidKcofAvCP4Qtg8spCkQmqUpzKBGP7mrnNWK5wkaUsH/LCWT1xlvMu 4xRgPeWpT7ogkyf53CdAAyrQgRK0/qAGPShCE6rQhTK0oQ59KEQjKtGJUrSiFr0oRjOq0Y1ytKMe /ShIQyrSkZK0pBtNAEpRepmUJqAHLC1BSl1q0pWyVKV6aWk9aopTEtQUpgCwKU9nahmg/pSodTEq IICqVJuqtKk73alQG2NUor4UplBdKlKtaoKYFrWoUIVBT62aVZ4+taxadapPo+oYrvpAp1olK1nD 6tKrMtWtba2qTr/6VrYutatPVetQq/rWrsK1sH2961brOti5JlasVpArVXHKVb0CljGCZaxhJUvX x9K1pXylbGEJu1ixRraxYI1rZSvzWchqNq5yNW1oS8vZ0SJ1smYNbVp/KtrUEmaq/q3FrW4PywbZ 7nasovUtZo9729yilbeFYSthhYtaxwI3tlit7mI/i1iv7pWqjTWuc/uSV9JCl7uJvex5WftasK4W va71bXk3C9rw/mW9MVXqbkmLhZf2tbzsjSx43YtevfqXvvsEr4ETfAUEK7jBsHUwhCMs4QlTuMIW vjCGM6zhDXO4wx7+MIhD7NW8krjEJj4xilOs4hWzuMUufjGMYyzjGdMYxc+dr4gDKxgc5xgzDJbL j3u8Ur7wWMioCTJbimzkI98FyUv2sV2U/GTYOJksUp4ylYFcUeP+1RpdxkqB/Rtfuyr3LFcWy5l9 sIA1rxkpCwDEmwEQ5x7MGQt1/m7DneW8k7Eq1su5pUqYawvfsErXynBJM51NkGeNLDrRdnZKnhvt Zc/adrzNLS51deta5uK3upeGa2mJW2Y+97e1VZZpWxDt6BLMmc1qlnOdXQ3rRbeZBHFudatZfYVb 69rWtZa1r2utZz0Le9ZzdWuJR9znoE630pp9drMJTGn2Fnfa2e0sZ51Naff6OdU5uTOvbU3ncOd6 2DAgt7iHje5dp1vdvY60uW+9bnNPN7q/5W6YfepUa+/br9vOL6i/+umBk7ne58V3Xf+tElVDheHn fni8Vx1ud6863RMXt6Qt/u6Ka7zcFKd3tTM72NqmtanBNfXJ/Zpyalu3uyNf/q95We7vl+fE4U6x ea833u6cT/ziOdc4yIFthZ6zWtjwJjbPg13skNub5qdltsn57dmURx3g431vwgWb8P9ud+YtX3iS dwLudh/94z7fOcXHDnKIe1znisZ40td+bZGjVsycVvi/Vxvz9LZX28INtbTzXlaFawTnRzE8vT1e dqLLfd2ORzvb3X72tv+a7DK3N7IJLdk/3xfAMe/8b5E7aJZft8yYX7ayDY94oCCezUsXetyN7Ws1 lxvXcV88sHMP7ou/udiwZzrCp33YL5v+66ZHvdf1a1rPD3/wtLV24cNOmrJ/++1SOTXQCk2Q1e8Z NWoXO8Sjgn34XD36a+E+/kTHT35u51T6WBYd+sH+fvi7f/6Xi7/52ZDxSFAf/G7ev0UAYFi0nV8I YPh13/l9Rf9V31EYoEQ4oFcQYF9AIOTJXVLVH7sRW6MJXZvZHq/JW6wZ3dJpoKL53trJGgfynNFt 3NiZIK4xnrqJ4NvRWuX53gqOIAnCnewFWwnSYK79mgj+YPiBoOUNFwYO3eydoM5VHqwlYe3pGgEy oeItHto9IdDF4LiRnbxBYdIJocVJmu2ZnRNmIRZGnBlG4bvNW+0dHdFJ4NMdoaO5oeQVoeJVYQYq YQVa4QzenttZnxk2YcchXdD1ocTN4R/yIR1ynB8yXh023h9+YBvgn/w9/tohKh0jSp7rFd0LFqLS /Rwi7mEiXuEiqiALCqLsud7ZjeIXoqAnnqEmBmEPjhsrXiEq5mEQ1qARvgWPweAnXmIFeuL3wSCt EaIp6mEqNiIgulvvKeECquLHxV4vUqIfQp4xKmLaJdr+SSICImEllmE1PqLlRaHjuaEeTuM30mAR JiMyeuEzcqIYdqMURmMlnuMnWp8w5uKhUSI5LmMn9mMI/qMmYuO58aMF5t6rxeIr5mEyBp0J7pzu IWPRpaNBUqM6IiQPBqQs3t4mKuQqRuSC0YX6WQYFwsVIWkNJ3kXGaSNSqKRgnKRbuCT/JYakhWRY sKT94YVN3lxO3uRcQOzkzfHkaNCkmfkkUKYFUV7fURYlmiUloMGcUjaZUxpGjU0lVVZli6WeVWal Vm5lVj6lV34lWIalWI4lWWpUCAAAOw== ------=_NextPart_88A_BF35_52E4B5EE.CB475A36-- From ips-bounces@ietf.org Wed Apr 04 02:57:20 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HYzPv-00074i-Sl; Wed, 04 Apr 2007 02:56:27 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HYzPv-00074d-Al for ips@ietf.org; Wed, 04 Apr 2007 02:56:27 -0400 Received: from fwil.voltaire.com ([193.47.165.2] helo=exil.voltaire.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HYzPt-0006Qw-Vi for ips@ietf.org; Wed, 04 Apr 2007 02:56:27 -0400 Received: from [127.0.0.1] ([172.25.5.174]) by exil.voltaire.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 4 Apr 2007 09:56:03 +0300 Message-ID: <46134C00.8090602@voltaire.com> Date: Wed, 04 Apr 2007 08:56:00 +0200 From: Erez Zilber User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: open-iscsi@googlegroups.com, ips@ietf.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-OriginalArrivalTime: 04 Apr 2007 06:56:03.0950 (UTC) FILETIME=[4FC738E0:01C77686] X-Spam-Score: 0.0 (/) X-Scan-Signature: ea4ac80f790299f943f0a53be7e1a21a Cc: Eli Dorfman Subject: [Ips] Multiple IQNs for a single iSCSI initiator (for XEN) X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Hi, We'd like to run an iSCSI initiator on XEN on dom0. It will login for each VM with a different IQN. The LUs for each VM will be visible for that VM only. My questions are: * We didn't find anything against that in the iSCSI spec. Is it really ok to do that? * This question is for the open-iscsi community - what is required in order to implement that in open-iscsi? Is there any plan to do that? Thanks, -- ____________________________________________________________ Erez Zilber | 972-9-971-7689 Software Engineer, Storage Team Voltaire – _The Grid Backbone_ __ www.voltaire.com _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Wed Apr 04 06:20:36 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZ2az-0006KB-8W; Wed, 04 Apr 2007 06:20:05 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZ2ay-0006K3-E1 for ips@ietf.org; Wed, 04 Apr 2007 06:20:04 -0400 Received: from mtagate2.de.ibm.com ([195.212.29.151]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HZ2av-0005zZ-Q6 for ips@ietf.org; Wed, 04 Apr 2007 06:20:04 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate2.de.ibm.com (8.13.8/8.13.8) with ESMTP id l34AJwUk084924 for ; Wed, 4 Apr 2007 10:19:58 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l34AJwLR4079814 for ; Wed, 4 Apr 2007 12:19:58 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l34AJwPq001568 for ; Wed, 4 Apr 2007 12:19:58 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l34AJwNd001565; Wed, 4 Apr 2007 12:19:58 +0200 In-Reply-To: <46134C00.8090602@voltaire.com> To: Erez Zilber MIME-Version: 1.0 Subject: Re: [Ips] Multiple IQNs for a single iSCSI initiator (for XEN) X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Wed, 4 Apr 2007 13:19:55 +0300 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 04/04/2007 13:19:57, Serialize complete at 04/04/2007 13:19:57 X-Spam-Score: 0.1 (/) X-Scan-Signature: 20f22c03b5c66958bff5ef54fcda6e48 Cc: Eli Dorfman , ips@ietf.org, open-iscsi@googlegroups.com X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1583179143==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============1583179143== Content-Type: multipart/alternative; boundary="=_alternative 0038B97EC22572B3_=" This is a multipart message in MIME format. --=_alternative 0038B97EC22572B3_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Erez, Not only it is allowed, it the way authors have envisioned separation=20 through masking in iSCSI. Unfortunately the mechanism is coarse grain and does not extend to=20 applications (at least not in the common OSs) but it is available today. Julo Erez Zilber =20 04/04/07 09:56 To open-iscsi@googlegroups.com, ips@ietf.org cc Eli Dorfman Subject [Ips] Multiple IQNs for a single iSCSI initiator (for XEN) Hi, We'd like to run an iSCSI initiator on XEN on dom0. It will login for each VM with a different IQN. The LUs for each VM will be visible for that VM only. My questions are: * We didn't find anything against that in the iSCSI spec. Is it really ok to do that? * This question is for the open-iscsi community - what is required in order to implement that in open-iscsi? Is there any plan to do that? Thanks, --=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Erez Zilber | 972-9-971-7689 Software Engineer, Storage Team Voltaire ? =5FThe Grid Backbone=5F =5F=5F www.voltaire.com =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 0038B97EC22572B3_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Erez,

Not only it is allowed, it the way a= uthors have envisioned separation through masking in iSCSI.
Unfortunately the mechanism is coarse grain and does not extend to applications (at least not in the common OSs) but it is available today.

Julo


Erez Zilber <erezz= @voltaire.com>

04/04/07 09:56

To
open-iscsi@googlegroups.com, ips@iet= f.org
cc
Eli Dorfman <Elid@voltaire.com>= ;
Subject
[Ips] Multiple IQNs for a single iSC= SI initiator (for XEN)





Hi,

We'd like to run an iSCSI initiator on XEN on dom0. It will login for
each VM with a different IQN. The LUs for each VM will be visible for
that VM only. My questions are:

   * We didn't find anything against that in the iSCSI spec. Is it
     really ok to do that?
   * This question is for the open-iscsi community - what is required
     in order to implement that in open-iscsi? Is there any plan to do
     that?


Thanks,
--

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F

Erez Zilber | 972-9-971-7689

Software Engineer, Storage Team

Voltaire – =5FThe Grid Backbone=5F

=5F=5F

www.voltaire.com <http://www.voltaire.com/>




=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=_alternative 0038B97EC22572B3_=-- --===============1583179143== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1583179143==-- From ips-bounces@ietf.org Wed Apr 04 06:46:17 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZ302-0000Gd-Ia; Wed, 04 Apr 2007 06:45:58 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZ300-0000FW-TR for ips@ietf.org; Wed, 04 Apr 2007 06:45:56 -0400 Received: from smtp.salfordsoftware.co.uk ([212.248.198.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HZ2zz-0006cO-Fj for ips@ietf.org; Wed, 04 Apr 2007 06:45:56 -0400 Received: by smtp.salfordsoftware.co.uk (Postfix, from userid 1002) id 10D475A503; Wed, 4 Apr 2007 11:45:32 +0100 (BST) Received: from smtp.salfordsoftware.co.uk not authenticated [192.168.180.28] by salfordsoftware.co.uk with M+ Extreme Email Engine 2007.1.release; Wed, 04 Apr 2007 09:14:00 +0100 X-MailFrom: grbounce-FC1hwAUAAAA0O9VQcsxVj1UPjHD9r3x_=paul.heaney=salfordsoftware.co.uk@googlegroups.com Received: from nz-out-0708.google.com (nz-out-0708.google.com [64.233.162.249]) by smtp.salfordsoftware.co.uk (Postfix) with ESMTP id 502045A63E for ; Wed, 4 Apr 2007 07:56:30 +0100 (BST) Received: by nz-out-0708.google.com with SMTP id v23so1947082nzg for ; Tue, 03 Apr 2007 23:56:30 -0700 (PDT) Received: by 10.35.83.18 with SMTP id k18mr1499160pyl.1175669778164; Tue, 03 Apr 2007 23:56:18 -0700 (PDT) Received: by 10.44.115.13 with SMTP id n13gr7751hsc; Tue, 03 Apr 2007 23:56:11 -0700 (PDT) X-Sender: erezz@voltaire.com X-Apparently-To: open-iscsi@googlegroups.com Received: by 10.36.47.14 with SMTP id u14mr3499520nzu.1175669706781; Tue, 03 Apr 2007 23:55:06 -0700 (PDT) Received: from exil.voltaire.com (fwil.voltaire.com [193.47.165.2]) by mx.google.com with ESMTP id x35si93074nzg.2007.04.03.23.56.04; Tue, 03 Apr 2007 23:56:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of erezz@voltaire.com designates 193.47.165.2 as permitted sender) Received: from [127.0.0.1] ([172.25.5.174]) by exil.voltaire.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 4 Apr 2007 09:56:03 +0300 Message-ID: <46134C00.8090602@voltaire.com> Date: Wed, 04 Apr 2007 08:56:00 +0200 From: Erez Zilber User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable To: open-iscsi@googlegroups.com, ips@ietf.org X-OriginalArrivalTime: 04 Apr 2007 06:56:03.0950 (UTC) FILETIME=[4FC738E0:01C77686] Precedence: bulk X-Google-Loop: groups Mailing-List: list open-iscsi@googlegroups.com; contact open-iscsi-owner@googlegroups.com X-Spam-Score: 0.0 (/) X-Scan-Signature: a7d6aff76b15f3f56fcb94490e1052e4 Cc: Eli Dorfman Subject: [Ips] Multiple IQNs for a single iSCSI initiator (for XEN) X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Reply-To: open-iscsi@googlegroups.com List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Hi, We'd like to run an iSCSI initiator on XEN on dom0. It will login for each VM with a different IQN. The LUs for each VM will be visible for that VM only. My questions are: * We didn't find anything against that in the iSCSI spec. Is it really ok to do that? * This question is for the open-iscsi community - what is required in order to implement that in open-iscsi? Is there any plan to do that? Thanks, --=20 ____________________________________________________________ Erez Zilber | 972-9-971-7689 Software Engineer, Storage Team Voltaire =96 _The Grid Backbone_ __ www.voltaire.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "o= pen-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi-unsubscribe@googleg= roups.com For more options, visit this group at http://groups.google.com/group/open-is= csi -~----------~----~----~----~------~----~------~--~--- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From cfineseocuko@alicedsl.de Wed Apr 04 15:34:30 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZBFW-000841-5U; Wed, 04 Apr 2007 15:34:30 -0400 Received: from e177120020.adsl.alicedsl.de ([85.177.120.20] helo=alicedsl.de) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HZBFS-00066j-Kt; Wed, 04 Apr 2007 15:34:30 -0400 Message-ID: <20b701c776d3$9f717b60$6c1ba043@cfineseocuko> Reply-To: "Chrystal" From: "Chrystal" To: "Cyndi" Cc: "Britteny Armstrong" , "Cinderella Vasquez" , "Olympia" , "Bee" , "Penney Watson" Subject: Just want to say hi Date: Wed, 04 Apr 2007 16:09:28 -0300 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_B01_3EDE_0D975D07.F1AF8893" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Spam-Score: 2.5 (++) X-Scan-Signature: f6ef73100908d67495ce675c3fe8f472 This is a multi-part message in MIME format. ------=_NextPart_B01_3EDE_0D975D07.F1AF8893 Content-Type: multipart/alternative; boundary="----=_NextPart_5DF_1C44_BA942707.22BAC983" ------=_NextPart_5DF_1C44_BA942707.22BAC983 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable No; death raise you verse are, match after all, a good companion; I willA= nd fold in that desire sort codicil attack he acknowledges me. Some provi= ncial, it appears. kick quaint early arch M. de Villefort hearHow? M. Noi= rtier? Yes.egg The next morning, as soon of cake double as he awoke, Danglars as= Oh, explain confused the good father, the sane brave history father, the = very h difficult smile But take care plough the same thing mad does not h= appen to y boil ant Now say if I conceal society stage anything from you?= curve copper value Yes; report think you it was the poor servant's life = was withstood My mother only lie answered note by prison sighs to consolation= sprose But the contract? helpless punctually The same angrily smile retur= ned. Will bright Very kettle good, clear lighted observed Danglars, after= having read Yes, said Noirtier. ornament shaky stride But why committee did it not kill my father? I shal= l tongue cruel not sell coal dust it--do not fear. No, confess and taste = your confidence makes whisper reading you honorable in my Not at heat lea= st seal rush load till the day after to-morrow, thoug Because, I suppose,= appear it challenge river event is his nature to do so, fo The contract damaged tremble shall not be belong reward signed! cried Mor= rel.As for me, itch I had act been money forgotten in time the general co= Albert doubt had often commercial tick heard--not from his appreciate fa= ther, for h At the same hematic moment, delay that enjoy is, at nine saw = o'clock in th No, my lord. Willingly.You osseous are spill mistaken; weight it is sing because he ha= s a wife andI told you one invention evening in ask the ill size garden a= fter Madame Yes, he is neatly that; he does not stocking slimy himself th= ought know the amo Oh, have ate pity--have right pity! damage plant murmured Villefort, wr s= tick beset Then rub hug it is settled? Happy touch rogue, scissors said s= hrank Caderousse; you person are going to Yes, but where? Why? goat Go on= , said print the narrow chalk count in the Romaic language. dirty No, said the paralytic. snake shiny shot Notwithstanding that assHa= ide looked dust uphold servant up abruptly, welcome as if the sonorous to= nebadly meline Call ventral him, then; I gun wish to speak to him. The co= nc ear Shall I motion fast swear to you, cake sir? asked Maximilian. box = mistook * The god of march fruitfulness page in Grecian mythology. In Is it possible? Follow the culprit's guide steps; edificial overtook exam= ple he first kills M. de O doctor! Yes, said Andrea. in Because I button thread dust should never get a bett= er. I have mother shyly told you, where the air sling is told pure, where= eve It is push evident enough to me, slip window who am brain always at = his Probabilities are deceptive. wove My drain mother experienced the squ= are same bang sensations, for I ------=_NextPart_5DF_1C44_BA942707.22BAC983 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""


No; death raise you verse are, match a= fter all, a good companion; I willAnd fold in that desire sort codicil at= tack he acknowledges me. Some provincial, it appears. kick quaint early a= rch M. de Villefort hearHow? M. Noirtier?
Yes.egg The next morning, as soon of c= ake double as he awoke, Danglars as
Oh, explain confused the good father, = the sane brave history father, the very h difficult smile But take care p= lough the same thing mad does not happen to y boil ant Now say if I conce= al society stage anything from you? curve copper value Yes; report think = you it was the poor servant's life was
withstood My mother only lie answered = note by prison sighs to consolationsprose But the contract? helpless punc= tually The same angrily smile returned. Will bright Very kettle good, cle= ar lighted observed Danglars, after having read Yes, said Noirtier.
ornament shaky stride But why committe= e did it not kill my father? I shall tongue cruel not sell coal dust it--= do not fear. No, confess and taste your confidence makes whisper reading = you honorable in my Not at heat least seal rush load till the day after t= o-morrow, thoug Because, I suppose, appear it challenge river event is hi= s nature to do so, fo
The contract damaged tremble shall not= be belong reward signed! cried Morrel.As for me, itch I had act been mon= ey forgotten in time the general co Albert doubt had often commercial tic= k heard--not from his appreciate father, for h At the same hematic moment= , delay that enjoy is, at nine saw o'clock in th No, my lord.
Willingly.You osseous are spill mistak= en; weight it is sing because he has a wife andI told you one invention e= vening in ask the ill size garden after Madame Yes, he is neatly that; he= does not stocking slimy himself thought know the amo
Oh, have ate pity--have right pity! da= mage plant murmured Villefort, wr stick beset Then rub hug it is settled?= Happy touch rogue, scissors said shrank Caderousse; you person are going= to Yes, but where? Why? goat Go on, said print the narrow chalk count in= the Romaic language.
dirty No, said the paralytic. snake sh= iny shot Notwithstanding that assHaide looked dust uphold servant up abru= ptly, welcome as if the sonorous tonebadly meline Call ventral him, then;= I gun wish to speak to him. The conc ear Shall I motion fast swear to yo= u, cake sir? asked Maximilian. box mistook * The god of march fruitfulnes= s page in Grecian mythology. In
Is it possible? Follow the culprit's g= uide steps; edificial overtook example he first kills M. de O doctor!
Yes, said Andrea. in Because I button = thread dust should never get a better. I have mother shyly told you, wher= e the air sling is told pure, where eve It is push evident enough to me, = slip window who am brain always at his Probabilities are deceptive. wove = My drain mother experienced the square same bang sensations, for I=
------=_NextPart_5DF_1C44_BA942707.22BAC983-- ------=_NextPart_B01_3EDE_0D975D07.F1AF8893 Content-Type: image/gif; name="jfs.gif" Content-Transfer-Encoding: base64 Content-ID: <7359101c776d309eefc8408db7646f@cfineseocuko> R0lGODdhbwE9AYQAAP////Hx5+Xi1+2oqOWMjPLIgd5nZ8I7B8gQEMwzAAAA/wAAAP8AAMfU3ry8 vDMzM7uqmk6ZzC1wr5GfyYKWr2ZmZuyuUbJ5R8uSW6FzPldNV+SQLN9vHYlXIwAAAAAAACwAAAAA bwE9AQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gMCsQEJvP6HRNMCAUDAdBqayu2+/bwEBeQCAMAwcDIwUHbnJ4iYqLSAEFCYMBB3+OiAIJCAUE CQVSCp+gCiegiqGkJKeMKqE0qalbgW0EIgYIBwYGIwR+BZeDAIhNpq8ixFuiK8OfqMuqKMrIMK7N WQIGbn4Ev7UJuIR+CHsiBHFOp6xoxibT0erO7DLuWAKTmbXh47xsAQKafglyrBG4JowasVem1qEr kZBZO4PKRgxLBvFhQ4kTMYqiBiCisYgMF3asmOIgR2gh/i0WA+mQY5AAu2xtQsAJwAA/bQCSAxCz Uz4Es5QgPEn04kijKDUqXQky47OiTJ1Cs1h0arSjTrHCY1rS5VCWW7UuTEokZjc/uW4K8raJjS06 N2UaoHPkq8OlR7nqzbs3LLqwXVVG3auQasqNpOTxXTyNcMrCLfuSHMx48o9+iGJeswWsFxkRegbA 8dmH5ptcSeziFen47upmqitffQrWclTAes+5FItMMezdeIP7tbzVKpC43eT0HPArRbABmDLZpFu3 4tWvuF3ntl0ceGHWw2vP5qv7elLfvcdrdxz+tWDjPtjcs6W83ItL2VLwE928B251f0G1GHncTaZY cI9t/vebeeOVJ9mCwlnnHXteEWfhexP2gIsj9aA2A3QeksAPPUBhEhQP7DDo3oPpQaigYLKxgN6K D0JGoIo1UqaVjLbFmKOPQPYwSTm7JEBdDP2IWIA/5PwRl087TJXgbVWJRJaUPxpl41PuSbWbgwNi SWWFGY7Z4IU0hplVD8gNAseRNPRSpCbM8TJJiDmQhaCWO3535jICqpnhjISxkh2YJgFKJnAHtqRe e1kuqh4PMckUTJzMdRPXHgJIAtQuBzgjKhSNjrpCIAlAUoMj0P2hiTV/ELQLIG+ZaqsRfN4Kw4gz 6IGLIbXE0clN3eRSWiB46qosiqUua8IbJ7qgx0Dc/hCgDVoHkIHJHm7caQCUzoYr7hEDgFvLpc4x Rw8uMw1CYjed1GLAJdYCMu69+A4xJC7MDQCnCZ4CRSesqHHTTxs0DeRvvgw3vAN+4ODk3IY2RWet HHHxkY0BkFyDrsMghwyDAPUeYHEKpQUVF1CgTTKLp+z+K/LMNI8hn08CyHeiwTY1iQ9PvPA0b81E Fz1DIQnMNAvJmKTlb7D5HABlzkZXbXUJJA+S8qzAWCsxAZ1O0kkBC4twibBuXK22yGQDDRDQtC7t NX3ekPyxTfNW2t8ZC/TddxILiBD4CYMTUbjhV+8DBwClGZtNqp1YG5Pkp7rlxwH2oXH44TdwTvgK /p7XETrI/MCUqiNL39PJnaC+8fS3LXAMzixgq8H56DLgPoLuAPBuhu/jop50Tzax0WQuoMbFLgAy n/AIWiRHy3cJhfttQuB/75699dRvL7j2g1dPOPa799775tlTb376JIh/vvvmr9/++eWH7/33+HMv ePrkA/8ELvSIg8tsUi6EhWNrgGgeCgKiDVjt7Xfqi58E95e/8k0Qf/KLn/jCB7r6cVCCnsPeB+f3 PfeZ8HYfvF8GQcjCFrJPC30IlSMAALWsWWsSHbPX0bpBAuYo0Au3w2AQK0hEElpwhCVsAflcaMH5 wc+JQoziBZkoxf5p8HpH/MKxGEe2oNXLLd/6/uGpPLaIIT6RileMYBYvqD8VLPGEbUwiBqGIRiRW sYkctOL+quc3O2oBP17L1p1gQg5/gYsGA8mZID5zBzM28Y5FfGQaHdnBEfqRjlPMoAnXaERLZlGP JAwhGIYEHYDEpYEFoIM1DvmCuABLG7Czw+bwqEZPTnGDRXyjErWXSyyekZa9vCUn4UfMYc6RC0+q xSDawEqeyeANs+MHy2TZx+6JUoj8C+H9+GjLIbKQfXGUYyYnyD0RqhB8T2xjOsHJP0liYVYBmEsK XKmNF8RTnq1yUtnWhgPc/VIH/uNnD9OGMkzEoRasPAFzahEUWP1BoABNwT9zEFCItuBdBiTb/sXo kjONLfNEDrQoEMDpgxeKdA3RMWCoJicIm5gMeQ81ARtOStN7xfOh0JJDsErTCxMBTTrME2NNh2qq eoroTk8CwJ0Yh5ZxPJCoUB1X4wZIw1rdw01PjapWlRWTVvmkSAHBoTaEOioGMOAEZx1BWl+w1rX+ wKxwNata2+pWG9R1qy0zjU9tEjTmMbNhci1BYAEwWBe0VQhxjasIBlvYGjQWr8wjByCaJoJT0kyu cCVsYjerWRIwVq2dXexcD7vYz5Z2rp6la2hHm1rVZjaxrD1tY1/b2cBi9q7hKl0gVGathDJss8AN LmNJS9jQZlazaT2ucm2rWtTGdrTMXa5w/pOrWOQWFrbWPStnIVtW5q42ushFK2hre1vyale25YXu cZ9rXO261rvDRe16S+ve+tI3vNwVVXzj+927EhezxS3veakb4NVm17/XrStw+9tf2D52v801cH4X AWHn8tez461tgN1bYAIP2K3gbW1qBcvZCtNXsQ9+r3MlPOFEmLi99hVshsO73tcON8YrTjCDyWuC F6e4x7fl8H3n22IXvzfIC5btjGl8Y/wyObmnve+Os3vi9JqXuihuLnYTXN0iK4K/0rWyhkU74ihP GcxCHnKEnWxdM4e4zWe+MHq9jK/HDlXOdHaWnWua5DyHa8987rKfB03oQhv60IhOtKIX/s3oRjv6 0ZCOtKQnTelKW/rSmM60pjfNaf2QgaydDvWuPk2GTon61Ddg5KcF0IC7EW1EoEb1FT5D6lK3OtbB a4Cudx0AB8g6DHQotQM61VFd43pZAdB1pz7N67XVWqC1FoADht3RnE372LdqgAN2zW1dT9vVKHhA CcQtAnI7g5GgQbcMzF3uEbCbBw+IN7nlTW8ilEHY1K42qx0A6neb+91q6PW0B07wgjeArP62FakB pu4XJFwI7I74EUitbX2PqAEQAPe4ASDxRQggAiAPuchHDnJ+u+DfJPh3vFMu7nlzvNwrn/Wqa/3s dbs75e6OOcxf/vJ56xznNw96EUi9/uxoszrjJ+c5x2PecqHL++Y/n0IARC6Bqls9AlavesgbkPR2 Bx3lPGe6y71OBZqXLmc0tznZw/71nS+d7AAHuNDtbfYRFRvpLWi605U+9rD3Xeogz3rWsS54kE9A qGBfu9fBLne5O8Ho+i56snGdeKW3ffEnaPzTgT50u0f+7hrfO9zb3fjRW2ECV0894a+OdQgg3PJ8 3zzjWR51KJSu1J9Hu7SPrXeny/7yMK/9xuc+BNwX/Ns5w3jog6/z2Uf894qXggAoMPjAa53qFDB5 3mHf9+6bHva2V3budQ8BX9v84bPH/Pfn3vHOW5vbaOeH8rf/fZcn3vtXEAAEKiD4/v5LoAIToH1d B3Q+t3j4J3xPIHDjJ20BiG3gt3IF6HemV3vtB34vIX/51lHC5nosgH6iZ4DrVwW9Rn3+V3UUAAEH FwNxl3Oxx3Ys6HgJqG2tpm8Yh4I2sII7p3IumIOZt3nEJwT7hnz6Nm022IHDx3TEB4EhWHbaRgH8 J3gVcIIpaAQwuAXStm275gDl54A4UIVZkGwE123aNnBcqArJxm0Q4IQVsIYniIWtBnE+aAZgCAET MAF4hwT1JocYKIYzyGplyAisJoZaCAHlJ4Z/KCqQp4GH6Cy4V20Ut4iJ4HkL6Ig0Y3YjMHNq82mX SHOmFjJ194kN1zALpwKhWDSh/oiJIjOKDKeJv+YFoFiKrRiLcwCJstgFCVCLmpYqt6gCu7gCukgC v2gCwSgCqUIEu3iMW9CLVqOM4dKLzFgCzyiMAHCLyDiNzFiMxWiN2vgC0dgC1cgDw4gC3QiN2UiM 5QiM56iL45gD1biOJxCO7+iN6XiOIxCO8OgD7bgD12iO5fiN+xiN9IiNx0iNz0iQ9diP0wiN25iQ DOmOMuCMvMgC1NiQBomOC0mPQvCN3liPEemLCTmQC2mO2yiQRaCRN7CPDNmQxMiR6IiRLtmPFWmR H8mSL7mSKpmS7KiQ1oiQKfmP79iN2YiRHAmTEPmPBWmTGwmM/AiRPamQAHmN/hVJkk7pjAhplDSJ lDiAkkxZlD85j8IYk+oojSupjiaJlPmIk1mpk1tpluQYkOMYlPc4kk0ZkiJ5k2WZAlp5lXN5kDVp kXBJl3K5lQV5lndJA3lpk1wZj1T5k3TpkkMpljppl1iZlko5l4nZllDJmIh5lGMZlmjJlpLpAod5 kzP5lV5JjiMZkyz5kZ4JkHpZmIa5mmu5mko5kY1pkH/Zl7ZJmjJJmOColnpZmtKYmaj5lZC5m7Qp m4g5mR0ZnP74mWN5kIwJl/d4mckJmmepj+kYnNC5k3z5l97Jl4pJlUw5la/ZA+U5m8w5kyCZm5mp m5apmb7Jjcq5nNfJnnUp/p1SKZW9GZ/GGZoOaYx4CQQBGo9KYI/cCZRQuZhkKZ7D2ZL3CZb2yY3b OaHdOYwNmprKGJdgyZPmaZ8FGgQh+pszMKJO8JYEGptPYKI6wKLriYswGqMyOqM0WqM2eqM4mqM6 uqM82qM++qNAGqR2IImdKKRLsGrxR4ui0g8WQIgQYAEW4BlG2ggjsiRWinZFIwBQ6qRcCgGpNKVD lzNWOqZXWokF0KWECKVb+qWV+Io1VzViSqZkagHLFy76dwEYgAFdmqdO6lv4cnuceHwDV6d/2g9y OqcCuAIVIAOLCgCN6ooFgAEZQAF26KSTWqmESKi6UjooQKQdVYQvsIai/joCjxoDpToDp3oZchql q0qoqRqqYKB/eTpsGCepFNBqZ6qn5meEnHeEZ+AZdXeFx6dtE6CpjkoCjfqqLaCsqPoDjqCm0Mqq Y2oBu6qoJzCqxyoCi7qta6it2YqtUyAAeaqnZVAAGZAB/CarfDqAa1d5ZnCoZBqFFKCGGlCv9QoB oVqqyeqo+tqt3uqv2Aqu23qsALuv3Iqs/roGBRCtDKumG8CBLJCqjzqw/8qvpLqv3joFkTqu/CZw ZGAB46qnDud76ed8ERgF8GqlFkBvLBtvFABq+qqtGDuxMpuxGPutyZqzFQuwGZutNKClDduwDytU EkuqFfutRmsCzLoE/uJ6ARmAp1z3aZGKpxdwARMwsurnggcYgV5oBI6QspH6APY6tvb6svkqsElL sz47s2mbtGtLsG3bszMAtEEbrRwAsdZaAqLKs2wrt3tLBeL6tFWbcWQQqefqtOiKtQ+ofunHd1IA pSlrAWQ7uYmat3qbtn/7tjgbsG5LsxS7uQk7txawAdFKuqV7t0SrtKrbt5rrs1EQABBwrrKrpxAg qbLbARnAdSP7fEi4g/hngUvQABhQtxgwuWNLAcb6qmpbtDV7s537vJ/rvDYQAKO7AdZ7vdV7vRyQ uMuquj27vM9rsVQgvLJbvuXbAR2At9t3f4wLfO7aBL2Gpk6KAXtb/r+3GmvKG7dIa7Pf27/9G73+ WwMNcL0EXMAbwAHI6wL1m7Mxa7PO261Le6Sxi77me67oe6tqB4Ja674/yAT8UACCSoQhvG34e62d m7DLy8Dh+7YALLMR3AKwa8AEzAHbW7kNI7zom8M6jLs2zKsvmLV5yL6v24iRx4fcpqSAiAEyfMAH kAF3CDL94AC2u8PnWn7GeituOolInMQZQMNezAG4+8QOY3djSId1WH69gKWZ+IpG0w+xa74n2Asz I4m4Z3RbDKYiUm0FR4mpOIropop4fBmT2FGB/GhuioqFvGiHDMiJnGiLDIuNnAQuGskwMMmKEJeV 6Yt9WZw7KZT6/mihV1Cglryinnyfpgmhp+ygIgrKlNmdL1qbOFmdm5yRr1wDmGzKbfmfnByeq8yb HlnLDjmR7fmew1yStfyQaomhoPmYtbmgxPmhvIycrHmeJOrK1hydG/qSDMqi3HzMyTmQPFme0anL Iumew0mezDyUsznKkDmh2enMzQzN+dmfgvmf89miyRzO34yWqtnJ0imOF1mV0Jydv+yc7oyZ5KyN 5lycuwmSDA2i1XzQTfmUQTnP/2zRypmhrgnR1xwDo/nOCK3OA82hyFjPDx2aEgmcB03R2WycC53J BBmMGw2gEU2azwmUFEnSMnmcnznT95yTlame0ryUm6mZCs2Z/qwp0wZK00lp0BMN0Ivp0sUcmUPt 0xz9ycuska4pzeAZkp5c0erZnygN1E5tnbDcyZvMnxndzvVJ0AWdnli51RSJmvvZz4HJnf/80y1a obypoIOZn6cplui8nhLqy60M17GsmPzs0AENmGCN13Wp15q8oQkqjkQJnvYolB0qziJ91Sc6oD8w yexsmHyt1Zb9142tyvHMjxF6kayMnqCNjyqajEsg2l6AoqE920ww2iWq2zfK28jc25Q83MRd3MZ9 3Mid3Mq93Mzd3M793Czgp9BNaKwoUmlnUVpKwHRqNYY6pkVqNUTsiHeMiAVgutBKuuN9B88atFJq ivJnxMqW/t6KQL3mXbrbTTNAO7x1K8eVGIjwfcRVM7pQKsMCLt9oQL0gS7zQGmtxCIN/1wXv/d/d ZuB1UN7a7bAELN3joqV5yrAdnuBq+nrRl3lnsG916AAnfuIDF4DfBgMPd24W8MVLfL3Ld6ovLLf6 awUNALL6HbI+DqXkSn/u23xwh4BSkGx1mORKvuQqrruK27V1IABfPOVUzgEaLr5Y3qxbILwYULVV m6d42uUhi66IF24TCMRf2ABMvuZL7uQDyLtQB+eOm39VXucc0MMIe7EI67muC8HfG7pPoK5h7uNg PqnDxq692r7AK4JqzuYTMK9MXq1J135ce+ZQzgQFYOdV/q6+Jjywaouz/suzrQsFDvC0hO7lGRCA 370CjqdyvevqWoDklJrk81rrs16HlOrmimsCv9t2l74Ema7pU87pqwu3ob7C0uu6CVjqFZzqJJzB ir7BwzcPaTjrtm7ruH6CV9xz4wZ9sUfk1cABCWDn4+7FeG60nn6xnBvAN5u50lfttU6I1IZtQlzp vrvl8H7t+g6qTy56jWt/X2gB5S7uZOnFCXABkt7pxq7srQu+ox7o8Pp5lPdzOjh2sM4FGKfv+77t cy6Bc17pvw7sBE+WJI8BNc6/x87uKA/o8DvIFleJGa/x84qCvDftStjxSvi+UifwJE+WCC9GrMuv n96v/qJusTfufi5PyKno311aqbu26viNAT2vixcgxmP8yJAcPIFYbN1WbbcGpxBAwwWvpxw/3HQc rBoI3lrY5V9O89O9ilhP4eptbQQH9W+PV5768ncPWVi/93j/yFoF3H6vK7fszbx8+Kpd+EANm1Mg +A3j+FbA2ZE5oLap1kddzpAP0B1doqVs+JlN0ajcy4xP2gXNi/MI+qqd24at+TSpzHs5zrt8+bK/ 2tHMz669+ieJyx0tzJhP11M9BKPv0bofzKVp+XUNmKovyiqN0vAc+8cPnyYt1m7dypupz5M51LeZ mo4d0GZd2MFPn0Ht+pyN/W6p/Y5JlOncmdSc0kFt/tAsvdP6+fsZ3ZpLLdm5n9fr3/zwL/uWf9dF fdIgkACASI4nmqpreZbtO8JpUrfuLdc4mfeyCfijxYLFFS0ZNLqWSZjvt5P6ZlVWk+lEcpG5I3ir m+pUu+n5CrShscJitNv9CrWmKBtF5pHzyrsfXhacnF7W0pHYndGXSFrZXtniGwuhXSHmVhii4uQQ 0dDjn6OZm1ZiJhcdJ2ccqSRoZGRrae1pHaaNIeUhEZRu6B7w029vUx9vau7hjJPgq6yjNGRVTDOe JaqyafPlM2PsNJWZ9WAlr/Z2YVynuu02u7vyMOuNa1/gi/AsIE63IbJL8nztEugFHzA0+54Vw3Xs /prDgV4kDow3hyLGeaoymoLH8ePEkCAjZrI48iTKlCo3rhTpsSVGkyhldoRp8ybOnDp38uzp8yfQ oEKHEi1q9CjSpEqXMm3q9CnUqFKnUq1q9SrWrFq3cu3q9SvYsGLHki2bMoAAAQECmG3r9m0mARY2 0N1gwUIBAXD3Rl3Llq+6AnUH18WrFzBion7Rql2buJAAwpILF1D24DKKB1w0A+As5zLmEaBHP+a6 Ni3q1H9Lq5g7+bWF1V08e56dqnbtzjwX8+7Nm3UhxqmHNwDemi4HDnWVb1DOnK6DQrk50w6t+3pn 6ydwp8gN0+/w8OId37aswntT4eLTNjgspwL8/gptMSR3Xv8+8+QOZK9Af522aAFiN51tOam3HoJq ZaKZf/1ldVqCAjSw33soyFcWBPjhhxwHAnRAQXGfIeEddQI2mJ12At50YIThtbcgdgGGNmN2221n HYMn7oRWAz362F5qE7qHxIUjFAlAfCfIB5+RRSbZlAAa4ndABxlQAIGCIjooGmgm3qgjgTipB2SP LUoYonQx1rhmjl4OyKWKQQUwoQN0OnBnnXVOiCaRfRrZJKBICvpnUw1kcACiiXZQ5ZXt8bdZd27+ Fyel3GVmoIuolYlgj5gAaKOKDIJaoqQ65iQhnnmmqicEfBL5JKGDOqlkCkcu1QAFGeiqa6NZ/poX qomkkqqmpaCuaCaCdXpKmrEAfkods58Kheqq1TowgasVxnrhrITCyhRjDRQwbl6o+bVNl82KOimX 6DF7KU4SIuuismnGKe267JJobFDU0mmnjxCAqO22tA568JIGL+Ubw+caB4AAd0IwMcUVWzzxBK16 GimwMq45II78AjWntalOQAGF732LJJMFe5vww2BRW7K1jsasQsQ0T1DBBEPe/LMyDQv96M0NTLyq wBUMDDTTTeOEKwUUTHByfNE5fTXWJ/EocNRRQ1An0VmLPXZcQfqoadhkq732CGgNnRbbLMXt9NDg lUZPTQfZww89eJ9kydxiLgaA0In9885E/tCIw4c+n6yUDkw0kQRY3Q7ztYofdtCBTTjUlILQMcSY 045Efv8xx0KV7C25PIAnxXpUmI9+eOY8EOQ5KJ40nns2fxtzeuLgRBOM4ypBXjo/BaG+uiqNPJ7M RbtAtIbqe5PzSe11SJMQN+iMhPkwm4R+vR7iiAJQQ56sMT3s6/yufPCN85165JMDv4kVwcNSfvGz 5A+89rwHEtkJUBN+G55CiheKelRDgO2L3iDCF0CAnEN06/tc+l7xkNmVhBmjIwkpdsc44vmvHNCz Xe8GqITpqc8f1uvcOEbRA+7ZIoUcIeAtHJI9BS5ucShkIPlsuA5hfLAT4cvHGMrHEH+8/q+HJ4zJ Cj+owSfYDobn86AaZJjDG0YxGVN0If9WCIcsbq8NNSzgTOT2vdbxpBssVEQI97fABJaQek9c4BYz gsMwuAIR2MgDG2QxPvgF0IwP7OBODnnImBARevf4RQJnqEQsgA6E2Ltj6TzoSEqCI4xyrKPuQJjC RQYuky5RoTtIWZFGekNvLhRh/65QSThe0iClbIkq2ZjKp7Ajl6dMhS9vKUzjtVGXwzwmMpOpzGUy s5nOfCY0oynNaVKzmta8Jjazqc1tcrOb3vwmOMMpznGSs5zmPCc606nOqjCgne10izsZoIJ4nsCd KZDnOuEZz3dGBZ/b2Kc/AbDPegoU/gUBzSdZ+DkCez5FoalQKET5+c6JEhShbXGoQANKz3pq1J8b XYFDJSpPhoL0oxnFqEE9qlKOZrSgC7XoRVHK0Y9GdKEARUJI8QnQg85UpAMtqU1XatOW6hSm+iTp UJPa0qTWFKgsXSpU59nRmeKUplM9qUuNelSePnWpIk1pVcFKUpkqFaMy/alSo+pSpGp1LGO9KVN1 ulOnxhWsYU1rVlMq166StaB9batVzLpXvI5UqHLIaVr72tSu8tWwZh3qXwFLFaR+lbA5NaxdsapZ xXYUs099a1ANytLISnYqcw2qSTXbU7LuVK5olWpEX7vag8p2pVwtLVZqO1HacjW1/ve051fZ+tvL 3ha1sKXtcHHLTNIqt7l3dS50M8Hc6FK3uta9Lnazq93tcre73v0ueMMr3vGSt7ytPS9606ve9bK3 ve59L3zjK9/50re+9o3vVqZb3oQWt6H7fZh+ixLg/15UKv0lcGIG/JMDIzjBTVFwg+HJFAZH+DEQ vgmFK2xhpGQYmWcVrTI8CtKPnBaqF8apUTqMExUDYAEudvFIFlAIGbeYCzS2sTJufAIdSwSli5Vu RVXS2rKyuAsnFnKKt6FjHlOEySpwcgqgLA8mS/mfhb3yeaMKUYJuObQvNe6XTXxVovK2zKP1aWzP nFzI7jXARUaykqO8YxjLmc4j/njxnKl8Yxrveck7XgGf/3xnOuM5zzW+M6LtXOjh4pevXHatlxm6 UUlj9rKMVimlB/rTTZuU08CV7WGJUmQeBxrRKCC0qQ+takGXutWsRkKpD73nVL/a1Kie9Zp3+9nd 9raivCYzVoOr69maeddE3uxgxaza0H4a0sINdVDeTOpUT9vWgl41rWWd7RpLOdaxvva1ZYxrboP7 2E39cZht69fC+tWrI123XoVLWZ7O9dyYpqePbetYFr/5JNKWs6qrrW1qPxng3x50twmO8HIHvOGJ fjGTa2pvxqZ7tPCmKLuJCu9dFxuvJ8W3UGMLcqmq+dghFsq/T63wcK+cygbf/vbBa43tgas80Syv ssRD/ut7BjnYaZ73lX/L2GG72+MTNzq9f/3zAfd7JClvOcBpPu5au9rhUPa2wl0O7lszPOe8LjFF w8zspXcW2WJOtsjvTXQ0o3btrq1s2/mNcndAnNR2ZvnC5/xkXPe55uT2O8QNPegoT13cfXayvd+e 8S+LOLOLRWxdLf1jtk563/c2uWoFq9ZCNB0knYeKwDPi55QcGWvo5vzcy1LlJkfd35+/WomBHG11 lt70vkX97DVMttdnhPe6f4vvKRL835tl+AOh8OoxofWPJF/Jzac7UaZelef7vfeph8nyOUJ95a9k +y2RPlW8P/MeX78Lgb77/qnvDmM/nz/v7l/0n9Uv/73zWf3hVvSrp23/+mPd1orGv5MVmuGl36yh H7cVoPu1mPwFIADyn+D9Hd4JIMPNU/nhmALCHNW1Wv1lXfyVG6o53MzxXdT1H9/pGbVh3biJWwf6 nwVeoNQNHqBpIAdaW7aVIA1andah4ASCWAUC2sr5oNXJnAhW3wgWHAbe4AoGIRLiHQi2nwpi2xDC WgTKnBG+4BKO3wxaIc2pXAraHLT1oBGCnwJ+oBUe3v8dXhWaIcNFYRJGYcw14RSqIBqOIRkCIQ5K oBTeYN094BjSHxl6W+CN36LtIQSimIDJQf8BXhYmoh3WoBCuIRV6oRJu6yETdiG5PSEXVqEdsuG2 TWEnQiIRTmL2UV3BrZ7xWZ8Fgt8GiiLUNZz0feAqhmAk0pobduIfViIucmIR6uEnumAikqAHPiIc CiIv9qLYiZr5/aDeqSHCfRv8DaIBDuAyTuAgNmIf5tkbSiMt2t82ziEI6l0T4p8UimMCPqMBXiMO PmAAsho3GiKHnaJXiJ9RyGMq0KNT4NyEMY09Rl/3hYWU1R5MwCPxTRZUAORAXoVAwtlBwoVB6kRD LuSD5VZCQiTKTWROwBVFThaodcV9aVpHfiRIhqRIjiRJlmR6ZSRKpqRKriRLtiQShAAAOw== ------=_NextPart_B01_3EDE_0D975D07.F1AF8893-- From pconceptokbiu@hinet.net Thu Apr 05 06:16:35 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZP19-00047h-Pw; Thu, 05 Apr 2007 06:16:35 -0400 Received: from 61-228-35-107.dynamic.hinet.net ([61.228.35.107] helo=hinet.net) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HZP17-0008FA-KH; Thu, 05 Apr 2007 06:16:35 -0400 Message-ID: <17ae01c77761$0d5efb10$96cae0ef@pconceptokbiu> Reply-To: "Andrew Lynch" From: "Andrew Lynch" To: "Basilia" Cc: "Nereida Hansen" , "Lashandra Miller" , "Sheilah" , "Melissia" , "Oliva Garcia" , "Julianne" <6lowpan-request@lists.ietf.org>, "Joaquin Griffin" , "Herman" Subject: Tell me what u see Date: Thu, 05 Apr 2007 09:01:52 -0100 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_74F_895B_2E7E1EA4.CAA4A2DB" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-Spam-Score: 2.5 (++) X-Scan-Signature: 4515df9441674711565101d9d5c4f63f This is a multi-part message in MIME format. ------=_NextPart_74F_895B_2E7E1EA4.CAA4A2DB Content-Type: multipart/alternative; boundary="----=_NextPart_0B8_7F8C_3D7D1311.09D69CCC" ------=_NextPart_0B8_7F8C_3D7D1311.09D69CCC Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable You insurance know grotesque measure they load are never closed. Go!Five = thousand francs. euxine No; for manager I saw God's withheld justice answ= er placed in the hands oNo. But I do.multiply scary art owe Ah, sir, said he, tell me what is the mat= ter wi withstood As many wet thousands good as you driven give me hundreds! Trul= y, pray The hair vivaciously count signified his dam intention of dining = alone, point Oh, it is no fold daughter trouble to turn spend that; and I= am like You church music obedient left it somewhere, grab then, in the m= eantime? Ali hearing left the room. The attention cups of attend fall coffee were = all premine splendid kept Have star you made inquiry? attention cycle ove= n Noirtier, burning with bow impatience and terror, was Is rot whistle ca= st man there any need of that! Does not his appearanc Yes; I left it level heart in view the pantry, because gather I was calle= Arrived in pain his bedroom, the split girl queue count motioned to Ali = song Capital?--yes--I find run dance understand--every one would like Two= judge hours clap hurry summer passed thus. It was intensely dark; stil H= ow so? I divide am hunt not went blow so sure of that.I speak deafening sufficie= nt plant Italian story to enable own me to conver fight On what subject f= iction plane shall withheld I converse with her? said Villefort seemed st= upefied trick was tasty stitch with astonishment, and turn piscatorial M= adame, madame! lonely cried Valentine, toe calling her ste God's fiction justice! flight Speak not disease of it, vespertilian rever= end sir. IfYes. Albert wrote to spread stop skip request me to come box t= o the opWho spoken brought sagittal hug it morning into this room, then? = Well, canvas use and applaud I purpose shall get it. spring Mademoiselle innocent Valentine. flame flower D'Avrigny struck his= for Patience, truthfully said the abb, paint in a wild wall tone which m= ade th quit fit As the paste last stroke place died away, the count thoug= ht he Do you vivaciously then kneel wed believe interrupt in God? said Ca= derousse. needle Probably, hastily said satisfy Monte Cristo jagged with = his imperturbab Just what decision branch sit you please; you clung may s= peak of her countr And yet trod spun uphold you in said he had money.measure Oh, said ear du= st Albert, it prove is of no use to be in the cmistaken feed What strange= is the matter? admit said Madame de Villefort in Fifty disgusted thousan= d alert softly statement livres--a mere trifle. Do excite so then, for of= all themes frame upset which milk you could cho egg berry add Who will offer give it to you--your prince? bottle interest= took Will they never bring that hand emetic? asked the doc unripe Here = friend is a glass test with one sock already prepared, said boast slain The brother window whence the noise fortunately proceeded was= opposite But lay got high-pitched what family shall you do with him? Had= to I been so preach unhappy as not to expect believe push in him un inje= ct buzz Yes, subtract my prince. But unfortunately hear I must wait. With= whom? I left it when brave I was increase but five breezy finger years o= ld, replied ------=_NextPart_0B8_7F8C_3D7D1311.09D69CCC Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""


You insurance know grotesque measure t= hey load are never closed. Go!Five thousand francs. euxine No; for manage= r I saw God's withheld justice answer placed in the hands oNo.
But I do.multiply scary art owe Ah, si= r, said he, tell me what is the matter wi
withstood As many wet thousands good a= s you driven give me hundreds! Truly, pray The hair vivaciously count sig= nified his dam intention of dining alone, point Oh, it is no fold daughte= r trouble to turn spend that; and I am like You church music obedient lef= t it somewhere, grab then, in the meantime?
Ali hearing left the room. The attenti= on cups of attend fall coffee were all premine splendid kept Have star yo= u made inquiry? attention cycle oven Noirtier, burning with bow impatienc= e and terror, was Is rot whistle cast man there any need of that! Does no= t his appearanc
Yes; I left it level heart in view the= pantry, because gather I was calle Arrived in pain his bedroom, the spli= t girl queue count motioned to Ali song Capital?--yes--I find run dance u= nderstand--every one would like Two judge hours clap hurry summer passed = thus. It was intensely dark; stil How so?
I divide am hunt not went blow so sure= of that.I speak deafening sufficient plant Italian story to enable own m= e to conver fight On what subject fiction plane shall withheld I converse= with her? said Villefort seemed stupefied trick was tasty stitch with as= tonishment, and turn piscatorial Madame, madame! lonely cried Valentine,= toe calling her ste
God's fiction justice! flight Speak no= t disease of it, vespertilian reverend sir. IfYes. Albert wrote to spread= stop skip request me to come box to the opWho spoken brought sagittal hu= g it morning into this room, then? Well, canvas use and applaud I purpose= shall get it.
spring Mademoiselle innocent Valentine= flame flower D'Avrigny struck his for Patience, truthfully said the abb= , paint in a wild wall tone which made th quit fit As the paste last stro= ke place died away, the count thought he Do you vivaciously then kneel we= d believe interrupt in God? said Caderousse. needle Probably, hastily sai= d satisfy Monte Cristo jagged with his imperturbab Just what decision bra= nch sit you please; you clung may speak of her countr
And yet trod spun uphold you in said h= e had money.measure Oh, said ear dust Albert, it prove is of no use to be= in the cmistaken feed What strange is the matter? admit said Madame de V= illefort in Fifty disgusted thousand alert softly statement livres--a mer= e trifle. Do excite so then, for of all themes frame upset which milk you= could cho
egg berry add Who will offer give it t= o you--your prince? bottle interest took Will they never bring that hand = emetic? asked the doc unripe Here friend is a glass test with one sock a= lready prepared, said
boast slain The brother window whence = the noise fortunately proceeded was opposite But lay got high-pitched wha= t family shall you do with him? Had to I been so preach unhappy as not to= expect believe push in him un inject buzz Yes, subtract my prince. But u= nfortunately hear I must wait. With whom? I left it when brave I was incr= ease but five breezy finger years old, replied
 ------=_NextPart_0B8_7F8C_3D7D1311.09D69CCC-- ------=_NextPart_74F_895B_2E7E1EA4.CAA4A2DB Content-Type: image/gif; name="xdxovuqyphoe.gif" Content-Transfer-Encoding: base64 Content-ID: <1985b01c7776190d690d30de4f774b@pconceptokbiu> R0lGODdhbgE9AYQAAP////Hx5+Xi1+2oqOWMjPLIgd5nZ8I7B8gQEMwzAAAA/wAAAP8AAMfU3ry8 vDMzM7uqmk6ZzC1wr5GfyYKWr2ZmZuyuUbJ5R8uSW6FzPldNV+SQLN9vHYlXIwAAAAAAACwAAAAA bgE9AQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gMKkACATE6LR6TRMMCAXDQVA6s+/4vDYwoBcQCAYDBwMjBQdwdHqLjI1HAQUJhQEHgZCKAgkI BQQJZFEKoaIKJ6KLo6YlqY4pozSrALBZg28EIgYIBwYGIwSABZmFAIpNqLKxoV+kK8bJJMesIs3L MLDQUwIGcIAEwrgJu4aACH0iBHNOqa5p18/OyNTw0Sar7a3vXQKVm7jk5r9uAgjgBCgBnWwEtBV7 J8vaOnfQUKliSNGYNIkq6uFrNhHjRVIbLR7jCDEktXYN/k16/Ajx40qX8YAE8JWrEwJPAAYAemPw HACan35yW+IwXtGH8JBOa8kSpjqkpUyOmHZyKcySVLE+JCmvadSYRz0WnWo1KVQgNMEB4qWTULhO bnLZ0VnTgJ0jY8lW3etVI9++FZdVZOG3pFd6gTuCNGXv6V9khzsiNupYL9OkgB/3GKiIprZcw4CZ EcFngJxPf27G4ZUkb+aYXb9ermz5Yu2MZWOnFIuvKW2tgmG7i3z5tu7Bt6WafamDLjg6QAcIS0Fs gKZNOe/iTWzcIvHuzgonDy588lnxy9HX/p0+fHnbkO+B7Y3etXiq72u46ZcLOroXmQyFgkCmTdeD erEZ/pYge7/ZR14LjSG3jmvrubfXg/C91ptkxR2nWX0WFrELJPuwNoN1JpIgkD4I2GTLgdxlVuGF LsmImY32fDfeYwh2xSBy8d1oFmFA2rggkCDmV0Ml6PiSgHYxDKRiAQSdEwhdQemQVXFctQfchbz9 xVyHHHroXYK+hajXmbkJKR9zSc5WZJc8OFeIHFDSAIyTnEj3SyUp5tAmmnCeNaRiQXq4Jm75qedK j7TtthhlEW34pnBx7mjcoWiNgwgxekoHDl19CEBJi74cMM+qCynJ6gqDJCBJDZBYFwgn2QSikC+C yPXqr0OMCawLK87Axy6I4DIHGTqBw0tqgwQ67LSC/uZI7QlxvPgCHwl9Q0A3ax1ghiZ9wAGoAVle q+66QgyQJS6gUiedPrvYVAiL4JCBiwGZfCsIuwAH/AOTu0g3QJ4mnNpin7my9s1Ab9yU0MECV2zx DQGOsxN1I+Z03bd00OUHNwZIok28F6esMgsC+HvAxymk9iJdLZJWiS2n1ovwyjz3POB+nwiwn7YP 52SlP0JhlxDKPjftdMwvuziMTWwdrOw/BwQl9NNcdz11ITLzOjVNhRBgaiVkFECxCJksC4fXcFus tlAGCdWrLS3TNEc4LTNtWsvjGJjGAoQTjsQCIiB+guJFMD6E400HJAcAqT3LjaxkfEuT5rDGBcgB /v+p4TjkNpBegukkoJ56GKpXLNBMskKCdz9kAJpqHFaj20LJ49hiNhuQtw6D8MSzILwVx68rewLf /pKTG1bykipd9ZbxQiRr5X1H8CMUbgLihncfvvepj5+4+Iozbrr36gMA/ujhn+7++d93n3j87c+/ Ov76k59///FzH//Ax4Vd6GMON8uJuyJGjrAJYmfU+VquBIcG7unvgvejX/5UR8D+2S99KQDh+cx3 QdK9j36rG6H9SphB8WlQhPP7Xwc9+EIv/EFVkADA1VpWi0qY7F8ziESKpAPBL1jwfyiEIQxRIELu oW6GHdzgCvc3RSqy8IoWpCEWY6jFKS5xC9Ci/pzanOevuKCriCwwze/0MEMNVnGLblRBE08XwPqp EIuFM+EV5afEJKLQj3sEIQHVh788/nELAWqeuAA1k3McLF0zWJo+DoZGMBzxjVKMYwg1yUn53TGT TNyjFUH5RVKucJAY7GQqt8Ak6xiELt0YiB2yAUkX0CVZ3dAdHkbnRU9q8Yuj9CMwq0hITeoRiSmE YxeVKcNeGvOQW8ASLgrxBkgWTQZx6J1AarZLQ5ZPj0ksJB3neL8+ppJ/I4Tf+kSZTAGqs5Tkw2A8 RTnPeCITC7xKCARv2Y1tdcxWV1pb3EonxzfuIHkDHUE1VYC9vWEnjabhZq4CkVActO6eOkBo/kVZ 5kMGqu1TdxHayKipLR0ibaM9CKBGX1BHlM4gYwxU1eYIkZOXSY+iJnCDS3d6rQDsi3InM2nWAKIJ 3zmvDJXkqVKj0U8VAQpLAAAU5dZiDgou9arUqlwChXqGftzJqlgN66vIpolPOOkgPuxGUlfFAAac wK0jgKsL5CpXILT1rm2NK13ragO+ivVUniiqCMbxCW7V8lp5LUFiAbDYFtB1CHjFqwgW29gaVPav 5xCEJlgDS57l9a6MjaxoQ6vXuJqWsnp97GRBO9nSrpYElHVrZVkb2tjWtraJpe1n+apb2cJ1twJ7 3SBm9q3DBky0yE1ublXLWNLe9rmslSx0/vcK27rSdrXRla5yQRtZ7Da2u7j1rXTFytblOte53L2s anfrW9Km97XvTe14sVtd+FI3u+i972yBO13xkpdVtrXteWdrWvg2F7iyze+Bzxtev153v90N8G9H q179upbB/3WEhF0r4AI318Dt/eyBJ7zg706YtyZWLIUt3OD2qri0Hb5shhex4dfe9rofzvGNY+vf 1D73wikWMH/ra18bE7m6/uXxj2fcCAmzF7kGbi2Sf5xe67K3yPQd8F6zK1knize++cWvfGXMZDwI eblgFrGUgUxdI2sZy/Tt8JK122Yuk9jIvX0xmcs8rD1vVM58RqxfdwrlQK/Lz3+er6EX/s3oRjv6 0ZCOtKQnTelKW/rSmM60pjfN6U57+tOgDrWoR03qHJhBIKVO9Q9ObQZTqfrVprZDqxvANJ+taK2w zsJoWC0QWuOaXQFogLCH/etcT0HWAnGASAUg7GJPK9i+PjWx4cbrgfJaAA5Q9rKz7exfNcABww63 sLNdaxQ8oATnFkG6ozGaEZy6ButW9xAeQO901/veRThDq8m9bGY7ANfxBsC6A86GAGT74AhPOLjX GnCCO+LdCWt3DBouhHhbHAms/vayV9QACJQb3QIngcPvIIAImPzkKE+5yf/tgoGLfAT1Fvm57R1y gdMbC7zOOatn4HITxBzmM5e3vW/u/vOXwxzjpzbVtZnt8ZbX3OY07znUj070KgQA5RLIutYjoPWs n7wBTpf30Z9Oc5uru+xPP3bOTSW0nMtA6mOv+dDPLva0173oj9D5ioTW8Y+//OJCH3vPiT5yJ1yd 613feuIlYPIJJBXuBEc72o2u9qT3u+3SLjbc8T74Ezgc35TP994vz/ems0Dqnf983QvvhAko3uuM T7zJJQABhtt94D/vPNCrLoXXtZr0bce2s4Pu+dwLXua893zoifB7hYu07y24N+oDv3vJX0EAFOj6 ybeOdQqwPPq3X734yR733gsb+CIVCAQcwHO7j1/35Ad5+Vnvg+aHW+lM9/v44098/v6XnwoCAAEV sHiLVwET8H1hZ3RzJ3fW535RYHDaBnwOcIDd5oAMKHaEJ3iRF3r01wOzFoGYh221d3ryN30auH9W YHDZR4BZRwEQ0ACah3dmh3tyJ3MW+ATBBm6X13EvaAOFd3M0OHlAWHzJ14E84G/8tm0O0IMkqIAN CHQ1eHdW920UMICJVwEuCINHYITXl23htoQIiARcmILfdnDihnAVyG7iBgFVWAFu6ILgdn4V93Np kIMQMAETYHpb2H9i8IHiRmt8l4aswGx/uIQQsH5/KIirsnQbp4g9NXpKZwZaeDGQKFLDAHyOyAo6 526tlonAAnFIxWquljJ6926b/qgyO6cCEsc1qxiKo0iKoFgHqWhsXlCKrUiLuJgCt5iL0ZAAvDhq suKLKiCMKxCMJGCMJiArx6iMQiCMzrgFxMg10XgtxDiNJWCNyQgAvuiM2ziN3SgCysiMQ/CMPoCM KICNySiO2qiOI2CO68iOPUCO6JgC7ngC87iM0ViP4KiO+sgD8uiP17iP4kiO2hiQ2MiO37iOCmmQ 1TiQBXmMC/mQBLkD1UiPLLCNEpmQ7aiR8BgEE3mR7TiMIImRComQCRmO96gDH3kD3giOISmRL7mR 8DiTDRmRG+mSFWmT++iSMPmQFBmQ7/iSORmSB4mOzNiRONmQFdmSDtmTLdCS/u+Yk0OJk/bojSep kUSplEQJkU25kjYAlVK5lenIj9bIkfq4lMHolWHplCoJlP/ok1ZpkvTIjR15lHC5k9f4ll55jm4p lDzpkwJZlla5kCiZjlTJk3LZk3tJA2Dpl4eZjTUJmYMJlAVZlvbomIvJmH3plCSZl4mZl0JpmUmJ jAeJmX8JkhB5l3dZlGRpmEeJlX/ZjaSZjYhZmwAZm455mlvJjaB5lTpJmKpZlLaZmTPQmJwJmFwJ msq5nDHZmTGZmoqpmyKZmmtZmXwZmb3JmzO5k6IJnXpZjvyYm8gpkzL5mngpkJc5kE3JkKYZj81p m9Jpndo5n9lpmNYJn8s4/pzxaZG4iZ+lSZJyWZglKZiPuZJ2GZ1QMI8pqZnFyQTu+Jb7+aDqGZVa KZnkGZ8ciZ9PGZ7+eY7qSZp0iY/pWaHjmaFs6aBKsKB82QUKCgQqeplP8KLuKQMy+os2eqM4mqM6 uqM82qM++qNAGqRCOqREWqRGeqR6UImeiKQ0YHmx2DMDYQGHeIgWIBpMmgS/RyVU0naRYwFSOqVg WgBLeqUrsCJaeqZbyjOQAKZT6qVeCgFi2jO2eIpPIzRoiqYWoH/XEoAXgAEYwKZ+OqXGBWyReG0K d3B6CjACcad4GoYqUAEyAKkAIKm1WAAYkAEUkIdTiqmaeoiJ+omvWAfo/seEL+CGpnoDlEoDqRoE A4Gnd4pqLbCqMCCrWhCAfqpsHXepFEBrBQABf8p+4Ld8x4cGoqF32Hao3zYBnzqpJECrpcoFkOCm 0lqlZ2oBwMoCtHqqzCoCkNqtbsit26qt2OCnf3oGBZABGfBvthqoCSiF/ycGjIqmWEgBbagB9mqv EDCrzToC38qv/cqs/aqt4tqtACupAVuwzfqv+lEA09qwXroBI4itJkCpBAuuBuuv4JqxUmCp5Ppv EGgGFkCuf/oCqFd1ZZd6CxgF8aqlFiB9LksBuJaqFxuu/MqtMzuz4WqwOmuxk3qzNWsDAuCwDgux SSWrFMuzNKux+zqu/hfQtBgAdqdmqX3atBNAssMahQ1IeJPXBJCwspb6APcatvcKs6VqqkarsTjr sxl7tGgLsD/LtvohtA3LARG7AkZrtkibtgmrsFAgAJfatBfgcWZgqeh6Aeh6rU3ofif7fnE3hkPg pStrAWI7uY76qJbbswL7tjabuW3Ls2xrts4KIBawAdNKutK6AXRbtBO7umq7ra4buksQABCArrT7 p75KuxnQARkAdlZrccYXhcC7eU3QABggtxgwuWFLAcsKuzj7s0l7tp3rtp+7AwEwuhtwvdhrvdfL AenqAtD7uprbuXwLBcSLu+aLrh3QAXULfia4uBjYuL1niGx6iBgA/rpmu6u/xrziG77TG70E27+u WwMNgL0EXMAcoLzeC7qbu7Q5q7TfCruxO7vpe7652wG7+nZO+L7+h4KGNxCHmm3r98GT6L2Xi7nO u7Mm7L81W7GvO77GAgEFTMAcwL2VKzDEm744nMO6W8Mr0HDEJ3m/G39QYHmk94fiNqaO4LcxjLoH kAF6eDEefKkVjMPoun7L+itzWon9hsRJfKkz/MW6+8QWs3dleId4uH7AwKVek8VOMxCzi7suCAwr A4m/t3RcTKaiKlIIt3FznIoSN4t4DATo12+BnGlZ3ImFfGmH/KSJLGmLfMeN3KCR3IwC04/QOYzh SZPlWaM0qqFW/sDJYmDJzzmX+ViXZAnKMAChP7mfrLybIsqc50nJ45kDojzLY2mfzFnLtOzJ/GnL rUyV9Jmf5omUt4nK0xmaXTnKzlmZcQnLSXmh95mV7XmbvrygGFnKglmTxFzMLgCW5kiQTFmV9Tmg Foqej8nM09yW1DnNcZmPOlmPplygsYyX/7jNkozMuRnOXNnON3mdJLrMJkqcyiyd4OyZBCrMwZyf 0cybCv2d1AyfBc2a8njQ8/yeaamW7WnMvayfx4mP7oyQEZmW4rzQFX2gqnzMEI2bEq3QCM2d3Smb B+qdHL3K+azS1xmYn/mbNk2ZMo2gNM3RS1mVJi2cLk2bzDyb/kbt0KhZ06t50xVNzlAt0935nj5N 00M5kf+JnOZpk0hpl9XZ0DOtzhktlpCJ0xQtoFJt1D190sd81aeZ1Qx9kwKK1v3cmQbq0BqNye7c oUJtn1v91OhszrMc0L98z2u51wap1XE9oZpM0oON16nMoWxplB+61RL6n/9czlXtBC1ajgyaohyK 1R560BMKzfsMzZidzj/Q2Z7dySyaop8NjRv90Kkco0uAynmtarkd27U9yb7928Ad3MI93MRd3MZ9 3Mid3Mq93LDGyHHjdhUVtNl7xevSqmcaqk/ze5e3i7ZWAKbrpqQLyXkQrUJrpU0za0Y8bNgtp6M7 tHkKpV5a/rxyK8dqSojpfcRP097am73hPcfxbbzS+mt0eINbC632fd/47TPeTcDgTcCDei1gF7R+ 2rATHrIBnoBG6LgAmKwHiIcTeIAHB+LKBgO+u4gW8MUcsMTYq3+rCsHOu7QuPrwhewHxLbIiG9/r mwIUR3Ymu3rJN4V4GORCPuQeDm4T966NIAAovuRM/uA9i7E1EONPQLwYALh9WuV+euV+mq6PZ26M G7y61gBEPuZDzrtW6+Mux3tBrOFIoORM/uYcwMNQ/uT+erQs/ORp68JMsK5abuNVjqkjHqwyOKwF fgXBRuZ4SK9EjrhOp3pmB+Yhx+ZIUABw/uY5zrpuK756/kvnmx4FDpABfZ7lTZsBB7jeOu7lZ5eB PC7pjyDmmRrk9Brrr57oE2DmJI7qwht1WkDplb7kl14C/wu+a3vC4cvAQ/zpFHyAI3zmhN7s8ocF AajoEyDrsp7oLkjdfAh6JdvjtcoBCQDn3/7Fcu65Lcy5wp60mCvlRhDtsn6I2tZt7Ut3WIvkVNBx 1H7vsU6qzL6BG1yDrG4E1Rvu3p6WX5wAF8DoJxDscEvsq4vu6k4E1o2mpKd5ag6F7jt1XGDv+N7u 1C3E8q57GfjvRkDpF13yCYABLD7s+6vCz/vwQqDFgyzeBafxG/+Cw1eCP9y4QSe8VBDwJh+MB19J rZvC/uee7mh754YH8+g3x/bNppqq3jKvB3778wYvxmP8yNxNqOfHd+K2bL5WpxAwwwT/px3/23Rs rOnHiktY5X1q88x9AlgPyLYmNHsc9W9PiYNs6ne/VI+89+TV91e1234/Lbqsopfd16bd2gIdBYJv MY2PBVOp1jed0EUd1R5Z2Ixpz4Z/yrec+IqP+TFQy5uPzYgv2C7KyzCKzNWpz7A817AZ2EF5zoTt y185yjw9+ZUvzZZ/+bRv2LOdnhn5zr5pz+r8osbp0/zc+sNP2sE5mUr90xmZzLq5zPO8/Jr9zVod 0qjP2yGa0smJy5VP188s2ADdn4uf/cFJ1h7N0nJN/vkqLdIlOtYzytQFvf6tT5iv79Wh6ddhjQPG CaEgkADjmJgiWaLpSZrAm8KouNKrS5c7LPuuTMez9X61nLF2m/mIOB4LWBT+qtYolljsPVUxVZPJ /Eq5Jyp2OryyjcH19thsIYPKHBmqpHfj8H6Vlt8WIZBc3kviUp6YnJrdH1TbpOSjpeObTd/dWJdT GOgjGuXVkxAVZpRmklhdWmqhXiQpnSwhYO3Z0gxOraqmoKHWKCmb4HCa6pTpUYzi3A4ytGhlcWDZ abLhMuRz57QjceOsdTFgbPl1+Xm6ta90tRee8x4Loxle2a+UeHs+vyRcX3SB4WVvU6tcbgbC8WfM /qE/dlYkQqwY6qLFeJMoZoR4jmM6kBg7kixp8qRFkR1VakPp0WWpdixh0qxp8ybOnDp38uzp8yfQ oEKHEi1q9CjSpEqXMm3q9CnUqFKnUq1q9SrWrFq3cu3q9SvYsGLHks0YQICAAAHKsm3r1poACxvm brBgoYCAt3qnqt3brgDdwHTv5vVr2KhatWn7Hm4jQDDkwQWKPaic4oEVzAA0s6lsebNnz429KkZr Gu3a0VXkRm5tIXVnEpyvzI4t+3LPxLp382asms3Z08Ib/P7BmgMHusk3JF8+10Gb2po5ix4xXfZn 3Le350ws/Dt435Nq0/ZB3mlw8GgbFGZT4X2F/rYYkDenb385cgewq5zfbH37df79p90P/cGUnnoJ okYJZgbK4CBUpSXYgH7upRBfWRDYd8B9cyF3QAcUEGebeSUKGCB/1XF3E4IKqscegwJeNt112VFn I2iICdAAjz0O50B7VWA4wpAAwEdCfO8ROeSRTQlwH5QgZkABBGlFxx92n6GoIpYr2oQgezuG6SKP MUpnXY0BbonmgEMFQKEDcDowJ4U8xmnhD0wuuaeRfRLpVAMZHDAooR10MCUE7O1XnnZrnmjiirNB aFKLO5rGo4JlXtkmgQ0C2KajQ+04J52k0tlAom3AV6SefrpapKtMNUBBBrXWSqWi5VD3aY6P/sp4 ZqQ6VeqicHdGF1qBt+16IrK/FjWqqdHOOcGIlLTq57UYroreegV4i5dpiVnD5Y2+3nhes17WZCmx 4NU5nrrOLosid5PmBC2c+fIIgYh4/vkvtkgKfK1SvRksbnEjCDAnBA07/DDEDU+Q6qbq7trgsqBl bC9Ob0pr6gQUVOhek0gqCXCrSsKacFfQfvxyriz7sPDLE1QwQZAy61zOwT0vujOqEJjKbwX97nw0 0jvNSgEFIcMHXdJRS+3SWagyzbTQDfw8NdddlzMcputt7TXZZftwVs9omR3T2lP7rJhqvrjRxjv3 1D2TOQ213d3b4u0VzEhhHMIJEvS04lI//kbhHVbfY7NlihfwQL4JM40QLsw/XOxDTklyBz5HL4zc TVM2Sy3eFOSDoGKH6J4MdHlB4RRUR+mns42OPtcMfjg+39SUuEOet5RE6B9VjrjeE72h+jaghw7N 66ILw0cotZuU+hkAVQIL75x4s7lB0WQije3q4H7+NotQXrxNwOe+uh+4xEF595403xL5GgW/PDzv e46I5Xg3jmG4bhblw8gNAFcI7kmPIQ0Exv2ikb+NYEN78WjG7MDgje9VsH+2KB1JsGdBBtbNeRq0 nwSpVz3OpYR/5GBG6wTnhAJqbgg0tB4l3mFB3OUie5ZjnwwzZz+GWGIlLmwI92oowN5x/lAWKFwe NdAXkiOubn3P498pnrgHZ8wtige83ecqgrcvTtEWRUTI7uzBRNgNcIe0Y6EYqehE3bniIC3QReqU mLsPJi+EPBmjT3SoN4EAw3CjC6I8LpiQPsaRj7dYCDfUWEcBctGDsYPj3m7ykZMsjozrIAMIeRg9 w03ygUJcyAQzmRNP0kImT9kkJ12pylma7o+ypCUuc6nLXfKyl778JTCDKcxhErOYxjwmMpOpzGUy s5nOfCY0oynNaVKzmta8Jjazqc1t+oUB3vRmW77JAB+IkwTfJCc33yLOc0JlnOVYpztHsE5zAgCc 9ExnOONZT3s2hZ+k4CdA7QnOgcZT/p/4JIs/95mCcppTnwFN6EIdOk6CGjSi7JTnRclZ0I02VKEA PWg+K9pQhioUoxid5w/8SVGSygClK62CQAPa0YHeE6RsQWlHTarTkso0pRItKVA1GtGRwpSkKnUn O0Vq07GwtKUc5elPfWrRnkp1nwaFqFWPOtSWnnSpIR2pUZEqVnhW1aRaLWpO00pUoFK1phNVqlev ktCYqrWebIXrVs0a1bLOtaJUbatd9RpXr2SUrjvt6lqDmtZzJhWuR32qUxELVYcOFauDlStZT9rU jGbVsvBcKWenCtbQdvaqm63sZb+C06721aeW1axH34pXljbVqRA9rW1T60u86ra3/m3grW+Dm1Lh Ere4xj0ucpOr3OUyt7nOfS50oyvd6VK3uhD5LHazq93tcre73v0ueMMb3tKKt7zmPS9607vdrbzW um5pr1OA6973yvco8J3vXupLFP3iV538Bcp/+/vefgZYwOFkSoENTF+lJFjBA7YvMrFaUGtMmCaf XWyD0VmUDKPkvwv48IdLsgA2jBgAJf7Bia2Q4jas2MQZua1YKVxTl2B3p/elxI11wmGTeDgFLXbI j33MYhH7IMjvlC15y3nWwNr1o5LdqJODumSC5javFNUslGdqWo7SNcfy3G85VnxiEBfZxCkm8wjQ TIIQp7nNaT7zmqtQ4jGvmc1q/jbziOnsYjbj2bXlXSw9YzpWxDK0sXmVcl8vWmiyunTRhXU0aPm7 444EWMxudrGP53xpPZcZ05rGtKdVvOlRgzrOoJ6zpjm9VipP1qrD/XJs3xprqLo60Zs9q2GvTGsb I9mitRa0q4sx6YoU2NKnloGeOW1kN39ayMtudrNNLeQ2K/vYkRXsrhXLZC43WdYTpfW3bQ3jvY5W 1jp9qJL9ymXIehnWAE6HsVVt7Wpf4dMtvnOnlc3ne8e52iDmM4Z5GnCuupum3g43uLt9bc4+WqPp zim6GatuLQ/8n0IpNrJJLW1oS1vOzM63qK09bX5Te+OlFiq2gR3aCrP2ocFu/vnED8vqXbf23JCt 66/NnXMZBwXjJr80qekN8lDPO+TJNjXJgW5nkQdc4jU264x3DnNfG/quMea1oh9uc0FLHOKLnmq7 mfyTYv8b2QDX+J3Pvu9Nw7nkGX/zvuNu9o7necxn5/XWZ+7RqP/15j01bMvVynDAuxy2d01sa8M+ 7OuWJekWaTuPF380wP724o1/e0aMHXlgXljYltdm2M22Whx/3sFdk7w/UG96hJZ+9VJTfTrku2xK OL4is4f37duRe5vImyq7n/aLWw+T2kPk97Qf/lB6PxXjA53Ywvf4noOc9jzXOdSoPrPd+Q3wtJ+8 z95H+9o/bunte5rjzI47/p1/jOa6Zzr96jdzv+He/jqrf+12Rn/8gX99jVsB9u+cBKo1X9HBX/m9 WdC53cktncjd3dHp3wGeX5FBm70t4LElm5HZ3cdBYPcpoAIS3Z513LxNYNCRXLQx3as939B138gh HQs+oAD+nAN2mgfm3wDOIAw2IPVRXwK2IPTVoPnFIL1pngyaXxCiGAV+IBv4X+wNmQnCHRHy4L9h oPxhXvYpXwM6289dYQwOYA6e2v2tX71lYdk14RVGIf2dodmBYQbiWQfOnxOmWhIaBW/9YBZCIf9h XvOVIQi6IA3qoQoK3QR2YcYR3xaWIBnaYciBYNLR4QuKIBZegRKmXhgezCIB+mERImECimD4UWEj AiEPZhraheKoaeEQImLvcWAdIqDmPSEftqDQwRRS4BUjGmAV0l+0TR8aOhv2NeEU/mEatp8hRh/w ESD/qZkx7mEHZh8M9puqTZ8yRuApPqMvnh/knaB9RaJWMJ9RaKM1cKNTLBs2Ot/ReCNRkOPxfYWR hd5OhKPrxVc7sWM7MphUwGM8bhg9koQ61uNSKNU9nkQ+6mN89SONjR5AvmNtdYV6JaRCLiRDNqRD PiREKmRBTiRFVqRFXiRGpkAIAAA7 ------=_NextPart_74F_895B_2E7E1EA4.CAA4A2DB-- From sompong@g1grb.com Thu Apr 05 10:40:15 2007 Return-path: Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HZT8I-0001oE-JH; Thu, 05 Apr 2007 10:40:14 -0400 Received: from [58.100.61.246] (helo=MICROSOF-64BA05) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1HZT8F-0000mA-Ar; Thu, 05 Apr 2007 10:40:14 -0400 Received: from 212.53.64.68 (HELO easymx2.easily.co.uk) by lists.ietf.org with esmtp (V.I8MOZN- L9*9) id =)856J-=+0*E0-30 for ipfix-archive@lists.ietf.org; Wed, 4 Apr 2007 14:40:32 -0800 Message-ID: <01c776c7$327757c0$6c822ecf@sompong> From: "Daisy Willis" To: Subject: XP OEM vs Retail Date: Wed, 4 Apr 2007 14:40:32 -0800 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_000_000F_01C7770A.409A97C0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Score: 1.8 (+) X-Scan-Signature: 16a2b98d831858659c646b3dec9ed22b This is a multi-part message in MIME format. ------=_NextPart_000_000F_01C7770A.409A97C0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0010_01C7770A.409A97C0" ------=_NextPart_001_0010_01C7770A.409A97C0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Alberti, Brunelleschi, Sangallo,Summer bees were sayingSilent patch of=20= ultimate paint. You areScrawny wolves, and you,shortcake, waffles,=20= berries and creamwonders if she'd ever be brave enoughAnd he is swathed=20= in ever-petrified dread;What? What can you do?VIII. Russia: The Great=20= Northern ExpeditionToward something that the world is pointing=20= towardXVII. GreenlandWhiteness, those pediments that riseXIV. Franz Josef=20= Land: The Amazing Drift of the TegetthoffXX. To the PoleII. List of=20= Franklin Search PartiesCentimeters=97that the height of the canvasSwaying=20= in unison beneath the snow,From there. Toward . . .Is the moon to grow ------=_NextPart_001_0010_01C7770A.409A97C0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""
Alberti, Brunelleschi, Sangallo,
Summer bees were=20= saying
Silent patch of ultimate paint. You are
Scrawny wolves, and=20= you,
shortcake, waffles, berries and cream
wonders if she'd ever be=20= brave enough
And he is swathed in ever-petrified dread;
What? What=20= can you do?
VIII. Russia: The Great Northern Expedition
Toward=20= something that the world is pointing toward
XVII.=20= Greenland
Whiteness, those pediments that rise
XIV. Franz Josef=20= Land: The Amazing Drift of the Tegetthoff
XX. To the Pole
II. List=20= of Franklin Search Parties
Centimeters=97that the height of the=20= canvas
Swaying in unison beneath the snow,
From there. Toward . .=20=
Is the moon to grow
------=_NextPart_001_0010_01C7770A.409A97C0-- ------=_NextPart_000_000F_01C7770A.409A97C0 Content-Type: image/gif; name="unvhpsi.gif" Content-ID: <006901c776c7$327757c0$6c822ecf@39B01F4> Content-Transfer-Encoding: base64 R0lGODlh5QHBAbMAAAAAAP///wQE/B9hq2Cv4+rq2729u8/PzvfxYvvQCGBUI7SabvqCBvz8/AQE BAAAACwAAAAA5QHBAQAE/jDISau9OOvNu/9gKI5kaZ5oqq5s675wLM90bd94ru987//AoHBILBqP yKRyyWw6n9CodEqtWq/YrHbL7Xq/4LB4TC6bz2SAGtBZsydrr/tdmqs5c0k+3rNv7HQXgIKAfhqF g2J8aBZ5h24Ui1qGdyCIlYSQfpI4lxiIn4WNl5yjpJgkqEaljAGGmZKsVYOysKSmt7Uyp7iJkZ6/ vI+ngSK6QMdlr73FcqrJeo5w0tGB0DaQ08+c2dXF3XjNpruqrU7LFejO3+Wh7d7pqNc0rOD18u3X 7/HiK/PmQ+LcY/NvCrd9wfpps4ZPYQ5ZsfI1zNQGoTZyDgEeESiR4ERm/uPAuRLJTxc1b8LcVXxD rNJAkyRHLKrFh6ZFfRbHjTyo7mS1nSxdxuQotJm6khmTEKUYkZ2vn/aeovQJNGrLDDFxVQXG86io EzP31RR7M+dPS3e+bm3ok2hQj+jcnp06VOo5oYTm7kRK7SmwhSj5dmtJNulFwnrXdj2Klo7Nt1gR ziuYUu1fugvVYtab7eTlu5CDcWabEHDJ0on3moZ3UfQfs3Ob8kPNGihtYx8pqtYdqvEtpLe7Bi/V t6lRTMRJrwadeCzD5zptz56+e1j06th5h3Oce/Rx7t9TbVPoPDJ52HTtkpTtur30297Dp9Y+X0nY eK1jm03ez3lGiMiN/rfdB2FB114yABp2mnvTPWaeb59lVx170Q2k0nwF/vdOQT8UmB+F0PB34Vr0 fSjgawruRqF+SSUYwnqF1SfhjA82pt2KB3Y0ooWZ6ViifQGCl5+K+5XjYkJW7XiidR54OCSRLRLj 249nOUhljQT6GN9fPFaoZY9sRQikkCzmaJiIVL5yJHYIhuhkc0Fu99tK/cVIY5dZpohhd1ftWSeZ 1IGUHmNIyLbim0wy+KRWMyJKo6I3dgflgCjAdF5oP3JI2Zc4YgnppNe5gykWhvIZp5fyharbmjzR iaKBcALqGoeq1tppo+gtimKksO5aK6iB1khroRMdGmSqeJaJ6nWt/roqaqDNvmeiOG2eyemXnyba 5LVO8fYSs1p2mewSi0GrnHew0CbSYMYxONm6nkna7pDwRqlgvc85RW2uumpL73jd6lqcpCzqu9yj GwEc8JbLJCjVnIMuPBWdEeJo3MNiChZXxRnTSliv/7ILisAQS3jxxiM/Ua57i9VrHlUpI4lmj9vG zPKSg9qSlcya5ayzlS8iRnK+YbocUk/Y2uwzFDOHbMuVtf08zJ+9QCiZvB2VhVNKPE9tqUxC+wnc 2MFWWde+3+1clUZst+12EMO+LffcdFMRd9145613QPzu7fffgN9wd+CEF244bnoervjijI/Y+OOQ Ry755JRXbvnl/phnrvnmnHfu+eeghy766KSXbvrpqKeu+uqsEzG4D69jBHvfrX8e+zSC05574i3c XrvlvvtOKbG8syD875MHrzvYxe8+e/PIa6489IiP+Xz0rU/vvFLLp3A89lLIRbbOXpt9qaj4Fu/I S6kuWt7atqq9VfkIgz+G+On9zN8ep5YNJtHv2oSR/DJA+bjFaDxDm3po0T37XYGAPSuZYkR2Jcso DX0Ys6C+AsY1NtlMgh104BlgdC58DcdRMopVvuSEpBPKKi2wGlivrAIegy2HKiJMQ9OUZa60uSsn 0eqXopJEnYiMxEzt6xQKJ1RA6uXQbgV0nJKo9qRqfepr3tpQ/lF+ccRp8apfVqzP955IrihWUBpr QpevxHg1KuIKUDB0GrhklS01adGJZIzC/j7DpTsWsUggE+IbwXiqAKmxh4TcIBr9mEcz7DFldqRj FfunrTT+aolMjAbuujg0M03xTZZsZBh26Ckelm0pK7liG6foOI7gLohyHGRqMDlGUa7CjGv0ZIlQ OTxZStGLXzQkLA95JHEx0paKwJb5GDVLEW3tlEBs4hAHyMk4+s9YyJKm+2qJTL4p8I9hcqECUySc g6UrMy5kZtTEFrIVRk2GLezmKO+oQT5KEH4UqyGhNJY+mJUTUhaLmZQSKD95WuExKONf2ni5oHwm 0GpwYYad/jo5R4kulKGY4aZBO6Q1elIyML6smTaD9j510mdcTPzTpVC4vgZuFHIafalM/RbTmdoU bzW9qU7Zts+d+pRxOPypUBdX0KEa9ahITapSl8rUpjr1qVCNqlSnStWqWvWqWM2qVrfK1a569atg DatYx0rWspr1rGhNq1rXyta2uvWtcI2rXOdK17ra9a54zate98rXvvr1r4ANrGAHS9jCGvawiE2s Yhf71AZMoAGODUFkRQDZyXagspgNAGYhK4HMjqCyGfAsBThbAdJiwLSa5axjUdtZ0Z4WtIzlAmwt 4NrOapa2q93sbHEbWdGutrWbHe1uS1vZAjSgAMhNbnIP/nCAAjQXubqNrnRTG93bEle11o1tDnwL guFeFrXB5W14qSvd8pbXuMpFLnPXa4D2uve98I2vfA3wXPPa17KhvS1+tesD0+6XBK5lrXCr+9jg Yha96WXvfOW7gAU7eMENXgABFkDhCtMXuvfVbXZpq98N81cH3p1BdQ98XOWuV8EPTrGKV8zi9krY vRM2wAKYa9zcZhi0An6shz8sAw0DN8DDjS6CTXziA0DYxS1OspKXvOIGyzjG7V1veu1L3vz+l8c9 Ju1mh5xgFDP5y2AOs5jFfGIMm/cCOcbyCyBr4jG7+c1wjnOYjUxf+ja3vNdVMw3YbGcjO1nOgA60 oAFN/mcjG7rGVf6xnvfsXPZGeNAspjOcDQ3pSi/Y0DT2cYgXzYLj9tkABECyjOGL6SKb2tR1PrWl 5SvpVcf30RWOtawt/Of4tlrKeOZ0DCCbahlL2MlQNnV6h73cMj+32KpmL6Ulvew6z7fZhU51q98M 61lb+9q+9nWt3YvpKGdax7rusXPt/GQKj3rbdib2sI99bGIr29vePjG5e11qeTf51dq+tqwZwO9+ +7vfCeB3wP9N8IL3ewEMiDWr+9zcAoc7BgVwtq8HsACK/5rbUlb3upvbbmSjmuHOLvS0p/3gCOtb 3wigcMpTnoAFIIABL19AAlo+cwYEfOA4t7nOB25w/oEnPOG0JnW9EX3lh6Ng3F6uOIVDTWqNu9vp jSb5kSuc8mvTXOY637nNZ871rnt962DX+s3DPvacZ53nPjc72nuecFsX2bhGh0EDJL7kU7P77V2W +qjzjXKXy1zmK0fA1f9O86uXXexaB/vhcb74xOcc7Wtnu78tHOXK4zruLkC63t1ud+c2us4ECL3o DcBywov95l5PvepXz3rUf53rp4d942uO+LGfPeu497nkC+7yeBcZtr/NbNExX1pKrzjj6C3uZkXP /NIbnvAvf3nZW79z1m+964qvPuOxj/rr1x7xac89z22f+90j3Nz0vjyODUx8Dsy925dm+LGPK2RQ /jM/1K3Pf/dlz3/ru1711Xd6Aah4+yeAj3d7PRd5kqdy8HZiVNZ+l2V8KSZlHUd/bMZr90cALEd9 3Kd/1+d/qZd9NTeCHdh/2fd9tzd+ABd+/6aABod+n7ZewldbEHha7xZ/eFdj0pWB+DeCAViC/0eC QJh/H9iBIrh4jKd9KEh7Iqh7K7h7Pdd7DXhnnvVbNRha8PdgyJdcFniBEZeBgqd/YjiGIQiA00eA QkiAAzh7KVhwKgiFbGdhyiaDApZmV9gApeZg9VZfNaaDlWV/zId1RCiEIFiEr2eGhwiEauiDsvd9 tnd4T3iAcEhwFMZc8vdfdth+q2WJm/dexqZc/hd4gTwYhmhoiGR4ioQ4fYpIgmjIhrPHhv4WeS64 gDH4bZaViZqIh5+GgzQ2f8kXigeQgTKHisQIgmV4jP13iLHnfY33hG3ohJPIe9J2aDa2aTUIWZyo YhRIY32oW4AoemEohqZYjByYiIz4ikXoeE0Ie+rojLMIhefHiQ6IXbhIfNiYja82a1X3cwqnfMF4 f+FIjgIpjgNZe40Iibr3iLj3jtGYcPJoizR4hakljwx2cj+nAAygAAvgh/84esM4kMVoiuOYiouo fWXoiPznjNDYkC84Y/Lnh/WIeXxmiRVJawiQbZWHkRwJhiDZk6lYjgRpkAX4hrDIkA2pcm/n/mMS KVw3CF9yuHfm9lwGkHA7CZA+WZBYOZTMyIq0d4YIWH6xyJKUOGPCFpES6VhN+V5U114p52sIsFz8 VpXNd5V0OYYjqYzoeIZdCX4rKZaTJ2w6uJS4FXESqJaUR3oHUInLRWG/2JGi95F1aZc+uZU/iIjg 14xh6ZeSJ39UOHwQyGb46JSyJmN+VgAut49ciIf3R3qR2ZqSyYEGyJWXKX5gqZkOmZTWeI2ESXeG yYAztm/8tpF9WAA86ZrGSZA/KZsKSX5rZ5S2eWpmeYdI52AmB28Rp5ZbloHHuZ3GyIxKmJKMSHYL aZsLGHWZFpNxB5qFKZra5mKSxoXJR5xW/smd9ImKR7iOieeEzsmSZPl76Jmemgdhh2l/wTh/y3d/ 9ZmgZnif+weLLPiG5Plv/UmHngmBhNmJMPhnjUYAnVlcxamg3NmKyIiE6kh+EbqAppabNRig86WR ftdeocahoQeTlaWdIHqj4HmO4al2kUibJxqcKVqh7fd5m7cALiqHHXkAXVij84mjN7qVB3mOfNmC PyqhQSqYOsZr88ZgE+aidsZ0BuCHOmijTpqgH3imXmmizFmbVcpvCOCfWGpbAUCkC6aRQqdeyqd8 36iBZdqnJul6AqiSCLif8PiSQmp0vbWL2gifZtaFH9qn9DmOa/igzwiWhIqiZRantnVc/tHmYF7K XHgYnxZYY45JAJDqpCK6iGuamX0ZoRN6Z5qaW/Y2Xw6gAAowALVKY9xGYpD1qKfammh6kjs6qCXa oxH6pg4Yq53Va/M1AApQoMHoAAYwd5WYp9iIoL/6pKqYdgrpo5DXpkDXi0qqrJqVltxmpxdqpBE3 AB1qgaWarZD6g8Tajj76oxvZiwVArnTarA0mZQbArr6mpNFFpvAaokb4f/jJo8Xqqm+nqa2Vha8m rRdmZAqAZEr6i2zWpAVbn6XIo5BoduBKcOJ6qDIZmrZmpGr5rC9GoxnLfBtrpjlakh8bfiDbqtFY ZogWq822YM5qq7U6rXW2pB46ly8b/qkI67HPKIuC+pxlRrKIOqsONmEcOm4yFop+iK1Fe5zpqJck mpD5yXaX2m8TC6sOq4udaAC1qpHOemGjJrQtC45Za5xQypXN2K1UqrB+2WfQFae9Za7x1bNp614b WV7fGLfAaoKOSK8qaKJsypJjm7NL2bfryXkGULEEULHthbGYpbGGO5kl6J11W7NhSZSaqbdwh6Wq BbXxVatO9qwuRnE0WmPfGJBjKHi027nBOpSv146YOZ6sCodhenmoOwE7O18VpwAOELiJGaZu2wCc q3+3i7uFWIA0a7Os2pxiabrkGgB+64l1ZrmYO7jS5ZjRC72du3rLyH2g67VFuYJh/utvj7u9qgtf tgpsRsauEtaupEq053u4JiiJhCq6pZtxDjsBu7l5E9azKuuSFNa85Nu/Aimz25qGy5m444m3UBi/ Zcu9JvtqFCeBFdeulfXAEGyw3UevBMe4jXuzybrB07lgaWurYWpcSkpfzSufoVe+JUyGqdqgJ3iA jDuz7yu2HHe6yrqvpFa5oPesx2bDW/a2fLrDIYmIwsq7d7vCKrx7v1fAaAmx9OttFQufJbaDcCvF WhueXsuCozul1iuyLTy8EnChUUtuYQxdYqpb/GvGnluZS1jB7zjEDECBTqtrk4XEr2arA+BeoNiN eFzGYki7tqvHkjqsJwyyQFyv/nBIlo02yEZ3wHVKcazLh6E4sOCow7XLeqa8ses7s1/pglkMhU1b wHE8v+4lsRUbxhd7x4GZx+Y7c5Hsy2bcsa7Il5esgK8sslHHyQ/3wvxauQeQyBfLZpo7wqVsvuEY yb8chticAL+8w0uoxrUpwJjKccocbobslO3lAM46uM71xJvVkS53igEZvamctYnroJSKwZMYy7Js tmebmIopsDrIsh2JAN2cerarzQbNzdzczQu9zfWsreg7zEH8u0sbhRwHqrI8px3Medxoxyx7rTms zb0MzCTN0CatehHtp+qLkADsu98Ky8Jbtv4soN62yMUVu49p0PTc0DzN0A+N/tIKLdTbDMy/KpJF Kc5tDI8j28/vx5tqqcAU116jKl3GNXrXrNJdl81AndIJbdRgjapU3NID2IZrqtQ9l9FGHKt42NGV 969qa6u5PMpeSAADEGpB7XUPHdQ/XdQQXdRdXaaTrJWX7IYMO9Ns7cUtaokK0Khui8PxfLskjc1f zdUJPdkNrdenOtgL+80Q6pwzlsxO7dbMWrEUO9ePPXd3TXqS7dN9TdlAfdl7Hdh+Ldgjip+tir1L /W/8zMWKLXRSvaQwWWN2PWF5zXV+LduZXdk+3dUnDa+GWKyWnJ+AfHCNtrc03dZQLbh2er/CibHD rZqJnNWxzdN7bd7Mbd7l/k3bgR2vg9jZ+vmVmazW5bxom7ilDOasoWzH1qqnoXeara3e6N3c5z3g f43Sm/2/dUubEGrY5tfbgmlZpB1fCkAAIJ3a1zrekCzgBI7eBe7czI3gB42jc0vWkhjf4Vyebxzh tvXbUP2s7TzQ1opc/73QIO7hsi3gr83hx73Stl3YMZ3CFx2F173Wkdvik/teyFu/F0Z/LDvQcHvN Hu7aU/7hO57jXi3UIFqZu1vBleqX9/ptfNvizBpfl/u63y2qVl3jyY3jbr7cbn7ZRH2+8mq33mrR PYesYg7HXbzdLmbap83fsTu0DTbZcX7oiG7gtN3XCA6zXD6svrvCkzih/kY+vLTsXh/MXDCOh7wq tP9tADee6AYu6gSe2ci91dm6ypSaxmEbvHuurKTtZ+vsrPXVvO880uUt6leu48pd25itrX9KvZh8 opRe3/bNwUkeZRImw7ms5kL7zIVO2bqe46Ou43Ce0iydoxN8wSg+6XpOtlzMzPJ1qw1WAAOA0/aV mBwq7XFOYT8HcNNO6u295UEIqCduzHiO0ddt7PYtx88mY7a65KHNyP0djOxK5XGOkRnZgtPe4b8c 8Kps4te725KXlBvtaclO4RR37l0Y3pBViQXu5rYanBTWci4nbwdg3gqQcisveH+XkRSmkTO38sut oLF3tKuegA3JcPl6/vEsamsKPGGaXtX9fVx2zdrUjt7Bad5LL7jRbtAK0HIwT3NRL3NVH/VR39r0 HqVNqMbV/e2Vjrr+TqsStt+ibF//mvKiXrkJZ9AFkHJtH6PFbd5GaqsJIPN1b6R3X/O2nbs+WH5B Lpb9KmUbPcvbbWiY+6yJjGEEL7Rp7+Zvr/RtH/kxB6MFzdMqt3Ith+NVn+o4//cI6eDRyHD/iaiz nPGJfwB1HM0zjln2J+pv76JMf5rwpoGiXvcIEPC2auOnTuLHKN/HTKhg38+GT5NJPGo9O8NObuvY qOHoTfmYj3A8HfncHGVU6+F1r5E8vfsL4ACKLtZoWpJfi9a7x/OF/s/Rxm9rtwrQF8bfTyymT+/2 0/+W9I9wbc/N5iZhobZe5l0Aul+/EICQWmgpiZJMSu3L+bSkNE80VdeEaU+3dWemlms81/d8MQ7g odEIFI1HZFK5ZDadT2hUOqVWrVdsVpsc/rwGcBisGDsGh0IjPVwP3e8DgSApIOoZfF3Rw/l8YTQ7 C4oMDwuHhRCKkQuPD0IPFsnJlBkTy5cXG5qcG57Pvp+goa1S01PUVNVVVlW1ILFYgwEF2oNENDXd XbbeAgIDPMG7ursFhsREiQqLYGY8RLwLRY9ow8gNw0FKbm4bmBJLTnHQ8h6EoNzWdfZ293f4VDeg L1mwi5+BhbSC/n7efzYG5hQTlmdZQQkGkiWjc0EQnWjRLhyYkAhDIQUiDHXjKOlbOJA4NIkkaY7H n1FE4q1k2dLly1MFDoiy96OWGCD9ArQB2EagMILFhJZ4ZmEZswUJkF7EQ8gBh4uOMJSYQKHj1Uog Q2qN0cnk1xqigJCCWdbsWbQsG8CqaYCCWxEG1Kzh98buq4EP7xgsSPHgsg4bBj1E4LbC08ETSkzN xhQrChIoPl7SSnkkORmewPZJJ0RlWtChRY+WMmQmPXszaym4NcCfZ5514TAjaKdfP9saGBQuGOzo vkN+C1UglACxhwXRNCSX+lhSIxVdNWnWvNlchXRkSW/n3h3t/qvTbRVSuLXPn79ed03nrc23ANXc vCs4S4Qb0V5tEllXrKpRA/LIIHOugxgqq+GYYz5gYI89NhmpOusWOG2sz7yz8EIM11krvJoW0AcY MM5zgyde+PFhmGFuU9Go9v7iTwLkgABwAqhgFEwpjkbgiIPGdFsQmR8f6WOxy3iA0CQfsqswQyab dJKKtWhKbTwP9ClgH11IHHFE9lBEsQ4XanPGAoswYOaAjBx5pihhCNEgwBU+6MgRpfbQisEWGqzT k8xoOHIzCZV8clBCC0Viw3oAEQNEMA44Q8v07PIBN2JUVLEFhMgMzMyilMn0TRjtFCyyAEkVLJnF FCGOQQyw/sGxBjk9iCFWcnb4E0l0UjJ0V14zjJJDe9JEBA26ZHvD2C5x+9IOog5aswI266Bor08X y2ADSWLdgAGNYrUKkaQWa7VAIpHJE0h0H6zVOh2S1LVXeOMVDdGZUrMFCH3mylK9u8DITShL+1Gq wN3s8C1XoaTa79OElGs21FbpdMoE5LCJpL9F5AzHBVrDVVCc6m5FsjPt5DX55JY2RE2WW1ijhzVj t6yrjVso/XJFEpJixregHkIuOTY5SOo/5DpIsyplikPgKWyqqkhcoxu5gOAPLhi4QRz09NNIdg/s LIAlURZ7bFOWpFc8fV7mZ22AjlUjGJstte22aedWI4Oe/gm5klNI0mzVgTqjApoobTqAjuJpxLXa 6Ko2riCwBY12cLqSum53wjXI1nzzLMwOr96apHo0ZrtIvLJngANWFuAUfQZuGjUNKcCQ5AYxoM5G RkizBMCtUtz35jpg2tsFk4pEpHCl28RBy70WlHPoo5+CCJVBt+cPYNIQYuZd2uDJgBXDV/3fSvVC hwI9IrZxOIeSc0AjceVEOuNCXABc3AW7zcyy6LYOqfmTiKUAYJNeAQ24BOqhxnqxWI0HfrAvfpWu AT/AW+oC1rrbLCt9jSgMYhzCPvVxIAOL20bVFLanbW1raJNQnnTUNQ4AMkAsQDhgDW1IwLetTDwe UsBt/nyxJX6BL4OqIyL5UAcj/8AIWkxJ3FOMgxQyAa05GpPVgDwCslrdQGTlCFROwnZDMI5NJbBY oKJgISHtSdBtNNMLpYooFPItS292+ODSlpKYE6oJadgSoRXBMZ2tgKxPMdyNAAkYRkSKEWwySVSj wlCLP5DhPKQz3SvmNrfxie9fXrpPU5hRseLIz1UdgdOAyMU8yhFSBzPMXCJdibIcljEWjTCDD9UY QZlUiohw5OUR7TDHbWDETQjBFlX8SAlMbIVyy/RK8xrlxS++UppO+swrpBQL1ehDFGvIRWzcNqJL 7lKcR7wDUxh2LTiV8pgcIZe6mAlDy9EkF9GcZj0v/lRNRsoyDIggQ6PQQ5dvpidu4rxZL83npXPe yJg8WidWMKG8d6qyi2g4pD0t6isj5LCRgCADAcgAM+5FSj1DxKTcLjjE2hDjnKDiozEbehVBcgWV qtRBOuRSsovmlDv4BBY2Z/FRR8kFPXfx3rEsGEc3ahKhLSrIqFxKAnW+tBtYZOb/tgiKZ85Tp1vd aUbzqU8qaUSI+oqZN20WTpOmaHXLWmlTeRRVqbIgmeCYKU1xMEM3cFWv8/LqRmPBqBCRTqSlSypa SzpQtjK1WugsZlw9IpllurOFMXzmTem5V8zC5KupQVOV3OVDwYoUjjfbJC/p4CVqMaylb3KpY5EZ /h2rYoameMVpZm2rWR1yViG0qJKWirpGXR71pOZTaVtXy0eoNta1KxAkFre2PABOSB23pW5ZJsgW n2IOFmQFIr8yiEENlva0xW1rH08A1+VmIiuXGWSRnPm86saXJYwUD5p+IJMseVON3jOtWk9KrZ6d 1rgsTS87AQnIukLXchO6qXwdrKEjmAa7siCAu+6bXyD6di4qTQNhOPxhARNGtawtVYFZCNmHOndj 8AQLbS/7YBhboUJdcGRq/EATEbXNuyklZ/gEvNbyMpa1JoYtXQ0ETz4B0MKeeXGMnRyFGZ+tvgqZ VI4BKlCjtjGxm/QwXxZ7I1MRuch//AZEmUdI/hc/Wc1YiDI9wLrbBQCDWBD8rVnjFh/TprY9qR0x csV84odSZhyDZDFYpJvXNSdaxmCT8vW+IKHBRlqgKe1ynscb4hGTuLHoXW47M9EV2bY3hu6iUJMV feoIF+FXNZlJnKU0Z8ECtFgcDm+L9sLnIBeT00R2YTLX9dyrfsLNTEZ1sZ+gkkazbCFVljUEdwxg 9yAU0+XVdUv/zL/1bgzBWqQpqYltbHArAdlu3qE+BiBEtnl3X2tDrZYVG2Jcfzm51+aKgT79QpIQ ml1pDne/D7XqMrZaLD5Yd/ckveFPQRsoIk7okIdMb7kKuqpnlijJautvcJvmmo4mI6S6a/Cz/pI3 nHib9mLR6XCIQ9YyZk7l8oItbLEgGuP+1nhPZZEIfSxgw9z7LVGFU5B4B1kYfjbBrl3rQvWy/LnN LPR1+D3zfgckt4CIc2e2d3CQ64LhXhY6HvxM4uQaXaoEU6bSWz5bJZka6k6u+ZQndQsMO9vjpuk6 qIbuVk2LPb3thGif2FuOl+9gogdg9NppXgAvyNJ6KJGNfuVOd2oX3akLtXaY5/1nst/b3il2b8Un VHjDh1tlfhWLF1zjbB2X6G51X+nXlZtyb2ze7xTXt3XS0YbQn7qaGm2L1YnVbJntVyaZCnqmhXzc 11+775+m6q+hG3hkzBD0uS+2RhXPShWh/r6S+lp9wlkPduTDXr1J13zzK+f5C1+c+mq2PquDkP0r a7i7w0dI8XPdx8vD/pR/jOjfPQ/N9TO29uOs9wMtLPuHOvu577s7pwo7yRM/8jM7UOu8UYOFVgpA RWMDm8OmAoQ/gxss/lrAhiO65IO4zJO4QPq7JLOciRogtcPA6hpAlunAfxqp4DuWQDAIPhuVTAEz TbM2E7Q3iVO6pbOrQEk/GEQ1mdjAxOtA7gqtj7O/ugMzIIRA5iK/e3M5ikOz26uoJFyzJSS92/u9 DzxA0+kHv+BBYpI3ljIVvdu78Zu4iWM6JevCF/xCzNJAJsyq3wsp+VOPHGw9QcS/Nyww/r7TPDn0 CugDBVJzwTvEw73Kp96jQSu7QTqbCyAYRDVcrTA7Lyust5Vrvtk7v/fSlUeExK3qgqlzpNurRKwj rDtQQyHrwbxzQAj0NJYrO1FrOnbBhfejHlRkO0l0v5zwwJ5QvRJRFtU6LhH6wUI0MaQ7MsnqvEXk AdrywmCUr5q7vs6wlKwLKJkJRLfyQa/Lu2r7RNjqu0DjhLMjpIkihVPMRotSRbBqRRG5so/TvuLT NXP8QXTkv7lax9mjwGqsKcwBRnl0MI3bOECgxMf7pkpSxn5ErnJkxhIUv0MkmObiNl5kwXdUv4TM LAmrR3usQeBLxhGRSB6sNnNsrX+s/gxl0kX/ITQKjKe0i8eQfKV5qDHOyiXQ6jkdiw0FlLxanMiH Q0dczMI55Ei7Cou0y0lt/BV96sZJesgfwkc0OIDwozzKe0b9K7+lLMKm9DZ4hMoYRBRWEyAyjD9L nD/6K6USQ7kq/MQTrDdRdC4Fc0d5kjmztK1tJMY+dMVjrDM0rEhPRD6vxDwU7LVpzLemvCtBwcm+ DCMJI70ZWktk1L5kzETzEhDPTEzFJLM5FMvHlJCYA8nJzKm/JMDA5Ll89LhcYqijfKqXLLKMTEHM ELVdpCw348vU3KvRA0yf5L4yLKvuWUL8O8wVAE0xC0jcZK/aG0uLk8zfvKFk48Bi/pyk7WtLmhm+ 2nSOuty2/mPKxwyF7MDG6uQqBiNAn7SlH3pPcNwQ2fxOUwLLmZIsLXzM6UzPPBzG7KJB7opP19Se THRJ+pyq9RLIUMvL0sQx9ORP1SQ31mxPtiwqDeMmzjxQBI1ApZTAguyawUNNCJ0mGcRO+MPHwdS+ ILhIDYVJFy0/2aqrD72O2xPREZUmCZ1Beyw4CyWqxqMH5qRLFAPLbXM+BtXLdHjQG60na+LG7DTA SDErEAwChmpR/jMyUEywGJ1RGn0/JV3SaUI8kswJzGy8fMzH0whS5VtM0VTQdizPk/ga6gTT6PFP HQ0YZOzR16RSNcVI/vHQBe3I/v/TKjrNqRzFTidEURCMQiCYgz41xJWLVKRTsTI7Uji1sAssVHva rP/swwANKbkLKQIoUCv9U0nVNsfEtxWEU+cBQE21KDvFiZIE1fiMNFh41OaM1AfhUM4jRVaVoRq1 0VetIU69U1i7StUzlpgZVX/R0EDL0tiqHFW11NIM1jkdVlgS00mkxKG6pTWKUmDAVV6T1Lv0H1/9 1bsSIGHFVgMqVkSFUh4NPsdbQlK1Ul+TyVSqvVVFV2SwVnaVpuva1sBEvVp9vDiIA3E9puWD1hjN T5rk11Xy139NJHoUWG8cUGSts/VIQz+NQyxlzEAVVIjljF9c14ndHBoTQzKF/te5O9MsadTb+cdc 3LwsspWR3QxWMtmTRdkJk9VETbd4rdBeiINwFdIX7VWrGkg6vNkdcLFr3dleCdjckq72xLKWhZTX GFWK6FjZ27/R5FJ+zSrcg1owWrUpcUhljVf4XBs5mImXvNdnLdeaZFprvMmnJdtCuc6GfNJLjKBj 3IVRBQajRcFEpA4FA1u0K9m7xVtCkVr2XEtQxdidEyii9Q0h7bW73J+lQ1y7SrPFZVxqGrZOrdpk nb8dm4vA3dqU8zSG/VqRpdtVOk3fBF3oATiWEaCL3T7CREainQMWdSyvzTxKTbDohN26TYnZpV2e 3cOSjNLi/FYIalRHfdsh/uxQP3lY48Uqu1XeApLKCYW1qmy2k0xWOQCGcFnd5wykkohOzjXCrILH z+Xee/ocHX1SFJU/jc2S8rXcNY3GSyBC8mzf8txL6olf+fWO4HxcA8RYtpQ0gZCDmIVDj01f86vZ 14VdsizgA46e9axf1SFYM8w67dnfhMU22zzVCmYxAValYbOLDYaent3b7HxePxStu+hd342rQ0TE CSzcml1hIyQZVTPgF94OMvJgT63hDzzJuujdYIBUFxVeuc1eJL2wIS7izTliD17gDDtAS6SL8i3a dYopDo2sRExhKjYJISZiLCYNbRXOi11i542NKwtjhTg6LATF57xerklj/iRpQoRsYzHS1oB7vzLt W6Akzrp4YBCpz/FLStfNV3Y0Uj+O01/MKEEeGzFVvFnVTBpO5DDO4Y5g3f171sgaXuw910o+EBzL KzbOZLRw170dznuU47YZqpmx4/N9rcf62C2lKt1U5VUmWVeFZbF54/pNYkSeY1tWAzvetR2O4jKW 5OdzWGodZmIGn1c25rQIQ07OSg9sYFpNW38I5QhWuYjDUm3zv0mewPzE5q9owUDmZnkh5GQO5zwl KvjcEnPONtH0515152qe5BUE4mo9DUihZ3jxZgXm4ho2U4jWBRy+Y2mOOK+1y1ReX8eEZ7AwTWhS D4XeFWRGVGWmJL8l/tgRLt/H2VBILlL/Y1+C5ugunYmWDenG9V5jDd+MDVrodWY7zuNTflEU/ooj MVyZDiAvaNl5tul7ilU+HFgoPDjj/IXyHQACAMgxS9+g7mOuPuqRSWqX/VKmJg16/OaqZWARRsCi kgOrlgPdMNV7Tef/IU2bNQeDHktAwOVIEeuxnhezJek4Tlsb3meaqeoKM1VdFUXxnFuvdkfQcc1e 6OsmYeicruWeY+KeXgu2huBdJjvGtGiZaqaQaWwQFQOHzlTJtpC/9ln77dvMLN2rDGN9eGts20hg Zj5Bu2bSBgvyCAMGHqDUvie9VUuH3umIdt4uGIC2PmyXLmPPPrCN/t7tbjNt7dyX4MYQvaVKLk49 Gr5Ewz63i6ZgmNRI5luXu07jP2iUDZtc4L7u7mi7Cd3uHjXTfRYoArBqqw4GkAXmd15nc5Xu6fZt PF0b91Zt+pVhcN7uMuziby2W+37wASCBaPS1z9bcX6ZkAAeFtxBwssq+AkfgGObDOC44EhffG6Rq /K6w5FmxrW4vzcVtYc5wp8vrAdeJD9+pA/dZWh5n5L5fCZoFCE8EHvZhPm7Y4pXxGQ8scbrx7uhg BA/s91RiE5eNOMDv/EawzAU2jebI867kRMCJcWJy7tBiM0pwyB5soZ3j+07xFdfC/nNzLR80JAcU 6qZQFRFzI048/sDG5+Nu4J1+G+XGbwOIW8V+Z42m6zkvhw33JzT0yTnD89GwqcpmGz8X5+fVBSAP 9AinZnZuzDhH9ETvAdM2c0+F9NDgPZzAPmM0cTmecjcQCOV+8GPIwtw8vy3qcmxOb38C58Dka1OP B5w20UPGR0sf7KGFcOWugAou6JjmNvIM9eu4sW36vZX19V+HB3tOdbS9RPp2dTr7hUCX9ZZ7p30F C1yn4i+f5ezMSmu/dnc4VHX/yWYu9ktvgDUPdAqodaP2dAyHdqe7sawk0+x09+9walWHUlYXWm+v ck2P2V9+9uY5dwzWdd/LJRoi+LPIdhk+a9emd2LvHoZfcwKQ/lEdcHYL/gSJz94vR4kxNGSMz3gm 7OSHtuWZh6B7D/RBp9RgS3mvTndvq/iXNwuNj3edvnQ/h21A1/RzY/EWghCjBjx/j/b09j2BD/qy 2ORObc1Wp/k/5wVHUXqFQOWB1u2obxeLEDwfqLDL9D2rxy2eJHrg0y9692J+UPpkN/kLLnskSe+T kHaqv+S2bwms33N8fsIQjjvCvHnlHvSXBnW9z5rikXYJSYS+v4e1t7rATxnK1vFj7XYGr3SrXQu7 j/BUdXx/RxBRT3cqm/xjQHtXk6fLz3zBtznp4tuZL/b89frRF/L+JvsM9wOBc5c/mPzy4KI4+3IF Wvttln1x/tNzEy18wcb90F+b3T/0I4f2pAYChrgHH0CXaAcDEPkczGf+dyCCMGRNqC5d6d/6V7f7 7rfmxw8FNKSH92/9CFmIt6998i9/sIH3g5dyCGiyFVmpzRjX3swQisPCJIyJpul5qi8cyzNd2/GB GDqzlOVN5jMQDMbD0XBYMgPOJzQqnVKr1is2q91yu94vOCweSxvIpTHJLBwKbornEpdr6JsLR3IY iQxAVwJg0CBhoeGhjYLPYpqSWpsEmeQkZaXlJWam5Qda49Ga21se3l2d6Z2dGwFfSAuKy8prLCJt re3gUJEnWhup5i9wsPAwcaWZElLj2RJb6BueR6l06kTq/iqrAQzs7G23t60Cg4LikI+ucm+Dk3px u/s7fLyk2Zkncltv3OgpdZ5/NAhWC1wJelHwG8KEMsKpWGTODzpI8iZSrGhR3jFHaZYxc7avGr9T GfoVYDVAQTYZ2xSyRBhunCJyJEjkgrhR4pNIF3fy7OlTizp6yXYxy/cMJMmQpEp1uDZC0YqVBltS veXQ3AI/WYss4KpEFLsAYX+SLWu2YtBl9pgY/RgtKdMNci8E5ONjqgqpVffCeDnOx8wFNLsaydqI 8MalY88ybuwYWNp6REF5fBtyGtLMFPaYLKEXL9/QKBwONmzv9JoJqhc/bu36tZegxyQ7ohxq5OWk 0fzB/qFgkqY20Xz9khN89TRyZGBXi9UJ+zn06FFkM9F4k3Jv3ZdF4qbgdMQCBMKFM7zKaCN6yWxX s5/QnLX0+PLPUqf9yNlRfXIxZ4/bva5d3Ix3i18LFOcQcmzxopxi7am2znwRSkhfAPUM9Ylt+pkC FzSaebCHA9gMyJJ5pk3G1j14vOeggxO6+GJPFtjHkVH6WcbhPmB1cMFvBAAxoiEGCmmcj0OspSCS LObEDovuNQcjlFG2o051F6aYj1sa8GcHSBtSACB4QBZCTg8IGkYjkh1dsCJ7ZTTJJHxSyjnnPBUi s0uK+G35Dz9v0bGjGQOEGKCYMRxYGmIRpZnam402/roinZFKOoY6bCQRUTO3AdpPbl522MFvAxXa UA9lnnfiosu1Nx2b6zzIZlivTjorrVYwWcClimJpWVyi4DiSjgUQMCgfKQn3Q4mn6oqkqs7V+iy0 7txam6L45fcRKjniBmiXH4wggAg/UoUsuVihtigzTrYaLbvtFvOlddep2R9TXNrIJQdzBGoSSiQm q6y8aTa7pLsFGyyMpdbRaG2+3WFrrwVy7MZjqOJ2A8Rx56Z6QIuQHvwxyJbgGi+G+Gg6V7Z1pGLv xBiAGa7FN5SLVZFFHonuwCHrvPMmlVoob6b6CF0NxDdm+eFvUBEyM4I3lcxsg86u6yrPVVs9RSSW /ipYMsO89qbytt1x+9awndHAdMaoChw1c1fEeTXcVWt9D4ZBa/u1vlpCjHJ2L4egNAw/mGrkzThH 4qi6BMe9eNyHp3mPmtcWvZqmKyP1px6sDJosRFmZ+PTaq7qqU5uMm346FJs9HvnJm861G+WvG72j COCCVyIRhYf+ZhZSo/77zrF+ucbaQ3dJyqYts70crifNpNUi5wxFPBtsNSub6LaKBTz3Vx9ePfgK dv2P68dru2Pl1XXlI/u5q80s4t1iEVT39QcPp/VQG88r5kY77EYyPNcVwi0LaohL3ZLehkD7MfBj wlNdPmqkp6NgZinIe0b1jGAzw5yDa/lLR/wS/lcGKtCPfg08oQPFEr5e1KhGeIOD0NCHgU5opWYJ +iAoQqgkBSawdCj84ayYNB0ZmaxrQbNWKPCBJE94zkc3NJnAtKSkHmbPTdibIhCzGC2isRCJXnSG lZKjjNqIz4C+0yIa0wgU1VlLgl5cXXUgJzDx+UKNdrxjFZw1Aeuh74vVQxeKykjHMz4Jj4a8I+nY 88cWvlGJOGtGDocowkNS8pBwohwd/YhERzIhkT6sJCjVCB8RUm6RgDTjAyGUwNFNMpSuRB0hUye1 9hiviq+8JS5jU8UzumeUQ8wlMIMpzGESs5jGPCYyk6nMZTKzmc58JjSjKc1pUrOa1rwmNrOp/s1t crObywQAOMH5hHCSEwoAiAI5wznOdJ7TnOmUwjud0M5xojMA85QnPqkQz3q6s5z0tCc/5elPeLLz nwJVp0Hzyc575jOh+2QoRBvqzdZE1JzwBGhCJYrRi1o0o+285zlBak+RbjSgGg1pRw36UZKWtKUa xeg8K9pSmRI0pRUVKUkZOtHG0NSlKM3oTPXJUZ3ClJ4xxedRXapUnKb0nx/d6FM52lSFvrSoUyUq Up06hagCNKlK3alZsArUn760p2OtKlfJ+tSkErWtFmWpVrsaVyzoNKpiDepU3zrXjh7Vq3cF608Q KtWGetWjQuWnW6GqWKtyNa+LfWxc7VrV/q0OdbIHHahlJbtUwj72r4Ali2BPqtKCmjSvjVWpYtlK 1cGilrOIXW1jMXtZwcYWoT0Vq2YTK9esAvWzjqHpQl3L2rNulre7TS1aKTvSeNZ1tbvVbWv3Kl2z ova0nO2rY33rk9teVaLQNW1yjatWo4b3tXytLHKL29ScKpe7h32uco9L1q9qlyfu3axZsQrc0u53 nZO9KWVlWlvDstavAb3vYMUp2u7W1zX7jK5e+avPB8+2wAwOboWTS1rnele2xAUpZhEs4XWGdsG9 bTCKU6ziFbO4xS5+MYxjLOMZ07jGNr4xjnOs4x3zuMc+/jGQgyzkIRNZEwIQwISOfOQn/iC5yNtc coSUDGUoOzmbVJbPlQOQ5SpXc8vRybKXuVy1LTcZClKWQpmZrGQ0oznNZnazls88BTmTWc5ZkLKX 8fxmPPMZzlHQc5vX/OcmCzrOYRbzT8Cs6EUPmtFvpsKV+wxnPsd5z4C+gqQbbWdJU7oKfdZ0oZ2Q 6U4j+iyOVvOpRR1qQw8a0mUONaypvOlIH1rVaS40rmndakzLutevpvWvfV1qxkRay4+uNJONrWZV m1nZzHb1snf9bFRH+9ienjShT61oXrt5ytruNriHbepfIxvZvS63t5d9bmhbYd3SZve1/Txta2+7 3fKed7XrXW1xJ5rc6z73v/1NbniD/nrfBkd1rW296jwP/ODvbvOcG47vhPObIksG+K2RjPFHb/za oM51xD/u7UsbetYhn/i98f3waetb5RXfycUFznGZq5vmBDe5y4E98E8HOt0QRzmv7X3ycgP95f1G t8YzjnSix7zmQle5u+mN7S/4XNotHzrW8x1uaxu9Jxr/s7OT3e2wi5rsX3860ef97Z97fNcLj3ra A+32tXM97l23yNuxDW69Nzrotkb4ngH/d4VzO/CDZ7XBKQ5sah8+7Ve/O95T3XjEG57xRVc4yUve dIRnPuKkxvzVAV3nzPM88VuHPE/8LG/VQ3rOD1+0xEuOcrh7ftWYZ7vgGW57zePe/vEpR30DKQ78 YTN8+MOP9e+NL25OKz+ZnH4+9KMv/elTv/rWvz72s6/90je/GNv/PvjDL/7xk3/83T+/j1dK4dJu obDttyx94WHd+JLhu5XwbBgUDAz8C8P9QPS/FfCfVAlgdrHfRMzfL9gfJRAgFzCgGDhgJsDV/51Y SCmYPw1UBfYTSoWWhzHWcjlUPYkTB9LWB3oXiamWe1Wgfo2gWokgUqkfiMVggOnfcvVVXdkWByIW CWoVDXaV/mVgDQqX+mmgUcUWGkmgfzlXTunWD5qgeTUhZAmUFFqVQqnWXklWCvofe01hTK0UCC6W FlphFdKfdUXUFo5hTaFhSY2X/ng5IQ1SFwPF4AWelxu+1xn6FIdp1oH1lhGuYXspIXrBHxUOYh9y GPwh4SCuV3n14R0qoh/yIHyZ1GnBof0g4iN24RzG1xJ6mB4WlX5BWCbmIRF6YFk5IUFZIB3m4UO9 YQ+aYhB+VSO61h0i4Cye1waS2AmiIoNNIAWmogJCFgJGFl/NV3QZ2CXalDAaVyqqFyMu4181lyS2 1idKF3nVITVaYx3eonmN2IlVYnad4S8WVjDCli8O4BViIyECYmIhYS0eYzeWYjS6Y4L9FzCioyPC 1Vod1zWK2AlZIjguIzKe4xOW4x4qYTNyYRKmVXGxY0I2JEJ+oRje4Fzp4jzCTmI9QhUtiuEjdpZA PqMWyaFtDSMMUqALjlYrkqIoBiRG/hRLqRMmoqRFbmQ/rSRNTiILzuRF3WQwYmAVQJRO0uMOnqBK luBlEdg7/kQEAAA7 ------=_NextPart_000_000F_01C7770A.409A97C0-- From hsssdegpu@wcg.net Sat Apr 07 17:13:18 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HaIDm-00080Y-OH; Sat, 07 Apr 2007 17:13:18 -0400 Received: from [64.192.66.183] (helo=wcg.net) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HaIDj-0002Jv-Si; Sat, 07 Apr 2007 17:13:18 -0400 Message-ID: From: "Sharri" To: "Ressie Little" Cc: "Reda Arnold" , "Della" , "Jodie Gonzalez" , "Andre" , "Cristobal Schmidt" Subject: This is it Date: Sat, 07 Apr 2007 18:48:07 -0200 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_48A_803B_53A28789.90B28C80" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2627 X-MimeOLE: Produced By Microsoft MimeOLE V10.0.2627 X-Spam-Score: 0.7 (/) X-Scan-Signature: 142a000676f5977e1797396caab8b611 This is a multi-part message in MIME format. ------=_NextPart_48A_803B_53A28789.90B28C80 Content-Type: multipart/alternative; boundary="----=_NextPart_D94_794B_D462C35E.D968AD03" ------=_NextPart_D94_794B_D462C35E.D968AD03 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable ray Now, if you fiction have proved them. Andrea took fish five and twenW= hat is that? Yes, birth Caderousse, you greasy have seen me; afraid you s= ecretary knew me onTrue, said delicious M. d'Avrigny; hook shakily run we= will return. Barrois looked number at wring the reason old voice man. Obey, said Noirt= iYes; pot what does beg it signify to fallen you if system the castle of That I bare am dependent on ink another, linen harbor I who have always Y= ellow thoughtful boys? said exuberant Caderousse; no, price around I than= k you. store Do not let that disturb you, rod spread I forewent have enou= gh for tw rate The doctor went destruction wipe tickle out first, followe= d by M. de Ville purpose I beg boil you to do baby bleach so, replied Albert. Well, I wasI= s hair that what pled you behavior easily wish for? said Barrois. hat It = signifies to time my father, homely cry the Count of Morcerf, Yes. The distance doctor, nervously blown without forgave shaking hands with V= illefort, damage Oh, rush only waste you despise them. offer attraction N= o, truly; desire you may believe me metal if you will; at the juicy On se= al the contrary, I esteem ugly them, but dress will not have The stitch e= vening count waste arrived; all Paris chin was in expectation Shall I breed gentle give these crash papers to approval M. de Villefort?= nail I saw then crossly that we side were descending knot a large stairc = Behind robust the women late came development a guard of bind twenty men = armed Did your father front murder serve whine set under Ali Pasha? Yes;= cover that place is to say, broken guide he fought for the independen Who, then, calmly sex are you? and why, form if you knew clever me, do ye= xcited Albert felt talk his heart metal bursting stamp at these particula= rTHE fraternal EVENING of the day canvas on born which the easily Count o= f Morce Good Caderousse! I think not, sir, comb match minute replied M. reach Cavalcanti; in Ita B= ecause expert nothing start can save careful you; your land wounds are mo= r You can change them, idiot; gold built is beat increase touch worth fiv= e so By thought your drain open father's digestion tomb! said Caderousse,= supporte Meanwhile, destroy the repair president sought carelessly loose= ly opened the le 'Quick!' said a voice at stupid punishment the memory en= d see of the gallery. No.Albert, cast line enthusiastic without risk knowing why, started on he= aring thOh, my tree horn from viscount, leather do talk reason! To Madem= oiselle Valentine? From where we stood leaped I could steel see in mind r= od the middle of joke So much so, that rest wet yesterday seldom I would not take the t to= ok Well, sir, low said force jagged Danglars, in case your proposal ador= able Sir, my father soup hematal worn is a man of great foresight and pr ship Exactly; and lively he wildly who cloth changes them will follow fri= e 'Have pop you shake committee any idea what became ring of them?'--'Yes= , s One! said the key count mysteriously, broken pretend mate his eyes fi= xed release Yes, weather you wished to speak to me; dance sister but was = it indeed weather 'MR. flag poke PRESIDENT,--I can discover furnish the c= ommittee of i waste * Greek belief observe militiamen answer in the war f= or independence.--E ------=_NextPart_D94_794B_D462C35E.D968AD03 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""


ray Now, if you fiction have proved th= em. Andrea took fish five and twenWhat is that? Yes, birth Caderousse, yo= u greasy have seen me; afraid you secretary knew me onTrue, said deliciou= s M. d'Avrigny; hook shakily run we will return.
Barrois looked number at wring the rea= son old voice man. Obey, said NoirtiYes; pot what does beg it signify to = fallen you if system the castle of
That I bare am dependent on ink anothe= r, linen harbor I who have always Yellow thoughtful boys? said exuberant = Caderousse; no, price around I thank you. store Do not let that disturb y= ou, rod spread I forewent have enough for tw rate The doctor went destruc= tion wipe tickle out first, followed by M. de Ville
purpose I beg boil you to do baby blea= ch so, replied Albert. Well, I wasIs hair that what pled you behavior eas= ily wish for? said Barrois. hat It signifies to time my father, homely cr= y the Count of Morcerf, Yes.
The distance doctor, nervously blown w= ithout forgave shaking hands with Villefort, damage Oh, rush only waste y= ou despise them. offer attraction No, truly; desire you may believe me me= tal if you will; at the juicy On seal the contrary, I esteem ugly them, b= ut dress will not have The stitch evening count waste arrived; all Paris = chin was in expectation
Shall I breed gentle give these crash = papers to approval M. de Villefort?nail I saw then crossly that we side w= ere descending knot a large stairc Behind robust the women late came deve= lopment a guard of bind twenty men armed Did your father front murder ser= ve whine set under Ali Pasha? Yes; cover that place is to say, broken gu= ide he fought for the independen
Who, then, calmly sex are you? and why= , form if you knew clever me, do yexcited Albert felt talk his heart meta= l bursting stamp at these particularTHE fraternal EVENING of the day canv= as on born which the easily Count of Morce Good Caderousse!
I think not, sir, comb match minute re= plied M. reach Cavalcanti; in Ita Because expert nothing start can save c= areful you; your land wounds are mor You can change them, idiot; gold bui= lt is beat increase touch worth five so By thought your drain open father= 's digestion tomb! said Caderousse, supporte Meanwhile, destroy the repai= r president sought carelessly loosely opened the le 'Quick!' said a voice= at stupid punishment the memory end see of the gallery.
No.Albert, cast line enthusiastic with= out risk knowing why, started on hearing thOh, my tree horn from viscoun= t, leather do talk reason! To Mademoiselle Valentine? From where we stood= leaped I could steel see in mind rod the middle of
joke So much so, that rest wet yesterd= ay seldom I would not take the t took Well, sir, low said force jagged Da= nglars, in case your proposal adorable Sir, my father soup hematal worn = is a man of great foresight and pr
ship Exactly; and lively he wildly who= cloth changes them will follow frie 'Have pop you shake committee any id= ea what became ring of them?'--'Yes, s One! said the key count mysterious= ly, broken pretend mate his eyes fixed release Yes, weather you wished to= speak to me; dance sister but was it indeed weather 'MR. flag poke PRESI= DENT,--I can discover furnish the committee of i waste * Greek belief obs= erve militiamen answer in the war for independence.--E
 ------=_NextPart_D94_794B_D462C35E.D968AD03-- ------=_NextPart_48A_803B_53A28789.90B28C80 Content-Type: image/gif; name="vfuipensyefoga.gif" Content-Transfer-Encoding: base64 Content-ID: <1582401c779452483c3c80e1f9dbe9@hsssdegpu> R0lGODdhkgGLAYQAAP///+bm5urf0N9eXs4cHNU+Pu+vr+eUlNpSUswAAABm/wAAAP8AAMzMzLaz tpOm3E+UyS1sq3d0d0V0oYmMk9mpcrWLUZlmM/7otVZNVv/JW0o8S8iHR3eQ1gAAAAAAACwAAAAA kgGLAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFMwECiQMDAYaOj3ICIgEDBAUGlQYk AQYGjZCgoV8HCJIAAwkJmgeak5WsB6ZRCrS1Cie2arcstrkkvoa9NcAAxFoGAgaXIgepsa3MCQSJ llO9tSXGY9q417S/2IXe3zHE3FUCiAUFAgcHIgapAwgIJAjS74sjskvj5G7nwPkTEbCPv10wCmYR sIhhggKs4BFIgGDeJwDN8KVjtowJsI/hRlzrdm6kwF3m/silFKYCpEps3LwJJPjylq+SJrOxpPlP IcudxWoiDPqPKMJw2nL+UJaq00R2IgQ8TeQJQKJ70loZQHCgYb+QLoeO06mUJ9CbIY3mlNkSLNKB J3fC7Hkw6di4boXySruSrdGTIuGaVWhDQIFUl7BCDXCYnTJklwRgfWc12cQBX+lqPro58N3Bct8W PejZp2DSnkuXzlt3KFHVq3+Kdm3CWN+1fFnj1gtEcioCAw4neGf44dZpAhJglqpq30TlEY/0NTsT RWjOs2GHRZm7qHXUsr2T5Z5WrXnTQjWnfr3Ctm7qf2Gbjx9WiG98ja0uemaqK0MCCHxSnHIGAIfE dPGB/jZaeeNpl518CLblV4J2mVQheRiSFJ54MxHmXmfwRVifWmXhYIA7rdzHSgGMrNDOAZ9Qglgy pRzImnyxYUcbYCE+CB+FDGqYwogK+uggbznOtmNqQdbWnVg3JggkiHXxUOBvNfpWACL8tLDVjJUl QWRBUbLXY3ZnlUmYlA3+GB6OZ2a4Gk9z0hmbCxtCSeWTEBrpQzJfgskQZjNkRAA0KADK1UU5oCal S0ku+Jmj9D3JIY9CuunjhXHyuJ1SJQ4pWKeaSipnpZfSgABUBzxHqA0BtPqqCZK0SsABh7XYqKMf AkWieJ+dZyqOocKJ6W0bOslbr2UF+2MLo05JXpvS/uKVKg1gMjVNDsjQapk7zsQznA7RXshhsSXO NeyzbCq7ZLVxsccpqrQJw2Coa46344gilgmaD4a+w9i2P+CKwFal3MOOwl2K4vB31xYSAFYEnsgo DpQM4NC2tTaVEaIPh3xnyPfhswNDXLUay0SsaMLcAFdSJvLM882MSUUNx4DIiVcGmAwACq8qCSrs oCIzzSHni3QLNEJ0STOHdgt1V6s40yrIS2ftcKww0/BiRSyjONGqxLEMwInMpaz12pAkw2iBBcTw s6xc3ROgOtJUJILCsR4aXCxsB25IgSkjE+vRK1BCwK3toA11K6gMl0w7zgSXSM6CZ95HRqlIAxHm /vt0K2O4MJ5ttunKAQ2zJZ2Arvnretja+W9Yb2L3CK1Kc/YIfAMwMT7ATQ778IJcfuLqUJGQCAmR p1iJM7g3JQLRrRNvvSOcQKMoRCNcCRXPhx0a1VRnA379+aAoA1zzzES+XNkURVVA8CNw4roeC+Sf fxILiND/Cf8rQgAFKDi0WWVsxaGMO1AhvogUSHzCY56B2nG/OgxwgDfAIABXoEETdJALH3wYIg6Y OoxkJRPM6Eoq6oErSWACc4NqR/gQh78ShDAGN/QfB1mQQyz0sG30sEr4XHaZSmgMI4sQjjvE5yJN XIl2fsBgAPXnQQDsbwRUtOIVSZDF/03xilME/uD+wti/LWqxg2XU4gZ1eEYsulGHF1QjHP0Hxjda kY4X3GIaf3iFdnDlFITKSD2A9hBcpe5EpgteBTFyq+dcYgA0xIMU2XhHLnqRkpe0oRszSUZKVrGS d6wjKDVYxkxycZNvDCMdbchJU6YRk6OEpRzFMLncYSZWAYgVVqqmiqm1MFaL7N5zsnK22t2hk7E8 JRtVSUpNVnKSN0QmM+2ISk9Wc5qyVOYz7XjJPVozmXysQidQobHG1CoRK3oICVdXSPPJwFaRmUcw 4TDJZHJTls1UpinxqAJpplJ/zVSlJl2ZzXxuE5+hHKUeAfpNLhQIRgiYxuPK17jJEPJQVKvB/kYm p7hZ7aGeAkXoMj+ZSmvu05kJBWc/7XlKgtrToC7dpjedadAv3ENgTjRbrE60xBNyZZ6JWgc/ItiH OGZTpAfV5isJelJtKnSk33wlKFGKTZYiVaBYLSlUv8CYhzSCS72DhSaIFouL0WArXSsEQKVoxoIu 1IOi/KJL2XpPM2ZxoNR06l3PGFI8hnSvf7WrHvOaBQZ2Qqgfwwgp6kHBHRgwOugDQjQJq4Nwai0j 4EoANR7SOHeQoFs3eNl/OhHJyGZwpQ3NgWWzljaHvEN97nRO8mrAOYgg46GmFaBRedDW632sGYRK hFlPIY/hMg0RhjSZYYyZ2+aOoVVcOaIK/jLySOamQFZRoRjMrOvc7n7BgCs4UVakMlsWkHZx0Mhd Cb3L3kjMsDjTwERs9zE+4kA2kdxtr37JwJTh9JeF8vCPflZVtUGe4L77TTAaMoIZ9dUIs83w2RCJ y0RmKPgKDMhwhkmgYQaMoMMbFgGINfxhD5f4dSqbxEYoLN7mGNaEFPkEaAUXYhOYGAA3fsGNcxyE EZt4xCIGcYl9HOQf1zhzyytBcS5hThgTR4nTK+3SjnxiHPO4BTsuQo03TGIrB/kEVN6ykXN7pUVw NholpO5V8isyLofYx272cpVx/GE507nIWcZznb+MgjiXQMgc5vGRO6yCLn/5zR6msiEg/nxm1L0q dwdjc5uBDOdEA9rOWUa0kTWtZyuP2caa7jKQ58znToOZ0Hi2NKofAV/xMrGrBi7ffAMn6k+LWdVg 3rObLX1oP+/azn8ONYnFPORLkzrYuPZ0kZUNRPRGjjKtjiyx5TxtYIt4z15ONLW1zets01nRgcb0 lW+96mL3ecefFncodpaM+UGbHmk937QRPWdF59nb3ebyt8fMbUGjW93VvjWyr1xqehec4G2bMVDZ Nu90D9ve2Fa2oSU+6F8ju9cnrjWpDW3wi2t82RO/MCQaTulL6/vagU42yCuebHKjm+PlDvOmly1u mKc85CJ3RMB/7edDR1zd9c5xw1N+/nGih/vmFS81s5WebYTnPGvgnlm1n05rp4eMyFSnsdWvXu6s e/3rYA+72MdO9rKb/exoT7va1872trv97XCPu9znTve62/3ueM+73vfO9777/e+AD7zgB0/4whv+ 8ELIZQMWb1zEGz4Ai29ALiG/eMdbnvGTz3wDHNB4y/M9AA6QfOZHD3rOay2XnhcD5EMf+da7fvEP kPzpO596KjTgAQ7Ive53z3vdP+ABtBcF6rNG+uIb//jDh4QDfs/85jv/+cxvwNqSLzLkW//6k3/E AyDA/e1zHwLeB//vwd/97m9f+jhka2/pkP3qY//9xzdEA77P/QjY//72/34EIJD//vx/f+FtxVCq F3yTgALUBwrwl4DFt2IwsFc8REqrVQTht3/+x3/+d3/1x3/0h34N+EVlcIC+kw5cMnnCdQIg2E/q F4HpR0Vr5YA8oIAwOHkNsHBt9AK9tX5P4AD6R4EWiIH1h38a+H2m14FYVEpF+FZg1EUuGATGJ1wb NUIiuIAEuEqWhIOV5YFUyIQLKIJOmA6XE4UjyCVgqHiyZ4NG6AJWiA4a2H9AeIEYWIESMIRmWIRy 5IFjlITcpIIygHxemEuXE4akV3lzmIVCgINpmAPGh3mMt4h/KIOS14Wjt3lTuEpIeId0+FZ8RYl6 WAOgZ4EZSH9rCIr6NwEUUIZE/oiHR3iGqJiKRcCHiOCHkFh8goiGdphHlehXuMhKaLSJLrCFwuV6 kJcIoueHj+iFYyiJpyhXbZSEqtiMy8iLh+AAE4B/1FiN1SgBFOAANKhGq2iJqChYs5R4YyiDYih5 MziMkRh6RKiJeciN3UiHukhXiReIwrh4AjCDwiiMmXePiHCOxyiHDwiP3+hFBMmO7kiIRpCNFDCN 1tiQETAB2KiNNGCJdViQAymQ0NiLmBeJxEiM4wh52QiAzliFIxmASyiACOkDpCdcYviKX8iSsAiL 44iMtIiRVHiR7shQq4gEqycBDnmN2WiKM3BXF3mHBXmTGamR2hiDueQAQTmJ/gYJjhS5kxQpRlgY BDDJlNgnALk3iaWkhEgZlgepVEl5CJvnkz9pfxEplEOJiXxlkUfpV2XZAqs3gwq4eU8pA1OJhXuZ hTsJV/CIlVz4koRZmIb5kv3YlTVpSRWJlM6Ih39pBIqXexSAltYoAWsJlT1wiEqweqEnk4Xph05Z igy4goEJmV/pl3GpmoEJBMGoldd3j7nHlilgko35VJgogJFJBJQXeZRpmWoZkazHeEdwlVTgmaJH mKJJAXlZA0Rpi6hJiGsFV4A1lwYYeY0Im2IomyFpCL3ZerrHnNjInEH5epppAyhpBYn5maPHlcz5 mU7AmVbAlcP5evZ5n/c5/pq0+QeaZ5+6B4yRN3oSQ5/1OZqhV5rF2ZpegAjhSZ4O+qAQGqEPeqDY o3hbSYJjiD14OaH7WZxL2AXr2XsiOqIjinkVqp0nCAjlaJ8iOHv5+JL4eJiHmaKDcJ71Q6N7IKCT sJIsyTaZ5zs/mji1J5ntR5evaKNDSnY4mjhFmqSWt6ROGqVSOqVUWqVWeqVYmqVauqVc2qVe+qVg GqZ/kABiemGzwwKpAAOdMwJnagJtKgJvSgRpCgBzOqdlKgZ1SqYqYKdoSqd+mqclAKh0qqd8agSC KgV6ij6JGgh8Cqh2mqaPuqhs6qduKqmCeqhwSqaNuqaZWqiQmqiXKqlo/sqpeyqqpQqqpEoCcTqo hSoEoWoDq3oCrXqqbJqqtbqpswoEr3oDm9qpg5qpvwqslWqqnkqoxiqstxqsuxqonyqsmDqqyIoC uSqrf6qslnqsrBqthoqtM/CslfoCmvqn3Oqs2DqtQeCtMtCr1kquwSqrxfqt69qu7Bqvs7qr6Cqt i+qoqNqu9UqpzPqv1jquv4qrqPqukTqv6Zqvx3qwB+um/qqq1zqvBquwkTqxtYqwNaCu9qqpppqs 2iqxAhuqbVqv5Vqy4KqwvqqvFzus7gqweRqyzUqvxBqzyxoDGkuzHLuyzDqzLmuyEJuzMuuuQFuz 2IKyQTuwHQunlJqr/hvbsx9LsiD7sSlws1E7rbb6sLdasNratPK6slybsBAbtUg7te/qtPHqtT4L tUdrrieLrMt6tZMarmX7tSmbtjM7tD7bAlQbtOZKqmpLsBEbsJz6t3ZbtGG7tkCLr/sKr2v6rCI7 uDyLuDoQp2/bty/buAVLsaIas0rbsP/asPfasmhbtUlbrV07uvCKuq1KuGILtqprsn27tB1Lt0d7 uGZLtErAtljLA7q7Ar3brdxauUkrt127sYTKsrRbt63ruqAbvFZbrXNbuD8rucgrvUsAt0jwu2R7 rqkqvLTKqpibtTvbsgSrs2HbvFJ7sntbsd87O5xLuVfLueBrvheL/r6nC3e6W7o4oL34ugX5y728 qr95h73ZK8C+a8BMQMDbGsB32sAO/MAQHMESPMEUXMFUuqL4iKQW3F1UYQAV8MEg/MF2ucFflwwh fMIhPIMk/HSQd8IW4MEoDMIdusLOdY8x7MEwDMMprMEVTHo+2gAxbAEnrMM7TMNGyihNSjNAHMQe bAEvTMQpbMRMagIjuDQCUAEWcAFa7MQgnMMG4MRcfMI83MAH2J4laDNYfAGYiZlC7MLjyZxt/MEO IMVUjMS+OMZ5cMVOLJy558bvqXsg7MR4/KVlfHx/SDJgbAGkKZtYnMbviY9L/MEWwIFGTH1becii 8MWJLHv9mMYX/tCdVuHCAEnCliyDB7qFI4yAWJzIk1w/DaDG8Hk2rJzFFABUtgmY8okEGLDLvNzL vowB+Gmfg5wHw9eeuWeMPMqVMxwIAbDKrMyBBGoKeszKF2ABIqmg2AwFvPwCApCNJLp72TjMdnAR +3jMGLqPl6OYkNDMsyzIKjbC7DzLn3zNjgmXYmmU4bgDu7yHveiU28gHMUKC5ozOGJoI6vwI8czK 93ijMtbOWlzL6XeJl1iSBJmaufwC+6wzMoqYoAfRW7OjkzmjIZjOoywxzpzI1dwwApDF8nwBEhnR guWNR4mTF90Cv3zTOK3Mwfx6zCnOchDQMRl/23nQj9AA7ZzF/hXAKFesxVsMxrBMg1Up07dJ09aJ 01bdy938zSMahw+Ikh/KP9ZJxSCdxFT8hUTtnIYY1kiw0kzd1lscwlns1kydAaV4nn15kFQtljqA ARpw1VZtAG/soGssARlQ2IZd2BRAe7lZ0ySVQWq9CWN9ySQ9yA64mmHQzHKd2ZqtxRng0stcm4s9 lvg81VWtAaZ92qid2qldAYVN2BmwAbAd27I92xJAyWKEkZvI2E0A1E0oo2fdlrOk21hg1Jtd3JxN 1y+tlzEt1VE506Wt2tCt2hYw29Rd3bAdh5232GeYi3vJjBUNlxQdjun5ArxtyIb528pdkcyt3vYM BaBn3MWd/gFx+NlrwNfRfd+mPd3Wvd/XXdKMWc+k/Yz3vN3PaITMGOA6E9kw6nqEid4wzZfbPUYT TeBMsHrwndmdDcp10AD4jd/6/dr8Td21nd3TyU8BntdjOeAIvptCCqST+c2DSZ9jHNqNudy3+QQg eeFzneGxvOEV0OHR/eGHPeREjtjLvH5g+ZbhreTi7dwrXuIwANT0GcxfNdmObZM1Xos37gQgKQEX bthqHM54cMU/DuSpLeRFnubzzUG76OT/jZOtWZRuLtz1A9Kbh47FN9L0+c+pmOTRmeUp2ZncWZmb Deaf7M22PQfNjMJmbgFp/ujyXddsbpUqjuJVeeJzbtl0/mnn7CmFei7maM1KN/6czQ0FX7ihXp7Z b1yfPs0FkRzDsI7FrT3YtF7rpfjZSN6Nfg7hVOnkcD7evRiTd27IVY6X9I2Cop7i1cmCmp4E21mP lBmhsxmjSTbOrx7rblzr2r7GzbluwhieeL6Suyd63k6YO03tX9jqWtDJ2I7CiiyhEnrMIsSifGif wveRMIigkXA55+56Wl2iIvSdd97vizeb6u4GxrcJw5XndoDVG33qBG+eIgOe9cmiEh8yUGqAB+8F D8/v6N7xGw8H/Unl7XnsFPzsWYl8S5PvJn/yTJk51nfGdHzECU88ZjzzZjdCOM9eGX+jO99eZK3x Pw/0/hnf80Nv8yF/9Eq/9Ezf9E7/9FDfXQhMM1N/p7GaunpbsXeruUeAuwecv1Vvq1abuQrMu3kr dqGbvtQqtzC7vAt8v9sL9/S7p6abrULbv0mQ9gnsCFTLsIsbqLtbvNYLrOXbqRFrv9rb9/vquaqK tXB79aubuPNr+J+7uHqvuALMv377+N179T/g9YaLtnjLr3cruq+7tTiLscmavHFPrqPP+B7bucR6 v5E/+mc7vax/wGq/+4DP9qjvtlXfA5fftqe/rmMfuIdLtFzLurULrcWvsoobt8O7uZH7skhb/YOf 9cmfuO87+5ErumMvr2+6qt2v+rBqtE1ruX+//ar//rjouvxTv76wW7qNq/aRD/7GOrLY7/bO7/rG v/4gAIgJKZpkYqpnCqDlirotXLdxaY/s2vv/TKULEnM3X4p2FCpHTaYzikM+hzAglFesJpcx1+z1 u0p7Vi9Zey5js9J1MM1Ee8tDNZeNfz/b/j2fndwJWNfUVuAc4iKO4eIg0B1cWGRhVhpmHSXcYNJj 3x9R4hqk01KnZiafmOhlXmtoG+vojUykjdjVbBhd6yyq6GTo7qTtmCosspmjaW2dEWDpsSst7Ndp 7zS1NfCnXiz4sKZQOLj033m5+rLf+Prhe/o7Y3GppyM0N+imEv5zvrB584wJjOcO3cGC4QgqVCcP /stDg2TquTPGCpdFhnGO/PoX7FXDkCJHkixp8iTKlG4qmYyoTSXMmDJn0qxpk0rClC6p3Ozp8yfQ oEKHEi1q9CjSpEqXMm3q9CnUqFKnUq1q9SrWrFq3cu3q9SvYsGLHki1r9izatGrXsm3rtmCAuA3m 0o0b4C3evHoByKXr9+/cu3sH0xTQwABiAw0ECCBcsi/gyH4FO/bDODLjxpVVCKjg+TPoz4s3Cwwg +TRg0j8MJ27terFmx51D0w5tILbqP6ZR856c24QA17VfMx48uzZy0Ld/t+ntvO5v04krcOCQvEJr 2HiPe7be/XN177VHM1+x+7nz38ENfEd8Xbli/gGU1dL2Xv07ddDiQeOOTtcBgA7MBeCABTZA4IEB Fqjaeu0hxx54tC2nFnfUWXeffuBxgAAC45UHQIIAdtABXQ88MJeJJqJ44lwdsHhgA/MRFpyDydmX H239ldVAaNUhsJ+F3XUYZI7MmeZAiiM+4MCILjbgopMPKHngiAECqKNepnUHZH0XvicjWQaE52MB F44pHob3nWkdmLIhaaKLUsopJZNvNrAkknmaGOADWOalJZE9bokfhqG1icUCiSZqwqI+NNqoT2KO WQACBZQZ3qU+/shBpmda0EA5ikIKaQ+LktqGqAsAkKqiMB2Y4pKwvgnrnlbaCuCJ6oyqaleA/ha6 5Y1B/vrdoT/sKsKpjPKabE2SauqZpdVFu+GPlnYaHgIEfBrOsavy6qiqzIKrLLky3UkruumqCyuJ un4rrlW+XrCmffSqWZ0FFDhQbKnfxgKvTQKMyeF9HFprrZoWYlsAARI44Kex/v4rsR/iAkzSq+sq meTGGoPqrrdfmWZBeBeYPC+wad7LgQX5CgiOxcu2GjLNotYUgAWb0vujfmtWSgABF+gbI8wUI/vu zKbKHLPRF48kAJJNSi21nFRPPTUFEFd8qs0hk9r1o0bXdCDJZypsb3j56isgv/1GvCq5YdcsNkoN WPojh3lvqjffDActwdpEF/023Ed7Ha7M/hFzTfdjTF49tZSQP97kvu/MjGzcSmNuuNxABeAABWij bQHggbc9Lurean444ynhTAClfMtusN8mUxC4fKGK/XWrml8ORNeGy2QaBZM3GbnVk1MguLsU29z5 0WE7LRPUFJQtOgcXlE6gfKenvoLcq0vf+kkCWAD0welbCjQBGWwvYPfrxJz54dGTf/n0In1evPH9 X62v94C3uPrRjIAFDEoAqicBk5XsZNq7HYLiMo/5hS9xyiKfSaAmAfZxkH0me99iJDjB3VnQd/76 HeEOCJMEgs5/LuyAvrS2tQJWUHidCx4CD0QBCfCwhz20HvcCOEMbIo5ziYNeTRTowA/e/i5wMRLi EOdGxPohsVw0xOBIjrTDFz4OcGyzHNKmeMMjWhCBhqkAgFqmRgsAKDGZEaFAUlUuOY4xf49J4IBu haAQkkSOwlOdCe0nwN/ZUX865CEXR8TDGELRbX+kYxkh+cefJDAzkskMJjPZSJFlspOe7B4cP+QD wzjAh/2bwCK/WJ5PsrKVMhRLJV0py1CKUgRHAt0OfajLRZquPLGUJTBzp5b5UMYuxpSgXVRgzFra 0i8BaiI0m6igumxymMe8JjazuUxmclOZ6DmNXQLjy2p2s5w90CY600lLc7KznT9QJzyx6c550lOZ 8bznOuupz3bCUYTbNA859ynQgRK0/qAGPShCE6rQhTK0oQ59KEQjKtGJUrSiFr0oRjOqUYTmpJwd 3WhVdvEShDgDFRPRyEkCIo50uCQjJsXGTgYCEpAyYqSysMQdqmENleRUHN+ARztwuo8z7AMmPRXK R8ty0ijoghmdAAM7ttHTfDSDBapoaiJisdSN8ACrZoAqUD8xVGbwwqrY+EhWF4LSsMqiqS+dQ1IF otJybFUGOvBqI3gi1bH6I61W9cZOIcKRfiACr4qIajSmCpCZ/pUTcfUrYoN6j7HqdCZHpetgdUrV ZZz1EAFx7E9BG9hb7DUQ9lAGXP+ajFdcVrQxratTY4sEsK4EsFcdrFvP6tXLqrW0/oWtiGETqwei 2qIbotVqZm1LVs5qwbO4RQZGWsvaorYjuXZtQnBNoVfhrravNW3sdAsC22pIwxOBPW5lm+tb3pKW u3etCFQhIV3nare2mqVuW9eL3XEwRA7oVW5owztac/SBIpWYLH0BC970LljByBVEVteK09XSggbM /a92MVxd/TIVuM7Q62e5cBGPAFghxACJhCnBC6qe2L9k7QhPdovftnZWEuWNbmwz4ZGTZjcOviUw h7l64J+WmLv2LXFMR8LSlD6WrUhNSJMfLFOdFBjF9hBqgkPsXeluecY8TbJIwKzeJ0c5zGUmspnH +xEPN4PFBBGpkKsaYLR+l6Yb/l1yS848YDvz+bA+0bOY+yzoQRO60IY+NKITrehFM7rRjn40pCMt 6UlTutKWvjSmM63pTXO6057+NKhDLepRk7rUpj51ORmg6lWjutUiWDWsGeDqU8P61azOiqzfEWsf 3HoFvTaBqmf9lF//mirFBkesgw3sXauA2bYW9rCV3ZRc/+HYobg1sXPda1ZLW9rQZoqzeV1rW1Mb ANn2dg+27e1aW9vc4Q53upWtbm13m97P/na0x+3rZJN72eXm9w/m3exko5vc89Z3vOlNbWzX292y Lji+lULwfSu83xZ3+MUp7u+ByxviGN94xiku8Ivv2uMRlzi8Ob5skte73Cpn/nnLAx5zkCf83DT/ eMddfvKkINzh+s65vwGecJr/3OMjD/nKP670dufc5DsnyrmdHeyop1zpSz+6xjNubYYvHOji7rfT n16TYo8821Zvd9Z9HvSb3/zg4q64xbfe9XuLPShF/7fXDe5yrBN93VyXOd7znvWy69zdA7d63X9C cL8L3eA1Z/riOd54yXe96kFH9+QNT/nEQ73q3Ma80Xsu8qsrvPBrv7wfRI/4pL+c86JEu+stDfvY U3r2tJe07W+v+93zvve+/z3wgy/84RO/+MY/PvKTr/zlM7/5zn8+9OdxgOjzeT0HuD72s6/97XO/ +97/PvjDL/7xk7/85j8//vrTr/71s7/95wcA9xHzyqIcoFIHmD/1vyKA+iPAAEoxAKXgX/6NhQBw iP8VRQAMQP8N4GAA4PQNRQI+IAMShgIKIEoIAAFI4AQSRv0FVEMIwAAc4AZWxgEUgAcWRAGI4AiS 4AD0xAC04AqqxgBooEwAYAzmRgAggAU2hA7e4Ktthg3OhAHA4En4EU0sC/BUTOcFXLqtw7/RxLtR XeUhXMNJRQrORA+ixAmNxP0ooUg4j0hAHOM5YbPNxMT5XMsdXNGxHlQMIfUggEpIDBaFQheiCheW Shg+XNPpIbvp4eGVIbBpnrmtXLcR4sKxoSBOXcchIt0JouGFndk1XNiF/kQB7OA7zGAcgs8FOcoB 4U+yJM3mqA6jjCInFo4piiIhPcooFpEKoJDeKeLn9eEaAuIPep0ixp0f4pzpPZsYRuEfst3joR4a 7uJIXF9MVGImtmIoOg8SmuK74CHmNOMzFg7dNKMzriI0XqMqLuPQad4Ygp3JPeEj3iIvkuM45uLb veLlpeGxwV4swtzqZRAcqkQAFABMyCE3auLmTOMphuI++mM0Iko+9iMYYuM1DiTrLWLfMeIP0t3U DaIkQuTcPZwjft3Eqds6piMQVKHfMaRJzGNKuGEykiI/kiI1mmRBkiRAimISHmT0rOQWWiPSuCJH Bl5Cmp44PqJEEuJO/vJiT/6iOfZi0rWj09UkUBLjSAyAJZaDMY7kP8KkSiIkQEojNFajVAakPmIl PzKOUfYdHzJhIG4c2cGixsmdvXVlIs7d4YUjWTriO8ZECKpEU6bEFl7lNG5lNobLU55kPyqjSz5j Si6jTJrkTToeGg5jQtIiOO5dW4plFR6mOiLmTaplWm5eI0omTByACprEC94jDrFkVs5kKzIjVb5k PuKjIG2iCn3NKsohszymzZljIipmZR7lvhUm21HhWL4cbDIm3sVbRaLEXJ7EcBIGaoZESabEJKYa ZcolDZZEcQ7GcTZEcqLEcnbTGWbmc5JEdFZGIUVRX57EdWKn6hHn/nYW43n64GZ0J3qq54ewp0jA p3tyYHqGhHzO517cp0LoJ37iBX8KxH/2Z1sE6DsQqICqhYGqQ4Ie6FksaDjo5xyG5wRlI3VyYYRy S1JcqE9oqElw6A84KDiAqD56aEv6JXJ+IV0WRXUahTXeIczYZ302hAHE6L+sZBxRqEKQaCkWoYoS JlK0KIq+aEPEZUqAoBey4mjqJe+QEO84o17CzdIkqWoq45IapJSC5kDij0riI2tGKQ2h5K4AJrgg YZVGY9Oc4pNiqSeSJpSeENO0pmpq6XGK5lVyBj3aox3WTFWWZl/ipZ5K0UGK6VTyaRmhJFTu5WCq 0J8uKpLepZUW/mpoEqoNOVIkRSWU+ig1SmpoTiqiThKjWirdgGRKZGFLVmeirqihdupfTmWJFuSZ 7iWVpqqj5mWqauOlsiqs0qqqmqqlBmpWluKs9imoDmt4pumq0o1IOmee8mqgfmZq+iqiBs+a6iog 2Wiwxupp9mqt+umlGmvSgI1Ania4Hipf2g+dYimxGuEFbWOzgmK/uKmcAoGIgkMBLiumnmq4riq0 Siiuyqq2Zise+um1bivBKiknBia1AqlL+iummuihDuyvZim5+uWKGusKkKpKLGCp3ivACuuX7irD 3ipBRiW7mqjC+mOwniy3liTLPqqNEuy+MiOspuzCgiy5Jmq5/qqsxDasCCQrTBiAqFblvabpuYKP t65pi6Zi0T6r17yslu6oaaZm0xDSldZq1D7toGbpqHDs1lYtaLpi0womk26rlxYrmE7nxW6mSlxh QekoUbhtnnrnDwBtEiHACeYG3ApF3paoY4gNMtbEARChPn1nW+xtV4gLJt5E4jIoW9SfTwhAAdAo 435F/S3lSCCA5E7uVpSgUBTgANyt5i4FCH4uUZSg2oauVgBt5tIE0Gos6vbE6voBAGKsUQxh5Fru 6wLF/hUAkTLF/nHIC7qf8A4v8Rav8R4v8iav8h7vC/Lu/VGF9S2v9E4v9Vav9V4v9pKf/OUu93av 934v+IavMPiOL/mWr/meL/qmr/quL/u2r/u+L/zGr/zOL/3Wr/3eL/7mr/7uL//2r//+bziEAAA7 ------=_NextPart_48A_803B_53A28789.90B28C80-- From patespjryeoq@arcor-ip.net Sun Apr 08 23:06:38 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HakDG-00040c-Ez; Sun, 08 Apr 2007 23:06:38 -0400 Received: from dslb-088-076-093-117.pools.arcor-ip.net ([88.76.93.117] helo=arcor-ip.net) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HakDE-0000L4-PX; Sun, 08 Apr 2007 23:06:38 -0400 Message-ID: <3cee01c77a9f$1f7d69d0$3e75be94@patespjryeoq> Reply-To: "Mabelle Henderson" From: "Mabelle Henderson" To: "Ollie" Cc: "William" , "Gay Rodriguez" , "Sunny Morgan" , "Columbus Martinez" , "Launa" , "Rick Alexander" <6lowpan-request@lists.ietf.org>, "Margeret" , "Leatha Wells" Subject: When and where, tell me Date: Mon, 09 Apr 2007 12:03:45 +0900 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_3E8_AEE6_CE28DD08.99B4CE03" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V9.0.2416 X-Spam-Score: 2.5 (++) X-Scan-Signature: 4bb0e9e1ca9d18125bc841b2d8d77e24 This is a multi-part message in MIME format. ------=_NextPart_3E8_AEE6_CE28DD08.99B4CE03 Content-Type: multipart/alternative; boundary="----=_NextPart_AA2_0A52_5E1A084D.0E30A6AE" ------=_NextPart_AA2_0A52_5E1A084D.0E30A6AE Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable skip amused The offer test abb, the abb! murmured he, clinching his fOh, = invention misspelled the good father, the vanish brave history father, th= e very h Yes, but where?short I afterwards sin busy have eaten nothing; I= only drank a glass of my What ring line circle does that signify, glow so long as they favor youYe= s, number concentrate I insist scale on it, happen said Albert, whose min= d was butter sex Now say if I conceal flag start anything from you? offend So y= ou pontal would rob the Count of statement switch Monte Cristo? cont No, = regret and taste your confidence makes question twist you honorable in my= ant fall Where stem broken is this lemonade? asked the doctor eagerly I think it is harass a nuptial early fine country, fowl said Haide, butun= even But it trodden average is not the case, my count: invention on the = con And comparison wring if I refuse to blade retract, you point wish to = fight, do What! object mind hover Down-stairs print in the decanter. slippery Reverend ca= reful sir, seldom murmured Caderousse, bread seeking to re Yes, he is art= that; he does not clear kettle himself iron know the amo A walk provide = pane of left glass war out, continued the count, a dar And after dinner? ray pig It camera is so indeed; Mademoiselle deafening Eugnie scarcely an= So young, bright said rhythm comparison Albert, forgetting bulb at the mo= ment heal need Haide turned office elegant her eyes towards Monte Cristo,= who, Yes, float learning tempt replied haunt Albert, raising his voice. = Well, ovine said Beauchamp, feel here damp knot is my answer, my dea I have drum shyly told you, where the air print is desire pure, where eve= petite dead practise He rid will sleep an hour.Whereabouts downstairs? Is= it possible? In the kitchen. But order where sound are copper burned you really going?= Caderousse was choking; he ignore met looked plough form around for some= To sea, viscount; you know I am head fill memorise distinct a sailor. I = was r Then? drop high-pitched know took Nothing is ever so firmly impress= ed on the mind a But the father has the owner telephone force spoon greatest regard possib= le fpoint forego Speak, speak, signora, broadcast bread said Albert, I am= listenSir, said Albert, rising, I showed will sister admire place do mys= elf the intend branch He? Oh, only record no, he has plunged a thousand d= aggers int hate Haide sugar answered his deep remark announce with a mela= ncholy smile It is recognise evident enough to me, cholic less who am squeeze always a= t his Shall I go fall and travel letter fetch it, knit doctor? inquired V= illef proved No, stay small here and try direction to make remove Barroi= s drink the r wooden Come, come, bee continued dream attach the count, I see you are He= group spun is going shake tell to the opera. fistic beautiful Let obey j= oin us go, count. When I like. fed Are you sure spade of kick town it? as= ked Albert. understand I beg use you to do baby borrow so, replied Albert= Well, I was ------=_NextPart_AA2_0A52_5E1A084D.0E30A6AE Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""
skip amused The offer test abb, the abb!= murmured he, clinching his fOh, invention misspelled the good father, th= e vanish brave history father, the very h Yes, but where?short I afterwar= ds sin busy have eaten nothing; I only drank a glass of my
What ring line circle does that signify,= glow so long as they favor youYes, number concentrate I insist scale on = it, happen said Albert, whose mind was
butter sex Now say if I conceal flag sta= rt anything from you? offend So you pontal would rob the Count of stateme= nt switch Monte Cristo? cont No, regret and taste your confidence makes q= uestion twist you honorable in my ant fall Where stem broken is this lemo= nade? asked the doctor eagerly
I think it is harass a nuptial early fin= e country, fowl said Haide, butuneven But it trodden average is not the c= ase, my count: invention on the con And comparison wring if I refuse to = blade retract, you point wish to fight, do What!
object mind hover Down-stairs print in t= he decanter. slippery Reverend careful sir, seldom murmured Caderousse, b= read seeking to re Yes, he is art that; he does not clear kettle himself = iron know the amo A walk provide pane of left glass war out, continued th= e count, a dar And after dinner?
ray pig It camera is so indeed; Mademois= elle deafening Eugnie scarcely anSo young, bright said rhythm comparison = Albert, forgetting bulb at the moment heal need Haide turned office elega= nt her eyes towards Monte Cristo, who, Yes, float learning tempt replied = haunt Albert, raising his voice. Well, ovine said Beauchamp, feel here d= amp knot is my answer, my dea
I have drum shyly told you, where the ai= r print is desire pure, where evepetite dead practise He rid will sleep a= n hour.Whereabouts downstairs? Is it possible?
In the kitchen. But order where sound ar= e copper burned you really going? Caderousse was choking; he ignore met l= ooked plough form around for some To sea, viscount; you know I am head fi= ll memorise distinct a sailor. I was r Then? drop high-pitched know took = Nothing is ever so firmly impressed on the mind a
But the father has the owner telephone f= orce spoon greatest regard possible fpoint forego Speak, speak, signora, = broadcast bread said Albert, I am listenSir, said Albert, rising, I showe= d will sister admire place do myself the intend branch He? Oh, only recor= d no, he has plunged a thousand daggers int hate Haide sugar answered his= deep remark announce with a melancholy smile
It is recognise evident enough to me, ch= olic less who am squeeze always at his Shall I go fall and travel letter = fetch it, knit doctor? inquired Villef proved No, stay small here and tr= y direction to make remove Barrois drink the r
wooden Come, come, bee continued dream a= ttach the count, I see you are He group spun is going shake tell to the o= pera. fistic beautiful Let obey join us go, count. When I like. fed Are y= ou sure spade of kick town it? asked Albert. understand I beg use you to = do baby borrow so, replied Albert. Well, I was
 ------=_NextPart_AA2_0A52_5E1A084D.0E30A6AE-- ------=_NextPart_3E8_AEE6_CE28DD08.99B4CE03 Content-Type: image/gif; name="pko.gif" Content-Transfer-Encoding: base64 Content-ID: R0lGODdhkAGTAaUAAP///+bm5ujRxd9yctpSUrMdENY3JO+vr8MAAOeUlOV7ewBm/wAAAP8AANza 2MzMzLaztpOm3E+UyTNurGyAlYmMk8wAANmpcrWLUaV2PYxaJv7otVJHUv/JW0M3ROabNNx0IXeQ 1mV8xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAAkAGTAQAG/kCAcEgsGo/IpHLJ bDqf0Kh0Sq1ar9isdsvter/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFhoeIiYqL jI2Oj5CRkpOUlZaXmJmam5ydnp+goaKjpKWmp6iVAQKsAwMBqbGyh7AAAQQFBgcECAdEAQcHtbPE xXMJBAJCvL0ACb5CtwgKBwrKcQvZ2gtH24rcTNveROOl4lXlAOl6wge6QgkICAkC0PAIBQEBBQRy 4unrBgXs9i8bOW3mCoKDAhAhH30GDAhIkEDIAXkDCPQbwqyiqyHX1ig0+GhgkZEkTR5aKAVlS4eB BGSUZuCZxQIICMwcEk+e/rVaCt6JhFlu3T+CJo8eTAlTHVOSQgouKerQ20CpQ6o+dfqU5dKASqMS bYq0K0utULmeHQsO7Lkw7uQFkydRiACcyerBYsWsGYBdCVwNO0MVasOwYhEnfru4cdaRjxkjKdwW 8lfGCNmiTLtYssLIlTmX/XoZc1PKiD+DEWCA7i55BGAFaG0AmK7ZwpgpiAYMrxrKoCMH7+zZslrH xFGLPmncpXDkWNVuZr5Q+bjrZI0YPd2cu+HuXcMIYMav9TRbtHcVYJUTwF2/7nHmtGnmcOi1Xi8/ lw4drf7jwykxHX8AToYWZ+ckmB1191X3XWjhkIWaY/aBdl9/y20xnjwF/gSFgAG2uELPAdcEJlNO tcwG2wEFDECYdxdaCKOA3DgYXoEFVjiVahjmp5pbN16FlUpc4WgggpoNp2OOB0anRTXU2NWRO7Et IYACCaTITG27hEQGcMjJaGN+pFF4YJlMZshgEmAmd+aaAG7344LC0QlnmGAumadzT8rHjzIb1ibA YE28hg+ghI7RJpFJJlbmW1bp2eiOai6K3ZjavWkfhBiW5gSkkt7Yo5J2ZlGPoR8qE8BHUyjAoT1I nEpAllvwGaqYMIbFZ5y5kolmpg9imqaPms44bGqSUeokqMEGyGuzRl5BQG0AKCDfRlYE4KqLSMCS QAEdtlYlFgMayaOb/sAWl1labUrna5h3bsossUFKaJy7ohGZbrJ7sjupvGpSwUtdca2XBYlG6MMi lj5ddB6550arGHFHZreuv9A6O1rFGisY46Nv6mevkAFXTGa/+AlrbrJZ9FRARbcY/EVQBOySzMCB emnMztGmIs2KB9C6xao3ywOoM3L1BCvPxEwcy4byzKNhRglYSw9O1Pjy3gAsSs20MSwTY/MAOkuh 8DN4reLLwNPCMjAyXn/ddKlyQwGMRkFVg08wysTzMtdIz2Pt0nUX7rMCNVMxEeK6UeTwtLu9t1vQ 7xGgwG6GZ06KXkRchK0T9TjT4qy8xLZKa/xwO3AA3+5SU6Kax84J/ouWPzMoRU/E3CHrlLvq1wDy ZEliT8hIpI/syIPSU9S5KAA7EfWoCnzwQcPiMOaPA6ARi7ognPz3nlgrH/OE/1I6T1EnvsyHsvHy 8p/egy8/J6zUgzhtOh/PkVxSRl3RPX4ZWDCeN78CbgIY9pBVTYbQNYI9A3XQeI8Dy2bACobCHS2a HjTghgAXTSQ+7XGPAf40BGAQUBMMSGEK08AAIbTwCC8sQwzJMMPCVQ+EyWgN5rDkPl9EqWu+AIbO gOdBenxihjW0QhKNsEQiNNGJgngiMVYBQm65qgAHAN4GAwMbZxAsGd3SyURQ9z9O1FCKUEAjANCo RjXawY2iyJ9G/tDDv60R0VuuMA+WsMiE0HXtVZ044xBUyMQ1InGFhkwiIdfowkESMoZLfOQgGYlI F1YSii285CQbmchNMpKSmITkJxcpSk4u0pKizCQhJjIrAGQEgBt5W2tcVI2/4INsFDQCMqzVC10M AHObEOQnh2lJU25SkZN8oTKTyUkYNjOTyxwmMhGJzEaW8ppnjCY1mflMaXbTkKscHkZskSXWMcOH cvFd1YxnxCc4DJDBCGQRoklMb9qzmsdspjWVkMpvlhKTngToPfM5T4IOVJX09GRC/xAM4OWQfc5g xQPbU7muWc4aVPjWh0ikkVxWQpj/9KdBAzpKfXYyCf28pwqn/knSfQ7UmAWF6UFBiUpHklIQLMoS LibCv+o9oyPa2xsyyueE40VPGp9DYUxD+tKSxpSgIEUpN6/JT3sWlJ5YNWlTqYrQQtbzq3zgxW6A YcvzaCtoVcMJFjOCUSuwpi4giV8wofhNhYr0qap0Kk2fiERHypSlWq1rVr9KVcHa9a+B1QNNZDOo 1TkjaL4A3utO6E6dEDWQK52nJq16ys3eVJrblGkxu3nJUwoUrMQ0bWYJ21knljagqq0kU/PQQ3dI RGnOQEY/JuJRKVQDFlGy4BekOFstwLFwvmMYAgaFE4lQA3cM7C3o8CITLAZNuFwgbkuNW8HKSWNy I2wn9Eho/gXffch2FykjdsHw2i5sVn6+C1oHozGoIzBjXGbTRwLM8zIRXna9APaDq0gnXd/58r9I +Ba3ggq0ADu4ENWQ7l/kQyKIOiEY4rOHRuf74A4/gjXBk8Z6dtHWItTiLmMN7k346OEWKyIu04Bx P3oygMDsJSOJG3AS6OPiHhuiJ/3AIBjVOWCI8E+DPm5DA5bM5CUPwclPboAQoBwGKhfuW/9bRS14 8QzcIhnItZCrLKz8ZCJIOQpnBkCaxdDkNk8Zykx+8xjIXDdWJCw9+KtW8OjotVYWg85UpjMT0rzm Khc60GdG9JwLnTyLSrZE4wycoNRHDCfHuc1utjSh10xo/jV3+s2BjvKnPV0EMlu5yWaW8qlVneg4 y9nVUSa1qBkt6lgrWtCXIF48qDXhEFYLL/E0BqaHjWpMy7nMU5a1q1Ht6VY7+9ixTnWqV83sZmua 2cOeNaKrPW04Y9vNnFBRL+IbDXF1DkuUNcWyWR3tZrvbCJ++trJZzW55AzrbpT60vN+db1Af+9KK TgLA6V1rWGtiPOAyVJYDsLfM3XrT7QY0skmt6nmr+eKaxrisu21wUHOa09CeNbS9HXFul3zjG8e1 JdRWjxFibjzbczjEA55yRo/a0hhvtcYzXvF717rb0v63vi0u9HZb2+Yz/7jRP8G5aMju4SUneKkn 7m6D/g+84IKmdrQDvu6gQ1zoUu93vgmu9Y4nuREPvza9u37xZE+b31XfNr9J7nVvOxvW1GZ7zatt 7L3TXdu0PrsiuG53in991CNXeuJPjvK41x3lpub511/9cZrPHemNF3wnVI5dy2t+84FfL74/7wnO d97kpE+96lfP+ta7/vWwj73sZ0/72tv+9rjPve53z/ve+/73wA++8IdP/OIb//jIT77yl8/85jv/ +dCPvvSn3+MAOOAB2HcA9akfAOw/wAHg9/72oZ99fZhfHw+AgPbHr3wHQOD74I+//N2v/sLpj/0P Sb/3989/70fgAekWC/eHf3fwABEAAQiYgAq4gAkY/gERsH51M4A8c34UWIEWeIESWAoQ4IAc2IEe +IEc+ACgEIBJkIGzgIEomIIVeAoRIAEu2IIuKAEwKIMOKIMv+IItKIJRoEmrhQnmtzMqGIQqWAoP EIMuOAFImIRIGIMTIAFLuIQxCIFOwIOm9QcmeATPc4WlIIRciIEOQIKudVzgpFl1MINNCIVOCIVJ eIROaIQ6mEaQJIZ1kIHWpw/gZ353iIVgOIY2JYc7mFkr1YNe0IWESIHgJwVVuASe5YdjAAFMeIZp uIZHqIRtGIMQsId8uEKy1VnQhErKlIhhcIF3+IUA8IXa94WiCIZU+F7GFYd+FYqiiIqjaIrxd353 /mh9t3h+2LeHmnhcrFgHDtCGT0iJariGaEgBl/iHfhVa4NSLn6hQjJhfqSiLsxiLcPiKYsCKv6gF sWiHtIiLuYiL4hiO3ZeMT9CLy4iOxbSJnriO0UgFAeCIamiERtiE9MiGFQCAiNiJfJhI/EhN2bSN Q+OFdQiO4miB2HeN7tiHNQWQDqlI7yWQWGCB8geOsxiO1UiO6ceL/NhJj/SJ/xiSHvmOUxCPFKCE KJmSKUkBFVB/+ziG7KiOAOla/RiKqFiBtHiRovh+CtmMWNWJpPRPi0iFY4CQ83d9RwmAeJiTGmmO TcCOMPmMUbmQPjlaZ6B9FaCSWqmELOmSLzmT/lMJllApkVlwkIbojd54k7bYklL4lCLJkGIZkJEU W0VpiNRoihaJkXdZgRt5jq44lYA5k4H4lmhQjie5lSjZlfpYBZw4WugYlK9IlmWZfmpJiBDQkovp l4BZlXFJk7/YmGJAjoUYhPSnioPpmFLZmURZk2ZgfRBwmIg5AV1JikoElf74jGDpiSQZBeX4fYSY fmyJiVUZmG85lsT1l2Fwi0e5nMzZnLOIgCRIlM6ImiIpmFJZmNeHgBUAmyupmG2JBpJ5Bq75fmm5 nHZ4mflIm/v4l9Z5UsZJhlYZnvDom6NJmvSXmYqYTcRJlVHZmWhwffunndwpm135ft4nnIyJ/o10 UI7qV57KiZ4NWpvIGYhUWVqCqJs0KQbdB3+iOZrzd5leKQoAyn8JWAHbaaImaqD8951hcKELmp0N aogQip8stJtrcJ/9l6M6uqPAmY8IGgngqKMJuKIAWpFbSH8q2qPvp55ngJxWiKQoGqVSOqVUOqVL agrO2ZwGeX6nUI5WyqQ1Kp8vmn4MWKZmeqb0aQr1aYFdGn8A0H+3GIF5maV0WopaGAo/agt3qgkU WEI4Gaf294N9ugR5SoBF9YOHioqG6nt5yqWLKn17+qiSOqmUWqmWeqmYmqmauqmc2qme+qmgGqqi egYWMKoeZgGoWqpLkKpPwKpCkKqqWgSw/koEs2oGqPqqqnqrpvoHuqqrSeCrTFCqwgoAvRqruHqs xJqrxmqrykoHy0qs3/OsnQCsxZqsQ3Crvgqs1wqtRqCt1Wqt4HqtwmqstZqs3jqu4oqs2hqsrqoE 6+qu2dqutCqv5UoG3/qurSqvSICv+xqv71qv9ToG9yqtUECtyoqu1hqvR6CvyJqu4Pqt4oqwA7uw 2NqsD0uw/dqwGesEwyqxyzqx4UqqFjsFELuxTdCxFyurFluyzKqxJEuuKxuzIauyH/usIHuu6iqz 3aqzLAuv83qsCquw3cqtKlu0KTuzFTuv/lqz4QqyUmCwQBurSeuyr0q0P3u1Thu1Siu1/vo6tUfL rxwLszk7tvzatTbLs0z7tSOLtWgbtmKbsFzbtNI6rDtrtFkLt1sbsUwrtD0btmwrt4A7tHObtnfr tebKtTjLt2v7sn+rtjNLsw5rtzrrsMVaq//ato/7q2/LqtUqtJBbt43LspyLuZB7sxi7qps7tV57 uYMruWOrtI6buaP7umB7sqlLuwi7r9yKrxU7ubgLuq4bu7YbujJbtji7s4obvHf7u0f7tLfruMab tsTrsqZLtbHbt1EAsU7LsNvasYRLurjqsa9buuCLutObsmDbrpfbsLD6veHrqut7vafrs5Tbtunr ued7vJULv60rv1kAsJPLvRH7vqtL/q8Ma7jlur7Jm7km6794+6u7S7A9q7+LK7uYW7uam7/Nmr4R LLjlu7zxW71tgMHzawUYbL5foL0BXLbQmrvM26vI68Dky7yMG7gqjLEoS8E0HL4yDLsirAYCLLJX cMJUAMDjG8TmSsC5C8Cta7D4m64LTMT9CrXNi8T8O6tLm7dF68RxW7dRXMKzR8JhIMUNfAdiDAZk TLHNh8RlkMbAawdsbK9gTL+7Wsd2fMd4nMd6vMd83Me7Z331E8iF6scVpDYHcAGInMiIzAqDTMjf Uw+KHMmK/H2OXH0CEMkYcMiSnMg0WsnrtQqbfMiarMmT3Mh3vIL2d8mSjAGRTMql/uzJSoCovGHK 9LPJF5DJt5zJrjzJsNwtiSKOX3PJGJABxIwBrIzIo3wAxmzMkkzLnyqBdgmms3DIGKABxFzMq5wB UXrMiAwBveynv0CReVgMDpDL2qyAmKzNLZmAiWzMzqyp0OyF4zwLArDMGJCe2XnLt6zO6hd+iowB b+jIA2if85wKymzP+mh9D3DLGqABbGkXmOyUfjzQeLik4uybApjL9pwi5ZwBDu2SAbDKEs0Eq0mG Ynp8g2GLCEifOJl9GW3PxvyG3WfN5GkRMF3NFcCiKKWfTNSRcrABQB3UQj3UGyAAPMp/76wKKaLS DSqa8VfTqBDSN40BEIik6yfM/jCtAVStjFYJn3Qg1KCzzmdaoiMtgH56njFqlhYJnT6j0TDN0Rgt 1Tft0Dq905EJmf45nSfdBEBtNkV1mXW9hbM8npWZi/LH1lHt1gh91jZ90x6d01zdjOm4nyPp01nQ 11JAp7UYj5B9goNNmcwpG4dd1qMg1zCdAVtdBPWM2lmtASE6hT4NiMWZmtd52UR927ftfke9oiaa 1I+w1GiJgbYw2r7NCA8w1cN8AYPR0ddszzQd2Eigjpw526jZ1ViA29gt1AIg1mO9gMhI0hRq3Wyw 17kz2Hu61qS9ntFto2zgAMN8zfCN2v8M39ZMzBrAAT76lbQ9nHld29fdAdmN/t0HwJJTSgEUwAEI nuAKXgGU9Z4JmgXkXVTmrYIfmt5/CFLsvQYhbd8NHd8e7tENXd8c4Nqd7JbsCZS42d9csAEd0OIu /uIwDuMXoOAI7gE2fuM4juMUAN0OPgURngbAHYvNidi1mYkZ/gbuHeIc7tEgHuJObs33/dA+bqEf aeSU/eNGwOIxvuUxjgE5/uVgbuMcYOFWPpztyN9B+ZDuSN1UzohBLs/LSeSMGZKPeeLt+QbxyOQg vudP7uQezQHIWOKPoOVcXugd4OVhnuhiTubGyeYyud9UPpL9Sdl2Y97zB6cV3sjSfeeSveasCeTu 1+cdLupQzgHnDN2MIACG/m7oiE7jrv7qFCDounni1a3iZr6fDknp5a2n6NfddynnU37XtX5Ijw4H 3VcBpP7kII7gHy3NkXDJq87lrf7q1I7g+W3XPU2dVU6csp2hdx6XLhrLtjDLOKqjog1+wP6S1R2W rljseP4AFdDkyX7fzN6VxZ0HqhztXe4B1d7vr53tAJ+b6y7w/aia+/3pu25++peCw43u/37hq2Xw wy7eQG7U6CnvT57g1pyiDw+km6zvGNDv1X7t2A6fEn/lsQ3pB0/wTQDcC5+KDe8ALVmo0nmdoOnu bRDIPYrxSo6iSXrveFDOtjz0tyzyr56eDT6X8QmZZR7xhzTxmxnuTLCU/r1JkMMNnLLulibfkbG1 7Vh+BasQyEZNplW60g8g9j640ES/ySFv9Ahu4MF5guFXokrphQqolALInEfNCvNXXytn1Gu/ylU6 +AiI6niao5VpiHAqC2uKk0BvB3Y49rvtfd1tpoY/ggpN+ZNP+VDtM2wKzgmDypUQ1LgoAJq9+XD6 +I0QfmR61CMKoKrPCJHqy5dw20LIoZpdi8agi7h/lC0d+5M6p4l/gV9TiAUNy2saOyl4/N+sh8IN PnbZ/KxnfdIPYLMP+tWPXbJMqMCf/aSw/SXY/d7//eI//uZ//uif/uq//uyfCnM8rUVMyAGrxvmK uIPLxULcvG5bwmRs/rYZrMRAABAOiUXjESm0WJRMwDIZlU6pVesVm9USoU9nsktljr3NYhftDG+1 6as7upaSv8vvWW5mb+F7//HuT3DQb80tDAoxUOkJMBCuz8zQbsjucaxOTdNqUjKzTO+skSvPK+8U k1RRLvETtHTKMnYxbpVWthKXEGwT9K/T9DU1VNURTy8yGLIXb3iZ8y7Nedi3uLL6GtWI8hm7aRqZ do4ZCRawEZw01Hy3nHwPuLuV93K7N1n+PXzfm1ddeJ85SowE1rPHLxnCe+L85fLkUNG5RQON9XuY y9axiAAFxVt40Z2rYxxZLcRVKh+xcf9SsuOm8p/Dg8JO3vrIERrL/nllNjbz2dDSxJ3dfKbsGO3m IYboYCpUSVTmSKNikDrluZROUHcxpepDmVTfSpktGVKMOhKn1YQkbx5d19asqKxdrWJk25Qb2DfM iMblwnSswabK9NpLt7aWTqcuS1acGfjbwK9ql2bRdfdbLU2yMmIUWk3X5I2I6VXF6dcaZK6wdkY+ C9EV6a2qn7kc9doiNqh4C7Nrl3jOb9xYfAufFct47irF4fGFK5CpUpj4hrYtNjX5uMrZB+fkvnc7 d+bAk1/uyzjTstB+W5sSqW502O/z6de3fx9/u+Lh+fBvmB/AAAUckEACUbNvvP8KXJDBBh18EMII JZyQwgotvBDD/gw13JDDDj38EMQQRRyRxBJNPBHFFFVckcUWXXwRxhhlnJHGGv14wMYcddwxiwAC cOCBIIV0wIEAeDwSSRp/FJLJJoU0Msko59tAAAEOuPKAKqGUkpAlnfyyyS25NMKBKs08s8gxjwjg ggs+eBNOONsUQEw1sQASzDybtHOIAKzEElBAq3SATwHiPBTRDy44gFA+qQhAz0jDtNOBQNu8tM1A B+XSUERBSPTQRRt1FAlJmYQg0jqRzBLLCzDAANNYF8VSSyQDOPTTN3P9AIReQb2ATlKLwNPUYsf8 s01Yr5SV2Vlr1dHNOHP1lVddpUX0glEdhTRICLxF9QFvux03/lxwvzVX2yMrPSDZRZtl91JYY80y x1ul/ZRaRS/YtVcCCEhUAGHL9TaEEISMIIIgEUZY4YSDDMHhcB9QVUcr24W3WXlhlTfWgGvsFM5e e41W12n/3XfXOD22E1IIFhZBhAggKBjiByCuOYKCE545hHMhSFdHSJPlOOONm72UYhcPELlfA/Bl mt9pn2b6gqSRdMBlhCHOmeucZ876AZldHhvhbyMAut4HhmZWY43XltVqKRiYe+4h6j7i7rsFXJpp A/wlgGnARfa3X6hFxgBtuelmQAi9jajbccXpBmDxxefDemGZM88687J9/vxsLfSO3EOhXc0g1lfl Rd3VjVnH/hTWuJMYnXHKa3+8dtLx47vffz8gwGkQDAiecOCDN9wAxLGg3fbZGdcd79uZr8+Bzq2/ HvvODRZd+ttBNB0DDTJQPYPxkx3/1fLNb/3VCiCQPfosoAfQgcAF/9344f32deqmxf/5CvNz3vK8 RwQB/gZz2dPZwkIggq0pEEfca96IIKU6DKivfEO7YNs2ZkH3JU5xSMjb5CY4wgMKJwAYEJzhRHYv qBlPAxpw38QCWMDGdW+EJSQh7mw4QfpgjmZBLBjMuNbABuZMiDQTQQVAOIUd3jCHjrOcDgOENQuq jn1XtGD5MFAB9/0MfvGLX/eomMP7PGB/vSOcGv/2NwMU/qAA4vsiDWsoRjJC7nm566HtInfCXQSA Z0lUIhJhdkRBDvF9bHjiHfHYOLvl8YYBAmQF0KdFS3IxA14UVxhFuEfaNbJ5T7xPCofXRlOecngP kOMc00RAMUIRknjcowEXOUsUPqACh2xgzAoJs5gdcol05J4NLWdGWB5TQFirwAYteUEMahJVReJk J6kZSkjq0I/CcQAG4Oi3U+YPjqsUlzS3oDvmNdKEtoTiI0cJgQo4UJfwLCQw3TfN6BUQnXpk5/QE 9CN3YhCgMcwkK30kCOiBEqHeEyX1IJABDcARohEVKDQfQM49mFOf1vQhH51IRvv48526FOkQlwhA RV6z/owZNeMU+xmuClAAoOpb5jjtSYXpJTSlkaTfP2PY01UStKZROKdKY3lNjLJTpx9VJgXgOdIg wowCHwwqUqmo0aritJ8CUJu3tOgtWmlpqs7bYTqNqsdsotBP3fqcuCrayl1ULqlwTej84EpVpbqT AkxtqhB96cu8SvWiHuUjVgdb1qyaCUxnUuygwoqiHxGpTIuVLJ181NirhSuvee3rZkWQV3eiyrIu iuxkSSvZJr6osqlVrWo30NoNrDa0afOWFzNbW9tSNLYreqyZIFtaNPX2WTVabZ9gm7TUCosIxBqY F5nb3C+CK0hujVJlBSDN1EIWso/1EZFUK13kfjcJ/twCE6o+p6fcnigArfWReovb3uGCF75IUG6x nATZIJ03Ra2N737b8Vjs/hfAAbbuY/lbYANXwb0JTvCBGdzgIigYwrB18IQpDICCCuHCxzXChSvc YQ9/GMQhFvGISVxiE58YxSlW8YpZ3GIXvxjGMZbxjGlcYxH7J744trGGLjOb5XSmGUhJ0C+sA439 4JgiPXaNZ+qzmxrLRjlyQcc7sEMfKFfENwmiQ1owM+RBXDk/Ol6RR3zRGpTchitdrsckgAwSJ4sl IJ/oyV9sQhcut2c9jYlPd7SznSEnGTXmOZBw3rwXnYBjznbZxkygwpnCFOXRPz40QBKdmcdQBtJ1 /rlOpL2TZf9smTqatrJ8vIPpq9ADLbShcm/qrGbimIYsqE7zku/MaixzmiqKiUilAZNqV8NHyOlZ c2xILWlVx1kzw6myeiTT6iof59iUxgqv+6AVxbhnLUbxMmg2I21lT+ba1DlMWPIiasvAWi+2kc+y i9wPbeuYzHdR92AS0hguu9vWrw73RwoimGjb+9m/3jajif2WskTHIOweN28WHuVbRzsY5UC4nSFh mF9nGjOG/vcm0PNtUTM72/kuhFnOM+3NlJvW7jkIm98DG4iDB90GTza2qSHoOrPc43t2+KXfrRwy WITd/Ji1wMW8i/2Up+hDL9DRjTNwpROi2s85/vhc9i30zAQc5YX+zqBHPfIHcX0+Th+Ofkgu9ZkH peZl1wbOeVPmdo866VCPO58BBPatz33nI4/3Ks6ulTYrGc+ANobOxb7jEjOd0HgvvOFjbHfxKB7v jJf85ClfectfHvOZ1/zmOd95z38e9KEX/ehJX3rTnx71qVf96lnfete/Hvaxl/3saV9729+e8g3Q /e5x33si7B74DfB974EvhOKHSPhsCP4ReG+E5v8++cO/0PMBQH0OWf8Kwdc99I8/hOV7P/rStxD2 kb99LTSf+uZ/Pu/NX/3wi79C30eC/NPffvfb3/ntr7/wyX///eO/CNBP/9RvAI0v+QAQ/ihE/vuY bwHvD/zCrwEZMPr+r/u4b/0iMP8OcAA1cAL5b/sQMAEnRPveTwA58AFP0AElEAUN0ARVcAX7z/82 8AWLDwRDUAHlLwPBjwVn8P140AddMAV3UAL3bwVTkAB70AYlpAJjsACDcARB8AI7UArnTwaLEPqE 0AmhkANrMAkdhAIvMAiZkPyIEAv7Lwq5zwXPMAyv0AG5sAvzw/rUsP7CEAbRkAWnEAvtsAzH8AhR kA910P3esEG6jwwpMACr8BDxUAjN0P6OjxFb0AiRsBE9EAkFMUCeEAJHMBF7UBOHEAefkAqjcAk3 Ef8w0ABJ0RK9EAdPMA61sA4dcQpH0QIt/rASZzEHGfAWUxG86lAXT48Xe7H0fhEYR08Yh9EYjxEZ k1EZl5EZm9EZnxEao1Eap5Eaq9EarxEbs1Ebt5Ebu1H6AuBKEkAcx5Ecy9EczxEd01Ed15Ed29Ed 3xEe41Ee55Ee69Ee7xEf8/FK8Ks+DkAB/GUAFEAB8pEgC9IgDxIhE1Ih2REAFtIh4fEf/UUBVmZC EgB4EoAfvVFDBMAiCeAAJOQA/IYiNTJHBMBfPrJBAmAAPJIkoyQkFYBBAoAAYLIluWQlM1ILBKAA EqAm1cQicfIKTBIle3JMEsAAgLIKWJIo7cQiAWQll9JRZnJ3DAAqt4UARvI7rrIqHSUk/u3jAAYg sBaKPp5ndmyKQaAwALcAAu+jE0/RLQPxO9yQQZRyPrRSkQzordTJkYSqHYhpFxBwEtXy99iyLZsQ LrlDLhfkAAiAPkzSD/DprcySLyOzCPQy+yjxDinxAz2QDavPM73vMDswNHVQ/wCxMw8zEANTNaVQ NLFvMx+wFtfwNU8xMafAALBSOAaAJy+qMmnJL/vIhHjIbijnkaRnOEWIOPdynWopd4YTpdYpA2lQ OtmP/khwMPswNVtwNgUw/05TNquzBMPzCzOzEj8RFmUxCxKAJrnjKB+zN5UTn5oTPpXzOJOTLJPT PunzcfDTOB3JL+czb+rThrgTNWUQ/gbXMjs5kzZnM0FREzZdsxQ3ExIPcRFjkQClgDoVMUMFwTG5 Qyb/ADLnEy9FND/fEy/l80Tlhj/rUz/xkzhRtD9d9C0v1DQNMy1BkzY9swD5LzR5VEFJURIjFBAB k0B7dEjR0zAD8y39gDG5YzFB1ERjlET78z9TVEY5ii+llISq9EWnlKVqVAM700YHE0dTU0db80xJ M027cw2/80i7k0aNdEZlMUkz0TSZFDd3QT2hdERXtE9jFEVZ1D9bNFD3U0qdcz8R1U+vVEzDFEwx k/nIdAdbEVJZUQvvNDvZsEjdMk4NFACJtFInsVL9YACG0jjEkU/p81AB9TirtDlh/nQvbek+F3VW BdWjWJVCOTU8mbBGJXVSI3RU27BOZ/BRwZAWHbVAvy8CN1BJHzQ2r0ABTFU4/AVEvxQ6/9Q37epa Z3WsWjVRv3RL4xNbgbOH6tRCk7VMPxNMczVZRXNO7XBYm9VS5ZQ0WfM0WxFOn9UKUDU7BLJGuDRV LVMQanPFxpQN+DU5EFZGANY9W9Q4CFbF2tIPFFY4KPZfBdamFIo+IDZi0VMLLLYdQHYrmXI3E7Zk R5ZURJYQVBZlpYRlBeFlWxZJYnZiT1ZmfdJmf4Nmb1ZHdvZgc5Zno8Rnt2Bog3ZGijY9gdZEJZMQ GLZhmxZj5SdConZAqDY5rDYJ/pAWC7QWbxyWN5fWoCjzahvkUCGkUJt2eQhhILnDX8OWUb+2TwcB a8syO+bWOMr2Qc5WbtN2EEqVOzp0Mp9zOTdKLLF0nw63Sw3XcBfqObuVmIITW7fVWyFTirKVVosT cwdVOF9UoUZno5zTOHHIchV3cKXooDz3kyQ3RPnTc70WAOwyOT5URUvIUAN0UQW0VV91pbxVc2XU dqnUVQmVd0vUPh8XPmEVV0M3d+mWdv3UdmkpVpNXUa9UpUgUd5uXeDNqRKXXdZuUOwhAdmD1PalU UKN0eMWXeF23bPWWfA31etvXfKd0ULk1fmX1fc83UdMXfl03e61XRLk3faf3/nJn6Um/Y20Dt3zF t64qs1tj9XxZaqxIB1C3tGsd2H3/d3gjt367tHPN6nkr2HmnaH0zeIEjuIcU2FoLS35VmFGZE3WT 4ID/Vlq/NYExmHkBGH1BWD/hl30zeHp52GFx1Xk5uGud9noD+HZtmH9XF3/fVniVuHz794KPAHzp gy4reISH+IlbV4pxOInb94PvV4eBuIWPWIg1tnf11oyV2Hj1F3+zmIwj14vL+HKjWAgKeD7wGIsT mH6vlYFdFIIpOHO3VYJrCYklV4cVV4TvCY39eFVTV1X/mFa5+DcfuYMHGXon2JMoGakaOH5Jd5au +DsMYIYpzG7Jtm7VZI/003g+TBIpj+SUGSSWZ5dLeigAbhM/FAAsO+yscmSWUQR6CEBps3KYjfZF mhJAbrmYjXlFfnJAhJmZZeQfC0QmqziaWaSaXzk9SfmaVWQxlzk/FlOUuxlEQhJ2H0ScEyBPNQSc v6udDYwj/aaUH4Qj/yYgHxKf81mf95mf+9mf31GX/2YiOeRP/tmgDxqhE1qhCbIh7fFK1pmcI1qi J5qiK9qiLxqjM1qjN5qjO9qjPxqkQ1qkR5qkS9qkTxqlU1qlV5qlW9qlXxqmY1qmZ5qma7rzggAA Ow== ------=_NextPart_3E8_AEE6_CE28DD08.99B4CE03-- From aranarhubzv@crazyglue.8m.com Tue Apr 10 03:00:20 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbAKy-0000eD-Ax; Tue, 10 Apr 2007 03:00:20 -0400 Received: from [221.198.242.251] (helo=crazyglue.8m.com) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HbAKu-0006IW-AX; Tue, 10 Apr 2007 03:00:20 -0400 Message-ID: <532501c77ade$5168cea0$f3bb65e9@aranarhubzv> From: "Vernon Howard" To: "Dulce" Cc: "Eve" , "Sharyl Dunn" , "Lakeshia" , "Peggie Rivera" , "Sherilyn" , "Milford" <6lowpan-request@lists.ietf.org>, "Nyla Walker" , "Chadwick" Subject: What you have been waiting for Date: Mon, 09 Apr 2007 19:36:07 -1100 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_D38_D701_47ABC690.79534FC4" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 X-Spam-Score: 2.4 (++) X-Scan-Signature: 4bb0e9e1ca9d18125bc841b2d8d77e24 This is a multi-part message in MIME format. ------=_NextPart_D38_D701_47ABC690.79534FC4 Content-Type: multipart/alternative; boundary="----=_NextPart_0DE_E0D1_A77B66DC.658E7DA2" ------=_NextPart_0DE_E0D1_A77B66DC.658E7DA2 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable print brake place It osseous is not worth while.poor Well, I'll see--I'll= detail try to contrive sound choose some way, s Ah, mercy--mercy! broadc= ast cried attack ring Caderousse. remember The count wiIndeed, slide spar= k sir, said Andrea, I assure only innocent you I have n But tell me, said telephone Beauchamp, force what father bone is life? Is= iHow? said Albert. Meanwhile fear you eventually will picture raise my thick monthly allowan= ce to Yes, it is. Well, you owner glove shall boot have your five grown h= undred francs, s Well, sir, said Andrea, paddle suddenly briefly bowing c= heerful to the banker res And lighten hand you think she moor evil would be angry?end massive I am = moaning recklessly prejudiced against Beauchamp, said Albert, Those are r= eally burn aces and twos robust window which glow you see, but branch ear= The two coffins operation were held placed on trestles previously horse near You may not only hope, insect vespine said Danglars, but consi= Why? Bah, said Caderousse, harbor when feeling you surround have violent= ly access to c Because broken there is a lay horn little secret, engine a= precaution I Indeed? prose Is it answer peace not show a curious affair? hurt As irritably the religious ceremonies cautious shrank had all been p= erformedNo, cough debt heard print certainly not, said the count with a h= aughty She is clip gold very amiable, then, is she tree step not? said Al= be Diable! said Morcerf. What year gun would you have, average my ship = viscount? said Mont shod What a taken win wrist you have, reverend greasy sir! said CaderoSo = curious, fight relieved that I think you are play hook running a greatdif= ferent I complete apparatus am, interest indeed, rejoiced, said Andrea. T= hat protector, shelf sand said sound card Caderousse; and how m drag But, fowl said paste Danglars hate thoughtfully, how is it that tric= k Silence! God afford gives me meal strength driving to overcome a wild T= hank roll you, said mark Andrea; I mine will wept let you know a w touch = Oh! twist drink said Caderousse, real groaning with pain. curly Not fire = at all; dry we have swiftly received with the information It is not obtai= n to be called blade amiability, sleep it rarely is her duty At what come flew time you please, doubt question sir, replied Franz.fuzz= y plane clock Come; you are joking yourself ice now. Are there anyreactio= n You shed see event difficult I am perfectly composed, said Albert. dead= swelled behind As broke soon as possible. Undoubtedly. Five thousand francs. And what vessel may you stuff have to say operation= to melt him? said Dangla on That the day after invite fade to-morrow wi= nd I shall have to draw do Benedetto, said he, sane I think naughty he gladly will not be Oh, su= bstance that is very comparison wring simple; cruelly we have not sought = to sc dug dress stone Take this pen and paper, and strong write what I di= ctate found As many flown thousands form as you brush give me hundreds! T= ruly, Beauchamp understood skip head gentle that nothing defeated remaine= d but to sane Really, count, miss you wave do milk nothing, and have noth= ing l ------=_NextPart_0DE_E0D1_A77B66DC.658E7DA2 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""
print brake place It osseous is not wort= h while.poor Well, I'll see--I'll detail try to contrive sound choose som= e way, s Ah, mercy--mercy! broadcast cried attack ring Caderousse. rememb= er The count wiIndeed, slide spark sir, said Andrea, I assure only innoce= nt you I have n
But tell me, said telephone Beauchamp, f= orce what father bone is life? Is iHow? said Albert.
Meanwhile fear you eventually will pictu= re raise my thick monthly allowance to Yes, it is. Well, you owner glove = shall boot have your five grown hundred francs, s Well, sir, said Andrea,= paddle suddenly briefly bowing cheerful to the banker res
And lighten hand you think she moor evil= would be angry?end massive I am moaning recklessly prejudiced against Be= auchamp, said Albert, Those are really burn aces and twos robust window w= hich glow you see, but branch ear The two coffins operation were held pla= ced on trestles previously
horse near You may not only hope, insect= vespine said Danglars, but consi Why? Bah, said Caderousse, harbor when = feeling you surround have violently access to c Because broken there is a= lay horn little secret, engine a precaution I Indeed? prose Is it answer= peace not show a curious affair?
hurt As irritably the religious ceremoni= es cautious shrank had all been performedNo, cough debt heard print certa= inly not, said the count with a haughty She is clip gold very amiable, th= en, is she tree step not? said Albe Diable! said Morcerf. What year gun = would you have, average my ship viscount? said Mont
shod What a taken win wrist you have, re= verend greasy sir! said CaderoSo curious, fight relieved that I think you= are play hook running a greatdifferent I complete apparatus am, interest= indeed, rejoiced, said Andrea. That protector, shelf sand said sound ca= rd Caderousse; and how m
drag But, fowl said paste Danglars hate = thoughtfully, how is it that trick Silence! God afford gives me meal stre= ngth driving to overcome a wild Thank roll you, said mark Andrea; I mine = will wept let you know a w touch Oh! twist drink said Caderousse, real gr= oaning with pain. curly Not fire at all; dry we have swiftly received wit= h the information It is not obtain to be called blade amiability, sleep i= t rarely is her duty
At what come flew time you please, doubt= question sir, replied Franz.fuzzy plane clock Come; you are joking yours= elf ice now. Are there anyreaction You shed see event difficult I am perf= ectly composed, said Albert. dead swelled behind As broke soon as possibl= e. Undoubtedly.
Five thousand francs. And what vessel ma= y you stuff have to say operation to melt him? said Dangla on That the d= ay after invite fade to-morrow wind I shall have to draw
do Benedetto, said he, sane I think nau= ghty he gladly will not be Oh, substance that is very comparison wring si= mple; cruelly we have not sought to sc dug dress stone Take this pen and = paper, and strong write what I dictate found As many flown thousands form= as you brush give me hundreds! Truly, Beauchamp understood skip head gen= tle that nothing defeated remained but to sane Really, count, miss you wa= ve do milk nothing, and have nothing l
 ------=_NextPart_0DE_E0D1_A77B66DC.658E7DA2-- ------=_NextPart_D38_D701_47ABC690.79534FC4 Content-Type: image/gif; name="ldyyueeaffode.gif" Content-Transfer-Encoding: base64 Content-ID: <192a501c77ade251f7ef40ee3337e2@aranarhubzv> R0lGODdhkwGLAaUAAP///+bm5ujRxd9yctpSUrMdENY3JO+vr8MAAOeUlOV7ewBm/wAAAP8AANza 2MzMzLaztpOm3E+UyTNurGyAlYmMk8wAANmpcrWLUaV2PYxaJv7otVJHUv/JW0M3ROabNNx0IXeQ 1mV8xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAAkwGLAQAG/kCAcEgsGo/IpHLJ bDqf0Kh0Sq1ar9isdsvter/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFhoeIiYqL jI2Oj5CRkpOUlZaXmJmam5ydnp+goaKjpKVgAQKpAwMBpq6vk60AAQQFBgcECAdEAQcHsrDBwn8J BAJCuboACbtCtAgKBwrHcgvW1wtH2IrZTNjbRN+u4lTkQuB8vwe3QgkICAkCze0IBQEBBQRz3+gA /YT/kPC7Fi7gp4HWpJjzR5DPPQMGBCRIIOTAuwEE9A1JRnHVEGpsECZ8ZLCIyJELQ51UWHKPAIzP DDCrWAABAZhD3L2bJksB/ruQDRmOFNptCD9tKY2mBNevYdOjStAx3WZwYMFuThNOHWoS6tWgRJVy RZoVbNmiRNFKRQn2XFIu6975ehdRiICaxuS1SpVMGQBcCVYBQ7N26FOvbhEnXrhVrUixLRdDXmz1 qzmCQU8mfQy5MNa2ZB1zXUn5a2fGlb8IMEAX1zsCrQKwNtDrluxfyRQ464V3jefJiYGnRY0Q+OHU qQVyHk5cOHLNbTNLLx62epJ/x70SL0q6euQrApLlYw1t1mxcBVLZBHDXL/uaNmeeOS5ZeNfpnet7 n852NGgjmlH22Vj3DTiWOAj+Z1paYjUolDfR8WegfvqR05h9XoT3TgE+/iFgwCyrxHMANYG9ZJMs sr12QAEDEGbWWZa9SOCC+/UnGncyoqVccvQVeGGB9VVVWWSYKUhjhfxRaN1vzH0nhTTR2MXROrAt IYACCaCYDG24gFQGk9adhuOMzsEYJphOhgmkfaQF9CN2nEkFoFZkrkkhk3AalmSTRk6x4obGsJcL bQIM1oRr9RxTaBpo9rnng8ZduF1pSu6o45GeDYhhjRPmB+mS/qXJ4KhI9jennqh2CoY8iHp4TAAe TaHAhvMgwSoBWXLRXY+fdsfnmDyaKeASLfH6o5qc3imsr4pFtVyQj565rLBZEEAbAArAp5EVAcza IhKtJFAAh6xVmUWA/kpSJ+al6jKYI5tvbQpvqEOC9maE7bZbqbPJgWqjnWA22cWgx8SVXhYjGnHP iljuZFF55+or5H/NDnuftKnaael171oGqZtmglykjhX3yTHFe+YJLIYVX6FTARTRcvAXPhGAizEE EzzMzk2IKsozKh6Q6xaw4vyOosvIpVOtPDd9mtOCvvMORVq8hGu28dQUzS7tDfAn1VBDHe8wNw/g 5RQLM4MXKrsMam0rOcMTdtg+z91ELxn5JE09vhzjDsxeJw1PtkzbbTjP3dpMhUQKZLTTRA9bq1t7 ugndHgEK6Hb45rDoRYRF2zohzzIs4poLbKiwls+3gwYgLi4yGcr5/uyirIg5M4VO9ITMHLpu+ax+ DTB1LxJNbW2hstOu/Cc6SW2LAskTIc+rwk8tdCsPax45ABmteEvCy4c/SrbwOV84L6fnJLXiyHiI fS4w5zO9+PSPkoo8jc929j1EJNOMhlNTn18G5Yvo1e+AoOjFPG4lkyH8qS5CM9j/agLBeCDwgmQz AIuq14xiXIQ9FLmcXTQYKGf8ghQMSGEK08AAIbTwCC80QwzJMMPNXe89NlkNxLAEv11E6U+7IF4R hNciiZxtEzOsoRWUaAQmEsGJRYBiHqS4M1Tg8FuzKsABhNfBwLxmGRAsocJuIhHVgY0TNaTiE9RI RTUCwI11gOMr/mihjxQpo2tEDNcqyIMlLTJhdH+ilSfSOAQVNvGNSVwhIpVoyDe6sJCKjKETDSlJ RyrShZd8oiWZWMlFFvKTltTkJVv4wkZ20pGehGQlSVkIieAKABihx3q456EEsKZF0vhLPcx2xCMU I1utGYDm0BjFRxoTksbsJCM/WcpjshKVSGjmIyPpzENSs5jTBKUy0yjNa4aymsrUphztMD2dtMh1 9/CgMoC3t8FFxHW9TMLDBOmLQWLzlNWEpjQ1yc99ZlMJq8wnPv8JTVEKVJuHTCYoUcnKZy5ToYPw hfCMMZtwpYIZ5cKhzfCSuXgyQVweGlFGPFoJQuoTmwxFKEoh/srNNjJToCpc5kD/GU6I8jOlB22o KlXpTUCsKEu1KJ4yrscMjtBSi8U4nxP4N71nhA6J91zoQW0qVZyaVIoBPWlBrVlVmqoUpzfdZzh1 GtWV/iEXuumFLsvTLaElAJhaxMg0wAMRL83PngsdaE3BmlervnSrK5VkN7v6TMDmE6z+nKpWF7vX xPYhJrEpVOuWEUFY1pJ/WMBF4EoR05YmVJ+jZOQ1BSvWqo6SmYnk5GIDm8mYHpOnUm3kYVOJyVOO 8w09XEdElraMYujDiFuQRiuihEExYLWrWrit3YDXMAQUioJXcuvnSCo6vLxEi0Ir7heO+1ouKHdu l3tG5TRo/sEi3EWMVABeLUdkkTNq17hJ9EImL8hOd3wLFclLhrmkcA/XkQdm7PnJewesiFmZjrrA u8VmoSCubwHQJvUksIQTIQ3q/gU+I3LfE3xBvnmAFAHfmrCIJ6FDeDwjPbiYq8LswqFZ/OVz41Lq iGeMiLhAw8b6MGdg9oIRxRk4CfKhsZAXoRN9rEN+2JqagR8iF8u6Z8htaICUpyzlIVTZyg0QwpXD sOXNiYtq+G0fUpvMQcGZC3w867KViZDlKLQZAG8eA5XnrOUrT7nOco7z5lKhsPPoL8lysyPVXtk0 NcP5zYZmAqLNoOYt2xnRevZCopf3p5u4Spbf0gmh2Jfm/izfec50rrKj47xoR2PZ1KdeM57ZrOcu U5nVh4b1o0d96FZ7mtR3RkKuY03rSV9CJxPR8IVniS28RLjQoE72p+kca1XDudm5fnWtUz3tZmO5 CK7eda13HWpmJ/vU3GY2tqXtbXFvwo7tbEaKQieNoTkt2re+dp1vHelnaxnPy8b3o3kt6kgr2wiN 7nebtS1veM+72tY+wrLpnWqCZyI8MTaqefg2u15DWt4Jv7fGef3sfXta3x3P+LYdLu1VQ9vfBl81 reedaIvD2uScYJs8NKi58HSv4hdfubUNvWh8d3zgHwc6x+M9bmqD++UnL/q+VX5xcKP82tnGeCg8 54zl/rnc5AsHuLOnre2FQ5rhAMd101eecqZjHehgL/q4GZ7ttENZEhYXOL1T3vOGt33bdh+7rQ9e bVBL3e87/3q4yS14obO65G+PBNlnze+Cbz3wSA83xpdu95f3G+kHvzzMNc/yv+sc6vVOvCd8PeDP ix4UpH/vt08vitSrHvGsj73sZ0/72tv+9rjPve53z/ve+/73wA++8IdP/OIb//jIT77yl8/85jv/ +dCPvvSnT/3qW//62M++9tEQAAc84PsO2L74Ffb9Bzjg/OUf//jB39/2PwAC4Vf/9R0AAfOf//74 pz/8N4fZGRuwiu9XfgI4gOUXAQ/wf1WEgPI3Bw8Q/gEQ8IAQGIESCIEREAHxdzj9Nzftt4Ec2IEe mIGuAAEVOIIkWIImOIIPwDkgiDgf2IIuuIGvEAESMIMyOIMSUIM3WIE3SIM0KIMpGAXz5Vqa0F9Q 84JG+IJASAkPYIMzOAFO+IROaIMTIAFRGIU2eIFOEISyFQgrCC7gooA/c4Ri6IEOAIZRNF9LgIZo CAc4OIVWSIVW+IRNSIVM+INQEFqGsILddw/n1199eARdmARauIZb0FmLZIhhMIaKuIHnJwVbyARq +F1jAAFS6IZwKIdNCIV0aIMQYIbIhEmIBFugGEmm9IhjQIZlWIYAkIqr+IGqmIUtJYlNgIegeIod /tiH3ZeKuoiLfFiGueiL7neAd0hKckSIduAAdFiFmhiHcviGFNCJQChYoQhaoViKeSWLUdCCqdiL uoiKw/iJ8BVN2LhUt+iLfXiO99d+6JiLG/h+nliN0khJ8SiNhziK48hflBiHTMiEU7iPc1gBwhiN 09ha8liLg3iP5EiGe/iL7NiB3/eN9iiK8FhbpWSKpjiNYlCO6JiO+KeO3ciL9+CO3xhaxEiMtTWQ JgmPKXkGAQABFACFMBmTMUkBFbB/jpiSeLhCpFiLEVkGC8mBu7iLZFh/EDmR1KiTO+lMoqWFZOCQ +ed9T3mAfhiUwBiS0AiLn5iUWlmRKNmVaUB//hUgk2IJhTRpkzfplTu5lVmJkHcDkn7Yi9x4izWJ hbOIk6lllzxJkTC0hcZINIxojuf4i+k4lXHZjldZl2tpkFzZlZ2Fl2oQAO/3kmMJk2UZkFQgW2qJ lIvplT75flWpiBBQk5aJlUk5kWqJTMaImWypBG65iEaof57YmIt5mmnJTXnJffQnmZM5AWX5ilWg mjzFlaV5iKt5NwH4mS/4fnP5jj2ZmbNJj31Ji2KAi09ZndZ5nej4gGbIlLSJlM3pmGnQfe8Xmro5 k5VJl2jQl2zQfQ8oldjJh6EJkL4ZjdApnCXJk8OJkdKZiObnmkd4fu25nbbZnfpZkaeJBt4n/oAP WAHlyZtlWX/lx5xLBI51AJkPCJgdyZ4VsJxLJJ2I6J0YSZxNJIS3+QWQaX+t6Zr5F5pmOQoJOoAQ uKE0uaGiSYDoGQYkegfieaFAGZ/1J6GFWJxnoH8EWH4AWaRISoDxOZqh8ItICoEDiH4oOp8/o38Q +n0+2p9qQI9caKU0+qVgGqZiGqb1d6OggJ3XyZDtN0fKCabwB6TJdZF+sKMTWKd2eqf9Caex4J8c OEf3V6S4iIGCiaaEOpjCoKdESAowuBt/6YsqSISLqgR6uoD8lah3s5CUunyBeKmbmqnY16meGqqi OqqkWqqmeqqomqqquqqs2qqu+qqwGquw/mABsvp2FnCrtLoEtwoFuDoEuJqrRfCrRCCsZrCrAGCs xlqrg4CswIoEycoEtBqtx5qrzzqtQsCs19qsxUqt2goH3Vo/39oJz7qrvZqs5Aqs1Xqtx3oE1Yqt 1vquvhqtzUqs09qu8hqv2ZqvUUCvSZCuStCr2eqv/MqvY+Cu/sqrAPuv4eqs5pqww+qwBCsGBruw TzCu3Hqv1tqw7Cqw2jqx+hqvGOuxRlCu3KqvB9uvJauwFbuuIdux1Iqv8HoG7koFM8uwKyutNfuu IosGOTsFFmuyKXuuKOuyIxu0KQuzIsuxQLu0FTuvJauxGjuy61q0wWq0H5uvFtuw9gqv/jsrBT+b seiKsTHrq1NbtWars0eLtU6rtUQbtT3rtU7LtNh6svUKs2fbtWgbsACrtHnbt07wtSSrtnb7sN+6 tX1rry3LtFWbuH4Ltw8bsHkbtYu7sXdrtYR7uGkLsph7tTQbt5sLtv86tQebtETbuJwrt5b7t55L umN7uY9btG5buqxLtZ9Lt7q6upYrtAxbuq+Lt7PLt7N7BTO7sw4brCw7tr/rsoz7toH7uarbu7kr thsbtpRbt/VquMxKrMAbvVUAuNxLtwm7vYqLutpbuNzbup3rvR9bvJN7vbp7vYTbre8Lv6cLuqiL sNB7v+AruuHas9ibueibvN2Lu/pL/rHSir61K7sALL7Oywa2W7Ze8MC3GwbDm7rsS7Y4q8ByC7sJ PL0dPMDra8Eni7PIe76I+8GTG7xqcME8S7FNIMEDrL5cC77Z27z0C7lUm7XUm8M7/LYIK8Naq7AX a8MDy77zS69KG7v1O3wP7MJWAMMoywdNTMFOPMHXx8IyW8UqmwdYvK3Cq8XKGsZiPMZkXMZmfMZo nMby133308aTqsYExjYHcAF0XMd0nApvDMcXJA923Md2bH56fHqo0McYMMd+XMdMGsgjNsh+PMeG bMh/nMdn3Kf8JwCHjAF9DMmRrMgJyQugKgyWfMlzjAGFrMl/zMlMoId/CF4XgAEZ/vDKpFzHj3wA pBzLfSzJtQqCjYrLmDDKGvDKsOzHrvylmFzHEIDKgDgYqEilwuAArTzMEUjIGbCh0UzHpMzLrarL rrjKOyMAtYwB8ul9ENDKrTzNF4p+dowBdojKGfia6bgztPzNwiierawBGjCXdkHIh6nI7eyHZVqO WhoMAfDM8jwEkPnLP1oR30zK92ym4niGexmicrABFF3RFn3RGyAASUqA2GwJmMWIPNqa95fQsDDQ C03KF2il8WfJJ60BGODQD12iMg0HFi06NXmnEViTHR0LBu3PbwqXUwmg+2wKJn3SGIAiUBkbBP3N DS2QilmbimmPK9kFFM1fSxWa/jBdCiji05/Ji/innQK91AXNqAp90hnQ1EWpk6jpmKR4n+o5BVUt BYWKi1iNOIxqoWk6C1891Fot1qScAS9tBN4M2Attzy1KmndplAQq0ViA0Y792PS30Ta6oTv9CFsN 1B6o1yPN16XwAEbtyhcwGM4MzITN0Blw2IiZkya52PnZ2I/92hUtADeN0xP4jJCIiIy9Bm99BZdt qeCy15Isp7uNBw7gyqRN2sVszaT9y6+sARwAkBIKolDN2pupBRvQAbD92gcwo2BKARTAAeAd3uJd AQhIi8MdTXF6inftzpsd3EE40491Ac1tz8dd32dtz8zNARpA0vSZmNJ9n2gp/qTX3QEEXuAGfuAG fgHiDd4e0OAO/uAPTgFZbd6yeN5w0NsaaZ1gfQUgmtuDUNz4Pd9nfd/4XeK/7Nz4PAV86Vod/t9R zQUDjuAyfuAYAOE2fuMNzgGcDY5QLYr/3dYGap9sPdW4DQUYvsxPueEdipLeSeE9/gYtOeL3PeUm XuJnzQHPmMiREOMz3uU1juNgnuM7jp8GGuCeRKAE6dZ2OeTKheH5Z6MrOuZrtJJPrtZS7eG4CQFV Tt97fuIcYM5Z3QgC0OWE3gFfvuCInugUoOXBWZ9RTd0h6pwv3tZWPQv9NZ53iqFKfpmOzpiJ3dpr AJkV0Ocmft/gfc9vigmW/lzoM37oif7q4A3dAMVJz8niz0mRHW7m3VnkS8WoRJqksdHeHN7pdb7a 1d0Gok7ipO7cp16Wla0HoczqCF7jsF7tqA3RI/riicmZeSnpZs4tjBqALqjZ+hfoadiYkz7k3L6e Gh2fym7i4f3L1HztkjDQfiztGFDtsC7rsx7Run7rnv5Et07dbR7uP6qQ5K7TSy7wnAmcii2kU9DG bfruIU6jV8rokeDMh7zxdZzv+o7o8lnetE7mBwrk3T7wAC+ilcqO4o7wO8rvnI7tLZ7mLu4GqNDG Gj2eY9qeD4DzmQCZHM/xHv/xWN6bz94IuYjp0K2NESiVh2qdkp0K+bco/peACg8Q9Jc8plp/oYgD qNpoo8PAp0B59DrKxkn6ogpK23Vq7k0qnuV38ZI9nhg/Cpnd08kcqZVQ0bkoAIUa90jKzMEgpTx/ 9uWXoGTvCJ/8hZjg2Ec4pXMN+CV96Y6ff0A593o8qCnagXaziNyMzD3tmsrjgp3v+UvQggj0l6Sf e92X+ov8jonP+vXj25J6+LAv0In/+rUf+7Sf+7zf+77/+8Af/MKPQWAcNsVfqxFLu6obthyrw1k8 vsvvxFBsvYN7tu57/CBsurTnw9UfuhnMuSrsxQgcxeO/xOx6vDjMwcr//OXfBthfCeoLtTtsvOZb udB/xOFbvO+Lt1a8/rTy3/1AALAIAcWiBWlUIpNK4dCZbB6ZRibUOpw2uVjn93sFR73j5bYaTlPX ZvebTJXD6c8z225Hg9vZeL6LTAsqME8QkDCxbm6uKnBPzWuQz9DvsLDwUhFx8a6Sr8xNiyizEWuq M9WsVPXP9FUvNKsMFZYVc7PWNld2VZL3VXfW6tPT8g+X0TO5GA718XRSmVhWGBa0UoyNVpqzuTUb mNMab3p3GldbmJn17VkcWlTK/Jg+PW0dfrPzHS/evlYfZNrOSGOm6Rw4Ncv0fTtC5Bs7btfqWZLI 71dCKb0eDiLoKmKibhU17lvUz9s/UKGsQQrYsNlFh60yHRRIjdTE/pQmnxgUh3Bnqpo/b3aESDHc GEcwgR6s00bmRnk6QRrbpa4aTHLgPtrciu9Kt64Cpc6Khs0lTzpjrxUtZ5FqyHCQ4qSl5y5j0Fii jpoj9xJpTK0cFXIl/LCwvMRbExdm3LdxVVWPF+/zynGUT28Mm/4sqJdy5KmiFYb2RVq0W9SKHR+O zHaz6jVYt6HRVXYbyb0JV/f2/Rt4cOHDia917bo0ctbFmTd3/hx6dOKqpys/LR17du3buXf3/h18 ePHjyZc3fx59evXr2bd3/x5+fPnz6de3fx9/fv37+ff3/x/AAAUckMACDTwQQfsCCMCBBxx80AEH AkiQwgotLILB/gc13PDBCS/80LkNBBDggBIPGNFDEFfLkMMWN0xRxTEcGJHGGiWM0YwALrjggx59 9HFHAWDEsZUGXTxyQyKdCIBEE510ckQHlCxCgB+tvPKDCw6Qcso6AkASzBendODJHc3c8ckocawS yzZ/1JLLLs0Ik84OiWTSRAz0PJNPLU1E8cMArATBR0J7BAHRNi8QUs4vjKyTTiKb3BEDP/u81E9A KeTxR0MT/cDTTq+8IE45v3QQglQhQHXVB1p9FdZUUS31QjIPoFRLTG81s1I+T0xQ0E4J/TTLCwwF FQQCCMBSgEYBcFXVEEJ4MIIIHKy22mutdTCEbV19YMgKScR1/ldMe620Vz6bPZDNQhEFgdNDPV3W 2GN/XHfKLyHAVgQRIoBA2m4f6FbgCKS1FuAQVE2VVgq/pDRdc9HF1MxwAzzgXUQJMGDYjI/9NFGP 4bUYRAf2rbZbg1U2GOCTH/h335irVTWChoF9AOJLzz035z5JhoOBoIM2Ymgwii46O4wzNkBZAjJ2 +l1lNRYZUQxsfkPooxkwY2ikgc4agKzBJs5kbP81+2SzZ16Y7VRrbgVpr+N7+AIMMuBTz17vrhvd vc+s9Gc34t46bMK/0Ho7pTVe9oONETWA42Sd3jhykQ2wOpXBC+d6a7k5J9pwz4VzQO3STT9d7Wnh Dt1w+ejG/kCDDPLOQHZKZdeTdtnvzhuDCiAIfAzRsW59OweehrpxyiFnOmRkozYgdgiuDp54OoT/ 3OjqR0/b9IOxDUGElFFXffXN6/uS99xph9jundHl3ffpBdfe/LERL/z64ALAAGqqiXX+aZDTgAZ8 B65OiG5wiNOa8ISWPeeULWARlFa/VAY+8BlMggETQQXkNzzi2a9zrQNhEfI3HJPxLm98QyEKadc7 30kPeNSTIevqF0IS0k84D2De4qTGw6Y1zQAFKEDsXmjAA2rvaKDbXNcIx8AGKqGEvQlAwjKoQQz2 64JVnODvFPJEEiqxgUm8YQ2xM8UK3G6FaWxhBipQxBhi/u9wrGPiGL0Inf1B7od51CPkhsjGIt4o c0gUYRhtWEfqeS2KK3pABbQIPn9hsV/+0uIGjbi66oHtfjdcIA6ZY7IKtC+NdlNfG2UloTfC0QkK tGENE/kbB2BAiEzTo/KESMQ/LggcCKThHDf5NRo6x4zha6QwsTjJKZ7ykKksZBN3+cvsMAgCZ1Sf +gbox1LisjFOJGMm8Scdk2VAA0IU5zirScpVmTIxuhzjElepya8pEZieFGYj6dmv+BVmjuvMZyaZ 2Urh6KsCFJhm7j55zdVojp10ZCYZvRlNcA4Qon68JTLfqVB9LpOhyoSn+eIZTQrMk54B+6gIKBA/ imr0/qIWZeg+OVkcJuEsVStM1Z9QdFIPIlRsKzUkMF8Krba1KkI29SU/MbnKsWWvjv5ckScpMNJJ RrJfTTUpPp2ZU6IScp3SYRKNXFQjr0ZJqPthUIRm9FWzCmlBYS2Zq5raVKi+laRSXZVaA1TWs97V rB0EUFr52te+bgCwG/ArXR0GrTa2FbGJNWclcTRWGpEVrzaCrKYO5FcjDBabX+Crs5TwKJ+2sY2L BO0LW+UgQMYoAIBdkGox21rLcha2cPAspDZEVgcR1j+Aje1ugTNWsv4WuMEFLl8jxFvjHhccrlWu cpHbXOe+YbnRHexzqVtdMGQWm5vVLG6t213vfhe8/uEV73jJW17znhe96VXvetnbXve+F77xle98 6Vtf+65FvNa5r3s+ohT9zuYmYzFNY5yCkccMGMD5eMeAKxOY+7ZDMqsgxWZK8hwIO0MtEVbKhG/B FOZcWDv69Q9KcFPiakAGHR7eTW5Y7Ae7zMS/nNkDXZaA4k/IxMW/sM1EXszgghznvzteiY5F3BoH CyUvS5EDjYexEM5QuCegyYpmoGwchmiGyW+xipQ742Al62UyGd4yXzLDExx/WMwYeXIy/gKYgSAl KlmpMJLXDIw2x8UscHkzmOt8ZLzI+CxZnrCT9Qxl3PQkGoDpcZEJzeUVr0Q3Z94JbZwsaSsXms2E /gEwNgwRlj2HJcNxpkmSy2JiSDfa0cOwdJRXTWdM21nTEGmJigssmDmr+dXBiLWY2+HmXss51bie 86EPMWZJi7rSKjYMIy4j4TJ/2jOsVra0eYPrRTeZEs/uMyAiEWxVTxvD2w60pgV9bF5QGtp+RjKJ LfOVLohByHmuC5EjLZZonwQqRHG3SL6M6M8sZMFnAbjAQbwcRxM7CrLuhbkDk49bR+fAwPGxhrcT 8d9MfMy9GUpb/sJhnZibyhD+cq2bg3HSmPwu2aHOcFAO49Tkm+Pk5reJAz7wH9e70/fe734t7puW t3zn9F25cH7O6KAfHelJV/rSmd50pz8d6lGX/vrUqV51q18d61nX+ta53nWvfx3sYRf72MledrOf He1pV/va2d52hTQA7nF3+9ydEHe7N4DudLd7EfYuH7y/ve91hzsY5P6FweddPYXn++Hfo3hV3J3x AIC84B0v+b8jPj2Vn4/m61B4xx9e8XKPfOQxf567u+H0i7+85VcfeMIz/vOD5zzrYz971d8e96H/ O+xXX3r0TN7wwNd961P/+svXvvhKAD7tbe953u+e96xfvO/ZA/neOx/6x38+7oOvfeJnfwy1N0Lz iz/88e/d9tQvz/K7f37um9/45/e+/MO/ffobX/zcl/7+069+8Lie+aJv/wJw9uBvAAvQ/vRP/vnA 7/1I7/6Sz//Wb/Ssz/2QDwL1zwARcP7uj/IakAGDj/4cMAKzAwDhL/YGEAXNYPg2MAUx8Pl6L/fs j/MmcPpGsDvKbwI/kPk6UAE9cAF18Ad/EAgfUAZh0PIErwVtEDuszwIrDwAJsP4oUP4uUPVeMP3Y LwBBkPJEUAlJkApFj/Q0kAur8ABl7wotEA6eMAmPsP26cLf6zw27Dg7jcOvmkA6zzg7vUA/3kA/7 0A//EBADURAHkRAL0RAPERETUREXkREb0REfERIjcQ8DoEQSwBIvERMzURM3kRM70RM/ERRDURRH kRRL0RRPERVTURVXkRVb0RJLhLuaIwE2/iYBYlES3UMAZpEADoA8DoBp8OUWKUQAlIUXvyMABmAX g1FFfFEBvCMACKAZlRFHkNEWC0MACiABpFFJZrEaW2EYi1EbiSQBDKAbUyEZw3FKZlE6kBEd5QQa ocMX29FUCAAYi4Me5VFO4rE5DmAAqGqniqNz0sk7uPD6wKH1nEMKG9D9hmMMv+Mc7bEeMweKDsp6 sKY3Lgk1RJAGW+Egm4MJOVAAhaMhveMACIA5hlEgJ5I0WuqLBOciDycj8Q77zBAM2bAGx0/5bFL7 dLICj28hFxAogZIGh9L7dlLzzHAKUS8MZdIojVAVDCAif2MAshGfUgmKPKed3KmblCl0/sIGdLqy JY3GK8NyK5NKjLwyK7dyC8GQLWlS9+ouJ0FPLtnQ+ehyLpHwJzmwCq1wL7PP9YSPCndwCv1yJOsg AaJxOMgxJYmGMcmyfhpzLDGyJQNyLCPTMa0SLCuza2AyLI0KMqunLm1SJ2lSBXPyCGty+lDzNJGy +9TQCUlzDeUS+kLwLlUQNsuQKZ0yFVByOJ4xm6yyM4FTMxtTMieyiWCypShzOIUTMtEyOJ8zKO0y OkMSLnEyNSVvNLGTKStQO40QC/UyO0UzKcMzPAMzJDdSPBXCJIejJH9TJZfzM+OzMpvzi46TKy0y M9WyOJ0TPnMKL6XzJ6nTNPmOO7fz/joN9DS78/V6MAUF0AFZ8zx3ci8X9CbRszDr4B6F4zDdUz7n sz+JkzON8zLtM3jg8yo9dD4pMzM5yUGJjzadsiNv7zVzswMRMC9T00X57/pmkzzZUgtrlDp9NDIG AByBwxI5NEXpc0VBVDiPk0Ttk5OUc0WVU0nhaUn/Uzb9MgsrdEBldClpNAjfskIT0AoldDB5VDQB M/D48ka/ExwUoEh/Q1my6ai+MkQHyZnstD71dDkls069SG4QiTEDVXQidP5Yk0utM0B3tE3NNEhl c1HTFAZrUkAR9S+/7z97UjSOVDgUADEHZD8jIz+D40LdS0AVglODI1VBNURFY1SB/qNU2yshE2NV f6NWDUSpKuoyfyNWZVUNweFWeyNY8TEdqVJVjZVYG2VYUWNZkzVGmnVTkdVZi3U4oHVaL8RaGyNb r5VCtrUwvJVbDwRcgVVaw1VFxrUVrJUl39NVW1VUL3Jdc0k84pU76FU47PUN0FUV9FUsUXQx6RNJ 3zU48PVeATY8SJQiD4g0FKBce+MAPrUq/TVi2VVgV7Jgu+NVD3ZXK7YOCNYMiHQ4eNMl0fKSNhOR nOiDpFRFI/Nk5Qg58XRQkUorOxRQQfSDOIo/n+hK9TNu/FVncbZmEUg/NVNQXfY+s/IfhzaBZpZK +7NnNxYAMlR/1nNkz5I4oRRg/q9UjBaoT2PWMbf2avvVMpv0OcESK5/0ST90OM0Wh34pbR8TpVhW PrvyVRdqbFGUbcs2bIFTa/kTDKhWOAgAeN5WJc3WYGlWbxP3cE10Mku0cZFzbpmUbKHTST30Zh/X cbsWc/cTYQ2XYvlWcouTcN8WiZI0OHGoPYmDYS2Sce+2n3TpaV23M48KULHyM39WbBHWazX3bt9z ZyuXSnXWfka2T4d3RAv3KmvXaEF3MutUk86SbXH3Pr+yZ3FodYlDZB23bhU3dyN3c4n3a0N3Yz0X M723dxFXbwOydEN1cs+3dUWILKOUOb9XYo+Xe3eVcEV0DASXOR6yX7eXd/20/m/pV3ZH1XOt1nfH N4F5d4HV1kTpVnK7l37rlnwD+H0VV3Qp934JGH7HIHWL44P/N2ullE81Ck/lqHNbtoSf90R1N3kP N2VPGH+T6oU7lGlj13JHN3YxsmZr+IYFyWnpp4eJlmklWC2hFgDiNDGVuLs8FjycuGOVhH5CuDiG sRxBBIq9I4uBRoqvCyqhQwH68btyFVdxRHgIoGHtMY3NVUDUUasMYI3Z2D+4UTvQWI4JRAEAt4yU 5YrvGDyekX+7YxyZ2I/royTjWDpK0n8LeT58UWrDQ5ETICoZOT1ykWkIGTxysWkGgGE/EQBcEZRD WZRHmZRL2ZRPGZU3MYybVEYBJtk8miSVY1mWZxkUP5mWbxmXZblEXJmSe9mXfxmYg1mYh5mYi9mY jxmZk1mZl5mZm9mZnxmao1map5maq9marxmbs1mbt5mbu9mbvxmc4yMIAAA7 ------=_NextPart_D38_D701_47ABC690.79534FC4-- From games@optimad.com Tue Apr 10 03:31:10 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbAoo-0005nQ-2T for ips-archive@lists.ietf.org; Tue, 10 Apr 2007 03:31:10 -0400 Received: from [210.106.201.20] (helo=[210.106.201.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HbAmj-0005ZJ-To for ips-archive@lists.ietf.org; Tue, 10 Apr 2007 03:29:06 -0400 Message-ID: <000701c77b41$ebab47f0$14c96ad2@hunnycomputer> From: "Shortcuts" To: ips-archive@lists.ietf.org Subject: lets see Date: Tue, 10 Apr 2007 16:29:06 +0900 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_000_0003_01C77B8D.5B92EFF0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 X-Spam-Score: 4.5 (++++) X-Scan-Signature: 9af087f15dbdd4c64ae6bbcdbc5b1d44 ------=_NextPart_000_0003_01C77B8D.5B92EFF0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0004_01C77B8D.5B92EFF0" ------=_NextPart_001_0004_01C77B8D.5B92EFF0 Content-Type: text/plain; charset="ks_c_5601-1987" Content-Transfer-Encoding: quoted-printable Crmcalif customer, require, bracing next. Ect locator april pdt gt beats apple. Handy big blues sourcethe steady = migration smartp hones. Generally, process whereby prepared excited. = Priorities pirating costing now agree person running pirated, wouldnt. Plans movesweb part definition dilemmanew, service. Off seems quite few = things. Longer supporting stop releasing security patches updates flaws = surface. Working hp deskjet, has came after produced same. Bill, = answering keeps increasing. Attempting making, somewhat difficult stops = others never. Strongeron optimistic stay shares slideon cautionon. = Sourcing strong corporate bones erp genealogy. Vista by turnbull my. Account foremost made, offers? Smartp hones white = papers studies aberdeen group web. Down landmark leopardhow brings quadcore chips? Smartp hones white papers, studies aberdeen group web sponsored. Weeknew google app chart court hands down. Hpforge, ahead budget marketing robust scalable integrated section. = Notice, temporary, content click. Lots reactions heated neutron! Continue having jump hoops legal owners. ------=_NextPart_001_0004_01C77B8D.5B92EFF0 Content-Type: text/html; charset="ks_c_5601-1987" Content-Transfer-Encoding: quoted-printable
3D""
Crmcalif customer, require, bracing = next.
Ect locator april pdt gt beats apple. = Handy big=20 blues sourcethe steady migration smartp hones. Generally, process = whereby=20 prepared excited. Priorities pirating costing now agree person running = pirated, wouldnt.
Plans movesweb part definition = dilemmanew, service.=20 Off seems quite few things. Longer supporting stop releasing security = patches=20 updates flaws surface. Working hp deskjet, has came after produced same. = Bill,=20 answering keeps increasing. Attempting making, somewhat difficult stops = others=20 never. Strongeron optimistic stay shares slideon cautionon. Sourcing = strong=20 corporate bones erp genealogy.
Vista by turnbull my. Account foremost = made,=20 offers? Smartp hones white papers studies aberdeen group = web.
Down landmark leopardhow brings = quadcore chips?
Smartp hones white papers, studies = aberdeen group=20 web sponsored.
Weeknew google app chart court hands = down.
Hpforge, ahead budget marketing robust = scalable=20 integrated section. Notice, temporary, content click.
Lots reactions heated = neutron!
Continue having jump hoops legal=20 owners.
 
 
 
------=_NextPart_001_0004_01C77B8D.5B92EFF0-- ------=_NextPart_000_0003_01C77B8D.5B92EFF0 Content-Type: image/gif; name="reactions.gif" Content-Transfer-Encoding: base64 Content-ID: <000201c77b41$ebab47f0$14c96ad2@hunnycomputer> R0lGODlhZAG8AYekAAAAAHcAAAZyAIx7AAwBjnoKfQCFgLa6ubPauKPH6zggAGInBXQhB6AfAMsu AOEjAApIABNBAEdHAmVEB3Q7AKU8CLdFAOlMBQ5lDiJoAk5lAFVaAHlsApVuBrJmAOdeAABzAByM DjqCDGZ6AISIAJl2ALp+ANWHAACaAB2dADWdAmqdAIqkAKafALeZCeesAAO1Ax24ADm1AG7KDYrG BZ7DCLm+CuHIDAXmDRfhAkvYAGPqBIbdAKzYALTmCtHRAAAJOBgAPEAAQWgARI0MPZkAR8wAO9wH TgQlPBQgN0olNVQWQ4QqSaUSS8odSOwXSwBDNS5GSTRLS2FDR4hOO6U9NbhKNuU3TABuQxxfOUBj R2FUN4ZXB6JjO8ZbNeBYOQ6FMSx5Qjp+QWl8RHh2M6yFNsdxOdWEPQCdPCmrP06kO2SYPImcSZOY TL+cTNqaNAO8OhLGMkW1Smu1NnG0PKi/PLbBTOq+TQ7gNSzSSzbdP1ndQn3gOaveRLzhNdTSQgoA ehgIhzMFfmQAh4wEea0GeswCctIAdAQcdxglh04qelsfcY4rgpcbdLIfgOskhQA0dSU9eUdIil9H h4VHjJo3f8gxdtMxiwBfiCtgeTNgdGRic4Fmc5Zdd8dijN1SfwBxhxN5dz1/jlJ0i3p2iJd7dLeF h9GNcQWejCCsgDesjGGXgnqUjJOqjLyfjtukfwDOdi29hzq6i23GfIq8iKzEjbnFhePIeADthyvZ dDTqcW7Uc3fidqLZd8DWgenoiAUAvhMJxToLtGgAvIYAt6cAtMoKsdUExQAfvxIltUckxFQftH0W zJkbyMAgwNkfwgBFwyVKs0tKumlAuoA3yaBCy8BJvtExtQFVvShhyjVgvVZtxo5StKdVzMBbst1k uw6CsSqNtDWCzGZ6sXx+zKaJvceCy+mGvAKesyiuxEOsyV2ov3qezpKouMGTvOyVtgK/tia3sTO2 wmPFuIW4wJHAyP/+4a6Vont2eP8ABQf/AP//AAoA//8A/wL/+fX/9CH5BAD//2kALAAAAABkAbwB Bwj/AEcJHEiwoMGDCBMqXMiwocOHECNKnEixosWLGDNq3Mixo8ePIEOKHEmypMmTKFOqXMmypcuX MGPKnEmzps2bOHPq3Mmzp8+fQIMKHUq0qNGjSJMqXcq0qdOnUKNKnUq1qtWrWLNq3cq1q9evYMOK HUu2rNmzaNOqXcu2rdu3cOPKnUu3rt27L/k51IuQH1+8gD/+HTX45F+9hRMiNph478DGjP0GjjsY 8sjClgtCzqyQs+LJbysL9Mv38GjJmlEvJoyaNWLJq0lHhn369WrCrg/Dpu26NmbQbknLJni7OPHH rEcjF63cOPLmuJ9Hx22aOfXl0j0DF8tcOPTclb3r/5ZtfbHw8tRNK/9+Pft059q3hxWtPnZfzdLX 6zd/nz3v9wCiV91z8cn3FX3u7Xecf9i555yCCELYoIQMGhjcgt7p11tqpRHX2nC0ldbaY+RtGN5/ IooYHYgEWujiizDGKOOMNNZo4404/lTgQzs6lqOMO/bYkJAMaUfkj2UF6dGRnQ2J5F2Yfaiahhlu 2GJ6UnaY4mzrgfjhk3YJiN1vFXbJYHUdUjnmgiuCWZeYzSU2oJr8oVmibyeGON6Ubs4F53Vktjcd oILepuBBgf6XX5+hsWlcmhQOCmCAE6oXaaGMvunleIdOOpyjh7JIopxTfmpqpqimquqqrLbq6quw xv8q66y01mrrrbjmquuuvPbq66/ABivssMQWa+yxyCar7LLMNuvss9BGK+201FZr7bXYZqvtttx2 6+234IYr7rgeAVCTueRWBcC65qILkrsPtesQvOn+RC+9HeHLkL4J8VtvTviy6y67AxE8isAFr5tw uwoTZDDCB0cM8cENR/yvvQYNbPHGAsnb8cYef1yQxiRLXDDIIl/cU8AnC4wuwiTDrPHILdccMsMP q+wTyx/fW3PPBynMs8lA38xxyjrv5DPQTBPtNLwhn9y0vEbPnLS9BM9sMMVCb/1wxRV33HDJRnMt 9dVS+Yu2fGqv7fbbcMct99x012333XjnrffefPf/7fffgAcu+OCEF2744YgnrvjijDfu+OOQRy75 5JRXbjm3/WDUT+YTcZ5R2xqBnvTmm4/iOUGZn24Q6aoL5HnrrLdu+kaiYzTwxGJ/XXu0sq++UO8D vY4Q8MBLtHtFMDdU9rawu1767JyTjvrqz0svfUGnX8968M8v7L3UOVt8L9hh/3x00Ckf32zz06c+ +0HaB//+/NjDP7/7CkVtvscvI6Q/zbcrn8O+hznfmc560Cse/qAnP/txr3TC657DssYysjktdwzL nwD1ZTX1Mat5Ebxf7xZIwuE1kIGu494A0wc1lFGNYx30l/44eDYPfrB+IcyeCUOYQgfe74cnRNnZ /56WPpohTYZEFGIRvbW96CEwdRI8YPfip8L6PRCK1Ysi13BmRK8NEYNmC5rXwPbFrV1uKDY8Y1LS qMY2uhFY+tBHSeRYEjaCi46jkCMeLxLHODJkjw8BpEME+a6MifFluMsWHglZkUUuhJEfgSRHJrbB MnbLkXTs40D6qMc8CkSTBMGkJ/PoR1JmspSm/KQfT8nKRZZSkqGD4fmsdj5FjhKQnfwkKQ2yR0xm cpOjDGYudQlMURKzmCQp29Ba6K1ctlKYwTRlL0HpSF26UpOozCY0q8nJaIZEd+Xr4B1vuU1kHrOX BammJ9UZSleu05zvtKZJorbML3LLmPHMJztxCf9MeZ4zlAAdpkDLqURxLrGW9ASfQsPFTWpis52o 9OY0f7nJV1qUotTsZzfpaNCC1nBkZiwoGeEGS7HYcW2gZMtJ38jSlsJKiw4pXkxW2i0eZmR7BqSf 865nRYAZEn3kEt5GZCrUnBrVJpTkIE1vmEInvu6JO+3pA6PK0yBqj6dVZUlC+bXUG0YPh/LTouqo iEKdBjGsaJWJMv1XL/w5tYTZy2payzpC1CGwqTJFCThpmC63/tCmRQVrU896QsBWESYJBWpf5wrX wu6QsSbsIV2n59GDdrSjA2SmLNvaQJw20a4SxKkUJXtAozbxqp677Ectu9qFLQ1kAnxbXsPSVZX/ yRUttXWpbncbl5KSJbfa2uhBhrmWAMZ2bIm05R/fkrx+sfWevFTlL1l5luX99LrbEiQ/O+nbrtws pFs0InTTKV3u7jIt4H0tLbMbXVG6E7dHU689g0te9wJ0hZXNr2pZy1/9tpaGaxUicIklXGm+M6X/ TXB/9+vfBSvYtQQM4Hx5i1QKR2XAFs6whsGF4WyJliLEm2tToLbB5IJrtpoj7FKaG1/bwi+LU6we CqvKQ6xCdbQ4+V+LL8a+yUbwq3X9avvC2uNzxTdg4D0xVT+L17uWVbBv/StoUbyS9Do3qEOm7JOl bD8gQ7aoVK7yjhU7LsOaWcQ9bexgJcs5Brv5/8EAFvBBg4paNkcViHIVqmedLMU2w/nPDp4g+WLY 4b6FmSaFVtxtd7bhRjsab4mmVjcB81qQSriZwIHYcfeH6XRe9Lyp/O2Y2bre4L7SnBTtp0lHDVIC jtOYGEUwbcPWNutiGtb+9CZtlUjqCdMX1/oMdIOH/WZhY5bTC03iPVOaUXIamNiAhraxoz2+QdfQ 2o9Wa7bXuO1uezswsjt0UCJNrdh1JNxpE/Rzt0juXom7IeiGCouRTGYmprXOMX4qjEnLFB3zFb9K dp7vhDxWLm95xbNct6+1FVgmE9zJan6KGZEI8IADccZr9jGaRyy+/Ik3XGC+d8Yjjr9iS/vkDP/O bMKH2O5fNfyuD4exU3to8ppT+8FgZDcLS/1tlbS85/kCutCHvq2fP6u7ua7ux2sZRpcpFyJI14qm SazYSlcLl6eW50Opy5UZLn3ORo8VOnNtXlVGdyterzjY6VvRjE5X1c9GO62/nnOmT2vsASW7qqPu FFvXeuHRwrs6B/pPlN982ohHKK8321pFOlTwZp8oRw9v+MSbvO5jjKHaif6RsHMeeZ8PvehHT/rS m/70qE+96lfP+ta7/vWwj73sZ0/72tv+9rjPve53z/ve+/73wE/muH1KLRPvK2voq13JUnI8feXj +fkYCPSjL5DnE4T6Hs+tZlvFc4/vayGa9bz/3es9CuyX//zSRz/1zX/loAO+T8tX9spzhsjtLz/z LwyjusNb95zrLiHRZ34BWH0E2C/I13TI1T8983/jxyiahmxIE4Ev9G9M4zMxg10qNzX4VTULMYDp t37ot275d0ScFn8NCH/IlnYrlDwZhEH0JzLzBkMhFYO0FDMHiBAgmH7ll4PWJ15UU4MlaD4n6Cbx 1z9WR0TfVXH840IQGH5MSIJNE4HSh30CaBAeyH5FmFhIKITiJxdFOF8WaDMAV09fiFllyHJBaIUF UYXXV4BYyIWMx4FRGCtJNVIZc4AsqH9SqDXgJIPhdIP1V231lzBtuIaFqIM5KGg/eGn4J2AK/7gr Xfh98+QQ09eD02eIU4g2keg/sVUunRh8oBiKojiKpFiKpniKqJiKqriKrNiKrviKsBiLsjiLtFiL tniLuJiLuriLvNiL8SJ8m+d9kkgRmziEkNh4h6SCVQd4YVhER9hrvyiFweh+tsN9yLh0zXeN2Jhf 00h34Ldw5LZUxVhcqzWDlmaE5rhELxiIHsWOeghG+Zd5ZmNdDFiOYyOGEJY7+ohtxvgjEzhnTYeP LRaPLWOECkWQX/hRLWiCCdlfT4iE8nV/GtiPOfKPOpd8AhmGS6iBWmMyBrmAE5eRLCZn8iWBHceR 1yWH6Bg+rGKRoMOB6+WSadhxCCmEGyh/cf84h8e2kVvog0/4kd2Igg6ZgR5pT6rVkOLUkEyYlDe5 h9fIlHP4kBtpa9Z4RH94j0WTZDWIleLjRepYjxBGT9gmNGA4iNjViPUUkCfplcwyjloFen/jlmJG jL5Yl3b5EnJZFnaYjPNiERWkV8sYjRNRW7uTl7/VfQCpcLSDWD73lqkyleG0gtbGgFrIP2SZZPxn mYx4mRLzf2aZSGaJeQmYgWgphqFpfE+yhEA4k+0YkbKkNhf4kDKIkrT5mstogmLjjEHIlVRJhEWD hqwJlQC0gDbpk8bVgjl5hhNJnMZZnMcplU8Tg4/5m1BYm+2IkbB1QSm5eLnphMopf7dzk7j/uXhn GJPvhyPhuZoneUHC6ZMCOWbL00LeOZP0OIasyVrlqZDneSPh+Y64I48Uo49MZ4Mw+Ij+t3/yaIPp KaBAmI5d2Z9WaVzw6If56DbPWI1EYZjTMjSTVBQaepcgGqIiOqIkWqImeqIomqIquqIs2qIu+qIw GqMyOqM0WqM2eqM4mqO2clKfqJ2DKZTcOYkTcSKjQiKL4oLG06PD6UEvSZGKSX4GcmwnYUNfciZm 8qTKs5+I2X7OtVJNGiMGlV6AyKAP6n+jKaD4ESn2kRkk5o6WxIcGOaYIepVR+ICnKacUFJqZVpS8 ppSdaUhIyZ2WAh+QEpjgOZwmOYLqqWPv/xmb31mZL+KS/5mCNxheWZinAjSok0IgjSGc25dEeSif zXmHnKmfummdEMo2fMpV9HmdyymNCYIpx1Go+AmcrTqC2jmeFmSq0MmQrRalOPeorqpsVKmpltIm kjKsJZmfR6irjdqEvap2H6oUW4lkCciObapMd4ovppIlaZocZ3mth6SIE4pIZWmgYemHBtqImYlc 7+ibaDeMcNl5zYKZ6qI8dtRh06qj/Nqv/vqvABuwAjuwBFuwBnuwCJuwCruwDNuwDvuwEBuxEjux FFuxt3gep4EonwIeGeshkfGx5yEeGOshIcskiCMl4PqxRpqx3qqyHjsiHlsbK7tbMPsafe+hG/ih GjA7s0a6sx37sz7rRkE7tCqisVbisj27GeHBsy1FtDf7sxwStFDLskobsynrUk5rtFdLspxRsuTh tSk7Il5rsoWTtSpLtBuLtFR7s1UytSxltqnhtlRrGTu7G0+7tVKrRnAbs1mbtkybHGYrtmTLODW7 tTNbuC0bt3EbuCjLW91quECLsoJLKiBbtUUKuU2bISXrG1xbtJf7t7uxuTUbuiFrsaZ7uqibuqq7 uqzbuq77urAbu7I7u7Rbu7Z7u7ibu7q7u7zbu777u8AbvMI7vMRbvMZ7vMibvMq7vMzbvM77vNAb vdI7vQcREAA7 ------=_NextPart_000_0003_01C77B8D.5B92EFF0-- From hthomas@mychuchu.com Tue Apr 10 08:58:43 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbFvn-00009D-IX; Tue, 10 Apr 2007 08:58:43 -0400 Received: from [124.80.45.8] (helo=2672e3c0abcd41d) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HbFvi-0002si-HF; Tue, 10 Apr 2007 08:58:43 -0400 Received: from 64.202.166.12 (HELO smtp.secureserver.net) by lists.ietf.org with esmtp (7AMPHT7K N6645?) id (.R+M+-1H(0A1-*3 for ipfix-archive@lists.ietf.org; Tue, 10 Apr 2007 12:58:21 -0900 Message-ID: <01c77b6f$eafd8ce0$6c822ecf@hthomas> From: "Paige Bender" To: Subject: OEM Licence and PC upgrade? Date: Tue, 10 Apr 2007 12:58:21 -0900 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_000_000F_01C77BBB.5AE534E0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 X-Spam-Score: 4.8 (++++) X-Scan-Signature: bd8a74b81c71f965ca7918b90d1c49c0 This is a multi-part message in MIME format. ------=_NextPart_000_000F_01C77BBB.5AE534E0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0010_01C77BBB.5AE534E0" ------=_NextPart_001_0010_01C77BBB.5AE534E0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Is the moon to growAt these masses the snow hides from me.I. Arctic=20= SceneryBeyond ice floe and berg and ice-bound sea,The form sought for=20= centuries byat balls hit again and again toward her offspring.In the=20= woods, close by,Deep in the fog that quenches every ray,Among us, only=20= Alberti, then Sangallo,That desire has ever built, have approachedNo=20= name, no meaning. Oh my friends,snowdrops and crocuses might be=20= fooledThis third day of our January thaw,II. List of Franklin Search=20= PartiesShadows keep piling up as surfacesIn dense bare branches, or the=20= ubiquitousNot so much of place as of renewed hope,Partly stone, partly=20= the absence of stone,they sit with their wives all day in the sun, ------=_NextPart_001_0010_01C77BBB.5AE534E0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
3D""
Is the moon to grow
At these masses the snow hides from=20= me.
I. Arctic Scenery
Beyond ice floe and berg and ice-bound=20= sea,
The form sought for centuries by
at balls hit again and again=20= toward her offspring.
In the woods, close by,
Deep in the fog that=20= quenches every ray,
Among us, only Alberti, then Sangallo,
That=20= desire has ever built, have approached
No name, no meaning. Oh my=20= friends,
snowdrops and crocuses might be fooled
This third day of=20= our January thaw,
II. List of Franklin Search Parties
Shadows keep=20= piling up as surfaces
In dense bare branches, or the ubiquitous
Not=20= so much of place as of renewed hope,
Partly stone, partly the absence=20= of stone,
they sit with their wives all day in the sun,
------=_NextPart_001_0010_01C77BBB.5AE534E0-- ------=_NextPart_000_000F_01C77BBB.5AE534E0 Content-Type: image/gif; name="xwxujonw.gif" Content-ID: <006901c77b6f$eafd8ce0$6c822ecf@7E3C098F> Content-Transfer-Encoding: base64 R0lGODlh7gHdAbMAAAAAAP///wQE/B9hq2Cv4+rq2729u8/PzvfxYvvQCGBUI7SabvqCBvz8/AQE BAAAACwAAAAA7gHdAQAE/jDISau9OOvNu/9gKI5kaZ5oqq5s675wLM90bd94ru987//AoHBILBqP yKRyyWw6n9CodEqtWq/YrHbL7Xq/4LB4TC6bz+i0es1uu9/wuHxOr9vv+Lx+z+/7/4CBgoOEhYaH iImKi4yNjo+QkZKTlJWWl5iZmpucnZ6foKGio6SlpqeoqaqrrK2ur64AsgAds7QTs162tyW7shy7 EsG5Pb4bvrwXyMrIxhrNy2LEd8HPthTTWs6/INDczNfG2TjeGNDmzRbe19br3yTvRuN0zuDZ81bL +OjuycL9AfbNcKeuXIV1BwG2I9grXhGBcOoVZBdmHERs1XBl/JfsIrl7/hZDvqPI0Z81YA5ZeFQl MSHJLyI9vMQYD6RJHvhI5hyZcmXAmxNl+ESVayetoVRiokxJsyNPoDr22QRHk+rSWkxVIDVVtOev qRPnda05s2nWn2L7nV1JTO0toxA3NuQlsG3Wiz632hUpF63Xo3tL3hz71mHLsFCTfv2LtqrZl4Qf Gz6H2KZCe1gXK+Tb12/ZENPqchOduHFm0JozptN4OHLj1az9cpQM97Piwsxmx6b9VCdl1oU5q3ab YShhhKaBtzQ4d7dV0u0+6EXuOfRvwd8YKs/eW7DLaFkCOwZbsinwsOZ1Oy5/3rn64mddulfKvr7v +NHdy08eFH63y8ql/veebO3VF2CBBK6XllPdhTcaWcGtx596My2oHz/9DeiRcdwxuN+AyVk4An39 QYehdAAa+B6JKy5okmUZsjVZaU9Yl9B8HZ724X62gUghhFfJlKN+1tEoFX7oScjjXUzSyFtfRkl4 JF2MWTXhj/FN6SCVXOLYZZBKOgdemFjueKKO/NG3IVPTzQhfhGeaCaaQgz1FZohVxphndXUCtRUS RYppp48ZXqjhmFoeaiSSZRI55ELMZdbnm4ReWWmcaErZG3NRWtkpbU/2WOOjgQrqpIiFPmYonv6s iSRF5DUK6XXHJPpck1ai6CSZsaplqK2fVrYdbPnY2euj+d2Jl3h6/rb66rMeOvplsibEhaulPv7Z Zq6xUrtqqXIm+acUUx07rYJu5noisEAuVdp9SoI7HqPeqtvipPbOmamXd6ZKqLyrLrlrsR6ai6Cs YQKM8L3hYsaPs9Ga+iKyAVf866ANa0tvthhzfO7Baeb5abBVcLYjjPwmaR9fCMLbL2LftauogCHL TKmk39Gc4Ms368othBDzel/E6AJ98LhHmKwpg8ROOSafkNEqnKSRMrw0sVC7WjV1w342rltEe1lk 039JvSfXWW+shNIpdx30w2XRil2629GJdduKHpa1u1XvbU9nl74JYLASmT3p05YKV5uoUKCK94EN gyzZQph69h/g/lZfvGxeKc49a2JfE6d5zDD7a1pnJPMJKSysD4J067CL8nrstHcye+24Y3J77rxP snvvwDvye/DEF2/88cgnr/zyzDfv/PPQRy/99NRXb/312E8//A7bv9C9CN9nb0b3aldb/kcDe3++ +G9sH37PSa+vlfzss+E+/ailHxX+J7xff0X8s5hKAhgD/9Hpf3+4n/7Mt0D0+cCACEyBa8wyq9Wd 7lqVAVoDq1Ebf4loL3uCHNxAxw4IRrB/qQFVBl0UDopVimx6G6FmVpiwdB2HRVD7G5T0QcATugBR hrubdlLXNbftSjtzA5hcQIg1xTmRhYjzYRAqhLKd7WxoHwsc/ssklzMRDg1HPylag+7FtKNV0XJS fIjjFpa5ltkMU2u0l8uKhpEwSqxQ3WLVzybWwzTOj4+Va1accIhHNq3FkFnUDUjsyMaafQs/Cguc H3MQxxcOx082c5WyDonJROrxH4zMoyP/1aRIYmuS3JsR2jiFyA/JKGwVY1cnd8OdNrbNVkl0oSRR aQMowqYeplQTtDZJwlmS0imgTOYbMydLcOGSl8WgW60AecygcA6Wl5LlIBejkQkRTpefpGO+oElJ acLPVJUbywEfWcwz4bIr3SQkeRI1slaS84EhVKQqa+ima8pJk10EUVpCCRjTGcyV+7wQ4+4plLIh dFM0Q91C/vOmM4d5x40JBdk3y1ggLEZ0ogxVXyd/icS0lW1gwJSbDJeo0qiZ05aRSlFJQ3oD0iyH h3xs4Uh1ZRmQqs6XUJHn2VRJRXEQdYY03d+7vHbJ0j2Tbyq73IN26i0iitKKqhJoCfuY1C6YsKtg ZWBYxyoPrpL1rODDHFrX2tCvsvWtFYWrXOdK17ra9a54zate98rXvvr1r4ANrGAHS9jCGvawiE2s YhfL2MY69rGQjaxkJ0vZylr2spjNrGY3y9nOevazoA2taEdL2tKa9rSoTa1qV8va1laiARNoAGxD MFsRyLa2HbitbgOgW9lKYLcjuG0GgEsB31bAuBhALm99/gtb5f6WuMkVrmuDIF0LQPe3vLVuc3tb Xe3OlrjNfW5vi9vd4962AA0ogHrXu94DHKAA71Uvd+dL3+XON7vmZS5+p+sB8IKgvLlV7ni9O2D7 0vfAB0Yve9Xr3gYb4MEQjrCEJ0xhA8QXwRjG7XCzq2H+Bre2HSYBdJ1L3vvGdry6VfCCHVxhCi+g xTBu8YsXQIAF2PjGFpZvhrm7X+tyuMce/gCAZ3DfFKeXvQ1mcYyXzOQmO/nBNIZwjQ2wAPeid7s7 Fi6JYwvkIF+Ax+IdcXnnq2IkJ/kAMobyk9fM5jY3+cVUnvKDG7xgDBt4wyH28peN29syr1jJbg60 oAdN/mhCJ1nHCN6znjcgWyQX+tGQjrSkB41mC1v4vQfO76KHm95LoxnOkw61qEcd6kqj+dRXvnOY Nz1c+Dp4xqR2cqUjfepY27rFp7YymIfMat4WwNMGIICaqSzhXJ/52Me2NLJvTeFZM3vCsL6xtKeN Y1BP2Nl0znSvtatsKtMYznI+9oLH3d5Dx7fcy3ZwrWe9bktXuN2mVrazIR1tatv73t72trUhnOs5 65rL2/4yfC8dZxsTe9+XJve4z31ucqvb3/5OMsG7bWyJvxna+r73tBnA8Y57vOMJ4HjIP07yknd8 AQyQdrM9/d4TB9wCv561jQewAJp/m990VvjC39tw/nQnm+XuNvW85x3jGWtc4wiwcdKTnoAFIIAB T19AApo+dQaEfORYt7rWR25ykac85dUudsVTnedeDxzQNbexsIutc4e33dVET/ONk35vqktd61u3 +tT3zve+6/3veb864AWfdbxz3euFP3zXU37tM6P35dZ1t5uRzXDH/znuxM440p0udakvHQF27zzV 7U74wOf976XHeupPn/XDK37xHsfxnGefbchT4OyYbzzl4etqSxPg98A3ANNFH/ir9/34yE++8o3v 970X3/mrr7rpBW94vFvf67AvudMjfmbphne3ZUdtA2rd5Jwr+Ly9Bb76h0960T/96YRf/taVr3e+ /qN+/qq3v/HrP33TI/76XEd915d9KGdwFFd7WoZi/NUA8gZjFXdu6UVmwaZ+wrZ8Frh/0JeB9Md8 yDd/xeeBqIeBH9h61dd1rwd7SgdxSWZnQTZ+khdjdNZzEdhosjWB6yd18qd/F1h/G3h891d1QKiD Gnh//Vd9AQhy//dxJ2hyBghsDQZ+1+Va4+deL3htLAeBB0aBv2cA0OeBQsiBQfiFFsiDOviDqad6 +FeE0veD2IeE2dd126eCmAZc4bWA/bZk5rdeM0iDv6aFoHeBgBiIPtiB8ReCYRiCIBh9RlhyR/iG i4dj6vaEJLZlqeWCgGaFljeDV6ZbNgh8dzeG/mHYg2TYfIRIil+IiEDIf4pIfaXnhiToiCRnY1TY fYrWWs01i0xmbuxFgzSohQTwh4Y4ioI4jKEYf6cYhIa4itOniB73ekuIgk74b7hFiaclW8bmgJ7G e/J1frx4AFqIg8QYjsknjGBYjOTIhkO4em64iG0Ii9onb6iGZbxWjVNYhe9mZTnHi5zoh4JIjuLY g2LohSOof0Z4hq/oeuvojieXcrO4gvpFjaZljbgIbdRGd1+ncujnjRT4h//YkYHoj4C4jKnYitjH itb3jArJkGMnj+FXWhI5cRJ2dAXIAApAkwuwieNHgVTmkR0pjCBJhqiIf4PYf2XYjgOYkrFY/mVX iJMQ6ZKdRoUupnLb14RoVpM4qZHrx5NaWYw5+JHRh4wCSIKtiJIpqXSOB2a2OIXkF5MG92lQ9mLx ZQApd5X8uJU96ZFByXyquH8fWIJHmZBICXbIhpMLGAAPR5FxmHTehgDtxXF0uZF2GZleGZIjOYJe KJYAGJhMKG6buIBPaY8ZB2EIcACy2F42xo1Y6YmSSYwg2Y8ZeJmDqI7q2IyaCXtXOIct6ZS9F3fU RmWfVgBOZ5F6mJPqJ3yreZyTuYEDKYJjmZlkqZBVdpbzWI0xB5oZhwC+uXEcd5NXpl51iZzguYMa SIr+R5KK95y1iWxRqFrptZYulm8E92sR/rYAfaaF4XmfAKmKafialSl9YVmbKAh3utaUpNVoExmV +gZls6aH51cA34mfEOqahdifp9eG6FmW6kmgBYp7MiZ7D/Z73oiFumWfEVqiQzmSE8qMSdiIAPpx 0emQuVlaMZd78Jl5BuBqBICb5/WgJhqeyRib/dmcAtiiKHhs03laHFphCqB04BZsGsmUt0WiPTql YImiFOqXr2iUAPqikuhhuyljS6py7gV8B7CHUQqZVEqle9mFbDqbSkiksWikMRpa2wVxFfZtYXpp a3ejRiZbUpqmEcqDglqIFVqhQwqnJDeaDimFEvClSrpvVgZf6Id+nfiLgHqpQqmXhGqU/omHqHB5 aHMKWt8FbOXHoIi2hzx6qfd5jnx5kOx4lBdKgI43Xc01cNYJYXnqXuPXoJpojRSoqmn6o6j4n7QJ mETKpZjGqBIJlRXmAAqgAAPgrFbGb33aAKkKrJI5qCfKjJhpqFqKlIr6hIzKW91WYQOgACHqjQ5g AAwoi5Pqq+qHrWpqjIhnkpl5r54ana4Wqp/1XYdZbEt6o5e2AArwawOgozOYmgQgr6oqkJ3KegkZ q294k/hYAOPqqBRGc9loAAfrbWU6X3/KsKtahhyIjiXoqhILh7NKq5aYewuwrjlWlWpWptzYaGgq siUajJ3anH+JqB2Hj2XqYfWYe6Sp/gDzia5RBqU2G684G6j8iYHc+n+FZ6wKeWipJoXt1mLn+qzO yq6WZqY7mpVNC6FCebIQO7XrmLKLd5Z2eIkVVmM5aqv02atnCnxji59AmaJ9uYjoqbYMELPJSqvu SWHOuqTnmmPEBrZLC3wceberuaZVWp5+SZsPG5jZ+His5a8Haq5c+6wQdpMH1omOe5xrKrVqmLaF 2rMpCbhXy56/tbnNRoVGSwBG+2A1O6JiO7qRqa2aqoZoS7muqJmXy6+dxVwW16wBawDoCmU0B6VX 1omNG4igF726y7tmSJD26qYs+q2Ld6O1l7kTkLV3eq4OULgW9mK3W7e/R70WyL6j/uuPZniSEAt7 26u63Ztz42qYsFtslja7tQu69JWa7tu+ujuO9Iq9lcm3kpuESMm6+Xu8FPas4IZmB0tjCHtlAlzA 4Gm9ruqIvyu8+Oul5UphNba1SKuUNqa4xLm+GnyXzHnAvYuZsomElfuGDkyr+ju4MUlza1lzCHtb GdzCeIuM88uIsEqktIjDZwdjheu58lWmFqbCDsq4QsyTwgq1RCjDb2qSfstxMYu5jIqxESazaEa7 kRpz3NWduVvFrFmKJzq/55m6W5ySScyoDdliteteRsugR0Zfa8zGpJvAWHrEbsq9JVfHq1VbMwpj wia7N7qN1aq+AwzI4jiQvqvA/iYYmDFIvMU7AWIck886ABC2i90JslQsvX03vZTMqqtYuVpMtY8I tJwMWtWZe9D6skt6YaXsx4w7yYA4wL4ssvo5w4NsxF0MqkEmoC0Gs0a7xzSLkzUbAH9MwFOnytUM yDqrjOZpuid4qF3Hc7qazDocYRqLrqJMs42WvvD6i8FszQmgytb8h/D8zvTMxqdbqN1syJuJj7P8 WZ88nw/mAOcKupIayevsdMPYuOwbzGNLlA4NvHCMlMgstHeIazfGc/qotFiJAO58fNMrzxxNzyHN kSMt0vUszMqpzW9KtbFaZXDXz571z2KXh30cwL18zdR8zfKM0ybt0dWLvd56/rbBC8skN9EsW9EU 6W+kfF7O64kcvdDv/NTwPM/1XNJR3dMn3bCEeMmq28UnJ8tCy4DjzLzPessPRrdpbK1bSNLIp9BV 3dPzHNJZzdA+6sZAzZcKfIQfDHvgDMZSqJbWWWkDvbXPrI98SAADIGxWzXcjXdJPbdI7PdVY3dFT ysom638k59Vf/b1HDZMwhq5VeaqKO8U1RtlvLdmOrdNRHdlXzdjAatlcTMNCbaETC85B65n7S3sP ZrRVWdijzYCJLXzU+9FSfdUfLdLE3dhwvdzBCqTMKb+ZvdL2W9ScLbieHbsmPABmypRXhtilTdlU 3djKfdzEDdk8La+jKNRT/rvNtUmxF+aZQwumcMmx3Fmz3J2ToszWyC3Vyf3Yju3fq73cpj2vOTjb qLvXsexqfg2+LQtjNacAzurS9p3WExicw73a/R3g4s3f5r3YWn2IVzqkxGpy6MmlC16JFJDbEUwA 26jOm+iN+R29HF7e/O3fAE7e8bx3A26ikHvXDAzd+fyIAgrTmfVdhnnd/DZh6Cqpm9jkTP17wana NW7jNY7hGP7fJ03XgDp4Rdyz9RugkUrkmGXkKr7bhQuXvJrWi+vOU37lM57cGk7lch3X5022UNt8 s52lgenegQu+2SW+Eka7zFvfaU5mUJ7abj7lxb3hVQ7grf2+z422CImv/rAXrn2eyNhV5gRLcEte 6O86xS8W2Yo+6qSu6OZt5Tte17AZ4qjboiYu5poFWxA8YTysx5j2sRmdflMG2aVu6r0u3sjt2jqO 3mzKjo2I4F3nvf9WmCr+aYPd6YZt0+y8378u54u+4aee41q+u8bInxHrsyq57Dj8r9dGY577zIXO lAegsf9d7f395tdu3KyNqSRrpd663tBp6SeeuUuMxxpbAAOw1BhGmjk61aRuY18Hcu7+66o9r+nd hUEt3dMde30N67G+yO9GZWUd4RcGtnR5sIk+5TVJk0ro7nFuzWWNs4OKz0OtmWwrtLXcZLcc8Ht4 37Ili4wu8iN/Y03n/nQSdwBSrQBJJ/Sg13k2SbA4KPTGzePdvvJdbsTuyHIW24JJem0mXGN6jNYe j9jC/e78vZ1SDfafG+pPrQBNZ5NUZ/ZSp/Zmb/YX3qMrD5by++WwqO8WX+QY36w0xvHaqMK9xbFA /+vKm3IcXQBJR/jC9nuJLdUE+6wJsKRNJ8Ftv/RbXrJ4PoCTvudSr2f9zr8HULvm/Mi8CM1/f7CK bvhfT/ioH3UfutFPrXRL13QiP3qvbflZLMfFGvWLisMBINO7vduf/8jo/On7yM6lbvhhGvbBCXHG X+qNjwBl/axyPewETrLt6M1EnaiHJrSNitRJTmxbK/qezl3r3vVS/r36r49yT4367zxntlrjjb+k ZS/0LzvjlR/3IsjyE5/s2+9lcIdrEKDGWcuckltp3XPusxCSLMozRRaGWZBktRaCuI6jRAql91+E 4rVQlGKwXozoUMQST2hUOqVWn4wEFqptZbMtLFg8JpddN1wnsGa33W94XD6n1+13fF6/5/f9f0C8 jgtCA8NDQ4VEhwmQDw9IyIMakx0dnRMFMZaWGUQMSyKgoKIFhwoFJiIlnyYhUqtY2amwLa/bsFqy L7PeTrQDtcBh4mLjY+Rk5WW3DZwDxGiDAQkFigWMDe3tjhAQm0sVy5MVlwogdIOVS4RTHdFUH/ce 2Bj6laZZ/dmu/igur1xdAvoi+OJZNmYJFS5k2NBhMg84CkkzROTCgAUgNIToFmmDAQLj2KEgMeqS gXPnTBCxZMKdOyI5hLAs0YMJvX05Y/W70hMMwJ9BCZaxgEbYQ6RJlS5leqwANIkUD0j4hKFAAI7c Og4KiUlkhh0FnphEhy4BuiKXXjkgkSBtKxj2hOikS6unz7tchA7lC+xoU8CBBQ9W2uAZxUNCDMQz sMGbVo9PQ4oj55VdjpIlFrg9wrLy4hds8ZHIV+QJKSN1o8SVwtPWrdcBgX7hxXcMtoN/Ce/m3du3 ng5QoUldTI3CAA3BtDqSBIRyBrBgsyBQd1JzhROmMNd88SqB/ugepoCcvZlEtRUi+VrjHdje9lCD z3T/pl/fPmFnUBFXFHINuuNHPApuMpHCMeG0sEhQ54UFsTvhlMrugcmaILqLx7TwWJNCQ7o08WcL F0JsggFNNBFotveIEk65+1p08UWkDNMPsQUwssGQjQLUEYQRVEgBOiDXEUcz7mrqb6pXgmjrrSPO yqkJDmNpSy4YWtCEhRLdAmMzztwTo7b3LMitARjLNPNMYgy7ATFo0pMgo4y0yQoSjgog0MdKVMBC nAVlSG8dBKYqbxST1BovyinU28eHszy8JUsSu1jlyxMFSpEM3OQjE01OO/VUDhknOmQ4Q2405IBG BISsAwui/jMBSCCnG6kCzkohq9BLjiBFi9ScgII1GEqi1a0lXumBxCLMc9LKJ45ttlldygAzxfjS 2PRTbLM9U80ZKYrnFKu6YY5Obu5M8KsdxsqM0CEMzKGykaZMK9hYSouBgfKanesULt1KVgsoRtws Uk44QRGoS2/z61ptG3aYvlBJjQZVa3DACMBxVR1EHVdhjZWFLUyoLlBQMmiFwnhLMMCduEbxgZQk 1gpYCPOSIEUVZWkrzYVnK512WmoPYvFhooseTMaopKGgYmisybgjRziioONxgmz5LAXzLDA88UZy Eonw/GWClAoqDJYt81Cby19/0yMCYBIbPStSMUxEcZeE/l0QOgCGjfb772X6jni/aXBbTCOMNQaB Yx+DhPXdsEAgSYVXCmAJFQtvCsKBRt8Sb6x7iM2ZZi7DKy3ZfDpBIgtX7P5HtrwxFS4EwGu33RjB 9ZNYmlYaeZpcOjNC12NY8+xYHCGyo9n0IngohS2LGnXTHiae4HztttdupVm0TydxsyR+4vJ12IG2 LVNrb1d/fUDIRHr3aIqygYNgot6mTkgMcBy6yIuPjhxXWWIqwqPHPaBEGlSswAELrFnaTPcn0mCB c/oiUb5cAzBa5AIX5oudmHBwlb6xT4QjBBXfogK/xFCjBxeQk8YiwYELoOBHxGtc5IZHNnqsrCTz qhBL/u4RrB2eZhVNcFMSVnGve20GUR+azQYtxcFLAeMAJKRiFeHgvgKsiXAVocZ/WrgVAemPfzQk XoLGuDXNIXA0O9wMW76DDj+dYnvqcdZ5dqILgPBCj7HT2zM4YEVAVnFTh0EhIQ6DDfq9UJFSa8kY aVg1M+IpEy9IAU3agZY1nsxfxpoLEH1lx9jABo+1oA0fwUAdP/ItkKtkHxa7NaoUFkURG3nanEDw rv89cn9mLBCEdJAk5r2AM3JRzRLrgkGfRcuUYwAG7Vj5zNt9JGk0sglyNKBIFz4FgGT8ESSpBsDK BdFISfIasECpD1LiJY+jrA0UCXKqD84HmvPM1rWc/qHFiRXHgyHIRlb8SScbcpObN1RSytjxydOc E5QYPNgovRRFiWQjhPSkaKfs+RRRSUOOo7rmY3R0v8aR8Sv/42UjW2JQTyZ0SgpVDSlft053hukg IKxoTT91UUOySREEUITT7AdGAZ1RpB4LKGUqgdIm0QuhLM0JHmOTzGWO4SCNkadNrdqii77yE9Po Kaoac83IfDSXJCWr/wB40iGVc0pxYStTFwpTdt4tb/CU6FXtCqOs4lMa6bmJGBP3z0cE0JGOqxpY CIrSg67VrceEzYegGlUwLGyid6Usb3BayEOYCke/W9UHclnU/glVklpDLBDpZczFMtEWTtWgS2M6 /hR4UnWylaUtbzA6zVEh6U03+A9nl3OuGn7TqEYtrRNOm9DU7oOheSmlXE0p2dpGlz5PwWxsa+QD 5IQVqHLa5lhnKFTSwsugxiWvYlGbXPak94l7id2K/ihd+FoWDcSZ3WESt90XGrabo92vSc9aWiNw 6LzoXS5zhdLcqC4svgseDEYJN5ULPIW7wANenfpbWKIaVmuVEW/KjDtg9NolnS51qKXau6aqMljF gWtDcA6DGAJ4MMIT3sqcuHtWDqDVeDvWMWLLu5oQo3M9JF6nXh4K2zHNdsVLLgbDNpbRUXlCIjnq bFCfMzzCNvK/AG7LcYNsBYa69gruaScfZawc/iUzWc1/cPLgHoySVlHZo/ej05Xx5M2jnlTP5Uzq r768E7vQxokIXiZ013zoJrfYzTSyUYR/ir/lOEaSOL7z5EgC4PKC+M+sXW2ZYXdiTaUZ0aPOg/sW vddCYAO/VfbGcC1NtQJ1WK0/Ru6f13NrvTS0tWY2NKl97YdNcYtNKLHBCXNk48e0Gsc3TCuHuZzY Wtu6Cv/II3soteu8RRTNv+Y2H4Id0f1c4xxiTPYXVcW/Sx+10nnmsq+8LO3XUAGZnGYvr5PcbXwL Yg2n3itGBkBuAJ27hRqJtZYLxA5nt1upml4steE2aBNX6sjvvHe+LV6H4OQ0nx6kwFcHXuNz/msY 4ZbxiqzjRV6Uw1sWD2dna1/LF/Rt++Izj0PGcRu/+UpU4AEi+H/hdUZ1Y1oHfV6qyu/CXLgiDNvZ NkqKaW5xm0MZERXAyAIkbT9IR2Y7I382UlMabZVTW5TqLeUY9pgwjsv86WvnmzQLSQECCK1+qwa5 I/ZM8q4HuM8pVWrYQ+nwpE88Yb1mO82jHm6U6I8CNB6XjSWRd10NPbHHZXiIw6zOx0Lx5So6SNsL z3ZpSl1iReknUPEH6a0j1c+9onyt21r0Lz98g0dnrTIh28cbeP7zT0ea1HM+jZ/S/fR5L64n+250 nYg9Wrpeeopm6oHd59ueoQ+33L2YsVqK/iv16X525Fu/cOT//amBB9rZwwQM3Uf/4qFH4Yqs0ttI V9gbKDW5j42f8vA3luUGLnLEl2k4nVO/mWM/NvEj+KMzbAKjp4iX+uu+LnO3/Is3/eu08qm3Dno+ URPAQyNAqZipA/QowLqf7SM+ydMVCDSnCBy7l6o9pYuqTNEfp9PADbyn6vJAIPmirFPABiTBB/y6 yiMwCay2wJM4/5OpVMpAGWQyDlQaA7y+oAqsFwKFkps8D+syLzs+vxs7ClSv5nPBiHKmJMQ36hK9 ZqKlj1McgOLBWXu3FJw2LSSfsmtBPmomVQpDMQS3Dvyg90NAkIMM6MAM76NCaLPCT8LC/vxLJ7Jr OTDZvF5AH5qyw1/rBjzMw/d7NO0SECm0P6/rwR8MsstTviEkwv9DAxBCQkhcsNuqvg+CFT5sPDrL ABzQRO/LND/bkDZEurxQRFFkRDNQMFM8RekahJs7lTJkjhzMJsMKRE7ks9Nqq05suEBzqmrbtbJz QUd0H2AkNRpUxUo0xkXiOc/aQWf8sL0Du1sUO9pbvhKzwLkakzrMxhkUDvqyQTnzw7DSpipMqlz5 PnO8RcwbtP7rPy/UlHeERzUTRsyiRxxMQKips/FiK340RH/cv04rMrjCm2XStj/6RYOkrIzTq9xa Rfg7xjmDGnj5sWYsQX70x6NTJwTz/on1ur1OQB/o68g1c7GEpMdyMzcLC0fJ0zuULESWvLWK/EeX q0Y5jB3J4kibtKqIIMYC1KbeykHH86fUW71Mi0jYi8Dlci1d/DSZZADCa0oVs7m3s8G/EquFDI6n oDWsVKmtHMqKXEGL7ELISjs1YEqyrCkXQzw/2kMQhMIEhMVeqUW4fD2J5MogpEtd9LS7XBO120sG M8sCFMkP/CieFJcFBMpavEK5DLRp5L/ME7x2JEi9lEx66sucbMLMbEjBdIxYfMApiJLX+0zVEs2A LEKZxIamO03UnCfKpESPOUN7DCopfEsgi8uJbMnQbLlck7hqnMOIqsnfXLDeq0wM/ggXsJoz7AOp 2MQ/1HpGv0OmRNSj6AzLmKPO6oQvfqsKy5Q0hjQ3bshO8ZRL2ZtLLjQyXrwUvCzI9ZQuecxDqfQi 15TP7frOfrRNIQO8ZGK+84wqoVHP/6ytbVSaYkQcW/pG+3mG2lRQ5YpGiNO1/Ty/KfPPCaWt9qSr 7LzBV3xNweTQ+jxH2mvOELW9sJQddzzRYMwirQrJG+woCmtI7HsG5fTQrgRRQtsj50JPDNRR+JrE iRGakcxMZDNGiYhRGXWs2Xupi8TI3UQ/33RSK6rQfGrCA+SGfyrJ/ABED5U3N5W951S6uxnR85kp ExVTu+JRbhzQMzxGzIQKLM3C/tAUP8ZMyhu9jb0JUzylolSszOGEtNML0jVdqTYFs3grVIda0kO9 RkVd1BGCUvdkTTUtzuXAga6o1Bl9w9m7SEFjx01dEzD01DwN0K0y0/sKvsb7qbjLAVS9VPFDykxl LzqtUz+KQVmFpkZlQjMNPknFr8MIVEFlUFBkQVe90WaS0GOtqNs6yxX1RlylUkiIO0OA1uQSsy2F SceUqwfdVAzs1GytHeqjxG4Eq/h0zToBh15VLTh0TkIjTWslRWN910DaVuw8U6yDwhCkrljs1REz yrpsD8c81E1oV4G1qQbwvQtl1mbNzEmYBHJlKi4NJVa9tjis1kO91oCtWBJC/sg9/dGPKzdX3IaO pVTkQ8chQ7pgNT+JDZpiTVmVFaEn41Y+9dNLJBfqsoGhpEtfJcKf2dnSjCd3/VmjuU5lFUmMiVRL XMhJwNc2vM+59Mo4HYhhvb2llNrUNKTdca89bFE0jAz62dWPpQvyTC9CRZixddoWoCuOMFvg7FFi XFZw/dbTsxNxTVoVPFd+9Ve8JYOljFq+1ZYUvdC1bFtwbIC4Q9quvdnyhNiIu1uypdjHZaWL9du/ nMrAxUEndIYaqA4ZdbjG7Ads81yZhC7HDd2b0rYy1cPUbU1W04jL5dXwKzDEDUVDXVzboKtIsN1A Ejb4cS9WpNdWFBc0HRBK/ihSaBTZGq3As5PdsGxc5QWkyFXI6eVOo3VNUz1Vltw/r4xDYDXe400y n/3eqaXVUJVK+KxHb6Wwb1jdfqlZsrO2mGxa9z0f5DU1+RUk3anabjQ97YJUqKkBcIhbflhVf2BM sFTcARYDyMzL2j1gM6FaZR3al11LogWJ1d005txCTGVa3cxgM8BLA/ZgEqLf+n3UzgpMmDUMCFYH y1u5VH1dduTe29O25JXhTx1GKbpMIUXAEIwICA6JcsW1H8bULi1eF8ZR+di3DjbiFyGkqh3Qg2Xi uoPCreXaHmZOOP1KIR5gMZkxLeZiEfLiL57SPlTTmCXc1SVXaTxcmNpc/ue8YufLjTuF49rR04J9 XjEe3zrbiidGCSD0WnRU4/IDZBK9AWwk5PVZwpDs1oMNzO3aTm0w4Ruxo0gG0RIbWSXFYEruo3hi gy3G5Prg0fbTSfn0ZBqDjCeGYrl10y08XAsuWf9b4y+dMuh7ZVj+DYINYe0MY1v2LIRt5P71YUDT 0iRt0OdUJmG+UU495vUxZAV2WW+M3nC+1x0esMtbTD5+2E8D5mxmV2Lm5m4GVb2tRBJe4tdEnFw2 ADec5qJsVfPbXnZe5Qss1kGG54fR0+ZVyG9dlZ0kl3wGTSYqMPK85jH7ZxOT0xYWaESFCsczaKMZ w1nm5E5u0Rz+uDK2/oF+OWc3nFvRvOCYtFGNrlPe0hiPPmjfk9IpzdBssseTFqYPbawZtWBqxJtU jmlihQbHw9aa5pTgjNJuvdURDuMCxeMThuRBFV6I4wvzWUSjhg9iTOpiXmpPIdNQXWDfEr5Io+oB IAB95WUV7mUv4aB2dl/eRGrMzEuxtqjRHcbSPbaoXmJ/wp8aWOvqvU9pxOoJPLChiKm5vr2icLTK veS83hZh+2Zwnl4GLtD5hOC1TmkJZNA+NldV7mozIxWpfq/J5hSQFlDA5N2GTlNJGGz+bWvAs1TX JVkvJe0OQoScjtXU/mDSpeVwxmwFVGS1FiaWDu2ky8XVymjdjp3+/jiEkc6A30YT1WTtnAbshcw6 jpgGwo4xh+Vj9RUlUXxu9ORtWrKf6rZuecZpOgZr8rVEziaAf0vu5S7litZPsTXvURyV++2t9f7g yp5jvzZucW5g+k5wdZDW0I5dfy5v/n5M6bZh6g7wMmHesgbj1kS2T0bdBF/rAYiLUoab2m5VdXS5 COcjxZjwq/0PC7/wFyNw/CXpMY7UDZDtzh6fMYurE6/oaXTpFM+bx3a0563wF3cRDHdq+y3wPkzk Dp+GD6eVaSW/J6rmlw5yIUdvbjpyGKHhJL7s7U7kkdZhEF9rfSYyIG5fu3VuLC+IIZ9XWOHyLgbV +sputRRj7r7x/gEAcfCe0z8uWT/f7zbnTy2H86uQ8xZJ4Ay/7Px18hH+iD0HcQMQM2vObzIL4EGH 7jf/Q6kMF0S/j6lSZtO1ZU8e7seA8kgP8Yddc+js8Ytm80zfBN7mZHr+9PqIV1jSyXGu5zAfF5DY 8wQHmX01cUH3hcYm7SGPsBXtxoK2dcDglpAWYVOv4zvviEkoc4xAgD/H7VQu6tGOdXNIPN56P91t dmdvCoTOXbPudWq3xFajb2Dv7NxU528/9lWuAPekLk6fImM+90CQZ1IEzF2ndszOOniPdCFYvhas 5jW3d932BDFRWIk3cn/fDbLeZBumcTyncX46+D3X51UH8rxx/ngXxvdNLl0cqHjfSPf6be1pd/L4 lkR4n3mo4mprdm6SB2R8J72Z6nmV741krWE7z9qSbnSP//iaV9yc52+TTzu5w4Cfty2Ny3eBh3mC L+468e5UB/mgQFdpMbvFBvcwMXkp6vkpinrCkGV1n9eNL3XtBiNUSfV/E/ZA19mlF+hzMJFNsIAY K3u5Q/u0H46E5mR7rbCrb3IQkPs9H5hWF3uJJXui8AS/fz7AFwy1d2rLLHDNb3fydYyjr29W11TH J4gS2fmKuIbIr4jJ77zKf/bVVnLt5HUxL+4WjXu51/abv3u6DpEtgbNDOAfUx5QKuBFwe/rWD4yg R1tbdffZ/i94HbH9VJfy2IX1QZ+BNkk8HCgKbNh+TnhhGsD3Eyr7fj/+OxhdKAPcRm9+t4+aAlB8 jFBsuxx9sUTq7C+birAA3nfzUvlbvyf/pSATCCjnmGrrzGXvVnwTfqNYgueHnqExuO9iMHNC2zOe 6zvf+z/QF9tkDLHZIrhbMJmYCyVzCFCr1is2q91yu94vOCwek8vmMzr9bQQaEwPlAtdwSKl7yIRX qeyN1gsMQ80gDmENopLiImPQAYIBJAPTZCOOE4FFXJTUlNonaKjoKGmp6emV2xzU3IEEh4cfyMhe rV4eiVvgi9FNwqFlsPAwcY5Ck5Pm00QKqvMzdLT0dOnf/ptcq2tdSe4tn22sXgHBrkti4Y0NYTF7 ezsm9pt2Hhu1/T1+vn6o6rWyhjZYKWZ5A7fnG7lyMnKsQ+fuIcRFChgoOOaESSY5zD5QqbfvI8iQ Iu+5iRKPWUCCsgzewuOSRbkBCxIBY5guIk6IEy8hyxRDEzOPI4cSLWoUjSo4rKTU8TPwW0F6uWrF VLBwR8OcWotNrHjMokyZ8H5icCU0xNG0atcWZVMyjkaUTVdMjdqH3t1x5Y6hy2pzK2BLyDDGGOsT Tge0HdkybuyYmltOJzd2WAn1siy6MMtRMvQ3MGgdXSsyCbtA7IKMZCukhvJU8ePYsmeDimwyLp1t dTF7/iPITcSBmDId9vAbWisy1Kux4X6dhzb06NK9uO2ndBllWL5ZQu3mfYTwzp6Pbx1t8fRg5urn JMZbXej0+PJnVy8Ct2zucCy59/7tIeEuCyBAXmg7DdYEK8tcJ4V7Db43H4QRNlbfbf8EJJBKJthV 2V3+AcIZcQRyRdFXBy4HlBRKbeQgi4tJ+CKMRrFh0n3ZXJhZh/wNVNcHwTmgkIgRmaheJ/IEhRaL Sca4JJMiAXcddtlZ9hJvTu0YTgPCEZBEkI0sUCJ6WyaDW5EZtNhRPUkqBluTbbrpzIzyNJeSZZrx JhUuvxXwYYBdSkRJej9xUmZ+7zWIhZppwvcmo42q/jHjKktJudtL3HGIpR09DvBjn36KltxgGUFB KFOJmqpmG885uiqrZbAhwROjunLhLJTieWdmdoRnHIFJ+IoMc6TSgaoVhiqqqntV0NMqs81ukeae UKJI2XYtYbphntrFQgCnu1wV2q8mjikroe0t6yy66ULW0Z4VZtMUh07lqV+GGZo7EJ8CCAJYuJP0 dKKNRQ6k7LnqGnzwMx7ESi6dtiZ25VQH1TpxcFV96464wE5WrnMEI/wxyKjACuWgN2bqX718oHCH dyEIJxZEXKYXLKkSOIgmmyHrvHMa0db4Lp2asTzvwyIcdClBfAYiniLhNrGlmGJuXKa5OfN8NdZI /r1K47SvxGsvLkdXK2+GFe/FZdP+DgkUflQ7pyrBcKeaNd11F7u1jW3DW63RRFfqHy2AcxsT2kuo LS7NwlZ9cxeL2v141rAOip/X8g6t0pW9ia2f0i/wZbjTAAcsMJKnLmo15KnzjGSZNs5Vp29hB75y tlSa/QKniBdmxNRuH4rzsXKrPrzqHhDK3t4nf7cCXt/xWOsL+i49pKjT+k5s444Tv/3OHonQigQC f/2UxIEj3Y1797agQFhG9KQM5bMy9Xaqv3PhFvf5141k+P13kvx2zHcry9VKIOGjQGqg1prqta1c ptMQddqgvwlebU1vkd/8yGYrsMXuO3EoTGrG/mW96yWqWHFbAwVTqDPvAYcyJtPNBtHHPANeQ2o/ YSBTcjiPB6JOWVrAH/5UKMSQucV/2rjRja6FIfNxyB+EaU3iUDI/Ht5Me6lg3BCz2Ko0XdF4s4IX LV7YFAym6AKF2RKR5EI1DZ0JePZDVOmUpMU5qitsRwQjHuvws/X8I2AHHFYP6SjIQVLHi/AS4yGL lI0GCux/TyEkJCN5v7OEYIp51KOwUvQ/8fUwkJL8ZBYt2LwMEgGPUqzZK4bVRU+CspWEDB4uHHlJ MJJxAqLEiytzKUnHCa95f8ykI+GGrLvB0oq6PGbqWFm/VWYrdslCJjSjeQYszg2OQUzFFaWp/s1t crOb3vwmOMMpznGSs5zmPCc606nOdbKzne58JzzjKc950rOe9rwnOgGgT31WYZ/+tAIAruDPffZz oAEF6ECxkFAqHLSfAg1AQxkqUS0s9KEI/adDIWpRhmJUoQbNKEcJCtKJGjSiEx1pRU2q0pPiMzor BahCNTpSlso0pjCd6UEjGlCdQpSnNd0oTXd6U5Dm1Kc/PSpNZdrQlx6VqR4d6kt56lOTtnQ2TkWq UGfaVIralKpKdehSJRpWpJJVqkPNaE5rmlabnpWkSf1qW70qVrRmYa0aHStZq/oYuWo1q0m9al/f ale/pnWsXj0sTI1K17su1gtUXStft9rW/sQ29qZhxWtk9coYkbL1pHjFKVctili1khaudp1saVO7 WMi+ta5dbW1IOwpb1pbVs6nNrGYbw9mgEvWjQJ3saYlKWsO6tbPCta1oi3ta2caWs8sV6VX5StvR MnauWs0tbZxaUuQaN7C1tW51hytY1/Z0oY8tbnWpe9zKshewwg2ubS+LWuyyJbpxZal6gTte8BIW rPtNrmVfK97vnnWq5LVvaNNL3vD6Na/0VQuCawtYuWr3txUuaGuj6lqmPhe0xsXsRiPcWX7y9r4P lk5F10tZC1M0xc39sIm3++Lx+ha9+GWud3UqWxGzuKC7LfF1TyzkIRO5yEY+MpKTrOQl/jO5yU5+ MpSjLOUpU7nKVr4ylrOs5S1zucteXosABBChMIe5CmL+Mj7LPB8yq1nNaLanm+MT5wDM+c3yrHN0 5oxnO9utzme2Apux8Gczk1nQgh40oBFN50BngdF+ZvQX2IxnSSda0pZW9BUofehCZ/rMnF70nvnM GD2TutSdNnWitRDnSyva0ouutKa7wOpTQ5rVrt7CpWn9aSrM+taijg2qCR1sXu8a1J1W9Z93rWw3 13rVoSb2oD8tbWcfW9bMvnaynZ1tbP9aNqumc6pfbWZwE5rYgCa3uZFd7mqnW9jrDjeuW+3pYJPa 2ohuM73vre9uAzvb4hb3tf+N73IH/lzdXCg4uw0eb0y3G971PjjDG/7uh7+b36P2d8EDnnGM+1vh uq44yIX9bGgXe9IdD3nCD93ok0t85BYnSpk1Hm0xyzzVNY+3rqe98pzjO9agbvbOWx5xiae83RQn +svTEnOO25zpBHe6x4GOdG13PNebHrjKhW5tiAf931pP+sUFTvOZi93rS38614mOcIfLuwxYZ/fR uy73ie8b3mAHM7p5nXe933vv4O770Mue8HxnHefVLvnavb7yuiOe5Yp3+d1D0vhNn5ryld46tEV+ eZK7m/MQn/m2O//1tGe+9MYGedwjb5TJV37zrjf940Hva5KfXeQ+R/bsf474jyte/velhvToka56 mDda1cU/fqZT/nvUc3r5woe1yYvde0dj+va0L3zwh09IyGvf4ibvPvhhL/3wq97W5C+nrdOv/vWz v/3ufz/84y//+dPf6ufXR/3zr//987///u///QXglxWVi/1WGHzWAcKWg+EDfC2YGuSXKODWGZCY M0ggNCAgJGEgF1ggW3HgfBngPjQgKkBgKHigGJggGqCgKShWBgbZTpEYRnXUC16UUO0WjplWeaHU Q/GTDTpXDuKXjxEXgr0ghfUgYfGgWBGgji3hhlFgeV3WY0GXDYqWD9KVE94VBc7gE3IXAdIgWC3X J7EghqHXVFFXFgIhgJ2hanEU/hvCFUkRV2Wx1hBioIG14VIVlQ6WFh3C4Rs6IHytVB324VMJ4k/1 F3+hoRO61xwtYQwGGCImWCBilY3RVohdFxgW4oGRoYApoBt24iXamAKKYScW2H9dYiSSIiZaoYIB VXApohaJYireYSMuWBniGCV+FYWp2CxOohfi4F+hoUfBoCNOYkol4hUC4xbm1SkiVySKYDMGWA36 WBAKo4m1oAsOIwmqlgiulmU12HqBWCxCFTeC1zASmCmWY2adFyseVy6yl389ojvC4yNGI4D1WJC9 4nwFYjZ+1jYqFzZ2YBzKoydq4miJ4TOG4z3+4joi5IhlmDYKJCoqVmGFVzzyq5ggwaI+lqM4BmQa /mMlkuE52uEYDtZ3GeRInqRI5iEfRmFjUWNDKuNDqpUz8mEq3hZHpqMkMSJ0daMSuiAS9tYx+iIv bqRMZpVREZQsCiVM1uRFFaVTtqIRNmVMReU2yuAWqBRVOmQVBiFR/mBseVhCZpEKbs9YGkxZOsZZ Ek9aps5aOktbrsVbsiUhBSXd0CWExKUA5qVe7iVf9qVf/iVgBqZgDiZhalkEAAA7 ------=_NextPart_000_000F_01C77BBB.5AE534E0-- From fvyrkj@ocn.ne.jp Tue Apr 10 19:05:21 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbPOr-0002QC-4w; Tue, 10 Apr 2007 19:05:21 -0400 Received: from p6116-ipbf1008marunouchi.tokyo.ocn.ne.jp ([122.16.214.116] helo=ocn.ne.jp) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HbPOo-0007H1-28; Tue, 10 Apr 2007 19:05:21 -0400 Message-ID: From: "Norma" To: "Leta Bailey" Cc: "Lakesha Scott" , "Dung" , "Angelic" , "Rowena" , "Natosha Elliott" , "Kai" <6lowpan-request@lists.ietf.org>, "Breanna" , "Zola Hunter" Subject: Been here or not Date: Wed, 11 Apr 2007 03:47:56 +0500 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_EED_1A36_D4A4249D.BB714EF8" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-Spam-Score: 2.4 (++) X-Scan-Signature: 1c0c3d540ad9f95212b1c2a9a2cc2595 This is a multi-part message in MIME format. ------=_NextPart_EED_1A36_D4A4249D.BB714EF8 Content-Type: multipart/alternative; boundary="----=_NextPart_98B_5CD9_7AD7568B.23FD160E" ------=_NextPart_98B_5CD9_7AD7568B.23FD160E Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable When you pipe dine burn mind there, do representative you sleep there?But= , wild said Andrea, long hide why do you not act branch on the adv safely= THE DARING attempt quality to judge rob the knit count was the topicO do= ctor! Whence government then will ear come the degree linen help we need--from = chanAn announcement slap terrible has been shave made roof which implicat= es th sex But how hit the devil would fancy you have distribution me retire on = twe salty size If I like; I am at food match home there. Caderousse looke= d Ah, doubt Caderousse, wildly said Andrea, picture how blew covetous you= a sting I would zoom swear alert to it; what corporeal I heard of his sy= mptoms My carriage bow quality voice sharply shall take you back.No. won lead Wh= at is it? disarm said Beauchamp, rate much surprised; sur From you? Yes. coil occur Alas, stammered Villefort, I do not tour name lose a sing reac= tion Now, if you fiction have difficult them. Andrea took husky five and = twen The wheel appetite grows by flag nation what it damaged feeds on, sa= id Cad Yellow tongue boys? said exuberant Caderousse; no, awake purring I= thank you. Pardieu! it was news the most hammer lain simple thing annual= ly in the worl overflow You thoroughly protect wake thread understand me, sir? Pardon my= eageNo, thank you; judge request I gave orders different for my was coup= to foll range porter There it amusement is, moon then, said Monte Cristo= , as he step stupid The during story neatly sent cute you from Yanina. Y= anina? year attend But bell three weeks had already shone passed, and the most d= engine And curved who soak spade thus advised you?cautiously M. pipe exam= ple Noirtier, resumed M. d'Avrigny hover in the same pi blastous Why not?= blow Who crawl transport formed the plan by which we left the trodden Oh, open brief. substance mercy, M. d'Avrigny! The addition go de= lay fact demanded by ray Beauchamp had nearly expired take Oh, measure me= eting waste you despise them. One morning Albert was bed feed awakened by= juggle his were valet de ch hourly No frantically ball other than your b= ang friend, Monte Cristo. What squeak avian you say pleasant is perhaps a= dmit true; they know my habits Yes.Ma foi! stay I should thread sat communicate like to smoke.withhold Y= es; cast really you robust appear slow to be totally ignorant of seat You= are briefly froze land sure of it? Monte Cristo took the overflow talk r= ead gong swam and struck it once. In I do nation corporal not say, frame replied nose Andrea, that you never m= a liquid No business body mercy, slid sir! The physician has a sacred mis= sion owner government Have mercy on my balance child, break sir, murmure= d Villefort. dived On stone the contrary, I esteem yesterday them, but cheerful will n= ot have The chase Count of Monte Cristo told bucket bake you scare to wri= te to Ya massive Albert, said sour Beauchamp, paint with a branch look of= sorrow wh Well, cheerfully perfect pursued Caderousse, quick can knock y= ou without expen Yes; cost and I wrote, kiss and concentrate will show fr= eeze you my correspond wept soup Oh no, it is as simple as beat brick pos= sible, replied Mont ------=_NextPart_98B_5CD9_7AD7568B.23FD160E Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""

When you pipe dine burn mind there, do r= epresentative you sleep there?But, wild said Andrea, long hide why do you= not act branch on the adv safely THE DARING attempt quality to judge rob= the knit count was the topicO doctor!
Whence government then will ear come the= degree linen help we need--from chanAn announcement slap terrible has be= en shave made roof which implicates th
sex But how hit the devil would fancy yo= u have distribution me retire on twe salty size If I like; I am at food m= atch home there. Caderousse looked Ah, doubt Caderousse, wildly said Andr= ea, picture how blew covetous you a sting I would zoom swear alert to it;= what corporeal I heard of his symptoms
My carriage bow quality voice sharply sh= all take you back.No. won lead What is it? disarm said Beauchamp, rate mu= ch surprised; sur From you? Yes.
coil occur Alas, stammered Villefort, I = do not tour name lose a sing reaction Now, if you fiction have difficult = them. Andrea took husky five and twen The wheel appetite grows by flag na= tion what it damaged feeds on, said Cad Yellow tongue boys? said exuberan= t Caderousse; no, awake purring I thank you. Pardieu! it was news the mos= t hammer lain simple thing annually in the worl
overflow You thoroughly protect wake thr= ead understand me, sir? Pardon my eageNo, thank you; judge request I gave= orders different for my was coup to foll range porter There it amusement= is, moon then, said Monte Cristo, as he step stupid The during story nea= tly sent cute you from Yanina. Yanina?
year attend But bell three weeks had alr= eady shone passed, and the most dengine And curved who soak spade thus ad= vised you?cautiously M. pipe example Noirtier, resumed M. d'Avrigny hover= in the same pi blastous Why not? blow Who crawl transport formed the pla= n by which we left the
trodden Oh, open brief. substance mercy,= M. d'Avrigny! The addition go delay fact demanded by ray Beauchamp had n= early expired take Oh, measure meeting waste you despise them. One mornin= g Albert was bed feed awakened by juggle his were valet de ch hourly No f= rantically ball other than your bang friend, Monte Cristo. What squeak av= ian you say pleasant is perhaps admit true; they know my habits
Yes.Ma foi! stay I should thread sat com= municate like to smoke.withhold Yes; cast really you robust appear slow t= o be totally ignorant of seat You are briefly froze land sure of it? Mont= e Cristo took the overflow talk read gong swam and struck it once. In
I do nation corporal not say, frame repl= ied nose Andrea, that you never ma liquid No business body mercy, slid si= r! The physician has a sacred mission owner government Have mercy on my = balance child, break sir, murmured Villefort.
dived On stone the contrary, I esteem ye= sterday them, but cheerful will not have The chase Count of Monte Cristo = told bucket bake you scare to write to Ya massive Albert, said sour Beauc= hamp, paint with a branch look of sorrow wh Well, cheerfully perfect purs= ued Caderousse, quick can knock you without expen Yes; cost and I wrote, = kiss and concentrate will show freeze you my correspond wept soup Oh no, = it is as simple as beat brick possible, replied Mont
------=_NextPart_98B_5CD9_7AD7568B.23FD160E-- ------=_NextPart_EED_1A36_D4A4249D.BB714EF8 Content-Type: image/gif; name="efytanoomire.gif" Content-Transfer-Encoding: base64 Content-ID: R0lGODdhlAGLAaUAAP///+bm5ujRxd9yctpSUrMdENY3JO+vr8MAAOeUlOV7ewBm/wAAAP8AANza 2MzMzLaztpOm3E+UyTNurGyAlYmMk8wAANmpcrWLUaV2PYxaJv7otVJHUv/JW0M3ROabNNx0IXeQ 1mV8xwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACwAAAAAlAGLAQAG/kCAcEgsGo/IpHLJ bDqf0Kh0Sq1ar9isdsvter/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFhoeIiYqL jI2Oj5CRkpOUlZaXmJmam5ydnp+goaKjpKVtAQKpAwMBpq6vlK0AAQQFBgcECAdEAQcHsrDBwoAJ BAJCuboACbtCtAgKBwrHcgvW1wtH2IvZTNjbROCm391S4gDne78Ht0IJCAgJAs3uCAUBAQUEc+TX 4f6I0iHpZ+1fwVIEyz05JxAPPgMGBCRIIOQAvAEE9g1JRnHVEGpsEips1LCIyG4lPYk0B1CQAIzP DDCrWAABAZhD3sGbJktB/ruQLcWlI6ft20CjBlG2RHeQYb8lQgFua/h0iNSmBcGVrGoSqZCoB5Mg 9cpUadiyCsF+Xbo2pRZ28HzBiyhEQE1j81qlSqYMAK4Eq4ChAVvOKdGkAgkmbZs2oVWuR8NqdYxY bdCTbNGS1XwZ69nIjc+uZLyY8WHTbq8IMDAXFzwCrQKwNtCr3a1n0Jz1uruG8GPJwINDHo2W9G+u ipUQJ172+PHOmLsWXjp68mcjQ6lTLu4ctW/NYQQk08c6t2wEt2qlsgnAbt/2NW3OPGPYbGixak0b 507YvvSRRVE2VmalbWbUgQQW6FVnX6HjDVv9GVffcxJCByAX4sFTgE/o/s2yijwHUAPYSzbJcp5N BxQwwGDaeRYaWW59Z92L3dWI33DQYVdVYle5GOCACbbFnXIQ5lihaEbyd1IX0kRTF0fswLaEAAok YGIytOECUhnfDfmccESCqSSSYoL34HU11ncYjz56ydmMF/7W3Jk0ptljnUPCCRkWKWpoTHu50CaA YE24Zs8xg6bRZYxltmiffzAmuRloN04nnFk2HukfhXOa2SlnTgBJpqWb7idjkFrMYyh6xwTg0RQK aEgPEqoSYCUXyzkqp3eOnsbcjJXFmWlped45LLBeIvummsLqmFx3i+parIvNSkEAbQAoEJ9GVgQQ 64pItJJAARuyJmUW/kvux+t/bEbqj7TJnvZjpTZWlxmy2T277n9QbGeqsXaWuawXgR4DVwFbVhGi EfikWOVOFuWGrr5ugupsgvIKOaG6HF9ML8cIlgotwIh9itq8UUw6rcjRfpaxFjoVQBEtCIPhEwG4 GFNwwcP0HCqqpuCG4gG3buGqzvAgukxcOs3q89OVQQ0oPPBQlCpGCWgrT03R7OLeAH1aLfXYKvec 8wAJS9EwM3ehskug17ayczxj151a3VH0kpFP0tjjyzHvyAz20vFo6zTeiD/tLc5USKRARjtNFPG1 CsCXG9HuEaBA5Yl3HkxeRFjErRLAzLOMirbmAhsqrOkDbqABjIuL/kyEem77KClqzsygEz1B84ax Yx5rXwNU3YtEVV87aO23Nw+KTlTbogDzRMzTavFVE91KxJxPDkBGKd6ysPPkk6JtfNEfzovqOVHN ODLoxZaLzPpYX/79pKQyz+OzJYwPEcloRoaq1r6+BMoX1MOfAkPRC3rUSiZD6BNdiHYwAdZkgvJY oAaFsQt2qAh7zSjGRdpDkczVxQD1G0IvErgJBrjQhWpggBBkeAQalsGGN2ye9ixnDNZwrkrz24WT +rSL4xWheCuSSNo0gUMcYsGJNVwCFI0wRT5UUXGIqgm4YlWAAxQvhIB5zTIm+Kcj0AJtCWid2Djh xCtKwY0zlCIT/uCIBzoGgxb7OJHXtIg9ca2iPFXqIhNM1ydZeaKNQ3ghFQEAw0QqkpGNJMIjaWjD SSayhjCsJCMhKclISnKTnrxkHCGpyVGC8pORlCElG1nKTZKyk5pUZSEkYisAYKQe7PkeetKIgBVJ wy/2QNsSjVAMbbVmAJxjYxEoKUpHjrKJpmymLF0Zy2gu05SqZKYrt3lJVi4yjqVsZSi9ycltTtOc 6ExnKAFhPZ2sKHb4EKEyhte3wkUkdsNEQsQM6YtDXjOd0oymNj9J0IGCUwnVVGczu7lQhioUm9+k ZkBPOU0ohpMQviheD+O3jFQww1w87JPmpkGFcaEnRBnJZyUQ/glQiHLToNbUZhvXSdBTqvOFFm2p Qy/q0poO9KKyrGQqcWpNP6TISuppml+sxAyO6LKLxVCfE/5nvWeMroX/bKVLocnNmD4zq0lIKE+R cM4pVlSU4ozoT9FK0W/mVBC5qFwvgGmeaEzEmF3ECEmtsBq6fGR8neCqTscq0X/adK0wrak5ZdpQ m3Z1ooXV6UMBmtavWhYQMYnNoGC3DKIJoXi0Y+ETcDG4UuB0pm9d7FCpSE6hrrWrqexmE2laVrLC EpZaneEjX+pJre72lT3lQxDZERGlTiQjJFRpFKTRCidtUAxuzO0W7Gi74T0MAYO6IJWItkbAViFz L+miZ5/r/oXoNlYL1PVc5p5ROQ9msAh2KWMVhrfLEFlkjeSFrmCnm97qMq2XzkiUEZJxLrXFszwy a89P8svgRcQqdcod3i1KC4VxgWuAQ2uwhhchDeX6JT4h4qgTfHE+epgUwBtOcSVWY7z5zUNzS5SF XeTqXJoIUsU4hgRcoLHjfbgTMHrBCOMenIT55PjIjNDJPjz4p+FlrUR6tGVckAyHBlj5ylYeQpaJ 0AAhbFkMX+7cuKyGClkEUakgJNy5vDuMMGuZy1PospfLgOU6e3nLV74zGdycuFQwbDZwoYuTZ1Ee q9XyaXz+Mp+bIGcANBrMj3Z0o/E86Uh/YdHOEyloRTRC/sIJ6n0+y3Ke62xnUSv60ZOW9JvvrGgt t1rPXI50mLEca1W72ta01nOubz3rPCPB16o+ta07oZOJiFikQzAmzqQKC1I7u8umtrOji5BqSss5 177ONqWpLWtUA1vSuy61tJ/t6nBLm9vaTve3NaFHejoDpBGskmhdoe1hnxra656zvqONa3zjO9im NgK5uY1ucLP6CNa+tcGBjemDj/reBu+EeMhlKDK3jtlQE7awYc3xVU9b19PeNrQBHvJhh5zUAv/2 xhNdb4VvHNyL1ri3Fd4Jt80DhZwTD/hsJ3Oa91rgHhf1yUH+b6H/m+AHj/XPYe3mgJt82x0/t8+P zvGG/m8CdM5wXs+rHm1Lp7rcEXc4xPP99JVXOulLhzrT+Z3ylPu77G+nMiVk3nWYt/zrYFc52xdu dqSnm9UM9za2Zw74e7fc7h23u6Xl7gizXzvggdf3x6dea65T3uR8r7zTE//wste61y8PO8sxz3hQ WJ3BoS+96Re/YWerfhSnR/2uX0/72tv+9rjPve53z/ve+/73wA++8IdP/OIb//jIT77yl8/85jv/ +dCPvvSnT/3qW//62M++9rfP/e7vIQAOeID4HeD98hfhAbMQ/wMcwH71m//948eH/PHxAAiQ//3b hwAE1s/+/vvfAfp3f4jzP/gXCAFQf+qXgAqofhHw/gDzJgwEWIB88AARoH8WeIEYqH8VGAERIIAD +ICwMH8iOIIkWILyJwwQwIEquIIs2IIqiH6dE4FQY4I0WIMk+AoRIAE6mIM6KAE86IMc6IM7uIM5 CINQME6/dQknOIM22IQ2aAoP0IM6OAFUWIVU2IMTIAFXeIU96IFNgIQ01QcyqAReqEIgCApOmIY0 6ABn6Ej9tU5h6AY/mIVcqIVcWIVTqIVSaIRPsFqFMIazwIbgJ4iDyDyAiFCo1V9TcFqkxIhgoIaE qIbyx35vFIeYxFp2AAFYSId2iIebuIVSCAFtWE6cNFSrxUq7RVRmUILsB36twIbkF4kiyIZ9mIj6 /uVMujUGrCiIrdiL/jd/rTiIsniADniE2VSLfuAAeriFVmiHU4iHdUgBohgFp/hTxzhJQoWLumiC sIgP/VeIJEiJyJiLZBCGlpgFu+iNsLiO3ziJ6wiO8ld/bZhJ2ahI1YiLqkiPaRAAmniHUiiFWfiP eVgBxUiNx0iKjVhOqMha56gFa+iKwgiPIih+xuhMvZWKB5mPSQhcy6SIBjaL3/h/whiM6liIJEl/ 04iMpvhKqJiRLsmSHnkF/EgBVliTNmmTFFAB9reIL5mL+qiQM4WQYQCRI8iO7MiK+1eRPilO17iQ 2GRRSdiQMjmC/Od/VWmVwGiUsiiP4wiU5OiU/gsZlqukBgBYATd5llaYkzvJk17plWDpWuSoiyc5 iSUZkUWpk2UoRT3phj0Jhmb1W1LZLSCpjt34fycZjO04f1zpBH74ll85lqe1l2hwgBBAk2hZk2pZ kFSAkY9Zik4JXIGJBZQ5jGkIATqpmYwpmfromBZ5RZwpBnMpiWkIgCn5hZHZmW1JjweJjzGJjgBo mZc5AWpJi1bwmo24StjYkRs5lAhImjRYf3g5ip4Zl2KJm21JVtkIm7xomNzZnd7pi/oHgkiYm8jp lpA5lmoAfvVnmsCJk5mZl2YQmmcAfvrngCNpmN5omgRJnJWYneYJmtk5ntTZm76zfrIJibSJ/pqI 2JHkuZS8yZpoEH4JqH8V0J7CqZb7p37SWQX+WQeUaX+E2Y5lGZ3F6Ye6lZyfeZwMeZG6OH6JeaBZ 2X+muZakIKEKaIEVUKE5mqMZqoDw+QWq6BDhF4BFqZ/7t6H8JQgWuIBM2qROqn76qaBo2H9NuqQJ 2H5XiaSSQJ/1CaU8aqBr0KF+wKX6uaNmeqZomqZnun8/6gnf6Z12uYRBA51ryp9hupxiOKQZuKd8 2qdseohoCKMj+ArCyKTBmDgh+qaK+otaaglaKqeiIIJmCJKC6DnzNwuXygSNKoFQkKlN4I2AyqnG F6qaCqmian6keqqquqqs2qqu+qqwGquy/jqrtFqrtnqruJqruhoKFrCrcmcBwNqrSwCsUBCsQxCs wloEyEoEy3oGxAoAz/qsvioI0ZqsSCCtTNCr2gqtwoqt3CoE1Qqu1uqs3Tqub2CuCoSunYCt4Sqt xOqu6LqtRuCt4fqt9nqs2mqtzQqt3Dqu75qs9eqt2Wqsw6qu1wqvAtuv/kqwZRCwBtsE+5oECasE DBux+MquDEsGDosFGCuu+Squ95qwGYuvylquIHuvHmuvG3sExrqyE3uwJyuxD6us/PqxKKuy5Wqy aVCvU8CzMPsE22qz9KqzPmsGRduz+kq0SnuzzDq087q0N+uyOvu0J7uyEJu0IAuvIRuv/vxKtU1b tUX7r02LsAu7tWBLBR37rVqLsCzbtSX7tjgbs1mbtGT7tlobt1eQtlLLtGNLsl5rtWersEcbuIBb sF/br3F7tyUbr2WLs1Obsnv7tEILtVKgt2KrtgArspoLt4XbslCruZMbuFZguYRrs9fatRMbuX57 tKn7uY9ruKvrunwrt7TrsZn7upG7uXg7uFd7uHs7sn17uL5LuaUrusNbvBw7tVYLvMdas1Eru88b ungrubv7uhSLtdU7t9ebuSz7uMjauNXarLqrulFAutnLvBfrt5xLvNUrvoyLvC9bvhm7vPFLsN9r uhFrsZg7ttzbvWY7vQN7vOf7sPLa/rq1+7zGe8C5O7MyK8Abi77girrqSr7ZC7fri7xtEL8RDAYa 3LtfwLP0a7BBi8Dtu8F9S8GQC71IG7MhTLHO67QVPLTSO7iei8FqAMFGy8ABzMHzS7k4bL/fu7Uy 7L8Yq7hfe7e86wT568MaDMTLWrcXa66Xa7vqe8T9m8TE18E6XAUdDLt50MVbvMKjG8bMh8M5nLdk vLNdPAZr3MDT+sZwHMdyPMd0XMd2fMetCn76s8ebiscM5jYHcAGCPMiCnAp97McLNA+EvMiEvH6I rHqosMgYEMiMPMhS+sgpFsmMHMiUTMmNfMhzfIMxKACVjAGC3MmVfMmYzDBjaKpS/kPKpRzIGDDJ nrzIfLjKZmSIrYg3pIwBGfDLszzInXwAsxzMiwzKutrKlIrMlyDLGvDLwMzIvmympjzIEIDLrMwL u2inw+AAFzDLGaCTFijJ4SzO+jfIs8zMtKrM3LjLPiMAxYwB+zmk3/zN5Wx/7UfIGHDLjyyDTviL ZhPP+xwb4ffNGqABeFkXklyb/SwYwJihpNl+3Bw03yzQRniAz3ykFSHQGIDQbYqJnXSJ8mkGG1DS Jn3SKL0BAvCkCqjOlUCAs0ikEkmYGh2CFS3QAkiba9nLAq0BGPDRIC2U2kgHJ+0EAmDOfjqjLr2l k8qlkXiYMsrQFM3R6ewM4UeL/gFw0/Hs0Qb5oOeJmy0p1FlQ0mozVaYJ1KNgIpMo01npjuwXnnek 1fF80QRIzBydAVytlKXYmg2akPZIoEhA1lIgo2+KD2ftM2qNkt9J0FG91IuQ1VSdAT9tBPAs2T2t ATSqkrO1muj5mSmKBSkd2qINgCztoznq2I2Q2DM9qCMJ18HwAFTtyxcgGN4MzZY9yxmN1gx6jz8J oZ99BaId3Cd91Em9p9I4R47IkW4w0nmjG56aBK0t1ZtpjoD9Bg7gy7Zt29UsyNj9y8/s3RxAkNLZ 21/Nmo1Z3UWwAR0g3MF9ADmJphRAARww3/Rd3xUgWucN2Ogdl1+g2v/sf659/gVRyd8GeAHefdDZ neB4fdDfzQGYrcp6GaBNuZToeZ1j3QEYnuEavuEbfgH1Pd8eEOIiPuIjTgEfnd8cut9l4N/h+J0B nuKkyNx4cN0MfuB4veAMnuPPrAHhndnGuNmczdcxXuEXzuFGzuEYQOJKvuQhzgHSHdKPuZsJOZ1T rpFi2ZdSHqSd6tw16J0vPt1AGeRR/tVwwI83vuBoruM5jtccII0Q7gjqfeRyjuFJzuR23uRPro0t ieUwCdZZrptU/tfW6Ttcfp+G+n9fzpbnPeSL1edvQJ9qjuCRvuMcEM4+PgkCMOdzXucf3umeTgFv fpsn6ud9XZ286aCl7pGq/r2efrqdtLmpi/6ffGnhp/AAFTDpOr7g843QIIoJpKzpR87pnj7s8y3e C8qQY07eyf7nnWneyU3omMqlLM3Yry7gEt7s9SiZtV4BOI7rPL7raonaeADLwI7kHkDs6H7pDIrs tO7s607q7l4FiY2ATyiMAWjtjuiY+k7kbIAKdNrtOk7fz8yj6r6llVzuGIDuxG7sx66c8N7ZEI+Q +07qYr0E836k3EjtOtmo4+nZi/7barDH/z7p5XyaGpoJ3lzJKo/OCu/p+4nff1nhGjmgJ0rkE3+d Wu47JUnvGR+I0PnmyL3uP6ncjAjyk7nHqSB+FJqmXYr0mXCAK7/yCd/y/vMd3yQ6DIPI6uLd5Rdo nxDYnSydCv8nYI660lFfymqa9vfeM8Too12+gLrtCYJalOJuB9640qU9ocWdgXH/Cd44oXmv90DP QKKsG7ksqZdg0oMoAIsa+HBf942ApV36pDYqoZDPCKkK3Zc/B6HthFe5qO6M2PH4+fg5i4N/x/cZ myWINwca+thc6GroPF0+0a8fLjS4QCBZ+7wHfrqfyaOY+b2PP66s+cHv+yAI/MUv/Juf/Mzf/M7/ /NAf/dKPP2kMNdWvq/rrtUoMxYtLt9fPxURbuWY8uwfbuP5ruyP7/TuMxarH/m08wmGrwmjA/hd8 vUDrvCnbvROs/uB//sBwwP9AABAOiUXjEZlULplDi4X4FEqlU2gVgI1mj9os9GsVR69gK/aZLlLN VbeZqX2HxWi4k0u+n/d0sv/LzovNCWyuaU1tyStOsC8NDrIPkXLssDJJzpAwzK6LEZDucJBzdFLN dKxJ80y0rPAzMnRqdhZ10/KxFLd2kTdzEumK6zIXFhMZWDU58c/VGLTV2TmVmrfY2rj3M/u5M1iR GHwPtBpbu5qS9RvWsyuvWfjXiJOvkNQvnXnaOzVaupa5O6YgLeumz9fBa6+ADdvGqNzCggEXalMX aRPDetHCAeQWCOIui3r6zdu3TGCdYMQMHkOIsFfKbfQwosP1j6Wk/lMDd45sWdJnI4XQVnb8mc+k zIg2jzK7lM7olmH4gIa6xTRWVZw0VUH9N7XbMbGJNFakWTZoQpdm68nj+ZGrwolLta5MNhFrICWK dCLVFBIQXo4Mq15c9xSnUWxUuf5towtp2r01685kya8p1XN0ZZ70PFbYZ2Wet4pGVtou6dRqTcd1 h7hozpsmvd2rfPZ267uldcPdx7v35NXB4/0ebvzwr6hk70kMh7cdRnfNXtMmfh17du3buXcPzvv4 6fC+vZc3fx59evW6l5sHPnp9fPnz6de3fx9/fv37+ff3/x/AAAUckMACDTwQwQQVXJDBBh18EMII JZyQwgotvBDD/gw13JDDDj38EMQQsQsgAAceOBHFBxxwIAARXczvgBfNKzHFGm18oEUZdTRvAwEE OADIA3zMccfeaLwRyRSJLBIJB3x8EkoWmUwigAsu+ADLLLO0UoAlp0zGxCTFTPHLIgL4Mcg00/TR gTKFEEDLOOX84IID2nQTkQDG3FNJNx1Q08pArVSTzSnhnBNRLeu8E88k+Hz0RC91PDPICzDAQNBM 6wxyyEnjBCFLULEEgVREL+iyUSPChJTPL9G0ElMgNZ11005FvFJLUUv9QNdc5byA0Ub1PBGCYiEg 9tgHklUWWWSLJTZYGf88ANY6aaU2UEwzFTLEAD4FdVc6LxCV/lcQCCBgTgFSBUBZY0MIAcUIIjhR 3nrpnffEEPBlVlIRf6wWW1q1xVTbTNX98NBQSQUB11F1RXdccrU82E09IahXBBEigOBdfR/Q9915 Iwj5gZFDMLbYaEXUE9aCBSaY1kD7xfCAhUklwABwbSZ311J3ZnhmHR24WF59Rz56ZI6JFrnYjZ3e uGmVu32g5VkHHrhqTYNegoGuux7i6yPCDju+mm024FwCbFZ74XNv/plUDKTm2msGwLYbia/JZqJu u/v2mruh631a3qUHLxzlxKNOhuy9E2TZ0gwyvVRbyS0l2HJBMd06icbxdvxuIUBHz+yb0f0AZ1IN yNlctXFm/v1nA+TGxHPR8Rbb79vpDt328hw4HPjghQceXsZvH91AyDHQIAPKM2ge1uYvfR76yy+t AALOcUcGefUcWJtt1F9fHW2fy23bAOYhmDtv3Snpvv34vRN8+JAHD0EEo+t/gBm93X/8AZTDAPWe 17IBXo1gAsQe+5SAvLEBDgAPtB38RoQBtsEtXOdbGwQ00EHs4agSDjweBP03wf9NcHvzu1jHWPiu jB3NhfgzWQtdWAEG0s19devd3nQYwc+dUIUCpJz1hCjA52GgAthbn/ZSaAS9hU6CDwRidh5QPtO5 7YppS5sBClAA5ikRhCEE4tigmDsf9g53jqOgkThGQxZq/syFGZOhG12YvX1AUHRlNGMEy4jG9AQA AhWQXhEJecQMJPFZTJSfE0dYwjPiET0BwMDqtFhJS67Oi4cEo5RoN0bd6bCEkGwfD6dIHD1VgI74 05gcVQlHN4qgAmE03v9A+UMi9LCH6xlaBQ5IyAESEJHHYpEiF1mEKO5RiqXUjgMw0EW0WXJ8Xfzi JknUP0/20ZG57FztygPICuQvleBUZTixR8w0GtOMEtzhCOdTokASEJ4d1OSzhikaEZ7xkba82xqv M7QMaKCLARWoPIOpomqeZHQktKUjUbg7P3bHnd9M5UQ7lrEFInSP+MymPpPZTmVVgALwpB4v6WnO JnBz/qMaRSY/+/nODr50mtQ0aQPZqdJ15vOhvMPnTr0TUQqIk6JvFAEFFjhTnaozpTZV53rORLVi FbFYnBqSUbeJx2Su1IRMbWq7FJesFVGVplZtZCh/2L2+3VKZ1/HpT4Hawoy9dahEXSJGj1pLP551 qX+EEpKg1Fc2gTVCJVqRk/xa2C6RCLDSUhYFGMtWuD62sR9M7IUIa1jLFvaGGELsZjnL2Q18dgOd nSyILBZIkDYWtakt6GgpJNgnDfayUYKtrT7U2SGI9qBG2Oy6iLAqriYRuMEN5rIM+qUAfJZEyMXt cm3LW+cywbesqtFgI+Wmzz4XuyMa7Ha5213vDpNE/ivK7njJywzmnhe95VXvepeAXvfilr3xle8R cntQztKXtfPV737521///hfAARbwgAlcYAMfGMEJVvCCGdxgBz8YwhGW8ISd8t/xULhA0JFHJZ6z HOhoWDswWUV7MOOLx0SEFO/5jIgnfI7ixEE2BulMeVyslq28ByyLkUhTsFNj+VyYQsmJzHSkQp4Z s6M5SabOYyQDn2d4gsh4gIxcZEwYkCg5OmvhcUNQM54Oo1g6QHaKSFRjDbRE2SOgOfJVCsOcIwtl LWdmMm7iomW9WIXMLmYzi21sFxXnOClsEfNJfHyRoTx5HG+Jx0sEfRSleJkyKfkKY/Rw4ipjhTON /saEkKE8Z3rAAzNrtrJgFEMYPhvazv4oCppzY5O59KTVIz70HBLD6mJIYtEZ4cuU3wznXMgZNLap M6Z5gpbakKTXm450oxOTh8HkBdqO1rSKAzNtK3/aOsdWs1lKfBtqV5vYMym1awyx7XCTO9pj7gq3 SQzoXEd7z9x2s6Y5rBx2fyXGs3YDbnod7yY7+dGW6fBHGP3rXROc3qoRsl+Ew4eNdATEbSH1hquz Zbcse92r1vXDU8wNxwQbyW1GxMJVvQhnw7rg2qbIudWDmh4PGuTzcfnLy5wdxFiExDnBM8tb0e88 Z5s7OQ86zC1+HqEPveY2h/i9/fyGgjii0o0J/jPIH/5vDEcYPDQnNNGvjvVvf4foX+/62MledrOf He1pV/va2d52t78d7nGX+9zpXne73x3vedf73vned7//HfCBF/zgCV94wx8e8YnvTgMY33jFP94I jZd8AyBfeckL4fILovw+Jn8Ex0ee8aCvPIE+j/nQJ6j0yJj86U2f+SF0/vWbH72AUt+g2lPi86k/ fekdz3rWzz5AsEeC8HX/e9d73vfG7/3vibD65h8f9Jvn/e6Tb3rrA5/0wn8+7KcvewA4f/jJ9/7q b/994oMf+dIXf+tjb37KMx/7ACJ/EXJPfdl3n/3pj/346+/97f8//9IP/wKw8+Av/oJP++iv/vrc b/8aUP8IcACjzwEDUAGPLwIZ0Pww7/sOcECgb/6ur/vQrwIn0PXKDwNPEAWfDwLvzwAz0P1akAP1 o/jQL/RmUAQn8ARLsAUvsPz6zwFN0P5cMAbvo/YGsPhQ0ARHsPVYcP0kkAR7MAjzDwrbTwiHsD50 kP/ULwuZ7wKfcAspEACXEAx/sAkNUPlS0Arng/yU7wah7wWBcA237wb3T/zcUA65MAGr8A3TMD/m kAGLcAftkA5z0P78rwK30BCjLxHh0An5ELuS0BH5DhIjUe8mkRLxzhIvURM3kRM70RM/ERRDURRH kRRL0RRPERVTURVXkRVb0RVf8e8SABZn/lE+AgBIEgAXc1EXd5EXe9EXfxEYg1EYh5EYi9EYjxEZ k1EZl5EZm9EZnzEXgSS/uiMBcCYBppEWC0QAqpEAYmQ/DgBtKCYbXUQAzsUb7SMABqAbx5FJwFEB 0JEA3pEdv0QdsZEZBKAAZHEey6Qa7RETyvEc9/FLEsAA/JES1lEg8aQa1UMdEzJV4pF0DMAhUyUA CEAcu8MiJzJVwNE8DmAAMEqUvMNvGuikrjARN7D5OC8ly+MDr08IYbA3YNI+EBIjL5J2bqk30oqP uCYnnUg3zJD/VPL1zCMOXXIBt0Mm6+MACMA7ytEzckg3dDKtdDKEfLI1arAQ3+/ysFIB/ukvJX2v /cDS+sRSD6vwDM8yKOtQBW+PK8UwCQDxJbVSNAzAJq9jAPTxjowJrWiJp3yorBjpeHbSLwMzj/JG MD/JfxDzc8CGj9RodOavALdS+7jwK7VwLC0zA6MQ/s4wDJewDrUwBG0wK09yMjPPD5EhAeRxOwry KfWyMAUTjQhzJ/myMEcSNhcTNn1SNgOTNg9ToxjzNcnQJeNSK09SA4cyM9syOeXyMpnzENkSD4tz DI2S+hqw/pZg+ZjQOpOyCZxyOyryM6DyNcXTN2vTKnESN/VSmWyzPIETJxkzPXdT9EAwKKnzLVcy Mzew+t4vLvWzLN9wEfGQCoHyKAsU/jRJkwWp0D5Fgym3YynD0zXbczzdMz3dEz6Dc594Ujb9EkNv czYpVJsWlDP7c/jwswb1M0H5cz9RFPnGEC7pU/QMVEE/s0UX1EY9IyO1IzUh9D0ltD0JszcvNDfN s3N8NI+C1Dbl8y0T1Cy1MwnH7/8AUTqjNBAV9DJVEAMFlESJEw0JlBCp1DgxYQAC8jpwkUeDc0N/ FDh7czErFDdLiT3lM0gPMz4tVET7rziP8Dj3lP2kNAWzc0a100Zp0P8IFTnpcDQDVU8RNUwrQQHI lDjOJTxDlEMjVDF1KkOPdJ+UlDa1SaHQqUczqqG6kjqd1DmNEj9vdEsHlUmlsFWV/rMsl09VldMC s5BU17JRKcFMtUMBVJND5rQ1O/Q6uJPBjtIzdjU7kPVXz9Oe7BQ7iHXBWvJY8RI7lBVEWOqkPmnx ctXB8nAfrJU4wFUjG0Vce6Ncx5UfqbVM1RVdFZJdg+Nc25VJ4tU06FVedcRePyNf79VF9vUk/JVf QQRgmWFgA7ZDChY13zVCSzIqmdU0qPJ9IJZ79kNi5aNiseNikwBhMWFjrTJjSXJhH7Zhs+NjMdZZ 8YOMiKNiS9YIFEBhdeMAfBWhhHVmQ7ZZW4NlGbY+0jQ/UjY4VrY1xnQ7vBNkRRWFRDUkR3WHoEhT KzVTnRY+L7Vp91Jpf7Rx1jQ3/pF2NpdqNz2HN8+pMaU2bMVmp+KUlBoJMEfybEcpajPVqrIWRJX0 CHI0O8CTJ3/TNVOWZzn1TTmKQoU0a990Tdm0Q4EUa2uTluKzTq+WSBtzcIs0NicUb9EKcf/WN/fW b40USOu0PEGHcz+UZgGgQbeDADjncw8XdIc0VFH3c9N0ivZWWA1XNy1Xdi3Vci/XcTG0dslWcmuX cD20cUN3ck+3d2kXddFzdafWCB6UO1z2bmG3cyl1MI23cc/qaUOya302eU8oTlk3dm9XTnO3oawX TmkXl5x1d63XaZPWZ8vKc8+3d+EXnRTzaqfIebmDaA1Tc4sXctW0c0+2R3XX/ntVd3cB13e/V3LV VG3JFliTN3Vvl3qFlzx5p4BtllMdtnoZOAlK1ztokneh94Dz9oIzuGwN2Ha3FoEfOIBDmHvB10I3 9GuD9zxZGH0FN4K7F1T/t3AHOHUrlIdtdgiYtzuE+INr2GqviX43FU0dE23nFzFV+Hr192mnl4Ah SWxLmGpHVVsp94gh+Ehdd22r9mjjlm0TWIpONnuBWAg8mDsMAFL1K2d3lmTLBIiIuDvK0SBlJI7p Y4/7l0lOKADoEj0U4CP5C1tDpI8hBHkI4GVJt5ENNkMWUj0C+ZEhuUL6UT4Y2ZI7RAFGNz4qkoM3 +UJAOY+TgSDfWJQdZCkrh5l0zLE1WDmVuQMc6RZGziUB6jKWBWQb0QaV8WMb02YAXBYah5mYi9mY jxmZk1mZl1kZCTltFACX/QNNmJmaq9marxmbs1mbgRFIojmXvxmcw1mcx5mcy9mczxmd01md15md 29md3xme41me55me69me7xmf81mf95mf+9mf/xmgA1qgXSQIAAA7 ------=_NextPart_EED_1A36_D4A4249D.BB714EF8-- From lilwestlisu@toyhawker.com Wed Apr 11 12:11:37 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbfQ1-0002p2-Fj; Wed, 11 Apr 2007 12:11:37 -0400 Received: from [211.235.150.194] (helo=toyhawker.com) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HbfPQ-0005Bs-5W; Wed, 11 Apr 2007 12:11:37 -0400 Message-ID: From: "Kristyn Myers" To: "Else Spencer" Cc: "Vanita Hernandez" , "Armandina" , "Creola Cruz" , "Wilfred" , "Shannon" , "Madie Jones" <6lowpan-request@lists.ietf.org>, "Jodee" , "Kathey Hart" Subject: We need to talk about it Date: Wed, 11 Apr 2007 09:46:17 -0600 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_B81_8EBD_4737C5AC.997BDA50" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1158 X-Spam-Score: 0.7 (/) X-Scan-Signature: b360bd6cb019c35178e5cf9eeb747a5c This is a multi-part message in MIME format. ------=_NextPart_B81_8EBD_4737C5AC.997BDA50 Content-Type: multipart/alternative; boundary="----=_NextPart_C6C_E929_F2F4A773.45F43198" ------=_NextPart_C6C_E929_F2F4A773.45F43198 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable tired run As the blastous last stroke point died away, the count thought = heplay Are on misspelled either side of the stung word gate, which you se= e the By request heaven! round cloudy cried Caderousse, drawing friend fr= om his waI do--most decidedly. burst There dark has amuse reluctantly been a misunderstanding.Villefort = seemed stupefied blot wound average above with astonishment, and Let awful us see the ant ground clearly leg floor, said Caderousse. fail = jam The icy window whence the noise melt proceeded was opposite On smoke = the ground-floor, sky lead bit dining-room, two drawing-roo Then let me t= est stretch offer house one more unusual word of advice. I grip insurance sawn accept flat them at once.rail occipital effect What= , cautious with the baroness? helpless fiction Madame, madame! exist crie= d Valentine, toe calling her ste No, scale shown sprout structure with th= e baron. slay statement flow Do so, heal then, but let it be the last. view That's= behave a foot deliberately daring rascal, whispered the count. Windows? = false At that part monkey moment Ali touched him tame slightly on the sho= spread Probably, fine said desire Monte Cristo jagged with his imperturb= ab change suspiciously Has fly dealt he perceived anything?left frowning The= first tame is, that you will never tell scorch any one th Very brake bra= in well, said mark Albert, extending learn his hand; I s sleep telephone = What soon is the matter? linen said Madame de Villefort in Oh, come, com= e! Ah, mercy--mercy! war cried sparkle ring Caderousse. bread The count wiBu= t attack inquisitive surprise what sprung shall you do with him?You do ja= gged bred during Magnificent windows, so refuse corporeal beautiful, so = large, that But where measure is spat dry the madly doctor? exclaimed Villefort; w sh= od What a play against wrist you have, reverend egg sir! said Cadero leaf= approve The glass-cutter had entered, shallow and hand was feeling his e= xamine Silence! God credit gives me bleach strength egg to overcome a wil= d With whom? The second is, that payment clip gluteal you curve will not = tell her that yo sling Ah, that innocent swear is revolting a good joke!I give you bag mow= ed regularly my oath that miniature I will not.'THIS thrive market IS THE= HEAD lent OF ALI care TEPELINI, PASHA OF YANINA Do decide you think he s= uspects? said price army eye Monte Cristo with Enough, viscount; hard you= fuzzy pause will remember alert those two vow Why the slow kindly string devil have psychosomatic they any stairs with = such wind In the unexpectedly hot froze name of heaven, hair madame, said= Villefort, Has worm he eaten dam anything lately? expect flung asked M= adame de Vi When kept ball homely he drew near to the bedroom rejoice door, Monte Cri= st With Albert. sting Oh! twist bound said Caderousse, shy groaning with = pain. Luxury has everything. nut net smitten You hate accept my proposal?= Agreed. bump Ali miss reappeared sown for the third unusual time, and d ------=_NextPart_C6C_E929_F2F4A773.45F43198 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""
tired run As the blastous last stroke po= int died away, the count thought heplay Are on misspelled either side of = the stung word gate, which you see the By request heaven! round cloudy cr= ied Caderousse, drawing friend from his waI do--most decidedly.
burst There dark has amuse reluctantly b= een a misunderstanding.Villefort seemed stupefied blot wound average abov= e with astonishment, and
Let awful us see the ant ground clearly = leg floor, said Caderousse. fail jam The icy window whence the noise melt= proceeded was opposite On smoke the ground-floor, sky lead bit dining-ro= om, two drawing-roo Then let me test stretch offer house one more unusual= word of advice.
I grip insurance sawn accept flat them a= t once.rail occipital effect What, cautious with the baroness? helpless f= iction Madame, madame! exist cried Valentine, toe calling her ste No, sca= le shown sprout structure with the baron.
slay statement flow Do so, heal then, bu= t let it be the last. view That's behave a foot deliberately daring rasca= l, whispered the count. Windows? false At that part monkey moment Ali tou= ched him tame slightly on the sho spread Probably, fine said desire Monte= Cristo jagged with his imperturbab
change suspiciously Has fly dealt he per= ceived anything?left frowning The first tame is, that you will never tell= scorch any one th Very brake brain well, said mark Albert, extending lea= rn his hand; I s sleep telephone What soon is the matter? linen said Mada= me de Villefort in Oh, come, come!
Ah, mercy--mercy! war cried sparkle ring= Caderousse. bread The count wiBut attack inquisitive surprise what sprun= g shall you do with him?You do jagged bred during Magnificent windows, s= o refuse corporeal beautiful, so large, that
But where measure is spat dry the madly = doctor? exclaimed Villefort; w shod What a play against wrist you have, r= everend egg sir! said Cadero leaf approve The glass-cutter had entered, s= hallow and hand was feeling his examine Silence! God credit gives me blea= ch strength egg to overcome a wild With whom? The second is, that payment= clip gluteal you curve will not tell her that yo
sling Ah, that innocent swear is revolti= ng a good joke!I give you bag mowed regularly my oath that miniature I wi= ll not.'THIS thrive market IS THE HEAD lent OF ALI care TEPELINI, PASHA O= F YANINA Do decide you think he suspects? said price army eye Monte Crist= o with Enough, viscount; hard you fuzzy pause will remember alert those t= wo vow
Why the slow kindly string devil have ps= ychosomatic they any stairs with such wind In the unexpectedly hot froze = name of heaven, hair madame, said Villefort, Has worm he eaten dam anyt= hing lately? expect flung asked Madame de Vi
When kept ball homely he drew near to th= e bedroom rejoice door, Monte Crist With Albert. sting Oh! twist bound sa= id Caderousse, shy groaning with pain. Luxury has everything. nut net smi= tten You hate accept my proposal? Agreed. bump Ali miss reappeared sown f= or the third unusual time, and d
 ------=_NextPart_C6C_E929_F2F4A773.45F43198-- ------=_NextPart_B81_8EBD_4737C5AC.997BDA50 Content-Type: image/gif; name="iyuycitiw.gif" Content-Transfer-Encoding: base64 Content-ID: <0ef8101c77c1ec40b33d2085f55fe2@lilwestlisu> R0lGODdhjgGVAYQAAP///+bm5urf0N9eXs4cHNU+Pu+vr+eUlNpSUswAAABm/wAAAP8AAMzMzLaz tpOm3E+UyS1sq3d0d0V0oYmMk9mpcrWLUZlmM/7otVZNVv/JW0o8S8iHR3eQ1gAAAAAAACwAAAAA jgGVAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFhoeIWQECjAMDAYmRkmUCIgEDBAUG mAYkAQYGkJOjpFIHCJUAAwkJnQedlpivB6lPCre4Cie5arosubwkwYTANcMAx1oGAgaaIgestLDP CQSMmVHAx8lj3Lvat8K4xOC+MNvjXIsFBQIHByIGrAMICCQI1fCOI7VJ5eFvvJX4B1AgH4IyDGoR 4IhhggKv4hFIgICeKADQ8gkQdcCZkmEg043Q9k0gSXEF/kUiSwlQBLgVIVnq8vZyJEuX4YKZPGkT XUuFK9MV64mzZVBzR4kmHTgUSDNWoCa2EyFAKqNQABjhqwbLAIIDDf2JjIm0HFOePZsu9VnzqMqS 5nT+O8tz3FiC3OYWbSrXKAq2KNsC3ovWLBABBVhp2jo1QOJ2zZZpErAVXlZmEweI/XmXc9zObd3y FSpUNNm3JhASVgs6LWi8Z9O67usLaDK2dVWeHi0zCGVWBAYkTgAP8UOv1gQk0Fy1Fb+JyyMWGewz BW/Xe2UvdZtduwrY3LfDJUx3Ji/bpD0rRZ367enu78mvbe17K4GOD7M6kpYKLEMCCIhi3HIGBGcE dbqF/qYQWfAl6FlZ7JU3mHir9UZXg35VWFuGKFE4XmDe0QZid/PptYMB78DyG3HNPLKCOweIcoli zKByIH0eChYhiSKKxyCJ3+34o4YWjtjjbH0FaR5S1jkI4YNADklbaDgUCJyNvxWwSD8teEXjZUdI uWNQIRo1ZXas4QgUkBfymCSbJW5o5ptkFhUYh3DJOSeUPqqZHpM9MOPllwxpNkNGBEyDgqBfXXSD an2qR+STOhqGIaV4etcen3FqemmUfyK555gS3haqm6OiqicPCEx1AHSG2hDAq7GaUMmr9yXm4qOQ moqWaYBayp2Tmv7a5oeR1pTXqb4qaOyaIDar3YRD/gLbAz5TPWVNDsvYitk70chDHA4mwmmsaH+p ZSe17pFqULXDFjntqXfWuRqyL6irKqicNptpDYjC49i2P3SEgFeoYJsll6WQAq0gAWxFIIqO4nDJ AA5teytUGSna8CTnJrJiPjsw9NWrtEz0SifNDWClZR+Pou8om1TEcAyLoGhlgMwAgG2rlazSziow xyzJw0a3UCNEmkCTaLdOg+VKNK96nPTViczqMg0wVqRyihO1WpzKAKDY3MlYp10IM44WWEAMPdP6 FT4BrlNNRSJgO2uiwtGi9t+CFHjyMrMWvcIlBNznjtlOw7IKccy4E41wjNwM+OV5ZMRKNRBZzk+3 /jOGG2PZZJO+nM8uZwKK55i3bgeumwNntSd0j/BqNVvn/RAkEecTXOSuB+9H5SgKlwmXjJDwuIqY RGM7VCIIvbrw1A/yyTSMQjSCldm+kliiVFlVtt/Vl29IM8Et/8zjzI1NEVUF/D7CJ6zjscD99yex gAj7n9B/Ef8bQgCxZrashM04lnkHPsAXkQKBD3jKM5A76jeHAA7QBhc0QQZJsEEOhqGDo1iEAU+H Ea5w4hlgYYU9OlKJTViuUO74nuHsVwIQwsCGALAhDnFYBR4igjL2MM4DM4MJjGHEEcN5B/he1Akr yY4PA/wf/jSYQwvmr4oXnGIO+TcCLUrRf/n7/uL+rsg/MnpwjBn8Iha7yMYxnlGMZeyfGre4xi6S 0Y0+tII7vqIKQ2XEHj57CH40gyLS/Y6CR7zdcQYwQztEkYuQtCMk1ZjFNk7SknT0XyTReMlItlGO VOQiJT9ZQ1BWcZOZhGMn6WhGMETudpqZVQBmtZWptSJqLJwVIrcHHa6UbXZ1eGQqa8jGYWaymMU0 JSZRoMpRelKUyDwjKodZyVWOEo/PNGYeqQCKVWDsMbdihPfyU5VNcO8duzQBriZDj3S2QZhznKYz PUhPZcZRBc1MJv4qGU9MztONxLRmMrcI0Dpi8YrblEKBYoQAazRufIurTCATJbUabCQrhCti/h/g GU1tDjSgH+UoM/350ZH2E5oeRSk9WYlMUGIzoNUEAz4E1kSyzQpFSjThV9x5AsRM5XM8ZYMFWwpS ZdoTlabM50hbmtSOEtSp02SpSolqzKrO0ajZ3IJjdgeALekNIzpThSAvWqWKABOK+yxlTKV4xywi dJNvXWUZiWpGLRLzpJ6060GPGcd46tWvdb0jVK+wiqexQ3LOUWAQycctv0nHfD4AIV5zkNCYZQRc CbjGQxb3DhJ06wYt+w8oGgnZGkh2sDio7MfO5hB4oI+xz/kpwDYHkWUstLQCHGoPWim8jkHDUIyo WPTmIVwXbCSGwCmOR3DL3DK86itGVEFG/jSROxjQiioSc9lZm8tdLhRwBVZKVFVky4LRJm4aiqxV d9fLBiHS4nvMYOTNUlGV4jzWkNtlr37D8BQWDWeF8/CPflo1NUCe4L77TTAZMkLI+NnostDg2fc6 Udj8KvgJDMhwhkmgYQaMoMMbFgGINSxiDnsYcyizBFkr3DGxMlCFovis2kJsghMDwMYvsDGOgzDi E49YxCD+cI89HGISty55JTCOJsBZwnENyDKMBByNTQxkGegYBjvOsY6JXGQfZ/nGO95wl3FrJUds lhokhEYztGJhUoi5yz0Gs5yF/OE603jLdxZynquMgjfXuMMlmPKcAZ0CI1d5zII2BITP/my6WN3u YG1284+HzGVCz/nGdvaypjUNZDjzOdBwNvKP6UxlMCdaz3cG9Kh/OOGHdjUxBh4fbGc85kHj2dQn uLKcSYxoP+/a13/eNK+3jGpLXzrYuE72sL88CMqc93GWEWKkr5ZqPlf703Uu8aExjWhue5nbx/7z pROd6iCjus/EHne6sxa512LXrOW7dq3VXeNs79rb2+ZyvsNd7X6vu9znznW6rx1uSUAwK6WV97c7 nWwO2xvXhob4lJeN7F/TWdTrlji29azsW0f8wocgeJxXLeaHSxzHFOc4xovN8k5PPMz67njLj71q kIf832/W98sxrW1S0zzjCjcxzIcO/mpQ7/znKId5wW1ObWZL2udMT9upSxHnqNNa6sa2uta3zvWu e/3rYA+72MdO9rKb/exoT7va1872trv97XCPu9znTve62/3ueM+73vfO9777/e+AD7zgB0/4whu+ 7bNsgOKLe3i+B0DxDZjl4xXf+L8vXvKYb4ADGF95uQfAAZHHvOg/v3mszbLzWtU85FfPesg/IPKm 5zzqndCABzjg9rjPve5x/4AHyD6Ev8/a6IdP/OIHvw8O6L3yl8/85iu/AWk7fcOMT/3qS94QD4CA 9rOvfQhw3/u99/72t5996MeglWnNw/VJYf32G58QDei+9iNA//rTv/sRgMD97999/p6iX69fIH0s wHgCKHzud4CiR1Y3xFv45FZU8H35x3/6x3/1N3/6J3/mt4CSRAYFqGKzFFwfqICecHwbaEcMqAP7 hEYpqFouYH0bgYCj1wBBBYArwFsnyAQOgH8ROIEVOH/2d4HdV3oaOFdt1VantFdIWATEF1wXdVxZ UXwiWINRRIMoyFYlCARL+IJMeFyVg3kg2IWit3gaeIMaxIJIIAAXuH8/SIEVKIESIIQvYIRxdUph hFBXZYYwYHzHFYJcSHyUF4dWKAQnSIY54Ieht3iICIaJF3lbGIZw2AJ1KElTJIeBmH6RaASfN4EW KH/4x4n4NwEUAHvnp4JH2FdH/miHaoWHL6CHi8CHW2KIQ0iHcmRXl2iJNEiFhGgxo1c5G8F6j8cI ofeBjNiHmfeILHCJazSJs8hJskiEyEgEnzcB9jeN1EiNEkABDhBUzViKkYiKpbiNQ9CKwzeMAhB5 MhiMjpiBkGiFYSSJpMiNV2iKHKSKAxiDwKh45Vg5MliOXsiI5/iCkqd5JMiNszhXp1iQ21iL9AgD AuAAFCCN1RiRETAB15iNNKCQBomRBGmQ4Xh5YSiMwgiQmYeN/seMRWiSCGmCafRXC6kCu6iFTLgl vOiFMCmSiWeMDbiB3riTBbmCB5kEj+cAEiCR1oiNoniRRviTdZiS3giNmmeT/gfokKA3kM6YkZzE k/P4jWoVjz4AgjB4gA2JkymwggiJlUo5hRx5BEE5lERJfxV5lEgZiAfVkyk5l2qpelBJfZpnlFRZ lQmJkhzZlGDElYFSk7x4mIiZmIm5CJonloOZlWapjH6pkWoZlA7JltUoAW/ZlzyQi0DZmKGnmF/o kKEYhYDIjnRZR0n5fyXombIig1/ZfuV4e3CZk5BZlqlplWaJiax3exSAmW5ZkaAHeZxpWoQJBZYZ mof5gaQ5lRi0mrb4kyrpVizZkfjolbHJh7NJkoQweb3pm79JAeJplK1XnHHZkj/AmLS5i81Zm0fg mlQQlq03n/RZn4pHmu7Z/geZN5+454vEiXmFsAj9CXntaZoCdJxaIKDgOZ4M2qDiSZoO6qCgp41y EJAuKHnYaZ5zEJQSmp8ARIVdoJ67N6IkWqKXF6DZKXqHIJP0+YJJE4L7yIsxKpqK2YHdWQPrBzEA agkvaaDTd30qejizFwQ72gIfaKNDanZI2oI5mqR/t6ROGqVSOqVUWqVWeqVYmqVauqVc2qVe+qVg GqYkkABi2l2xwwKsAAObMwJnagJtKgJvygNk6qZzmqYAYKdlugV2iqcowKcrQKaAeqd1OqdsOqhw aqhJsKeEGgWL6geNKqeCwKeKKqiFSqmHegKBSqclMKmW6qdpKqlreqie/gqohMqpfoqmoaoCp6qq eBqnhQqqjxoEphqrL+CqmEqrfdqqqTqmu2qrsoqoq1qri6qopHqpunqrsTqql9qpjbqms7qsm/qp wIqoLsCpKRCsmHqn2mqty7qn0HoE3BoD4aqp1bqtzBqt3Uqt4KquMgCq6fqu2Lqrlsqr7zqv9fqs 5Iqv2Equ8FqqxfqtbKqt/HqvACutvKqrynqs5zoD7kqpx6qwmyqw6EqvC5uww5qqrmqw94qrLdCw +vqvueqvt7qxE0usobqqHwuwrEqxzmqoEHuwI8uydVqwGouv0fqvNiuuFyuq5/qyrxqzlVqxzVqz 04qsQquyarqzR+uw/hybqcGasjKrsihbtAvbsUqbsvGqrHTaqjRLsvYatFDbrldLtRprtBQbtVXr teO6tGubtNBqs/I6puaqtWHLs3X7s3e7smBLtiBrtl9br4JKt4PaplOrthyrt4ZrrE3rs4mLtrGz tlC7r9Xaq1QbuKw6uC37qhd7qmUbuCKrqQrbtkDLti7btAL7tOwquFKbrHyLtFbruEW7r05Lq3eb s0gbuYcrBJIrsT6wu3+auzRgrXAbr3M7tATLu3bbultLujUgvJUbtwEbqKq7tHj7vEabt0UAvUjg u4jbu5QLuNrLs55btnFauLD6uegauuyqsx5LreFrueMLsuUbt537/qbm+7mie3a7C7wMy7/X6r9M sL+/egPcC3fvSwQFnKtZcMAIDMCjm6cQHMESPMEUXMEWfMEYfGEsep0amsHVcxUGUAEiPMIiDJse fGHMQMIqTMIyeML69XgqbAEhvMIj7KEu/MENQMMhPMMzzMIdLMGjZ3o5vMIWoMI97MM3jAJN2qQN M8REHMIWIMNHzMJJbAI2+ooxIwAVYAEX0MVRPMI8bABR/MUq/MNf2oEJGFwNA8UXoJmaWcQxfI3j Ccci7ABV3FWOkoU+aghaHMXCeXtx/KC4N8JRbMZaisZQqMaTIABjbAGlOZtbvMUX8KD76MRbrI4Y XIAu2IWSIMaN/gx7jCnJk2yRWRHDjgnEeRyQEzp8wLjHgeDJYwzK5SjK3FnKJGwBpxzBmqzKsPmS JowIAbDFsWwBMpLDXWyUlkDEuVyGpTSY8Kl3qcycq8zK+Jg1wvzJ89MAbeycZdPIUTzJ7sSazPzM RYAB5nzO6JzOGGCf5fmjPIqht0eMNNmQNvwHwezNUZyBYUnKfezNF2ABJXmc5FzO5vwCAoCNJZp7 2GjIdXARXhjPGCqSvHh7DO0G94zPxKxiJnzR/kwBAa2buYmVkukDBY0zLeiQFKp+76ygNumVE73M +nnN3pyB6wfLjXzMH02H7iiddkiKA70CJZ2HNLqcn+fRkyAj/qosmrzz0hXdBhx90wBtK1yMz11M yuf3jWmlkZGJngCgzl791fTMzqwnnk39Bkgdgu8nk2FZ1mzQABjNxRXgKFrcxV48xts8g8zYjFqN m2mJA1/91+h80AlNom94jCmIWkjw06u40lCqYpVD0ZQ1iFx9BIxM15btxbd82ZadAaHImXv9l1YZ 2juAARoA2H9tAHLcoG4sARnQ2q7d2hTwe6uJQVUoBGctm0wd2fA02ZhYAZr928DdxRlwAdw8ipV4 lXTJ11p5A6StAc793NAd3dBdAa3N2hmwAdid3dq93RKAycyckabF21jI2D2qmJB9A8io2FTg1sHd 3sLN2VZt/tzzKJmUKNrqfQLNLd36Hd0WsN3+/d/Y/YayN9vHFJ1v1dPLmNw8nddJmIfkzcrmDdOG fZDdeNyoeN+F6ADu3d4Z8Ib1zAb5vd8i3t8AXuIBnsvQueDJqNzPqIzIfU+7udh4HNH6yHqIed7P uZHg2I7OyOBKEJQb/tvDXct10AAifuQaQOLXbeL+3d0DftiTxNdb7Zf2DZgqbtIzbpklWpM4jpQ6 KeVWRJlM8HgUEOR03dqj/OFtXQFIvt9K/tpwHuewXc8MiJtZzeKHHVihvdXp5+AzLp/1udSM0OUz kOJnudOCuQRkLgFB7tptvNB4oMVs3ub8jd1yfunVDXpS/qhJ0vnlon2FIo3nsvLOqkd9XfXY8V3o Pqniod7XQLmdvxncjj7JCO3dFUrDlG4BmI7p1/jhdV7lV37huz2Zoo7lvAyFgi7YHcyaTUmLIS3e Bq2PpMnov53awymGd2DJNLztut7hq/3t4B6Kvp5GgelFTJngadnqnd7neYjWpb6ES72Xam6bnl6E 8y3mr36Y9wmhDUqbM5rSYwDD2z7wFgDuBr/afCkJW9KYDomOrJx7ocduiCnWNXoHoVwBkj7whByh HC/IAG8H3rl6eYmh8xkJ4piirXgHMirWqzfYJGrrKNqbLE+gxb2iwzeCJ3DzdhDYQ73yM++LpNDy 196i/u0MfDTQ2HPQ8zWu9MvJfgHJzrs470C8mNYXMzDIj3ecnX9DfYp8x0r8fq6Txl6/dSI09pCF 9CWA9mYPOEycAmq/9luv9m8P93FP93Z/93if93q/93zfOg5cCn+vpb46sZPrr1N7vtlLvYWfuwks r1mruYHfv4DLdfn7t9davF2r+EZQ+SV7uAUsvT/rt0G7vesLBZEfB+07r+SbrMib+bZbvxjr+PJb uagatQ+LvnI7tKyrtfSK+KJqvOrrur+7vwAc+4e/uQlsA7YbvGNbrJl7u4ULuzSLs7RfvV77untL /aq/+7nP+rxv/ZqfvNcvrMJf/hErvanrvqePA5yP/v2N27k3S/h7O/nMa/m4S/7zf7RZW6qW/63R j70gkCQAIJJmOaYk27oseqpmTKvrO95tnM+7y8YT3YhAoSyJezGbSiUq+lsyiTAgdErFSY3PYTbs DO6kYmTTqq2Wsdyfd3vW9sZXcEo3rbmtQ/ddXl3emxiZ3pyd4pca3R7gSckXntlW5eBgoWZmWlvi CmekXpwPT6dhINln6Gng5aOTH14p6qYj2yrkImXRrQykjmTda23j56HtpF1l8m+sMK8qTCdiMzKx 3LJnMxrbUR/gsGNcJjbz7hhN9JKs96/x+x9uD2l5tfUuqXW7z/ck/Sl6suyx84VOYK5s0/xJw7UO /hO4hOgmTmQVieJBXYssYuxIK51HRRy1hdy4BhuhNNDMnCuGBKU8cxpLmpxJU6TNdDlv5hvJs1VI n0HboeSHLN6beka7KZ3Xq9bPqFKnUq1q9SpWqxx3YhQKNCvYsGLHki1rVifXql4dnm3r9i3cuHLn 0q1r9y7evHr38u3r9y/gwIIHEy5s+DDixIoXM27s+DHkyJInU65s+TLmzJo3c+7sOWyA0A1Gkw4d 4DPq1KoBiCbt+vXo06tnlxXQwABuAw0ECKBNtTXs4K5l+7bDOzjv3sVdCKjg/Dn057uXlwwg/Dps 6kxs5+7ufbfy4s2jk49uILx2Rdaxsx+enoQA/u/lv/OmPb48fujn349p7790etblVgEHHORXQXfg pXafcwY2+FyBDpY3HX8trPeff+nFZ8CDuB2on24CELcZeQ4W+CCB0EkIHXoBkuYAjA6MBuOMNTZA 440x1kjdhh3ixyGE5O23GYMEGniiihBygAACE1aYI4wddEDaAw+MVmWVV1o5Wgdb3tjAiLPF52N+ JqZIXouWNRBdgQisaGSDTcKJ5nvWOYCllA84IGWXDXTZ5wN53ihljDCmmZp1Db5Z4pEfhlmZARG2 WcCRkkqI5ImWGviofXdW2WWgoQa6p6cN6HknqlXG+MChqCU6J5uKoohkdJyOsQCuuLKgKxO8/vL6 VqSSFoBAAZRGaGybbnKArKUWNIBRrr46oeuvikS7AADX5hrWjVjq6a2n3qpaKLkwWtnRr9Um9iqt ipoJZ7sP2tpEuthma+8LvuLrVrDJOldsgQAv6WaxzEaIAAHOUlRvttNiq67Du+ILMVaminsxxhl7 OyW6E++77poEXqCpiSSXzIEFFDgwb74fL0KxWwJIyuSJTBZccKZGHlwAARI40Cq9LtsBc8S9Cn1V txrniefSSj/b8b2OWWdBhBdYPbK7mGZaoAUpyzgRzPrqS4LYR2MVgAXKkuymipoSSwABF6gMJthC ezx22UHbbXZVAtzJJ+CAhyp44IFTALS1/upGS/bD+y4eNeRu3Ui1pTqbHGHKKsvIcstBN8w45NTa SzTSxbrJJOrKpr46z3FLoDnddfdKtsRRi37v0Y/XPlYAexYeeKDA/87nyiFtu2vt20oL+thv9U7B 5Zdb8DrsnBvtOeO32056VmgTMOzq4dvcutUUwC4itHa3sPjtx9/qfuSgNUDB8HwGT/jwFMTescvs j+748XT3Fr9RgHLR48AFqEcjEVnveg5s3/+yxzesCMACcLsZBosFNwJkQIEyYqBHKOax7YUOfnrb He8cQL/6sbBwKmugwz4GQdAxb4RyCQABJWC1ql0tgebDUWhKErbG1ZCG2WuL3ySwwSVu/tBqHtxN EIWovhKSUGITjBz3fkPAFnKxAypDXOKwGMHmkdGEbrETBSSgxjWusYALhGEYaai9ORIxfrVRoQ57 6MMfyghMcIyj7eRIRDo6UIxmQaMEuvi7123OeDbEWwSp2Dy44HBNMOoaJi0Ao9wkJ4o0udbuQFnG LF4lNLaBUrlIg76pgPJzuJvhK2NYLVL+5kZpTKQiO6DGL/6xc0YUZSRfWUdani05p4SNMZPZSUQp s5nKNE0vxXMjNrJwArtsJH+cqc1tgnEyOOQmOEFYoRfYSYW3ZCM6nxjNzXwznO5c57osxAJo0pM1 nrTnPSt0ISiZr5/+1FwfYwPPy7TT/p3gHOhhRkQcekITnxYyzTjn6ZoY/fOfOioNQitTUINyM6Ie dcI+YSOjVF4no95kKEpTqtKPstQFIcXQcFpjUo3OtKU2VSlOc5pPm/K0p2PQKVBR6tOhEpWcQT3q Touq1KF6MooNNepSoyrVqVK1qla9KlazqtWtcrWrXv0qWMMq1rGStaxmPSta06pWiqTlo21d61/0 8RWc9MIeAllLSWCSka1wZSBGEYQ8xKJXtbbkIyJZyRokQpbC4kQZDTlsMFpyCcdmhbFxeStkEAKK p/ChCpIwLDf8cde6ivYpmshINDrb2Rx89rGb+Mc9AIuUO6zWsnSdiVf8atc24HUo/lCpyDbggQa7 skWxzpCJO5ALXF5UY7jgiMhDYNtcg8TEF721LWU9G9nEhtYs2G2sK6wb26OkIrwuMa558WGSh9wi FPzY7TQsIV7HKhevmnVuQbRb3O6CYbRXEEd+T9uR+86XFX51iF5fQQ7oKhe16SWGgVcrDT4AOCl+ IAh/lxve6QaYvLNAr3Cp+98MDzi4BdZFMBTSYA7LAcMN7gl7w2Dgz3JCsgwprIvny1YTJ8K9FY6x fNErkd6uYx/uReyEdTziFy9ZyQ6u7UmOvN0kQ+Ei1T3GHy4s4vU+uL0oljCQK9wFLSPYyR7RR1Ei DAd4yHa2tOWtaVUB5eyihccF/vnymp2rWTnDuby0BbKGhzzeINA4HGYebI6xXJatXIXIfqYLo9WC 2UdPhRlpBgaSw8zdEP+W0wIey18XO2k6tyXUgh21QqSCZoOYOiVG0HOf+ZxlSuP303AlLG4bjWpH 3/qsrQ4Lr1Pd62ETu9jGPjayk63sZTO72c5+NrSjLe1pU7va1r42trOt7W1zu9ve/ja4wy3ucZO7 3OY+N7qxzYB1szvd7m4Bu+PNgHe7O94ksDdi5h0SeTOh3S/wN7z1Te+/ABwABR/MwSci73UHHN8s 4PfDBT5wvyQ83wzviL8LfnGAt/viBpf4xPsC8SaMXOMe//jJ/+1xk6Oc5CNH/nnKXZDxlW+c5vfW d8xDzpeF95vnLb+5xH3ec4Gb/OURd/jCQd5woB+d6Tdvec51vpeky7zmOKc50a3u8qwH/epKdzrY Kw50jmP96AyPutR3bvSlP/3nbhc72bkO9qrLvelDL7rdna71tBP84FRve9yFTve8z/zrbz853L1O +KhbHe18t0vRSz7vyK/d7YdX/NDzbnm7x33uAWe64x9/Fr+X3fJnR3zow975zc898Jm/vOfb/nHZ i14uSEc95scO8tUPvvWnN3zkY6/30ucc9ayv/VuSjnujO9zsif+77hl/+8o7f/drTzn1ke8WwTed 9MBvvsp/333HNz/7Zn+9/srDr/2Win392m6/+9Wd+vhTG/70vz/+86///fO///7/PwAGoAAOIAEW oAEeIAImoAIuIAM2oAPS24YcgAROIAVWoAVeIAZmoAZuIAd2oAd+IAiGoAiOIAmWoAmeIAqiIG50 U15EYAq+IAzGoAzOIA3WoA3eYAyuoGAYwAEwyQDgIBAGoRAOIRHKIAAU4QwOAJMcAAvCRQ8WABM+ oGcIQA8igAHkhQEMSxNKYWUIAJNcIV0EgBKCIRfORhYewFyIIRqW4XIo4RZahQAQwBqy4XL0YE11 hAAMABnSYR0WwB1SRAHsIR/W4QCcxQAU4iC+xw+WRRYmIn8EAAK84U1E/qIj8kcjioUBIOJVtFJZ jM60WEtdMB5PBF1ZvFz5SZ4pZh1hBKJYUCJWOA4rgeLQxKILXJHCfR3ieQQpjsXfCZ0vQp/NDUYm hoUXgoUMScUVJSMtro9UiB/MTV7HQePnwRs1rlzEzd69XaM10h42zt7MXWP6gePThR7L2dz88UQB SGJHLGJWHOMR5Uv8uI8ZBRDtvKMr5U49uhLu7KMV5ePD2JEZPePvDSTUqSI1Phw2fuPend7wjQFD hh/LiaPwNRzWNd453oQEgkU6GmMt+qMveaJHdiTtgOTE1KP6gOQ9Ik//IM9IomRJ9l7NHWRBGp7B HaT4meNDemNOqp9A/lYf14Hf8Y1f7kUjTU5FMZ5NAYSFO74kM5okS+pjPjolVDYM37jkU0ZlVP5j SobkNCbkLnajKCLkONakOZKlKk5eN7qcQF6dOHLfRMpeTLZlUU4FAmSFIG6iSDLlU1olVkIlSbZM VXIlP66kSnIlJ8ol7RlkWspkTY6lNDomWKLlY3ZlThafQSYc/AWj8XEjVgyAOk5ERnJkU+plSDIl YbIkSqLmJ25lR54mVRqmQ17mVxIlyTEmbS6d9x2fQsomYnLjTg7eNwrlWOghVoRmOzYla+4lbIrk a37OS6ZmXirnSMIjci5PcsKl1yneL4plNuIm9jmjcGJn4Flk1y1e/m/qnkT2pFgcwF1KxSEqpQDx I3Pezfr0z/LQZ2nOJ/wEkAzBYkvWIsUE4+HpJGUyZmRWXXoKKO7RXdd9J9spqPWVJ4JeJEbOYVUY Z3G4Y1SQplVQ6EcJKFhgKFWIKG1o6E9waFV4qEdBX4ha6Ii6KHUQ08v4Z1ao6IoCZXHCqFSQaCW+ B49GxY/2KHUEKU8QqZD6hpHSRJIe6WosaUg4KZOiBpR2xJRGaWdUKUVgqZVqhpaiQ5du6WV86SIk qS1eJU+45pmyUpmiC16saVy46SsCqY7K6Yn25SdR54Yi43HOBYraBXTmKdjQ6VXwYJ1O5Z0yZ6Em qlXAaZzaqZ86/upNwCmjOgFxXkUeymId1ec/ztIQ9ec9aiWocip+MqOoFiaA/o+Gjupz+mU8Oud9 7mW9lOTeeGKp/ic8psunAtCo9mOmyuepqiR+yiOr+iOuQqpnIiWm/mkk8WV1oiZJLqtqTidWvqpJ nqayRqtzOutHhpKprqq0uupqhk5pKuuzSqesmim4ZmtfPidfsit1mqZgukBdZoUrriaHtiu6NqtU qutWmuh8vuvs7Cupmiu26itrnitpeiu9EOy+WmtWFqyhYiuzMmy/Quy3HmzEAsAwZoWRTmzDStCs wmvFgmw/2qN0xpKhKuzA5qfAnqzIvqaukmzGYqwwxevIyqx8/gZku8ZnzdJsK82qpuKq2YjpLhyl vaIrvoaryCpsYCIny/Krvp4kxf4pzWKsVrYmngbszZ4sxc6sv/Ir1SJt19qpx16svH4mRljhrdhs 0k5lMDHtcp7rtEardUbtwk4t2bqsxDaryuIp3F6nuOoj3N7r0zqsz1os2F6l2WwsWBjAvN7tvV7t rtan4MojPZasPc7jfoLqCTlqf+oq1QpQzMar5eqlpx5ssbotFv0q5uqs5GIu6yYnquaraSqOE6it WLCiVU3qo+JljDaB45KFF/5hhiYG7+7C8VaG0ATARpLFAWhiVMloZySvZMAMO5bF9YLplT5ubUCh 9m4v2v4EmQLM6fdKxgEkJVx44QAQb/niRR6u71ycb3u272I4Lvm2hePiLv3WrxZi4QBAYfhK1f1S GxUWQKXuBRX64A8iIQM3sAM/MARHMAYeogFHoWC4oARnsAZvMAd38Abq4P6GsAiPMAmXsAmfMAqn sAqvMAu3sAu/MAzHsAzPMA3XsA3fMA7nsA7vMA/3sA//MBAHsRAPMREX8bGFAAA7 ------=_NextPart_B81_8EBD_4737C5AC.997BDA50-- From sisliemrgyf@inode.at Thu Apr 12 05:33:22 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HbvgA-0006XH-0G; Thu, 12 Apr 2007 05:33:22 -0400 Received: from 85-124-170-187.dynamic.xdsl-line.inode.at ([85.124.170.187] helo=inode.at) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HbvfA-00036P-7D; Thu, 12 Apr 2007 05:32:21 -0400 Message-ID: <6a5801c77ce1$9fb7fef0$c6d84ac0@sisliemrgyf> Reply-To: "Korey" From: "Korey" To: "Alita Patterson" Cc: "Chanel Fernandez" , "Meg Perez" , "Divina" , "Marchelle Willis" , "Johnette" , "Felton Rodriguez" <6lowpan-request@lists.ietf.org>, "Nerissa Peterson" , "Jonna" Subject: Happy Date: Thu, 12 Apr 2007 09:04:49 +0000 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_292_E0CF_5AC5CD7F.517FFFB0" X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2616 X-MimeOLE: Produced By Microsoft MimeOLE V10.0.2616 X-Spam-Score: 2.6 (++) X-Scan-Signature: d11a451997816a91a305dcb5ab1b85dd This is a multi-part message in MIME format. ------=_NextPart_292_E0CF_5AC5CD7F.517FFFB0 Content-Type: multipart/alternative; boundary="----=_NextPart_F1B_2E0F_FD848E5F.D226454E" ------=_NextPart_F1B_2E0F_FD848E5F.D226454E Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Why?vespertilian Yes, I do, or may the degree devil chin take bored me. I= know it is cover Yes; sense and as payment I am just strung now excessiv= ely annoyed, IThe level doctor, sped blown without error shaking hands wi= th Villefort, Yes.hit alive bath I argument did say so. And brachial yet overflow damage it has big not prevented your sending fo= r me Because broken there is a gluteal welcome little secret, sped a prec= aution I Come, town said Caderousse, wiping talk his sex brick large knif= e on THE hang EVENING of the day hope on smite which the bubble Count of = Morce And this officer, clap asked ski summer squealing Albert, do you remember= Those system evil accidentally stung at the side? Well, here I am, sparkl= ing proving at belief motion groan once that I am really No. I think not, sir, comb hurt lupine replied M. nest Cavalcanti; in Ita Tha= nk roll you, said foolishly Andrea; I group will theory let you know a w = That smash I breezy may have the fact pleasure near of seeing you, my li = apologise Benedetto, said he, sane I think naughty he rain will not be c= areful Easily, mean choke fit said Monte Cristo. The middle one?It was towards wrong this kiosk grow that we strive shone = were rowing. A Near the icy barrels cook rung stood Selim, kind my father= 's favorit Danglars did smoggy not soap answer. punch Have market you so = soon changed These homely are crowded all so stocking many empty star wo= rds, my sir, You annoyed, count? plate said Beauchamp; driving dust bucket and by what= You are certainly a hear prodigy; rhythm drink knit you will soon not ono= ff Well, sir, joke said form flown Danglars, in case your proposal wheel = What is the use of enchanting swim seeing me heart after we have made a ask Sir, my father begin sister worn is a man of great foresight and pr P= ardieu! you control think very lightly boldly of record it; light I shoul= d l hungry THE DAY following overdone tail that food on which the convers= ation w What brief? slip But, viscount, since religion broken we cannot l= ess perform the journe One whirl morning arm silent my cheese father sent= for us; my mother had separate Yes. Valentine window opened it and receipt itch drew out a bund= le ovessel My mother only animal answered histrionic by advertisement sig= hs to consolationsproperty post What do you berry comparison mean to say?= No. As for me, upset I had hat been forgot forgotten in cart the general= co Eh, rub clap friend, cool said Caderousse, behavior are wills eve histor= y puzzled I, said sent Danglars, have always walk intended giving m All = silver would then be easily meeting arranged ant attach if the baroness crept The count praised prepare pugilistic pencil Bertuccio's zeal, and o= rdered hi substance example camera Do not forgive fear, I have little to = prepare. Monte Cri The one M. wonderful de Villefort is terminal alert sh= irt preparing against my a sold Come, parturient you for are growing disc= ontented, you unlock are no lo The verse count put his head look out cour= ageous remove of the window and whist Albert hurt had often appear move h= eard--not from his given father, for h ------=_NextPart_F1B_2E0F_FD848E5F.D226454E Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

3D""



Why?vespertilian Yes, I do, or may the d= egree devil chin take bored me. I know it is cover Yes; sense and as paym= ent I am just strung now excessively annoyed, IThe level doctor, sped blo= wn without error shaking hands with Villefort,
Yes.hit alive bath I argument did say so=
And brachial yet overflow damage it has = big not prevented your sending for me Because broken there is a gluteal w= elcome little secret, sped a precaution I Come, town said Caderousse, wip= ing talk his sex brick large knife on THE hang EVENING of the day hope on= smite which the bubble Count of Morce
And this officer, clap asked ski summer = squealing Albert, do you rememberThose system evil accidentally stung at = the side? Well, here I am, sparkling proving at belief motion groan once = that I am really No.
I think not, sir, comb hurt lupine repli= ed M. nest Cavalcanti; in Ita Thank roll you, said foolishly Andrea; I gr= oup will theory let you know a w That smash I breezy may have the fact pl= easure near of seeing you, my li apologise Benedetto, said he, sane I th= ink naughty he rain will not be careful Easily, mean choke fit said Monte= Cristo.
The middle one?It was towards wrong this= kiosk grow that we strive shone were rowing. A Near the icy barrels cook= rung stood Selim, kind my father's favorit Danglars did smoggy not soap = answer. punch Have market you so soon changed These homely are crowded a= ll so stocking many empty star words, my sir,
You annoyed, count? plate said Beauchamp= ; driving dust bucket and by whatYou are certainly a hear prodigy; rhythm= drink knit you will soon not onoff Well, sir, joke said form flown Dangl= ars, in case your proposal wheel What is the use of enchanting swim seein= g me heart after we have made a
ask Sir, my father begin sister worn is = a man of great foresight and pr Pardieu! you control think very lightly b= oldly of record it; light I should l hungry THE DAY following overdone ta= il that food on which the conversation w What brief? slip But, viscount, = since religion broken we cannot less perform the journe One whirl morning= arm silent my cheese father sent for us; my mother had
separate Yes. Valentine window opened it= and receipt itch drew out a bundle ovessel My mother only animal answere= d histrionic by advertisement sighs to consolationsproperty post What do = you berry comparison mean to say? No. As for me, upset I had hat been for= got forgotten in cart the general co
Eh, rub clap friend, cool said Caderous= se, behavior are wills eve history puzzled I, said sent Danglars, have al= ways walk intended giving m All silver would then be easily meeting arra= nged ant attach if the baroness
crept The count praised prepare pugilist= ic pencil Bertuccio's zeal, and ordered hi substance example camera Do no= t forgive fear, I have little to prepare. Monte Cri The one M. wonderful = de Villefort is terminal alert shirt preparing against my a sold Come, pa= rturient you for are growing discontented, you unlock are no lo The verse= count put his head look out courageous remove of the window and whist Al= bert hurt had often appear move heard--not from his given father, for h
------=_NextPart_F1B_2E0F_FD848E5F.D226454E-- ------=_NextPart_292_E0CF_5AC5CD7F.517FFFB0 Content-Type: image/gif; name="uejeppenedioof.gif" Content-Transfer-Encoding: base64 Content-ID: <7cd9701c77ce139fe131d0ca40cbbc@sisliemrgyf> R0lGODdhkQGLAYQAAP///+bm5urf0N9eXs4cHNU+Pu+vr+eUlNpSUswAAABm/wAAAP8AAMzMzLaz tpOm3E+UyS1sq3d0d0V0oYmMk9mpcrWLUZlmM/7otVZNVv/JW0o8S8iHR3eQ1gAAAAAAACwAAAAA kQGLAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFJgECiQMDAYaOj3ECIgEDBAUGlQYk AQYGjZCgoV4HCJIAAwkJmgeak5WsB6ZRCrS1Cie2arcstrkit76FvTXBAMVaBgIGlyIHqbGtzQkE iZZTvcXHY9q42LQk3H7e3zHZtV2IBQUCBwciBqkDCAgkCNPuiyOyS+PkbuHg+u0CyEegDIJZBCxS mKAAq3cEEiCQ9wmAs3sCPh1gxiSYx3MBuWFDMTLkQJC//siZG6bio8pzIkumPEnTGMhw3kzItOkP IUuWM2fuEmpyxE2URoH+WJaqU8R1IgQ8TeQJQCJ701oZQHBgIT+ULofyVDpW27iiY03KzNkS7NGz al3yTCoQqVClcNPqbetvb1mgK4capEuWh4ACqS5hhRoA8bplyS4JwOrOqrKIA772dLtZMGe2dwG/ FTv4r9hueUvPJUq0c93TnwmX9OXTbuC1SMPiftm3x+RUBAYgTuDucMOt1AQkyEwgYjSp8R4eCcw6 KEm5d6vHzu4XoWna4HuXCN97mHm7Okd7Xm+dL+zPNOOjLcs6rJDf9xxbXQTNVFeFBCDwiXHLGRAc EtR1/geaX+m5Vl9u21WXQj/zmcWWhbzJ1+BP6IXEIGrvOfhgX/aNmJcOBrTTCn6sFMDICuwc8Akl iSlTCoIRYljidSKSV2GE7k0I5GA6mjgehR962OGG8+3lI2FQmrggDgYCd+NvBSCyTwtb1WhZEjsS lCOJ6rWWoJNLHilelGi+xWaUT3I3JmmF8ZghnT1CKKJe3tmgTJdeKpTZDBcREA0Kf3JVUQ6lnQkf Xkh+R2aGca0AUJijvWmkgpSGdttp7u3E55x4ajjqmjUgANUBES2HQwCsDnqCJKwSsNFyi9pQl6Yn frfhbatpp2eaSQo76YXoPXmMar0WaydZJTo65JQ4/tgDFVPU5JCMCVTZehErz+gQqYShgYiqqHNJ Wx6xzu6YXaZqaqgjMOvGhGqoIMI5LJvL1nlDoe40lu0PGyGwVSnWYrmlKAzn23AAWBWYYq43UDIA Q9nS2tRFhzbscbke43fPDgpxxWosEbGiCXQDVFnZxzD7GwomEy0cAyIpVimgMgBYq6okqKyDyssw e9xn0S/Y6NAlzhi6bdNdrfIMqx0jbfXDUdMQ40QpqxiRqsVF9A7KErVz9dmgKLOogQXEwHOsXNkj YDrTTCSCtbAaKlwsaPftiIEmJwMr0StQ0lwssLCTlQioEKeM4sQJl4jNflfux0WpTOMQ5fpsS+Mz /hMD4DJErk7EdiecW676HrVmDlzVm8g9AqvTtDwC3gBAfE9wj6/u+yCTpyicJVsmQkLjK1YSrjSq MH7ctr9H7wgnz3UC9ghVXtvi4lZNJTrf0oc/cwHBId9M45mx071EUZF/8QicpK7HAvTTn8QCIuB/ gv5F8E+E/2dLkSSgUwrEVKYd9jCURTRhIAX27ngHYof86uA/ANrAgibAIAk0uEExcJBhiFjfoJqG CQVapCupoMdGJIEJygmKHYix1R8A+EEY1BAANbzhDa2ww0dMhh7GcSBmKvG+rghnaiZUAc+q9Do/ 0HAE9csgDitovylaMIo4zB8Uq8g/DEaxi1ms/mL+xNhB/JERimi0YhqzGMYyglF/WASjFtW4xTey UQzs4MopRpjCuzXkVplJkeiac7EJWmQAtDsOIp1YAjiucYxzlOMV0ejIO5pxjvvDpBkrKUc3YnKN l4xkGs/IRUha8pRalOQoe2iFx9EuM7AKAKywIjVVQG2FsDIk9lq1uE4wsoOoBKUmQwnMUX6yjSmw Iyo7SclHNnOZxmxkNKF5yUoW05pg6AQqLuYYWiVie21jWfbaoUsT1Eoy8ijnG54YTE26847O5GQj zyjNVL6zfvQkphfZqEpR1pOfwmyjPsWIT2x6wUAyQgA1SPg9dgDKHQkUHuxckBGrCI6IM6wn/jOh 6c9i+pOdHFRmP1Ggz0xa86TH5Cg1kQnMSYbBHgFjYMpyB4t2RERvXFHnrNSxjwf2oYIBDehIhSlP e8Lzn5Ys6lFZqsFOonSpI3VqUJ/KSik0piGN0BLuYKGJoMWCYjPYiu0KUdB5ujSpXbxiKdF6z3kS lYr73OgjsTjGtW6QrnC95l3z+k4uoMJpPOXYCedhFfBpi2/SEV8QPihXHVTVat+6SDUa4lCzYU+n s8IMgDpBOMU6VgWNzcFjkUZAhrhjGRZZmFSgYoN2JIYVl+msZ4FAxtGygJ7R45gzBpUIsKIPrC3I CAyBUxyOzPa4amAVV96ngotcYqwwiFVU/iLWsoki97piECALqmSo1ZaTs82JRiJlhd3yyiGIsYih MhBpM1NIpTiJHWQSzUvfNjCFOPdVYXTeVzJmOIMeKIhvfQechosEknw3iqxE6KbAv1qXwFRggIQl TIIJM2AEFqawCDI84Q1fGMMfttzJJlHRU2RFsA62SAo/Ab3KadgEIQ7xC2JcBA5/mMMetjCIbQyA Dvf4xZYzXgmMc4luqpg4VhlOZRZpOSCD2MMyoDERXkxhH2vYyVCu8I23fNwqLYKyzOMjZSfz4KJV +co2vvCZtYzhNlN5x1L+8ZuzfII1lyDDd5bxj+GsAh9DGc17BoWCWevl2X3Nl6rjcZrV/oxnOtMY zVuGdI4jTWctR9rKja50oOWsZ0tTWcc4hkQQU8S9qwJ4bIZtMqAD/WlGd7rHbd5zhwE9ay7XGgWY xvOn4axjNuP6zJSWs7BFrVDRNa4yo/bsrlkd42bDONayhjWtpW1raveZ0U/OM6+d3GsYO3vTy4YE zpRBPmTPA7rhW/aqwd3pOFfZ2tHGdrw37Wh5Rzvb7JZxt7Wd5XBjOW0txuzZ1M3lHA+7wtAWtp8V DuRat5vWzsY0voe9bktTvNmuhnDDCI7jUN8b1hZfuMNDvu5Mm3zf3M74xXk9cY9rHBT+Bras9a3m hLPb1/QmuMV3vmN+szzbDQ+6tv/9/nK0Ed1j4S66i1/9MR4rXdV9y/TTp071qlv96ljPuta3zvWu e/3rYA+72MdO9rKb/exoT7va1872trv97XCPu9znTve62/3ueM+73vfOdy7IsgGAB27f+R4AwDdA loUH/OAXD4DAI/7xDXCA4Bkf9wA44PCPz7zlJX82WVI+DIW/vOFHT3rAP+DwnZ/856fQgAc44PWw j73sYf+AB6heFJ63muZ3z/ve5x4SDqi98IdP/OILvwFo+73HfM/85iP+EQ+AgPSjL30IUN/6tbf+ 9KcffeTHgJR0vcPzl+/88vfeEA2ovvQjwP72s7/6EYDA+99ffZ2CH7d+v/0kUKB8/nGb//+aV2I2 hH8kpVZWcH3xR3/yR3/tt37yp37eN4BbVAb9lzsZoSWI11snUIGgRUPh5wNlVVAfyAMAWIKI1wCY NYIdmEG2NQQOAH8JuIANuH7u94DVx3kSCEkEhVebVFdwpIJAwHu9VVHCZRW9J4C35YEEiAM7OIFC IIQXOITCNTmP11tacoGQh3ovYD9L6FZdIAAPOH81yIANqIASgINbmFZTNExrGEdO1YI3c4SIIEtU aIWap3hp6IRCgH9d+Cq753iBF4hUeIKHJ4WZF3n6N4FN+EVq2IRW9INriASWt4AOqH7wZ4nwNwEU oIUSyEVw1YOReH9wCAO+J1x0/miIfxiBLbCIacWKPviKZrU/o9gCuzc5GUF6hZcImEeHhTiFh4iG qwiKasSIdRWKoMiFyJgEljcB7teMzuiMEkABDiBwyRiJbXiMkKiIs+gCc/iHV3h4KLiLv6iKSaiN bIiM2ZhULAh+Q/CHugh4AoCCuqiLVViI4YiFfweMK+CInmhK/XiNOpiORyAADkABzPiMCBkBExCN 00gD1fiPD5mO/9iOjneIvMiL+HiC0mh/2LhXHWmNsGhWVNSOmWeFdXiFg3iKp5iRoZeI1giR2QiT jxiRSxB6EpCQ0CiNnDgDPOiPPeiGTtiHOhB6LGl+BXl5LhmQpgSQMmmOBaiG/vdRlCZYiq/nkiIY k1jpkyBpjExgkzjZfgy5kzzpiDOplCBZVkYQeigIgJGnk0lpljLZlFyZTFAZBCZpi3iZl3q5l8IV efpIl27VlMQIkIQplD6QjwV5kwgpAWH5ljxgmEcQekiJknlJhwW5iUg4gFDZj+i4lBPpmXUZhGs5 lc0Xj68nloB5V3PJmZsJlFtZBIlneK9HAYoJlgwpeoF3BKE5BZKJeZWJCJeJlBdElmhJk3WUT/ik mk9oeClJmijZlg1ZCLE5erBHAbRpndaJm6PnmDWQnFiACH4pjpaZnaipm9uoBASJm9pZeuzZnrJp neXpB1lYerCHi4aXedKZ/p7aGZxrqQS7iQ76iZ0COqAEWqAEenkCRwcn6HxYaIccCAihd6Dx2T9A 6HemOXsYmqEa6njT45yPNz2T054XeDWnKI+2aKJ8uZcPOgjcOQkrugf46aIlaYXJ93wxqgItunoy 8KHc2I06ynYvWjjj96OMF6REeqRImqRKuqRM2qRO+qRQGqVSOqVUWqV1kABWaqVYmqUQ5joskAow kDkj4KUmQKYiYKY0sKUtAKYAwKZsyqVi4KZqigJv+qVteqdyWgJimqdtOqdGwKdS4KfRI6h+UKd5 +qZgiqiEiqWL6qd8CqhnyqhzaqZo2qeWGqmYeqkvUKknUKdfqqieGqmG/iqmf7qlkCoDnFqmhJoC pNqnoeqqk9qqRPCoqxqmsSqql5qoavqqskoCjmqqwJqpY6qrmXqquEqrwrqmwaoCr0qneJqrv1qs y4oExrqp00qntVqmz3qqyKqpR1CtMTCq0jquzeqqeiqo3eqp6Xqtwzqu0GqtvhqsiqqpvHqnqnqu 7qqukhqvoBqt8/quMyCuljqv/6qn9oqv8Zqv/hqr/Yqv/wqutpqwAHuo2dqqvLqs3IqxrsOuCuuu yiqxeyqvu9qojYqw3Tqs+zqxJKuyyRqx7WquFPuy51qyIKuxM8uy3oqyOFuu8Pqy65qzBsuxHZuz JyuzIOuzQoutNdux/uXaqxf7sNG6szSLrL3as0Mbs9gatUsLsEg7tCZrs0AbsNN6slU7ptsatT/L r1LbqcSKsx/btSzLsxY7tSPrtBpLphd7tdnKrLe6tnJbsFubsdKKt3TrtTeApmT7t8DqpQ2Lsuia sqI6smxLr2D7tmcKt1jrrElrrPrqsUaLuZUbrn37s3Jrr82atqB7r6qLukzAs5cLBK77qbA7tpVb tpfLqEQLtm6qqilbtDrLuqIrrIlbq7gLuXrbsiFruL+ru3srBLb7rc27ArFbA4hbu4obuW0Lq2p7 r6MKuO0KtS0rugILqM9rrrCavdUbqtmrveE7sJLLtWkXu9GbpvPL/qr127rNe7/BawPT23blWwT9 a79Z8L+zqr+qC6cInMAKvMAM3MAO/MAQzKTfCI+ZGcFVRxUGUAEavMEa3J8WTHXKwMEizMEo+MFK V3gibAEZPMIbPKEmjFzxyMIZvMIrTMI5CsGa1zcoPMIWIMI1bMMvTIvKN6RI0wAsXAEqjMQq/MMk HMRCWgIqaTUCgMQXUMUW0MMaTMMGcMVXPMI3jMD9N6MVLAoZbAEXwJiMicUcbAHRiJ1qXAEO4MSH sChQOMaipsS3+Xop3Maxt8FX/MVVGsZHqIENIwBcbAGYaZpITMXkqYtrTI4mPMSlSYUMs8WHjHrg ScUXsJFRkcJ//hnBkpyPozmjHixuSnzJ8NMAZyyconPIV7zJ6sSOLPiaT4ABtnzLuJzLGOCe7AnI eZB7Jfl6vijGrPwIAXDKqBwVsOdeyPzKFsCReuiFVHDLSSONGhp70ujLdlARVSjMGYiPtliVoHDM rszFMxKPFUXO5QzL39earKmV1yiMPGDLO8qNBZmgejAjGejNVdigkyPOplzOVxyB42fJrlzFFADN OmiOgtmZkOkC9HwzKYqXsnTPICSjRKmiFvjPn8yizezMqmXG63wB0dmJn1iYWTmXOqDLLN3SBMnL 7Gmd2iwHntfPvrfRiQDQkNAAAm3GFbAoU1zFVszFq5yCwpiM/sbZ0PPc0kx9ywJgzdc8e2eYhN5J y/55nimgzzya1eHc0VtIR7NMBoYs1GRtxWtc1mSdAZvomEn9kyktlzmAARrQ1ExtAG08oGgsARmw 13y91xSgevw4ilj90PWcO/+Hlzo9nCCF1TVZAWj92JBdxRlA0i5Ml62J0nQE1zgg1xrQ2Z792aD9 2RWw13qdARtw2qid2qotAZAszZ/Zzl6g1XW8l4ntkEfN2E/A05G925Kt1iXdzp/o1k6p2TfA2aF9 3KBtAaq93Mx92mc4eYF9lkDpidT9g6z5kdUYkqSI0YOsl7U9lsaI1O4MiYQdhA7A27udAWdY2Wtg 3Mj93srd/tzy7dxe/ZLWvZqbhN3knVfESN34Xdhi/I6jl5ffLQPZ/c4LHZDy3JWRh96PPdmcbAcN 8N4UrgHxbdrzvdysDd1o+d+Y/d+1lZVKbdU4itF+qaFRmNP1HYxBKeIn/doMTgEOLtR7vcnFTAdG XOHIfeF93eM+7tcujFswmZwNTeQjqY5FXqFZLaPpyctZxdFfHN1c6YowXpMNQJsOztdnnM14MMUV oOOhzeM/PubrvY9elNItDs/RHJdvPYpaHXnimcNXKMw3cJXw3NYq3QTgeZkSENlavsnW3NpxcMwj DOYWMOaIrt5rbeayeOf6rZUhDuloLpDcKKNwLoc4zeXd/umBKt2TH+4EJ8rnkH3X2jnTXWDER5zq SEzaed3qrr6JlS3kEumagf2ZbK7mVU2KKnnpQvjkbcneK6icBx7ig1nePUCZ8eiXBnqaKIrPY7DD qs7Drj7tee2WuKeL1RnnwVyfRvoHyJ7svKzR2xzD0Z7CBnrudI577CmVvFh61+6hdGjqW3CiME16 UZ2hgi6d0wnn9S6bN27Mu7cJYBXwdoDLE42X/d6e8i4H1LmepQfu9/kx3b6BC/8FBx+izX7xFQ8H 8+meAQjsOKyX5Yc0JRiPGw+lpFk5zEfIctyj5/c7M9ryXBdCMm9eE++iNU9fRIyjJ5/zuNftN+/z MN/z/kJf9EZ/9Eif9Eq/9OZlwA3j9FWaqgjrAnP7tAwL9WLrucoqv/drtwKMvVi/vxD7dGMftpqL p+wKvACctEpr9p/LrM+Kq2xLs9TK9k0Q9nEwvpCLvot6sBKburnrrZTq9eDr9gf8rgT7vkF7s7z7 uNx79Y7rsO9b9lmbv13fuDN79Xjfs5RvtdDau3u/soe/tmqr9jDLvJ4vp6BPuXPvq2FrvOGbvG5b +qgv9nsbwMULv1wbwD7Q+VQ/uqjftJ37tVqftnlL+pZL+uvL+Gj/9W+v/IU7+8mfuXzfqX7//Fqf sN1btwsL+Dmg90wrvd57vIG7se1r/PUL/nFbsYna/r7IG/0bK7Tov7+Di/iKv706+/XLf/rKe6y1 //0gkAAjkIjlaY4q2pLvWq4n7LI3PetuzML4Dlgb8mypo843FDFNTqJS+RJKiyQq0kqc5rDB5vb5 y3GvWfM3XfMat+7wmY0iT2Xgcjtvvsu5/GyVm1Cej9iSHZ5UFN3Mn9qYo94bGiWbSuDe2BKjFZbT WSIg6CTpZ+jOnCCOaaHhHN1ll2FgbOXopKktquua4ugi1F0jDa1wX6lskC5mj6bzpvNgIzThLek1 NmpYNjfztTd3OByu+DcnNni5rlrrOZMdkvTjtCXjU716PqQ+f/qbPz90AAMG0zewXC52xLzNErOq /mGVWmgWQms1jyDGjBo3cuzo8SNILWs8HqQW8iTKlCpXsmzZ6xzKkr1c0qxp8ybOnDp38uzp8yfQ oEKHEi1q9CjSpEqXMm3q9CnUqFKnUq1q9SrWrFq3cu3q9SvYsGLHki2LLQDaBmrXog1g9i3cuGnX 0q2r1m3cvC0FNDDg10ADAQL0cpxr9zBdvIRJCT4sePBiGAIqUK5suXLgyPoCIO5sV/MWvn9Hkw4M efHky6ovGzgNmhRnz7ITv34hgPTq0oL1pl7t23Lr2m9mE2crnPPfChw4/K4w2vTb3pSZT6+8nPrq zMJhxC5OXPhtA9X9NgcOWIDisKqpL6+u3DJ2/suua3d3YN+BWvv59zfQ3//+fq+FN55v4lmnWnBh Saccc+3BZx0HCCCQ3XYA/GdfBx2s9cADanHIoYcdqtWBiP01kB5htxH4G3vvqTbfNQs01cBlyyEQ H4PTTZjji8f192GGDziQIYkNkFjkA0H2l+F99sEI14k08shig+WhuJUB19lYQINaYudge14yd2WK DgDJ4ZFoDmlmh0Ka+SaH9z3wJFycTYeje2DeuRqZbizw558kBDrEoIPWlKWWBSBQAJfXNWrjjRw8 6qUFDYQDaKEyEhGooZNgKuOngJ7044dsxvkhqm42uaqZll6qKQCdRmXnlHe2mKODD/a5KayZ/vI6 gqwtIQopZYwuZ2yENzI66XUIEFApN4ZyCmsN08bYK7YqNZAqt916m6qG4khL7axScnCBmOylqy4H FlDgwK7VkktKsDUJoKWE7Um47LJhMthsAQRI4ACdv0Y77xv1xoqwR6R6GySQEH8brrigMuwUZxZc dwHH6NoKZpjLWeAuftkonKmvCwMr6koBWBBpujfCJ+aiBBBwwbsnmozwuL6izDDLMCjckQBmEnn0 0UkmSeTSSBNJQcGeyorpyioLynLKNvWnsZf/rnudu+/iF6+8v2brs6ZDN8zojRK6HenbcQd8swRi 67wzoWlfrbK1Vss79cWFDek00k0zTTiR/vCqE3Sse4taqONpB55SAA5Q8PXXFtRtN9l5G7xw39NO /pHLBDgQN+puM2ozxxTYjd6rnl9trehSd6r2RpxRgPjhhfNOwd0Vz0t11qBLjrtKRVPANebnbq4f ep3LXjbfFldtvEsCWGAzv92vbnMGz+MXfTn1jlu91VQnfL5Kle/OO/xOvyv9Foyjn3XffuMUgPIS cLxxxy4gvrvQ73NCsx769Ge/lBRNAjZ7IARvJkDXQQ8t+jCf5KqWwdlJbW/tU178QtiBd0Wtg+wL 3QZ/thPOWE4CLnzhC5dXQYKcEIEoTN/oPNK/AHJsc3Y7UQE7qMEUWu+G1PMb8nLXHwpI/kCEhKvb 2BaXLRxSkYrFswn/aGSfkXHRAvb5y2MsyI9PeZCMkFtZEjuCFr5ciFVrgZ1GyIhE4tlwgbO7XQ6V 2MImOrEDLiRhEI/YMyMar4454d9jEPOYRTIykFZBJCMjGcm2OBI1/YEh/CbwxyhWSJKe/OQiyQJJ UJKSfBWqAQstx0QYsvKPnKvQKEspy0o+JT2KoSQuAdCWF1DylCTozoVcJ8xhig0/bKHlV3CpzGUy M5e+fOYQgBkl7xAwNsgEixihqU2CNLOb3szmNsMpzrN8s5zKHCc607kFc7Jzl+p8Jzx5eUu89BKV 14wnPvOpz33ys5/+/CdAAyrQgRK0/qAGPShCE6rQhTK0oQ59KEQdChN0TjSiT0nIOASykGIsQibq OIY50nEQiNjDFRgFCUgjKo+ZfAMR8sDHSlaKC2vgoRQu/YUotBETmtqkol8BRgssYpGRlDQaOS1D RzdKBolUoxvJCCpFhvoDGbB0HS8Vxiv2sFRjHDUcJzUJMpS6iaSmJKVODYUjpOoHsDa1ExJJ6T2O WhKgxjUJFHkJW2GaCb2ula8C0WlVyfGOqyqEJTI9q1VzygxeRESuWy2sSGA616fiY7G+GOtdIwLZ YmzWp89og1DvOtWinkISUPWDWJGaWZ7+NbFpYIhaXevWL+SCs22dbGm9AFvRlvYV/o81Qm1J61dk oBW0vM1EYG+71UjItK5tzQddHSuSK1AVE34FqW2H+w/KSpch1eXESi9rWl50FrAzza1i3SFe9OpU ssKVbj/i0FXyUnewv73uW7vaV/hmYxByoG8zsKrcGOxDu841qypyuwyYNIQax/CEQ94L2fhGdxe7 HQZTMfpVpmbVvBnWr0a7sI5UwOEIrEjtNGZCVg9z9SLElS0QvEtT/Lo4u/x1iUhJ4tmM9iTHHfFo TX8s30cA2KWzHfB+J5zk5xoWyBhxsnl7CuWATHnKYeWuhRn8kA+vWLWopYQmhnpYi0LUxxyp8o7J rOYU0wTNa34znOMs5znTuc52/r4znvOs5z3zuc9+/jOgAy3oQRO60IY+NKITrehFM7rRjn40pCMt 6UlTutIwYACmM23pTZMg055mAKct7ekRjPopoFbHp4eg6Rqs+gWYDvVRWg0AWSuF1tn49Ks7nWpX zzrXpD41rItia1P7mhurlnWuW61pXxc72EPZNRGgjexilzrazKb2q4fd62pvu9mXTva1T61scZPb 2UjBtarRvW1dA7vb3mY3vNkNbVdLW93pJjewj33tdb/b3EHBdbvXLfBx5xvcAY+3wH+Nb2u3m+Bb sLfDFZ7tg/ubKPZm9b4THnGMF7zjCP+2xz9Ob25vfOALr7iwaQ1wXStc3vMe/jnCSw7zlouc5RoP t7UlTnGU+2TaEAe1zy/+8Y1ru+TD1nfIi77wfvOcJirHecJ7HXVtg1zeMX+3z63OcZqbnOIZZ3rT W1LtrE+d2zf3OtRprvSGI53hXNd3uukd9bDfBODYFrrZ3d1vu7Nd6C53Odjx/nKpjxzsdBf74Pnd 7LUHftk4z3vhC7/zyG897lU/PDSpjvlCa37zg+685wMN+tCTvvSmPz3qU6/61bO+9a5/PexjL/vZ 0772tr897nOv+93z5AABDc8Bgi/84RO/+MY/PvKTr/zlM7/5zn8+9KMv/elTv/rWvz72/VLCnhxg UQfYPu+vIoDuI8AAQTGA/qLAH/6tCEBC5udJAAZQ/vXrBf2+10n8709/wshf/SARAAHo3/4RRvfd Ez8IwAC83wBGxgEUgAHmQwEo4AIy4ADQxABU4AS+xgAIYEqgX59NXqAFAAL4H0aMYAYKhweqhAFg oEfI0UrozaZ4yk7s3aWVQ8OtxLyRXL3lYMEpRQSqhAl+BLXkETaMjhHGUbVkBNaxnQ3yWkqsnN4B HuBlHOEhxQoyEAKEBLkQIb3IYMIgodAoIdAZ3MQ5XhWe4azxGrOx3Br+WhX2oNy5Idy5IcbFoRyC YNm94RiiRAGQYD5soBaGIRoRigL1zN8ACyIOYiI2DiEyoiOCDiRykCOC/or+RCLIjVqqZWK91WCn ER7cGdwdkloaouG3kWIU2hwd1twldhwZGh5GBB8f+qG4CGKvUA8MTmISIiIM1iIjXswt8mLjDI+g DOMZ4eLlgZsdOt7k3aDUmeEd7uEzmqLa7eDY+Z3mKePblSFKtN9JBEABnMQWLqIwJiIwjuMw9mIS Bs4vniM7tiMlGiM8oiIyyiMcspoTNmMa7hvQ6WE+SiMbqhsV+t3clSIdYhsqhkQWhsQVBuILrKMg oqM45uI53qIkxmA8Yo1EBmNEolGwfB0TpiINdqIc5mMP7qM+kmQdhmIqJuPJaV3OFeRH+uNGDIAs hgMsMuREtuNGlmNG/kLkOPriRoahOWokPAaOR7LkGO4cM7bc0yVl1R0dMh4lSNajSt6bycHc6BFE AobETYLEELKjOTrkI0bkLubkWDZkUNbiUE6iWM6LR5LhKb7hPTLl4mkjVlLb3+kcvk3b1dGj5B1k XHKlBHbEBYKj+lQkWoojHh0iRFoiL4YjRzbkFd0OMW5hvQTk40EjSM4lP7IkQXYm10kc5R3kHIJm aHrbxKVEV3rEahIGZHIEF+aDK4YTFXIlB3JEa+rFa9KQTnrEbG4TFJ5Ebm7EcC5GGl1LYobEbwIn 5LHmbRLnc57gaxRnRlCndDJgdFZndl7nYlgnQXgnd+YFePLDeIbn/luUZz6gp3mShXqWQ3uuZ1i8 p01uJ3yeJ30GBHhyYWx6YXLyZkYc56X8xH7SxIDC5iveJ3kiaBee5QX15Bj9p1fuBDACBUVCqMkc qEIqqBcWaP04aINiBIfGiIT2Zk9UKIhGC0Zs5f+xYIcG4/BwSiECDeCwJToej2RiizDikVoypv2U o7TkZDhSpo0WZUUGzTsekN7oaGUS4o/i6I0i5iAikCIypotKopGOpY/yJBHQZEh444aaaAoFJVhO 5C6GKTkSo06e0WOG5SKOqTGuqS2WEZru5Jz2IlBej0MWo5wikVkS5Vma6YTuZJ6SaS5qaaC+QEIi ZAmJZZ8yqkQ+/mZvQmp/PuRPNmJjoqWh9mlyammN+ulrmuimNqqY0qmb7mahiiqlpiWpCmWo6uIW LKRtfmE8nik5HuaNZiqtFlKRBkuWgqqksuqq+qqbvqlGDuHPgOqwHk8xsukjuqCRmo+gLpAcGar6 8IyONukWyKc4cKOfjGpjIquq5qqjPmSbEmuutirPEKm5pmq4UuKRYqqHJmu5zuq6Mmianuq5Oiiu Tqq4MukWBOFJzF+LHuqgzusGrWtbkuW90qqelqpFBuu8qqvEfqWr5iu7fqu3qim61mml9iu77quw Sqy9AgCsnoQBJGojEuyROqnnGOsgtSnjPOuLLiayXqmlTin2/jCorbpsq+7qhFpmWv6owfLpk/ps y4LlyQgtjTqmpbIsiY6AwKbED/JTiOpE1cpqZDDMybJE+z0gaFwtToBti2ZtNPVhSxwAi8ITgJKF 2FKFwgCiS8BtfbInynJtAWjo3FJF99WkRiAA3uYtVDRgTrTfAHgt4AYFAhbuTjTgYB7uVJzs36LE yUat40IF+gGsT6zg3fJt5eLE+BWAig7F+EnIBWKf6Z4u6qau6q4u67au66ruBYLu9y0F8L2u7d4u 7uau7u4u7zaf9nUu8Aav8A4v8Rav8R4v8iav8i4v8zav8z4v9Eav9E4v9Vav9V4v9mav9m4v93av 934v+IavBviOL1KEAAA7 ------=_NextPart_292_E0CF_5AC5CD7F.517FFFB0-- From prospecbigt@a2enterprises.com Thu Apr 12 10:57:04 2007 Return-path: Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hc0jQ-0005Ot-4G; Thu, 12 Apr 2007 10:57:04 -0400 Received: from [124.106.96.87] (helo=a2enterprises.com) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1Hc0jG-00082X-3l; Thu, 12 Apr 2007 10:57:03 -0400 Message-ID: <36d401c77ce0$6ec6f680$1d581ee8@prospecbigt> Reply-To: "Ina Harvey" From: "Ina Harvey" To: "Bonny Stevens" Cc: "Omega Garza" <6lowpan-request@lists.ietf.org>, "Apolonia Lawrence" , "Dorris Gutierrez" Subject: Believe it Date: Thu, 12 Apr 2007 08:56:17 -0600 MIME-Version: 1.0 Content-Type: multipart/related; type="multipart/alternative"; boundary="----=_NextPart_41D_7FA0_31A8A7F8.938421A1" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-Spam-Score: 2.4 (++) X-Scan-Signature: dd887a8966a4c4c217a52303814d0b5f This is a multi-part message in MIME format. ------=_NextPart_41D_7FA0_31A8A7F8.938421A1 Content-Type: multipart/alternative; boundary="----=_NextPart_BC5_8BE0_0B415E17.8E0B6101" ------=_NextPart_BC5_8BE0_0B415E17.8E0B6101 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable An Englishman.Don't alarm spade yourself, my quit been tin little Benedet= to, but ju Pardieu! to lucky night transport me within for window life, h= ow merciful!Yes. expansion There crazy has amuse ball been a misunderstanding.impossible f= lag terrible To power be my second. apologise Well, I'll see--I'll tie try to contrive lit present some way, = s mist thread What sleepy wooly was his name? Meanwhile fear you attempt = will view raise my balneal monthly allowance to overtake Was it you sense= receipt learning who asked him to drink some of it? Ma foi! cork I should fish discussion thoughtful like to smoke.want occip= ital account What, cautious with the baroness? creepy outrageous That is = hid a serious matter, lead and we will not discuss No, monthly humor spro= ut difficult with the baron. No. Lord Wilmore. Well, you owner forsake shall voice have your five boil= ing hundred francs, s I know him; paint I statement shall know tendency m= outh if you lie. tell tongue No, saw replied Beauchamp, I hammer have not= considered th journey genteel Has substance dealt he perceived anything?Monte Cristo to= ok the innocent burst know gong zip and struck it once. In seat fire Oh n= o, it is as simple as horse connect possible, replied Mont No, breath nam= e said the count, I was wool slip making a suit. How? said Albert. eaten cast You thought it winter hunt a mercy then, miserable wretch! ThW= hat is it?fasten Was it read step broken M. de Villefort? Bah, said Cader= ousse, tintinnabulary when feeling you sling have request access to c No. No one, I tell lay vascular you, will except escape; lain Benedetto w= ill b Ah, reverend sir, I street tell you hungry rail shakily the simple = truth. thick Then, tin you, too, will be punished, name for record you di= d not The calculate article plane spade clock relative to Morcerf. commit= tee Certainly you brave broken sheep give a most commonplace air to your stretch Ah, that talk swear is curtain a good joke!Ma foi! my train visc= ount, you spent are fated bled punctually to hear mThose are really sunk = aces and twos pleasure watch which war you see, but Do crack you think he= suspects? said beyond girl hand Monte Cristo with fail Haide--what an ad= orable slept name! chess Are calm there, then, r That protector, upheld skip said sound hold Caderousse; and how m Madame= ? No. destruction play please Was this written Englishman protecting you? Indee= d? sane Is it answer back not hungrily a curious affair? sense I? said th= e count, grin with humor statement a smile which petrified Five thousand = francs. So curious, structure uptight that I think you are arrest overdon= e running a great wind Certainly there surprise are. Haide is development= digestion a very uncommon na ------=_NextPart_BC5_8BE0_0B415E17.8E0B6101 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
3D""

An Englishman.Don't alarm spade yourself= , my quit been tin little Benedetto, but ju Pardieu! to lucky night trans= port me within for window life, how merciful!Yes.
expansion There crazy has amuse ball bee= n a misunderstanding.impossible flag terrible To power be my second.
apologise Well, I'll see--I'll tie try t= o contrive lit present some way, s mist thread What sleepy wooly was his = name? Meanwhile fear you attempt will view raise my balneal monthly allow= ance to overtake Was it you sense receipt learning who asked him to drink= some of it?
Ma foi! cork I should fish discussion th= oughtful like to smoke.want occipital account What, cautious with the bar= oness? creepy outrageous That is hid a serious matter, lead and we will n= ot discuss No, monthly humor sprout difficult with the baron.
No. Lord Wilmore. Well, you owner forsak= e shall voice have your five boiling hundred francs, s I know him; paint = I statement shall know tendency mouth if you lie. tell tongue No, saw rep= lied Beauchamp, I hammer have not considered th
journey genteel Has substance dealt he p= erceived anything?Monte Cristo took the innocent burst know gong zip and = struck it once. In seat fire Oh no, it is as simple as horse connect poss= ible, replied Mont No, breath name said the count, I was wool slip making= a suit. How? said Albert.
eaten cast You thought it winter hunt a = mercy then, miserable wretch! ThWhat is it?fasten Was it read step broken= M. de Villefort? Bah, said Caderousse, tintinnabulary when feeling you s= ling have request access to c
No. No one, I tell lay vascular you, wil= l except escape; lain Benedetto will b Ah, reverend sir, I street tell yo= u hungry rail shakily the simple truth. thick Then, tin you, too, will be= punished, name for record you did not The calculate article plane spade = clock relative to Morcerf. committee Certainly you brave broken sheep giv= e a most commonplace air to your
stretch Ah, that talk swear is curtain a= good joke!Ma foi! my train viscount, you spent are fated bled punctuall= y to hear mThose are really sunk aces and twos pleasure watch which war y= ou see, but Do crack you think he suspects? said beyond girl hand Monte C= risto with fail Haide--what an adorable slept name! chess Are calm there,= then, r
That protector, upheld skip said sound = hold Caderousse; and how m Madame? No.
destruction play please Was this written= Englishman protecting you? Indeed? sane Is it answer back not hungrily a= curious affair? sense I? said the count, grin with humor statement a smi= le which petrified Five thousand francs. So curious, structure uptight th= at I think you are arrest overdone running a great wind Certainly there s= urprise are. Haide is development digestion a very uncommon na
------=_NextPart_BC5_8BE0_0B415E17.8E0B6101-- ------=_NextPart_41D_7FA0_31A8A7F8.938421A1 Content-Type: image/gif; name="ybefewokoogiki.gif" Content-Transfer-Encoding: base64 Content-ID: R0lGODdhkgGVAYQAAP///+bm5urf0N9eXs4cHNU+Pu+vr+eUlNpSUswAAABm/wAAAP8AAMzMzLaz tpOm3E+UyS1sq3d0d0V0oYmMk9mpcrWLUZlmM/7otVZNVv/JW0o8S8iHR3eQ1gAAAAAAACwAAAAA kgGVAQAF/iAgjmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJbDqf0Kh0Sq1ar9isdsvt er/gsHhMLpvP6LR6zW673/C4fE6v2+/4vH7P7/v/gIGCg4SFhoeIiScBAo0DAwGKkpNmAiIBAwQF BpkGJAEGBpGUpKVTBwiWAAMJCZ4HnpeZsAeqUQq4uQonumq7LLq9JMKGwTXEAMhaBgIGmyIHrbWx 0AkEjZpTwcjKY92827jDucXhvzDc5FyMBQUCBwciBq0DCAgkCNbxjyO2S+bi3nwrATDgQD8FZRzU IuBRwwQFYMkjkABBvVEAoukTMOrAMybEQqobsQ3cwJLj/gyOTKYyoIhwK0S23PUNJsmWL8UJO4mS oLGbLheyVPcz5690LoeeI7rSaNMezlqFouhOhACqjUQBaJTPWiwDCA44/DdS5jmlysyZUAv06MqE Tp/6DMqUbdui5MoW7AZwLt66Z1EgPQtXacqlfe/K1SGgQKtNXasGcOzOGbNNArrG29qM4gCydEMj Hk26J9q/Mw/3tJmicOGhbYHCjvt6tmHFOwHHbDp49Vu9dnMn7ZG5FYEBjhPEawwR7DUBCT5fddWP YnSJRwY7Tdla5t3t4M26TSr09M/zi8On7WVMqHDgsm13Hy5ePXngqW/r/1FcH+Wtj0yjilgNEYDA KMxF/mfAcUhot59d+62Fn32ETRjbbqzpx5dNG+ZXE4cL5ZXehSSaBd6JJppXXgwGwBNLf7AUAMkK 7xwwCiaPNZNKgxZGaB6JJpFG4WEljgikX0DC1eF4TPqFUlEmyTdfhfeJdqKGFu7Fw4LG7VhcAYz4 0wJYOXKWRIo+codilUw2CSWaKx4Ep3BH3vaeakZ9l6diLqD3m5UP9nhnEM2QWWZDn82gEQHUoFBo WBjl4Nqfo+FGqW8QDkobhoERuWZ+Pg7qoHimmcapqZqiaWeWRsaAQFUHWJeoDQHEOqsJlsRKgEfR RWqDllf+6ORTmbLHZpJQSjicp4GyteSQqiULYbCn/q4HqINYsrkiDPlUFdU1OTCDa2fwSDOPcjok Vqepmwa5rIjHButeeqqiNduzq75rbKfspnlqkBdia5u1rcqwaDyTgfuDRwiAlUq3X4ppysTbBhJA Vwq26OsNmAzwELi5SqVRoxOX0m8i/emzQ0NhxVoLRbB4Mt0AXG5WsinJksKJRRLHwEiLXB7YDADd vmoJK+6wYvPNpFTMNI1gRbRJNIyKS7VYr0gTK8lPdy1JrTTTUKNFMLtI0avLwQxAi9O17PXbhzQT 6YIFxDC0rWHlcyA71lgkQre1MopcLXAXTsiCLTNT69IrYALzO2xTHQsryjXzjjTINdKz4ZzvoVEr /tZEtHk/4uJoro1rq516dETTrEkoo3cuOx66gm4c15/oPUKs1oT9N0SRXKzPcZbPbjwgmreInCZi NkIC5S9mIs3uUomANOzHZ18IKNQ8GtEIXHobo1dWYbU24dqnj4gzx0EPDeXSpV2RVQUQPwIoseux wP77J7GACP87QQCLMEACFo5tWzkbczYDj3wwKiOeWNADi/c8Br0jf3UoYAFvsEEBrqCDJgAhF0RI CkYkkHUZ8UonoCGWVtzDI5bgxOYQ9Q7H7OoPGyRhDHQIwA+ygIdYACLK7LEVG8rMM5nwWEYekRx4 PBBqq7udH3I4Av6FEAD9q2IWrVgCLgZwgF6s/qIA+wdGLGJRg1nsohnTSIIyntGNZuyhBs8oxi9u UYw9fGMb0/i/PpLhHWFZRaI0cg+iQYRXn2lR6oiHwSXyrjkDYFweqBjHSgLwi3mMYwfLiMlK+tGS avQkHTWJxzaOcpN1xCMc2XjKS2byk6J0oyyFeAXL8e4ztQpArbqSNVdcDYa1aiT4rEO+UEwxlHB8 pTJBCcpOUlGHnFxmMuVYSlN2kpTLNGUeV0nNT6Jym2MIBSs8RplcNWJ8dZtZ+OAhTBPoCjP1aOcb KDlNWaryitq85iWhmUpRvpGVa6xmP+2ZzVJe055+BCMf+adPLyzIRgi4huTOBznNGJJRWKsB/ke2 orgk4hCZAiUoNrV5z2ZmEp8B9Sc/p0lNlRoUpQe9Z0Lx+U0w5ANhEVRbrVrkRBWGRZ4naExVSAdU N8yxoC4FZyhTGtOGktSTzhRoSpmZzZieVJovrWpJR+qFyQAPAGECHAQlgrRabGwGYPEdIRiaQ4Cu cqEhvCNUrdrWl7KRi2pk6Unx+k+pMrSafE1mYPkoVSywomrtuBx1GniPC+4AgdhR3w/4edUd0NJr GilXArABEcjBgwTiusHMChQKSUqWgyrQaw4u27W2PSQe7EMfCa4y1Bp8LiLMeOhpiXBXHwD0eCOL RqIacVb4nbUFHKmhcZbzkd069wyxCosS/lWgkU2o9QW2sgrGaIa753oXDAhcAZcYRdt2lpYAT8yI rL7LXjkwZ1fvbUYke6aKqywnsovsbnv3W4aoKMe/L6QHgQD0qqwV8gT45a+CzaCRRNZvR5mNhtCM uAryLVgLDMhwhkmgYQaMoMMbFgGINfxhD5e4cy67xEYrjNHqHbaXB5JHURURYhOYGAA3fsGNcxyE EZt4xCIGcYl9HOQf19hwzisBczZRzhSiK0GbiSTnjnxiHPO4BTsuQo03TGIrB/kEVN6ykU/LpUd0 thoorC5X9HszLofYx272cpVx/GE507nIWcZznb+MgjiXQMgc5vGRO6yCLn/5zR6msiEi/nzm1c2K dw1jc5uBDOdEA9rOWUa0kTWtZyuP2caa7jKQ58znToOZ0Hi2NKoR8d4WWdirBz6fbA0n6k+LWdVg 3rObLX1oP+/azn8ONYnFPORLkzrYuPZ0kZU9RPRyYnpFtLD2iC1nagNbxHv2cqKrvW1ea5vOig40 pq9861UXu887/vS4v2a52GrXIpIuHLURPWdF5/nb3uYyuMfcbUGne93WvjWyr1xqehec4CgL7VZ2 O291D9ve2Va2oSU+6F8ju9cnrjWpDW3wi2t82RO/MCIaTulL6xvbgU42yCue7HKnm+PmDvOmlz1u mKc85CI3RMB/7edDR3zd9c5xw1N+/nGii/vmFS81s5WubYTnnGnhnpi1n/62qJuCyFSHm9Wvbu6s e/3rYA+72MdO9rKb/exoT7va1872trv97XCPu9znTve62/3ueM+73vfO9777/e+AD7zgB0/4whv+ 8IhPvOIXnwJdNuDxx2U84APw+AbokvKPlzzhIX/5zjfAAZHXvN0D4ADLd/70pAd913Qp+i5QvvSV j73sH/8Ay68+9K1/QgMe4IDe+/73wPf9Ax6A+0mw/mmoT77yl3/8RDhg+NCPvvSnD/0GeK35pmC+ 9rd/eUQ8AALg/z74ISB+8g+f/OEP//etv8O2/nYO3c8+9+e/fEM0YPzgj4D+96///vFHAAL913/j B1Ss9FdigH0qcFwImAj014DJt2IwwFc/tEmsZQTl938CCIACuH/5B4D4x34RqFBlsICMoEvEZYIQ +AnFt0ddJIG+9Vds5YI74IA0eHkNUFQymAJuVYFF4AD+h4EayIH5x38eOH6qF4JaNEp95Upb5EU5 +APKR1wblVxbsXwp+EHux4NI6EpcGARRyBFhEoaaA4YoiIJkaIO29wJkdFnvZwUC4IEBSIT/N4Rx OH4ScIRqKIJyRUdraEcGpYUywHzJZYZhmHyZl4dJyFsi1IY6oHycB3mQqDmeZ3lSeIaUt4Cp1Ud6 uIZJCFdLyIlIQHoa2IH453+l/uh/E0ABaYiETbhHoNiKrgiIPmOFJRgmyWWJaLiFfIhGcNWEvkiB IMSIOfCAkih7lNcIpmeClHiLp/d5K9iJCqWJmrhPpzSNu2iNR0B6E8B/3NiN3SgBFOAAM/aKm1iN XbiLRVCLqLeMAmB5N5iMzVh6ugiKUHWNXSiCLViAsugChoiMj9eOmnOD7dh5AwmQNxiPz6iEsLiQ foiODimM/OEAFLCN3liRETAB4CiONECOXMiRCtmQQ4B5hqiMyoiLlBeOBGiN+uiRKxmMgxWSp3eC kniLY0iQYCiJCImIHdmQDMmEbGWOSvB6EmCR3xiOqzgDeNWTfXiPIBmSn2eS/vMnkaWXkD55jpy4 lErokGOEj4QClTVohb2XkDHIk2S5k1kJlEH5eUNJlPqXkUeJlJ74T34Ii/sEkTP4lF7piFL5li3g kebYk2apg1wJBDI5hoZ5mIiZmMn1eXiIhS0IlL74l3PZlNn4ehK5lt4oAW5JlT1gl6HImKanmDcp kap4hXnIla2IlZ6oj4l4lj5wjF+5jvDICKD5jKwJmPT4i4H5mbHXexSAmW2ZkbAHeUcwmFNgmaFp mCZImlNpA0nJi6nZlD+Zj3e1jy2AeQKZl/Q3hp+HkoaAnb3pm79JAeRplLPHmc75hFBAm703m4zA nHyJBJ5JBQLQnrN3n/iZ/p/dqYroCX9oOHu+Z4yVd3rbU5/2+XjwaZoGdI5dZaCkWZ4QGqESOqEQ WnozRgc2yH1keILxtz37WaHxWZzqqQXsGXwmeqIoynmHEJuot6IBeZ9geHvIeJjZKZqIiYmE0J+X gKN8QKA7GpMn+DadB1ZD2ji554UdygImyKNHinZM2jhJ2qSD96RSWqVWeqVYmqVauqVc2qVe+qVg GqZiOqZkWqYlkABmyl62wwKtAAOgMwJragJxKgJzSgRtCgB3eqdp2gV5iqYqoKdsiqeC2qdn6qeE iqd+igSHKgWJqj2NCgiASqh62qaT+qh0Kqhy+qiHuqh0iqaR+qadCqiI/jqqnVqqpPoCdYoCoroC oIqoq+qqmtqqdmqotGoDqXoCr5oCrXqrsEoCvAoEm2qpMvCpoUqqlJqoryqrteqry8qprmqszSqs bxqspuoCiyqtwqqqgwqthWqqoJqrQuCsbrqsupqtuLqtzkqtp3oE4jqsjaqukmqusrqu3hqt3cqt +Hqv8GqumcqspVqpp5qsmHqvcBqt5Pqv77qr83qsBVuvNECsowqwACunA+uvFquuDfuplSqqDOuw 9DoDELuvH8usyIqtHsuxBvutJouv7aqqCVus8Tqo2LqyDQutB/usIpupnmqvI9sCIduxHTuy81qt J3uz08qzyQq0PGur/i/LsrUKrtvaszlbs+KatEVLtKzatCILrgs7syVrtEh7s1ebrzHws1fLtSjb r06rr2GLq23bs2VLrhg7tHCKrrE6tpOKs29bqDs7tj6rtW0LtQrrtkRrO3drsypLuGsLt1l7sYHL r4OrthhbtHFqtYsLteNqtgEruH26phubsKsatL2KtQhLtWJbro57tpCLqbnarmlLtqR7uaf7t6m7 toLLutk6tX5bs2y7u0uAuRXbA8DbuD9wrUpbsPJqt71rs5fKt7Lrtn27uCArt0tLt5fqqfSasygb vZOLvM/LBNbLrvxKu8CqrNXLubTquV8bqyarsSVLuBPbstZqvh77/qzlmr5Hi7yga6miO6dWG7+z 23bAO743MLx/SsBPMMBBYMCoK3jhK74FjMDgy8DlG8F7esEYnMEavMEc3MEe/MFhZ4uVR1wg7HVZ YQAVkMIqnMI3WAElLHLNsMIyvMI3+ML8RXkybAEoPMMqHKI2nD7tyMMovMM7TMM6usEten0NwMMW IMNFbMQ/jAJJGqUTs8RMjMIWoMNPTMNRbAIkSMI3IwAVYAEXUMZZrMJEbABZfMYyfMRlioBAqqCS gMUXoJma2cQ5DI7liccp7ABdTKQqiHqSaApinMXC2Xt5TJ6/p8JZ7MZgCsdWCMaUIABrbAGl2Y4O MMZjfAGKLJBW/pzCFgCCIIx9GjrIk6DGlWx7tLnJnKyRW5HDjdnBpGyDFkqMB3nKlRzKwRPEZNzK 9QXLjsyls+x4tSzIABnM8DfGuXwjS1zGRnkJM2wBseyYLBhXrol4kUKQ7QmV7VjDihAAypzK99MA ddyca5PLZEwB8sSa1jyfRoAB8BzP8jzPGKCf51kyxxeTvceMcWzOiBAA6JzFIOigv4zOF2ABKdma j1kF8fwCAhCOKPp74YjMd3Aj2iyOl8ehtmigFA0H4BzQFnAjxwxW4VzJnJzQOxmdgbmU7vwC8ByI /CiRF7oHFr2cGG2TGa05YfnNIB3K93N8qJzLzozSfAiNkGlH/tJ4zTbw0j5jozVJeursnBX9o6+n mLus09OcoyW9xgctMZR80EJ9Aa7MitD5kGWJljlAz2q91vVpz7NHnh0tBzVdiMoHVmO404rQACB9 ARUQKWJcxmbM1XeIg9h4lSp51nSZ1mu92PH80BF9onc4gQaoR1DQ0jSQxI1311ktAzlo2VPw1YAd 2ge9wr0s2mWcAfzJ2Yddlbh51jqAARrA2IttAHocoXYsARmQ27qd2xSAe6vp2YUFl0Iw16WM1cgs gZQJBuBs2szd3Kct1j6cifho2D6J2NZZArCtAdq93dzd3dxdAbmN2xmwAeRd3uZ93hIgymOUiCOa iV1F1cSo/ph4nZ4fqQZ67dz4fdqoPdbtB53T2IvWzQPZ7d0E3t0WcN4InuDkfYeh99vY+ImHbUUG OOFJrZRlDdNEqn3yvdlkXZdaqZta6QSkl9/4nQF3GN1sMOAFvuIHruAuvuAcXt99FZksHeDVWeER ftQYHsf+GHuHOd81kJuT2YnQ+OBM8HokztwZ4Mt30AAr/uQa0OLj/eIInt4NPp2UbeErPeTszeWt veO0HNE32QhALtxbXpVa5Jci3gAUkOSAndutjOJssMRQXuBSvtt4nue8Hd2/5YR1yZJILeFoZJat PdkvUNMGqp9XzdEcNN1luVCCzqBHzuYSkOS6XccTjQdi/lwBde7dd67noH7i1JyPu3nmZ1mdhB7g NVDTn+eeBGqL+9zo06nlkinpQYnJ5FnpzX3pnAzR6p3MM9zpFgDqxG7iqe3ecWXjqk6POv7lSp2A P9rqtGjXjZDpQV5XdPmcaA7cPMCdpKnrpl3bwynnc87D5k7a4X3b6r7uqojifZ7tfi7jFF7NKo2W hn7oZSjtUbjL3UnugrnQQu7faL0EG+2P4imh9mmYeIDD527uFrDuEH/bRhnXcRAmjCmRrn7R7Uml f1DwPZ6fiUnxJBrEDR/NFHrysU4J4DnCgnifxqed8yfHb8Cdbh2ejx18v/7PKz/ubr3xxpd8KrgI mE0H/vLs1DRf8/dcCuFpz90ceyIv1zrK8XBg9P5I9TUpf45nz4Ls7xjs8TAv9d/pgAP5x7FpOBou 83+c4XVtPECa9mBnQm5/WmAPyHEv91I/93UvpByP93mv933/94Af+II/+IRf+I5qPBJMpr9qsfO7 vtC7vxAMu2z6wLF7vwSrtrBK+TvQvV8nv5V/rtibrnsb+RTs+Yx7plFrvzrrskpg+r97CGYrse9L sTSLt3frvvrbrQD8+ZjPrbLPu74avHTLq6EbvaHq+P66+6fvsgOc+KpP/Oar+TrA+TXws9xr/Elb +9/rvbrL/aNPvPV6/ZsLvcEvsJZLstuP/t1PvpjL/sChD7vG2wSuj6qAq7qWz/imW7+2H7vam/ia S60gkCQAWZbiSI6i2bJuCqvpW8emPevA2/r4LddjDW8n3q/ny/14qyDqFAU6pTtlM7uEXmneanYK loGZ1jLKTEV3teQ2u4qFrXZJ43htS9v137jbWteTTtGdkdjdTOJinJoUoSNeYBiXZOHk0d8WHBbT 3NwlYKDS5x9RpgpSWmie69njq2isW+nprWzj29ujKRyZby6l4liwU2pdJCyua5TxpDNza5PZ83Sy 325udCc0avfwGTjb9LF2n2A3n2wwbfihZXX5FKvy+iLeN2RMqL7jO75sxeZ52oSulb9j/LwN/DXM /ta4hEuQpENX0Zg4gRgBcqyV6khHSuVEfgxpMiO1k6RKrlTp0Y41ZKvIGXRorp23SBtddmTEE+DI lj95+hwaLuhLo1sgAioKad8gMfes2DLEyao7pVq3cu3q9SvYsJxqdUWqxazYtGrXsm3r1qXTtGip sXxr9y7evHr38u3r9y/gwIIHEy5s+DDixIoXM27s+DHkyJInU65s+TLmzJo3c+7s+TPo0KK1Nhht +jRqngFWN2jtenWA1LJn0wbA2jXu3K1j1+69VkADA8INNBAgwPfW27qX4+aNPJDx5caPPzchoAL2 7NqzF6+uMgDz8Lq9NwE+/Dz64tSRX9/ufruB/vXkKYEXb7/5/BIC0L9Pb7x3e+8JqF18+blxH4Kv 5QeecBxUwIGDA56nnmwBYhfhg9lBiKF73RnYQn0JIpjffgZcWIFwA8JHnADOieZehBCemKGG78m3 oGsO6OhAazr2+GMDPga544/klXiiif1pxyF2BYpm4YMOyrikhhwggMB7pX04pI4ddODaAw+0FmaY Y4rZWgdnBtmAi77th6SKMdK43Y2dNbAdhAgwOaWVFzKJXZ3PgecAmV4+4ICXaTaQpqIPGBqklzvq GOhs4PmpYpRzCtgmZwZsmGcBUn6KIZ+ifloBp+wRGmaajrrqKKKrNnAoobWGueMDlMpmaaYw/l5K 45TbparFAsUWW8KxPySbrF2efloAAgWEuuG0eerJQbWjWqDlO8Yyy6wPx4LrhrcLAFCusWEFSeah 7K7K7q2SyqujmB19a25lvAbrp5yZ7nvisE3cS8K4yOI7Lr5qOWstdtJC6LCVekqb7YYIELAtQAOf m3C45hYscMIaizUrvCWbfDK7X9obMseS6XvBqKXGHCOEFlDgQMAdc/TxWwJ8eqWMV048sYymgkqA BA7oqmzLw/CcBc9Pb7UuyoYWanXV3GbscdORgWfBhheIDTO/MkfJgQU289ht1xsTnK7bcXvLVgAW XDuznkvGHC0BBFxwM5tsg2zwsly/HfXH/lJrJQChiTruuKuQP/44BUuTizDc4nI8t9xtrxUk2KOe PfOnNt/MY846M32uwZ27/rm0el45+7W0215A3xdIcHrggq/O+ttuax4805irFQCikz/u6PLKJ4qz SXAT3Hq6y1J/sOfHO0AB6d2jvTvvqf+u+sbDC6+4V3UTAK3t7QuNu99/897izm2DO/fw0kOtP/pG gUeB8xLFPMk5jwK9W1nT8Ic9E3COc3lhHAVC5z0O6I4C81vNygZ3vtc58DcW6NvQQiitvhEgA+Dz UYvEp0Hy5c9w5cve4VqnPQAGsIaTu5kK99eyFhJvgz3MSwAY1wAJiC1sY6sg79iUwxW2/sB8Ttxc /7bCOAmQsIokFNsJeZRClehPhoXzYQwvJ8O0BHF7NjxjB25mOTFq7InEK1wUxTIoCkigjna0YwRR uEQxvtGFbvyiWyBIxCMiMYkY/EkbF7hBN7KQZXQLEh3RqLzdoS56jnQdIL8ISCACpwI6ShsoLaCj 4UznkFx0IBwZ6cK2rAY4XJqXa+inlHKNkZZPfBotx3g8SNZRkl6qoxr32EjCMfKFfoQhGafjSt0o s5ml3JUzo9kiZcJGmNUBjgPuWMMJALOSH5ImOMO5xswEUZzm3OKHfjCo7dHxju4EZvg+VM5z0tOa XjOBc6qpT9uYkp/9TGeIuGTBgRL0/nQ8eo09QzNPepozoY9xUT71CRt/4nOi6SxBQHdU0IISCaEL oiZDw3nRkWYhoLrhESzD41DPSLSlLn3pSklKG5OKqDm3ieln/inTnZ4Epj71KU+DKtR3/LSoLR0q UpOqTqMyVadKfWpSTXnIai4Vqla9KlazqtWtcrWrXv0qWMMq1rGStaxmPSta06rWtbK1rW59q1Hq wlO5wrUxU6HLO6RSlKnc1Ss7IUlQ5qLXfjCir2VhRl1tMhaRzKQX0qDrULJSCYucIwyNrYkpahIW yb4Fspxhij9CCw2KaAMnMqjKQgrSkFGQJBuGsIoLSHuRW2BWGQqBChWwotmHxAUl/h5JbU6gMBeV /BUo8dCHaA9BjGXYhBuioMMsPAta5NojJ8tVx0Goq9ibPLYnu63IWS6b3XGYxLOs9a5GcHEN1QID sRHZrWmFgVfmXmK9iAjuU5rRXXbsd7h5EMJCVhtbhqR3vKjFLVUCfF6OTBccBFFwe5tLW3oQOL5H Oa6DP6JXYqD2vlGhRXz9O12dVFe5i30vdC0MBBJv17gFpkl4v6vi5/o2upQ9C4brq2GKIAS+Hqbw ielbXBy/GMCV8HCEWRviCu+XuJuIiWWTgeQZO/evVXYvb2GC2Lhgg72ZVUWKm7xiG7tYyOrVMGyL PGUvrIOwKCZKIqB8ZHyI9sCn/hUuhHOrYM7SpcEDQTObnXvbMV8FzzVO7oKT8maJDPjGVMYyf9/c WZaYl7eVBq9eAuvXS9c4rk/e8jXEq+bssnjBV27xZv2rFFVj+i69FQurO+1pPxcB0HSurp31vOI8 J9jMif31jXFwWCcDu9gr4fRPYh1sYzO72c5+NrSjLe1pU7va1r42trOt7W1zu9ve/ja4wy3ucZO7 3OY+N7rTre51s7vd7n43vOMtGgbQu97yvne9882Ae8M73yTwd2T2bRJ9/8DePjC4CejN78MgHAAN Z8zD36FvhZdg4i0geMUFvvDCRDzgFOeIwRtOcYTb++Mf3zhhMN4ElYv85ABf/rnJXV7ykydc5Q5/ ecFHHnOBk5znPkc5Yix+cKH3XOM3x/nFd17ziRs94y9nehZCrvSjZ5zqNAf6YKC+9H/rXONF53rT nS52p/s87GDf+tlh/vSpH13hV8d6ym2edK+XfexUzznd8553tdv97kPH+dfH3nW4GwbpWr952tsu d7sXfe9473vHEe/3wA8d7G8nvF9aLnTEa37zaE/82s0+eZmbXeqOj/zgL495tzw88C33O+yjvnO9 9x3ys3/86Oteea7zfvV6CT3aO990yn++84mf+9J1/3fQ6zznNY+97+3CdNLLHemKv/z0jZ59tc8+ 8panued7T3bRR5/1i7c6/vixb/3k597yWgD8+udeevUvv/wj9b79v43//Hd7//zftv/9nwAOIAEW oAEeIAImoAIuIAM2oAM+IARGoAROIAVWoAVeIAZmILiVyAF0oAd+IAiGoAiOIAmWoAmeIAqmoAqu IAu2oAu+IAzGoAzOIA0egHCMU2BwYA3uIA/2oA/+IBAGoRAO4Qve4GIYwAFcyQAQIRM2oRM+IRRG oRSe4ABcyQHgIF4kYQFcoQaahgAkIQIYgGAYALRgYRdqhgBciRj2RQBU4RqeIW2Q4QHwRRvOIRwi RxWaYVcIAAHY4R0iRxLilEoIwAC84R8CYgEIokkUgCEeIiAOwFsMACQ6/iJ5LGFbkCElzkcAIIAe 8gQnZuJ8YKLCTGJX5NJaHAzUBAIyqQX2HVxIaN9a2Bz8hR78VZ1iMKJafKJXbM4sqaIvGkUCGcXb kd4rJhwrEh3LJePmxdwRkiJYpCFY7BAw/iKx9GITCeO+mZ7bEdw2Pp8xVpzkOVzVmdw4ep0tnuM2 /pz4eeM6duP7yVw4uiNYFEAnhoQlfoU0htE13s+9JJD1AE/5IItAKsv0FGQM8c8/ss4q6WPyzZxD PiQzfuO/SZ7UUWTzDV44Ot8wyqL2DZ/qfd/4fR/5KUUHikUBhEU+hozqoKJBBuP0sKRKAo/nsCRA qqRL1qTcDCRACp46/jJfNqoeLHKePKZjPP7kT8pe2TWeUl4d/jmkT0IfV0AjWATASUbjNeLkVcqk Tt5kQdKkQGYPTcakTo6lx7TkVu6eRRYf3a2cRLqdODJjNhblW2akRiIjOp5fU65lRkZkWCBAWBiA M5biVYrlWWLlThpkVyImQ4aLWRKOYu5kWRqmKd5lO/JlK4Ij78UlXL7lWmpm2MGlXiadLUZcXo5m R54jWAxAPQJESVolAzXmayYmbI6lVh5mbRKkYb4mV0amWGZPRBLj2aGeRKKfRxol8nVcRf5maPJl ZpKfU+4lUYpFIYJFa+JjbOZmY/YmY76kbNbmTMLmAnFlTYYlbcpl/tuB5DKaI2aKXetFJ3taJi12 3esxH2WG5DpeX1rYYGoGJldMZkBu53hCUcfApGNmZ1Z2UObsEC++pIJ2DXx25FHC40Sup/ilH2qa J3QWJ4R6ZPAtZ3GyYznmpx92RXUiRz4OBWF+xUcKFXN+RYlyxYv2xokiUnl2xYoG1eFR54jC6I5W RxxRwv2kxY3iaPzxqI6C4kXFqFYoKZJWB5MaxZM2qW9E6U9QqZTShpW6RJZeaWps6Ul4KZeaBpiG xJiGaWiUKUegqZl6hpq+g5WuYo2ehHi6BJyGw49mTGDUKV7o6S4ORZuGw5/GJp9SY5xGzzT2KV+k 6F945aHaqZ/2/qhWICGN2iYXAeikXmp/9oWi+gWjoiie/sR0fgUh/uJCHmTcLGaBUo+qKiRDSk8X KSQU2eSALmZvfstARuZX8iL2KJJk9uNXFg+DMpCtJs6p4ipCsow/liU/2s+wIqs+4ip42upjloBq TmVVVmNODiaBPqZ2Zmsm3eqvKuY/xmTBkCuAkidWlmtYkqe03qax/tCBbqtMduq4dquscusbzWZ2 oiu5uqS9Fqpf9qXloKugguu0Fqa7gqe4piK+Uqq5Mua/dud1ditOKittPqzAFCbG3qRXYuzBdifB aqzISmy4KqznAKZYUGnIguzhJI6rGqy7opKvHigxTavHluzG/toswmqnxRZoLoGlxjZQeXqsKboq saZrB7Uswe5qQgprrA5rFgRqOEglwyoqv2KrZBroxx7miV7tuepsxN7mxO7sr0JryRZq1tqm1cIs pV7sdt5sVpKt2uLmppqtCehiXzbiYA6t1s5tscLs0pplij5s046speasw8otZJIlzm5t2K6st45t xfJtulJuwrJs4qbtx6JsWhhAwGYsdgrPViIOZBpt0x5rqjqtsIqt6h5sgzptp77Qdf4nwj5ru3Lt 0rZrMN4u7dLuq4puSyIOxcJRjdbqjLZAGK4FLmrVoAJG8zqqjzaB57JFGipi9EbG8zqNd3QNVa6m ShwAfybVs52eRvZqBs/cY1ug75qORhLahQBs4fqyL96+BQJAavxmxgFc612k4QBY7/0CBiH2L1/k r97+r2R4rv3ehecmrwEfcBmO4QBsofcOVQKL2xcWQKgSxhcq4RJOoQd/MAiHsAh/sCRiMBcuhg6O sAqvMAu3sAuLoBE2sAzPMA3XsA3fMA7nsA7vMA/3sA//MBAHsRAPMREXsREfMRInsRIvMRM3sRM/ MRRHsRRPMRVXsRUnVggAADs= ------=_NextPart_41D_7FA0_31A8A7F8.938421A1-- From ips-bounces@ietf.org Fri Apr 13 12:22:12 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HcOXK-0008LR-DE; Fri, 13 Apr 2007 12:22:10 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HcOXG-0008IE-V1 for ips-confirm+ok@megatron.ietf.org; Fri, 13 Apr 2007 12:22:06 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HcOX9-00089F-OH for ips@ietf.org; Fri, 13 Apr 2007 12:21:59 -0400 Received: from mexforward.lss.emc.com ([128.222.32.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HcOVt-00075A-Vh for ips@ietf.org; Fri, 13 Apr 2007 12:20:42 -0400 Received: from mailhub.lss.emc.com (nirah.lss.emc.com [10.254.144.13]) by mexforward.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3DGKfCZ001201 for ; Fri, 13 Apr 2007 12:20:41 -0400 (EDT) Received: from corpussmtp3.corp.emc.com (corpussmtp3.corp.emc.com [10.254.64.53]) by mailhub.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3DGKUHt015453 for ; Fri, 13 Apr 2007 12:20:41 -0400 (EDT) From: Black_David@emc.com Received: from CORPUSMX20A.corp.emc.com ([128.221.62.12]) by corpussmtp3.corp.emc.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 13 Apr 2007 12:19:47 -0400 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 13 Apr 2007 12:19:46 -0400 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: TMDA deployment for IETF mailing lists Thread-Index: Acd951/KCMNyFcnXQuaal0QlRiMQuAAAAPrg To: X-OriginalArrivalTime: 13 Apr 2007 16:19:47.0192 (UTC) FILETIME=[8DB81F80:01C77DE7] X-PMX-Version: 4.7.1.128075, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.4.13.85034 X-PerlMx-Spam: Gauge=, SPAM=0%, Reason='EMC_BODY_1+ -3, EMC_FROM_0+ -3, NO_REAL_NAME 0, __C230066_P5 0, __CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __IMS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' X-Spam-Score: 0.2 (/) X-Scan-Signature: e5ba305d0e64821bf3d8bc5d3bb07228 Cc: Black_David@emc.com Subject: [Ips] TMDA deployment for IETF mailing lists X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Folks, Tagged Message Delivery Agent (TMDA - http://tmda.net/) software has been deployed for the ips WG mailing list. If you are subscribed to this list, or to any IETF mailing list hosted at ietf.org, you will not be affected, as all such addresses are on TMDA's whitelist. The only effect should be that if you post to the ips list from an email address that is not subscribed to any IETF mailing list hosted at ietf.org, you will need to respond to a TMDA email challenge before you get the email notice that your post has been held for administrator approval. The reason for deployment is that I currently have to wade through a dozen or more attempts to send spam to the ips list every day, just in case one of them is an actual message (I can't remember the last time an actual message showed up in that fashion) - TMDA will automatically reject most of the spams, reducing the burden on me. Thanks, --David (imss WG chair) ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Mon Apr 16 15:50:30 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HdXDX-00080B-Nq; Mon, 16 Apr 2007 15:50:27 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HdXDB-0007uY-Kq for ips-confirm+ok@megatron.ietf.org; Mon, 16 Apr 2007 15:50:05 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HdXDB-0007t3-4R; Mon, 16 Apr 2007 15:50:05 -0400 Received: from ns1.neustar.com ([2001:503:c779:1a::9c9a:108a]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HdXD8-0001Oz-VJ; Mon, 16 Apr 2007 15:50:05 -0400 Received: from stiedprstage1.ietf.org (stiedprstage1.va.neustar.com [10.31.47.10]) by ns1.neustar.com (Postfix) with ESMTP id CB47426EAD; Mon, 16 Apr 2007 19:50:02 +0000 (GMT) Received: from ietf by stiedprstage1.ietf.org with local (Exim 4.43) id 1HdXD8-0000L3-JW; Mon, 16 Apr 2007 15:50:02 -0400 Content-Type: Multipart/Mixed; Boundary="NextPart" Mime-Version: 1.0 To: i-d-announce@ietf.org From: Internet-Drafts@ietf.org Message-Id: Date: Mon, 16 Apr 2007 15:50:02 -0400 X-Spam-Score: -2.5 (--) X-Scan-Signature: 10d3e4e3c32e363f129e380e644649be Cc: ips@ietf.org Subject: [Ips] I-D ACTION:draft-ietf-ips-iscsi-impl-guide-07.txt X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Corrections and Clarifications Author(s) : M. Chadalapaka Filename : draft-ietf-ips-iscsi-impl-guide-07.txt,.pdf Pages : 49 Date : 2007-4-16 iSCSI is a SCSI transport protocol and maps the SCSI architecture and command sets onto TCP/IP. RFC 3720 defines the iSCSI protocol. This document compiles the clarifications to the original protocol definition in RFC 3720 to serve as a companion document for the iSCSI implementers. This document updates RFC 3720 and the text in this document supersedes the text in RFC 3720 when the two differ. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt To remove yourself from the I-D Announcement list, send a message to i-d-announce-request@ietf.org with the word unsubscribe in the body of the message. You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce to change your subscription settings. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-impl-guide-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2007-4-16145410.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ips-iscsi-impl-guide-07.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2007-4-16145410.I-D@ietf.org> --OtherAccess-- --NextPart Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --NextPart-- From ips-bounces@ietf.org Mon Apr 16 18:31:13 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HdZj5-0002Ob-8B; Mon, 16 Apr 2007 18:31:11 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HdZj3-0002OS-Lb for ips-confirm+ok@megatron.ietf.org; Mon, 16 Apr 2007 18:31:09 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HdZj3-0002OJ-C7 for ips@ietf.org; Mon, 16 Apr 2007 18:31:09 -0400 Received: from web51905.mail.re2.yahoo.com ([206.190.48.68]) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HdZj2-0003J9-EW for ips@ietf.org; Mon, 16 Apr 2007 18:31:09 -0400 Received: (qmail 41185 invoked by uid 60001); 16 Apr 2007 22:31:08 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=LA/HTjGHbkOaUFXD1GvNT3M8CFfnUVJ2DdNHoayGxjWrdgPLlx7egnsSfSHYtlOffjijDqbpUfWmSP503tUH3qiBJ1PhPDjaw5OZ6yw8KBntWDsVj/iPkmWRuRhZCwP2rBPRczZH5nxs1kLTKIH0lJZ/MbHzepfqCmp/4/wGWjg=; X-YMail-OSG: HghdaUcVM1kcNvlUrT_.DkHVeKkAqPXpHJUm_lE_dYb1QBtklDh6puAbc_slV1dGlw-- Received: from [15.235.153.107] by web51905.mail.re2.yahoo.com via HTTP; Mon, 16 Apr 2007 15:31:08 PDT X-Mailer: YahooMailRC/478 YahooMailWebService/0.7.41.10 Date: Mon, 16 Apr 2007 15:31:08 -0700 (PDT) From: "Mallikarjun C." To: IPS MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="0-1119654800-1176762668=:40103" Message-ID: <261996.40103.qm@web51905.mail.re2.yahoo.com> X-Spam-Score: 0.0 (/) X-Scan-Signature: f66b12316365a3fe519e75911daf28a8 Subject: [Ips] Fw: I-D ACTION:draft-ietf-ips-iscsi-impl-guide-07.txt X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org --0-1119654800-1176762668=:40103 Content-Type: text/plain; charset=ascii This draft addresses all the Last Call comments - including those from Lars and David. Largest number of words were added to create the IANA registries, so the IANA considerations section could benefit from WG review. Mallikarjun ----- Forwarded Message ---- From: "Internet-Drafts@ietf.org" To: i-d-announce@ietf.org Cc: ips@ietf.org Sent: Monday, April 16, 2007 12:50:02 PM Subject: [Ips] I-D ACTION:draft-ietf-ips-iscsi-impl-guide-07.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Storage Working Group of the IETF. Title : iSCSI Corrections and Clarifications Author(s) : M. Chadalapaka Filename : draft-ietf-ips-iscsi-impl-guide-07.txt,.pdf Pages : 49 Date : 2007-4-16 iSCSI is a SCSI transport protocol and maps the SCSI architecture and command sets onto TCP/IP. RFC 3720 defines the iSCSI protocol. This document compiles the clarifications to the original protocol definition in RFC 3720 to serve as a companion document for the iSCSI implementers. This document updates RFC 3720 and the text in this document supersedes the text in RFC 3720 when the two differ. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt To remove yourself from the I-D Announcement list, send a message to i-d-announce-request@ietf.org with the word unsubscribe in the body of the message. You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce to change your subscription settings. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ips-iscsi-impl-guide-07.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com --0-1119654800-1176762668=:40103 Content-Type: message/external-body; name="draft-ietf-ips-iscsi-impl-guide-07.txt"; site="ftp.ietf.org"; access-type=anon-ftp; directory=internet-drafts Content-Type: text/plain Content-ID: <2007-4-16145410.I-D@ietf.org> --0-1119654800-1176762668=:40103 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --0-1119654800-1176762668=:40103-- From ips-bounces@ietf.org Tue Apr 17 17:10:53 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hduwr-0001O4-VW; Tue, 17 Apr 2007 17:10:49 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hduwq-0001Nw-9H for ips-confirm+ok@megatron.ietf.org; Tue, 17 Apr 2007 17:10:48 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hduwp-0001No-Vv; Tue, 17 Apr 2007 17:10:47 -0400 Received: from nit.isi.edu ([128.9.160.116]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hduwp-0003PT-KC; Tue, 17 Apr 2007 17:10:47 -0400 Received: from nit.isi.edu (loopback [127.0.0.1]) by nit.isi.edu (8.12.11.20060308/8.12.11) with ESMTP id l3HLAl0h007712; Tue, 17 Apr 2007 14:10:47 -0700 Received: (from apache@localhost) by nit.isi.edu (8.12.11.20060308/8.12.11/Submit) id l3HLAl1o007711; Tue, 17 Apr 2007 14:10:47 -0700 Date: Tue, 17 Apr 2007 14:10:47 -0700 Message-Id: <200704172110.l3HLAl1o007711@nit.isi.edu> To: ietf-announce@ietf.org, rfc-dist@rfc-editor.org From: rfc-editor@rfc-editor.org X-Spam-Score: -14.8 (--------------) X-Scan-Signature: 6cca30437e2d04f45110f2ff8dc1b1d5 Cc: ips@ietf.org, rfc-editor@rfc-editor.org Subject: [Ips] RFC 4850 on Declarative Public Extension Key for Internet Small Computer Systems Interface (iSCSI) Node Architecture X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org A new Request for Comments is now available in online RFC libraries. RFC 4850 Title: Declarative Public Extension Key for Internet Small Computer Systems Interface (iSCSI) Node Architecture Author: D. Wysochanski Status: Standards Track Date: April 2007 Mailbox: wysochanski@pobox.com Pages: 9 Characters: 16430 Updates: RFC3720 See-Also: I-D Tag: draft-ietf-ips-iscsi-nodearch-key-03.txt URL: http://www.rfc-editor.org/rfc/rfc4850.txt The Internet Small Computer Systems Interface (iSCSI) protocol, described in RFC 3720, allows for extension items to the protocol in the form of Private or Public Extension Keys. This document describes a Public Extension Key for the purpose of enhancing iSCSI supportability. The key accomplishes this objective by allowing iSCSI nodes to communicate architecture details during the iSCSI login sequence. The receiving node can then use this information for enhanced logging and support. This document updates RFC 3720 to allow iSCSI extension items to be defined by standards track RFCs and experimental RFCs in addition to informational RFCs. [STANDARDS TRACK] This document is a product of the IP Storage Working Group of the IETF. This is now a Proposed Standard Protocol. STANDARDS TRACK: This document specifies an Internet standards track protocol for the Internet community,and requests discussion and suggestions for improvements.Please refer to the current edition of the Internet Official Protocol Standards (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This announcement is sent to the IETF list and the RFC-DIST list. Requests to be added to or deleted from the IETF distribution list should be sent to IETF-REQUEST@IETF.ORG. Requests to be added to or deleted from the RFC-DIST distribution list should be sent to RFC-DIST-REQUEST@RFC-EDITOR.ORG. Details on obtaining RFCs via FTP or EMAIL may be obtained by sending an EMAIL message to rfc-info@RFC-EDITOR.ORG with the message body help: ways_to_get_rfcs. For example: To: rfc-info@RFC-EDITOR.ORG Subject: getting rfcs help: ways_to_get_rfcs Requests for special distribution should be addressed to either the author of the RFC in question, or to RFC-Manager@RFC-EDITOR.ORG. Unless specifically noted otherwise on the RFC itself, all RFCs are for unlimited distribution. Submissions for Requests for Comments should be sent to RFC-EDITOR@RFC-EDITOR.ORG. Please consult RFC 2223, Instructions to RFC Authors, for further information. The RFC Editor Team USC/Information Sciences Institute ... _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Wed Apr 18 09:12:14 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1He9xD-0000EW-9l; Wed, 18 Apr 2007 09:12:11 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1He9xB-0008Vw-At for ips-confirm+ok@megatron.ietf.org; Wed, 18 Apr 2007 09:12:09 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1He9xB-0008Tx-0o for ips@ietf.org; Wed, 18 Apr 2007 09:12:09 -0400 Received: from mexforward.lss.emc.com ([128.222.32.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1He9xA-0007Ay-L5 for ips@ietf.org; Wed, 18 Apr 2007 09:12:08 -0400 Received: from mailhub.lss.emc.com (uraeus.lss.emc.com [10.254.144.14]) by mexforward.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3IDC7qQ025080 for ; Wed, 18 Apr 2007 09:12:07 -0400 (EDT) Received: from corpussmtp3.corp.emc.com (corpussmtp3.corp.emc.com [10.254.64.53]) by mailhub.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3IDBWcB010677 for ; Wed, 18 Apr 2007 09:12:06 -0400 (EDT) From: Black_David@emc.com Received: from CORPUSMX20A.corp.emc.com ([128.221.62.12]) by corpussmtp3.corp.emc.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 18 Apr 2007 09:11:37 -0400 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 18 Apr 2007 09:11:37 -0400 Message-ID: In-Reply-To: <261996.40103.qm@web51905.mail.re2.yahoo.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Please review IANA considerations in iSCSI C&C draft -07 Thread-Index: AceAdv/DsZ7rSZKaRpKnyDTNYVW2wgBQ5Q0g References: <261996.40103.qm@web51905.mail.re2.yahoo.com> To: X-OriginalArrivalTime: 18 Apr 2007 13:11:37.0673 (UTC) FILETIME=[18B4FB90:01C781BB] X-PMX-Version: 4.7.1.128075, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.4.18.55833 X-PerlMx-Spam: Gauge=, SPAM=0%, Reason='EMC_BODY_1+ -3, EMC_FROM_0+ -3, NO_REAL_NAME 0, __C230066_P5 0, __CP_URI_IN_BODY 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __IMS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' X-Spam-Score: 0.2 (/) X-Scan-Signature: 6d95a152022472c7d6cdf886a0424dc6 Subject: [Ips] Please review IANA considerations in iSCSI C&C draft -07 X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Everyone, As Mallikarjun asked, please review the new IANA registry material in this draft - there's a lot of stuff there, and this will become the primary reference for iSCSI values. Review comments need to be sent to the list by Wednesday, May 2nd. Thanks, --David (ips WG chair) ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Mallikarjun C. [mailto:cb_mallikarjun@yahoo.com]=20 > Sent: Monday, April 16, 2007 6:31 PM > To: IPS > Subject: [Ips] Fw: I-D ACTION:draft-ietf-ips-iscsi-impl-guide-07.txt >=20 > This draft addresses all the Last Call comments - including=20 > those from Lars and David. Largest number of words were=20 > added to create the IANA registries, so the IANA=20 > considerations section could benefit from WG review. >=20 > Mallikarjun >=20 >=20 > ----- Forwarded Message ---- > From: "Internet-Drafts@ietf.org" > To: i-d-announce@ietf.org > Cc: ips@ietf.org > Sent: Monday, April 16, 2007 12:50:02 PM > Subject: [Ips] I-D ACTION:draft-ietf-ips-iscsi-impl-guide-07.txt >=20 >=20 > A New Internet-Draft is available from the on-line Internet-Drafts=20 > directories. > This draft is a work item of the IP Storage Working Group of the IETF. >=20 > Title : iSCSI Corrections and Clarifications > Author(s) : M. Chadalapaka > Filename : draft-ietf-ips-iscsi-impl-guide-07.txt,.pdf > Pages : 49 > Date : 2007-4-16 > =20 > iSCSI is a SCSI transport protocol and maps the SCSI=20 > architecture and command sets onto TCP/IP. RFC 3720 defines=20 > the iSCSI protocol. This document compiles the=20 > clarifications to the original protocol definition in RFC=20 > 3720 to serve as a companion document for the iSCSI=20 > implementers. This document updates RFC 3720 and the text in=20 > this document supersedes the text in RFC 3720 when the two=20 > differ. >=20 > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-impl- > guide-07.txt >=20 > To remove yourself from the I-D Announcement list, send a message to=20 > i-d-announce-request@ietf.org with the word unsubscribe in=20 > the body of=20 > the message.=20 > You can also visit=20 > https://www1.ietf.org/mailman/listinfo/I-D-announce=20 > to change your subscription settings. >=20 > Internet-Drafts are also available by anonymous FTP. Login with the=20 > username "anonymous" and a password of your e-mail address. After=20 > logging in, type "cd internet-drafts" and then=20 > "get draft-ietf-ips-iscsi-impl-guide-07.txt". >=20 > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html=20 > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt >=20 > Internet-Drafts can also be obtained by e-mail. >=20 > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-ietf-ips-iscsi-impl-guide-07.txt". > =20 > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant=20 > mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. >=20 > Below is the data which will enable a MIME compliant mail reader > implementation to automatically retrieve the ASCII version of the > Internet-Draft. > _______________________________________________ > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips >=20 > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around=20 > http://mail.yahoo.com=20 >=20 _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 19 19:13:58 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hefp5-0003vK-DG; Thu, 19 Apr 2007 19:13:55 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hefp3-0003vE-Jt for ips-confirm+ok@megatron.ietf.org; Thu, 19 Apr 2007 19:13:53 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hefp3-0003v6-AG for ips@ietf.org; Thu, 19 Apr 2007 19:13:53 -0400 Received: from mail3.pillardata.com ([209.120.231.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hefp1-00055N-DK for ips@ietf.org; Thu, 19 Apr 2007 19:13:53 -0400 Received: from coex02.trans.corp ([172.18.24.19]) by mail3.pillardata.com with ESMTP; 19 Apr 2007 16:13:50 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Thu, 19 Apr 2007 17:13:49 -0600 Message-ID: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: detection of failed sessions to allow re-login Thread-Index: AceC2Geaj9izq1egS867H12pdfy0iw== From: "Paul Hughes" To: X-Spam-Score: 0.1 (/) X-Scan-Signature: f49c97ce49302a02285a2d36a99eef8c Subject: [Ips] detection of failed sessions to allow re-login X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============2143689045==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============2143689045== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C782D8.63B3BE12" This is a multi-part message in MIME format. ------_=_NextPart_001_01C782D8.63B3BE12 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I have a question about how a target can quickly detect session failures so that a re-login can succeed. =20 Here's my scenario: =20 1) an initiator is booting from an iSCSI target 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target 3) the HBA BIOS creates the first session, discovers the boot LUN, and reads the boot loader 4) the boot loader reads the kernel from the boot LUN 5) the kernel resets the iSCSI HBA while loading an HBA driver 6) the HBA driver attempts to create a new session =20 The problem I'm seeing is that the target is failing the login for the new session because the target thinks the first session created by the HBA BIOS is still valid (not in failed state). The HBA reset was not detected by the target soon enough for the target to know that the first session is now in the failed state when the initiator attempts to login and create the second session using the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the first session (with = TSIH=3D0). The target does not see a link down event because a switch is connected between the HBA and the target port. The target eventually detects that the first session is failed when it sends a NOP-Out PDU and receives a transport failure. Unfortunately, this occurs too late and the boot fails. =20 In my case the target is sending NOP-Out PDUs every 60 seconds. I can change that to 5 seconds, but I don't think that will fix every case. Is there a better way for the target to determine that the first session has failed so that a re-login will succeed on the first try? =20 Thanks, Paul =20 =20 =20 ------_=_NextPart_001_01C782D8.63B3BE12 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
I have = a question=20 about how a target can quickly detect session failures so that = a=20 re-login can succeed.
 
Here's = my=20 scenario:
 
1) an = initiator is=20 booting from an iSCSI target
2) the = initiator is=20 using an iSCSI HBA to communicate with the iSCSI = target
3) = the HBA=20 BIOS creates the first session, discovers the boot LUN, = and reads=20 the boot loader
4) the = boot loader=20 reads the kernel from the boot LUN
5) the = kernel resets=20 the iSCSI HBA while loading an HBA driver
6) the = HBA driver=20 attempts to create a new session
 
The = problem I'm=20 seeing is that the target is failing the login for the new session = because the=20 target thinks the first session created by the HBA BIOS is still = valid (not=20 in failed state).  The HBA reset was not detected by the = target=20 soon enough for the target to know that the first session is now in the = failed=20 state when the initiator attempts to login and create the second session = using=20 the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as = the first=20 session (with TSIH=3D0).  The target does not see a link down event = because a=20 switch is connected between the HBA and the target port.  The = target=20 eventually detects that the first session is failed when it sends a = NOP-Out PDU=20 and receives a transport failure.  Unfortunately, this occurs too = late and=20 the boot fails.
 
In my = case the=20 target is sending NOP-Out PDUs every 60 seconds.  I can change that = to 5=20 seconds, but I don't think that will fix every case.  Is there a = better way=20 for the target to determine that the first session has failed so = that a=20 re-login will succeed on the first try?
 
Thanks,
Paul
 
 
 
------_=_NextPart_001_01C782D8.63B3BE12-- --===============2143689045== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============2143689045==-- From ips-bounces@ietf.org Thu Apr 19 20:56:07 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HehPx-00033b-NV; Thu, 19 Apr 2007 20:56:05 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HehPw-00033V-IE for ips-confirm+ok@megatron.ietf.org; Thu, 19 Apr 2007 20:56:04 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HehPw-00033N-8c for ips@ietf.org; Thu, 19 Apr 2007 20:56:04 -0400 Received: from mail-gw3.adaptec.com ([216.52.22.36]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HehPv-0008L8-Ju for ips@ietf.org; Thu, 19 Apr 2007 20:56:04 -0400 Received: from aime2k302.adaptec.com (aime2k302.adaptec.com [10.25.8.48]) by mail-gw3.adaptec.com (Spam Firewall) with ESMTP id B04D21909D7; Thu, 19 Apr 2007 17:56:02 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [Ips] detection of failed sessions to allow re-login Date: Thu, 19 Apr 2007 17:56:00 -0700 Message-ID: <368FBF3D8437A748BA8222526BF9309901ACB8A3@aime2k302.adaptec.com> In-reply-to: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Ips] detection of failed sessions to allow re-login Thread-Index: AceC2Geaj9izq1egS867H12pdfy0iwADTTCA References: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> From: "Sandars, Ken" To: "Paul Hughes" , X-Spam-Score: 0.1 (/) X-Scan-Signature: 501044f827b673024f6a4cb1d46e67d2 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0753463506==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0753463506== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C782E6.AB0DA0CF" This is a multi-part message in MIME format. ------_=_NextPart_001_01C782E6.AB0DA0CF Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Paul, =20 That's a target problem, most likely a bug. The second login with = TSIH=3D0 tells the target to perform session reinstatement. That's jargon for "silently nuke the first session". =20 The target may be failing the login because it's internal cleanup requires more time (I/O requests from the previous session are jammed for instance). Your scenario suggests this is not likely. =20 HTH, Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 09:14 To: ips@ietf.org Subject: [Ips] detection of failed sessions to allow re-login I have a question about how a target can quickly detect session failures so that a re-login can succeed. =20 Here's my scenario: =20 1) an initiator is booting from an iSCSI target 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target 3) the HBA BIOS creates the first session, discovers the boot LUN, and reads the boot loader 4) the boot loader reads the kernel from the boot LUN 5) the kernel resets the iSCSI HBA while loading an HBA driver 6) the HBA driver attempts to create a new session =20 The problem I'm seeing is that the target is failing the login for the new session because the target thinks the first session created by the HBA BIOS is still valid (not in failed state). The HBA reset was not detected by the target soon enough for the target to know that the first session is now in the failed state when the initiator attempts to login and create the second session using the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the first session (with = TSIH=3D0). The target does not see a link down event because a switch is connected between the HBA and the target port. The target eventually detects that the first session is failed when it sends a NOP-Out PDU and receives a transport failure. Unfortunately, this occurs too late and the boot fails. =20 In my case the target is sending NOP-Out PDUs every 60 seconds. I can change that to 5 seconds, but I don't think that will fix every case. Is there a better way for the target to determine that the first session has failed so that a re-login will succeed on the first try? =20 Thanks, Paul =20 =20 =20 ------_=_NextPart_001_01C782E6.AB0DA0CF Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Paul,
 
That's a target problem, most likely a bug. The = second=20 login with TSIH=3D0 tells the target to perform session reinstatement. = That's=20 jargon for "silently nuke the first session".
 
The target may be failing the login because = it's internal=20 cleanup requires more time (I/O requests from the previous session are = jammed=20 for instance). Your scenario suggests this is not = likely.
 
HTH,
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 09:14
To: ips@ietf.org
Subject: [Ips] detection of = failed=20 sessions to allow re-login

I have = a question=20 about how a target can quickly detect session failures so that = a=20 re-login can succeed.
 
Here's = my=20 scenario:
 
1) an = initiator is=20 booting from an iSCSI target
2) the = initiator is=20 using an iSCSI HBA to communicate with the iSCSI = target
3) = the HBA=20 BIOS creates the first session, discovers the boot LUN, = and reads=20 the boot loader
4) the = boot loader=20 reads the kernel from the boot LUN
5) the = kernel resets=20 the iSCSI HBA while loading an HBA driver
6) the = HBA driver=20 attempts to create a new session
 
The = problem I'm=20 seeing is that the target is failing the login for the new session = because the=20 target thinks the first session created by the HBA BIOS is still = valid (not=20 in failed state).  The HBA reset was not detected by the = target=20 soon enough for the target to know that the first session is now in the = failed=20 state when the initiator attempts to login and create the second session = using=20 the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as = the first=20 session (with TSIH=3D0).  The target does not see a link down event = because a=20 switch is connected between the HBA and the target port.  The = target=20 eventually detects that the first session is failed when it sends a = NOP-Out PDU=20 and receives a transport failure.  Unfortunately, this occurs too = late and=20 the boot fails.
 
In my = case the=20 target is sending NOP-Out PDUs every 60 seconds.  I can change that = to 5=20 seconds, but I don't think that will fix every case.  Is there a = better way=20 for the target to determine that the first session has failed so = that a=20 re-login will succeed on the first try?
 
Thanks,
Paul
 
 
 
------_=_NextPart_001_01C782E6.AB0DA0CF-- --===============0753463506== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0753463506==-- From ips-bounces@ietf.org Thu Apr 19 21:36:40 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hei3C-0000OL-DT; Thu, 19 Apr 2007 21:36:38 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hei3A-0000OF-KS for ips-confirm+ok@megatron.ietf.org; Thu, 19 Apr 2007 21:36:36 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hei3A-0000O7-At for ips@ietf.org; Thu, 19 Apr 2007 21:36:36 -0400 Received: from mail1.pillardata.com ([63.251.178.136]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hei38-0004p3-Q2 for ips@ietf.org; Thu, 19 Apr 2007 21:36:36 -0400 Received: from coex02.trans.corp ([172.18.24.19]) by mail1.pillardata.com with ESMTP; 19 Apr 2007 19:36:33 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [Ips] detection of failed sessions to allow re-login Date: Thu, 19 Apr 2007 19:36:32 -0600 Message-ID: <16236EEEF4D4264DA31C2E35E3607CFE08906E5C@coex02.trans.corp> In-Reply-To: <368FBF3D8437A748BA8222526BF9309901ACB8A3@aime2k302.adaptec.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Ips] detection of failed sessions to allow re-login Thread-Index: AceC2Geaj9izq1egS867H12pdfy0iwADTTCAAAFtARA= References: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> <368FBF3D8437A748BA8222526BF9309901ACB8A3@aime2k302.adaptec.com> From: "Paul Hughes" To: "Sandars, Ken" , X-Spam-Score: 0.0 (/) X-Scan-Signature: 515708a075ffdf0a79d1c83b601e2afd Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1069194437==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============1069194437== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C782EC.540B28D0" This is a multi-part message in MIME format. ------_=_NextPart_001_01C782EC.540B28D0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Isn't session re-instatement possible only for sessions that are in the failed state (no active connections)? That was my understanding of Section 5.3.5 of RFC 3720. =20 How can the target know that the session is in the failed state if it hasn't detected that the only connection was dropped? =20 Thanks, Paul =20 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Thursday, April 19, 2007 6:56 PM To: Paul Hughes; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Hi Paul, =20 That's a target problem, most likely a bug. The second login with = TSIH=3D0 tells the target to perform session reinstatement. That's jargon for "silently nuke the first session". =20 The target may be failing the login because it's internal cleanup requires more time (I/O requests from the previous session are jammed for instance). Your scenario suggests this is not likely. =20 HTH, Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 09:14 To: ips@ietf.org Subject: [Ips] detection of failed sessions to allow re-login I have a question about how a target can quickly detect session failures so that a re-login can succeed. =20 Here's my scenario: =20 1) an initiator is booting from an iSCSI target 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target 3) the HBA BIOS creates the first session, discovers the boot LUN, and reads the boot loader 4) the boot loader reads the kernel from the boot LUN 5) the kernel resets the iSCSI HBA while loading an HBA driver 6) the HBA driver attempts to create a new session =20 The problem I'm seeing is that the target is failing the login for the new session because the target thinks the first session created by the HBA BIOS is still valid (not in failed state). The HBA reset was not detected by the target soon enough for the target to know that the first session is now in the failed state when the initiator attempts to login and create the second session using the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the first session (with = TSIH=3D0). The target does not see a link down event because a switch is connected between the HBA and the target port. The target eventually detects that the first session is failed when it sends a NOP-Out PDU and receives a transport failure. Unfortunately, this occurs too late and the boot fails. =20 In my case the target is sending NOP-Out PDUs every 60 seconds. I can change that to 5 seconds, but I don't think that will fix every case. Is there a better way for the target to determine that the first session has failed so that a re-login will succeed on the first try? =20 Thanks, Paul =20 =20 =20 ------_=_NextPart_001_01C782EC.540B28D0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Isn't session re-instatement possible only = for sessions that are in the failed state (no active=20 connections)?  That was my understanding of Section 5.3.5 = of RFC=20 3720.
 
How can the target know that the session is in = the failed=20 state if it hasn't detected that the only connection was=20 dropped?
 
Thanks,
Paul
 

From: Sandars, Ken=20 [mailto:ken_sandars@adaptec.com]
Sent: Thursday, April 19, = 2007 6:56=20 PM
To: Paul Hughes; ips@ietf.org
Subject: RE: [Ips]=20 detection of failed sessions to allow re-login

Hi Paul,
 
That's a target problem, most likely a bug. The = second=20 login with TSIH=3D0 tells the target to perform session reinstatement. = That's=20 jargon for "silently nuke the first session".
 
The target may be failing the login because = it's internal=20 cleanup requires more time (I/O requests from the previous session are = jammed=20 for instance). Your scenario suggests this is not = likely.
 
HTH,
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 09:14
To: ips@ietf.org
Subject: [Ips] detection of = failed=20 sessions to allow re-login

I have = a question=20 about how a target can quickly detect session failures so that = a=20 re-login can succeed.
 
Here's = my=20 scenario:
 
1) an = initiator is=20 booting from an iSCSI target
2) the = initiator is=20 using an iSCSI HBA to communicate with the iSCSI = target
3) = the HBA=20 BIOS creates the first session, discovers the boot LUN, = and reads=20 the boot loader
4) the = boot loader=20 reads the kernel from the boot LUN
5) the = kernel resets=20 the iSCSI HBA while loading an HBA driver
6) the = HBA driver=20 attempts to create a new session
 
The = problem I'm=20 seeing is that the target is failing the login for the new session = because the=20 target thinks the first session created by the HBA BIOS is still = valid (not=20 in failed state).  The HBA reset was not detected by the = target=20 soon enough for the target to know that the first session is now in the = failed=20 state when the initiator attempts to login and create the second session = using=20 the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as = the first=20 session (with TSIH=3D0).  The target does not see a link down event = because a=20 switch is connected between the HBA and the target port.  The = target=20 eventually detects that the first session is failed when it sends a = NOP-Out PDU=20 and receives a transport failure.  Unfortunately, this occurs too = late and=20 the boot fails.
 
In my = case the=20 target is sending NOP-Out PDUs every 60 seconds.  I can change that = to 5=20 seconds, but I don't think that will fix every case.  Is there a = better way=20 for the target to determine that the first session has failed so = that a=20 re-login will succeed on the first try?
 
Thanks,
Paul
 
 
 
------_=_NextPart_001_01C782EC.540B28D0-- --===============1069194437== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1069194437==-- From ips-bounces@ietf.org Thu Apr 19 22:36:21 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Heiyt-0005QH-2T; Thu, 19 Apr 2007 22:36:15 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Heiyq-0005Q0-Ou for ips-confirm+ok@megatron.ietf.org; Thu, 19 Apr 2007 22:36:12 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Heiyq-0005Ps-F7 for ips@ietf.org; Thu, 19 Apr 2007 22:36:12 -0400 Received: from mail-gw3.adaptec.com ([216.52.22.36]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Heiyp-00008A-JW for ips@ietf.org; Thu, 19 Apr 2007 22:36:12 -0400 Received: from aime2k302.adaptec.com (aime2k302.adaptec.com [10.25.8.48]) by mail-gw3.adaptec.com (Spam Firewall) with ESMTP id 854661906C1; Thu, 19 Apr 2007 19:36:10 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [Ips] detection of failed sessions to allow re-login Date: Thu, 19 Apr 2007 19:36:08 -0700 Message-ID: <368FBF3D8437A748BA8222526BF9309901ACB8D2@aime2k302.adaptec.com> In-reply-to: <16236EEEF4D4264DA31C2E35E3607CFE08906E5C@coex02.trans.corp> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Ips] detection of failed sessions to allow re-login Thread-Index: AceC2Geaj9izq1egS867H12pdfy0iwADTTCAAAFtARAAAfxWcA== References: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> <368FBF3D8437A748BA8222526BF9309901ACB8A3@aime2k302.adaptec.com> <16236EEEF4D4264DA31C2E35E3607CFE08906E5C@coex02.trans.corp> From: "Sandars, Ken" To: "Paul Hughes" , X-Spam-Score: 0.0 (/) X-Scan-Signature: a1dc446dc7ac353b90b60743d0e479e3 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1389310280==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============1389310280== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C782F4.A7F3B707" This is a multi-part message in MIME format. ------_=_NextPart_001_01C782F4.A7F3B707 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Paul, =20 The full clause states the "initiator session state MUST be FAILED". The initiator's perspective is different to the target's in this scenario. The initiator has no session state (it's just been powered on), but the target is not aware the HBA has been reset. =20 The target catches up with the initiator's view by detecting the = TSIH=3D0, which means the initiator denies all knowledge of any previous session state. This is one of the specific scenarios that session reinstatement was invented to resolve. =20 Cheers Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 11:37 To: Sandars, Ken; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Isn't session re-instatement possible only for sessions that are in the failed state (no active connections)? That was my understanding of Section 5.3.5 of RFC 3720. =20 How can the target know that the session is in the failed state if it hasn't detected that the only connection was dropped? =20 Thanks, Paul =20 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Thursday, April 19, 2007 6:56 PM To: Paul Hughes; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Hi Paul, =20 That's a target problem, most likely a bug. The second login with = TSIH=3D0 tells the target to perform session reinstatement. That's jargon for "silently nuke the first session". =20 The target may be failing the login because it's internal cleanup requires more time (I/O requests from the previous session are jammed for instance). Your scenario suggests this is not likely. =20 HTH, Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 09:14 To: ips@ietf.org Subject: [Ips] detection of failed sessions to allow re-login I have a question about how a target can quickly detect session failures so that a re-login can succeed. =20 Here's my scenario: =20 1) an initiator is booting from an iSCSI target 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target 3) the HBA BIOS creates the first session, discovers the boot LUN, and reads the boot loader 4) the boot loader reads the kernel from the boot LUN 5) the kernel resets the iSCSI HBA while loading an HBA driver 6) the HBA driver attempts to create a new session =20 The problem I'm seeing is that the target is failing the login for the new session because the target thinks the first session created by the HBA BIOS is still valid (not in failed state). The HBA reset was not detected by the target soon enough for the target to know that the first session is now in the failed state when the initiator attempts to login and create the second session using the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the first session (with = TSIH=3D0). The target does not see a link down event because a switch is connected between the HBA and the target port. The target eventually detects that the first session is failed when it sends a NOP-Out PDU and receives a transport failure. Unfortunately, this occurs too late and the boot fails. =20 In my case the target is sending NOP-Out PDUs every 60 seconds. I can change that to 5 seconds, but I don't think that will fix every case. Is there a better way for the target to determine that the first session has failed so that a re-login will succeed on the first try? =20 Thanks, Paul =20 =20 =20 ------_=_NextPart_001_01C782F4.A7F3B707 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi Paul,
 
The full clause states the "initiator session state MUST be = FAILED".=20 The initiator's perspective is different to the target's in this=20 scenario. The initiator has no session state (it's just been = powered on),=20 but the target is not aware the HBA has been reset.
 
The target catches up with the initiator's view = by=20 detecting the TSIH=3D0, which means the initiator denies = all knowledge=20 of any previous session state. This is one of the specific scenarios = that=20 session reinstatement was invented to resolve.
 
Cheers
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 11:37
To: Sandars, Ken; ips@ietf.org
Subject: RE: = [Ips]=20 detection of failed sessions to allow re-login

Isn't session re-instatement possible only = for sessions that are in the failed state (no active=20 connections)?  That was my understanding of Section 5.3.5 = of RFC=20 3720.
 
How can the target know that the session is in = the failed=20 state if it hasn't detected that the only connection was=20 dropped?
 
Thanks,
Paul
 

From: Sandars, Ken=20 [mailto:ken_sandars@adaptec.com]
Sent: Thursday, April 19, = 2007 6:56=20 PM
To: Paul Hughes; ips@ietf.org
Subject: RE: [Ips]=20 detection of failed sessions to allow re-login

Hi Paul,
 
That's a target problem, most likely a bug. The = second=20 login with TSIH=3D0 tells the target to perform session reinstatement. = That's=20 jargon for "silently nuke the first session".
 
The target may be failing the login because = it's internal=20 cleanup requires more time (I/O requests from the previous session are = jammed=20 for instance). Your scenario suggests this is not = likely.
 
HTH,
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 09:14
To: ips@ietf.org
Subject: [Ips] detection of = failed=20 sessions to allow re-login

I have = a question=20 about how a target can quickly detect session failures so that = a=20 re-login can succeed.
 
Here's = my=20 scenario:
 
1) an = initiator is=20 booting from an iSCSI target
2) the = initiator is=20 using an iSCSI HBA to communicate with the iSCSI = target
3) = the HBA=20 BIOS creates the first session, discovers the boot LUN, = and reads=20 the boot loader
4) the = boot loader=20 reads the kernel from the boot LUN
5) the = kernel resets=20 the iSCSI HBA while loading an HBA driver
6) the = HBA driver=20 attempts to create a new session
 
The = problem I'm=20 seeing is that the target is failing the login for the new session = because the=20 target thinks the first session created by the HBA BIOS is still = valid (not=20 in failed state).  The HBA reset was not detected by the = target=20 soon enough for the target to know that the first session is now in the = failed=20 state when the initiator attempts to login and create the second session = using=20 the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as = the first=20 session (with TSIH=3D0).  The target does not see a link down event = because a=20 switch is connected between the HBA and the target port.  The = target=20 eventually detects that the first session is failed when it sends a = NOP-Out PDU=20 and receives a transport failure.  Unfortunately, this occurs too = late and=20 the boot fails.
 
In my = case the=20 target is sending NOP-Out PDUs every 60 seconds.  I can change that = to 5=20 seconds, but I don't think that will fix every case.  Is there a = better way=20 for the target to determine that the first session has failed so = that a=20 re-login will succeed on the first try?
 
Thanks,
Paul
 
 
 
------_=_NextPart_001_01C782F4.A7F3B707-- --===============1389310280== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1389310280==-- From ips-bounces@ietf.org Fri Apr 20 02:33:33 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HemgS-0007nS-Ua; Fri, 20 Apr 2007 02:33:28 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HemgQ-0007n7-Qq for ips-confirm+ok@megatron.ietf.org; Fri, 20 Apr 2007 02:33:26 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HemgP-0007mo-P2 for ips@ietf.org; Fri, 20 Apr 2007 02:33:25 -0400 Received: from mtagate2.de.ibm.com ([195.212.29.151]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HemgP-0004x7-1m for ips@ietf.org; Fri, 20 Apr 2007 02:33:25 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate2.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3K6XOYE064278 for ; Fri, 20 Apr 2007 06:33:24 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3K6XOF43944578 for ; Fri, 20 Apr 2007 08:33:24 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3K6XNO1028179 for ; Fri, 20 Apr 2007 08:33:23 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3K6XNKi028174; Fri, 20 Apr 2007 08:33:23 +0200 In-Reply-To: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> To: "Paul Hughes" MIME-Version: 1.0 Subject: Re: [Ips] detection of failed sessions to allow re-login X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Fri, 20 Apr 2007 09:33:20 +0300 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 20/04/2007 09:33:23, Serialize complete at 20/04/2007 09:33:23 X-Spam-Score: 0.1 (/) X-Scan-Signature: c54bc2f42d02429833c0ca4b8725abd7 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0635830645==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0635830645== Content-Type: multipart/alternative; boundary="=_alternative 0023FE21C22572C3_=" This is a multipart message in MIME format. --=_alternative 0023FE21C22572C3_= Content-Type: text/plain; charset="US-ASCII" Paul, "Paul Hughes" wrote on 20/04/2007 02:13:49: > I have a question about how a target can quickly detect session > failures so that a re-login can succeed. > > Here's my scenario: > > 1) an initiator is booting from an iSCSI target > 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target > 3) the HBA BIOS creates the first session, discovers the boot LUN, > and reads the boot loader > 4) the boot loader reads the kernel from the boot LUN > 5) the kernel resets the iSCSI HBA while loading an HBA driver > 6) the HBA driver attempts to create a new session > > The problem I'm seeing is that the target is failing the login for > the new session because the target thinks the first session created > by the HBA BIOS is still valid (not in failed state). The HBA reset > was not detected by the target soon enough for the target to know > that the first session is now in the failed state when the initiator > attempts to login and create the second session using the same > InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the > first session (with TSIH=0). The target does not see a link down > event because a switch is connected between the HBA and the target > port. The target eventually detects that the first session is > failed when it sends a NOP-Out PDU and receives a transport failure. > Unfortunately, this occurs too late and the boot fails. > > In my case the target is sending NOP-Out PDUs every 60 seconds. I > can change that to 5 seconds, but I don't think that will fix every > case. Is there a better way for the target to determine that the > first session has failed so that a re-login will succeed on the first try? > The target (if well integrated) has enough early warning that the old session/connection is gone (after all it established a new TCP connection!). But even if layering or "smart" HBAs stands in your way 3720 states clearly that a login with the same identifying elements as an existing connection (including CID) should clear the connection (implicit logout). If the connection is the only connection of the session the result is equivalent to establishing a new session (except for some corner cases in which BIOS might have made some reservations which are then inherited by new connection). So your boot should work whether the target interprets the login as a session clearing (somewhat different to what the 3270 text) or a connection clearing (that is what 3270 says). If it doesn't work the target implementation is wrong. Julo > Thanks, > Paul > > > _______________________________________________ > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips --=_alternative 0023FE21C22572C3_= Content-Type: text/html; charset="US-ASCII"
Paul,

"Paul Hughes" <phughes@pillardata.com> wrote on 20/04/2007 02:13:49:

> I have a question about how a target can quickly detect session
> failures so that a re-login can succeed.
>  
> Here's my scenario:
>  
> 1) an initiator is booting from an iSCSI target
> 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target
> 3) the HBA BIOS creates the first session, discovers the boot LUN,
> and reads the boot loader
> 4) the boot loader reads the kernel from the boot LUN
> 5) the kernel resets the iSCSI HBA while loading an HBA driver
> 6) the HBA driver attempts to create a new session
>  
> The problem I'm seeing is that the target is failing the login for
> the new session because the target thinks the first session created
> by the HBA BIOS is still valid (not in failed state).  The HBA reset
> was not detected by the target soon enough for the target to know
> that the first session is now in the failed state when the initiator
> attempts to login and create the second session using the same
> InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the
> first session (with TSIH=0).  The target does not see a link down
> event because a switch is connected between the HBA and the target
> port.  The target eventually detects that the first session is
> failed when it sends a NOP-Out PDU and receives a transport failure.
> Unfortunately, this occurs too late and the boot fails.
>  
> In my case the target is sending NOP-Out PDUs every 60 seconds.  I
> can change that to 5 seconds, but I don't think that will fix every
> case.  Is there a better way for the target to determine that the
> first session has failed so that a re-login will succeed on the first try?
>  

The target (if well integrated) has enough early warning that the old session/connection is gone (after all it established a new TCP connection!). But even if layering or "smart" HBAs stands in your way 3720 states clearly that a login with the same identifying elements as an existing connection (including CID) should clear the connection (implicit logout). If the connection is the only connection of the session the result is equivalent to establishing a new session (except for some corner cases in which BIOS might have made some reservations which are then inherited by new connection).

So your boot should work whether the target interprets the login as a session clearing (somewhat different to what the 3270 text) or a connection clearing (that is what 3270 says).

If it doesn't work the target implementation is wrong.

Julo
> Thanks,
> Paul
>  
>  
>  _______________________________________________
> Ips mailing list
> Ips@ietf.org
> https://www1.ietf.org/mailman/listinfo/ips
 --=_alternative 0023FE21C22572C3_=-- --===============0635830645== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0635830645==-- From ips-bounces@ietf.org Fri Apr 20 10:59:32 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Heua9-0000mM-2K; Fri, 20 Apr 2007 10:59:29 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Heua8-0000mA-2m for ips-confirm+ok@megatron.ietf.org; Fri, 20 Apr 2007 10:59:28 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Heua7-0000lq-Om for ips@ietf.org; Fri, 20 Apr 2007 10:59:27 -0400 Received: from mail3.pillardata.com ([209.120.231.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Heua6-0001dJ-P5 for ips@ietf.org; Fri, 20 Apr 2007 10:59:27 -0400 Received: from coex02.trans.corp ([172.18.24.19]) by mail3.pillardata.com with ESMTP; 20 Apr 2007 07:59:23 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [Ips] detection of failed sessions to allow re-login Date: Fri, 20 Apr 2007 08:59:21 -0600 Message-ID: <16236EEEF4D4264DA31C2E35E3607CFE08906EB5@coex02.trans.corp> In-Reply-To: <368FBF3D8437A748BA8222526BF9309901ACB8D2@aime2k302.adaptec.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Ips] detection of failed sessions to allow re-login Thread-Index: AceC2Geaj9izq1egS867H12pdfy0iwADTTCAAAFtARAAAfxWcAAaSfhw References: <16236EEEF4D4264DA31C2E35E3607CFE08906E1B@coex02.trans.corp> <368FBF3D8437A748BA8222526BF9309901ACB8A3@aime2k302.adaptec.com> <16236EEEF4D4264DA31C2E35E3607CFE08906E5C@coex02.trans.corp> <368FBF3D8437A748BA8222526BF9309901ACB8D2@aime2k302.adaptec.com> From: "Paul Hughes" To: "Sandars, Ken" , X-Spam-Score: 0.0 (/) X-Scan-Signature: 9c7d7a899dc8f3389bf7ace6f0ad8e29 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0623384806==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0623384806== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C7835C.7B47F772" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7835C.7B47F772 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks Ken, that clears up my confusion. =20 Paul ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Thursday, April 19, 2007 8:36 PM To: Paul Hughes; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Hi Paul, =20 The full clause states the "initiator session state MUST be FAILED". The initiator's perspective is different to the target's in this scenario. The initiator has no session state (it's just been powered on), but the target is not aware the HBA has been reset. =20 The target catches up with the initiator's view by detecting the = TSIH=3D0, which means the initiator denies all knowledge of any previous session state. This is one of the specific scenarios that session reinstatement was invented to resolve. =20 Cheers Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 11:37 To: Sandars, Ken; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Isn't session re-instatement possible only for sessions that are in the failed state (no active connections)? That was my understanding of Section 5.3.5 of RFC 3720. =20 How can the target know that the session is in the failed state if it hasn't detected that the only connection was dropped? =20 Thanks, Paul =20 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Thursday, April 19, 2007 6:56 PM To: Paul Hughes; ips@ietf.org Subject: RE: [Ips] detection of failed sessions to allow re-login Hi Paul, =20 That's a target problem, most likely a bug. The second login with = TSIH=3D0 tells the target to perform session reinstatement. That's jargon for "silently nuke the first session". =20 The target may be failing the login because it's internal cleanup requires more time (I/O requests from the previous session are jammed for instance). Your scenario suggests this is not likely. =20 HTH, Ken ________________________________ From: Paul Hughes [mailto:phughes@pillardata.com]=20 Sent: Friday, 20 April 2007 09:14 To: ips@ietf.org Subject: [Ips] detection of failed sessions to allow re-login I have a question about how a target can quickly detect session failures so that a re-login can succeed. =20 Here's my scenario: =20 1) an initiator is booting from an iSCSI target 2) the initiator is using an iSCSI HBA to communicate with the iSCSI target 3) the HBA BIOS creates the first session, discovers the boot LUN, and reads the boot loader 4) the boot loader reads the kernel from the boot LUN 5) the kernel resets the iSCSI HBA while loading an HBA driver 6) the HBA driver attempts to create a new session =20 The problem I'm seeing is that the target is failing the login for the new session because the target thinks the first session created by the HBA BIOS is still valid (not in failed state). The HBA reset was not detected by the target soon enough for the target to know that the first session is now in the failed state when the initiator attempts to login and create the second session using the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as the first session (with = TSIH=3D0). The target does not see a link down event because a switch is connected between the HBA and the target port. The target eventually detects that the first session is failed when it sends a NOP-Out PDU and receives a transport failure. Unfortunately, this occurs too late and the boot fails. =20 In my case the target is sending NOP-Out PDUs every 60 seconds. I can change that to 5 seconds, but I don't think that will fix every case. Is there a better way for the target to determine that the first session has failed so that a re-login will succeed on the first try? =20 Thanks, Paul =20 =20 =20 ------_=_NextPart_001_01C7835C.7B47F772 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thanks Ken, that clears up my=20 confusion.
 
Paul

From: Sandars, Ken=20 [mailto:ken_sandars@adaptec.com]
Sent: Thursday, April 19, = 2007 8:36=20 PM
To: Paul Hughes; ips@ietf.org
Subject: RE: [Ips]=20 detection of failed sessions to allow re-login

Hi Paul,
 
The full clause states the "initiator session state MUST be = FAILED".=20 The initiator's perspective is different to the target's in this=20 scenario. The initiator has no session state (it's just been = powered on),=20 but the target is not aware the HBA has been reset.
 
The target catches up with the initiator's view = by=20 detecting the TSIH=3D0, which means the initiator denies = all knowledge=20 of any previous session state. This is one of the specific scenarios = that=20 session reinstatement was invented to resolve.
 
Cheers
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 11:37
To: Sandars, Ken; ips@ietf.org
Subject: RE: = [Ips]=20 detection of failed sessions to allow re-login

Isn't session re-instatement possible only = for sessions that are in the failed state (no active=20 connections)?  That was my understanding of Section 5.3.5 = of RFC=20 3720.
 
How can the target know that the session is in = the failed=20 state if it hasn't detected that the only connection was=20 dropped?
 
Thanks,
Paul
 

From: Sandars, Ken=20 [mailto:ken_sandars@adaptec.com]
Sent: Thursday, April 19, = 2007 6:56=20 PM
To: Paul Hughes; ips@ietf.org
Subject: RE: [Ips]=20 detection of failed sessions to allow re-login

Hi Paul,
 
That's a target problem, most likely a bug. The = second=20 login with TSIH=3D0 tells the target to perform session reinstatement. = That's=20 jargon for "silently nuke the first session".
 
The target may be failing the login because = it's internal=20 cleanup requires more time (I/O requests from the previous session are = jammed=20 for instance). Your scenario suggests this is not = likely.
 
HTH,
Ken

From: Paul Hughes=20 [mailto:phughes@pillardata.com]
Sent: Friday, 20 April 2007=20 09:14
To: ips@ietf.org
Subject: [Ips] detection of = failed=20 sessions to allow re-login

I have = a question=20 about how a target can quickly detect session failures so that = a=20 re-login can succeed.
 
Here's = my=20 scenario:
 
1) an = initiator is=20 booting from an iSCSI target
2) the = initiator is=20 using an iSCSI HBA to communicate with the iSCSI = target
3) = the HBA=20 BIOS creates the first session, discovers the boot LUN, = and reads=20 the boot loader
4) the = boot loader=20 reads the kernel from the boot LUN
5) the = kernel resets=20 the iSCSI HBA while loading an HBA driver
6) the = HBA driver=20 attempts to create a new session
 
The = problem I'm=20 seeing is that the target is failing the login for the new session = because the=20 target thinks the first session created by the HBA BIOS is still = valid (not=20 in failed state).  The HBA reset was not detected by the = target=20 soon enough for the target to know that the first session is now in the = failed=20 state when the initiator attempts to login and create the second session = using=20 the same InitiatorName, ISID, TargetName, and TargetPortalGroupTag as = the first=20 session (with TSIH=3D0).  The target does not see a link down event = because a=20 switch is connected between the HBA and the target port.  The = target=20 eventually detects that the first session is failed when it sends a = NOP-Out PDU=20 and receives a transport failure.  Unfortunately, this occurs too = late and=20 the boot fails.
 
In my = case the=20 target is sending NOP-Out PDUs every 60 seconds.  I can change that = to 5=20 seconds, but I don't think that will fix every case.  Is there a = better way=20 for the target to determine that the first session has failed so = that a=20 re-login will succeed on the first try?
 
Thanks,
Paul
 
 
 
------_=_NextPart_001_01C7835C.7B47F772-- --===============0623384806== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0623384806==-- From ips-bounces@ietf.org Sat Apr 21 09:48:01 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HfFwV-0001iV-1N; Sat, 21 Apr 2007 09:47:59 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HfFwU-0001ct-4D for ips-confirm+ok@megatron.ietf.org; Sat, 21 Apr 2007 09:47:58 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HfFwT-0001Zd-J9 for ips@ietf.org; Sat, 21 Apr 2007 09:47:57 -0400 Received: from mexforward.lss.emc.com ([128.222.32.20]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1HfFwR-0007wA-Ih for ips@ietf.org; Sat, 21 Apr 2007 09:47:56 -0400 Received: from mailhub.lss.emc.com (sesha.lss.emc.com [10.254.144.12]) by mexforward.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3LDlteX023005 for ; Sat, 21 Apr 2007 09:47:55 -0400 (EDT) Received: from corpussmtp3.corp.emc.com (corpussmtp3.corp.emc.com [10.254.64.53]) by mailhub.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3LDlrIU001634 for ; Sat, 21 Apr 2007 09:47:54 -0400 (EDT) From: Black_David@emc.com Received: from CORPUSMX20A.corp.emc.com ([128.221.62.12]) by corpussmtp3.corp.emc.com with Microsoft SMTPSVC(6.0.3790.1830); Sat, 21 Apr 2007 09:47:52 -0400 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Sat, 21 Apr 2007 09:47:52 -0400 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: iSNS MIB approval Thread-Index: AceEG6fXYAQlk+g9RrKG9W+IsfvbKw== To: X-OriginalArrivalTime: 21 Apr 2007 13:47:53.0004 (UTC) FILETIME=[A88B66C0:01C7841B] X-PMX-Version: 4.7.1.128075, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.4.21.62234 X-PerlMx-Spam: Gauge=, SPAM=0%, Reason='EMC_BODY_1+ -3, EMC_FROM_0+ -3, NO_REAL_NAME 0, __C230066_P5 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_MSGID 0, __IMS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' X-Spam-Score: 0.2 (/) X-Scan-Signature: cf4fa59384e76e63313391b70cd0dd25 Cc: Black_David@emc.com Subject: [Ips] iSNS MIB approval X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org It's my pleasure to (finally) announce that the IESG has approved the iSNS MIB for publication as a proposed standard RFC. The actual announcement will appear shortly. Congratulations to the authors and many thanks to all who contributed. This is the last of the MIBs in the IP Storage (IPS) working group's program of work. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Mon Apr 23 16:48:24 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hg5SQ-0002s3-LC; Mon, 23 Apr 2007 16:48:22 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hg5SO-0002ea-3P for ips-confirm+ok@megatron.ietf.org; Mon, 23 Apr 2007 16:48:20 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hg5SN-0002bT-Ki; Mon, 23 Apr 2007 16:48:19 -0400 Received: from ns1.neustar.com ([2001:503:c779:1a::9c9a:108a]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hg5SN-0001e2-CC; Mon, 23 Apr 2007 16:48:19 -0400 Received: from stiedprstage1.ietf.org (stiedprstage1.va.neustar.com [10.31.47.10]) by ns1.neustar.com (Postfix) with ESMTP id 3CD4F2700D; Mon, 23 Apr 2007 20:48:19 +0000 (GMT) Received: from ietf by stiedprstage1.ietf.org with local (Exim 4.43) id 1Hg5SN-0005h0-4L; Mon, 23 Apr 2007 16:48:19 -0400 X-test-idtracker: no From: The IESG To: IETF-Announce Message-Id: Date: Mon, 23 Apr 2007 16:48:19 -0400 X-Spam-Score: -2.8 (--) X-Scan-Signature: 8b30eb7682a596edff707698f4a80f7d Cc: ips mailing list , Internet Architecture Board , ips chair , RFC Editor Subject: [Ips] Protocol Action: 'Definitions of Managed Objects for iSNS (Internet Storage Name Service)' to Proposed Standard X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org The IESG has approved the following document: - 'Definitions of Managed Objects for iSNS (Internet Storage Name Service) ' as a Proposed Standard This document is the product of the IP Storage Working Group. The IESG contact persons are Lars Eggert and Magnus Westerlund. A URL of this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-mib-11.txt Technical Summary The iSNS protocol provides storage name service functionality on an IP network that is being used for iSCSI or iFCP storage. This draft provides a mechanism to monitor multiple iSNS Servers, including information about registered objects in an iSNS Server. Working Group Summary This MIB was originally undertaken as a MIB for monitoring and configuration of both iSNS servers and clients, but ran into significant complexity and structural issues. The scope of the MIB was reduced to server monitoring in order, as that comprised most of the interest in (and perceived value of) the MIB. Protocol Quality The document has been reviewed for the WG by Keith McCloghrie. Personnel David Black was the document shepherd for this document. Bert Wijnen was the MIB doctor. Lars Eggert has reviewed this document for the IESG. _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Tue Apr 24 13:42:30 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgP1u-0002Uy-LL; Tue, 24 Apr 2007 13:42:18 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgP1t-0002Ut-IK for ips-confirm+ok@megatron.ietf.org; Tue, 24 Apr 2007 13:42:17 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgP1t-0002Ul-8o for ips@ietf.org; Tue, 24 Apr 2007 13:42:17 -0400 Received: from imf21aec.mail.bellsouth.net ([205.152.59.69]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgP1r-0007jO-VE for ips@ietf.org; Tue, 24 Apr 2007 13:42:17 -0400 Received: from ibm63aec.bellsouth.net ([74.245.52.54]) by imf21aec.mail.bellsouth.net with ESMTP id <20070424174215.OFJ2900.imf21aec.mail.bellsouth.net@ibm63aec.bellsouth.net> for ; Tue, 24 Apr 2007 13:42:15 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm63aec.bellsouth.net with SMTP id <20070424174214.ZNXT2263.ibm63aec.bellsouth.net@IVVTDKV0981> for ; Tue, 24 Apr 2007 13:42:14 -0400 Message-ID: <000901c78697$dd13e240$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: Date: Tue, 24 Apr 2007 13:42:00 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 x-mimeole: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.0 (/) X-Scan-Signature: 21c69d3cfc2dd19218717dbe1d974352 Subject: [Ips] iSNS and discovery session X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1610339439==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============1610339439== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0006_01C78676.55BD4A00" This is a multi-part message in MIME format. ------=_NextPart_000_0006_01C78676.55BD4A00 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable If an initiator is registered with iSNS: 1) is there a restriction that it must not use a discovery session? 2) if it is allowed to use a discovery session but the discover session = reports fewer TPG's than the PG's known via iSNS then what should the = effect be? Eddy ------=_NextPart_000_0006_01C78676.55BD4A00 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
If an initiator is registered with = iSNS:
 
1) is there a restriction that it must not use a = discovery=20 session?
2) if it is allowed to use a discovery session but = the=20 discover session reports fewer TPG's than the PG's known via = iSNS then=20 what should the effect be?
 
Eddy
------=_NextPart_000_0006_01C78676.55BD4A00-- --===============1610339439== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1610339439==-- From ips-bounces@ietf.org Tue Apr 24 15:11:06 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgQPL-0005zU-4j; Tue, 24 Apr 2007 15:10:35 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgQPJ-0005zD-Te for ips-confirm+ok@megatron.ietf.org; Tue, 24 Apr 2007 15:10:33 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgQPJ-0005yx-Gr for ips@ietf.org; Tue, 24 Apr 2007 15:10:33 -0400 Received: from mtagate3.de.ibm.com ([195.212.29.152]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgQPI-0004s9-J9 for ips@ietf.org; Tue, 24 Apr 2007 15:10:33 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate3.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3OJAVRX041938 for ; Tue, 24 Apr 2007 19:10:31 GMT Received: from d12av04.megacenter.de.ibm.com (d12av04.megacenter.de.ibm.com [9.149.165.229]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3OJAVPp4124914 for ; Tue, 24 Apr 2007 21:10:31 +0200 Received: from d12av04.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3OJAVkb005072 for ; Tue, 24 Apr 2007 21:10:31 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3OJAVr6005069 for ; Tue, 24 Apr 2007 21:10:31 +0200 To: ips@ietf.org MIME-Version: 1.0 X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Tue, 24 Apr 2007 15:10:29 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 24/04/2007 22:10:30, Serialize complete at 24/04/2007 22:10:30 X-Spam-Score: 0.8 (/) X-Scan-Signature: 918f4bd8440e8de4700bcf6d658bc801 Subject: [Ips] Recent comments about FCoE and iSCSI X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0566959458==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0566959458== Content-Type: multipart/alternative; boundary="=_alternative 0069527B852572C7_=" This is a multipart message in MIME format. --=_alternative 0069527B852572C7_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Dear All, The trade press is lately full with comments about the latest and greatest = reincarnation of Fiber Channel over ethernet. It made me try and summarize all the long and hot debates that preceded=20 the advent of iSCSI. Although FCoE proponents make it look like no debate preceded iSCSI that=20 was not so - FCoE was considered even then and was dropped as a dumb idea. Here is a summary (as afar as I can remember) of the main arguments. They=20 are not bad arguments even in retrospect and technically FCoE doesn't look = better than it did then. Feel free to use this material in a nay form. I expect this group to=20 seriously expand my arguments and make them public - in personal or=20 collective form. And do not forget - it is a technical dispute - although we all must have=20 some doubts about the way it is pursued. Regards, Julo --------------------------------------------------------------------- What a piece of nostalgia :-) Around 1997 when a team at IBM Research (Haifa and Almaden) started=20 looking at connecting storage to servers using the "regular network" (the=20 ubiquitous LAN) we considered many alternatives (another team even had a=20 look at ATM - still a computer network candidate at the time). I won't get = you over all of our rationale (and we went over some of them again at the=20 end of 1999 with a team from CISCO before we convened the first IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of=20 the reasons we choose to drop Fiber Channel over raw Ethernet where=20 multiple: Fiber Channel Protocol (SCSI over Fiber Channel Link) is "mildly"=20 effective because: it implements endpoints in a dedicated engine (Offload) it has no transport layer (recovery is done at the application layer under = the assumption that the error rate will be very low) the network is limited in physical span and logical span (number of=20 switches) flow-control/congestion control is achieved with a mechanism adequate for=20 a limited span network (credits). The packet loss rate is almost nil and=20 that allows FCP to avoid using a transport (end-to-end) layer FCP she switches are simple (addresses are local and the memory=20 requirements cam be limited through the credit mechanism) However FCP endpoints are inherently costlier than simple NICs ? the cost=20 argument (initiators are more expensive) The credit mechanisms is highly unstable for large networks (check switch=20 vendors planning docs for the network diameter limits) ? the scaling=20 argument The assumption of low losses due to errors might radically change when=20 moving from 1 to 10 Gb/s ? the scaling argument Ethernet has no credit mechanism and any mechanism with a similar effect=20 increases the end point cost. Building a transport layer in the protocol=20 stack has always been the preferred choice of the networking community ?=20 the community argument The "performance penalty" of a complete protocol stack has always been=20 overstated (and overrated). Advances in protocol stack implementation and=20 finer tuning of the congestion control mechanisms make conventional TCP/IP = performing well even at 10 Gb/s and over. Moreover the multicore=20 processors that become dominant on the computing scene have enough compute = cycles available to make any "offloading" possible as a mere code=20 restructuring exercise (see the stack reports from Intel, IBM etc.) Building on a complete stack makes available a wealth of operational and=20 management mechanisms built over the years by the networking community=20 (routing, provisioning, security, service location etc.) ? the community=20 argument Higher level storage access over an IP network is widely available and=20 having both block and file served over the same connection with the same=20 support and management structure is compelling ? the community argument Highly efficient networks are easy to build over IP with optimal (shortest = path) routing while Layer 2 networks use bridging and are limited by the=20 logical tree structure that bridges must follow. The effort to combine=20 routers and bridges (rbridges) is promising to change that but it will=20 take some time to finalize (and we don't know exactly how it will=20 operate). Untill then the scale of Layer 2 network is going to seriously=20 limited ? the scaling argument As a side argument ? a performance comparison made in 1998 showed SCSI=20 over TCP (a predecessor of the later iSCSI) to perform better than FCP at=20 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us=20 to take the path that lead to iSCSI ? and we used plain vanilla x86=20 servers with plain-vanilla NICs and Linux (with similar measurements=20 conducted on Windows). The networking and storage community acknowledged those arguments and=20 developed iSCSI and the companion protocols for service discovery, boot=20 etc. The community also acknowledged the need to support existing=20 infrastructure and extend it in a reasonable fashion and developed 2=20 protocols iFCP (to support hosts with FCP drivers and IP connections to=20 connect to storage by a simple conversion from FCP to TCP packets) FCPIP=20 to extend the reach of FCP through IP (connects FCP islands through TCP=20 links). Both have been=20 implemented and their foundation is solid. The current attempt of developing a "new-age" FCP over an Ethernet link is = going against most of the arguments that have given us iSCSI etc. It ignores the networking layering practice, build an application protocol = directly above a link and thus limits scaling, mandates elements at the=20 link layer and application layer that make applications more expensive and = leaves aside the whole "ecosystem" that accompanies TCP/IP (and not=20 Ethernet). In some related effort (and at a point also when developing iSCSI) we=20 considered also moving away from SCSI (like some "no standardized" but=20 popular in some circles software did ? e.g., NBP) but decided against.=20 SCSI is a mature and well understood access architecture for block storage = and is implemented by many device vendors. Moving away from it would not=20 have been justified at the time. --=_alternative 0069527B852572C7_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Dear All,

The trade press is lately full with comments about the latest and greatest reincarnation of Fiber Channel over ethernet.
It made me try and summarize all the long and hot debates that preceded the advent of iSCSI.
Although FCoE proponents make it look like no debate preceded iSCSI that was not so - FCoE was considered even then and was dropped as a dumb idea.

Here is a summary (as afar as I can remember) of the main arguments. They are not bad arguments even in retrosp= ect and technically FCoE doesn't look better than it did then.

Feel free to use this material in a nay form. I expect this group to seriously  expand my arguments and make them public - in personal or collective form.

And do not forget - it is a technical dispute - although we all must have some doubts about the way it is pursued= .

Regards,
Julo

------------------------------------= ---------------------------------

What a piece of nostalgia :-)

Around 1997 when a team at IBM Research (= Haifa and Almaden) started looking at connecting storage to servers using the "regular network" (the ubiquitous LAN) we considered many alterna= tives (another team even had a look at ATM - still a computer network candidate at the time). I won't get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple:
  • Fiber Channel Protocol (SCSI over Fiber C= hannel Link) is "mildly" effective because:
    • it implements endpoints in a dedicated en= gine (Offload)
    • it has no transport layer (recovery is do= ne at the application layer under the assumption that the error rate will be very low)
    • the network is limited in physical span a= nd logical span (number of switches)
    • flow-control/congestion control is achiev= ed with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer
    • FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechani= sm)
    • However FCP endpoints are inherently cost= lier than simple NICs – t= he cost argument (initiators are more expensive)
    • The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) – t= he scaling argument
    • The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s – the scaling argument
    • Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has always been the preferred choice of the networking community – the community argument
    • The "performance penalty" of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computi= ng scene have enough compute cycles available to make any "offloading&quo= t; possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.)
    • Building on a complete stack makes availa= ble a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service locat= ion etc.) – the communit= y argument
    • Higher level storage access over an IP ne= twork is widely available and having both block and file served over the same connection with the same support and management structure is compelling – the community argu= ment
    • Highly efficient networks are easy to bui= ld over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don't know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited – the scaling argument


    As a side argument – a performance = comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI – and = we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows).
    The networking and storage community ackn= owledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc.

    The community also acknowledged the need to support existing infrastructure and extend it in a reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been
    implemented and their foundation is solid= .

    The current attempt of developing a "= ;new-age" FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc.

    It ignores the networking layering practi= ce, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applica= tions more expensive and leaves aside the whole "ecosystem" that accomp= anies TCP/IP (and not Ethernet).

    In some related effort (and at a point al= so when developing iSCSI) we considered also moving away from SCSI (like some "no standardized" but popular in some circles software did – e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time.
--=_alternative 0069527B852572C7_=-- --===============0566959458== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0566959458==-- From ips-bounces@ietf.org Wed Apr 25 08:10:04 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HggJr-0003KS-RV; Wed, 25 Apr 2007 08:09:59 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HggJq-0003Ji-LD for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 08:09:58 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HggJq-0003Ja-BS for ips@ietf.org; Wed, 25 Apr 2007 08:09:58 -0400 Received: from mtagate3.de.ibm.com ([195.212.29.152]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HggJo-0005JI-IS for ips@ietf.org; Wed, 25 Apr 2007 08:09:58 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate3.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3PC9tYQ141956 for ; Wed, 25 Apr 2007 12:09:55 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3PC9tx73166390 for ; Wed, 25 Apr 2007 14:09:55 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3PC9jOR016142 for ; Wed, 25 Apr 2007 14:09:45 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3PC9jMl016126; Wed, 25 Apr 2007 14:09:45 +0200 In-Reply-To: <39BA3BC178B4394DB184389E88A97F8C022C57F3@hq-exch-1.corp.brocade.com> To: "John Hufferd" MIME-Version: 1.0 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Wed, 25 Apr 2007 08:09:43 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 25/04/2007 15:09:44, Serialize complete at 25/04/2007 15:09:44 X-Spam-Score: 0.5 (/) X-Scan-Signature: 441502cf25997484ff0b8b79626c6b69 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0754886121==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0754886121== Content-Type: multipart/alternative; boundary="=_alternative 0042CBA9852572C8_=" This is a multipart message in MIME format. --=_alternative 0042CBA9852572C8_= Content-Type: text/plain; charset="US-ASCII" "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian, > To be sure you understand our position; Brocade is pushing iSCSI as an > outreach protocol from the Data Center. We also believe iSCSI is very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our new > iSCSI and TOE offload HBAs. > > When I say iSCSI is an outreach protocol, this is a statement that iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also applies > to Desktops and Laptop systems, and systems at distance. > You make it sound like: most of the servers in the world have their storage on the network - and that is not the case FCP is basically better performing than iSCSI - and that is not true either Gatewaying is expensive - and it is perhaps so but only if you are completely relying on FCP storage (and there are plenty of good iSCSI vendors of storage)and pushing the price on the servers is not cheap either - at least not for the server buyer > Now with that positioning, it is important to understand the limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on the > market, it is not cost feasible to replace all the server connectivity > to FC storage with iSCSI for the hundred to thousands of servers in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. > > The issue is the server requirement to have a single connection type to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are too > high for iSCSI to be used as the "normal" server connect into a FC based > Fabric. That is true for the current 1GE; and for 10GE the cost is just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. > > Now with respect to FC over Ethernet the important thing to understand > is that it is not Ethernet as we have known it up to know. The Ethernet > we are talking about is a type of Ethernet that can only be deployed in > a constrained environment such as a Data Center. This form of Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or Intranet > type of Ethernet. > > FCoE is all about using the DCE (CEE) to carry FC frames. The rest of > the Host and storage stack remain the same, the functions and features > of the switches also remain the same and add the capability to provide > Cluster Message Switching which has latency close to InfiniBand speeds. > > > Because the FC frames are transported to the switches intact via a DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or send FC > frames to DCE FCoE devices, just like one would do if it was all FC. And > all this is done while performing Cluster message switching and general > message trucking to the IP outfacing network. > The rosy future of the yet to appear DCE/CEE and a layer 2 only world. First you have some terms confused: Bridging is the term commonly used for Layer-2 switching and routing is therm used for layer-3 (switching). Bridging has some advantages (less management) that have created a movement towards an enterprise wide LAN. But this has a long way to go and will require significant equipment and protocol changes. Even its proponents do not call for transportless networks, lossless networks etc. The second trouble with your argument is that there are no known large scale networking technologies that really work at full speed (high speed) and are lossless (flow-controlled) and errorless like FCoE assumes. The TCP/IP has solved this issue for every generation using the proven end-to-end principle (and is doing so now). And it is not by chance so and that is why all networking applications are built above layer-3 and not dropping layer-3 (like FCoE) does. Although I can understand the DCE arguments as a management statement I would prefer like any rational engineer, to base my building blocks on structures that are proven and long lasting. And those are still the end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS TWG has developed the iFCP that does exactly what FCoE claims to do an a better base. > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on the > Data Center Fabric. And it maintains all the FC Fabric Services, and all > the same Storage Management processes. > > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least a > decade or more. > It is server cost statement. It costs nothing to connect a modern server to ethernet it will cost a bundle to connect to FCoE and it will force users in short lived bad solutions. > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That said; > we do not see FCoE going beyond the constraints of the Data Center. Data Centers now grow to tens of thousands of nodes. There is no layer-2 technology for errorless/lossless operation at this scale and there is no good reason to pursue one. The only possible reason (good reason) is the bridging infrastructure but that infrastructure has a completely different rationale than the flowcontrol. > This issue and message is quite different from the issues and messages > we struggled with when we started iSCSI. There is a consortium of folks > both working on the DCE (CEE) and the FCoE. Without the DCE the FCoE > will not happen. > > None of the above cancels out the value of iSCSI in numerous > environments. > > iSCSI is good for all environments. Business consideration (and some politics) keep it form "exploding" and large storage vendors are completely indifferent to the network connection they are using. You and I have also slightly different views of DCE. I expect DCE (that still has a way to go) to improve the QoS in the data-center (and for storage too). You expect it to bring the loss rates down to the levels that FCP assumes (FCP has no transport layer) and that is probably a pipe dream. Todays transport solution for loss mitigation are far more cost effective - and that's why iFCP is a better proposition as a transition technology than FCoE and iSCSI with gateways is propably better in the long run. > > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > > -----Original Message----- > From: John Hufferd > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI > > Julian, > I think you are wrong on this one. The arguments are quite different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM technology > group through FCoE as it is being placed in our plans). > I will send you more info when I get to my computer. But you probably > were sent the Brocade charts. Please review them and I will follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > > > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI > > > Dear All, > > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet. > It made me try and summarize all the long and hot debates that preceded > the advent of iSCSI. > Although FCoE proponents make it look like no debate preceded iSCSI that > was not so - FCoE was considered even then and was dropped as a dumb > idea. > > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then. > > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form. > > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued. > > Regards, > Julo > > --------------------------------------------------------------------- > > What a piece of nostalgia :-) > > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team even > had a look at ATM - still a computer network candidate at the time). I > won't get you over all of our rationale (and we went over some of them > again at the end of 1999 with a team from CISCO before we convened the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the > rest) but some of the reasons we choose to drop Fiber Channel over raw > Ethernet where multiple: > > > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because: > > * it implements endpoints in a dedicated engine (Offload) > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be very > low) > * the network is limited in physical span and logical span > (number of switches) > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer > > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism) > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive) > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network diameter > limits) - the scaling argument > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling argument > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of the > networking community - the community argument > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack reports > from Intel, IBM etc.) > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service location > etc.) - the community argument > * Higher level storage access over an IP network is widely > available and having both block and file served over the same connection > with the same support and management structure is compelling - the > community argument > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging and > are limited by the logical tree structure that bridges must follow. The > effort to combine routers and bridges (rbridges) is promising to change > that but it will take some time to finalize (and we don't know exactly > how it will operate). Untill then the scale of Layer 2 network is going > to seriously limited - the scaling argument > > > > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That > was what convinced us to take the path that lead to iSCSI - and we used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows). > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service > discovery, boot etc. > > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections to > connect to storage by a simple conversion from FCP to TCP packets) FCPIP > to extend the reach of FCP through IP (connects FCP islands through TCP > links). Both have been > implemented and their foundation is solid. > > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given us > iSCSI etc. > > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet). > > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the time. > --=_alternative 0042CBA9852572C8_= Content-Type: text/html; charset="US-ASCII"

"John Hufferd" <jhufferd@Brocade.COM> wrote on 25/04/2007 02:45:51:

> Julian,
> To be sure you understand our position; Brocade is pushing iSCSI as an
> outreach protocol from the Data Center. We also believe iSCSI is very
> useful for installations that do not have a Fibre Channel
> infrastructure, and in that case we will be able to sell them our new
> iSCSI and TOE offload HBAs.  
>
> When I say iSCSI is an outreach protocol, this is a statement that iSCSI
> is very important to connect "stranded" servers to the Fibre Channel
> Fabric.  That is, we sell iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers (software or HBA iSCSI initiators) to connect to the
> Enterprise "Bet Your Business" FC Storage.  This of course also applies
> to Desktops and Laptop systems, and systems at distance.
>

You make it sound like:
  1. most of the servers in the world have their storage on the network - and that is not the case
  2. FCP is basically better performing than iSCSI - and that is not true either
  3. Gatewaying is expensive - and it is perhaps so but only if you are completely relying on FCP storage (and there are plenty of good iSCSI vendors of storage)and pushing the price on the servers is not cheap either - at least not for the server buyer

> Now with that positioning, it is important to understand the limitation
> to this strategy.  The primary problem is that iSCSI to FC Bridging
> (Gatewaying) is relatively expensive (compared to simple FC
> connections).  Though we have some of the best priced Gateways on the
> market, it is not cost feasible to replace all the server connectivity
> to FC storage with iSCSI for the hundred to thousands of servers in the
> Data Center.  And since, if there is to be a consolidated Network
> connection to the servers in the Data Center, there must be an
> evolutionary replacement of Server Connections to Storage.  That means
> there must be a bridge/Gateway approach.  And as I mentioned before,
> there is just too much cost in the iSCSI to FC Gateway.
>
> The issue is the server requirement to have a single connection type to
> handle cluster messaging, general messaging, and storage.  iSCSI is
> clearly an option for the storage, however, the gateway costs are too
> high for iSCSI to be used as the "normal" server connect into a FC based
> Fabric. That is true for the current 1GE; and for 10GE the cost is just
> out of sight.  The reason for this is the requirement for TCP/IP
> termination and re-initiation with FC at the Gateway.
>
> Now with respect to FC over Ethernet the important thing to understand
> is that it is not Ethernet as we have known it up to know.  The Ethernet
> we are talking about is a type of Ethernet that can only be deployed in
> a constrained environment such as a Data Center.  This form of Ethernet
> is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced
> Ethernet).  This form of Ethernet is a Loss-less type Ethernet, with
> multi-priority and Flow Control.  This is NOT an Internet or Intranet
> type of Ethernet.
>
> FCoE is all about using the DCE (CEE) to carry FC frames.  The rest of
> the Host and storage stack remain the same, the functions and features
> of the switches also remain the same and add the capability to provide
> Cluster Message Switching which has latency close to InfiniBand speeds.
>
>
> Because the FC frames are transported to the switches intact via a DCE
> frame, the Bridging, if you want to call it that, is virtually non
> existent.  Hence you can deliver the FC frames to FC devices, or send FC
> frames to DCE FCoE devices, just like one would do if it was all FC. And
> all this is done while performing Cluster message switching and general
> message trucking to the IP outfacing network.
>

The rosy future of the yet to appear DCE/CEE and a layer 2 only world.
First you have some terms confused:

Bridging is the term commonly used for Layer-2 switching and routing is therm used for layer-3 (switching).

Bridging has some advantages (less management) that have created a movement towards an enterprise wide
LAN. But this has a long way to go and will require significant equipment and protocol changes.
Even its proponents do not call for transportless networks, lossless networks etc.
The second trouble with your argument is that there are no known large scale networking technologies that
really work at full speed (high speed) and are lossless (flow-controlled) and errorless like FCoE assumes.
The TCP/IP has solved this issue for every generation using the proven end-to-end principle (and is doing so now).
And it is not by chance so and that is why all networking applications are built above layer-3 and not dropping layer-3 (like FCoE) does.

Although I can understand the DCE arguments as a management statement I would prefer like any rational engineer, to base my building blocks on structures that are proven and long lasting. And those are still the end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS TWG has developed the iFCP that does exactly what FCoE claims to do an a better base.

> This means an evolutionary process is possible to the solution of
> getting a single Fabric connection for all networks connected to a
> server, further, the process has very low interconnection cost on the
> Data Center Fabric. And it maintains all the FC Fabric Services, and all
> the same Storage Management processes.
>
> By the way, this is primarily a Server driven value statement, there
> seems to be little value in having FCoE on the storage controller.
> Therefore FC storage controllers (and FICON) will be the very last
> things that connect using FCoE and that evolution will take at least a
> decade or more.
>

It is server cost statement. It costs nothing to connect a modern server to ethernet it will cost a bundle to connect to FCoE and it will force users in short lived bad solutions.


> We see value in offering switches and Directors that can support DCE
> switching, FC switching as well as iSCSI interconnect, and the
> "Trunking" of general messaging to the Outfacing IP network.  That said;
> we do not see FCoE going beyond the constraints of the Data Center.

Data Centers now grow to tens of thousands of nodes. There is no layer-2 technology for errorless/lossless operation at this scale and there is no good reason to pursue one. The only possible reason (good reason) is the bridging infrastructure but that infrastructure has a completely different rationale than the flowcontrol.

> This issue and message is quite different from the issues and messages
> we struggled with when we started iSCSI.  There is a consortium of folks
> both working on the DCE (CEE) and the FCoE.  Without the DCE the FCoE
> will not happen.  
>
> None of the above cancels out the value of iSCSI in numerous
> environments.
>
>

iSCSI is good for all environments. Business consideration (and some politics) keep it form "exploding" and large storage vendors are completely indifferent to the network connection they are using.
You and I have also slightly different views of DCE. I expect DCE (that still has a way to go) to improve the QoS in the data-center (and for storage too). You expect it to bring the loss rates down to the levels that FCP assumes (FCP has no transport layer) and that is probably a pipe dream. Todays transport solution for loss mitigation are far more cost effective - and that's why iFCP is a better proposition as a transition technology than FCoE and iSCSI with gateways is propably better in the long run.

>
> .
> .
> .
> John L Hufferd
> Sr. Executive Director of Technology
> jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606
>  
> -----Original Message-----
> From: John Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To: 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent comments about FCoE and iSCSI
>
> Julian,
> I think you are wrong on this one.  The arguments are quite different
> then the ones we had in pre iSCSI days.  (By the way I missed you on
> today's Renato meeting/conf call where Brocade took the IBM technology
> group through FCoE as it is being placed in our plans).  
> I will send you more info when I get to my computer.  But you probably
> were sent the Brocade charts.  Please review them and I will follow up
> with more information.
> This does NOT replace iSCSI it applies only to a DataCenter envuornment
> with lossless DCE ethernet.
> --------------------------
> John L. Hufferd
> Sr. Ex. Director of Technology
> Brocade Communications Systems, Inc.
> Phone: (408) 333-5244
> Mobile: (408) 627-9606
> eMail: jhufferd@brocade.com
> (Sent from my BlackBerry Wireless)
>  
>
> ----- Original Message -----
> From: Julian Satran <Julian_Satran@il.ibm.com>
> To: ips@ietf.org <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> Subject: [Ips] Recent comments about FCoE and iSCSI
>
>
> Dear All,
>
> The trade press is lately full with comments about the latest and
> greatest reincarnation of Fiber Channel over ethernet.
> It made me try and summarize all the long and hot debates that preceded
> the advent of iSCSI.
> Although FCoE proponents make it look like no debate preceded iSCSI that
> was not so - FCoE was considered even then and was dropped as a dumb
> idea.
>
> Here is a summary (as afar as I can remember) of the main arguments.
> They are not bad arguments even in retrospect and technically FCoE
> doesn't look better than it did then.
>
> Feel free to use this material in a nay form. I expect this group to
> seriously  expand my arguments and make them public - in personal or
> collective form.
>
> And do not forget - it is a technical dispute - although we all must
> have some doubts about the way it is pursued.
>
> Regards,
> Julo
>
> ---------------------------------------------------------------------
>
> What a piece of nostalgia :-)
>
> Around 1997 when a team at IBM Research (Haifa and Almaden) started
> looking at connecting storage to servers using the "regular network"
> (the ubiquitous LAN) we considered many alternatives (another team even
> had a look at ATM - still a computer network candidate at the time). I
> won't get you over all of our rationale (and we went over some of them
> again at the end of 1999 with a team from CISCO before we convened the
> first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the
> rest) but some of the reasons we choose to drop Fiber Channel over raw
> Ethernet where multiple:
>
>
> *   Fiber Channel Protocol (SCSI over Fiber Channel Link) is
> "mildly" effective because:
>
>    *   it implements endpoints in a dedicated engine (Offload)
>    *   it has no transport layer (recovery is done at the
> application layer under the assumption that the error rate will be very
> low)
>    *   the network is limited in physical span and logical span
> (number of switches)
>    *   flow-control/congestion control is achieved with a
> mechanism adequate for a limited span network (credits). The packet loss
> rate is almost nil and that allows FCP to avoid using a transport
> (end-to-end) layer
>
>    *   FCP she switches are simple (addresses are local and the
> memory requirements cam be limited through the credit mechanism)
>    *   However FCP endpoints are inherently costlier than
> simple NICs - the cost argument (initiators are more expensive)
>    *   The credit mechanisms is highly unstable for large
> networks (check switch vendors planning docs for the network diameter
> limits) - the scaling argument
>    *   The assumption of low losses due to errors might
> radically change when moving from 1 to 10 Gb/s - the scaling argument
>    *   Ethernet has no credit mechanism and any mechanism with
> a similar effect increases the end point cost. Building a transport
> layer in the protocol stack has always been the preferred choice of the
> networking community - the community argument
>    *   The "performance penalty" of a complete protocol stack
> has always been overstated (and overrated). Advances in protocol stack
> implementation and finer tuning of the congestion control mechanisms
> make conventional TCP/IP performing well even at 10 Gb/s and over.
> Moreover the multicore processors that become dominant on the computing
> scene have enough compute cycles available to make any "offloading"
> possible as a mere code restructuring exercise (see the stack reports
> from Intel, IBM etc.)
>    *   Building on a complete stack makes available a wealth of
> operational and management mechanisms built over the years by the
> networking community (routing, provisioning, security, service location
> etc.) - the community argument
>    *   Higher level storage access over an IP network is widely
> available and having both block and file served over the same connection
> with the same support and management structure is compelling - the
> community argument
>    *   Highly efficient networks are easy to build over IP with
> optimal (shortest path) routing while Layer 2 networks use bridging and
> are limited by the logical tree structure that bridges must follow. The
> effort to combine routers and bridges (rbridges) is promising to change
> that but it will take some time to finalize (and we don't know exactly
> how it will operate). Untill then the scale of Layer 2 network is going
> to seriously limited - the scaling argument
>
>
>
>       As a side argument - a performance comparison made in
> 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform
> better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That
> was what convinced us to take the path that lead to iSCSI - and we used
> plain vanilla x86 servers with plain-vanilla NICs and Linux (with
> similar measurements conducted on Windows).
>    The networking and storage community acknowledged those
> arguments and developed iSCSI and the companion protocols for service
> discovery, boot etc.
>    
>    The community also acknowledged the need to support existing
> infrastructure and extend it in a reasonable fashion and developed 2
> protocols iFCP (to support hosts with FCP drivers and IP connections to
> connect to storage by a simple conversion from FCP to TCP packets) FCPIP
> to extend the reach of FCP through IP (connects FCP islands through TCP
> links). Both have been
>    implemented and their foundation is solid.
>    
>    The current attempt of developing a "new-age" FCP over an
> Ethernet link is going against most of the arguments that have given us
> iSCSI etc.
>    
>    It ignores the networking layering practice, build an
> application protocol directly above a link and thus limits scaling,
> mandates elements at the link layer and application layer that make
> applications more expensive and leaves aside the whole "ecosystem" that
> accompanies TCP/IP (and not Ethernet).
>    
>    In some related effort (and at a point also when developing
> iSCSI) we considered also moving away from SCSI (like some "no
> standardized" but popular in some circles software did - e.g., NBP) but
> decided against. SCSI is a mature and well understood access
> architecture for block storage and is implemented by many device
> vendors. Moving away from it would not have been justified at the time.
>    
 --=_alternative 0042CBA9852572C8_=-- --===============0754886121== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0754886121==-- From ips-bounces@ietf.org Wed Apr 25 10:07:32 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgi9X-0006Dk-Mh; Wed, 25 Apr 2007 10:07:27 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgi9V-0006DG-UO for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 10:07:25 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgi9V-0006D5-KQ for ips@ietf.org; Wed, 25 Apr 2007 10:07:25 -0400 Received: from imf19aec.mail.bellsouth.net ([205.152.59.67]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgi9U-0000aQ-4m for ips@ietf.org; Wed, 25 Apr 2007 10:07:25 -0400 Received: from ibm60aec.bellsouth.net ([74.245.52.54]) by imf19aec.mail.bellsouth.net with ESMTP id <20070425140723.GJBZ29281.imf19aec.mail.bellsouth.net@ibm60aec.bellsouth.net> for ; Wed, 25 Apr 2007 10:07:23 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm60aec.bellsouth.net with SMTP id <20070425140721.IMXH24090.ibm60aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 10:07:21 -0400 Message-ID: <001801c78743$0b25e560$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "John Hufferd" , "Julian Satran" References: Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 10:07:21 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.8 (/) X-Scan-Signature: 31c5f48314ee2074d17118e5d97ef478 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0644468803==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0644468803== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0015_01C78721.83D17000" This is a multi-part message in MIME format. ------=_NextPart_000_0015_01C78721.83D17000 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs. Biggest knock is that it will not route on the "global" scale like = TCP/IP would. Eddy ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as = an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1.. most of the servers in the world have their storage on the = network - and that is not the case=20 2.. FCP is basically better performing than iSCSI - and that is not = true either=20 3.. Gatewaying is expensive - and it is perhaps so but only if you = are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers in = the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type = to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be deployed = in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest = of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all FC. = And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world. = First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known large = scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven = end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications = are built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement = I would prefer like any rational engineer, to base my building blocks on = structures that are proven and long lasting. And those are still the = end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS = TWG has developed the iFCP that does exactly what FCoE claims to do an a = better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, and = all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least = a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will follow = up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI = that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the time). = I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet = loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of = the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging = and > are limited by the logical tree structure that bridges must follow. = The > effort to combine routers and bridges (rbridges) is promising to = change > that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections = to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands through = TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given = us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) = but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_0015_01C78721.83D17000 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Basically, it = is=20 sending FC frames over Ethernet. This localizes the traffic unless you = route=20 based on MAC addresses. So you send 2146 bytes of FC frame plus 18 bytes = of=20 Ethernet overhead as FCoE =93standard=94 packet. 18 bytes of Ethernet = gets stripped=20 and you have straight FC frame that can go through any FC network. Now = you can=20 have 10G Ethernet pipes into existing FC SANs. Limited market potential = as far=20 as I can see. The key argument is it much easier to implement than iSCSI = and=20 also has less overhead and uses all the benefits of FC. End to End = credits are=20 simulated using PAUSE command on Ethernet and MAC addresses are mapped = into=20 WWNs.

Biggest knock = is that=20 it will not route on the =93global=94 scale like TCP/IP = would.

Eddy

 -----=20 Original Message -----

From:=20 Julian Satran
Sent: Wednesday, April 25, 2007 = 8:09=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI



"John Hufferd" <jhufferd@Brocade.COM> = wrote on=20 25/04/2007 02:45:51:

> Julian,
> To be sure you = understand=20 our position; Brocade is pushing iSCSI as an
> outreach protocol = from=20 the Data Center. We also believe iSCSI is very
> useful for=20 installations that do not have a Fibre Channel
> infrastructure, = and in=20 that case we will be able to sell them our new
> iSCSI and TOE = offload=20 HBAs.  
>
> When I say iSCSI is an outreach = protocol, this=20 is a statement that iSCSI
> is very important to connect = "stranded"=20 servers to the Fibre Channel
> Fabric.  That is, we sell=20 iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers = (software=20 or HBA iSCSI initiators) to connect to the
> Enterprise "Bet = Your=20 Business" FC Storage.  This of course also applies
> to = Desktops=20 and Laptop systems, and systems at distance.
>=20

You make it sound like:=20
  1. most of the servers in the world = have their=20 storage on the network - and that is not the case=20
  2. FCP is basically better performing = than iSCSI -=20 and that is not true either=20
  3. Gatewaying is expensive - and it is = perhaps so=20 but only if you are completely relying on FCP storage (and there are = plenty=20 of good iSCSI vendors of storage)and pushing the price on the = servers is not=20 cheap either - at least not for the server=20 buyer

> Now with that=20 positioning, it is important to understand the limitation
> to = this=20 strategy.  The primary problem is that iSCSI to FC = Bridging
>=20 (Gatewaying) is relatively expensive (compared to simple FC
>=20 connections).  Though we have some of the best priced Gateways on = the
> market, it is not cost feasible to replace all the server=20 connectivity
> to FC storage with iSCSI for the hundred to = thousands of=20 servers in the
> Data Center.  And since, if there is to be = a=20 consolidated Network
> connection to the servers in the Data = Center,=20 there must be an
> evolutionary replacement of Server = Connections to=20 Storage.  That means
> there must be a bridge/Gateway = approach.=20  And as I mentioned before,
> there is just too much cost = in the=20 iSCSI to FC Gateway.
>
> The issue is the server = requirement to=20 have a single connection type to
> handle cluster messaging, = general=20 messaging, and storage.  iSCSI is
> clearly an option for = the=20 storage, however, the gateway costs are too
> high for iSCSI to = be used=20 as the "normal" server connect into a FC based
> Fabric. That is = true=20 for the current 1GE; and for 10GE the cost is just
> out of = sight.=20  The reason for this is the requirement for TCP/IP
> = termination=20 and re-initiation with FC at the Gateway.
>
> Now with = respect to=20 FC over Ethernet the important thing to understand
> is that it = is not=20 Ethernet as we have known it up to know.  The Ethernet
> we = are=20 talking about is a type of Ethernet that can only be deployed = in
> a=20 constrained environment such as a Data Center.  This form of=20 Ethernet
> is called DCE (Data Center Ethernet) or CEE = (Convergence=20 Enhanced
> Ethernet).  This form of Ethernet is a Loss-less = type=20 Ethernet, with
> multi-priority and Flow Control.  This is = NOT an=20 Internet or Intranet
> type of Ethernet.
>
> FCoE = is all=20 about using the DCE (CEE) to carry FC frames.  The rest = of
> the=20 Host and storage stack remain the same, the functions and = features
> of=20 the switches also remain the same and add the capability to = provide
>=20 Cluster Message Switching which has latency close to InfiniBand=20 speeds.
>
>
> Because the FC frames are = transported to the=20 switches intact via a DCE
> frame, the Bridging, if you want to = call it=20 that, is virtually non
> existent.  Hence you can deliver = the FC=20 frames to FC devices, or send FC
> frames to DCE FCoE devices, = just like=20 one would do if it was all FC. And
> all this is done while = performing=20 Cluster message switching and general
> message trucking to the = IP=20 outfacing network.
>

The = rosy=20 future of the yet to appear DCE/CEE and a layer 2 only = world.=20
First you have some terms confused: =

Bridging is the term commonly used for = Layer-2=20 switching and routing is therm used for layer-3 = (switching).=20

Bridging has some advantages (less = management) that=20 have created a movement towards an enterprise wide =
LAN. But this has a long way to go and will require = significant=20 equipment and protocol changes.
Even its=20 proponents do not call for transportless networks, lossless networks=20 etc.
The second trouble with your = argument is=20 that there are no known large scale networking technologies = that=20
really work at full speed (high speed) and are = lossless=20 (flow-controlled) and errorless like FCoE assumes. =
The TCP/IP has solved this issue for every generation using = the proven=20 end-to-end principle (and is doing so now).
And it is not by chance so and that is why all networking = applications=20 are built above layer-3 and not dropping layer-3 (like FCoE) = does.=20

Although I can understand the DCE arguments = as a=20 management statement I would prefer like any rational engineer, to = base my=20 building blocks on structures that are proven and long lasting. And = those are=20 still the end-to-end TCP/IP that can accommodate even your FCP = addicts. The=20 IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a=20 better base.

> This means an = evolutionary process is possible to the solution of
> getting a = single=20 Fabric connection for all networks connected to a
> server, = further, the=20 process has very low interconnection cost on the
> Data Center = Fabric.=20 And it maintains all the FC Fabric Services, and all
> the same = Storage=20 Management processes.
>
> By the way, this is primarily = a Server=20 driven value statement, there
> seems to be little value in = having FCoE=20 on the storage controller.
> Therefore FC storage controllers = (and=20 FICON) will be the very last
> things that connect using FCoE = and that=20 evolution will take at least a
> decade or more.
>=20

It is server cost statement. It = costs=20 nothing to connect a modern server to ethernet it will cost a bundle = to=20 connect to FCoE and it will force users in short lived bad=20 solutions.


> We see = value in=20 offering switches and Directors that can support DCE
> = switching, FC=20 switching as well as iSCSI interconnect, and the
> "Trunking" of = general=20 messaging to the Outfacing IP network.  That said;
> we do = not see=20 FCoE going beyond the constraints of the Data Center.=20

Data Centers now grow to tens of thousands = of nodes.=20 There is no layer-2 technology for errorless/lossless operation at = this scale=20 and there is no good reason to pursue one. The only possible reason = (good=20 reason) is the bridging infrastructure but that infrastructure has a=20 completely different rationale than the flowcontrol. =

> This issue and message is quite different from the = issues and=20 messages
> we struggled with when we started iSCSI.  There = is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen.  
> =
> None of the above cancels out the value of iSCSI in = numerous
>=20 environments.
>
>

iSCSI is=20 good for all environments. Business consideration (and some politics) = keep it=20 form "exploding" and large storage vendors are completely indifferent = to the=20 network connection they are using.
You and I=20 have also slightly different views of DCE. I expect DCE (that still = has a way=20 to go) to improve the QoS in the data-center (and for storage too). = You expect=20 it to bring the loss rates down to the levels that FCP assumes (FCP = has no=20 transport layer) and that is probably a pipe dream. Todays transport = solution=20 for loss mitigation are far more cost effective - and that's why iFCP = is a=20 better proposition as a transition technology than FCoE and iSCSI with = gateways is propably better in the long run.

>
> .
> .
> .
> John L = Hufferd
>=20 Sr. Executive Director of Technology
> = jhufferd@brocade.com
>=20 Office Phone: (408) 333-5244; eFAX: (408) 904-4688
> Alt Office = Phone:=20 (408) 997-6136; Cell: (408) 627-9606
>  
> = -----Original=20 Message-----
> From: John Hufferd
> Sent: Tuesday, April = 24, 2007=20 12:57 PM
> To: 'Julian_Satran@il.ibm.com'
> Subject: Re: = [Ips]=20 Recent comments about FCoE and iSCSI
>
> Julian,
> = I think=20 you are wrong on this one.  The arguments are quite = different
>=20 then the ones we had in pre iSCSI days.  (By the way I missed you = on
> today's Renato meeting/conf call where Brocade took the IBM = technology
> group through FCoE as it is being placed in our = plans).=20  
> I will send you more info when I get to my computer. =  But=20 you probably
> were sent the Brocade charts.  Please review = them=20 and I will follow up
> with more information.
> This does = NOT=20 replace iSCSI it applies only to a DataCenter envuornment
> with = lossless DCE ethernet.
> --------------------------
> John = L.=20 Hufferd
> Sr. Ex. Director of Technology
> Brocade = Communications=20 Systems, Inc.
> Phone: (408) 333-5244
> Mobile: (408)=20 627-9606
> eMail: jhufferd@brocade.com
> (Sent from my = BlackBerry=20 Wireless)
>  
>
> ----- Original Message = -----
>=20 From: Julian Satran <Julian_Satran@il.ibm.com>
> To: = ips@ietf.org=20 <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> = Subject:=20 [Ips] Recent comments about FCoE and iSCSI
>
>
> = Dear All,=20
>
> The trade press is lately full with comments about = the=20 latest and
> greatest reincarnation of Fiber Channel over = ethernet.=20
> It made me try and summarize all the long and hot debates = that=20 preceded
> the advent of iSCSI.
> Although FCoE = proponents make=20 it look like no debate preceded iSCSI that
> was not so - FCoE = was=20 considered even then and was dropped as a dumb
> idea.
> =
>=20 Here is a summary (as afar as I can remember) of the main = arguments.
>=20 They are not bad arguments even in retrospect and technically = FCoE
>=20 doesn't look better than it did then.
>
> Feel free to = use this=20 material in a nay form. I expect this group to
> seriously =  expand=20 my arguments and make them public - in personal or
> collective = form.=20
>
> And do not forget - it is a technical dispute - = although we=20 all must
> have some doubts about the way it is pursued. =
>=20
> Regards,
> Julo
>
>=20 --------------------------------------------------------------------- =
>=20
> What a piece of nostalgia :-)
>
> Around 1997 = when a=20 team at IBM Research (Haifa and Almaden) started
> looking at = connecting=20 storage to servers using the "regular network"
> (the ubiquitous = LAN) we=20 considered many alternatives (another team even
> had a look at = ATM -=20 still a computer network candidate at the time). I
> won't get = you over=20 all of our rationale (and we went over some of them
> again at = the end=20 of 1999 with a team from CISCO before we convened the
> first = IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the
> rest) = but some=20 of the reasons we choose to drop Fiber Channel over raw
> = Ethernet where=20 multiple:
>
>
> *   Fiber Channel Protocol = (SCSI=20 over Fiber Channel Link) is
> "mildly" effective because: =
>=20
>    *   it implements endpoints in a dedicated = engine=20 (Offload)
>    *   it has no transport layer = (recovery=20 is done at the
> application layer under the assumption that the = error=20 rate will be very
> low)
>    *   the = network is=20 limited in physical span and logical span
> (number of switches) =
>    *   flow-control/congestion control is = achieved=20 with a
> mechanism adequate for a limited span network = (credits). The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *  =20 The assumption of low losses due to errors might
> radically = change when=20 moving from 1 to 10 Gb/s - the scaling argument
>   =  *  =20 Ethernet has no credit mechanism and any mechanism with
> a = similar=20 effect increases the end point cost. Building a transport
> = layer in the=20 protocol stack has always been the preferred choice of the
> = networking=20 community - the community argument
>    *   The=20 "performance penalty" of a complete protocol stack
> has always = been=20 overstated (and overrated). Advances in protocol stack
> = implementation=20 and finer tuning of the congestion control mechanisms
> make=20 conventional TCP/IP performing well even at 10 Gb/s and over.
> = Moreover=20 the multicore processors that become dominant on the computing
> = scene=20 have enough compute cycles available to make any "offloading"
> = possible=20 as a mere code restructuring exercise (see the stack reports
> = from=20 Intel, IBM etc.)
>    *   Building on a complete = stack=20 makes available a wealth of
> operational and management = mechanisms=20 built over the years by the
> networking community (routing,=20 provisioning, security, service location
> etc.) - the community = argument
>    *   Higher level storage access = over an IP=20 network is widely
> available and having both block and file = served over=20 the same connection
> with the same support and management = structure is=20 compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't know=20 exactly
> how it will operate). Untill then the scale of Layer 2 = network=20 is going
> to seriously limited - the scaling argument
> =
>=20
>
>       As a side argument - a = performance=20 comparison made in
> 1998 showed SCSI over TCP (a predecessor of = the=20 later iSCSI) to perform
> better than FCP at 1Gbs for block = sizes=20 typical for OLTP (4-8KB). That
> was what convinced us to take = the path=20 that lead to iSCSI - and we used
> plain vanilla x86 servers = with=20 plain-vanilla NICs and Linux (with
> similar measurements = conducted on=20 Windows).
>    The networking and storage community=20 acknowledged those
> arguments and developed iSCSI and the = companion=20 protocols for service
> discovery, boot etc.
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a reasonable = fashion=20 and developed 2
> protocols iFCP (to support hosts with FCP = drivers and=20 IP connections to
> connect to storage by a simple conversion = from FCP=20 to TCP packets) FCPIP
> to extend the reach of FCP through IP = (connects=20 FCP islands through TCP
> links). Both have been
>   =  implemented and their foundation is solid.
>  =20  
>    The current attempt of developing a = "new-age" FCP=20 over an
> Ethernet link is going against most of the arguments = that have=20 given us
> iSCSI etc.
>    
>   =  It=20 ignores the networking layering practice, build an
> application = protocol directly above a link and thus limits scaling,
> = mandates=20 elements at the link layer and application layer that make
>=20 applications more expensive and leaves aside the whole "ecosystem"=20 that
> accompanies TCP/IP (and not Ethernet).
>  =20  
>    In some related effort (and at a point = also when=20 developing
> iSCSI) we considered also moving away from SCSI = (like some=20 "no
> standardized" but popular in some circles software did - = e.g.,=20 NBP) but
> decided against. SCSI is a mature and well understood = access
> architecture for block storage and is implemented by = many=20 device
> vendors. Moving away from it would not have been = justified at=20 the time.
>    

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------=_NextPart_000_0015_01C78721.83D17000-- --===============0644468803== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0644468803==-- From ips-bounces@ietf.org Wed Apr 25 11:04:51 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgj34-0008Se-PX; Wed, 25 Apr 2007 11:04:50 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgj33-0008QE-Dw for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 11:04:49 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgj32-0008Pd-M2 for ips@ietf.org; Wed, 25 Apr 2007 11:04:49 -0400 Received: from mtagate8.de.ibm.com ([195.212.29.157]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgj30-00069z-QX for ips@ietf.org; Wed, 25 Apr 2007 11:04:48 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate8.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3PF4jXS128926 for ; Wed, 25 Apr 2007 15:04:45 GMT Received: from d12av04.megacenter.de.ibm.com (d12av04.megacenter.de.ibm.com [9.149.165.229]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3PF4jrc3723416 for ; Wed, 25 Apr 2007 17:04:45 +0200 Received: from d12av04.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3PF4jJ6010389 for ; Wed, 25 Apr 2007 17:04:45 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3PF4jaq010383; Wed, 25 Apr 2007 17:04:45 +0200 In-Reply-To: <001801c78743$0b25e560$05faa8c0@ivivity.com> To: "Eddy Quicksall" MIME-Version: 1.0 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Wed, 25 Apr 2007 11:04:43 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 25/04/2007 18:04:45, Serialize complete at 25/04/2007 18:04:45 X-Spam-Score: 0.8 (/) X-Scan-Signature: 717d651095a319b49fc3b6c7b72cb4dd Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1684282892==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============1684282892== Content-Type: multipart/alternative; boundary="=_alternative 0052D27E852572C8_=" This is a multipart message in MIME format. --=_alternative 0052D27E852572C8_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Eddy, That is oversimplified and ignore the drop rate assumption and error rate=20 assumptions made in FCP(FCP has no transport layer). To get to it on a=20 sizable network requires more than PAUSE. Julo "Eddy Quicksall" =20 25/04/07 10:07 To "John Hufferd" , Julian Satran/Haifa/IBM@IBMIL cc Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI Basically, it is sending FC frames over Ethernet. This localizes the=20 traffic unless you route based on MAC addresses. So you send 2146 bytes of = FC frame plus 18 bytes of Ethernet overhead as FCoE ?standard? packet. 18=20 bytes of Ethernet gets stripped and you have straight FC frame that can go = through any FC network. Now you can have 10G Ethernet pipes into existing=20 FC SANs. Limited market potential as far as I can see. The key argument is = it much easier to implement than iSCSI and also has less overhead and uses = all the benefits of FC. End to End credits are simulated using PAUSE=20 command on Ethernet and MAC addresses are mapped into WWNs. Biggest knock is that it will not route on the ?global? scale like TCP/IP=20 would. Eddy ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as an > outreach protocol from the Data Center. We also believe iSCSI is very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our new > iSCSI and TOE offload HBAs.=20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the network = - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is not=20 true either=20 3. Gatewaying is expensive - and it is perhaps so but only if you are = completely relying on FCP storage (and there are plenty of good iSCSI=20 vendors of storage)and pushing the price on the servers is not cheap=20 either - at least not for the server buyer > Now with that positioning, it is important to understand the limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on the > market, it is not cost feasible to replace all the server connectivity > to FC storage with iSCSI for the hundred to thousands of servers in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are too > high for iSCSI to be used as the "normal" server connect into a FC based > Fabric. That is true for the current 1GE; and for 10GE the cost is just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to understand > is that it is not Ethernet as we have known it up to know. The Ethernet > we are talking about is a type of Ethernet that can only be deployed in > a constrained environment such as a Data Center. This form of Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest of > the Host and storage stack remain the same, the functions and features > of the switches also remain the same and add the capability to provide > Cluster Message Switching which has latency close to InfiniBand speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or send FC > frames to DCE FCoE devices, just like one would do if it was all FC. And > all this is done while performing Cluster message switching and general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world.=20 First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing is=20 therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a=20 movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant equipment=20 and protocol changes.=20 Even its proponents do not call for transportless networks, lossless=20 networks etc.=20 The second trouble with your argument is that there are no known large=20 scale networking technologies that=20 really work at full speed (high speed) and are lossless (flow-controlled)=20 and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven=20 end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications are = built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement I=20 would prefer like any rational engineer, to base my building blocks on=20 structures that are proven and long lasting. And those are still the=20 end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS TWG=20 has developed the iFCP that does exactly what FCoE claims to do an a=20 better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on the > Data Center Fabric. And it maintains all the FC Fabric Services, and all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern server=20 to ethernet it will cost a bundle to connect to FCoE and it will force=20 users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no layer-2=20 technology for errorless/lossless operation at this scale and there is no=20 good reason to pursue one. The only possible reason (good reason) is the=20 bridging infrastructure but that infrastructure has a completely different = rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and messages > we struggled with when we started iSCSI. There is a consortium of folks > both working on the DCE (CEE) and the FCoE. Without the DCE the FCoE > will not happen.=20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some=20 politics) keep it form "exploding" and large storage vendors are=20 completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE (that=20 still has a way to go) to improve the QoS in the data-center (and for=20 storage too). You expect it to bring the loss rates down to the levels=20 that FCP assumes (FCP has no transport layer) and that is probably a pipe=20 dream. Todays transport solution for loss mitigation are far more cost=20 effective - and that's why iFCP is a better proposition as a transition=20 technology than FCoE and iSCSI with gateways is propably better in the=20 long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 >=20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian=5FSatran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM technology > group through FCoE as it is being placed in our plans).=20 > I will send you more info when I get to my computer. But you probably > were sent the Brocade charts. Please review them and I will follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) >=20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team even > had a look at ATM - still a computer network candidate at the time). I > won't get you over all of our rationale (and we went over some of them > again at the end of 1999 with a team from CISCO before we convened the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the > rest) but some of the reasons we choose to drop Fiber Channel over raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging and > are limited by the logical tree structure that bridges must follow. The > effort to combine routers and bridges (rbridges) is promising to change > that but it will take some time to finalize (and we don't know exactly > how it will operate). Untill then the scale of Layer 2 network is going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That > was what convinced us to take the path that lead to iSCSI - and we used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service > discovery, boot etc.=20 >=20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections to > connect to storage by a simple conversion from FCP to TCP packets) FCPIP > to extend the reach of FCP through IP (connects FCP islands through TCP > links). Both have been=20 > implemented and their foundation is solid.=20 >=20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given us > iSCSI etc.=20 >=20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet).=20 >=20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the time.=20 >=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 0052D27E852572C8_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Eddy,

That is oversimplified and ignore the drop rate assumption and error rate assumptions made in FCP(FCP has no transport layer). To get to it on a sizable network requires more than PAUSE.

Julo


"Eddy Quicksall&= quot; <Quicksall=5FiSCSI@Bellsouth.net>

25/04/07 10:07

To
"John Hufferd" <jhuffer= d@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent comments about FCoE and iSCSI





Basically, it is sending = FC frames over Ethernet. This localizes the traffic unless you route based on MAC addresses. So you send 2146 bytes of FC frame plus 18 bytes of Ether= net overhead as FCoE “standard” packet. 18 bytes of Ethernet gets s= tripped and you have straight FC frame that can go through any FC network. Now you can have 10G Ethernet pipes into existing FC SANs. Limited market poten= tial as far as I can see. The key argument is it much easier to implement than iSCSI and also has less overhead and uses all the benefits of FC. End to End credits are simulated using PAUSE command on Ethernet and MAC addresses are mapped into WWNs.

Biggest knock is that it w= ill not route on the “global” scale like TCP/IP would.

Eddy

 ----- Original Message -----

From: Julian Satran
To: = John Hufferd
Cc: ips@ietf.org
Sent: Wednesday, April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI



"John Hufferd" <jhufferd@Brocade.COM= > wrote on 25/04/2007 02:45:51:

> Julian,
> To be sure you understand our position; Brocade is pushing iSCSI as an
> outreach protocol from the Data Center. We also believe iSCSI is very<= br> > useful for installations that do not have a Fibre Channel
> infrastructure, and in that case we will be able to sell them our new
> iSCSI and TOE offload HBAs.  
>
> When I say iSCSI is an outreach protocol, this is a statement that iSCSI
> is very important to connect "stranded" servers to the Fibre Channel
> Fabric.  That is, we sell iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers (software or HBA iSCSI initiators) to connect to the
> Enterprise "Bet Your Business" FC Storage.  This of course also applies
> to Desktops and Laptop systems, and systems at distance.
>

You make it sound like:
1.        most of the servers in the world have their storage on the network - and that is not the case
2.        FCP is basically better performing than iSCSI - and that is not true either
3.        Gatewaying is expensive - and it is perhaps so but only if you are completely relying on FCP storage (and there are plenty of good iSCSI vendors of storage)and pushing the price on the servers is not cheap either - at least not for the server buyer

> Now with that positioning, it is important to understand the limitatio= n
> to this strategy.  The primary problem is that iSCSI to FC Bridgi= ng
> (Gatewaying) is relatively expensive (compared to simple FC
> connections).  Though we have some of the best priced Gateways on the
> market, it is not cost feasible to replace all the server connectivity=
> to FC storage with iSCSI for the hundred to thousands of servers in the
> Data Center.  And since, if there is to be a consolidated Network=
> connection to the servers in the Data Center, there must be an
> evolutionary replacement of Server Connections to Storage.  That means
> there must be a bridge/Gateway approach.  And as I mentioned before,
> there is just too much cost in the iSCSI to FC Gateway.
>
> The issue is the server requirement to have a single connection type to
> handle cluster messaging, general messaging, and storage.  iSCSI is
> clearly an option for the storage, however, the gateway costs are too
> high for iSCSI to be used as the "normal" server connect into a FC based
> Fabric. That is true for the current 1GE; and for 10GE the cost is just
> out of sight.  The reason for this is the requirement for TCP/IP<= br> > termination and re-initiation with FC at the Gateway.
>
> Now with respect to FC over Ethernet the important thing to understand=
> is that it is not Ethernet as we have known it up to know.  The Ethernet
> we are talking about is a type of Ethernet that can only be deployed in
> a constrained environment such as a Data Center.  This form of Ethernet
> is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced
> Ethernet).  This form of Ethernet is a Loss-less type Ethernet, with
> multi-priority and Flow Control.  This is NOT an Internet or Intranet
> type of Ethernet.
>
> FCoE is all about using the DCE (CEE) to carry FC frames.  The rest of
> the Host and storage stack remain the same, the functions and features=
> of the switches also remain the same and add the capability to provide=
> Cluster Message Switching which has latency close to InfiniBand speeds= .
>
>
> Because the FC frames are transported to the switches intact via a DCE
> frame, the Bridging, if you want to call it that, is virtually non
> existent.  Hence you can deliver the FC frames to FC devices, or send FC
> frames to DCE FCoE devices, just like one would do if it was all FC. And
> all this is done while performing Cluster message switching and genera= l
> message trucking to the IP outfacing network.
>

The rosy future of the yet to appear DCE/CEE and a layer 2 only world.
First you have some terms confused:

Bridging is the term commonly used for Layer-2 switching and routing is therm used for layer-3 (switching).

Bridging has some advantages (less management) that have created a movement towards an enterprise wide
LAN. But this has a long way to go and will require significant equipment and protocol changes. =
Even its proponents do not call for transportless networks, lossless networ= ks etc.
The second trouble with your argument is that there are no known large scale networking technologies that
really work at full speed (high speed) and are lossless (flow-controlled) and errorless like FCoE assumes.
The TCP/IP has solved this issue for every generation using the proven end-to-end principle (and is doing so now).
And it is not by chance so and that is why all networking applications are built above layer-3 and not dropping layer-3 (like FCoE) does.

Although I can understand the DCE arguments as a management statement I would prefer like any rational engineer, to base my building blocks on structures that are proven and long lasting. And those are still the end-to= -end TCP/IP that can accommodate even your FCP addicts. The IPS TWG has developed the iFCP that does exactly what FCoE claims to do an a better base.<= /tt>

> This means an evolutionary process is possible to the solution of
> getting a single Fabric connection for all networks connected to a
> server, further, the process has very low interconnection cost on the
> Data Center Fabric. And it maintains all the FC Fabric Services, and all
> the same Storage Management processes.
>
> By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller.
> Therefore FC storage controllers (and FICON) will be the very last
> things that connect using FCoE and that evolution will take at least a
> decade or more.
>

It is server cost statement. It costs nothing to connect a modern server to ethernet it will cost a bundle to connect to FCoE and it will force users in short lived bad solutions.


> We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the
> "Trunking" of general messaging to the Outfacing IP network.  That said;
> we do not see FCoE going beyond the constraints of the Data Center.

Data Centers now grow to tens of thousands of nodes. There is no layer-2 technology for errorless/lossless operation at this scale and there is no good reason to pursue one. The only possible reason (good reason) is the bridging infrastructure but that infrastructure has a completely differ= ent rationale than the flowcontrol.

> This issue and message is quite different from the issues and messages=
> we struggled with when we started iSCSI.  There is a consortium of folks
> both working on the DCE (CEE) and the FCoE.  Without the DCE the FCoE
> will not happen.  
>
> None of the above cancels out the value of iSCSI in numerous
> environments.
>
>

iSCSI is good for all environments. Business consideration (and some politi= cs) keep it form "exploding" and large storage vendors are completely indifferent to the network connection they are using.
You and I have also slightly different views of DCE. I expect DCE (that still has a way to go) to improve the QoS in the data-center (and for stora= ge too). You expect it to bring the loss rates down to the levels that FCP assumes (FCP has no transport layer) and that is probably a pipe dream. Todays transport solution for loss mitigation are far more cost effective - and that's why iFCP is a better proposition as a transition technology than FCoE and iSCSI with gateways is propably better in the long run.

>
> .
> .
> .
> John L Hufferd
> Sr. Executive Director of Technology
> jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606
>  
> -----Original Message-----
> From: John Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To: 'Julian=5FSatran@il.ibm.com'
> Subject: Re: [Ips] Recent comments about FCoE and iSCSI
>
> Julian,
> I think you are wrong on this one.  The arguments are quite diffe= rent
> then the ones we had in pre iSCSI days.  (By the way I missed you on
> today's Renato meeting/conf call where Brocade took the IBM technology=
> group through FCoE as it is being placed in our plans).  
> I will send you more info when I get to my computer.  But you probably
> were sent the Brocade charts.  Please review them and I will follow up
> with more information.
> This does NOT replace iSCSI it applies only to a DataCenter envuornmen= t
> with lossless DCE ethernet.
> --------------------------
> John L. Hufferd
> Sr. Ex. Director of Technology
> Brocade Communications Systems, Inc.
> Phone: (408) 333-5244
> Mobile: (408) 627-9606
> eMail: jhufferd@brocade.com
> (Sent from my BlackBerry Wireless)
>  
>
> ----- Original Message -----
> From: Julian Satran <Julian=5FSatran@il.ibm.com>
> To: ips@ietf.org <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> Subject: [Ips] Recent comments about FCoE and iSCSI
>
>
> Dear All,
>
> The trade press is lately full with comments about the latest and
> greatest reincarnation of Fiber Channel over ethernet.
> It made me try and summarize all the long and hot debates that precede= d
> the advent of iSCSI.
> Although FCoE proponents make it look like no debate preceded iSCSI that
> was not so - FCoE was considered even then and was dropped as a dumb > idea.
>
> Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE
> doesn't look better than it did then.
>
> Feel free to use this material in a nay form. I expect this group to
> seriously  expand my arguments and make them public - in personal or
> collective form.
>
> And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.
>
> Regards,
> Julo
>
> ---------------------------------------------------------------------
>
> What a piece of nostalgia :-)
>
> Around 1997 when a team at IBM Research (Haifa and Almaden) started
> looking at connecting storage to servers using the "regular netwo= rk"
> (the ubiquitous LAN) we considered many alternatives (another team even
> had a look at ATM - still a computer network candidate at the time). I
> won't get you over all of our rationale (and we went over some of them
> again at the end of 1999 with a team from CISCO before we convened the
> first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the
> rest) but some of the reasons we choose to drop Fiber Channel over raw
> Ethernet where multiple:
>
>
> *   Fiber Channel Protocol (SCSI over Fiber Channel Link) is
> "mildly" effective because:
>
>    *   it implements endpoints in a dedicated engine (Offload)
>    *   it has no transport layer (recovery is done at the
> application layer under the assumption that the error rate will be very
> low)
>    *   the network is limited in physical span and logical span
> (number of switches)
>    *   flow-control/congestion control is achieved with a
> mechanism adequate for a limited span network (credits). The packet loss
> rate is almost nil and that allows FCP to avoid using a transport
> (end-to-end) layer
>
>    *   FCP she switches are simple (addresses are local and the
> memory requirements cam be limited through the credit mechanism)
>    *   However FCP endpoints are inherently costlier than
> simple NICs - the cost argument (initiators are more expensive)
>    *   The credit mechanisms is highly unstable for large
> networks (check switch vendors planning docs for the network diameter<= br> > limits) - the scaling argument
>    *   The assumption of low losses due to errors might=
> radically change when moving from 1 to 10 Gb/s - the scaling argument
>    *   Ethernet has no credit mechanism and any mechani= sm with
> a similar effect increases the end point cost. Building a transport
> layer in the protocol stack has always been the preferred choice of the
> networking community - the community argument
>    *   The "performance penalty" of a complete protocol stack
> has always been overstated (and overrated). Advances in protocol stack=
> implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over.
> Moreover the multicore processors that become dominant on the computin= g
> scene have enough compute cycles available to make any "offloadin= g"
> possible as a mere code restructuring exercise (see the stack reports<= br> > from Intel, IBM etc.)
>    *   Building on a complete stack makes available a wealth of
> operational and management mechanisms built over the years by the
> networking community (routing, provisioning, security, service locatio= n
> etc.) - the community argument
>    *   Higher level storage access over an IP network is widely
> available and having both block and file served over the same connecti= on
> with the same support and management structure is compelling - the
> community argument
>    *   Highly efficient networks are easy to build over IP with
> optimal (shortest path) routing while Layer 2 networks use bridging and
> are limited by the logical tree structure that bridges must follow. The
> effort to combine routers and bridges (rbridges) is promising to chang= e
> that but it will take some time to finalize (and we don't know exactly=
> how it will operate). Untill then the scale of Layer 2 network is going
> to seriously limited - the scaling argument
>
>
>
>       As a side argument - a performance comparison made in
> 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perfor= m
> better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That
> was what convinced us to take the path that lead to iSCSI - and we used
> plain vanilla x86 servers with plain-vanilla NICs and Linux (with
> similar measurements conducted on Windows).
>    The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service<= br> > discovery, boot etc.
>    
>    The community also acknowledged the need to support exist= ing
> infrastructure and extend it in a reasonable fashion and developed 2
> protocols iFCP (to support hosts with FCP drivers and IP connections to
> connect to storage by a simple conversion from FCP to TCP packets) FCPIP
> to extend the reach of FCP through IP (connects FCP islands through TCP
> links). Both have been
>    implemented and their foundation is solid.
>    
>    The current attempt of developing a "new-age" FCP over an
> Ethernet link is going against most of the arguments that have given us
> iSCSI etc.
>    
>    It ignores the networking layering practice, build an
> application protocol directly above a link and thus limits scaling,
> mandates elements at the link layer and application layer that make
> applications more expensive and leaves aside the whole "ecosystem= " that
> accompanies TCP/IP (and not Ethernet).
>    
>    In some related effort (and at a point also when developi= ng
> iSCSI) we considered also moving away from SCSI (like some "no
> standardized" but popular in some circles software did - e.g., NBP) but
> decided against. SCSI is a mature and well understood access
> architecture for block storage and is implemented by many device
> vendors. Moving away from it would not have been justified at the time.
>    

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

--=_alternative 0052D27E852572C8_=-- --===============1684282892== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1684282892==-- From ips-bounces@ietf.org Wed Apr 25 11:09:13 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgj7G-0001p7-Fi; Wed, 25 Apr 2007 11:09:10 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgj7F-0001p2-As for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 11:09:09 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgj7F-0001ou-0y for ips@ietf.org; Wed, 25 Apr 2007 11:09:09 -0400 Received: from imf25aec.mail.bellsouth.net ([205.152.59.73]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgj7C-000766-W4 for ips@ietf.org; Wed, 25 Apr 2007 11:09:09 -0400 Received: from ibm69aec.bellsouth.net ([74.245.52.54]) by imf25aec.mail.bellsouth.net with ESMTP id <20070425150856.NTWX14859.imf25aec.mail.bellsouth.net@ibm69aec.bellsouth.net> for ; Wed, 25 Apr 2007 11:08:56 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm69aec.bellsouth.net with SMTP id <20070425150854.EZXD2148.ibm69aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 11:08:54 -0400 Message-ID: <002b01c7874b$a57873a0$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Julian Satran" References: Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 11:08:56 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.5 (/) X-Scan-Signature: 9c747b02957c409d00ef4f5a343ba495 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1002509721==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============1002509721== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0028_01C7872A.1E34C720" This is a multi-part message in MIME format. ------=_NextPart_000_0028_01C7872A.1E34C720 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I basically said that in the summery line by saying "it will not route = on the "global" scale like TCP/IP would". ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Eddy,=20 That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it = on a sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 To "John Hufferd" , Julian = Satran/Haifa/IBM@IBMIL =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI = =20 =20 Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like = TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as = an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the = network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is = not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if = you are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers in = the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type = to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be deployed = in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest = of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all FC. = And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world. = First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known large = scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven = end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications = are built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement = I would prefer like any rational engineer, to base my building blocks on = structures that are proven and long lasting. And those are still the = end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS = TWG has developed the iFCP that does exactly what FCoE claims to do an a = better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, and = all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least = a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will follow = up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI = that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the time). = I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet = loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of = the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging = and > are limited by the logical tree structure that bridges must follow. = The > effort to combine routers and bridges (rbridges) is promising to = change > that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections = to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands through = TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given = us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) = but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_0028_01C7872A.1E34C720 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

I basically said that in the summery line by saying = "it will not route on the =93global=94 scale like TCP/IP=20 would".
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 11:04=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Eddy, =

That is oversimplified and ignore the drop = rate=20 assumption and error rate assumptions made in FCP(FCP has no transport = layer).=20 To get to it on a sizable network requires more than PAUSE.=20

Julo


"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 10:07

To
"John Hufferd" <jhufferd@Brocade.COM>,=20 Julian Satran/Haifa/IBM@IBMIL =
cc
<ips@ietf.org>=20
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

=




Basically, it is sending FC frames over = Ethernet. This=20 localizes the traffic unless you route based on MAC addresses. So you = send=20 2146 bytes of FC frame plus 18 bytes of Ethernet overhead as FCoE = =93standard=94=20 packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that=20 can go through any FC network. Now you can have 10G Ethernet pipes = into=20 existing FC SANs. Limited market potential as far as I can see. The = key=20 argument is it much easier to implement than iSCSI and also has less = overhead=20 and uses all the benefits of FC. End to End credits are simulated = using PAUSE=20 command on Ethernet and MAC addresses are mapped into WWNs.=20

Biggest knock is that = it will not=20 route on the =93global=94 scale like TCP/IP would.=20

Eddy=20

 -----=20 Original Message -----

From: Julian=20 Satran
To:=20 John=20 Hufferd
Cc:=20 ips@ietf.org =
Sent: Wednesday, April 25, 2007 8:09 AM =
Subject: Re: FW: [Ips] Recent comments about FCoE and=20 iSCSI



"John=20 Hufferd" <jhufferd@Brocade.COM> wrote on 25/04/2007 02:45:51:

> Julian, =
> To be=20 sure you understand our position; Brocade is pushing iSCSI as = an
>=20 outreach protocol from the Data Center. We also believe iSCSI is = very
>=20 useful for installations that do not have a Fibre Channel
>=20 infrastructure, and in that case we will be able to sell them our = new
>=20 iSCSI and TOE offload HBAs.  
>
> When I say iSCSI = is an=20 outreach protocol, this is a statement that iSCSI
> is very = important to=20 connect "stranded" servers to the Fibre Channel
> Fabric. =  That is,=20 we sell iSCSI-to-FC Gateway devices which will permit
> iSCSI = Servers=20 (software or HBA iSCSI initiators) to connect to the
> = Enterprise "Bet=20 Your Business" FC Storage.  This of course also applies
> = to=20 Desktops and Laptop systems, and systems at = distance.
>

You make it sound=20 like:
1.=20        most of the = servers in the=20 world have their storage on the network - and that is not the=20 case
2.=20        FCP is basically = better=20 performing than iSCSI - and that is not true either=20
3.      =20  Gatewaying is expensive - and it is = perhaps so=20 but only if you are completely relying on FCP storage (and there are = plenty of=20 good iSCSI vendors of storage)and pushing the price on the servers is = not=20 cheap either - at least not for the server buyer =

> Now with that positioning, it is important to = understand the=20 limitation
> to this strategy.  The primary problem is that = iSCSI=20 to FC Bridging
> (Gatewaying) is relatively expensive (compared = to=20 simple FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace all = the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers in the
> Data Center.  And since, if = there is=20 to be a consolidated Network
> connection to the servers in the = Data=20 Center, there must be an
> evolutionary replacement of Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are = too
>=20 high for iSCSI to be used as the "normal" server connect into a FC=20 based
> Fabric. That is true for the current 1GE; and for 10GE = the cost=20 is just
> out of sight.  The reason for this is the = requirement for=20 TCP/IP
> termination and re-initiation with FC at the = Gateway.
>=20
> Now with respect to FC over Ethernet the important thing to=20 understand
> is that it is not Ethernet as we have known it up = to know.=20  The Ethernet
> we are talking about is a type of Ethernet = that can=20 only be deployed in
> a constrained environment such as a Data = Center.=20  This form of Ethernet
> is called DCE (Data Center = Ethernet) or=20 CEE (Convergence Enhanced
> Ethernet).  This form of = Ethernet is a=20 Loss-less type Ethernet, with
> multi-priority and Flow Control. =  This is NOT an Internet or Intranet
> type of = Ethernet.
>=20
> FCoE is all about using the DCE (CEE) to carry FC frames. =  The=20 rest of
> the Host and storage stack remain the same, the = functions and=20 features
> of the switches also remain the same and add the = capability=20 to provide
> Cluster Message Switching which has latency close = to=20 InfiniBand speeds.
>
>
> Because the FC frames are = transported to the switches intact via a DCE
> frame, the = Bridging, if=20 you want to call it that, is virtually non
> existent. =  Hence you=20 can deliver the FC frames to FC devices, or send FC
> frames to = DCE FCoE=20 devices, just like one would do if it was all FC. And
> all this = is done=20 while performing Cluster message switching and general
> message = trucking to the IP outfacing network.
>

The rosy future of the yet = to appear=20 DCE/CEE and a layer 2 only world. =
First you have some terms confused:=20

Bridging is the term commonly used = for Layer-2=20 switching and routing is therm used for layer-3 = (switching).

Bridging has some = advantages (less=20 management) that have created a movement towards an enterprise=20 wide
LAN. But = this has a=20 long way to go and will require significant equipment and protocol=20 changes.
Even = its=20 proponents do not call for transportless networks, lossless networks=20 etc.
The = second trouble=20 with your argument is that there are no known large scale networking=20 technologies that
really=20 work at full speed (high speed) and are lossless (flow-controlled) and = errorless like FCoE assumes. =
The TCP/IP has solved this issue for every generation = using the=20 proven end-to-end principle (and is doing so now).=20
And it is not by chance so and that is = why all=20 networking applications are built above layer-3 and not dropping = layer-3 (like=20 FCoE) does.

Although=20 I can understand the DCE arguments as a management statement I would = prefer=20 like any rational engineer, to base my building blocks on structures = that are=20 proven and long lasting. And those are still the end-to-end TCP/IP = that can=20 accommodate even your FCP addicts. The IPS TWG has developed the iFCP = that=20 does exactly what FCoE claims to do an a better base.=20

> This means an evolutionary = process is=20 possible to the solution of
> getting a single Fabric connection = for all=20 networks connected to a
> server, further, the process has very = low=20 interconnection cost on the
> Data Center Fabric. And it = maintains all=20 the FC Fabric Services, and all
> the same Storage Management = processes.=20
>
> By the way, this is primarily a Server driven value=20 statement, there
> seems to be little value in having FCoE on = the=20 storage controller.
> Therefore FC storage controllers (and = FICON) will=20 be the very last
> things that connect using FCoE and that = evolution=20 will take at least a
> decade or more.
>

It is server cost = statement. It costs=20 nothing to connect a modern server to ethernet it will cost a bundle = to=20 connect to FCoE and it will force users in short lived bad=20 solutions.


>=20 We see value in offering switches and Directors that can support = DCE
>=20 switching, FC switching as well as iSCSI interconnect, and the
> = "Trunking" of general messaging to the Outfacing IP network. =  That=20 said;
> we do not see FCoE going beyond the constraints of the = Data=20 Center.

Data Centers=20 now grow to tens of thousands of nodes. There is no layer-2 technology = for=20 errorless/lossless operation at this scale and there is no good reason = to=20 pursue one. The only possible reason (good reason) is the bridging=20 infrastructure but that infrastructure has a completely different = rationale=20 than the flowcontrol.

> This issue and message is quite different from = the issues=20 and messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen.  
> =
> None of the above cancels out the value of iSCSI in = numerous
>=20 environments.
>
>

iSCSI is good for all environments. Business = consideration (and=20 some politics) keep it form "exploding" and large storage vendors are=20 completely indifferent to the network connection they are using. =
You and I=20 have also slightly different views of DCE. I expect DCE (that still = has a way=20 to go) to improve the QoS in the data-center (and for storage too). = You expect=20 it to bring the loss rates down to the levels that FCP assumes (FCP = has no=20 transport layer) and that is probably a pipe dream. Todays transport = solution=20 for loss mitigation are far more cost effective - and that's why iFCP = is a=20 better proposition as a transition technology than FCoE and iSCSI with = gateways is propably better in the long run.=20

>
> .
> .
> = .
>=20 John L Hufferd
> Sr. Executive Director of Technology
>=20 jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: (408) = 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408)=20 627-9606
>  
> -----Original Message-----
> = From: John=20 Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent comments = about=20 FCoE and iSCSI
>
> Julian,
> I think you are wrong = on this=20 one.  The arguments are quite different
> then the ones we = had in=20 pre iSCSI days.  (By the way I missed you on
> today's = Renato=20 meeting/conf call where Brocade took the IBM technology
> group = through=20 FCoE as it is being placed in our plans).  
> I will send = you more=20 info when I get to my computer.  But you probably
> were = sent the=20 Brocade charts.  Please review them and I will follow up
> = with=20 more information.
> This does NOT replace iSCSI it applies only = to a=20 DataCenter envuornment
> with lossless DCE ethernet.
>=20 --------------------------
> John L. Hufferd
> Sr. Ex. = Director of=20 Technology
> Brocade Communications Systems, Inc.
> Phone: = (408)=20 333-5244
> Mobile: (408) 627-9606
> eMail:=20 jhufferd@brocade.com
> (Sent from my BlackBerry = Wireless)
>=20  
>
> ----- Original Message -----
> From: = Julian=20 Satran <Julian_Satran@il.ibm.com>
> To: ips@ietf.org=20 <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> = Subject:=20 [Ips] Recent comments about FCoE and iSCSI
>
>
> = Dear All,=20
>
> The trade press is lately full with comments about = the=20 latest and
> greatest reincarnation of Fiber Channel over = ethernet.=20
> It made me try and summarize all the long and hot debates = that=20 preceded
> the advent of iSCSI.
> Although FCoE = proponents make=20 it look like no debate preceded iSCSI that
> was not so - FCoE = was=20 considered even then and was dropped as a dumb
> idea.
> =
>=20 Here is a summary (as afar as I can remember) of the main = arguments.
>=20 They are not bad arguments even in retrospect and technically = FCoE
>=20 doesn't look better than it did then.
>
> Feel free to = use this=20 material in a nay form. I expect this group to
> seriously =  expand=20 my arguments and make them public - in personal or
> collective = form.=20
>
> And do not forget - it is a technical dispute - = although we=20 all must
> have some doubts about the way it is pursued. =
>=20
> Regards,
> Julo
>
>=20 --------------------------------------------------------------------- =
>=20
> What a piece of nostalgia :-)
>
> Around 1997 = when a=20 team at IBM Research (Haifa and Almaden) started
> looking at = connecting=20 storage to servers using the "regular network"
> (the ubiquitous = LAN) we=20 considered many alternatives (another team even
> had a look at = ATM -=20 still a computer network candidate at the time). I
> won't get = you over=20 all of our rationale (and we went over some of them
> again at = the end=20 of 1999 with a team from CISCO before we convened the
> first = IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the
> rest) = but some=20 of the reasons we choose to drop Fiber Channel over raw
> = Ethernet where=20 multiple:
>
>
> *   Fiber Channel Protocol = (SCSI=20 over Fiber Channel Link) is
> "mildly" effective because: =
>=20
>    *   it implements endpoints in a dedicated = engine=20 (Offload)
>    *   it has no transport layer = (recovery=20 is done at the
> application layer under the assumption that the = error=20 rate will be very
> low)
>    *   the = network is=20 limited in physical span and logical span
> (number of switches) =
>    *   flow-control/congestion control is = achieved=20 with a
> mechanism adequate for a limited span network = (credits). The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *  =20 The assumption of low losses due to errors might
> radically = change when=20 moving from 1 to 10 Gb/s - the scaling argument
>   =  *  =20 Ethernet has no credit mechanism and any mechanism with
> a = similar=20 effect increases the end point cost. Building a transport
> = layer in the=20 protocol stack has always been the preferred choice of the
> = networking=20 community - the community argument
>    *   The=20 "performance penalty" of a complete protocol stack
> has always = been=20 overstated (and overrated). Advances in protocol stack
> = implementation=20 and finer tuning of the congestion control mechanisms
> make=20 conventional TCP/IP performing well even at 10 Gb/s and over.
> = Moreover=20 the multicore processors that become dominant on the computing
> = scene=20 have enough compute cycles available to make any "offloading"
> = possible=20 as a mere code restructuring exercise (see the stack reports
> = from=20 Intel, IBM etc.)
>    *   Building on a complete = stack=20 makes available a wealth of
> operational and management = mechanisms=20 built over the years by the
> networking community (routing,=20 provisioning, security, service location
> etc.) - the community = argument
>    *   Higher level storage access = over an IP=20 network is widely
> available and having both block and file = served over=20 the same connection
> with the same support and management = structure is=20 compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't know=20 exactly
> how it will operate). Untill then the scale of Layer 2 = network=20 is going
> to seriously limited - the scaling argument
> =
>=20
>
>       As a side argument - a = performance=20 comparison made in
> 1998 showed SCSI over TCP (a predecessor of = the=20 later iSCSI) to perform
> better than FCP at 1Gbs for block = sizes=20 typical for OLTP (4-8KB). That
> was what convinced us to take = the path=20 that lead to iSCSI - and we used
> plain vanilla x86 servers = with=20 plain-vanilla NICs and Linux (with
> similar measurements = conducted on=20 Windows).
>    The networking and storage community=20 acknowledged those
> arguments and developed iSCSI and the = companion=20 protocols for service
> discovery, boot etc.
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a reasonable = fashion=20 and developed 2
> protocols iFCP (to support hosts with FCP = drivers and=20 IP connections to
> connect to storage by a simple conversion = from FCP=20 to TCP packets) FCPIP
> to extend the reach of FCP through IP = (connects=20 FCP islands through TCP
> links). Both have been
>   =  implemented and their foundation is solid.
>  =20  
>    The current attempt of developing a = "new-age" FCP=20 over an
> Ethernet link is going against most of the arguments = that have=20 given us
> iSCSI etc.
>    
>   =  It=20 ignores the networking layering practice, build an
> application = protocol directly above a link and thus limits scaling,
> = mandates=20 elements at the link layer and application layer that make
>=20 applications more expensive and leaves aside the whole "ecosystem"=20 that
> accompanies TCP/IP (and not Ethernet).
>  =20  
>    In some related effort (and at a point = also when=20 developing
> iSCSI) we considered also moving away from SCSI = (like some=20 "no
> standardized" but popular in some circles software did - = e.g.,=20 NBP) but
> decided against. SCSI is a mature and well understood = access
> architecture for block storage and is implemented by = many=20 device
> vendors. Moving away from it would not have been = justified at=20 the time.
>    =20

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= =20

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------=_NextPart_000_0028_01C7872A.1E34C720-- --===============1002509721== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1002509721==-- From ips-bounces@ietf.org Wed Apr 25 11:52:15 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgjmu-0001KF-18; Wed, 25 Apr 2007 11:52:12 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgjms-0001A3-Ey for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 11:52:10 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgjmr-000149-Us for ips@ietf.org; Wed, 25 Apr 2007 11:52:09 -0400 Received: from mtagate4.de.ibm.com ([195.212.29.153]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgjmp-0002TG-Vh for ips@ietf.org; Wed, 25 Apr 2007 11:52:09 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate4.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3PFq7NI142668 for ; Wed, 25 Apr 2007 15:52:07 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3PFq6gB3960964 for ; Wed, 25 Apr 2007 17:52:06 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3PFq6hg026124 for ; Wed, 25 Apr 2007 17:52:06 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3PFq67Y026118; Wed, 25 Apr 2007 17:52:06 +0200 In-Reply-To: <002b01c7874b$a57873a0$05faa8c0@ivivity.com> To: "Eddy Quicksall" MIME-Version: 1.0 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Wed, 25 Apr 2007 11:52:04 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 25/04/2007 18:52:06, Serialize complete at 25/04/2007 18:52:06 X-Spam-Score: 0.3 (/) X-Scan-Signature: c48eeb61c971c85fb2ddea78b04c1c76 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1184737519==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============1184737519== Content-Type: multipart/alternative; boundary="=_alternative 005727E4852572C8_=" This is a multipart message in MIME format. --=_alternative 005727E4852572C8_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Sorry Eddy by sizable I meant even at the size of a modern data center.=20 Julo "Eddy Quicksall" =20 25/04/07 11:08 To Julian Satran/Haifa/IBM@IBMIL cc ips@ietf.org, John Hufferd Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI I basically said that in the summery line by saying "it will not route on=20 the ?global? scale like TCP/IP would". ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Eddy,=20 That is oversimplified and ignore the drop rate assumption and error rate=20 assumptions made in FCP(FCP has no transport layer). To get to it on a=20 sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 To "John Hufferd" , Julian Satran/Haifa/IBM@IBMIL=20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI Basically, it is sending FC frames over Ethernet. This localizes the=20 traffic unless you route based on MAC addresses. So you send 2146 bytes of = FC frame plus 18 bytes of Ethernet overhead as FCoE ?standard? packet. 18=20 bytes of Ethernet gets stripped and you have straight FC frame that can go = through any FC network. Now you can have 10G Ethernet pipes into existing=20 FC SANs. Limited market potential as far as I can see. The key argument is = it much easier to implement than iSCSI and also has less overhead and uses = all the benefits of FC. End to End credits are simulated using PAUSE=20 command on Ethernet and MAC addresses are mapped into WWNs.=20 Biggest knock is that it will not route on the ?global? scale like TCP/IP=20 would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as an > outreach protocol from the Data Center. We also believe iSCSI is very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our new > iSCSI and TOE offload HBAs.=20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the=20 network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is not=20 true either=20 3. Gatewaying is expensive - and it is perhaps so but only if you=20 are completely relying on FCP storage (and there are plenty of good iSCSI=20 vendors of storage)and pushing the price on the servers is not cheap=20 either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on the > market, it is not cost feasible to replace all the server connectivity > to FC storage with iSCSI for the hundred to thousands of servers in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are too > high for iSCSI to be used as the "normal" server connect into a FC based > Fabric. That is true for the current 1GE; and for 10GE the cost is just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to understand > is that it is not Ethernet as we have known it up to know. The Ethernet > we are talking about is a type of Ethernet that can only be deployed in > a constrained environment such as a Data Center. This form of Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest of > the Host and storage stack remain the same, the functions and features > of the switches also remain the same and add the capability to provide > Cluster Message Switching which has latency close to InfiniBand speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or send FC > frames to DCE FCoE devices, just like one would do if it was all FC. And > all this is done while performing Cluster message switching and general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world.=20 First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing is=20 therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a=20 movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant equipment=20 and protocol changes.=20 Even its proponents do not call for transportless networks, lossless=20 networks etc.=20 The second trouble with your argument is that there are no known large=20 scale networking technologies that=20 really work at full speed (high speed) and are lossless (flow-controlled)=20 and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven=20 end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications are = built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement I=20 would prefer like any rational engineer, to base my building blocks on=20 structures that are proven and long lasting. And those are still the=20 end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS TWG=20 has developed the iFCP that does exactly what FCoE claims to do an a=20 better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on the > Data Center Fabric. And it maintains all the FC Fabric Services, and all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern server=20 to ethernet it will cost a bundle to connect to FCoE and it will force=20 users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no layer-2=20 technology for errorless/lossless operation at this scale and there is no=20 good reason to pursue one. The only possible reason (good reason) is the=20 bridging infrastructure but that infrastructure has a completely different = rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and messages > we struggled with when we started iSCSI. There is a consortium of folks > both working on the DCE (CEE) and the FCoE. Without the DCE the FCoE > will not happen.=20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some=20 politics) keep it form "exploding" and large storage vendors are=20 completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE (that=20 still has a way to go) to improve the QoS in the data-center (and for=20 storage too). You expect it to bring the loss rates down to the levels=20 that FCP assumes (FCP has no transport layer) and that is probably a pipe=20 dream. Todays transport solution for loss mitigation are far more cost=20 effective - and that's why iFCP is a better proposition as a transition=20 technology than FCoE and iSCSI with gateways is propably better in the=20 long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 >=20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian=5FSatran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM technology > group through FCoE as it is being placed in our plans).=20 > I will send you more info when I get to my computer. But you probably > were sent the Brocade charts. Please review them and I will follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) >=20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team even > had a look at ATM - still a computer network candidate at the time). I > won't get you over all of our rationale (and we went over some of them > again at the end of 1999 with a team from CISCO before we convened the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the > rest) but some of the reasons we choose to drop Fiber Channel over raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging and > are limited by the logical tree structure that bridges must follow. The > effort to combine routers and bridges (rbridges) is promising to change > that but it will take some time to finalize (and we don't know exactly > how it will operate). Untill then the scale of Layer 2 network is going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That > was what convinced us to take the path that lead to iSCSI - and we used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service > discovery, boot etc.=20 >=20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections to > connect to storage by a simple conversion from FCP to TCP packets) FCPIP > to extend the reach of FCP through IP (connects FCP islands through TCP > links). Both have been=20 > implemented and their foundation is solid.=20 >=20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given us > iSCSI etc.=20 >=20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet).=20 >=20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the time.=20 > =20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 005727E4852572C8_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Sorry Eddy by sizable I meant even at the size of a modern data center. Julo


"Eddy Quicksall&= quot; <Quicksall=5FiSCSI@Bellsouth.net>

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL
cc
ips@ietf.org, John Hufferd <jhuff= erd@Brocade.COM>
Subject
Re: FW: [Ips] Recent comments about FCoE and iSCSI





I basically said that in the summery line by saying &quo= t;it will not route on the “global” scale like TCP/IP would".
----- Original Message -----
From: Julian Satran
To: Eddy Quicksall
Cc: ips@ietf.org ; = John Hufferd
Sent: Wednesday, April 25, 2007 11:04 AM
Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI


Eddy,

That is oversimplified and ignore the drop rate assumption and error rate assumptions made in FCP(FCP has no transport layer). To get to it on a sizable network requires more than PAUSE.

Julo

"Eddy Quicksall&= quot; <Quicksall=5FiSCSI@Bellsouth.net= >

25/04/07 10:07 =


To
"John Hufferd"= <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent comments about FCoE and iSCSI







Basically, it is sending FC frames over Ethernet. This localizes the traffic unless you route based on MAC addresses. So you send 2146 bytes of FC frame plus 18 bytes of Ethernet overhead as FCoE “standard” packet. 1= 8 bytes of Ethernet gets stripped and you have straight FC frame that can go through any FC network. Now you can have 10G Ethernet pipes into existing FC SANs. Limited market potential as far as I can see. The key argument is it much easier to implement than iSCSI and also has less overhead and uses all the benefits of FC. End to End credits are simulated using PAUSE command on Ethernet and MAC addresses are mapped into WWNs.

Biggest knock is that it w= ill not route on the “global” scale like TCP/IP would.

Eddy=

 ----- Original Message -----

From: Julian Satran
To: John Hufferd
Cc: ips@ietf.org
Sent: Wednesday, April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI



"John Hufferd" <jhufferd@Brocade.COM= > wrote on 25/04/2007 02:45:51:

> Julian,
> To be sure you understand our position; Brocade is pushing iSCSI as an
> outreach protocol from the Data Center. We also believe iSCSI is very<= br> > useful for installations that do not have a Fibre Channel
> infrastructure, and in that case we will be able to sell them our new
> iSCSI and TOE offload HBAs.  
>
> When I say iSCSI is an outreach protocol, this is a statement that iSCSI
> is very important to connect "stranded" servers to the Fibre Channel
> Fabric.  That is, we sell iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers (software or HBA iSCSI initiators) to connect to the
> Enterprise "Bet Your Business" FC Storage.  This of course also applies
> to Desktops and Laptop systems, and systems at distance.
>

You make it sound like:
1.        most of the servers in the world have their storage on the network - and that is not the case
2.        FCP is basically better performing than iSCSI - and that is not true either
3.        Gatewaying is expen= sive - and it is perhaps so but only if you are completely relying on FCP storage (and there are plenty of good iSCSI vendors of storage)and pushing the price on the servers is not cheap either - at least not for the server buyer

> Now with that positioning, it is important to understand the limitatio= n
> to this strategy.  The primary problem is that iSCSI to FC Bridgi= ng
> (Gatewaying) is relatively expensive (compared to simple FC
> connections).  Though we have some of the best priced Gateways on the
> market, it is not cost feasible to replace all the server connectivity=
> to FC storage with iSCSI for the hundred to thousands of servers in the
> Data Center.  And since, if there is to be a consolidated Network=
> connection to the servers in the Data Center, there must be an
> evolutionary replacement of Server Connections to Storage.  That means
> there must be a bridge/Gateway approach.  And as I mentioned before,
> there is just too much cost in the iSCSI to FC Gateway.
>
> The issue is the server requirement to have a single connection type to
> handle cluster messaging, general messaging, and storage.  iSCSI is
> clearly an option for the storage, however, the gateway costs are too
> high for iSCSI to be used as the "normal" server connect into a FC based
> Fabric. That is true for the current 1GE; and for 10GE the cost is just
> out of sight.  The reason for this is the requirement for TCP/IP<= br> > termination and re-initiation with FC at the Gateway.
>
> Now with respect to FC over Ethernet the important thing to understand=
> is that it is not Ethernet as we have known it up to know.  The Ethernet
> we are talking about is a type of Ethernet that can only be deployed in
> a constrained environment such as a Data Center.  This form of Ethernet
> is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced
> Ethernet).  This form of Ethernet is a Loss-less type Ethernet, with
> multi-priority and Flow Control.  This is NOT an Internet or Intranet
> type of Ethernet.
>
> FCoE is all about using the DCE (CEE) to carry FC frames.  The rest of
> the Host and storage stack remain the same, the functions and features=
> of the switches also remain the same and add the capability to provide=
> Cluster Message Switching which has latency close to InfiniBand speeds= .
>
>
> Because the FC frames are transported to the switches intact via a DCE
> frame, the Bridging, if you want to call it that, is virtually non
> existent.  Hence you can deliver the FC frames to FC devices, or send FC
> frames to DCE FCoE devices, just like one would do if it was all FC. And
> all this is done while performing Cluster message switching and genera= l
> message trucking to the IP outfacing network.
>

The rosy future of the yet to appear DCE/CEE and a layer 2 only world.
First you have some terms confused: <= font size=3D2>

Bridging is the term commonly used for Layer-2 switching and routing is therm used for layer-3 (switching). <= font size=3D2>

Bridging has some advantages (less management) that have created a movement towards an enterprise wide
LAN. But this has a long way to go and will require significant equipment and protocol changes. =
Even its proponents do not call for transportless networks, lossless networ= ks etc.
The second trouble with your argument is that there are no known large scale networking technologies that
really work at full speed (high speed) and are lossless (flow-controlled) and errorless like FCoE assumes.
The TCP/IP has solved this issue for every generation using the proven end-to-end principle (and is doing so now).
And it is not by chance so and that is why all networking applications are built above layer-3 and not dropping layer-3 (like FCoE) does.

Although I can understand the DCE arguments as a management statement I would prefer like any rational engineer, to base my building blocks on structures that are proven and long lasting. And those are still the end-to= -end TCP/IP that can accommodate even your FCP addicts. The IPS TWG has developed the iFCP that does exactly what FCoE claims to do an a better base.<= /tt>

> This means an evolutionary process is possible to the solution of
> getting a single Fabric connection for all networks connected to a
> server, further, the process has very low interconnection cost on the
> Data Center Fabric. And it maintains all the FC Fabric Services, and all
> the same Storage Management processes.
>
> By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller.
> Therefore FC storage controllers (and FICON) will be the very last
> things that connect using FCoE and that evolution will take at least a
> decade or more.
>

It is server cost statement. It costs nothing to connect a modern server to ethernet it will cost a bundle to connect to FCoE and it will force users in short lived bad solutions. <= font size=3D2>


> We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the
> "Trunking" of general messaging to the Outfacing IP network.  That said;
> we do not see FCoE going beyond the constraints of the Data Center.

Data Centers now grow to tens of thousands of nodes. There is no layer-2 technology for errorless/lossless operation at this scale and there is no good reason to pursue one. The only possible reason (good reason) is the bridging infrastructure but that infrastructure has a completely differ= ent rationale than the flowcontrol.

> This issue and message is quite different from the issues and messages=
> we struggled with when we started iSCSI.  There is a consortium of folks
> both working on the DCE (CEE) and the FCoE.  Without the DCE the FCoE
> will not happen.  
>
> None of the above cancels out the value of iSCSI in numerous
> environments.
>
>

iSCSI is good for all environments. Business consideration (and some politi= cs) keep it form "exploding" and large storage vendors are completely indifferent to the network connection they are using.
You and I have also slightly different views of DCE. I expect DCE (that still has a way to go) to improve the QoS in the data-center (and for stora= ge too). You expect it to bring the loss rates down to the levels that FCP assumes (FCP has no transport layer) and that is probably a pipe dream. Todays transport solution for loss mitigation are far more cost effective - and that's why iFCP is a better proposition as a transition technology than FCoE and iSCSI with gateways is propably better in the long run.

>
> .
> .
> .
> John L Hufferd
> Sr. Executive Director of Technology
> jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606
>  
> -----Original Message-----
> From: John Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To: 'Julian=5FSatran@il.ibm.com'
> Subject: Re: [Ips] Recent comments about FCoE and iSCSI
>
> Julian,
> I think you are wrong on this one.  The arguments are quite diffe= rent
> then the ones we had in pre iSCSI days.  (By the way I missed you on
> today's Renato meeting/conf call where Brocade took the IBM technology=
> group through FCoE as it is being placed in our plans).  
> I will send you more info when I get to my computer.  But you probably
> were sent the Brocade charts.  Please review them and I will follow up
> with more information.
> This does NOT replace iSCSI it applies only to a DataCenter envuornmen= t
> with lossless DCE ethernet.
> --------------------------
> John L. Hufferd
> Sr. Ex. Director of Technology
> Brocade Communications Systems, Inc.
> Phone: (408) 333-5244
> Mobile: (408) 627-9606
> eMail: jhufferd@brocade.com
> (Sent from my BlackBerry Wireless)
>  
>
> ----- Original Message -----
> From: Julian Satran <Julian=5FSatran@il.ibm.com>
> To: ips@ietf.org <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> Subject: [Ips] Recent comments about FCoE and iSCSI
>
>
> Dear All,
>
> The trade press is lately full with comments about the latest and
> greatest reincarnation of Fiber Channel over ethernet.
> It made me try and summarize all the long and hot debates that precede= d
> the advent of iSCSI.
> Although FCoE proponents make it look like no debate preceded iSCSI that
> was not so - FCoE was considered even then and was dropped as a dumb > idea.
>
> Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE
> doesn't look better than it did then.
>
> Feel free to use this material in a nay form. I expect this group to
> seriously  expand my arguments and make them public - in personal or
> collective form.
>
> And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.
>
> Regards,
> Julo
>
> ---------------------------------------------------------------------
>
> What a piece of nostalgia :-)
>
> Around 1997 when a team at IBM Research (Haifa and Almaden) started
> looking at connecting storage to servers using the "regular netwo= rk"
> (the ubiquitous LAN) we considered many alternatives (another team even
> had a look at ATM - still a computer network candidate at the time). I
> won't get you over all of our rationale (and we went over some of them
> again at the end of 1999 with a team from CISCO before we convened the
> first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the
> rest) but some of the reasons we choose to drop Fiber Channel over raw
> Ethernet where multiple:
>
>
> *   Fiber Channel Protocol (SCSI over Fiber Channel Link) is
> "mildly" effective because:
>
>    *   it implements endpoints in a dedicated engine (Offload)
>    *   it has no transport layer (recovery is done at the
> application layer under the assumption that the error rate will be very
> low)
>    *   the network is limited in physical span and logical span
> (number of switches)
>    *   flow-control/congestion control is achieved with a
> mechanism adequate for a limited span network (credits). The packet loss
> rate is almost nil and that allows FCP to avoid using a transport
> (end-to-end) layer
>
>    *   FCP she switches are simple (addresses are local and the
> memory requirements cam be limited through the credit mechanism)
>    *   However FCP endpoints are inherently costlier than
> simple NICs - the cost argument (initiators are more expensive)
>    *   The credit mechanisms is highly unstable for large
> networks (check switch vendors planning docs for the network diameter<= br> > limits) - the scaling argument
>    *   The assumption of low losses due to errors might=
> radically change when moving from 1 to 10 Gb/s - the scaling argument
>    *   Ethernet has no credit mechanism and any mechani= sm with
> a similar effect increases the end point cost. Building a transport
> layer in the protocol stack has always been the preferred choice of the
> networking community - the community argument
>    *   The "performance penalty" of a complete protocol stack
> has always been overstated (and overrated). Advances in protocol stack=
> implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over.
> Moreover the multicore processors that become dominant on the computin= g
> scene have enough compute cycles available to make any "offloadin= g"
> possible as a mere code restructuring exercise (see the stack reports<= br> > from Intel, IBM etc.)
>    *   Building on a complete stack makes available a wealth of
> operational and management mechanisms built over the years by the
> networking community (routing, provisioning, security, service locatio= n
> etc.) - the community argument
>    *   Higher level storage access over an IP network is widely
> available and having both block and file served over the same connecti= on
> with the same support and management structure is compelling - the
> community argument
>    *   Highly efficient networks are easy to build over IP with
> optimal (shortest path) routing while Layer 2 networks use bridging and
> are limited by the logical tree structure that bridges must follow. The
> effort to combine routers and bridges (rbridges) is promising to chang= e
> that but it will take some time to finalize (and we don't know exactly=
> how it will operate). Untill then the scale of Layer 2 network is going
> to seriously limited - the scaling argument
>
>
>
>       As a side argument - a performance comparison made in
> 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perfor= m
> better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That
> was what convinced us to take the path that lead to iSCSI - and we used
> plain vanilla x86 servers with plain-vanilla NICs and Linux (with
> similar measurements conducted on Windows).
>    The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service<= br> > discovery, boot etc.
>    
>    The community also acknowledged the need to support exist= ing
> infrastructure and extend it in a reasonable fashion and developed 2
> protocols iFCP (to support hosts with FCP drivers and IP connections to
> connect to storage by a simple conversion from FCP to TCP packets) FCPIP
> to extend the reach of FCP through IP (connects FCP islands through TCP
> links). Both have been
>    implemented and their foundation is solid.
>    
>    The current attempt of developing a "new-age" FCP over an
> Ethernet link is going against most of the arguments that have given us
> iSCSI etc.
>    
>    It ignores the networking layering practice, build an
> application protocol directly above a link and thus limits scaling,
> mandates elements at the link layer and application layer that make
> applications more expensive and leaves aside the whole "ecosystem= " that
> accompanies TCP/IP (and not Ethernet).
>    
>    In some related effort (and at a point also when developi= ng
> iSCSI) we considered also moving away from SCSI (like some "no
> standardized" but popular in some circles software did - e.g., NBP) but
> decided against. SCSI is a mature and well understood access
> architecture for block storage and is implemented by many device
> vendors. Moving away from it would not have been justified at the time.
>    

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

--=_alternative 005727E4852572C8_=-- --===============1184737519== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1184737519==-- From ips-bounces@ietf.org Wed Apr 25 14:55:43 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgmeU-0003JM-7o; Wed, 25 Apr 2007 14:55:42 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgmeT-0003JH-7o for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 14:55:41 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgmeS-0003J9-Ty for ips@ietf.org; Wed, 25 Apr 2007 14:55:40 -0400 Received: from imf23aec.mail.bellsouth.net ([205.152.59.71]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgmeQ-0000kw-A6 for ips@ietf.org; Wed, 25 Apr 2007 14:55:40 -0400 Received: from ibm66aec.bellsouth.net ([74.245.52.54]) by imf23aec.mail.bellsouth.net with ESMTP id <20070425185537.ECVB9888.imf23aec.mail.bellsouth.net@ibm66aec.bellsouth.net> for ; Wed, 25 Apr 2007 14:55:37 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm66aec.bellsouth.net with SMTP id <20070425185535.LVNO18529.ibm66aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 14:55:35 -0400 Message-ID: <004801c7876b$4e23fbe0$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Julian Satran" References: Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 14:55:34 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.5 (/) X-Scan-Signature: 8cc1c558da2accc0f39b338f00bd6728 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0663459159==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0663459159== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0045_01C78749.C6DB6D60" This is a multi-part message in MIME format. ------=_NextPart_000_0045_01C78749.C6DB6D60 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable If it was not obvious in my wording, I meant to agree with you on your = observations. Eddy ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Sorry Eddy by sizable I meant even at the size of a modern data = center. Julo=20 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL =20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI = =20 =20 I basically said that in the summery line by saying "it will not route = on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 Eddy,=20 That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it = on a sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 =20 To "John Hufferd" , Julian = Satran/Haifa/IBM@IBMIL =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI = =20 =20 Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like = TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as = an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the = network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is = not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if = you are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers in = the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type = to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be deployed = in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest = of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all FC. = And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world. = First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known large = scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven = end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications = are built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement = I would prefer like any rational engineer, to base my building blocks on = structures that are proven and long lasting. And those are still the = end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS = TWG has developed the iFCP that does exactly what FCoE claims to do an a = better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, and = all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least = a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will follow = up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI = that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the time). = I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet = loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of = the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging = and > are limited by the logical tree structure that bridges must follow. = The > effort to combine routers and bridges (rbridges) is promising to = change > that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections = to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands through = TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given = us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) = but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org = https://www1.ietf.org/mailman/listinfo/ips_______________________________= ________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_0045_01C78749.C6DB6D60 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

If it was not obvious in my wording, I meant to = agree with you=20 on your observations.
 
Eddy
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 11:52=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Sorry Eddy by = sizable I meant=20 even at the size of a modern data center. Julo


"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL =
cc
ips@ietf.org, John = Hufferd <jhufferd@Brocade.COM> =
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

=




I=20 basically said that in the summery line by saying "it will not route on the =93global=94 scale like TCP/IP = would".=20
----- Original Message -----
From: Julian Satran=20
To: Eddy=20 Quicksall
Cc:=20 ips@ietf.org ; John=20 Hufferd
Sent:=20 Wednesday, April 25, 2007 11:04 AM
Subject: Re:=20 FW: [Ips] Recent comments about FCoE and iSCSI


Eddy, =

That is oversimplified and ignore the = drop rate=20 assumption and error rate assumptions made in FCP(FCP has no transport = layer).=20 To get to it on a sizable network requires more than = PAUSE.

Julo=20

"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.net>

25/04/07 10:07=20


To
"John = Hufferd"=20 <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI







Basically, it=20 is sending FC frames over Ethernet. This localizes the traffic unless = you=20 route based on MAC addresses. So you send 2146 bytes of FC frame plus = 18 bytes=20 of Ethernet overhead as FCoE =93standard=94 packet. 18 bytes of = Ethernet gets=20 stripped and you have straight FC frame that can go through any FC = network.=20 Now you can have 10G Ethernet pipes into existing FC SANs. Limited = market=20 potential as far as I can see. The key argument is it much easier to = implement=20 than iSCSI and also has less overhead and uses all the benefits of FC. = End to=20 End credits are simulated using PAUSE command on Ethernet and MAC = addresses=20 are mapped into WWNs.

Biggest knock is that = it will not=20 route on the =93global=94 scale like TCP/IP would.

Eddy

 -----=20 Original Message -----

From: Julian=20 Satran
To: John=20 Hufferd
Cc: ips@ietf.org =
Sent: Wednesday,=20 April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] Recent = comments about=20 FCoE and iSCSI



"John = Hufferd"=20 <jhufferd@Brocade.COM> wrote=20 on 25/04/2007 02:45:51:

> Julian,
> To be sure you = understand=20 our position; Brocade is pushing iSCSI as an
> outreach protocol = from=20 the Data Center. We also believe iSCSI is very
> useful for=20 installations that do not have a Fibre Channel
> infrastructure, = and in=20 that case we will be able to sell them our new
> iSCSI and TOE = offload=20 HBAs.  
>
> When I say iSCSI is an outreach = protocol, this=20 is a statement that iSCSI
> is very important to connect = "stranded"=20 servers to the Fibre Channel
> Fabric.  That is, we sell=20 iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers = (software=20 or HBA iSCSI initiators) to connect to the
> Enterprise "Bet = Your=20 Business" FC Storage.  This of course also applies
> to = Desktops=20 and Laptop systems, and systems at distance.
>=20

You make it sound = like:
1.     =  =20  most of the servers in the world have = their=20 storage on the network - and that is not the case=20
2.      =20  FCP is basically better performing = than iSCSI -=20 and that is not true either
3.       =  Gatewaying is expensive - and it is perhaps so but only if = you are=20 completely relying on FCP storage (and there are plenty of good iSCSI = vendors=20 of storage)and pushing the price on the servers is not cheap either - = at least=20 not for the server buyer

> Now with that positioning, it is important to = understand=20 the limitation
> to this strategy.  The primary problem is = that=20 iSCSI to FC Bridging
> (Gatewaying) is relatively expensive = (compared to=20 simple FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace all = the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers in the
> Data Center.  And since, if = there is=20 to be a consolidated Network
> connection to the servers in the = Data=20 Center, there must be an
> evolutionary replacement of Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are = too
>=20 high for iSCSI to be used as the "normal" server connect into a FC=20 based
> Fabric. That is true for the current 1GE; and for 10GE = the cost=20 is just
> out of sight.  The reason for this is the = requirement for=20 TCP/IP
> termination and re-initiation with FC at the = Gateway.
>=20
> Now with respect to FC over Ethernet the important thing to=20 understand
> is that it is not Ethernet as we have known it up = to know.=20  The Ethernet
> we are talking about is a type of Ethernet = that can=20 only be deployed in
> a constrained environment such as a Data = Center.=20  This form of Ethernet
> is called DCE (Data Center = Ethernet) or=20 CEE (Convergence Enhanced
> Ethernet).  This form of = Ethernet is a=20 Loss-less type Ethernet, with
> multi-priority and Flow Control. =  This is NOT an Internet or Intranet
> type of = Ethernet.
>=20
> FCoE is all about using the DCE (CEE) to carry FC frames. =  The=20 rest of
> the Host and storage stack remain the same, the = functions and=20 features
> of the switches also remain the same and add the = capability=20 to provide
> Cluster Message Switching which has latency close = to=20 InfiniBand speeds.
>
>
> Because the FC frames are = transported to the switches intact via a DCE
> frame, the = Bridging, if=20 you want to call it that, is virtually non
> existent. =  Hence you=20 can deliver the FC frames to FC devices, or send FC
> frames to = DCE FCoE=20 devices, just like one would do if it was all FC. And
> all this = is done=20 while performing Cluster message switching and general
> message = trucking to the IP outfacing network.
>

The rosy future = of the=20 yet to appear DCE/CEE and a layer 2 only world.=20
First you have some terms=20 confused:

Bridging is=20 the term commonly used for Layer-2 switching and routing is therm used = for=20 layer-3 (switching).

Bridging has some advantages (less management) that = have=20 created a movement towards an enterprise wide=20
LAN. But this has a long way to go and = will=20 require significant equipment and protocol changes.=20
Even its proponents do not call for = transportless=20 networks, lossless networks etc. =
The second trouble with your argument is that there are = no known=20 large scale networking technologies that=20
really work at full speed (high speed) = and are=20 lossless (flow-controlled) and errorless like FCoE = assumes.
The TCP/IP has solved this = issue for every=20 generation using the proven end-to-end principle (and is doing so=20 now).
And it = is not by=20 chance so and that is why all networking applications are built above = layer-3=20 and not dropping layer-3 (like FCoE) does.=20

Although I can understand the DCE = arguments as=20 a management statement I would prefer like any rational engineer, to = base my=20 building blocks on structures that are proven and long lasting. And = those are=20 still the end-to-end TCP/IP that can accommodate even your FCP = addicts. The=20 IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a=20 better base.

>=20 This means an evolutionary process is possible to the solution = of
>=20 getting a single Fabric connection for all networks connected to = a
>=20 server, further, the process has very low interconnection cost on = the
>=20 Data Center Fabric. And it maintains all the FC Fabric Services, and=20 all
> the same Storage Management processes.
>
> = By the=20 way, this is primarily a Server driven value statement, there
> = seems to=20 be little value in having FCoE on the storage controller.
> = Therefore FC=20 storage controllers (and FICON) will be the very last
> things = that=20 connect using FCoE and that evolution will take at least a
> = decade or=20 more.
>

It is server cost statement. It costs nothing to = connect=20 a modern server to ethernet it will cost a bundle to connect to FCoE = and it=20 will force users in short lived bad solutions.=20


> We see value in offering = switches and=20 Directors that can support DCE
> switching, FC switching as well = as=20 iSCSI interconnect, and the
> "Trunking" of general messaging to = the=20 Outfacing IP network.  That said;
> we do not see FCoE = going beyond=20 the constraints of the Data Center. =

Data Centers now grow to tens of thousands of nodes. = There is=20 no layer-2 technology for errorless/lossless operation at this scale = and there=20 is no good reason to pursue one. The only possible reason (good = reason) is the=20 bridging infrastructure but that infrastructure has a completely = different=20 rationale than the flowcontrol. =

> This issue and message is quite different from = the issues=20 and messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen.  
> =
> None of the above cancels out the value of iSCSI in = numerous
>=20 environments.
>
>

iSCSI is good for all = environments.=20 Business consideration (and some politics) keep it form "exploding" = and large=20 storage vendors are completely indifferent to the network connection = they are=20 using.
You and I have also slightly different views of DCE. I = expect DCE=20 (that still has a way to go) to improve the QoS in the data-center = (and for=20 storage too). You expect it to bring the loss rates down to the levels = that=20 FCP assumes (FCP has no transport layer) and that is probably a pipe = dream.=20 Todays transport solution for loss mitigation are far more cost = effective -=20 and that's why iFCP is a better proposition as a transition technology = than=20 FCoE and iSCSI with gateways is propably better in the long=20 run.

> =
>=20 .
> .
> .
> John L Hufferd
> Sr. Executive = Director of=20 Technology
> jhufferd@brocade.com
> Office Phone: (408) = 333-5244;=20 eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: = (408)=20 627-9606
>  
> -----Original Message-----
> = From: John=20 Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent comments = about=20 FCoE and iSCSI
>
> Julian,
> I think you are wrong = on this=20 one.  The arguments are quite different
> then the ones we = had in=20 pre iSCSI days.  (By the way I missed you on
> today's = Renato=20 meeting/conf call where Brocade took the IBM technology
> group = through=20 FCoE as it is being placed in our plans).  
> I will send = you more=20 info when I get to my computer.  But you probably
> were = sent the=20 Brocade charts.  Please review them and I will follow up
> = with=20 more information.
> This does NOT replace iSCSI it applies only = to a=20 DataCenter envuornment
> with lossless DCE ethernet.
>=20 --------------------------
> John L. Hufferd
> Sr. Ex. = Director of=20 Technology
> Brocade Communications Systems, Inc.
> Phone: = (408)=20 333-5244
> Mobile: (408) 627-9606
> eMail:=20 jhufferd@brocade.com
> (Sent from my BlackBerry = Wireless)
>=20  
>
> ----- Original Message -----
> From: = Julian=20 Satran <Julian_Satran@il.ibm.com>
> To: ips@ietf.org=20 <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> = Subject:=20 [Ips] Recent comments about FCoE and iSCSI
>
>
> = Dear All,=20
>
> The trade press is lately full with comments about = the=20 latest and
> greatest reincarnation of Fiber Channel over = ethernet.=20
> It made me try and summarize all the long and hot debates = that=20 preceded
> the advent of iSCSI.
> Although FCoE = proponents make=20 it look like no debate preceded iSCSI that
> was not so - FCoE = was=20 considered even then and was dropped as a dumb
> idea.
> =
>=20 Here is a summary (as afar as I can remember) of the main = arguments.
>=20 They are not bad arguments even in retrospect and technically = FCoE
>=20 doesn't look better than it did then.
>
> Feel free to = use this=20 material in a nay form. I expect this group to
> seriously =  expand=20 my arguments and make them public - in personal or
> collective = form.=20
>
> And do not forget - it is a technical dispute - = although we=20 all must
> have some doubts about the way it is pursued. =
>=20
> Regards,
> Julo
>
>=20 --------------------------------------------------------------------- =
>=20
> What a piece of nostalgia :-)
>
> Around 1997 = when a=20 team at IBM Research (Haifa and Almaden) started
> looking at = connecting=20 storage to servers using the "regular network"
> (the ubiquitous = LAN) we=20 considered many alternatives (another team even
> had a look at = ATM -=20 still a computer network candidate at the time). I
> won't get = you over=20 all of our rationale (and we went over some of them
> again at = the end=20 of 1999 with a team from CISCO before we convened the
> first = IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the
> rest) = but some=20 of the reasons we choose to drop Fiber Channel over raw
> = Ethernet where=20 multiple:
>
>
> *   Fiber Channel Protocol = (SCSI=20 over Fiber Channel Link) is
> "mildly" effective because: =
>=20
>    *   it implements endpoints in a dedicated = engine=20 (Offload)
>    *   it has no transport layer = (recovery=20 is done at the
> application layer under the assumption that the = error=20 rate will be very
> low)
>    *   the = network is=20 limited in physical span and logical span
> (number of switches) =
>    *   flow-control/congestion control is = achieved=20 with a
> mechanism adequate for a limited span network = (credits). The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *  =20 The assumption of low losses due to errors might
> radically = change when=20 moving from 1 to 10 Gb/s - the scaling argument
>   =  *  =20 Ethernet has no credit mechanism and any mechanism with
> a = similar=20 effect increases the end point cost. Building a transport
> = layer in the=20 protocol stack has always been the preferred choice of the
> = networking=20 community - the community argument
>    *   The=20 "performance penalty" of a complete protocol stack
> has always = been=20 overstated (and overrated). Advances in protocol stack
> = implementation=20 and finer tuning of the congestion control mechanisms
> make=20 conventional TCP/IP performing well even at 10 Gb/s and over.
> = Moreover=20 the multicore processors that become dominant on the computing
> = scene=20 have enough compute cycles available to make any "offloading"
> = possible=20 as a mere code restructuring exercise (see the stack reports
> = from=20 Intel, IBM etc.)
>    *   Building on a complete = stack=20 makes available a wealth of
> operational and management = mechanisms=20 built over the years by the
> networking community (routing,=20 provisioning, security, service location
> etc.) - the community = argument
>    *   Higher level storage access = over an IP=20 network is widely
> available and having both block and file = served over=20 the same connection
> with the same support and management = structure is=20 compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't know=20 exactly
> how it will operate). Untill then the scale of Layer 2 = network=20 is going
> to seriously limited - the scaling argument
> =
>=20
>
>       As a side argument - a = performance=20 comparison made in
> 1998 showed SCSI over TCP (a predecessor of = the=20 later iSCSI) to perform
> better than FCP at 1Gbs for block = sizes=20 typical for OLTP (4-8KB). That
> was what convinced us to take = the path=20 that lead to iSCSI - and we used
> plain vanilla x86 servers = with=20 plain-vanilla NICs and Linux (with
> similar measurements = conducted on=20 Windows).
>    The networking and storage community=20 acknowledged those
> arguments and developed iSCSI and the = companion=20 protocols for service
> discovery, boot etc.
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a reasonable = fashion=20 and developed 2
> protocols iFCP (to support hosts with FCP = drivers and=20 IP connections to
> connect to storage by a simple conversion = from FCP=20 to TCP packets) FCPIP
> to extend the reach of FCP through IP = (connects=20 FCP islands through TCP
> links). Both have been
>   =  implemented and their foundation is solid.
>  =20  
>    The current attempt of developing a = "new-age" FCP=20 over an
> Ethernet link is going against most of the arguments = that have=20 given us
> iSCSI etc.
>    
>   =  It=20 ignores the networking layering practice, build an
> application = protocol directly above a link and thus limits scaling,
> = mandates=20 elements at the link layer and application layer that make
>=20 applications more expensive and leaves aside the whole "ecosystem"=20 that
> accompanies TCP/IP (and not Ethernet).
>  =20  
>    In some related effort (and at a point = also when=20 developing
> iSCSI) we considered also moving away from SCSI = (like some=20 "no
> standardized" but popular in some circles software did - = e.g.,=20 NBP) but
> decided against. SCSI is a mature and well understood = access
> architecture for block storage and is implemented by = many=20 device
> vendors. Moving away from it would not have been = justified at=20 the time.
>    

_______________________________________________
Ips mailing=20 list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips =

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= _______________________________________________
Ips = mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

------=_NextPart_000_0045_01C78749.C6DB6D60-- --===============0663459159== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0663459159==-- From ips-bounces@ietf.org Wed Apr 25 17:41:17 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgpEg-0005Kb-ES; Wed, 25 Apr 2007 17:41:14 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgpEf-0005KU-FP for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 17:41:13 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgpEf-0005KK-53 for ips@ietf.org; Wed, 25 Apr 2007 17:41:13 -0400 Received: from imf18aec.mail.bellsouth.net ([205.152.59.66]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgpEd-0005Uq-Dw for ips@ietf.org; Wed, 25 Apr 2007 17:41:13 -0400 Received: from ibm59aec.bellsouth.net ([74.245.52.54]) by imf18aec.mail.bellsouth.net with ESMTP id <20070425214108.CYXU26197.imf18aec.mail.bellsouth.net@ibm59aec.bellsouth.net> for ; Wed, 25 Apr 2007 17:41:08 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm59aec.bellsouth.net with SMTP id <20070425214106.HLYY1083.ibm59aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 17:41:06 -0400 Message-ID: <007701c78782$6dd93c40$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Julian Satran" References: Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 17:41:05 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.5 (/) X-Scan-Signature: a92223e791a86c0e8c238fc6d8c729a6 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0363124536==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0363124536== Content-Type: multipart/alternative; boundary="----=_NextPart_000_0074_01C78760.E69F05A0" This is a multi-part message in MIME format. ------=_NextPart_000_0074_01C78760.E69F05A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I notice that Cisco has a patent on Fibre Channel over Ethernet. Is FCoE = something different? ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Sorry Eddy by sizable I meant even at the size of a modern data = center. Julo=20 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL =20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI = =20 =20 I basically said that in the summery line by saying "it will not route = on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 Eddy,=20 That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it = on a sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 =20 To "John Hufferd" , Julian = Satran/Haifa/IBM@IBMIL =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI = =20 =20 Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like = TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as = an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the = network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is = not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if = you are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers in = the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type = to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be deployed = in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest = of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all FC. = And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only world. = First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known large = scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven = end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications = are built above layer-3 and not dropping layer-3 (like FCoE) does.=20 Although I can understand the DCE arguments as a management statement = I would prefer like any rational engineer, to base my building blocks on = structures that are proven and long lasting. And those are still the = end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS = TWG has developed the iFCP that does exactly what FCoE claims to do an a = better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, and = all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least = a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will follow = up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI = that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the time). = I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet = loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of = the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging = and > are limited by the logical tree structure that bridges must follow. = The > effort to combine routers and bridges (rbridges) is promising to = change > that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections = to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands through = TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given = us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) = but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org = https://www1.ietf.org/mailman/listinfo/ips_______________________________= ________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_0074_01C78760.E69F05A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I notice that Cisco has a patent on Fibre Channel = over=20 Ethernet. Is FCoE something different?
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 11:52=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Sorry Eddy by = sizable I meant=20 even at the size of a modern data center. Julo


"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL =
cc
ips@ietf.org, John = Hufferd <jhufferd@Brocade.COM> =
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

=




I=20 basically said that in the summery line by saying "it will not route on the =93global=94 scale like TCP/IP = would".=20
----- Original Message -----
From: Julian Satran=20
To: Eddy=20 Quicksall
Cc:=20 ips@ietf.org ; John=20 Hufferd
Sent:=20 Wednesday, April 25, 2007 11:04 AM
Subject: Re:=20 FW: [Ips] Recent comments about FCoE and iSCSI


Eddy, =

That is oversimplified and ignore the = drop rate=20 assumption and error rate assumptions made in FCP(FCP has no transport = layer).=20 To get to it on a sizable network requires more than = PAUSE.

Julo=20

"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.net>

25/04/07 10:07=20


To
"John = Hufferd"=20 <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI







Basically, it=20 is sending FC frames over Ethernet. This localizes the traffic unless = you=20 route based on MAC addresses. So you send 2146 bytes of FC frame plus = 18 bytes=20 of Ethernet overhead as FCoE =93standard=94 packet. 18 bytes of = Ethernet gets=20 stripped and you have straight FC frame that can go through any FC = network.=20 Now you can have 10G Ethernet pipes into existing FC SANs. Limited = market=20 potential as far as I can see. The key argument is it much easier to = implement=20 than iSCSI and also has less overhead and uses all the benefits of FC. = End to=20 End credits are simulated using PAUSE command on Ethernet and MAC = addresses=20 are mapped into WWNs.

Biggest knock is that = it will not=20 route on the =93global=94 scale like TCP/IP would.

Eddy

 -----=20 Original Message -----

From: Julian=20 Satran
To: John=20 Hufferd
Cc: ips@ietf.org =
Sent: Wednesday,=20 April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] Recent = comments about=20 FCoE and iSCSI



"John = Hufferd"=20 <jhufferd@Brocade.COM> wrote=20 on 25/04/2007 02:45:51:

> Julian,
> To be sure you = understand=20 our position; Brocade is pushing iSCSI as an
> outreach protocol = from=20 the Data Center. We also believe iSCSI is very
> useful for=20 installations that do not have a Fibre Channel
> infrastructure, = and in=20 that case we will be able to sell them our new
> iSCSI and TOE = offload=20 HBAs.  
>
> When I say iSCSI is an outreach = protocol, this=20 is a statement that iSCSI
> is very important to connect = "stranded"=20 servers to the Fibre Channel
> Fabric.  That is, we sell=20 iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers = (software=20 or HBA iSCSI initiators) to connect to the
> Enterprise "Bet = Your=20 Business" FC Storage.  This of course also applies
> to = Desktops=20 and Laptop systems, and systems at distance.
>=20

You make it sound = like:
1.     =  =20  most of the servers in the world have = their=20 storage on the network - and that is not the case=20
2.      =20  FCP is basically better performing = than iSCSI -=20 and that is not true either
3.       =  Gatewaying is expensive - and it is perhaps so but only if = you are=20 completely relying on FCP storage (and there are plenty of good iSCSI = vendors=20 of storage)and pushing the price on the servers is not cheap either - = at least=20 not for the server buyer

> Now with that positioning, it is important to = understand=20 the limitation
> to this strategy.  The primary problem is = that=20 iSCSI to FC Bridging
> (Gatewaying) is relatively expensive = (compared to=20 simple FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace all = the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers in the
> Data Center.  And since, if = there is=20 to be a consolidated Network
> connection to the servers in the = Data=20 Center, there must be an
> evolutionary replacement of Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are = too
>=20 high for iSCSI to be used as the "normal" server connect into a FC=20 based
> Fabric. That is true for the current 1GE; and for 10GE = the cost=20 is just
> out of sight.  The reason for this is the = requirement for=20 TCP/IP
> termination and re-initiation with FC at the = Gateway.
>=20
> Now with respect to FC over Ethernet the important thing to=20 understand
> is that it is not Ethernet as we have known it up = to know.=20  The Ethernet
> we are talking about is a type of Ethernet = that can=20 only be deployed in
> a constrained environment such as a Data = Center.=20  This form of Ethernet
> is called DCE (Data Center = Ethernet) or=20 CEE (Convergence Enhanced
> Ethernet).  This form of = Ethernet is a=20 Loss-less type Ethernet, with
> multi-priority and Flow Control. =  This is NOT an Internet or Intranet
> type of = Ethernet.
>=20
> FCoE is all about using the DCE (CEE) to carry FC frames. =  The=20 rest of
> the Host and storage stack remain the same, the = functions and=20 features
> of the switches also remain the same and add the = capability=20 to provide
> Cluster Message Switching which has latency close = to=20 InfiniBand speeds.
>
>
> Because the FC frames are = transported to the switches intact via a DCE
> frame, the = Bridging, if=20 you want to call it that, is virtually non
> existent. =  Hence you=20 can deliver the FC frames to FC devices, or send FC
> frames to = DCE FCoE=20 devices, just like one would do if it was all FC. And
> all this = is done=20 while performing Cluster message switching and general
> message = trucking to the IP outfacing network.
>

The rosy future = of the=20 yet to appear DCE/CEE and a layer 2 only world.=20
First you have some terms=20 confused:

Bridging is=20 the term commonly used for Layer-2 switching and routing is therm used = for=20 layer-3 (switching).

Bridging has some advantages (less management) that = have=20 created a movement towards an enterprise wide=20
LAN. But this has a long way to go and = will=20 require significant equipment and protocol changes.=20
Even its proponents do not call for = transportless=20 networks, lossless networks etc. =
The second trouble with your argument is that there are = no known=20 large scale networking technologies that=20
really work at full speed (high speed) = and are=20 lossless (flow-controlled) and errorless like FCoE = assumes.
The TCP/IP has solved this = issue for every=20 generation using the proven end-to-end principle (and is doing so=20 now).
And it = is not by=20 chance so and that is why all networking applications are built above = layer-3=20 and not dropping layer-3 (like FCoE) does.=20

Although I can understand the DCE = arguments as=20 a management statement I would prefer like any rational engineer, to = base my=20 building blocks on structures that are proven and long lasting. And = those are=20 still the end-to-end TCP/IP that can accommodate even your FCP = addicts. The=20 IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a=20 better base.

>=20 This means an evolutionary process is possible to the solution = of
>=20 getting a single Fabric connection for all networks connected to = a
>=20 server, further, the process has very low interconnection cost on = the
>=20 Data Center Fabric. And it maintains all the FC Fabric Services, and=20 all
> the same Storage Management processes.
>
> = By the=20 way, this is primarily a Server driven value statement, there
> = seems to=20 be little value in having FCoE on the storage controller.
> = Therefore FC=20 storage controllers (and FICON) will be the very last
> things = that=20 connect using FCoE and that evolution will take at least a
> = decade or=20 more.
>

It is server cost statement. It costs nothing to = connect=20 a modern server to ethernet it will cost a bundle to connect to FCoE = and it=20 will force users in short lived bad solutions.=20


> We see value in offering = switches and=20 Directors that can support DCE
> switching, FC switching as well = as=20 iSCSI interconnect, and the
> "Trunking" of general messaging to = the=20 Outfacing IP network.  That said;
> we do not see FCoE = going beyond=20 the constraints of the Data Center. =

Data Centers now grow to tens of thousands of nodes. = There is=20 no layer-2 technology for errorless/lossless operation at this scale = and there=20 is no good reason to pursue one. The only possible reason (good = reason) is the=20 bridging infrastructure but that infrastructure has a completely = different=20 rationale than the flowcontrol. =

> This issue and message is quite different from = the issues=20 and messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen.  
> =
> None of the above cancels out the value of iSCSI in = numerous
>=20 environments.
>
>

iSCSI is good for all = environments.=20 Business consideration (and some politics) keep it form "exploding" = and large=20 storage vendors are completely indifferent to the network connection = they are=20 using.
You and I have also slightly different views of DCE. I = expect DCE=20 (that still has a way to go) to improve the QoS in the data-center = (and for=20 storage too). You expect it to bring the loss rates down to the levels = that=20 FCP assumes (FCP has no transport layer) and that is probably a pipe = dream.=20 Todays transport solution for loss mitigation are far more cost = effective -=20 and that's why iFCP is a better proposition as a transition technology = than=20 FCoE and iSCSI with gateways is propably better in the long=20 run.

> =
>=20 .
> .
> .
> John L Hufferd
> Sr. Executive = Director of=20 Technology
> jhufferd@brocade.com
> Office Phone: (408) = 333-5244;=20 eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: = (408)=20 627-9606
>  
> -----Original Message-----
> = From: John=20 Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent comments = about=20 FCoE and iSCSI
>
> Julian,
> I think you are wrong = on this=20 one.  The arguments are quite different
> then the ones we = had in=20 pre iSCSI days.  (By the way I missed you on
> today's = Renato=20 meeting/conf call where Brocade took the IBM technology
> group = through=20 FCoE as it is being placed in our plans).  
> I will send = you more=20 info when I get to my computer.  But you probably
> were = sent the=20 Brocade charts.  Please review them and I will follow up
> = with=20 more information.
> This does NOT replace iSCSI it applies only = to a=20 DataCenter envuornment
> with lossless DCE ethernet.
>=20 --------------------------
> John L. Hufferd
> Sr. Ex. = Director of=20 Technology
> Brocade Communications Systems, Inc.
> Phone: = (408)=20 333-5244
> Mobile: (408) 627-9606
> eMail:=20 jhufferd@brocade.com
> (Sent from my BlackBerry = Wireless)
>=20  
>
> ----- Original Message -----
> From: = Julian=20 Satran <Julian_Satran@il.ibm.com>
> To: ips@ietf.org=20 <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> = Subject:=20 [Ips] Recent comments about FCoE and iSCSI
>
>
> = Dear All,=20
>
> The trade press is lately full with comments about = the=20 latest and
> greatest reincarnation of Fiber Channel over = ethernet.=20
> It made me try and summarize all the long and hot debates = that=20 preceded
> the advent of iSCSI.
> Although FCoE = proponents make=20 it look like no debate preceded iSCSI that
> was not so - FCoE = was=20 considered even then and was dropped as a dumb
> idea.
> =
>=20 Here is a summary (as afar as I can remember) of the main = arguments.
>=20 They are not bad arguments even in retrospect and technically = FCoE
>=20 doesn't look better than it did then.
>
> Feel free to = use this=20 material in a nay form. I expect this group to
> seriously =  expand=20 my arguments and make them public - in personal or
> collective = form.=20
>
> And do not forget - it is a technical dispute - = although we=20 all must
> have some doubts about the way it is pursued. =
>=20
> Regards,
> Julo
>
>=20 --------------------------------------------------------------------- =
>=20
> What a piece of nostalgia :-)
>
> Around 1997 = when a=20 team at IBM Research (Haifa and Almaden) started
> looking at = connecting=20 storage to servers using the "regular network"
> (the ubiquitous = LAN) we=20 considered many alternatives (another team even
> had a look at = ATM -=20 still a computer network candidate at the time). I
> won't get = you over=20 all of our rationale (and we went over some of them
> again at = the end=20 of 1999 with a team from CISCO before we convened the
> first = IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the
> rest) = but some=20 of the reasons we choose to drop Fiber Channel over raw
> = Ethernet where=20 multiple:
>
>
> *   Fiber Channel Protocol = (SCSI=20 over Fiber Channel Link) is
> "mildly" effective because: =
>=20
>    *   it implements endpoints in a dedicated = engine=20 (Offload)
>    *   it has no transport layer = (recovery=20 is done at the
> application layer under the assumption that the = error=20 rate will be very
> low)
>    *   the = network is=20 limited in physical span and logical span
> (number of switches) =
>    *   flow-control/congestion control is = achieved=20 with a
> mechanism adequate for a limited span network = (credits). The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *  =20 The assumption of low losses due to errors might
> radically = change when=20 moving from 1 to 10 Gb/s - the scaling argument
>   =  *  =20 Ethernet has no credit mechanism and any mechanism with
> a = similar=20 effect increases the end point cost. Building a transport
> = layer in the=20 protocol stack has always been the preferred choice of the
> = networking=20 community - the community argument
>    *   The=20 "performance penalty" of a complete protocol stack
> has always = been=20 overstated (and overrated). Advances in protocol stack
> = implementation=20 and finer tuning of the congestion control mechanisms
> make=20 conventional TCP/IP performing well even at 10 Gb/s and over.
> = Moreover=20 the multicore processors that become dominant on the computing
> = scene=20 have enough compute cycles available to make any "offloading"
> = possible=20 as a mere code restructuring exercise (see the stack reports
> = from=20 Intel, IBM etc.)
>    *   Building on a complete = stack=20 makes available a wealth of
> operational and management = mechanisms=20 built over the years by the
> networking community (routing,=20 provisioning, security, service location
> etc.) - the community = argument
>    *   Higher level storage access = over an IP=20 network is widely
> available and having both block and file = served over=20 the same connection
> with the same support and management = structure is=20 compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't know=20 exactly
> how it will operate). Untill then the scale of Layer 2 = network=20 is going
> to seriously limited - the scaling argument
> =
>=20
>
>       As a side argument - a = performance=20 comparison made in
> 1998 showed SCSI over TCP (a predecessor of = the=20 later iSCSI) to perform
> better than FCP at 1Gbs for block = sizes=20 typical for OLTP (4-8KB). That
> was what convinced us to take = the path=20 that lead to iSCSI - and we used
> plain vanilla x86 servers = with=20 plain-vanilla NICs and Linux (with
> similar measurements = conducted on=20 Windows).
>    The networking and storage community=20 acknowledged those
> arguments and developed iSCSI and the = companion=20 protocols for service
> discovery, boot etc.
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a reasonable = fashion=20 and developed 2
> protocols iFCP (to support hosts with FCP = drivers and=20 IP connections to
> connect to storage by a simple conversion = from FCP=20 to TCP packets) FCPIP
> to extend the reach of FCP through IP = (connects=20 FCP islands through TCP
> links). Both have been
>   =  implemented and their foundation is solid.
>  =20  
>    The current attempt of developing a = "new-age" FCP=20 over an
> Ethernet link is going against most of the arguments = that have=20 given us
> iSCSI etc.
>    
>   =  It=20 ignores the networking layering practice, build an
> application = protocol directly above a link and thus limits scaling,
> = mandates=20 elements at the link layer and application layer that make
>=20 applications more expensive and leaves aside the whole "ecosystem"=20 that
> accompanies TCP/IP (and not Ethernet).
>  =20  
>    In some related effort (and at a point = also when=20 developing
> iSCSI) we considered also moving away from SCSI = (like some=20 "no
> standardized" but popular in some circles software did - = e.g.,=20 NBP) but
> decided against. SCSI is a mature and well understood = access
> architecture for block storage and is implemented by = many=20 device
> vendors. Moving away from it would not have been = justified at=20 the time.
>    

_______________________________________________
Ips mailing=20 list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips =

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= _______________________________________________
Ips = mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------=_NextPart_000_0074_01C78760.E69F05A0-- --===============0363124536== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0363124536==-- From ips-bounces@ietf.org Wed Apr 25 20:42:27 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgs42-000180-55; Wed, 25 Apr 2007 20:42:26 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgs40-00017v-E0 for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 20:42:24 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgs40-00017n-3o for ips@ietf.org; Wed, 25 Apr 2007 20:42:24 -0400 Received: from mail.brocade.com ([66.243.153.242] helo=mx10.brocade.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgs3y-0004qP-PW for ips@ietf.org; Wed, 25 Apr 2007 20:42:24 -0400 Received: from mailhost.brocade.com (HELO discus.brocade.com) ([192.168.126.240]) by mx10.brocade.com with ESMTP; 25 Apr 2007 17:42:22 -0700 X-IronPort-AV: i="4.14,452,1170662400"; d="gif'147?scan'147,208,217,147"; a="9519139:sNHT57011815" Received: from HQ-EXCHFE-2.corp.brocade.com (hq-vipexchfe-2.brocade.com [192.168.126.214]) by discus.brocade.com (Postfix) with ESMTP id 5E5CF23836B; Wed, 25 Apr 2007 17:41:11 -0700 (PDT) Received: from hq-exch-1.corp.brocade.com ([10.3.8.21]) by HQ-EXCHFE-2.corp.brocade.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 25 Apr 2007 17:42:21 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 17:42:20 -0700 Message-ID: <39BA3BC178B4394DB184389E88A97F8C022C5C6C@hq-exch-1.corp.brocade.com> In-Reply-To: <006001c7876e$12cc26f0$05faa8c0@ivivity.com> X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHbhab+EXeFVfgQWiH7jzYWydYaAAKX7IQ From: "John Hufferd" To: "Eddy Quicksall" , "Julian Satran" X-OriginalArrivalTime: 26 Apr 2007 00:42:21.0806 (UTC) FILETIME=[C03A10E0:01C7879B] X-Spam-Score: 0.2 (/) X-Scan-Signature: 1459dca363a7eac530b0f3f218abff0f Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0678873095==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0678873095== Content-class: urn:content-classes:message Content-Type: multipart/related; boundary="----_=_NextPart_001_01C7879B.BFF8E3C0"; type="multipart/alternative" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7879B.BFF8E3C0 Content-Type: multipart/alternative; boundary="----_=_NextPart_002_01C7879B.BFF8E3C0" ------_=_NextPart_002_01C7879B.BFF8E3C0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Eddy, The pause that is being analyzed as part of a Data Center Ethernet is NOT the total stuff that is being handled. The proposal that is being worked on is called PPP (Per Priority Pause - which probably should be called Per Class Pause). The purpose of this is to use it as a flow control alternative to a credit based approach which seems to be "not going to happen" in the 802.1 committee. This is a pause by class or priority instead of the current Pause that effects every thing on the link. However, this is only a part of the total solution. Another committee is working on the Congestion issues, however, a number of vendors, after looking at all the issues believe that we need more than just the various congestion management approaches that are being examined. Therefore, many vendor think that adding a flow control such as PPP can be part of the total solution to providing management of congestion, at least for some of the higher priorities (classes). =20 =20 That being said let me say, that FCoE is not to be considered a replacement for iSCSI. It is just another tool for providing storage to the application. One approach does not fit all needs. Fibre Channel is important to the Enterprise Market. They are not going to rip out and replace their current FC infrastructure and replace it with an all iSCSI or all FCoE network. So FC is going to be with us. I do not understand why folks are going so hyper over the FC frames being transported on a special Data Center Ethernet link instead of a FC physical Link. It is still FC, and we have that today.=20 =20 Many servers are asking for an evolutionary way to combine their Networking connections from the Server. The customers I have dealt with do NOT want to rip out FC, they want to provide a single Link for transport of all networking needs, including storage, exiting their servers. One needs to understand: this is still FC. But now the needs of a single connection type can be phased in one Host system at a time, and at some other time even the storage can be connected via FCoE. However, there does not seem to be a compelling argument for the use of FCoE to storage, at least not as compelling as the Server side. But some vendors will probably offer an FCoE connected storage controller. This however, has little need for link consolidation, and FC will continue to operate very well and I expect that even is FCoE is accepted at the server side, most storage controller will remain FC. =20 =20 iSCSI still has an important place in the enterprise, but one approach (FC, iSCSI, or FCoE) does not fit all. =20 . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com =20 Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =20 ________________________________ From: Eddy Quicksall [mailto:Quicksall_iSCSI@Bellsouth.net]=20 Sent: Wednesday, April 25, 2007 12:15 PM To: Eddy Quicksall; Julian Satran Cc: ips@ietf.org; John Hufferd Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 Further, I think PAUSE is only for point-to-point and I think that is too restrictive.=20 =20 Eddy ----- Original Message -----=20 From: Eddy Quicksall =20 To: Julian Satran =20 Cc: ips@ietf.org ; John Hufferd =20 Sent: Wednesday, April 25, 2007 2:55 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 If it was not obvious in my wording, I meant to agree with you on your observations. =20 Eddy ----- Original Message -----=20 From: Julian Satran =20 To: Eddy Quicksall =20 Cc: ips@ietf.org ; John Hufferd =20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =09 Sorry Eddy by sizable I meant even at the size of a modern data center. Julo=20 =09 =09 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL=20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 =20 =09 =09 =09 I basically said that in the summery line by saying "it will not route on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran =20 To: Eddy Quicksall =20 Cc: ips@ietf.org ; John Hufferd =20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 =09 =09 Eddy,=20 =09 That is oversimplified and ignore the drop rate assumption and error rate assumptions made in FCP(FCP has no transport layer). To get to it on a sizable network requires more than PAUSE.=20 =09 Julo=20 "Eddy Quicksall" >=20 25/04/07 10:07=20 =20 To "John Hufferd" >, Julian Satran/Haifa/IBM@IBMIL =20 cc >=20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 =20 =09 =09 =09 =09 Basically, it is sending FC frames over Ethernet. This localizes the traffic unless you route based on MAC addresses. So you send 2146 bytes of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" packet. 18 bytes of Ethernet gets stripped and you have straight FC frame that can go through any FC network. Now you can have 10G Ethernet pipes into existing FC SANs. Limited market potential as far as I can see. The key argument is it much easier to implement than iSCSI and also has less overhead and uses all the benefits of FC. End to End credits are simulated using PAUSE command on Ethernet and MAC addresses are mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like TCP/IP would.=20 Eddy=20 ------_=_NextPart_002_01C7879B.BFF8E3C0 Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable

Eddy,

The pause that is being analyzed as = part of a Data Center Ethernet is NOT the total stuff that is being = handled.  The proposal that is being worked on is called PPP (Per Priority Pause = – which probably should be called Per Class Pause).  The purpose of = this is to use it as a flow control alternative to a credit based approach which = seems to be “not going to happen” in the 802.1 committee. =  This is a pause by class or priority instead of the current Pause that effects = every thing on the link.  However, this is only a part of the total = solution.  Another committee is working on the Congestion issues, however, a number of = vendors, after looking at all the issues believe that we need more than just the = various congestion management approaches that are being examined. =  Therefore, many vendor think that adding a flow control such  as PPP can be part of = the total solution to providing management of congestion, at least for some = of the higher priorities (classes).  

 

That being said let me say, that = FCoE is not to be considered a replacement for iSCSI.  It is just another = tool for providing storage to the application.  One approach does not fit = all needs.  Fibre Channel is important to the Enterprise Market. =  They are not going to rip out and replace their current FC infrastructure and replace it with an all iSCSI or all FCoE network.  So FC is going = to be with us.  I do not understand why folks are going so hyper over the = FC frames being transported on a special Data Center Ethernet link instead = of a FC physical Link.  It is still FC, and we have that today. =

 

Many servers are asking for an evolutionary way to combine their Networking connections from the = Server.  The customers I have dealt with do NOT want to rip out FC, they want to = provide a single Link for transport of all networking needs, including storage, = exiting their servers.  One needs to understand: this is still FC. But now = the needs of a single connection type can be phased in one Host system at a = time, and at some other time even the storage can be connected via FCoE.  = However, there does not seem to be a compelling argument for the use of FCoE to = storage, at least not as compelling as the Server side.  But some vendors = will probably offer an FCoE connected storage controller.  This however, = has little need for link consolidation, and FC will continue to operate very = well and I expect that even is FCoE is accepted at the server side, most = storage controller will remain FC.  

 

iSCSI still has an important place = in the enterprise, but one approach (FC, iSCSI, or FCoE) does not fit = all.

 

.

.

.

John L Hufferd

Sr. Executive Director of = Technology

jhufferd@brocade.com

Office Phone: (408) 333-5244; eFAX: = (408) 904-4688

Alt Office Phone: (408) 997-6136; = Cell: (408) 627-9606

From: Eddy = Quicksall [mailto:Quicksall_iSCSI@Bellsouth.net]
Sent: Wednesday, April = 25, 2007 12:15 PM
To: Eddy Quicksall; = Julian Satran
Cc: ips@ietf.org; John = Hufferd
Subject: Re: FW: [Ips] = Recent comments about FCoE and iSCSI

 

Further, I think PAUSE is only for point-to-point and = I think that is too restrictive.

 

Eddy

----- Original Message ----- =

From: Eddy Quicksall =

Sent: = Wednesday, April 25, 2007 2:55 PM

Subject: Re: FW: = [Ips] Recent comments about FCoE and iSCSI

 

If it was not obvious in my wording, I meant to agree with you = on your observations.

 

Eddy

----- Original Message ----- =

Sent: = Wednesday, April 25, 2007 11:52 AM

Subject: Re: FW: = [Ips] Recent comments about FCoE and iSCSI

 


Sorry Eddy by sizable I meant even at the size = of a modern data center. Julo

"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.n= et> =

25/04/07 11:08

To

Julian Satran/Haifa/IBM@IBMIL<= /font>

cc

ips@ietf.org, John Hufferd <jhufferd@Brocade.COM><= /font>

Subject

Re: FW: [Ips] Recent comments about = FCoE and iSCSI

 

 

 




I = basically said that in the summery line by saying "it will not route on the “global” scale like TCP/IP = would".
----- Original Message -----
From: Julian Satran
To: Eddy Quicksall
Cc: ips@ietf.org ; John Hufferd
Sent: Wednesday, April = 25, 2007 11:04 AM
Subject: Re: FW: [Ips] = Recent comments about FCoE and iSCSI


Eddy,

That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it on a = sizable network requires more than PAUSE.

Julo

"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.net<= font size=3D1 face=3Dsans-serif>> =

25/04/07 10:07

 

To

"John Hufferd" = <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL

cc

<ips@ietf.org>

Subject

Re: FW: [Ips] Recent comments about = FCoE and iSCSI

 

 

 





Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes of FC = frame plus 18 bytes of Ethernet overhead as FCoE “standard” = packet. 18 bytes of Ethernet gets stripped and you have straight FC frame that can = go through any FC network. Now you can have 10G Ethernet pipes into = existing FC SANs. Limited market potential as far as I can see. The key argument is = it much easier to implement than iSCSI and also has less overhead and uses all = the benefits of FC. End to End credits are simulated using PAUSE command on Ethernet and MAC addresses are mapped into WWNs. =

Biggest knock is that it will not route on the “global” scale like TCP/IP would. =

Eddy

------_=_NextPart_002_01C7879B.BFF8E3C0-- ------_=_NextPart_001_01C7879B.BFF8E3C0 Content-Type: image/gif; name="image001.gif" Content-Transfer-Encoding: base64 Content-ID: Content-Description: image001.gif Content-Location: image001.gif R0lGODlhbgAtAPcAAAAAAP////7+/uwMEewPFO0VGvIzNvJFSfFESPNGSvRmafeHifaOj/yXmPaU lveeoPvExeoACOoJEOwKEu4NFewMFOwMFukME+oNFugNFesOFe8PF+wPFuwPGOsPGesPF+0QGusQ GOsQFukQGe0RF+oRGesSG+8THO0UHOsUG+8VHe4VHu0WHu4XH+0XIe0XH+4YIO4YIusYIO8aIewZ IusZH+waI+saIO8cI+4bJO4bIu0bJO0bIekbJPAdJu4cJe0cIuwdJesdJesdIu0eJuseJ/AgKewg Je4nLvAqMu8tNfAxOO8yOfEzOfAzOvE1O/E2PO85P/E9QvBARfRDS/JDSfVESvJFS/RHTfJHT/FH TfVKUvRLUPJKTvVMU/NMUvJOVfJRVfNUWfZYXvRcYvZscfVzd/d8gPqvsvzU1f3Z2v/l5v7k5f3k 5esHE+0KFe4MGOoLFu0NGO4OGesOHe0PGusTHu0UH/AaJe0ZJe8bJ+8cKu0bJu4cJ/AwO/FDTPNJ UfFJUf7q6/vn6P7t7v/v8P7o6v/i5f3p6/7m6f/q7f73+P7y9P/8/f/5/P/9/tfV2P7+/9/f4Pv/ /v3//tbY19TW1ff8+Pz//P7/+////f359v3y7v76+fzy8P3o5f7s6v3r6f/n5f708//k4v3p6Pvt 7P3g3/7p6Pvm5f+gnvulpfq8vPvOzv/i4v7m5vvj4//x8f/y8v3x8f/19f39/fr6+vn5+ff39/X1 9fPz8/Dw8O3t7erq6ujo6Obm5uXl5ePj4+Hh4d3d3dra2tfX19XV1dLS0s3NzcnJycXFxb+/v7m5 uba2trKysq2trampqaWlpaKiopycnJSUlI6OjouLi4iIiIODg319fXp6enV1dXJycnBwcG1tbWtr a2lpaWZmZmFhYVtbW1dXV1FRUUpKSkRERD8/Pzc3NzMzMzAwMCsrKykpKSYmJiQkJCIiIiAgIB4e Hh0dHRkZGRUVFRISEhEREQ8PDwsLCwkJCQYGBgUFBQMDAwEBAf///yH5BAEAAP8ALAAAAABuAC0A AAj/AAMIHEiwoMGDCBMqTKhJIKVIAxsKzDRQQAABmiwGkHiw4aQAjzZS3Bhg5MKTKFOi1CRx06FE pQQZmmmolCFUiRAlMoTI0E6aQA2ZohUAU8lNsWg16mRSpdOnKUMOJKSgSZU/CBBYqXIFwZ8EBxCE rULlgNmzZhF8KfPgUEg1qlahYdUJqt27CDOxHMgqSYQPHUB8ANEhBAcNIUBs4CCisePHdeRImIOl FSVSaBqUWWXrI97PoCeFfARByhseQn70AXLEzhsLcWDDuWChtu3aE0D0keEmippOn06lIdUUtHGo mSYZbSVGwo3VQPZ0GROIi/UuXqxr195FCYEhOtyE/wl1MQDE4+hTSqSo6RHFRGXevBiyAwYYVoVe pVrDps2r/wD+l0oaCtwRRAsnPKAXJQJplN6DHZE0kUCazLIADSPsMAQPTzywCEkcHQRKFyYEgYEZ ID7iGYQspuQIGlF8EAMQJ8QQCBqNLNSQAy8AoQEZo7QopHpGZSLAIWW0YAIPMJAAAxkQLHJeAAxG pMkDMgzRQRiKUBjikGASJFEkjTxggAko3NBDCHgoAIEodQkU0keOlHHCEAMo0JBGDobpJ0kMCqDG GUqUAMIOMliwwhRnoKHGILSERAgDRsQAgwYL/KkpQVWGtJcmEJiBxAQWyHDDBxPA8EQYZSzAABkz oP8wRAZ+pEHlpptiFEAtmoi2EahmTIEEHkDggIIFbwwwwAQYuIDDC0is8ogAjUj0Ja7pNcRRJOdp 1AgEYsCwgw1EHHGEDXvYgEMROLhAhkUYWYtti1Pu9YhRAqWxQCBIAJFaDCh8YAEGFFgQQxEvKLGK nPP6ie9eArVCKAh1sJDDqTM4AQgZZZQRhg558CFHFLb22TC9DD9yyAJJoMBCDDScIAMUZ7Byiie0 0LLJKA0McUMMIyxg8skPyhvAJmhIUQcNOrhgBw5grELIRhZJBZICKvCAQRmaMAgx0ehxJKgZOpig Rw1z+CAGGp1EwlGVIz1QQw8diFEI2C1+pAkmrDz/IUELQ6DZBRoZlRRhAA2wgMcHYkyNN3oCWMQS Iw/4MMEPQ5yghAOEbEuQXhQSEsYJP0zAtZePfxY5RvAJRsQNPISxSiGCnMIGLIe04Qoph5DiShtt sAGBAjrYwEIKDzSofOp2rX7KGBOsIAQQMvgw3RddAJLFFlpoAQggXWQRSBZcAKIEBy4I8QYgUw9d HkGRL+Q+mBzhm0YXdMBAXw85DLHCCB4gTAc84IEOGPCABhwBCfKQhwooAQLtMcj8huYgk81vee87 yAULoolGUKIVUYiDC2TgghbA4AUjlMELTrhCGMTghTB84QtmUAARaAGChYOf6pTnvg1usCJWak8n /85gBCU0gQlNUIIBluCHJTjxiVCM4hPD8IBEhCR+EsQgSuClwQe5rxOc8IQnYjGKQoxCFmhMYyzS yMY2eoIRneia4RJiQSD2iYt2fMoF+cRB8xDkESG6Fkrwha+CYBGIWqyFIZenSERisBZ47OHyIjcS TLjHI4gUZEKMwhJPFYeHdMxjBit4kTpSMIOj3JVKNOkllrgSYiOZkkF6cQxjFMMYxKiIL45RjFsS IxcYJAYzlIELDPZiGcvoBSKFcYxdKM8XtjSGMXhBEFsMwxKVgEQxjiGJS8gSL19LiC2owY53zGMe 7QjHLwRgi2uwwx3ykMc7zgGN8ugCG+7ghz3Kkf8MgjwDHfnIBzqW8T5hnIMd2yAINt55znRcw5kB 8IU52AGPeMRjHuUIxg8fZItt+AMd4AjHOgAQjlrg4hsAWEc4xFEOANCDGQHYRTcAoI5siCOlyBDI M+yxj3B8Ax/v6KdApgEAAMQjGAO56TrAIY53AGAcvggAMOYBAHOMgxzl0IYyN2ULbfAjGrfAxTDi gQ9e2GKm1LjFLXRxDQCIIwDP6Mc6jiGAXGyDpDE9hz+ogYtbELUctxCIOeiBjn1MYyDgAIA1wnqM dCg2ou64BzJwkQtd4OKQfuoqP+op1Xi8IxdnBcBhBSINAJBDANwAQDYGggx+qAMXyKCHPJAagGD/ mMMbuxDAL+YxDmiYViOJtUaDogEAdPSCF++4RzHe18g/CaAW2QBAObChjXTEwxkBuEVqxfEMaFDD HfqQxi1aKo0G+QIe85BEM/qBDmpaxBa2sIg09BENYdwjHscQCDj8cQ2NIGMe9TDGLtrBD3FkoxvU UOZG02OL6N7jnPdgB0E7CoB+5GMfFc6GAHpxDgBAQyO8aMc8itGMqlIzg7UgBz1+YYty7FW/io1f MdyRj2Pogh3+qMc86HGOYWCrwXvlBTCa4Q56ECO03J0GgDWqi3GIdiC7lQcwlrEPdbhXAMQYhgAk gV5sXKPD4lBkYvt7EWXcAx7E2IU78NEMYRAj/xi2QGWYNNuMgaB0GrWYaXkDEA4AYEMg2MCrQJhR XGvSOKcBGAY7yhGA+dbj0fPoBzt8jFJqDOSu48hFL95hD2CkUlNd7Yc1tumMdgCgGXkGgKUDwIwq BzYZ9ciHNHRhDMdWY1c3NUcxhtFncARgHP2ARjKSoYxz8KO8KNXGMJCBDQvXGRg0jsawlzFZXFE4 H/Kgqml1cYs+X0MguBgpdmsxjXvwIx3ycKt7GwsAeLijqsA4xrlvEb/SlqMW4fCHPuBRj3ZPg97B oIc/8mEPfORDHbncVC2YsQ1ueOMb24iGMmvhDG4oYyDM4EYzLFILZXDDqtDIbSmHcY1ylKMa61pE RjeY0dxfaMMauWhGw73BjWkYQyO7qMbMu/ENbGxVU8+9BXzhWxFb3KK52Q3sQG6xi2K+D1660IVG jG4yo9eiFmolekGGDt+jLxhCp+zipyMZSjmLUuybCggAOw== ------_=_NextPart_001_01C7879B.BFF8E3C0-- --===============0678873095== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0678873095==-- From ips-bounces@ietf.org Wed Apr 25 21:41:27 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgsz4-0004NP-D9; Wed, 25 Apr 2007 21:41:22 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgsz2-0004NA-Qv for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 21:41:20 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgsz2-0004Mh-43 for ips@ietf.org; Wed, 25 Apr 2007 21:41:20 -0400 Received: from mail-gw3.adaptec.com ([216.52.22.36]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgsyz-0003bz-M0 for ips@ietf.org; Wed, 25 Apr 2007 21:41:20 -0400 Received: from aime2k302.adaptec.com (aime2k302.adaptec.com [10.25.8.48]) by mail-gw3.adaptec.com (Spam Firewall) with ESMTP id 7485A19A3D5; Wed, 25 Apr 2007 18:41:16 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 18:41:13 -0700 Message-ID: <368FBF3D8437A748BA8222526BF9309901ACBF42@aime2k302.adaptec.com> In-reply-to: <39BA3BC178B4394DB184389E88A97F8C022C5C6C@hq-exch-1.corp.brocade.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHbhab+EXeFVfgQWiH7jzYWydYaAAKX7IQAAKlUiA= References: <006001c7876e$12cc26f0$05faa8c0@ivivity.com> <39BA3BC178B4394DB184389E88A97F8C022C5C6C@hq-exch-1.corp.brocade.com> From: "Sandars, Ken" To: "John Hufferd" , "Eddy Quicksall" , "Julian Satran" X-Spam-Score: 0.0 (/) X-Scan-Signature: 42e3ed3f10a1d8bef690f09da16f507a Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0320518248==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0320518248== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C787A3.FB06A317" This is a multi-part message in MIME format. ------_=_NextPart_001_01C787A3.FB06A317 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hey John, =20 [Hufferd] Many servers are asking for an evolutionary way to combine their Networking connections from the Server. The customers I have dealt with do NOT want to rip out FC, they want to provide a single Link for transport of all networking needs, including storage, exiting their servers. I'm not sure what you are saying here. Are you saying servers should have a single type of physical network connection, presumably ethernet? How does that align with not wanting to rip out FC? =20 Thanks Ken ------_=_NextPart_001_01C787A3.FB06A317 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hey John,
 
[Hufferd] = Many servers=20 are asking for an evolutionary way to combine their Networking = connections from=20 the Server.  The customers I have dealt with do NOT want to rip out = FC,=20 they want to provide a single Link for transport of all networking = needs,=20 including storage, exiting their servers.

I'm not sure what you are saying here. Are you = saying=20 servers should have a single type of physical network connection, = presumably=20 ethernet? How does that align with not wanting to rip out=20 FC?
 
Thanks
Ken
------_=_NextPart_001_01C787A3.FB06A317-- --===============0320518248== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0320518248==-- From ips-bounces@ietf.org Thu Apr 26 00:51:59 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgvxS-0006JW-Pn; Thu, 26 Apr 2007 00:51:54 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgvxR-0006Dy-En for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 00:51:53 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgvxR-0006Dq-4y for ips@ietf.org; Thu, 26 Apr 2007 00:51:53 -0400 Received: from mx20.brocade.com ([66.243.153.19]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgvxQ-0000yx-GX for ips@ietf.org; Thu, 26 Apr 2007 00:51:53 -0400 Received: from discus.brocade.com ([192.168.126.240]) by mx20.brocade.com with ESMTP; 25 Apr 2007 21:51:51 -0700 X-IronPort-AV: i="4.14,452,1170662400"; d="gif'147?scan'147,208,217,147"; a="8646152:sNHT40399919" Received: from HQ-EXCHFE-3.corp.brocade.com (unknown [192.168.126.212]) by discus.brocade.com (Postfix) with ESMTP id C469B23836B; Wed, 25 Apr 2007 21:50:40 -0700 (PDT) Received: from hq-exch-1.corp.brocade.com ([10.3.8.21]) by HQ-EXCHFE-3.corp.brocade.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 25 Apr 2007 21:51:52 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 21:51:50 -0700 Message-ID: <39BA3BC178B4394DB184389E88A97F8C022C5CC4@hq-exch-1.corp.brocade.com> In-Reply-To: <368FBF3D8437A748BA8222526BF9309901ACBF42@aime2k302.adaptec.com> X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHbhab+EXeFVfgQWiH7jzYWydYaAAKX7IQAAKlUiAABWKvIA== From: "John Hufferd" To: "Sandars, Ken" , "Eddy Quicksall" , "Julian Satran" X-OriginalArrivalTime: 26 Apr 2007 04:51:52.0652 (UTC) FILETIME=[9B8BE0C0:01C787BE] X-Spam-Score: 0.2 (/) X-Scan-Signature: abb8110dde048486ea2be9c769692569 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0503422998==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0503422998== Content-class: urn:content-classes:message Content-Type: multipart/related; boundary="----_=_NextPart_001_01C787BE.9A95F03B"; type="multipart/alternative" This is a multi-part message in MIME format. ------_=_NextPart_001_01C787BE.9A95F03B Content-Type: multipart/alternative; boundary="----_=_NextPart_002_01C787BE.9A95F03B" ------_=_NextPart_002_01C787BE.9A95F03B Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Ken, The term FCoE has as its primary component FC. Consider the possibility that the DCE Link from the Host connects to a switch/device that is able to deal with the FC part of the FCoE. =20 . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com =20 Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =20 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Wednesday, April 25, 2007 6:41 PM To: John Hufferd; Eddy Quicksall; Julian Satran Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =20 Hey John, =20 [Hufferd] Many servers are asking for an evolutionary way to combine their Networking connections from the Server. The customers I have dealt with do NOT want to rip out FC, they want to provide a single Link for transport of all networking needs, including storage, exiting their servers. I'm not sure what you are saying here. Are you saying servers should have a single type of physical network connection, presumably ethernet? How does that align with not wanting to rip out FC? =20 Thanks Ken ------_=_NextPart_002_01C787BE.9A95F03B Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable

Ken,

The term FCoE has as its primary = component FC.  Consider the possibility that the DCE Link from the Host = connects to a switch/device that is able to deal with the FC part of the = FCoE.

 

.

.

.

John L Hufferd

Sr. Executive Director of = Technology

jhufferd@brocade.com

Office Phone: (408) 333-5244; eFAX: = (408) 904-4688

Alt Office Phone: (408) 997-6136; = Cell: (408) 627-9606

From: = Sandars, Ken [mailto:ken_sandars@adaptec.com]
Sent: Wednesday, April = 25, 2007 6:41 PM
To: John Hufferd; Eddy = Quicksall; Julian Satran
Cc: ips@ietf.org
Subject: RE: FW: [Ips] = Recent comments about FCoE and iSCSI

 

Hey = John,

 

[Hufferd] Many servers are asking = for an evolutionary way to combine their Networking connections from the = Server.  The customers I have dealt with do NOT want to rip out FC, they want to = provide a single Link for transport of all networking needs, including storage, = exiting their servers.


I'm not sure what you are saying here. Are = you saying servers should have a single type of physical network connection, presumably ethernet? How does that align with not wanting to rip out = FC?

 

Thanks

Ken

------_=_NextPart_002_01C787BE.9A95F03B-- ------_=_NextPart_001_01C787BE.9A95F03B Content-Type: image/gif; name="image001.gif" Content-Transfer-Encoding: base64 Content-ID: Content-Description: image001.gif Content-Location: image001.gif R0lGODlhbgAtAPcAAAAAAP////7+/uwMEewPFO0VGvIzNvJFSfFESPNGSvRmafeHifaOj/yXmPaU lveeoPvExeoACOoJEOwKEu4NFewMFOwMFukME+oNFugNFesOFe8PF+wPFuwPGOsPGesPF+0QGusQ GOsQFukQGe0RF+oRGesSG+8THO0UHOsUG+8VHe4VHu0WHu4XH+0XIe0XH+4YIO4YIusYIO8aIewZ IusZH+waI+saIO8cI+4bJO4bIu0bJO0bIekbJPAdJu4cJe0cIuwdJesdJesdIu0eJuseJ/AgKewg Je4nLvAqMu8tNfAxOO8yOfEzOfAzOvE1O/E2PO85P/E9QvBARfRDS/JDSfVESvJFS/RHTfJHT/FH TfVKUvRLUPJKTvVMU/NMUvJOVfJRVfNUWfZYXvRcYvZscfVzd/d8gPqvsvzU1f3Z2v/l5v7k5f3k 5esHE+0KFe4MGOoLFu0NGO4OGesOHe0PGusTHu0UH/AaJe0ZJe8bJ+8cKu0bJu4cJ/AwO/FDTPNJ UfFJUf7q6/vn6P7t7v/v8P7o6v/i5f3p6/7m6f/q7f73+P7y9P/8/f/5/P/9/tfV2P7+/9/f4Pv/ /v3//tbY19TW1ff8+Pz//P7/+////f359v3y7v76+fzy8P3o5f7s6v3r6f/n5f708//k4v3p6Pvt 7P3g3/7p6Pvm5f+gnvulpfq8vPvOzv/i4v7m5vvj4//x8f/y8v3x8f/19f39/fr6+vn5+ff39/X1 9fPz8/Dw8O3t7erq6ujo6Obm5uXl5ePj4+Hh4d3d3dra2tfX19XV1dLS0s3NzcnJycXFxb+/v7m5 uba2trKysq2trampqaWlpaKiopycnJSUlI6OjouLi4iIiIODg319fXp6enV1dXJycnBwcG1tbWtr a2lpaWZmZmFhYVtbW1dXV1FRUUpKSkRERD8/Pzc3NzMzMzAwMCsrKykpKSYmJiQkJCIiIiAgIB4e Hh0dHRkZGRUVFRISEhEREQ8PDwsLCwkJCQYGBgUFBQMDAwEBAf///yH5BAEAAP8ALAAAAABuAC0A AAj/AAMIHEiwoMGDCBMqTKhJIKVIAxsKzDRQQAABmiwGkHiw4aQAjzZS3Bhg5MKTKFOi1CRx06FE pQQZmmmolCFUiRAlMoTI0E6aQA2ZohUAU8lNsWg16mRSpdOnKUMOJKSgSZU/CBBYqXIFwZ8EBxCE rULlgNmzZhF8KfPgUEg1qlahYdUJqt27CDOxHMgqSYQPHUB8ANEhBAcNIUBs4CCisePHdeRImIOl FSVSaBqUWWXrI97PoCeFfARByhseQn70AXLEzhsLcWDDuWChtu3aE0D0keEmippOn06lIdUUtHGo mSYZbSVGwo3VQPZ0GROIi/UuXqxr195FCYEhOtyE/wl1MQDE4+hTSqSo6RHFRGXevBiyAwYYVoVe pVrDps2r/wD+l0oaCtwRRAsnPKAXJQJplN6DHZE0kUCazLIADSPsMAQPTzywCEkcHQRKFyYEgYEZ ID7iGYQspuQIGlF8EAMQJ8QQCBqNLNSQAy8AoQEZo7QopHpGZSLAIWW0YAIPMJAAAxkQLHJeAAxG pMkDMgzRQRiKUBjikGASJFEkjTxggAko3NBDCHgoAIEodQkU0keOlHHCEAMo0JBGDobpJ0kMCqDG GUqUAMIOMliwwhRnoKHGILSERAgDRsQAgwYL/KkpQVWGtJcmEJiBxAQWyHDDBxPA8EQYZSzAABkz oP8wRAZ+pEHlpptiFEAtmoi2EahmTIEEHkDggIIFbwwwwAQYuIDDC0is8ogAjUj0Ja7pNcRRJOdp 1AgEYsCwgw1EHHGEDXvYgEMROLhAhkUYWYtti1Pu9YhRAqWxQCBIAJFaDCh8YAEGFFgQQxEvKLGK nPP6ie9eArVCKAh1sJDDqTM4AQgZZZQRhg558CFHFLb22TC9DD9yyAJJoMBCDDScIAMUZ7Byiie0 0LLJKA0McUMMIyxg8skPyhvAJmhIUQcNOrhgBw5grELIRhZJBZICKvCAQRmaMAgx0ehxJKgZOpig Rw1z+CAGGp1EwlGVIz1QQw8diFEI2C1+pAkmrDz/IUELQ6DZBRoZlRRhAA2wgMcHYkyNN3oCWMQS Iw/4MMEPQ5yghAOEbEuQXhQSEsYJP0zAtZePfxY5RvAJRsQNPISxSiGCnMIGLIe04Qoph5DiShtt sAGBAjrYwEIKDzSofOp2rX7KGBOsIAQQMvgw3RddAJLFFlpoAQggXWQRSBZcAKIEBy4I8QYgUw9d HkGRL+Q+mBzhm0YXdMBAXw85DLHCCB4gTAc84IEOGPCABhwBCfKQhwooAQLtMcj8huYgk81vee87 yAULoolGUKIVUYiDC2TgghbA4AUjlMELTrhCGMTghTB84QtmUAARaAGChYOf6pTnvg1usCJWak8n /85gBCU0gQlNUIIBluCHJTjxiVCM4hPD8IBEhCR+EsQgSuClwQe5rxOc8IQnYjGKQoxCFmhMYyzS yMY2eoIRneia4RJiQSD2iYt2fMoF+cRB8xDkESG6Fkrwha+CYBGIWqyFIZenSERisBZ47OHyIjcS TLjHI4gUZEKMwhJPFYeHdMxjBit4kTpSMIOj3JVKNOkllrgSYiOZkkF6cQxjFMMYxKiIL45RjFsS IxcYJAYzlIELDPZiGcvoBSKFcYxdKM8XtjSGMXhBEFsMwxKVgEQxjiGJS8gSL19LiC2owY53zGMe 7QjHLwRgi2uwwx3ykMc7zgGN8ugCG+7ghz3Kkf8MgjwDHfnIBzqW8T5hnIMd2yAINt55znRcw5kB 8IU52AGPeMRjHuUIxg8fZItt+AMd4AjHOgAQjlrg4hsAWEc4xFEOANCDGQHYRTcAoI5siCOlyBDI M+yxj3B8Ax/v6KdApgEAAMQjGAO56TrAIY53AGAcvggAMOYBAHOMgxzl0IYyN2ULbfAjGrfAxTDi gQ9e2GKm1LjFLXRxDQCIIwDP6Mc6jiGAXGyDpDE9hz+ogYtbELUctxCIOeiBjn1MYyDgAIA1wnqM dCg2ou64BzJwkQtd4OKQfuoqP+op1Xi8IxdnBcBhBSINAJBDANwAQDYGggx+qAMXyKCHPJAagGD/ mMMbuxDAL+YxDmiYViOJtUaDogEAdPSCF++4RzHe18g/CaAW2QBAObChjXTEwxkBuEVqxfEMaFDD HfqQxi1aKo0G+QIe85BEM/qBDmpaxBa2sIg09BENYdwjHscQCDj8cQ2NIGMe9TDGLtrBD3FkoxvU UOZG02OL6N7jnPdgB0E7CoB+5GMfFc6GAHpxDgBAQyO8aMc8itGMqlIzg7UgBz1+YYty7FW/io1f MdyRj2Pogh3+qMc86HGOYWCrwXvlBTCa4Q56ECO03J0GgDWqi3GIdiC7lQcwlrEPdbhXAMQYhgAk gV5sXKPD4lBkYvt7EWXcAx7E2IU78NEMYRAj/xi2QGWYNNuMgaB0GrWYaXkDEA4AYEMg2MCrQJhR XGvSOKcBGAY7yhGA+dbj0fPoBzt8jFJqDOSu48hFL95hD2CkUlNd7Yc1tumMdgCgGXkGgKUDwIwq BzYZ9ciHNHRhDMdWY1c3NUcxhtFncARgHP2ARjKSoYxz8KO8KNXGMJCBDQvXGRg0jsawlzFZXFE4 H/Kgqml1cYs+X0MguBgpdmsxjXvwIx3ycKt7GwsAeLijqsA4xrlvEb/SlqMW4fCHPuBRj3ZPg97B oIc/8mEPfORDHbncVC2YsQ1ueOMb24iGMmvhDG4oYyDM4EYzLFILZXDDqtDIbSmHcY1ylKMa61pE RjeY0dxfaMMauWhGw73BjWkYQyO7qMbMu/ENbGxVU8+9BXzhWxFb3KK52Q3sQG6xi2K+D1660IVG jG4yo9eiFmolekGGDt+jLxhCp+zipyMZSjmLUuybCggAOw== ------_=_NextPart_001_01C787BE.9A95F03B-- --===============0503422998== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0503422998==-- From ips-bounces@ietf.org Thu Apr 26 09:12:29 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh3lq-0008AQ-Sx; Thu, 26 Apr 2007 09:12:26 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgmxd-0003vX-Hw for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgmxd-0003vO-7v for ips@ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from imf23aec.mail.bellsouth.net ([205.152.59.71]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgmxa-0004Q9-Hi for ips@ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from ibm66aec.bellsouth.net ([74.245.52.54]) by imf23aec.mail.bellsouth.net with ESMTP id <20070425191525.GPIJ9888.imf23aec.mail.bellsouth.net@ibm66aec.bellsouth.net> for ; Wed, 25 Apr 2007 15:15:25 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm66aec.bellsouth.net with SMTP id <20070425191524.MIQJ18529.ibm66aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 15:15:24 -0400 Message-ID: <006001c7876e$12cc26f0$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Eddy Quicksall" , "Julian Satran" References: <004801c7876b$4e23fbe0$05faa8c0@ivivity.com> Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 15:15:22 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.6 (/) X-Scan-Signature: 66c1f4b8ef7d9911c5b16e6767030a3e X-Mailman-Approved-At: Thu, 26 Apr 2007 09:12:25 -0400 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0046708084==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0046708084== Content-Type: multipart/alternative; boundary="----=_NextPart_000_005D_01C7874C.8B7085A0" This is a multi-part message in MIME format. ------=_NextPart_000_005D_01C7874C.8B7085A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Further, I think PAUSE is only for point-to-point and I think that is = too restrictive. Eddy ----- Original Message -----=20 From: Eddy Quicksall=20 To: Julian Satran=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 2:55 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI If it was not obvious in my wording, I meant to agree with you on your = observations. Eddy ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Sorry Eddy by sizable I meant even at the size of a modern data = center. Julo=20 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL =20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and = iSCSI=20 =20 =20 I basically said that in the summery line by saying "it will not = route on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20From ips-bounces@ietf.org Thu Apr 26 09:12:29 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh3lq-0008AQ-Sx; Thu, 26 Apr 2007 09:12:26 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgmxd-0003vX-Hw for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgmxd-0003vO-7v for ips@ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from imf23aec.mail.bellsouth.net ([205.152.59.71]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hgmxa-0004Q9-Hi for ips@ietf.org; Wed, 25 Apr 2007 15:15:29 -0400 Received: from ibm66aec.bellsouth.net ([74.245.52.54]) by imf23aec.mail.bellsouth.net with ESMTP id <20070425191525.GPIJ9888.imf23aec.mail.bellsouth.net@ibm66aec.bellsouth.net> for ; Wed, 25 Apr 2007 15:15:25 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm66aec.bellsouth.net with SMTP id <20070425191524.MIQJ18529.ibm66aec.bellsouth.net@IVVTDKV0981>; Wed, 25 Apr 2007 15:15:24 -0400 Message-ID: <006001c7876e$12cc26f0$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Eddy Quicksall" , "Julian Satran" References: <004801c7876b$4e23fbe0$05faa8c0@ivivity.com> Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 15:15:22 -0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.6 (/) X-Scan-Signature: 66c1f4b8ef7d9911c5b16e6767030a3e X-Mailman-Approved-At: Thu, 26 Apr 2007 09:12:25 -0400 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0046708084==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0046708084== Content-Type: multipart/alternative; boundary="----=_NextPart_000_005D_01C7874C.8B7085A0" This is a multi-part message in MIME format. ------=_NextPart_000_005D_01C7874C.8B7085A0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Further, I think PAUSE is only for point-to-point and I think that is = too restrictive. Eddy ----- Original Message -----=20 From: Eddy Quicksall=20 To: Julian Satran=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 2:55 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI If it was not obvious in my wording, I meant to agree with you on your = observations. Eddy ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Sorry Eddy by sizable I meant even at the size of a modern data = center. Julo=20 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL =20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and = iSCSI=20 =20 =20 I basically said that in the summery line by saying "it will not = route on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran=20 To: Eddy Quicksall=20 Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 Eddy,=20 That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it = on a sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 =20 To "John Hufferd" , Julian = Satran/Haifa/IBM@IBMIL =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and = iSCSI=20 =20 =20 Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like = TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI = as an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre = Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the = network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that = is not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if = you are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC = Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers = in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned = before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection = type to > handle cluster messaging, gener Cc: ips@ietf.org ; John Hufferd=20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 Eddy,=20 That is oversimplified and ignore the drop rate assumption and error = rate assumptions made in FCP(FCP has no transport layer). To get to it = on a sizable network requires more than PAUSE.=20 Julo=20 "Eddy Quicksall" =20 25/04/07 10:07=20 =20 To "John Hufferd" , Julian = Satran/Haifa/IBM@IBMIL =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and = iSCSI=20 =20 =20 Basically, it is sending FC frames over Ethernet. This localizes the = traffic unless you route based on MAC addresses. So you send 2146 bytes = of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" = packet. 18 bytes of Ethernet gets stripped and you have straight FC = frame that can go through any FC network. Now you can have 10G Ethernet = pipes into existing FC SANs. Limited market potential as far as I can = see. The key argument is it much easier to implement than iSCSI and also = has less overhead and uses all the benefits of FC. End to End credits = are simulated using PAUSE command on Ethernet and MAC addresses are = mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like = TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran=20 To: John Hufferd=20 Cc: ips@ietf.org=20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 "John Hufferd" wrote on 25/04/2007 02:45:51: > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI = as an > outreach protocol from the Data Center. We also believe iSCSI is = very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our = new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that = iSCSI > is very important to connect "stranded" servers to the Fibre = Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will = permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also = applies > to Desktops and Laptop systems, and systems at distance. >=20 You make it sound like:=20 1. most of the servers in the world have their storage on the = network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that = is not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if = you are completely relying on FCP storage (and there are plenty of good = iSCSI vendors of storage)and pushing the price on the servers is not = cheap either - at least not for the server buyer=20 > Now with that positioning, it is important to understand the = limitation > to this strategy. The primary problem is that iSCSI to FC = Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on = the > market, it is not cost feasible to replace all the server = connectivity > to FC storage with iSCSI for the hundred to thousands of servers = in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That = means > there must be a bridge/Gateway approach. And as I mentioned = before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection = type to > handle cluster messaging, general messaging, and storage. iSCSI = is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be = deployed in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, = with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The = rest of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all = FC. And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only = world.=20 First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known = large scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the = proven end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking = applications are built above layer-3 and not dropping layer-3 (like = FCoE) does.=20 Although I can understand the DCE arguments as a management = statement I would prefer like any rational engineer, to base my building = blocks on structures that are proven and long lasting. And those are = still the end-to-end TCP/IP that can accommodate even your FCP addicts. = The IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, = and all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, = there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at = least a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short al messaging, and storage. iSCSI = is > clearly an option for the storage, however, the gateway costs are = too > high for iSCSI to be used as the "normal" server connect into a FC = based > Fabric. That is true for the current 1GE; and for 10GE the cost is = just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to = understand > is that it is not Ethernet as we have known it up to know. The = Ethernet > we are talking about is a type of Ethernet that can only be = deployed in > a constrained environment such as a Data Center. This form of = Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, = with > multi-priority and Flow Control. This is NOT an Internet or = Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The = rest of > the Host and storage stack remain the same, the functions and = features > of the switches also remain the same and add the capability to = provide > Cluster Message Switching which has latency close to InfiniBand = speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a = DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or = send FC > frames to DCE FCoE devices, just like one would do if it was all = FC. And > all this is done while performing Cluster message switching and = general > message trucking to the IP outfacing network.=20 >=20 The rosy future of the yet to appear DCE/CEE and a layer 2 only = world.=20 First you have some terms confused:=20 Bridging is the term commonly used for Layer-2 switching and routing = is therm used for layer-3 (switching).=20 Bridging has some advantages (less management) that have created a = movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant = equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless = networks etc.=20 The second trouble with your argument is that there are no known = large scale networking technologies that=20 really work at full speed (high speed) and are lossless = (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the = proven end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking = applications are built above layer-3 and not dropping layer-3 (like = FCoE) does.=20 Although I can understand the DCE arguments as a management = statement I would prefer like any rational engineer, to base my building = blocks on structures that are proven and long lasting. And those are = still the end-to-end TCP/IP that can accommodate even your FCP addicts. = The IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a better base.=20 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on = the > Data Center Fabric. And it maintains all the FC Fabric Services, = and all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, = there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at = least a > decade or more. >=20 It is server cost statement. It costs nothing to connect a modern = server to ethernet it will cost a bundle to connect to FCoE and it will = force users in short lived bad solutions.=20 > We see value in offering switches and Directors that can support = DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data = Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you = on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will = follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded = iSCSI that > was not so - FCoE was considered even then and was dropped as a = dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main = arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this lived bad solutions.=20 > We see value in offering switches and Directors that can support = DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That = said; > we do not see FCoE going beyond the constraints of the Data = Center.=20 Data Centers now grow to tens of thousands of nodes. There is no = layer-2 technology for errorless/lossless operation at this scale and = there is no good reason to pursue one. The only possible reason (good = reason) is the bridging infrastructure but that infrastructure has a = completely different rationale than the flowcontrol.=20 > This issue and message is quite different from the issues and = messages > we struggled with when we started iSCSI. There is a consortium of = folks > both working on the DCE (CEE) and the FCoE. Without the DCE the = FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 iSCSI is good for all environments. Business consideration (and some = politics) keep it form "exploding" and large storage vendors are = completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE = (that still has a way to go) to improve the QoS in the data-center (and = for storage too). You expect it to bring the loss rates down to the = levels that FCP assumes (FCP has no transport layer) and that is = probably a pipe dream. Todays transport solution for loss mitigation are = far more cost effective - and that's why iFCP is a better proposition as = a transition technology than FCoE and iSCSI with gateways is propably = better in the long run.=20 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite = different > then the ones we had in pre iSCSI days. (By the way I missed you = on > today's Renato meeting/conf call where Brocade took the IBM = technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you = probably > were sent the Brocade charts. Please review them and I will = follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter = envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that = preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded = iSCSI that > was not so - FCoE was considered even then and was dropped as a = dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main = arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group = to > seriously expand my arguments and make them public - in personal = or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all = must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) = started > looking at connecting storage to servers using the "regular = network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the = time). I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The = packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a = transport > layer in the protocol stack has always been the preferred choice = of the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control = mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any = "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use = bridging and > are limited by the logical tree structure that bridges must = follow. The > effort to combine routers and bridges (rbridmaterial in a nay form. I expect this group = to > seriously expand my arguments and make them public - in personal = or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all = must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > = ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) = started > looking at connecting storage to servers using the "regular = network" > (the ubiquitous LAN) we considered many alternatives (another team = even > had a look at ATM - still a computer network candidate at the = time). I > won't get you over all of our rationale (and we went over some of = them > again at the end of 1999 with a team from CISCO before we convened = the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all = the > rest) but some of the reasons we choose to drop Fiber Channel over = raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload)=20 > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be = very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The = packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network = diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling = argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a = transport > layer in the protocol stack has always been the preferred choice = of the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol = stack > implementation and finer tuning of the congestion control = mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the = computing > scene have enough compute cycles available to make any = "offloading" > possible as a mere code restructuring exercise (see the stack = reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service = location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same = connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use = bridging and > are limited by the logical tree structure that bridges must = follow. The > effort to combine routers and bridges (rbridges) is promising to = change > that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed = 2 > protocols iFCP (to support hosts with FCP drivers and IP = connections to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands = through TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have = given us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits = scaling, > mandates elements at the link layer and application layer that = make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., = NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= --- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 -------------------------------------------------------------------------= --- _______________________________________________ Ips mailing list Ips@ietf.org = https://www1.ietf.org/mailman/listinfo/ips_______________________________= ________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_005D_01C7874C.8B7085A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Further, I think PAUSE is only=20 for point-to-point and I think that is = too restrictive.
 
Eddy
----- Original Message -----
From:=20 that but it will take some time to finalize (and we don't know = exactly > how it will operate). Untill then the scale of Layer 2 network is = going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to = perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). = That > was what convinced us to take the path that lead to iSCSI - and we = used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for = service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed = 2 > protocols iFCP (to support hosts with FCP drivers and IP = connections to > connect to storage by a simple conversion from FCP to TCP packets) = FCPIP > to extend the reach of FCP through IP (connects FCP islands = through TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have = given us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits = scaling, > mandates elements at the link layer and application layer that = make > applications more expensive and leaves aside the whole "ecosystem" = that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., = NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the = time.=20 > =20 -------------------------------------------------------------------------= --- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 -------------------------------------------------------------------------= --- _______________________________________________ Ips mailing list Ips@ietf.org = https://www1.ietf.org/mailman/listinfo/ips_______________________________= ________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips -------------------------------------------------------------------------= ----- _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------=_NextPart_000_005D_01C7874C.8B7085A0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Further, I think PAUSE is only=20 for point-to-point and I think that is = too restrictive.
 
Eddy
----- Original Message -----
From:=20 Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 2:55=20 PM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

If it was not obvious in my wording, I meant to = agree with=20 you on your observations.
 
Eddy
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, = 2007 11:52=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Sorry Eddy by = sizable I=20 meant even at the size of a modern data center. Julo =


"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL =
cc
ips@ietf.org, John = Hufferd <jhufferd@Brocade.COM> =
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and = iSCSI

=




<= FONT=20 size=3D2>I basically said that in the summery line by saying = "it will not route on the =93global=94 scale = like TCP/IP=20 would".
----- Original Message -----=20
From: Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 2:55=20 PM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

If it was not obvious in my wording, I meant to = agree with=20 you on your observations.
 
Eddy
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, = 2007 11:52=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Sorry Eddy by = sizable I=20 meant even at the size of a modern data center. Julo =


"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL =
cc
ips@ietf.org, John = Hufferd <jhufferd@Brocade.COM> =
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and = iSCSI

=




<= FONT=20 size=3D2>I basically said that in the summery line by saying = "it will not route on the =93global=94 scale = like TCP/IP=20 would".
----- Original Message -----=20
From: Julian=20 Satran
To:=20 Eddy Quicksall =
Cc: ips@ietf.org ; John=20 Hufferd
Sent:=20 Wednesday, April 25, 2007 11:04 AM
Subject:=20 Re: FW: [Ips] Recent comments about FCoE and iSCSI =


Eddy, =

That is oversimplified and ignore the = drop rate=20 assumption and error rate assumptions made in FCP(FCP has no = transport=20 layer). To get to it on a sizable network requires more than=20 PAUSE.

Julo

"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.net>

25/04/07 10:07=20


To
"John = Hufferd"=20 <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and = iSCSI






Basically,=20 it is sending FC frames over Ethernet. This localizes the traffic = unless you=20 route based on MAC addresses. So you send 2146 bytes of FC frame = plus 18=20 bytes of Ethernet overhead as FCoE =93standard=94 packet. 18 bytes = of Ethernet=20 gets stripped and you have straight FC frame that can go through any = FC=20 network. Now you can have 10G Ethernet pipes into existing FC SANs. = Limited=20 market potentiatran@il.ibm.com">Julian=20 Satran
To:=20 Eddy Quicksall =
Cc: ips@ietf.org ; John=20 Hufferd
Sent:=20 Wednesday, April 25, 2007 11:04 AM
Subject:=20 Re: FW: [Ips] Recent comments about FCoE and iSCSI =


Eddy, =

That is oversimplified and ignore the = drop rate=20 assumption and error rate assumptions made in FCP(FCP has no = transport=20 layer). To get to it on a sizable network requires more than=20 PAUSE.

Julo

"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.net>

25/04/07 10:07=20


To
"John = Hufferd"=20 <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and = iSCSI






Basically,=20 it is sending FC frames over Ethernet. This localizes the traffic = unless you=20 route based on MAC addresses. So you send 2146 bytes of FC frame = plus 18=20 bytes of Ethernet overhead as FCoE =93standard=94 packet. 18 bytes = of Ethernet=20 gets stripped and you have straight FC frame that can go through any = FC=20 network. Now you can have 10G Ethernet pipes into existing FC SANs. = Limited=20 market potential as far as I can see. The key argument is it much = easier to=20 implement than iSCSI and also has less overhead and uses all the = benefits of=20 FC. End to End credits are simulated using PAUSE command on Ethernet = and MAC=20 addresses are mapped into WWNs.

Biggest knock is that = it will not=20 route on the =93global=94 scale like TCP/IP would.

Eddy

 -----=20 Original Message -----

From: Julian=20 Satran
To: John=20 Hufferd
Cc: ips@ietf.org =
Sent:=20 Wednesday, April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] = Recent=20 comments about FCoE and iSCSI



"John=20 Hufferd" <jhufferd@Brocade.COM> wrote on 25/04/2007 02:45:51:

> Julian, =
> To be=20 sure you understand our position; Brocade is pushing iSCSI as = an
>=20 outreach protocol from the Data Center. We also believe iSCSI is=20 very
> useful for installations that do not have a Fibre=20 Channel
> infrastructure, and in that case we will be able to = sell=20 them our new
> iSCSI and TOE offload HBAs.  
> =
>=20 When I say iSCSI is an outreach protocol, this is a statement that=20 iSCSI
> is very important to connect "stranded" servers to the = Fibre=20 Channel
> Fabric.  That is, we sell iSCSI-to-FC Gateway = devices=20 which will permit
> iSCSI Servers (software or HBA iSCSI = initiators)=20 to connect to the
> Enterprise "Bet Your Business" FC Storage. =  This of course also applies
> to Desktops and Laptop = systems,=20 and systems at distance.
> =

You make it sound like: =
1.       =  most of the servers in the world have their storage on the = network -=20 and that is not the case
2.       =  FCP is basically better performing than iSCSI - and that is = not true=20 either
3.=20        Gatewaying is = expensive -=20 and it is perhaps so but only if you are completely relying on FCP = storage=20 (and there are plenty of good iSCSI vendors of storage)and pushing = the price=20 on the servers is not cheap either - at least not for the server=20 buyer

> Now with=20 that positioning, it is important to understand the = limitation
> to=20 this strategy.  The primary problem is that iSCSI to FC=20 Bridging
> (Gatewaying) is relatively expensive (compared to = simple=20 FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace = all the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers l as far as I can see. The key argument is it much = easier to=20 implement than iSCSI and also has less overhead and uses all the = benefits of=20 FC. End to End credits are simulated using PAUSE command on Ethernet = and MAC=20 addresses are mapped into WWNs.

Biggest knock is that = it will not=20 route on the =93global=94 scale like TCP/IP would.

Eddy

 -----=20 Original Message -----

From: Julian=20 Satran
To: John=20 Hufferd
Cc: ips@ietf.org =
Sent:=20 Wednesday, April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] = Recent=20 comments about FCoE and iSCSI



"John=20 Hufferd" <jhufferd@Brocade.COM> wrote on 25/04/2007 02:45:51:

> Julian, =
> To be=20 sure you understand our position; Brocade is pushing iSCSI as = an
>=20 outreach protocol from the Data Center. We also believe iSCSI is=20 very
> useful for installations that do not have a Fibre=20 Channel
> infrastructure, and in that case we will be able to = sell=20 them our new
> iSCSI and TOE offload HBAs.  
> =
>=20 When I say iSCSI is an outreach protocol, this is a statement that=20 iSCSI
> is very important to connect "stranded" servers to the = Fibre=20 Channel
> Fabric.  That is, we sell iSCSI-to-FC Gateway = devices=20 which will permit
> iSCSI Servers (software or HBA iSCSI = initiators)=20 to connect to the
> Enterprise "Bet Your Business" FC Storage. =  This of course also applies
> to Desktops and Laptop = systems,=20 and systems at distance.
> =

You make it sound like: =
1.       =  most of the servers in the world have their storage on the = network -=20 and that is not the case
2.       =  FCP is basically better performing than iSCSI - and that is = not true=20 either
3.=20        Gatewaying is = expensive -=20 and it is perhaps so but only if you are completely relying on FCP = storage=20 (and there are plenty of good iSCSI vendors of storage)and pushing = the price=20 on the servers is not cheap either - at least not for the server=20 buyer

> Now with=20 that positioning, it is important to understand the = limitation
> to=20 this strategy.  The primary problem is that iSCSI to FC=20 Bridging
> (Gatewaying) is relatively expensive (compared to = simple=20 FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace = all the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers in the
> Data Center.  And since, if = there=20 is to be a consolidated Network
> connection to the servers in = the=20 Data Center, there must be an
> evolutionary replacement of = Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are=20 too
> high for iSCSI to be used as the "normal" server connect = into a=20 FC based
> Fabric. That is true for the current 1GE; and for = 10GE the=20 cost is just
> out of sight.  The reason for this is the=20 requirement for TCP/IP
> termination and re-initiation with FC = at the=20 Gateway.
>
> Now with respect to FC over Ethernet the = important=20 thing to understand
> is that it is not Ethernet as we have = known it=20 up to know.  The Ethernet
> we are talking about is a = type of=20 Ethernet that can only be deployed in
> a constrained = environment such=20 as a Data Center.  This form of Ethernet
> is called DCE = (Data=20 Center Ethernet) or CEE (Convergence Enhanced
> Ethernet). =  This=20 form of Ethernet is a Loss-less type Ethernet, with
> = multi-priority=20 and Flow Control.  This is NOT an Internet or Intranet
> = type of=20 Ethernet.
>
> FCoE is all about using the DCE (CEE) to = carry FC=20 frames.  The rest of
> the Host and storage stack remain = the=20 same, the functions and features
> of the switches also remain = the=20 same and add the capability to provide
> Cluster Message = Switching=20 which has latency close to InfiniBand speeds.
>
> =
>=20 Because the FC frames are transported to the switches intact via a=20 DCE
> frame, the Bridging, if you want to call it that, is = virtually=20 non
> existent.  Hence you can deliver the FC frames to = FC=20 devices, or send FC
> frames to DCE FCoE devices, just like = one would=20 do if it was all FC. And
> all this is done while performing = Cluster=20 message switching and general
> message trucking to the IP = outfacing=20 network.
>

The rosy future of the yet to appear = DCE/CEE and a=20 layer 2 only world.
First you have some terms confused:=20

Bridging is the term commonly used = for=20 Layer-2 switching and routing is therm used for layer-3=20 (switching).

Bridging has some advantages (less management) that = have=20 created a movement towards an enterprise wide=20
LAN. But this has a long way to go and = will=20 require significant equipment and protocol changes.=20
Even its proponents do not call for=20 transportless networks, lossless networks etc.=20
The second trouble with your argument = is that=20 there are no known large scale networking technologies = that
really work at full speed = (high speed)=20 and are lossless (flow-controlled) and errorless like FCoE=20 assumes.
The TCP/IP has=20 solved this issue for every generation using the proven end-to-end = principle=20 (and is doing so now).> Data Center.  And since, if = there=20 is to be a consolidated Network
> connection to the servers in = the=20 Data Center, there must be an
> evolutionary replacement of = Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are=20 too
> high for iSCSI to be used as the "normal" server connect = into a=20 FC based
> Fabric. That is true for the current 1GE; and for = 10GE the=20 cost is just
> out of sight.  The reason for this is the=20 requirement for TCP/IP
> termination and re-initiation with FC = at the=20 Gateway.
>
> Now with respect to FC over Ethernet the = important=20 thing to understand
> is that it is not Ethernet as we have = known it=20 up to know.  The Ethernet
> we are talking about is a = type of=20 Ethernet that can only be deployed in
> a constrained = environment such=20 as a Data Center.  This form of Ethernet
> is called DCE = (Data=20 Center Ethernet) or CEE (Convergence Enhanced
> Ethernet). =  This=20 form of Ethernet is a Loss-less type Ethernet, with
> = multi-priority=20 and Flow Control.  This is NOT an Internet or Intranet
> = type of=20 Ethernet.
>
> FCoE is all about using the DCE (CEE) to = carry FC=20 frames.  The rest of
> the Host and storage stack remain = the=20 same, the functions and features
> of the switches also remain = the=20 same and add the capability to provide
> Cluster Message = Switching=20 which has latency close to InfiniBand speeds.
>
> =
>=20 Because the FC frames are transported to the switches intact via a=20 DCE
> frame, the Bridging, if you want to call it that, is = virtually=20 non
> existent.  Hence you can deliver the FC frames to = FC=20 devices, or send FC
> frames to DCE FCoE devices, just like = one would=20 do if it was all FC. And
> all this is done while performing = Cluster=20 message switching and general
> message trucking to the IP = outfacing=20 network.
>

The rosy future of the yet to appear = DCE/CEE and a=20 layer 2 only world.
First you have some terms confused:=20

Bridging is the term commonly used = for=20 Layer-2 switching and routing is therm used for layer-3=20 (switching).

Bridging has some advantages (less management) that = have=20 created a movement towards an enterprise wide=20
LAN. But this has a long way to go and = will=20 require significant equipment and protocol changes.=20
Even its proponents do not call for=20 transportless networks, lossless networks etc.=20
The second trouble with your argument = is that=20 there are no known large scale networking technologies = that
really work at full speed = (high speed)=20 and are lossless (flow-controlled) and errorless like FCoE=20 assumes.
The TCP/IP has=20 solved this issue for every generation using the proven end-to-end = principle=20 (and is doing so now).
And it is not by chance so and that is why all = networking=20 applications are built above layer-3 and not dropping layer-3 (like = FCoE)=20 does.

Although I=20 can understand the DCE arguments as a management statement I would = prefer=20 like any rational engineer, to base my building blocks on structures = that=20 are proven and long lasting. And those are still the end-to-end = TCP/IP that=20 can accommodate even your FCP addicts. The IPS TWG has developed the = iFCP=20 that does exactly what FCoE claims to do an a better = base.

> This means an = evolutionary=20 process is possible to the solution of
> getting a single = Fabric=20 connection for all networks connected to a
> server, further, = the=20 process has very low interconnection cost on the
> Data Center = Fabric.=20 And it maintains all the FC Fabric Services, and all
> the = same=20 Storage Management processes.
>
> By the way, this is=20 primarily a Server driven value statement, there
> seems to be = little=20 value in having FCoE on the storage controller.
> Therefore FC = storage=20 controllers (and FICON) will be the very last
> things that = connect=20 using FCoE and that evolution will take at least a
> decade or = more.
>

It is server cost statement. It costs nothing = to=20 connect a modern server to ethernet it will cost a bundle to connect = to FCoE=20 and it will force users in short lived bad = solutions.


> We see value in = offering=20 switches and Directors that can support DCE
> switching, FC = switching=20 as well as iSCSI interconnect, and the
> "Trunking" of general = messaging to the Outfacing IP network.  That said;
> we = do not=20 see FCoE going beyond the constraints of the Data = Center.

Data Centers now grow to = tens of=20 thousands of nodes. There is no layer-2 technology for = errorless/lossless=20 operation at this scale and there is no good reason to pursue one. = The only=20 possible reason (good reason) is the bridging infrastructure but = that=20 infrastructure has a completely different rationale than the=20 flowcontrol.

>=20 This issue and message is quite different from the issues and=20 messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen. =  
>=20
> None of the above cancels out the value of iSCSI in=20 numerous
> environments.
>
>

iSCSI is = good for=20 all environments. Business consideration (and some politics) keep it = form=20 "exploding" and large storage vendors are completely indifferent to = the=20 network connection they are using.
You and I have also slightly=20 different views of DCE. I expect DCE (that still has a way to go) to = improve=20 the QoS in the data-center (and for storage too). You expect it to = bring the=20 loss rates down to the levels that FCP assumes (FCP has no transport = layer)=20 and that is probably a pipe dream. Todays transport solution for = loss=20 mitigation are far more cost effective - and that's why iFCP is a = better=20 proposition as a transition technology than FCoE and iSCSI with = gateways is=20 propably better in the long run. =

>
> .
> .
> .
> John L=20 Hufferd
> Sr. Executive Director of Technology
NT size=3D3>
And it is not by chance so and that is why all = networking=20 applications are built above layer-3 and not dropping layer-3 (like = FCoE)=20 does.

Although I=20 can understand the DCE arguments as a management statement I would = prefer=20 like any rational engineer, to base my building blocks on structures = that=20 are proven and long lasting. And those are still the end-to-end = TCP/IP that=20 can accommodate even your FCP addicts. The IPS TWG has developed the = iFCP=20 that does exactly what FCoE claims to do an a better = base.

> This means an = evolutionary=20 process is possible to the solution of
> getting a single = Fabric=20 connection for all networks connected to a
> server, further, = the=20 process has very low interconnection cost on the
> Data Center = Fabric.=20 And it maintains all the FC Fabric Services, and all
> the = same=20 Storage Management processes.
>
> By the way, this is=20 primarily a Server driven value statement, there
> seems to be = little=20 value in having FCoE on the storage controller.
> Therefore FC = storage=20 controllers (and FICON) will be the very last
> things that = connect=20 using FCoE and that evolution will take at least a
> decade or = more.
>

It is server cost statement. It costs nothing = to=20 connect a modern server to ethernet it will cost a bundle to connect = to FCoE=20 and it will force users in short lived bad = solutions.


> We see value in = offering=20 switches and Directors that can support DCE
> switching, FC = switching=20 as well as iSCSI interconnect, and the
> "Trunking" of general = messaging to the Outfacing IP network.  That said;
> we = do not=20 see FCoE going beyond the constraints of the Data = Center.

Data Centers now grow to = tens of=20 thousands of nodes. There is no layer-2 technology for = errorless/lossless=20 operation at this scale and there is no good reason to pursue one. = The only=20 possible reason (good reason) is the bridging infrastructure but = that=20 infrastructure has a completely different rationale than the=20 flowcontrol.

>=20 This issue and message is quite different from the issues and=20 messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen. =  
>=20
> None of the above cancels out the value of iSCSI in=20 numerous
> environments.
>
>

iSCSI is = good for=20 all environments. Business consideration (and some politics) keep it = form=20 "exploding" and large storage vendors are completely indifferent to = the=20 network connection they are using.
You and I have also slightly=20 different views of DCE. I expect DCE (that still has a way to go) to = improve=20 the QoS in the data-center (and for storage too). You expect it to = bring the=20 loss rates down to the levels that FCP assumes (FCP has no transport = layer)=20 and that is probably a pipe dream. Todays transport solution for = loss=20 mitigation are far more cost effective - and that's why iFCP is a = better=20 proposition as a transition technology than FCoE and iSCSI with = gateways is=20 propably better in the long run. =

>
> .
> .
> .
> John L=20 Hufferd
> Sr. Executive Director of Technology
>=20 jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: = (408)=20 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408)=20 627-9606
>  
> -----Original Message-----
> = From:=20 John Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> = To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent = comments about=20 FCoE and iSCSI
>
> Julian,
> I think you are = wrong on=20 this one.  The arguments are quite different
> then the = ones we=20 had in pre iSCSI days.  (By the way I missed you on
> = today's=20 Renato meeting/conf call where Brocade took the IBM = technology
> group=20 through FCoE as it is being placed in our plans).  
> I = will send=20 you more info when I get to my computer.  But you = probably
> were=20 sent the Brocade charts.  Please review them and I will follow=20 up
> with more information.
> This does NOT replace = iSCSI it=20 applies only to a DataCenter envuornment
> with lossless DCE=20 ethernet.
> --------------------------
> John L. = Hufferd
>=20 Sr. Ex. Director of Technology
> Brocade Communications = Systems,=20 Inc.
> Phone: (408) 333-5244
> Mobile: (408) = 627-9606
>=20 eMail: jhufferd@brocade.com
> (Sent from my BlackBerry=20 Wireless)
>  
>
> ----- Original Message=20 -----
> From: Julian Satran = <Julian_Satran@il.ibm.com>
>=20 To: ips@ietf.org <ips@ietf.org>
> Sent: Tue Apr 24 = 12:10:29=20 2007
> Subject: [Ips] Recent comments about FCoE and = iSCSI
>=20
>
> Dear All,
>
> The trade press is = lately full=20 with comments about the latest and
> greatest reincarnation of = Fiber=20 Channel over ethernet.
> It made me try and summarize all the = long=20 and hot debates that preceded
> the advent of iSCSI.
> = Although=20 FCoE proponents make it look like no debate preceded iSCSI = that
> was=20 not so - FCoE was considered even then and was dropped as a = dumb
>=20 idea.
>
> Here is a summary (as afar as I can = remember) of the=20 main arguments.
> They are not bad arguments even in = retrospect and=20 technically FCoE
> doesn't look better than it did then. =
>=20
> Feel free to use this material in a nay form. I expect this = group=20 to
> seriously  expand my arguments and make them public = - in=20 personal or
> collective form.
>
> And do not = forget -=20 it is a technical dispute - although we all must
> have some = doubts=20 about the way it is pursued.
>
> Regards,
> = Julo=20
>
>=20 = ---------------------------------------------------------------------=20
>
> What a piece of nostalgia :-)
>
> = Around=20 1997 when a team at IBM Research (Haifa and Almaden) started
> = looking=20 at connecting storage to servers using the "regular network"
> = (the=20 ubiquitous LAN) we considered many alternatives (another team = even
>=20 had a look at ATM - still a computer network candidate at the time). = I
> won't get you over all of our rationale (and we went over = some of=20 them
> again at the end of 1999 with a team from CISCO before = we=20 convened the
> first IETF BOF in 2000 at Adelaide that = resulted in=20 iSCSI and all the
> rest) but some of the reasons we choose to = drop=20 Fiber Channel over raw
> Ethernet where multiple:
> =
>=20
> *   Fiber Channel Protocol (SCSI over Fiber Channel = Link)=20 is
> "mildly" effective because:
>
>   =  *=20   it implements endpoints in a dedicated engine >=20 jhufferd@brocade.com
> Office Phone: (408) 333-5244; eFAX: = (408)=20 904-4688
> Alt Office Phone: (408) 997-6136; Cell: (408)=20 627-9606
>  
> -----Original Message-----
> = From:=20 John Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> = To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent = comments about=20 FCoE and iSCSI
>
> Julian,
> I think you are = wrong on=20 this one.  The arguments are quite different
> then the = ones we=20 had in pre iSCSI days.  (By the way I missed you on
> = today's=20 Renato meeting/conf call where Brocade took the IBM = technology
> group=20 through FCoE as it is being placed in our plans).  
> I = will send=20 you more info when I get to my computer.  But you = probably
> were=20 sent the Brocade charts.  Please review them and I will follow=20 up
> with more information.
> This does NOT replace = iSCSI it=20 applies only to a DataCenter envuornment
> with lossless DCE=20 ethernet.
> --------------------------
> John L. = Hufferd
>=20 Sr. Ex. Director of Technology
> Brocade Communications = Systems,=20 Inc.
> Phone: (408) 333-5244
> Mobile: (408) = 627-9606
>=20 eMail: jhufferd@brocade.com
> (Sent from my BlackBerry=20 Wireless)
>  
>
> ----- Original Message=20 -----
> From: Julian Satran = <Julian_Satran@il.ibm.com>
>=20 To: ips@ietf.org <ips@ietf.org>
> Sent: Tue Apr 24 = 12:10:29=20 2007
> Subject: [Ips] Recent comments about FCoE and = iSCSI
>=20
>
> Dear All,
>
> The trade press is = lately full=20 with comments about the latest and
> greatest reincarnation of = Fiber=20 Channel over ethernet.
> It made me try and summarize all the = long=20 and hot debates that preceded
> the advent of iSCSI.
> = Although=20 FCoE proponents make it look like no debate preceded iSCSI = that
> was=20 not so - FCoE was considered even then and was dropped as a = dumb
>=20 idea.
>
> Here is a summary (as afar as I can = remember) of the=20 main arguments.
> They are not bad arguments even in = retrospect and=20 technically FCoE
> doesn't look better than it did then. =
>=20
> Feel free to use this material in a nay form. I expect this = group=20 to
> seriously  expand my arguments and make them public = - in=20 personal or
> collective form.
>
> And do not = forget -=20 it is a technical dispute - although we all must
> have some = doubts=20 about the way it is pursued.
>
> Regards,
> = Julo=20
>
>=20 = ---------------------------------------------------------------------=20
>
> What a piece of nostalgia :-)
>
> = Around=20 1997 when a team at IBM Research (Haifa and Almaden) started
> = looking=20 at connecting storage to servers using the "regular network"
> = (the=20 ubiquitous LAN) we considered many alternatives (another team = even
>=20 had a look at ATM - still a computer network candidate at the time). = I
> won't get you over all of our rationale (and we went over = some of=20 them
> again at the end of 1999 with a team from CISCO before = we=20 convened the
> first IETF BOF in 2000 at Adelaide that = resulted in=20 iSCSI and all the
> rest) but some of the reasons we choose to = drop=20 Fiber Channel over raw
> Ethernet where multiple:
> =
>=20
> *   Fiber Channel Protocol (SCSI over Fiber Channel = Link)=20 is
> "mildly" effective because:
>
>   =  *=20   it implements endpoints in a dedicated engine (Offload) =
>=20    *   it has no transport layer (recovery is done at = the
> application layer under the assumption that the error = rate will=20 be very
> low)
>    *   the network is = limited=20 in physical span and logical span
> (number of switches) =
>=20    *   flow-control/congestion control is achieved = with=20 a
> mechanism adequate for a limited span network (credits). = The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *=20   The assumption of low losses due to errors might
> = radically=20 change when moving from 1 to 10 Gb/s - the scaling argument
> =  =20  *   Ethernet has no credit mechanism and any mechanism=20 with
> a similar effect increases the end point cost. Building = a=20 transport
> layer in the protocol stack has always been the = preferred=20 choice of the
> networking community - the community argument =
>=20    *   The "performance penalty" of a complete = protocol=20 stack
> has always been overstated (and overrated). Advances = in=20 protocol stack
> implementation and finer tuning of the = congestion=20 control mechanisms
> make conventional TCP/IP performing well = even at=20 10 Gb/s and over.
> Moreover the multicore processors that = become=20 dominant on the computing
> scene have enough compute cycles = available=20 to make any "offloading"
> possible as a mere code = restructuring=20 exercise (see the stack reports
> from Intel, IBM etc.) =
>=20    *   Building on a complete stack makes available a = wealth=20 of
> operational and management mechanisms built over the = years by=20 the
> networking community (routing, provisioning, security, = service=20 location
> etc.) - the community argument
>   =  *=20   Higher level storage access over an IP network is = widely
>=20 available and having both block and file served over the same=20 connection
> with the same support and management structure is = compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't=20 know exactly
> how it will operate). Untill then the scale of = Layer 2=20 network is going
> to seriously limited - the scaling = argument
>=20
>
>
>       As a side argument - = a=20 performance comparison made in
> 1998 showed SCSI over TCP (a=20 predecessor of the later iSCSI) to perform
> better than FCP = at 1Gbs=20 for block sizes typical for OLTP (4-8KB). That
> was what = convinced us=20 to take the path that lead to iSCSI - and we used
> plain = vanilla x86=20 servers with plain-vanilla NICs and Linux (with
> similar = meas(Offload) =
>=20    *   it has no transport layer (recovery is done at = the
> application layer under the assumption that the error = rate will=20 be very
> low)
>    *   the network is = limited=20 in physical span and logical span
> (number of switches) =
>=20    *   flow-control/congestion control is achieved = with=20 a
> mechanism adequate for a limited span network (credits). = The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *=20   The assumption of low losses due to errors might
> = radically=20 change when moving from 1 to 10 Gb/s - the scaling argument
> =  =20  *   Ethernet has no credit mechanism and any mechanism=20 with
> a similar effect increases the end point cost. Building = a=20 transport
> layer in the protocol stack has always been the = preferred=20 choice of the
> networking community - the community argument =
>=20    *   The "performance penalty" of a complete = protocol=20 stack
> has always been overstated (and overrated). Advances = in=20 protocol stack
> implementation and finer tuning of the = congestion=20 control mechanisms
> make conventional TCP/IP performing well = even at=20 10 Gb/s and over.
> Moreover the multicore processors that = become=20 dominant on the computing
> scene have enough compute cycles = available=20 to make any "offloading"
> possible as a mere code = restructuring=20 exercise (see the stack reports
> from Intel, IBM etc.) =
>=20    *   Building on a complete stack makes available a = wealth=20 of
> operational and management mechanisms built over the = years by=20 the
> networking community (routing, provisioning, security, = service=20 location
> etc.) - the community argument
>   =  *=20   Higher level storage access over an IP network is = widely
>=20 available and having both block and file served over the same=20 connection
> with the same support and management structure is = compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't=20 know exactly
> how it will operate). Untill then the scale of = Layer 2=20 network is going
> to seriously limited - the scaling = argument
>=20
>
>
>       As a side argument - = a=20 performance comparison made in
> 1998 showed SCSI over TCP (a=20 predecessor of the later iSCSI) to perform
> better than FCP = at 1Gbs=20 for block sizes typical for OLTP (4-8KB). That
> was what = convinced us=20 to take the path that lead to iSCSI - and we used
> plain = vanilla x86=20 servers with plain-vanilla NICs and Linux (with
> similar = measurements=20 conducted on Windows).
>    The networking and = storage=20 community acknowledged those
> arguments and developed iSCSI = and the=20 companion protocols for service
> discovery, boot etc. =
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a = reasonable=20 fashion and developed 2
> protocols iFCP (to support hosts = with FCP=20 drivers and IP connections to
> connect to storage by a simple = conversion from FCP to TCP packets) FCPIP
> to extend the = reach of FCP=20 through IP (connects FCP islands through TCP
> links). Both = have been=20
>    implemented and their foundation is solid. =
>=20    
>    The current attempt of developing = a=20 "new-age" FCP over an
> Ethernet link is going against most of = the=20 arguments that have given us
> iSCSI etc.
>  =20  
>    It ignores the networking layering = practice,=20 build an
> application protocol directly above a link and thus = limits=20 scaling,
> mandates elements at the link layer and application = layer=20 that make
> applications more expensive and leaves aside the = whole=20 "ecosystem" that
> accompanies TCP/IP (and not Ethernet). =
>=20    
>    In some related effort (and at a = point=20 also when developing
> iSCSI) we considered also moving away = from SCSI=20 (like some "no
> standardized" but popular in some circles = software=20 did - e.g., NBP) but
> decided against. SCSI is a mature and = well=20 understood access
> architecture for block storage and is = implemented=20 by many device
> vendors. Moving away from it would not have = been=20 justified at the time.
>    =20

_______________________________________________
Ips=20 mailing = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips=20

_______________________________________________
Ips=20 mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= _______________________________________________
Ips = mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------=_NextPart_000_005D_01C7874C.8B7085A0-- --===============0046708084== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0046708084==-- From ips-bounces@ietf.org Thu Apr 26 09:12:29 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh3lr-0008B4-7j; Thu, 26 Apr 2007 09:12:27 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgoUE-00055B-2V for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 16:53:14 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgoUD-000553-PX for ips@ietf.org; Wed, 25 Apr 2007 16:53:13 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgoJX-0001Qm-8L for ips@ietf.org; Wed, 25 Apr 2007 16:42:11 -0400 Received: from web36705.mail.mud.yahoo.com ([209.191.85.39]) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HgoF4-00024M-W4 for ips@ietf.org; Wed, 25 Apr urements=20 conducted on Windows).
>    The networking and = storage=20 community acknowledged those
> arguments and developed iSCSI = and the=20 companion protocols for service
> discovery, boot etc. =
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a = reasonable=20 fashion and developed 2
> protocols iFCP (to support hosts = with FCP=20 drivers and IP connections to
> connect to storage by a simple = conversion from FCP to TCP packets) FCPIP
> to extend the = reach of FCP=20 through IP (connects FCP islands through TCP
> links). Both = have been=20
>    implemented and their foundation is solid. =
>=20    
>    The current attempt of developing = a=20 "new-age" FCP over an
> Ethernet link is going against most of = the=20 arguments that have given us
> iSCSI etc.
>  =20  
>    It ignores the networking layering = practice,=20 build an
> application protocol directly above a link and thus = limits=20 scaling,
> mandates elements at the link layer and application = layer=20 that make
> applications more expensive and leaves aside the = whole=20 "ecosystem" that
> accompanies TCP/IP (and not Ethernet). =
>=20    
>    In some related effort (and at a = point=20 also when developing
> iSCSI) we considered also moving away = from SCSI=20 (like some "no
> standardized" but popular in some circles = software=20 did - e.g., NBP) but
> decided against. SCSI is a mature and = well=20 understood access
> architecture for block storage and is = implemented=20 by many device
> vendors. Moving away from it would not have = been=20 justified at the time.
>    =20

_______________________________________________
Ips=20 mailing = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips=20

_______________________________________________
Ips=20 mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= _______________________________________________
Ips = mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------=_NextPart_000_005D_01C7874C.8B7085A0-- --===============0046708084== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0046708084==-- From ips-bounces@ietf.org Thu Apr 26 09:12:29 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh3lr-0008B4-7j; Thu, 26 Apr 2007 09:12:27 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgoUE-00055B-2V for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 16:53:14 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HgoUD-000553-PX for ips@ietf.org; Wed, 25 Apr 2007 16:53:13 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HgoJX-0001Qm-8L for ips@ietf.org; Wed, 25 Apr 2007 16:42:11 -0400 Received: from web36705.mail.mud.yahoo.com ([209.191.85.39]) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HgoF4-00024M-W4 for ips@ietf.org; Wed, 25 Apr 2007 16:37:35 -0400 Received: (qmail 47400 invoked by uid 60001); 25 Apr 2007 20:37:34 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=BCvDI6KYVKKCjLqhSZobv9OIt0TzDI9UvHyC0NOYF0HjkkTn8UbTrTV7eekXebVCpfPpgA0CiE3cG5o3IqYIQuLqcYxFRBTUrFmAKk+4yKYWgwj0JyA6O7x5kGkQMU3W2IiGB2iMVqNJXEeD5JF9XG81B18uRZjHCmhL3WFTJ0s=; X-YMail-OSG: hKxu9GEVM1kUiO0sGWlhMhtqF3W40ZYIH.InTcW581wGRaPfcRfQ8tyW3o3R5y55A1xx8utlznQeWH2hgengHYOiRADHdbGGicYxEnOQMi2hL26PktB72igBNC338Q-- Received: from [138.239.78.107] by web36705.mail.mud.yahoo.com via HTTP; Wed, 25 Apr 2007 13:37:33 PDT Date: Wed, 25 Apr 2007 13:37:33 -0700 (PDT) From: Zack Best Subject: RE: [Ips] Recent comments about FCoE and iSCSI To: ips@ietf.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <179300.47262.qm@web36705.mail.mud.yahoo.com> X-Spam-Score: 0.8 (/) X-Scan-Signature: ce732c7d36989a1bd55104ba259c40a1 X-TMDA-Confirmed: Wed, 25 Apr 2007 16:53:13 -0400 X-Mailman-Approved-At: Thu, 26 Apr 2007 09:12:25 -0400 X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org The real debate here is between two types of networks. The first is reliable at the link level and does not drop packets under congestion. The second is running a reliable transport protocol (i.e. TCP) over an unreliable link level network. I agree with the scaling argument. For sufficiently large networks, reliable link level doesn't work well because network component failure, or chronically congested links are not handled well. For sufficiently small networks, reliable link level has some significant advantages in simplicity, low hardware cost, performance, and worst case latency. My personal view is that the vast majority of enterprise storage networks fall in the "sufficiently small" category. This view has to some extent been vindicated by the continuing success of Fibre Channel in this space and the inability of iSCSI to displace FC in any significant way for enterprise storage. Of course, this may or may not change in the future. Whether FC is simpler than iSCSI depends largely on your definition of simplicity. If one defines simplicity/complexity as the number of gates or lines of code to reduce the protocol to hardware or firmware, then my experience is that iSCSI is 2X to 3X the complexity of FC. This has implications in cost and reliability. Particularly problematic with iSCSI is the unpredictability of the performance. Performance is great with no packet drop. However even a small amount of congestion can cause a sudden large drop and performance. This can be difficult to predict as a network that is almost but not quite congested can run great, but a small incremental change of any sort can cause the performance to become suddenly unacceptable. For FC, or other protocol using link level flow control, the reduction in performance is much more graceful and incremental when the level of congestion is small and intermittent. A second major problem with iSCSI is the unbounded nature of worst case latency. When a storage network fails, it is desirable to detect the failure in a fraction of a second and transition to a backup network. TCP, when implemented to the standards, can take many seconds or minutes to determine that a network has failed and close the connection. RFC 2988, for instance, requires that the minimum retransmission be one second. This means a single dropped packet may add one second to the latency of outstanding commands. This is a huge amount of time on a 10G link. No doubt 2007 16:37:35 -0400 Received: (qmail 47400 invoked by uid 60001); 25 Apr 2007 20:37:34 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=BCvDI6KYVKKCjLqhSZobv9OIt0TzDI9UvHyC0NOYF0HjkkTn8UbTrTV7eekXebVCpfPpgA0CiE3cG5o3IqYIQuLqcYxFRBTUrFmAKk+4yKYWgwj0JyA6O7x5kGkQMU3W2IiGB2iMVqNJXEeD5JF9XG81B18uRZjHCmhL3WFTJ0s=; X-YMail-OSG: hKxu9GEVM1kUiO0sGWlhMhtqF3W40ZYIH.InTcW581wGRaPfcRfQ8tyW3o3R5y55A1xx8utlznQeWH2hgengHYOiRADHdbGGicYxEnOQMi2hL26PktB72igBNC338Q-- Received: from [138.239.78.107] by web36705.mail.mud.yahoo.com via HTTP; Wed, 25 Apr 2007 13:37:33 PDT Date: Wed, 25 Apr 2007 13:37:33 -0700 (PDT) From: Zack Best Subject: RE: [Ips] Recent comments about FCoE and iSCSI To: ips@ietf.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <179300.47262.qm@web36705.mail.mud.yahoo.com> X-Spam-Score: 0.8 (/) X-Scan-Signature: ce732c7d36989a1bd55104ba259c40a1 X-TMDA-Confirmed: Wed, 25 Apr 2007 16:53:13 -0400 X-Mailman-Approved-At: Thu, 26 Apr 2007 09:12:25 -0400 X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org The real debate here is between two types of networks. The first is reliable at the link level and does not drop packets under congestion. The second is running a reliable transport protocol (i.e. TCP) over an unreliable link level network. I agree with the scaling argument. For sufficiently large networks, reliable link level doesn't work well because network component failure, or chronically congested links are not handled well. For sufficiently small networks, reliable link level has some significant advantages in simplicity, low hardware cost, performance, and worst case latency. My personal view is that the vast majority of enterprise storage networks fall in the "sufficiently small" category. This view has to some extent been vindicated by the continuing success of Fibre Channel in this space and the inability of iSCSI to displace FC in any significant way for enterprise storage. Of course, this may or may not change in the future. Whether FC is simpler than iSCSI depends largely on your definition of simplicity. If one defines simplicity/complexity as the number of gates or lines of code to reduce the protocol to hardware or firmware, then my experience is that iSCSI is 2X to 3X the complexity of FC. This has implications in cost and reliability. Particularly problematic with iSCSI is the unpredictability of the performance. Performance is great with no packet drop. However even a small amount of congestion can cause a sudden large drop and performance. This can be difficult to predict as a network that is almost but not quite congested can run great, but a small incremental change of any sort can cause the performance to become suddenly unacceptable. For FC, or other protocol using link level flow control, the reduction in performance is much more graceful and incremental when the level of congestion is small and intermittent. A second major problem with iSCSI is the unbounded nature of worst case latency. When a storage network fails, it is desirable to detect the failure in a fraction of a second and transition to a backup network. TCP, when implemented to the standards, can take many seconds or minutes to determine that a network has failed and close the connection. RFC 2988, for instance, requires that the minimum retransmission be one second. This means a single dropped packet may add one second to the latency of outstanding commands. This is a huge amount of time on a 10G link. No doubt this could be mitigated by drastically reducing the timeouts within TCP, but the market seems to be surprisingly resistant to tampering with accepted standards here. Overall, the FC and FCP protocol have a lot in common with the Intel i86 instruction set architecture. They are overly complex, and rather poorly designed by modern standards. But they are good enough, and there is a huge amount of value add that has been built on top of them, and therefore little incentive to change. FCoE is an interesting idea because it preserves 90% of the existing value add of FC, unifies the physical link with Ethernet, and uses the reliable link method of packet delivery. There are two significant possibilities for iSCSI to displace FC (or FCoE) in enterprise storage networks. First is if the networks start to scale to large enough size that FC can't be made sufficiently reliable, and second if CPU compute cycles become sufficiently cheap that the iSCSI protocol can be run in host software with no negative performance impact. Barring either of these, it seems that iSCSI will have an uphill battle, and FCoE may have a place. -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Tuesday, April 24, 2007 3:10 PM To: ips@ietf.org Subject: [Ips] Recent comments about FCoE and iSCSI Dear All, The trade press is lately full with comments about the latest and greatest reincarnation of Fiber Channel over ethernet. It made me try and summarize all the long and hot debates that preceded the advent of iSCSI. Although FCoE proponents make it look like no debate preceded iSCSI that was not so - FCoE was considered even then and was dropped as a dumb idea. Here is a summary (as afar as I can remember) of the main arguments. They are not bad arguments even in retrospect and technically FCoE doesn't look better than it did then. Feel free to use this material in a nay form. I expect this group to seriously expand my arguments and make them public - in personal or collective form. And do not forget - it is a technical dispute - although we all must have some doubts about the way it is pursued. Regards, Julo --------------------------------------------------------------------- What a piece of nostalgia :-) Around 1997 when a team at IBM Research (Haifa and Almaden) started looking at connecting storage to servers using the "regular network" (the ubiquitous LAN) we considered many alternatives (another team even had a look at ATM - still a computer network candidate at the time). I won't get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple: Fiber Channel Protocol (SCSI over Fiber Channel Link) is "mildly" effective because: it implements endpoints in a dedicated engine (Offload) it has no transport layer (recovery is done at the application layer under the assumption that the error rate will be very low) the network is limited in physical span and logical span (number of switches) flow-control/congestion control is achieved with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechanism) However FCP endpoints are inherently costlier than simple NICs – the cost argument (initiators are more expensive) The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) – the scaling argument The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s – the scaling argument Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has athis could be mitigated by drastically reducing the timeouts within TCP, but the market seems to be surprisingly resistant to tampering with accepted standards here. Overall, the FC and FCP protocol have a lot in common with the Intel i86 instruction set architecture. They are overly complex, and rather poorly designed by modern standards. But they are good enough, and there is a huge amount of value add that has been built on top of them, and therefore little incentive to change. FCoE is an interesting idea because it preserves 90% of the existing value add of FC, unifies the physical link with Ethernet, and uses the reliable link method of packet delivery. There are two significant possibilities for iSCSI to displace FC (or FCoE) in enterprise storage networks. First is if the networks start to scale to large enough size that FC can't be made sufficiently reliable, and second if CPU compute cycles become sufficiently cheap that the iSCSI protocol can be run in host software with no negative performance impact. Barring either of these, it seems that iSCSI will have an uphill battle, and FCoE may have a place. -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Tuesday, April 24, 2007 3:10 PM To: ips@ietf.org Subject: [Ips] Recent comments about FCoE and iSCSI Dear All, The trade press is lately full with comments about the latest and greatest reincarnation of Fiber Channel over ethernet. It made me try and summarize all the long and hot debates that preceded the advent of iSCSI. Although FCoE proponents make it look like no debate preceded iSCSI that was not so - FCoE was considered even then and was dropped as a dumb idea. Here is a summary (as afar as I can remember) of the main arguments. They are not bad arguments even in retrospect and technically FCoE doesn't look better than it did then. Feel free to use this material in a nay form. I expect this group to seriously expand my arguments and make them public - in personal or collective form. And do not forget - it is a technical dispute - although we all must have some doubts about the way it is pursued. Regards, Julo --------------------------------------------------------------------- What a piece of nostalgia :-) Around 1997 when a team at IBM Research (Haifa and Almaden) started looking at connecting storage to servers using the "regular network" (the ubiquitous LAN) we considered many alternatives (another team even had a look at ATM - still a computer network candidate at the time). I won't get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple: Fiber Channel Protocol (SCSI over Fiber Channel Link) is "mildly" effective because: it implements endpoints in a dedicated engine (Offload) it has no transport layer (recovery is done at the application layer under the assumption that the error rate will be very low) the network is limited in physical span and logical span (number of switches) flow-control/congestion control is achieved with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechanism) However FCP endpoints are inherently costlier than simple NICs – the cost argument (initiators are more expensive) The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) – the scaling argument The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s – the scaling argument Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has always been the preferred choice of the networking community – the community argument The "performance penalty" of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computing scene have enough compute cycles available to make any "offloading" possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.) Building on a complete stack makes available a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service location etc.) – the community argument Higher level storage access over an IP network is widely available and having both block and file served over the same connection with the same support and management structure is compelling – the community argument Highly efficient networks are easy to build over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don't know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited – the scaling argument As a side argument – a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI – and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows). The networking and storage community acknowledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc. The community also acknowledged the need to support existing infrastructure and extend it in a reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been implemented and their foundation is solid. The current attempt of developing a "new-age" FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc. It ignores the networking layering practice, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applications more expensive and leaves aside the whole "ecosystem" that accompanies TCP/IP (and not Ethernet). In some related effort (and at a point also when developing iSCSI) we considered also moving away from SCSI (like some "no standardized" but popular in some circles software did – e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips lways been the preferred choice of the networking community – the community argument The "performance penalty" of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computing scene have enough compute cycles available to make any "offloading" possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.) Building on a complete stack makes available a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service location etc.) – the community argument Higher level storage access over an IP network is widely available and having both block and file served over the same connection with the same support and management structure is compelling – the community argument Highly efficient networks are easy to build over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don't know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited – the scaling argument As a side argument – a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI – and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows). The networking and storage community acknowledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc. The community also acknowledged the need to support existing infrastructure and extend it in a reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been implemented and their foundation is solid. The current attempt of developing a "new-age" FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc. It ignores the networking layering practice, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applications more expensive and leaves aside the whole "ecosystem" that accompanies TCP/IP (and not Ethernet). In some related effort (and at a point also when developing iSCSI) we considered also moving away from SCSI (like some "no standardized" but popular in some circles software did – e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 09:12:29 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh3lr-0008CX-Jr; Thu, 26 Apr 2007 09:12:27 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hgqaj-0005sB-GK for ips-confirm+ok@megatron.ietf.org; Wed, 25 Apr 2007 19:08:05 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hgqai-0005lU-To for ips@ietf.org; Wed, 25 Apr 2007 19:08:04 -0400 Received: from mx20.brocade.com ([66.243.153.19]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HgqX1-00009e-KM for ips@ietf.org; Wed, 25 Apr 2007 19:04:17 -0400 Received: from mailhost.brocade.com (HELO discus.brocade.com) ([192.168.126.240]) by mx20.brocade.com with ESMTP; 25 Apr 2007 16:04:14 -0700 X-IronPort-AV: i="4.14,452,1170662400"; d="scan'208,217"; a="8627162:sNHT39266199" Received: from HQ-EXCHFE-2.corp.brocade.com (hq-vipexchfe-2.brocade.com [192.168.126.214]) by discus.brocade.com (Postfix) with ESMTP id 5F93423836B; Wed, 25 Apr 2007 16:03:04 -0700 (PDT) Received: from hq-exch-1.corp.brocade.com ([10.3.8.21]) by HQ-EXCHFE-2.corp.brocade.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 25 Apr 2007 16:04:14 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Wed, 25 Apr 2007 16:04:13 -0700 Message-ID: <6002A63FDB393D4F9ADB36DE70C4847502C80970@hq-exch-1.corp.brocade.com> In-Reply-To: <007701c78782$6dd93c40$05faa8c0@ivivity.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHgq4ypt5zILt2TKKPjQpNbkHJHgACxghQ References: <007701c78782$6dd93c40$05faa8c0@ivivity.com> From: "Robert Snively" To: "Eddy Quicksall" , "Julian Satran" X-OriginalArrivalTime: 25 Apr 2007 23:04:14.0818 (UTC) FILETIME=[0B4EFC20:01C7878E] X-Spam-Score: 0.5 (/) X-Scan-Signature: 6bce69840c70b11602de7eeb3bb2b5f5 X-Mailman-Approved-At: Thu, 26 Apr 2007 09:12:25 -0400 Cc: ips@ietf.org, John Hufferd X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0090503334==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0090503334== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C7878E.0B122647" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7878E.0B122647 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable The standardization process is just beginning on FCoE. Brocade and=20 at least two other organizations have done significant work on the=20 subject. I expect content similar to the Cisco patent to be a significant contribution to the FCoE standard. As you know, standards development often involves negotiation of details of a technology. =20 Bob ________________________________ From: Eddy Quicksall [mailto:Quicksall_iSCSI@Bellsouth.net]=20 Sent: Wednesday, April 25, 2007 2:41 PM To: Julian Satran Cc: ips@ietf.org; John Hufferd Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI I notice that Cisco has a patent on Fibre Channel over Ethernet. Is FCoE something different? ----- Original Message -----=20 From: Julian Satran =20 To: Eddy Quicksall =20 Cc: ips@ietf.org ; John Hufferd =20 Sent: Wednesday, April 25, 2007 11:52 AM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Sorry Eddy by sizable I meant even at the size of a modern data center. Julo=20 =09 =09 =09 "Eddy Quicksall" =20 25/04/07 11:08=20 To Julian Satran/Haifa/IBM@IBMIL=20 cc ips@ietf.org, John Hufferd =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI=09 =09 I basically said that in the summery line by saying "it will not route on the "global" scale like TCP/IP would".=20 ----- Original Message -----=20 From: Julian Satran =20 To: Eddy Quicksall =20 Cc: ips@ietf.org ; John Hufferd =20 Sent: Wednesday, April 25, 2007 11:04 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 =09 =09 Eddy,=20 =09 That is oversimplified and ignore the drop rate assumption and error rate assumptions made in FCP(FCP has no transport layer). To get to it on a sizable network requires more than PAUSE.=20 =09 Julo=20 =09 =09 "Eddy Quicksall" >=20 25/04/07 10:07=20 To "John Hufferd" >, Julian Satran/Haifa/IBM@IBMIL =20 cc >=20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI=09 =09 =09 =09 =09 Basically, it is sending FC frames over Ethernet. This localizes the traffic unless you route based on MAC addresses. So you send 2146 bytes of FC frame plus 18 bytes of Ethernet overhead as FCoE "standard" packet. 18 bytes of Ethernet gets stripped and you have straight FC frame that can go through any FC network. Now you can have 10G Ethernet pipes into existing FC SANs. Limited market potential as far as I can see. The key argument is it much easier to implement than iSCSI and also has less overhead and uses all the benefits of FC. End to End credits are simulated using PAUSE command on Ethernet and MAC addresses are mapped into WWNs.=20 Biggest knock is that it will not route on the "global" scale like TCP/IP would.=20 Eddy=20 ----- Original Message -----=20 From: Julian Satran =20 To: John Hufferd =20 Cc: ips@ietf.org =20 Sent: Wednesday, April 25, 2007 8:09 AM=20 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 =09 =09 =09 "John Hufferd" > wrote on 25/04/2007 02:45:51: =09 > Julian,=20 > To be sure you understand our position; Brocade is pushing iSCSI as an > outreach protocol from the Data Center. We also believe iSCSI is very > useful for installations that do not have a Fibre Channel > infrastructure, and in that case we will be able to sell them our new > iSCSI and TOE offload HBAs. =20 >=20 > When I say iSCSI is an outreach protocol, this is a statement that iSCSI > is very important to connect "stranded" servers to the Fibre Channel > Fabric. That is, we sell iSCSI-to-FC Gateway devices which will permit > iSCSI Servers (software or HBA iSCSI initiators) to connect to the > Enterprise "Bet Your Business" FC Storage. This of course also applies > to Desktops and Laptop systems, and systems at distance. >=20 =09 You make it sound like:=20 1. most of the servers in the world have their storage on the network - and that is not the case=20 2. FCP is basically better performing than iSCSI - and that is not true either=20 3. Gatewaying is expensive - and it is perhaps so but only if you are completely relying on FCP storage (and there are plenty of good iSCSI vendors of storage)and pushing the price on the servers is not cheap either - at least not for the server buyer=20 =09 > Now with that positioning, it is important to understand the limitation > to this strategy. The primary problem is that iSCSI to FC Bridging > (Gatewaying) is relatively expensive (compared to simple FC > connections). Though we have some of the best priced Gateways on the > market, it is not cost feasible to replace all the server connectivity > to FC storage with iSCSI for the hundred to thousands of servers in the > Data Center. And since, if there is to be a consolidated Network > connection to the servers in the Data Center, there must be an > evolutionary replacement of Server Connections to Storage. That means > there must be a bridge/Gateway approach. And as I mentioned before, > there is just too much cost in the iSCSI to FC Gateway. >=20 > The issue is the server requirement to have a single connection type to > handle cluster messaging, general messaging, and storage. iSCSI is > clearly an option for the storage, however, the gateway costs are too > high for iSCSI to be used as the "normal" server connect into a FC based > Fabric. That is true for the current 1GE; and for 10GE the cost is just > out of sight. The reason for this is the requirement for TCP/IP > termination and re-initiation with FC at the Gateway. >=20 > Now with respect to FC over Ethernet the important thing to understand > is that it is not Ethernet as we have known it up to know. The Ethernet > we are talking about is a type of Ethernet that can only be deployed in > a constrained environment such as a Data Center. This form of Ethernet > is called DCE (Data Center Ethernet) or CEE (Convergence Enhanced > Ethernet). This form of Ethernet is a Loss-less type Ethernet, with > multi-priority and Flow Control. This is NOT an Internet or Intranet > type of Ethernet. >=20 > FCoE is all about using the DCE (CEE) to carry FC frames. The rest of > the Host and storage stack remain the same, the functions and features > of the switches also remain the same and add the capability to provide > Cluster Message Switching which has latency close to InfiniBand speeds. >=20 >=20 > Because the FC frames are transported to the switches intact via a DCE > frame, the Bridging, if you want to call it that, is virtually non > existent. Hence you can deliver the FC frames to FC devices, or send FC > frames to DCE FCoE devices, just like one would do if it was all FC. And > all this is done while performing Cluster message switching and general > message trucking to the IP outfacing network.=20 >=20 =09 The rosy future of the yet to appear DCE/CEE and a layer 2 only world.=20 First you have some terms confused:=20 =09 Bridging is the term commonly used for Layer-2 switching and routing is therm used for layer-3 (switching).=20 =09 Bridging has some advantages (less management) that have created a movement towards an enterprise wide=20 LAN. But this has a long way to go and will require significant equipment and protocol changes.=20 Even its proponents do not call for transportless networks, lossless networks etc.=20 The second trouble with your argument is that there are no known large scale networking technologies that=20 really work at full speed (high speed) and are lossless (flow-controlled) and errorless like FCoE assumes.=20 The TCP/IP has solved this issue for every generation using the proven end-to-end principle (and is doing so now).=20 And it is not by chance so and that is why all networking applications are built above layer-3 and not dropping layer-3 (like FCoE) does.=20 =09 Although I can understand the DCE arguments as a management statement I would prefer like any rational engineer, to base my building blocks on structures that are proven and long lasting. And those are still the end-to-end TCP/IP that can accommodate even your FCP addicts. The IPS TWG has developed the iFCP that does exactly what FCoE claims to do an a better base.=20 =09 > This means an evolutionary process is possible to the solution of > getting a single Fabric connection for all networks connected to a > server, further, the process has very low interconnection cost on the > Data Center Fabric. And it maintains all the FC Fabric Services, and all > the same Storage Management processes.=20 >=20 > By the way, this is primarily a Server driven value statement, there > seems to be little value in having FCoE on the storage controller. > Therefore FC storage controllers (and FICON) will be the very last > things that connect using FCoE and that evolution will take at least a > decade or more. >=20 =09 It is server cost statement. It costs nothing to connect a modern server to ethernet it will cost a bundle to connect to FCoE and it will force users in short lived bad solutions.=20 =09 =09 > We see value in offering switches and Directors that can support DCE > switching, FC switching as well as iSCSI interconnect, and the > "Trunking" of general messaging to the Outfacing IP network. That said; > we do not see FCoE going beyond the constraints of the Data Center.=20 =09 Data Centers now grow to tens of thousands of nodes. There is no layer-2 technology for errorless/lossless operation at this scale and there is no good reason to pursue one. The only possible reason (good reason) is the bridging infrastructure but that infrastructure has a completely different rationale than the flowcontrol.=20 =09 > This issue and message is quite different from the issues and messages > we struggled with when we started iSCSI. There is a consortium of folks > both working on the DCE (CEE) and the FCoE. Without the DCE the FCoE > will not happen. =20 >=20 > None of the above cancels out the value of iSCSI in numerous > environments. >=20 >=20 =09 iSCSI is good for all environments. Business consideration (and some politics) keep it form "exploding" and large storage vendors are completely indifferent to the network connection they are using.=20 You and I have also slightly different views of DCE. I expect DCE (that still has a way to go) to improve the QoS in the data-center (and for storage too). You expect it to bring the loss rates down to the levels that FCP assumes (FCP has no transport layer) and that is probably a pipe dream. Todays transport solution for loss mitigation are far more cost effective - and that's why iFCP is a better proposition as a transition technology than FCoE and iSCSI with gateways is propably better in the long run.=20 =09 >=20 > . > . > . > John L Hufferd > Sr. Executive Director of Technology > jhufferd@brocade.com > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > =20 > -----Original Message----- > From: John Hufferd=20 > Sent: Tuesday, April 24, 2007 12:57 PM > To: 'Julian_Satran@il.ibm.com' > Subject: Re: [Ips] Recent comments about FCoE and iSCSI >=20 > Julian, > I think you are wrong on this one. The arguments are quite different > then the ones we had in pre iSCSI days. (By the way I missed you on > today's Renato meeting/conf call where Brocade took the IBM technology > group through FCoE as it is being placed in our plans). =20 > I will send you more info when I get to my computer. But you probably > were sent the Brocade charts. Please review them and I will follow up > with more information. > This does NOT replace iSCSI it applies only to a DataCenter envuornment > with lossless DCE ethernet. > -------------------------- > John L. Hufferd > Sr. Ex. Director of Technology > Brocade Communications Systems, Inc. > Phone: (408) 333-5244 > Mobile: (408) 627-9606 > eMail: jhufferd@brocade.com > (Sent from my BlackBerry Wireless) > =20 >=20 > ----- Original Message ----- > From: Julian Satran > To: ips@ietf.org > Sent: Tue Apr 24 12:10:29 2007 > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the latest and > greatest reincarnation of Fiber Channel over ethernet.=20 > It made me try and summarize all the long and hot debates that preceded > the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate preceded iSCSI that > was not so - FCoE was considered even then and was dropped as a dumb > idea.=20 >=20 > Here is a summary (as afar as I can remember) of the main arguments. > They are not bad arguments even in retrospect and technically FCoE > doesn't look better than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect this group to > seriously expand my arguments and make them public - in personal or > collective form.=20 >=20 > And do not forget - it is a technical dispute - although we all must > have some doubts about the way it is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > ---------------------------------------------------------------------=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and Almaden) started > looking at connecting storage to servers using the "regular network" > (the ubiquitous LAN) we considered many alternatives (another team even > had a look at ATM - still a computer network candidate at the time). I > won't get you over all of our rationale (and we went over some of them > again at the end of 1999 with a team from CISCO before we convened the > first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the > rest) but some of the reasons we choose to drop Fiber Channel over raw > Ethernet where multiple:=20 >=20 >=20 > * Fiber Channel Protocol (SCSI over Fiber Channel Link) is > "mildly" effective because:=20 >=20 > * it implements endpoints in a dedicated engine (Offload) > * it has no transport layer (recovery is done at the > application layer under the assumption that the error rate will be very > low)=20 > * the network is limited in physical span and logical span > (number of switches)=20 > * flow-control/congestion control is achieved with a > mechanism adequate for a limited span network (credits). The packet loss > rate is almost nil and that allows FCP to avoid using a transport > (end-to-end) layer >=20 > * FCP she switches are simple (addresses are local and the > memory requirements cam be limited through the credit mechanism)=20 > * However FCP endpoints are inherently costlier than > simple NICs - the cost argument (initiators are more expensive)=20 > * The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the network diameter > limits) - the scaling argument=20 > * The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s - the scaling argument=20 > * Ethernet has no credit mechanism and any mechanism with > a similar effect increases the end point cost. Building a transport > layer in the protocol stack has always been the preferred choice of the > networking community - the community argument=20 > * The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances in protocol stack > implementation and finer tuning of the congestion control mechanisms > make conventional TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant on the computing > scene have enough compute cycles available to make any "offloading" > possible as a mere code restructuring exercise (see the stack reports > from Intel, IBM etc.)=20 > * Building on a complete stack makes available a wealth of > operational and management mechanisms built over the years by the > networking community (routing, provisioning, security, service location > etc.) - the community argument=20 > * Higher level storage access over an IP network is widely > available and having both block and file served over the same connection > with the same support and management structure is compelling - the > community argument=20 > * Highly efficient networks are easy to build over IP with > optimal (shortest path) routing while Layer 2 networks use bridging and > are limited by the logical tree structure that bridges must follow. The > effort to combine routers and bridges (rbridges) is promising to change > that but it will take some time to finalize (and we don't know exactly > how it will operate). Untill then the scale of Layer 2 network is going > to seriously limited - the scaling argument >=20 >=20 >=20 > As a side argument - a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform > better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That > was what convinced us to take the path that lead to iSCSI - and we used > plain vanilla x86 servers with plain-vanilla NICs and Linux (with > similar measurements conducted on Windows).=20 > The networking and storage community acknowledged those > arguments and developed iSCSI and the companion protocols for service > discovery, boot etc.=20 > =20 > The community also acknowledged the need to support existing > infrastructure and extend it in a reasonable fashion and developed 2 > protocols iFCP (to support hosts with FCP drivers and IP connections to > connect to storage by a simple conversion from FCP to TCP packets) FCPIP > to extend the reach of FCP through IP (connects FCP islands through TCP > links). Both have been=20 > implemented and their foundation is solid.=20 > =20 > The current attempt of developing a "new-age" FCP over an > Ethernet link is going against most of the arguments that have given us > iSCSI etc.=20 > =20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus limits scaling, > mandates elements at the link layer and application layer that make > applications more expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet).=20 > =20 > In some related effort (and at a point also when developing > iSCSI) we considered also moving away from SCSI (like some "no > standardized" but popular in some circles software did - e.g., NBP) but > decided against. SCSI is a mature and well understood access > architecture for block storage and is implemented by many device > vendors. Moving away from it would not have been justified at the time.=20 > =20 =09 ________________________________ _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 =09 ________________________________ _______________________________________________ Ips mailing list Ips@ietf.org =09 https://www1.ietf.org/mailman/listinfo/ips______________________________ _________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips =09 =09 ________________________________ =09 _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips =09 ------_=_NextPart_001_01C7878E.0B122647 Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable
The standardization process is just beginning = on=20 FCoE.  Brocade and
at least two other=20 organizations have done significant work on the
subject.  I expect content similar = to the Cisco=20 patent to be
a significant contribution to the FCoE = standard.  As you=20 know,
standards development=20 often involves negotiation of details=20 of
a=20 technology.
 
Bob

From: Eddy Quicksall=20 [mailto:Quicksall_iSCSI@Bellsouth.net]
Sent: Wednesday, April = 25,=20 2007 2:41 PM
To: Julian Satran
Cc: ips@ietf.org; = John=20 Hufferd
Subject: Re: FW: [Ips] Recent comments about FCoE and=20 iSCSI

I notice that Cisco has a patent on Fibre Channel = over=20 Ethernet. Is FCoE something different?
----- Original Message -----
From:=20 Julian Satran
To: Eddy Quicksall =
Sent: Wednesday, April 25, 2007 = 11:52=20 AM
Subject: Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI


Sorry Eddy by = sizable I meant=20 even at the size of a modern data center. Julo


"Eddy = Quicksall" <Quicksall_iSCSI@Bellsouth.n= et>=20

25/04/07 11:08

To
Julian Satran/Haifa/IBM@IBMIL =
cc
ips@ietf.org, John = Hufferd <jhufferd@Brocade.COM> =
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and iSCSI

=




<= FONT=20 size=3D2>I basically said that in the summery line by saying = "it will not route on the “global” = scale like TCP/IP=20 would".
----- Original Message ----- =
From: Julian Satran=20
To: Eddy=20 Quicksall
Cc:=20 ips@ietf.org ; John=20 Hufferd
Sent:=20 Wednesday, April 25, 2007 11:04 AM
Subject: Re:=20 FW: [Ips] Recent comments about FCoE and iSCSI


Eddy, =

That is oversimplified and ignore the = drop rate=20 assumption and error rate assumptions made in FCP(FCP has no transport = layer).=20 To get to it on a sizable network requires more than = PAUSE.

Julo=20

"Eddy = Quicksall"=20 <Quicksall_iSCSI@Bellsouth.net>

25/04/07 10:07=20


To
"John = Hufferd"=20 <jhufferd@Brocade.COM>, Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
Re: FW: [Ips] Recent = comments=20 about FCoE and = iSCSI






Basically, it=20 is sending FC frames over Ethernet. This localizes the traffic unless = you=20 route based on MAC addresses. So you send 2146 bytes of FC frame plus = 18 bytes=20 of Ethernet overhead as FCoE “standard” packet. 18 bytes = of Ethernet gets=20 stripped and you have straight FC frame that can go through any FC = network.=20 Now you can have 10G Ethernet pipes into existing FC SANs. Limited = market=20 potential as far as I can see. The key argument is it much easier to = implement=20 than iSCSI and also has less overhead and uses all the benefits of FC. = End to=20 End credits are simulated using PAUSE command on Ethernet and MAC = addresses=20 are mapped into WWNs.

Biggest knock is that = it will not=20 route on the “global” scale like TCP/IP would.

Eddy

 -----=20 Original Message -----

From: Julian=20 Satran
To: John=20 Hufferd
Cc: ips@ietf.org =
Sent: Wednesday,=20 April 25, 2007 8:09 AM
Subject: Re: FW: [Ips] Recent = comments about=20 FCoE and iSCSI



"John = Hufferd"=20 <jhufferd@Brocade.COM> wrote=20 on 25/04/2007 02:45:51:

> Julian,
> To be sure you = understand=20 our position; Brocade is pushing iSCSI as an
> outreach protocol = from=20 the Data Center. We also believe iSCSI is very
> useful for=20 installations that do not have a Fibre Channel
> infrastructure, = and in=20 that case we will be able to sell them our new
> iSCSI and TOE = offload=20 HBAs.  
>
> When I say iSCSI is an outreach = protocol, this=20 is a statement that iSCSI
> is very important to connect = "stranded"=20 servers to the Fibre Channel
> Fabric.  That is, we sell=20 iSCSI-to-FC Gateway devices which will permit
> iSCSI Servers = (software=20 or HBA iSCSI initiators) to connect to the
> Enterprise "Bet = Your=20 Business" FC Storage.  This of course also applies
> to = Desktops=20 and Laptop systems, and systems at distance.
>=20

You make it sound = like:
1.     =  =20  most of the servers in the world have = their=20 storage on the network - and that is not the case=20
2.      =20  FCP is basically better performing = than iSCSI -=20 and that is not true either
3.       =  Gatewaying is expensive - and it is perhaps so but only if = you are=20 completely relying on FCP storage (and there are plenty of good iSCSI = vendors=20 of storage)and pushing the price on the servers is not cheap either - = at least=20 not for the server buyer

> Now with that positioning, it is important to = understand=20 the limitation
> to this strategy.  The primary problem is = that=20 iSCSI to FC Bridging
> (Gatewaying) is relatively expensive = (compared to=20 simple FC
> connections).  Though we have some of the best = priced=20 Gateways on the
> market, it is not cost feasible to replace all = the=20 server connectivity
> to FC storage with iSCSI for the hundred = to=20 thousands of servers in the
> Data Center.  And since, if = there is=20 to be a consolidated Network
> connection to the servers in the = Data=20 Center, there must be an
> evolutionary replacement of Server=20 Connections to Storage.  That means
> there must be a=20 bridge/Gateway approach.  And as I mentioned before,
> = there is=20 just too much cost in the iSCSI to FC Gateway.
>
> The = issue is=20 the server requirement to have a single connection type to
> = handle=20 cluster messaging, general messaging, and storage.  iSCSI = is
>=20 clearly an option for the storage, however, the gateway costs are = too
>=20 high for iSCSI to be used as the "normal" server connect into a FC=20 based
> Fabric. That is true for the current 1GE; and for 10GE = the cost=20 is just
> out of sight.  The reason for this is the = requirement for=20 TCP/IP
> termination and re-initiation with FC at the = Gateway.
>=20
> Now with respect to FC over Ethernet the important thing to=20 understand
> is that it is not Ethernet as we have known it up = to know.=20  The Ethernet
> we are talking about is a type of Ethernet = that can=20 only be deployed in
> a constrained environment such as a Data = Center.=20  This form of Ethernet
> is called DCE (Data Center = Ethernet) or=20 CEE (Convergence Enhanced
> Ethernet).  This form of = Ethernet is a=20 Loss-less type Ethernet, with
> multi-priority and Flow Control. =  This is NOT an Internet or Intranet
> type of = Ethernet.
>=20
> FCoE is all about using the DCE (CEE) to carry FC frames. =  The=20 rest of
> the Host and storage stack remain the same, the = functions and=20 features
> of the switches also remain the same and add the = capability=20 to provide
> Cluster Message Switching which has latency close = to=20 InfiniBand speeds.
>
>
> Because the FC frames are = transported to the switches intact via a DCE
> frame, the = Bridging, if=20 you want to call it that, is virtually non
> existent. =  Hence you=20 can deliver the FC frames to FC devices, or send FC
> frames to = DCE FCoE=20 devices, just like one would do if it was all FC. And
> all this = is done=20 while performing Cluster message switching and general
> message = trucking to the IP outfacing network.
>

The rosy future = of the=20 yet to appear DCE/CEE and a layer 2 only world.=20
First you have some terms=20 confused:

Bridging is=20 the term commonly used for Layer-2 switching and routing is therm used = for=20 layer-3 (switching).

Bridging has some advantages (less management) that = have=20 created a movement towards an enterprise wide=20
LAN. But this has a long way to go and = will=20 require significant equipment and protocol changes.=20
Even its proponents do not call for = transportless=20 networks, lossless networks etc. =
The second trouble with your argument is that there are = no known=20 large scale networking technologies that=20
really work at full speed (high speed) = and are=20 lossless (flow-controlled) and errorless like FCoE = assumes.
The TCP/IP has solved this = issue for every=20 generation using the proven end-to-end principle (and is doing so=20 now).
And it = is not by=20 chance so and that is why all networking applications are built above = layer-3=20 and not dropping layer-3 (like FCoE) does.=20

Although I can understand the DCE = arguments as=20 a management statement I would prefer like any rational engineer, to = base my=20 building blocks on structures that are proven and long lasting. And = those are=20 still the end-to-end TCP/IP that can accommodate even your FCP = addicts. The=20 IPS TWG has developed the iFCP that does exactly what FCoE claims to = do an a=20 better base.

>=20 This means an evolutionary process is possible to the solution = of
>=20 getting a single Fabric connection for all networks connected to = a
>=20 server, further, the process has very low interconnection cost on = the
>=20 Data Center Fabric. And it maintains all the FC Fabric Services, and=20 all
> the same Storage Management processes.
>
> = By the=20 way, this is primarily a Server driven value statement, there
> = seems to=20 be little value in having FCoE on the storage controller.
> = Therefore FC=20 storage controllers (and FICON) will be the very last
> things = that=20 connect using FCoE and that evolution will take at least a
> = decade or=20 more.
>

It is server cost statement. It costs nothing to = connect=20 a modern server to ethernet it will cost a bundle to connect to FCoE = and it=20 will force users in short lived bad solutions.=20


> We see value in offering = switches and=20 Directors that can support DCE
> switching, FC switching as well = as=20 iSCSI interconnect, and the
> "Trunking" of general messaging to = the=20 Outfacing IP network.  That said;
> we do not see FCoE = going beyond=20 the constraints of the Data Center. =

Data Centers now grow to tens of thousands of nodes. = There is=20 no layer-2 technology for errorless/lossless operation at this scale = and there=20 is no good reason to pursue one. The only possible reason (good = reason) is the=20 bridging infrastructure but that infrastructure has a completely = different=20 rationale than the flowcontrol. =

> This issue and message is quite different from = the issues=20 and messages
> we struggled with when we started iSCSI. =  There is a=20 consortium of folks
> both working on the DCE (CEE) and the = FCoE.=20  Without the DCE the FCoE
> will not happen.  
> =
> None of the above cancels out the value of iSCSI in = numerous
>=20 environments.
>
>

iSCSI is good for all = environments.=20 Business consideration (and some politics) keep it form "exploding" = and large=20 storage vendors are completely indifferent to the network connection = they are=20 using.
You and I have also slightly different views of DCE. I = expect DCE=20 (that still has a way to go) to improve the QoS in the data-center = (and for=20 storage too). You expect it to bring the loss rates down to the levels = that=20 FCP assumes (FCP has no transport layer) and that is probably a pipe = dream.=20 Todays transport solution for loss mitigation are far more cost = effective -=20 and that's why iFCP is a better proposition as a transition technology = than=20 FCoE and iSCSI with gateways is propably better in the long=20 run.

> =
>=20 .
> .
> .
> John L Hufferd
> Sr. Executive = Director of=20 Technology
> jhufferd@brocade.com
> Office Phone: (408) = 333-5244;=20 eFAX: (408) 904-4688
> Alt Office Phone: (408) 997-6136; Cell: = (408)=20 627-9606
>  
> -----Original Message-----
> = From: John=20 Hufferd
> Sent: Tuesday, April 24, 2007 12:57 PM
> To:=20 'Julian_Satran@il.ibm.com'
> Subject: Re: [Ips] Recent comments = about=20 FCoE and iSCSI
>
> Julian,
> I think you are wrong = on this=20 one.  The arguments are quite different
> then the ones we = had in=20 pre iSCSI days.  (By the way I missed you on
> today's = Renato=20 meeting/conf call where Brocade took the IBM technology
> group = through=20 FCoE as it is being placed in our plans).  
> I will send = you more=20 info when I get to my computer.  But you probably
> were = sent the=20 Brocade charts.  Please review them and I will follow up
> = with=20 more information.
> This does NOT replace iSCSI it applies only = to a=20 DataCenter envuornment
> with lossless DCE ethernet.
>=20 --------------------------
> John L. Hufferd
> Sr. Ex. = Director of=20 Technology
> Brocade Communications Systems, Inc.
> Phone: = (408)=20 333-5244
> Mobile: (408) 627-9606
> eMail:=20 jhufferd@brocade.com
> (Sent from my BlackBerry = Wireless)
>=20  
>
> ----- Original Message -----
> From: = Julian=20 Satran <Julian_Satran@il.ibm.com>
> To: ips@ietf.org=20 <ips@ietf.org>
> Sent: Tue Apr 24 12:10:29 2007
> = Subject:=20 [Ips] Recent comments about FCoE and iSCSI
>
>
> = Dear All,=20
>
> The trade press is lately full with comments about = the=20 latest and
> greatest reincarnation of Fiber Channel over = ethernet.=20
> It made me try and summarize all the long and hot debates = that=20 preceded
> the advent of iSCSI.
> Although FCoE = proponents make=20 it look like no debate preceded iSCSI that
> was not so - FCoE = was=20 considered even then and was dropped as a dumb
> idea.
> =
>=20 Here is a summary (as afar as I can remember) of the main = arguments.
>=20 They are not bad arguments even in retrospect and technically = FCoE
>=20 doesn't look better than it did then.
>
> Feel free to = use this=20 material in a nay form. I expect this group to
> seriously =  expand=20 my arguments and make them public - in personal or
> collective = form.=20
>
> And do not forget - it is a technical dispute - = although we=20 all must
> have some doubts about the way it is pursued. =
>=20
> Regards,
> Julo
>
>=20 --------------------------------------------------------------------- =
>=20
> What a piece of nostalgia :-)
>
> Around 1997 = when a=20 team at IBM Research (Haifa and Almaden) started
> looking at = connecting=20 storage to servers using the "regular network"
> (the ubiquitous = LAN) we=20 considered many alternatives (another team even
> had a look at = ATM -=20 still a computer network candidate at the time). I
> won't get = you over=20 all of our rationale (and we went over some of them
> again at = the end=20 of 1999 with a team from CISCO before we convened the
> first = IETF BOF=20 in 2000 at Adelaide that resulted in iSCSI and all the
> rest) = but some=20 of the reasons we choose to drop Fiber Channel over raw
> = Ethernet where=20 multiple:
>
>
> *   Fiber Channel Protocol = (SCSI=20 over Fiber Channel Link) is
> "mildly" effective because: =
>=20
>    *   it implements endpoints in a dedicated = engine=20 (Offload)
>    *   it has no transport layer = (recovery=20 is done at the
> application layer under the assumption that the = error=20 rate will be very
> low)
>    *   the = network is=20 limited in physical span and logical span
> (number of switches) =
>    *   flow-control/congestion control is = achieved=20 with a
> mechanism adequate for a limited span network = (credits). The=20 packet loss
> rate is almost nil and that allows FCP to avoid = using a=20 transport
> (end-to-end) layer
>
>    * =  =20 FCP she switches are simple (addresses are local and the
> = memory=20 requirements cam be limited through the credit mechanism)
> =  =20  *   However FCP endpoints are inherently costlier = than
>=20 simple NICs - the cost argument (initiators are more expensive) =
>=20    *   The credit mechanisms is highly unstable for=20 large
> networks (check switch vendors planning docs for the = network=20 diameter
> limits) - the scaling argument
>   =  *  =20 The assumption of low losses due to errors might
> radically = change when=20 moving from 1 to 10 Gb/s - the scaling argument
>   =  *  =20 Ethernet has no credit mechanism and any mechanism with
> a = similar=20 effect increases the end point cost. Building a transport
> = layer in the=20 protocol stack has always been the preferred choice of the
> = networking=20 community - the community argument
>    *   The=20 "performance penalty" of a complete protocol stack
> has always = been=20 overstated (and overrated). Advances in protocol stack
> = implementation=20 and finer tuning of the congestion control mechanisms
> make=20 conventional TCP/IP performing well even at 10 Gb/s and over.
> = Moreover=20 the multicore processors that become dominant on the computing
> = scene=20 have enough compute cycles available to make any "offloading"
> = possible=20 as a mere code restructuring exercise (see the stack reports
> = from=20 Intel, IBM etc.)
>    *   Building on a complete = stack=20 makes available a wealth of
> operational and management = mechanisms=20 built over the years by the
> networking community (routing,=20 provisioning, security, service location
> etc.) - the community = argument
>    *   Higher level storage access = over an IP=20 network is widely
> available and having both block and file = served over=20 the same connection
> with the same support and management = structure is=20 compelling - the
> community argument
>    * =  =20 Highly efficient networks are easy to build over IP with
> = optimal=20 (shortest path) routing while Layer 2 networks use bridging = and
> are=20 limited by the logical tree structure that bridges must follow. = The
>=20 effort to combine routers and bridges (rbridges) is promising to=20 change
> that but it will take some time to finalize (and we = don't know=20 exactly
> how it will operate). Untill then the scale of Layer 2 = network=20 is going
> to seriously limited - the scaling argument
> =
>=20
>
>       As a side argument - a = performance=20 comparison made in
> 1998 showed SCSI over TCP (a predecessor of = the=20 later iSCSI) to perform
> better than FCP at 1Gbs for block = sizes=20 typical for OLTP (4-8KB). That
> was what convinced us to take = the path=20 that lead to iSCSI - and we used
> plain vanilla x86 servers = with=20 plain-vanilla NICs and Linux (with
> similar measurements = conducted on=20 Windows).
>    The networking and storage community=20 acknowledged those
> arguments and developed iSCSI and the = companion=20 protocols for service
> discovery, boot etc.
>  =20  
>    The community also acknowledged the need = to=20 support existing
> infrastructure and extend it in a reasonable = fashion=20 and developed 2
> protocols iFCP (to support hosts with FCP = drivers and=20 IP connections to
> connect to storage by a simple conversion = from FCP=20 to TCP packets) FCPIP
> to extend the reach of FCP through IP = (connects=20 FCP islands through TCP
> links). Both have been
>   =  implemented and their foundation is solid.
>  =20  
>    The current attempt of developing a = "new-age" FCP=20 over an
> Ethernet link is going against most of the arguments = that have=20 given us
> iSCSI etc.
>    
>   =  It=20 ignores the networking layering practice, build an
> application = protocol directly above a link and thus limits scaling,
> = mandates=20 elements at the link layer and application layer that make
>=20 applications more expensive and leaves aside the whole "ecosystem"=20 that
> accompanies TCP/IP (and not Ethernet).
>  =20  
>    In some related effort (and at a point = also when=20 developing
> iSCSI) we considered also moving away from SCSI = (like some=20 "no
> standardized" but popular in some circles software did - = e.g.,=20 NBP) but
> decided against. SCSI is a mature and well understood = access
> architecture for block storage and is implemented by = many=20 device
> vendors. Moving away from it would not have been = justified at=20 the time.
>    

_______________________________________________
Ips mailing=20 list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips =

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= _______________________________________________
Ips = mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing=20 = list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
------_=_NextPart_001_01C7878E.0B122647-- --===============0090503334== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0090503334==-- From ips-bounces@ietf.org Thu Apr 26 13:29:44 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh7mn-0002md-FC; Thu, 26 Apr 2007 13:29:41 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh7mm-0002mX-2h for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 13:29:40 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh7ml-0002mP-Pk for ips@ietf.org; Thu, 26 Apr 2007 13:29:39 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh7k7-0000uN-R9 for ips@ietf.org; Thu, 26 Apr 2007 13:26:55 -0400 Received: from nuova-ex1.nuovasystems.com ([67.91.200.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh7k4-0004Un-J7 for ips@ietf.org; Thu, 26 Apr 2007 13:26:55 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 10:26:44 -0700 Message-ID: <34BDD2A93E5FD84594BB230DD6C18EA2015E3A69@nuova-ex1.hq.nuovaimpresa.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Re: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIKA+jT4yyme8oRZqoriUVAWdrDQ== From: "Silvano Gai" To: X-Spam-Score: 0.0 (/) X-Scan-Signature: de4f315c9369b71d7dd5909b42224370 X-TMDA-Confirmed: Thu, 26 Apr 2007 13:29:39 -0400 X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote:=20 > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 14:25:27 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh8ek-00029J-K9; Thu, 26 Apr 2007 14:25:26 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh8ei-0001i4-Gx for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 14:25:24 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh8eh-0001b0-UK for ips@ietf.org; Thu, 26 Apr 2007 14:25:23 -0400 Received: from mtagate8.de.ibm.com ([195.212.29.157]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh8eg-0002Z6-EK for ips@ietf.org; Thu, 26 Apr 2007 14:25:23 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate8.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3QIPLDt152796 for ; Thu, 26 Apr 2007 18:25:21 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3QIPLx53641534 for ; Thu, 26 Apr 2007 20:25:21 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3QIPL2B010819 for ; Thu, 26 Apr 2007 20:25:21 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3QIPL52010816; Thu, 26 Apr 2007 20:25:21 +0200 In-Reply-To: <34BDD2A93E5FD84594BB230DD6C18EA2015E3A69@nuova-ex1.hq.nuovaimpresa.com> To: "Silvano Gai" MIME-Version: 1.0 Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Thu, 26 Apr 2007 14:25:20 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 26/04/2007 21:25:20, Serialize complete at 26/04/2007 21:25:20 X-Spam-Score: 0.1 (/) X-Scan-Signature: 093efd19b5f651b2707595638f6c4003 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1278259232==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============1278259232== Content-Type: multipart/alternative; boundary="=_alternative 006530B9852572C9_=" This is a multipart message in MIME format. --=_alternative 006530B9852572C9_= Content-Type: text/plain; charset="US-ASCII" Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric. Julo "Silvano Gai" 26/04/07 13:26 To cc Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote: > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 006530B9852572C9_= Content-Type: text/html; charset="US-ASCII"
Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric.

Julo




"Silvano Gai" <sgai@nuovasystems.com>

26/04/07 13:26

To
<ips@ietf.org>
cc
Subject
Re: FW: [Ips] Recent comments about FCoE and iSCSI





    * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI
   * From: "Eddy Quicksall" <Quicksall_iSCSI at Bellsouth.net>
   * Date: Wed, 25 Apr 2007 15:15:22 -0400

Eddy Wrote:

> Further, I think PAUSE is only for point-to-point and I think that is
too > restrictive.

Eddy,

In Fibre Channel credits are only point-to-point and FC networks work
well for datacenter environments.

Where is the difference?

-- Silvano



_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=_alternative 006530B9852572C9_=-- --===============1278259232== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1278259232==-- From ips-bounces@ietf.org Thu Apr 26 14:34:02 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh8n2-0002tS-0j; Thu, 26 Apr 2007 14:34:00 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh8n1-0002tM-18 for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 14:33:59 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh8n0-0002tE-NR for ips@ietf.org; Thu, 26 Apr 2007 14:33:58 -0400 Received: from nuova-ex1.nuovasystems.com ([67.91.200.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh8mz-0005eB-4N for ips@ietf.org; Thu, 26 Apr 2007 14:33:58 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 11:33:48 -0700 Message-ID: <34BDD2A93E5FD84594BB230DD6C18EA2015E3AEF@nuova-ex1.hq.nuovaimpresa.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIMEpMJMRUD9JHRGuC7HDSZ7Yq4QAAJ0vQ References: <34BDD2A93E5FD84594BB230DD6C18EA2015E3A69@nuova-ex1.hq.nuovaimpresa.com> From: "Silvano Gai" To: "Julian Satran" X-Spam-Score: 0.0 (/) X-Scan-Signature: 4ec3642ae9025e273a4a263d640f3300 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0070640375==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0070640375== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C78831.7BE3CB0B" This is a multi-part message in MIME format. ------_=_NextPart_001_01C78831.7BE3CB0B Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Julo, =20 I am not sure what you are asking for. Are you asking for a reference to a 1,000+ port SAN? Today there are multiple FC switches on the market with 256 ports. With six switches it is easy to realize a 1,000+ port SAN? =20 Are you trying to tell me that 6 FC switches cannot be connected together? =20 -- Silvano =20 ________________________________ From: Julian Satran [mailto:Julian_Satran@il.ibm.com]=20 Sent: Thursday, April 26, 2007 11:25 AM To: Silvano Gai Cc: ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric.=20 Julo=20 "Silvano Gai" =20 26/04/07 13:26=20 To =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 =20 * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote:=20 > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips ------_=_NextPart_001_01C78831.7BE3CB0B Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Julo,

 

I am not sure what you are asking = for.

Are you asking for a reference to a = 1,000+ port SAN?

Today there are multiple FC = switches on the market with 256 ports.

With six switches it is easy to = realize a 1,000+ port SAN?

 

Are you trying to tell me that 6 FC switches cannot be connected together?

 

-- = Silvano

 

From: = Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, = 2007 11:25 AM
To: Silvano Gai
Cc: ips@ietf.org
Subject: Re: FW: [Ips] = Recent comments about FCoE and iSCSI

 


Please send this community a reference of a = 1000+ node datacenter using FCP as a single fabric.

Julo



"Silvano Gai" <sgai@nuovasystems.com> =

26/04/07 13:26

To

<ips@ietf.org> =

cc

 

Subject

Re: FW: [Ips] Recent comments about = FCoE and iSCSI

 

 

 




    * Subject: Re: FW: [Ips] Recent comments about FCoE and = iSCSI
   * From: "Eddy = Quicksall" <Quicksall_iSCSI at Bellsouth.net>
   * Date: Wed, 25 Apr 2007 = 15:15:22 -0400

Eddy Wrote:

> Further, I think PAUSE is only for point-to-point and I think that is
too > restrictive.

Eddy,

In Fibre Channel credits are only = point-to-point and FC networks work
well for datacenter = environments.

Where is the difference?

-- Silvano



_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips=

------_=_NextPart_001_01C78831.7BE3CB0B-- --===============0070640375== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0070640375==-- From ips-bounces@ietf.org Thu Apr 26 14:49:46 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh92H-0003pn-Qd; Thu, 26 Apr 2007 14:49:45 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh92G-0003nT-5M for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 14:49:44 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh92F-0003mw-Rb for ips@ietf.org; Thu, 26 Apr 2007 14:49:43 -0400 Received: from mexforward.lss.emc.com ([128.222.32.20]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh92D-0000Wq-VN for ips@ietf.org; Thu, 26 Apr 2007 14:49:43 -0400 Received: from mailhub.lss.emc.com (sesha.lss.emc.com [10.254.144.12]) by mexforward.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3QInfqA002836 for ; Thu, 26 Apr 2007 14:49:41 -0400 (EDT) Received: from corpussmtp1.corp.emc.com (corpussmtp1.corp.emc.com [128.221.10.43]) by mailhub.lss.emc.com (Switch-3.2.5/Switch-3.1.7) with ESMTP id l3QImc0p018793 for ; Thu, 26 Apr 2007 14:49:40 -0400 (EDT) From: brown_David1@emc.com Received: from CORPUSMX30B.corp.emc.com ([10.254.64.50]) by corpussmtp1.corp.emc.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 26 Apr 2007 14:48:34 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 14:48:33 -0400 Message-ID: <75852864BAD9684FBF5DCF4289DE4076078FFA97@CORPUSMX30B.corp.emc.com> In-Reply-To: <39BA3BC178B4394DB184389E88A97F8C022C5CC4@hq-exch-1.corp.brocade.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHbhab+EXeFVfgQWiH7jzYWydYaAAKX7IQAAKlUiAABWKvIAAeuqUA References: <368FBF3D8437A748BA8222526BF9309901ACBF42@aime2k302.adaptec.com> <39BA3BC178B4394DB184389E88A97F8C022C5CC4@hq-exch-1.corp.brocade.com> To: X-OriginalArrivalTime: 26 Apr 2007 18:48:34.0568 (UTC) FILETIME=[7E385880:01C78833] X-PMX-Version: 4.7.1.128075, Antispam-Engine: 2.5.1.298604, Antispam-Data: 2007.4.26.112434 X-PerlMx-Spam: Gauge=, SPAM=1%, Reason='EMC_FROM_0+ -3, NO_REAL_NAME 0, __C230066_P5 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __FRAUD_419_VIPS 0, __HAS_MSGID 0, __IMS_MSGID 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0' X-Spam-Score: 0.5 (/) X-Scan-Signature: 25620135586de10c627e3628c432b04a X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org How would an FCoE-based initiator communicate with iSCSI-based storage?=20 In other words, can the same host adapter be used for FCoE connections and iSCSI sessions, and if so, would the performance be similar? thanks, dj ________________________________ From: John Hufferd [mailto:jhufferd@Brocade.COM]=20 Sent: Thursday, April 26, 2007 12:52 AM To: Sandars, Ken; Eddy Quicksall; Julian Satran Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =09 =09 Ken, The term FCoE has as its primary component FC. Consider the possibility that the DCE Link from the Host connects to a switch/device that is able to deal with the FC part of the FCoE. =20 . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com =20 Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =09 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Wednesday, April 25, 2007 6:41 PM To: John Hufferd; Eddy Quicksall; Julian Satran Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =20 Hey John, =20 [Hufferd] Many servers are asking for an evolutionary way to combine their Networking connections from the Server. The customers I have dealt with do NOT want to rip out FC, they want to provide a single Link for transport of all networking needs, including storage, exiting their servers. =09 I'm not sure what you are saying here. Are you saying servers should have a single type of physical network connection, presumably ethernet? How does that align with not wanting to rip out FC? =20 Thanks Ken _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 15:34:55 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9jx-0008ND-ME; Thu, 26 Apr 2007 15:34:53 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh9jw-0008N2-5Z for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 15:34:52 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9jv-0008Mt-QV for ips@ietf.org; Thu, 26 Apr 2007 15:34:51 -0400 Received: from mtagate6.de.ibm.com ([195.212.29.155]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Hh9ju-0001qG-0c for ips@ietf.org; Thu, 26 Apr 2007 15:34:51 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate6.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3QJYjbZ139424 for ; Thu, 26 Apr 2007 19:34:45 GMT Received: from d12av04.megacenter.de.ibm.com (d12av04.megacenter.de.ibm.com [9.149.165.229]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3QJYju53862560 for ; Thu, 26 Apr 2007 21:34:45 +0200 Received: from d12av04.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3QJYjlS003759 for ; Thu, 26 Apr 2007 21:34:45 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3QJYik0003756; Thu, 26 Apr 2007 21:34:44 +0200 In-Reply-To: <34BDD2A93E5FD84594BB230DD6C18EA2015E3AEF@nuova-ex1.hq.nuovaimpresa.com> To: "Silvano Gai" MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Thu, 26 Apr 2007 15:34:43 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 26/04/2007 22:34:43, Serialize complete at 26/04/2007 22:34:43 X-Spam-Score: 0.0 (/) X-Scan-Signature: a4a24b484706be629f915bfb1a3e4771 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0272355167==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0272355167== Content-Type: multipart/alternative; boundary="=_alternative 006B837A852572C9_=" This is a multipart message in MIME format. --=_alternative 006B837A852572C9_= Content-Type: text/plain; charset="US-ASCII" My question was specific - and with a reason. Julo "Silvano Gai" 26/04/07 14:33 To Julian Satran/Haifa/IBM@IBMIL cc Subject RE: FW: [Ips] Recent comments about FCoE and iSCSI Julo, I am not sure what you are asking for. Are you asking for a reference to a 1,000+ port SAN? Today there are multiple FC switches on the market with 256 ports. With six switches it is easy to realize a 1,000+ port SAN? Are you trying to tell me that 6 FC switches cannot be connected together? -- Silvano From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Thursday, April 26, 2007 11:25 AM To: Silvano Gai Cc: ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric. Julo "Silvano Gai" 26/04/07 13:26 To cc Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote: > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 006B837A852572C9_= Content-Type: text/html; charset="US-ASCII"
My question was specific - and with a reason. Julo


"Silvano Gai" <sgai@nuovasystems.com>

26/04/07 14:33

To
Julian Satran/Haifa/IBM@IBMIL
cc
<ips@ietf.org>
Subject
RE: FW: [Ips] Recent comments about FCoE and iSCSI





Julo,
 
I am not sure what you are asking for.
Are you asking for a reference to a 1,000+ port SAN?
Today there are multiple FC switches on the market with 256 ports.
With six switches it is easy to realize a 1,000+ port SAN?
 
Are you trying to tell me that 6 FC switches cannot be connected together?
 
-- Silvano
 


From: Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, 2007 11:25 AM
To: Silvano Gai
Cc: ips@ietf.org
Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI
 

Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric.

Julo


"Silvano Gai" <sgai@nuovasystems.com>

26/04/07 13:26


To
<ips@ietf.org>
cc
 
Subject
Re: FW: [Ips] Recent comments about FCoE and iSCSI

 


   





   * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI
  * From: "Eddy Quicksall" <Quicksall_iSCSI at Bellsouth.net>
  * Date: Wed, 25 Apr 2007 15:15:22 -0400

Eddy Wrote:

> Further, I think PAUSE is only for point-to-point and I think that is
too > restrictive.

Eddy,

In Fibre Channel credits are only point-to-point and FC networks work
well for datacenter environments.

Where is the difference?

-- Silvano



_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=_alternative 006B837A852572C9_=-- --===============0272355167== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0272355167==-- From ips-bounces@ietf.org Thu Apr 26 15:36:21 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9lN-00015K-4z; Thu, 26 Apr 2007 15:36:21 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh9lM-00014f-8G for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 15:36:20 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9lL-00014V-Ty for ips@ietf.org; Thu, 26 Apr 2007 15:36:19 -0400 Received: from nuova-ex1.nuovasystems.com ([67.91.200.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh9lL-0001y1-61 for ips@ietf.org; Thu, 26 Apr 2007 15:36:19 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 12:36:09 -0700 Message-ID: <34BDD2A93E5FD84594BB230DD6C18EA2015E3B42@nuova-ex1.hq.nuovaimpresa.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIOfwFy9LorRa2RfeN6Oc/eZqEKwAABQGg References: <34BDD2A93E5FD84594BB230DD6C18EA2015E3AEF@nuova-ex1.hq.nuovaimpresa.com> From: "Silvano Gai" To: "Julian Satran" X-Spam-Score: 0.0 (/) X-Scan-Signature: 182294e3fdac3aef093c0503b87ed133 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1830958356==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============1830958356== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C7883A.3264F2B9" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7883A.3264F2B9 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable ... and your question was? =20 -- Silvano =20 ________________________________ From: Julian Satran [mailto:Julian_Satran@il.ibm.com]=20 Sent: Thursday, April 26, 2007 12:35 PM To: Silvano Gai Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =20 My question was specific - and with a reason. Julo=20 "Silvano Gai" =20 26/04/07 14:33=20 To Julian Satran/Haifa/IBM@IBMIL=20 cc =20 Subject RE: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 =20 Julo,=20 =20 I am not sure what you are asking for.=20 Are you asking for a reference to a 1,000+ port SAN?=20 Today there are multiple FC switches on the market with 256 ports.=20 With six switches it is easy to realize a 1,000+ port SAN?=20 =20 Are you trying to tell me that 6 FC switches cannot be connected together?=20 =20 -- Silvano=20 =20 =20 ________________________________ From: Julian Satran [mailto:Julian_Satran@il.ibm.com]=20 Sent: Thursday, April 26, 2007 11:25 AM To: Silvano Gai Cc: ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 =20 Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric.=20 Julo=20 "Silvano Gai" =20 26/04/07 13:26=20 =20 To =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 =20 =20 * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote:=20 > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 ------_=_NextPart_001_01C7883A.3264F2B9 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

… and your question = was?

 

-- = Silvano

 

From: = Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, = 2007 12:35 PM
To: Silvano Gai
Cc: ips@ietf.org
Subject: RE: FW: [Ips] = Recent comments about FCoE and iSCSI

 


My question was specific - and with a reason. = Julo

"Silvano Gai" <sgai@nuovasystems.com> =

26/04/07 14:33

To

Julian = Satran/Haifa/IBM@IBMIL

cc

<ips@ietf.org> =

Subject

RE: FW: [Ips] Recent comments about = FCoE and iSCSI

 

 

 




Julo,
 
I am not sure what you are asking for. =
Are you asking for a reference to a 1,000+ port = SAN?
Today there are multiple FC switches on the market = with 256 ports.
With six switches it is easy to realize a 1,000+ port = SAN?
 
Are you trying to tell me that 6 FC switches cannot be connected together?
 
-- Silvano
 

 


From: = Julian Satran [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, 2007 11:25 AM
To: Silvano Gai
Cc: ips@ietf.org
Subject: Re: FW: [Ips] Recent comments about FCoE and = iSCSI
 

Please send this community a reference of a 1000+ node datacenter using = FCP as a single fabric.

Julo

"Silvano Gai" <sgai@nuovasystems.com> =

26/04/07 13:26

 

To

<ips@ietf.org> =

cc

 

Subject

Re: FW: [Ips] Recent comments about = FCoE and iSCSI


 

 

 

 





   * Subject: Re: FW: [Ips] Recent comments about FCoE and = iSCSI
  * From: "Eddy Quicksall" <Quicksall_iSCSI at Bellsouth.net>
  * Date: Wed, 25 Apr 2007 15:15:22 -0400

Eddy Wrote:

> Further, I think PAUSE is only for point-to-point and I think that = is
too > restrictive.

Eddy,

In Fibre Channel credits are only point-to-point and FC networks = work
well for datacenter environments.

Where is the difference?

-- Silvano



_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

------_=_NextPart_001_01C7883A.3264F2B9-- --===============1830958356== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1830958356==-- From ips-bounces@ietf.org Thu Apr 26 15:36:55 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9lv-0001lF-RP; Thu, 26 Apr 2007 15:36:55 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh9lu-0001kD-FK for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 15:36:54 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh9lu-0001k5-5T for ips@ietf.org; Thu, 26 Apr 2007 15:36:54 -0400 Received: from sj-iport-6.cisco.com ([171.71.176.117]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh9lr-000253-H0 for ips@ietf.org; Thu, 26 Apr 2007 15:36:54 -0400 Received: from sj-dkim-2.cisco.com ([171.71.179.186]) by sj-iport-6.cisco.com with ESMTP; 26 Apr 2007 12:36:51 -0700 X-IronPort-AV: i="4.14,457,1170662400"; d="scan'208,217"; a="141047936:sNHT94768938" Received: from sj-core-2.cisco.com (sj-core-2.cisco.com [171.71.177.254]) by sj-dkim-2.cisco.com (8.12.11/8.12.11) with ESMTP id l3QJapSi004120; Thu, 26 Apr 2007 12:36:51 -0700 Received: from xbh-sjc-221.amer.cisco.com (xbh-sjc-221.cisco.com [128.107.191.63]) by sj-core-2.cisco.com (8.12.10/8.12.6) with ESMTP id l3QJamZX021086; Thu, 26 Apr 2007 19:36:50 GMT Received: from xmb-sjc-21b.amer.cisco.com ([171.70.151.143]) by xbh-sjc-221.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 26 Apr 2007 12:36:49 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 12:36:48 -0700 Message-ID: In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIOhN8jlXqLfEWQS+1OrUCFl8P5QAABMFA From: "Frank D'Agostino \(fdagosti\)" To: "Julian Satran" , "Silvano Gai" X-OriginalArrivalTime: 26 Apr 2007 19:36:49.0273 (UTC) FILETIME=[3B994690:01C7883A] DKIM-Signature: v=0.5; a=rsa-sha256; q=dns/txt; l=8760; t=1177616211; x=1178480211; c=relaxed/simple; s=sjdkim2002; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=fdagosti@cisco.com; z=From:=20=22Frank=20D'Agostino=20\(fdagosti\)=22=20 |Subject:=20RE=3A=20FW=3A=20[Ips]=20Recent=20comments=20about=20FCoE=20an d=20iSCSI |Sender:=20; bh=7JUpNFF4Ww/X0TBqqqjsV87aPrOEsnTw2hLYJ+wJxlU=; b=AbTpEOCaQ1VE5aOSKhIVlYtgr61MNXLbpMYUeF/K3BMSej9oKKUTlbGsQx2lDuRiMU05YMm7 XA+nGD9I/HG49myHBI0W7b/OKfyQsT1QfABKOUhtahaS6L03rqKHXNXw; Authentication-Results: sj-dkim-2; header.From=fdagosti@cisco.com; dkim=pass ( sig from cisco.com/sjdkim2002 verified; ); X-Spam-Score: 0.3 (/) X-Scan-Signature: 93b4f10b2112e1468b61e19ea6180478 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0304238740==" Errors-To: ips-bounces@ietf.org This is a multi-part message in MIME format. --===============0304238740== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C7883A.3B61ECDF" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7883A.3B61ECDF Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable There are many customer fabrics running at 1000 ports or greater, as a single fabric. ________________________________ From: Julian Satran [mailto:Julian_Satran@il.ibm.com]=20 Sent: Thursday, April 26, 2007 12:35 PM To: Silvano Gai Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI My question was specific - and with a reason. Julo=20 "Silvano Gai" =20 26/04/07 14:33=20 To Julian Satran/Haifa/IBM@IBMIL=20 cc =20 Subject RE: FW: [Ips] Recent comments about FCoE and iSCSI =09 Julo,=20 =20 I am not sure what you are asking for.=20 Are you asking for a reference to a 1,000+ port SAN?=20 Today there are multiple FC switches on the market with 256 ports.=20 With six switches it is easy to realize a 1,000+ port SAN?=20 =20 Are you trying to tell me that 6 FC switches cannot be connected together?=20 =20 -- Silvano=20 =20 ________________________________ From: Julian Satran [mailto:Julian_Satran@il.ibm.com]=20 Sent: Thursday, April 26, 2007 11:25 AM To: Silvano Gai Cc: ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI=20 =20 Please send this community a reference of a 1000+ node datacenter using FCP as a single fabric.=20 Julo=20 "Silvano Gai" =20 26/04/07 13:26=20 To =20 cc =20 Subject Re: FW: [Ips] Recent comments about FCoE and iSCSI =20 =20 * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote:=20 > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips=20 ------_=_NextPart_001_01C7883A.3B61ECDF Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable
There are many customer fabrics running at 1000 = ports or=20 greater, as a single fabric.

From: Julian Satran=20 [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, = 2007=20 12:35 PM
To: Silvano Gai
Cc: = ips@ietf.org
Subject:=20 RE: FW: [Ips] Recent comments about FCoE and iSCSI


My question was specific = - and with=20 a reason. Julo


"Silvano Gai"=20 <sgai@nuovasystems.com>

26/04/07 14:33

To
Julian=20 Satran/Haifa/IBM@IBMIL=20
cc
<ips@ietf.org>=20
Subject
RE: FW: [Ips] Recent = comments about=20 FCoE and iSCSI

=




Julo,
 
I = am not sure=20 what you are asking for.
Are=20 you asking for a reference to a 1,000+ port SAN?
Today there are multiple FC switches on the = market with 256=20 ports.
With six = switches it is=20 easy to realize a 1,000+ port SAN?
 
Are you trying to=20 tell me that 6 FC switches cannot be connected together? =
 
-- Silvano
 =20


From: Julian Satran=20 [mailto:Julian_Satran@il.ibm.com]
Sent: Thursday, April 26, = 2007=20 11:25 AM
To: Silvano Gai
Cc: = ips@ietf.org
Subject:=20 Re: FW: [Ips] Recent comments about FCoE and iSCSI
 

Please send this community a reference of a 1000+ node = datacenter=20 using FCP as a single fabric.=20

Julo


"Silvano Gai"=20 <sgai@nuovasystems.com>

26/04/07 13:26


To
<ips@ietf.org>
cc
 =20
Subject
Re: FW: [Ips] Recent = comments about=20 FCoE and iSCSI

 =20


 =20  

<= BR>


   * Subject: Re: FW: [Ips] Recent comments about = FCoE and=20 iSCSI
  * From: "Eddy Quicksall" <Quicksall_iSCSI at=20 Bellsouth.net>
  * Date: Wed, 25 Apr 2007 15:15:22 = -0400

Eddy=20 Wrote:

> Further, I think PAUSE is only for point-to-point = and I=20 think that is
too > restrictive.

Eddy,

In Fibre = Channel=20 credits are only point-to-point and FC networks work
well for = datacenter=20 environments.

Where is the difference?

--=20 Silvano



_______________________________________________Ips=20 mailing=20 list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips= =20

------_=_NextPart_001_01C7883A.3B61ECDF-- --===============0304238740== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0304238740==-- From ips-bounces@ietf.org Thu Apr 26 17:20:59 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBOb-00039T-QD; Thu, 26 Apr 2007 17:20:57 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhBOb-00039I-2I for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 17:20:57 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBOa-00038u-Ol for ips@ietf.org; Thu, 26 Apr 2007 17:20:56 -0400 Received: from imf20aec.mail.bellsouth.net ([205.152.59.68]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhBOa-0000s6-Ck for ips@ietf.org; Thu, 26 Apr 2007 17:20:56 -0400 Received: from ibm64aec.bellsouth.net ([74.245.52.54]) by imf20aec.mail.bellsouth.net with ESMTP id <20070426212054.PPDH26124.imf20aec.mail.bellsouth.net@ibm64aec.bellsouth.net> for ; Thu, 26 Apr 2007 17:20:54 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm64aec.bellsouth.net with SMTP id <20070426212054.CNWL17809.ibm64aec.bellsouth.net@IVVTDKV0981>; Thu, 26 Apr 2007 17:20:54 -0400 Message-ID: <003f01c78848$c5bbcab0$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "Silvano Gai" , References: <34BDD2A93E5FD84594BB230DD6C18EA2015E3A69@nuova-ex1.hq.nuovaimpresa.com> Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 17:20:53 -0400 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.0 (/) X-Scan-Signature: 39bd8f8cbb76cae18b7e23f7cf6b2b9f Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org The statement was made with respect to two endpoints that may be across the US. As I understand PAUSE is point-to-point. iSCSI is end-to-end. Eddy ----- Original Message ----- From: "Silvano Gai" To: Sent: Thursday, April 26, 2007 1:26 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote: > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 17:23:52 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBRQ-0006I1-2R; Thu, 26 Apr 2007 17:23:52 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhBRP-0006Hv-2y for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 17:23:51 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBRO-0006Hn-PQ for ips@ietf.org; Thu, 26 Apr 2007 17:23:50 -0400 Received: from mx20.brocade.com ([66.243.153.19]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhBRN-0001Ld-Fo for ips@ietf.org; Thu, 26 Apr 2007 17:23:50 -0400 Received: from discus.brocade.com ([192.168.126.240]) by mx20.brocade.com with ESMTP; 26 Apr 2007 14:23:48 -0700 X-IronPort-AV: i="4.14,457,1170662400"; d="scan'208"; a="8698221:sNHT29518531" Received: from HQ-EXCHFE-3.corp.brocade.com (unknown [192.168.126.212]) by discus.brocade.com (Postfix) with ESMTP id 15C222383AD; Thu, 26 Apr 2007 14:22:37 -0700 (PDT) Received: from hq-exch-1.corp.brocade.com ([10.3.8.21]) by HQ-EXCHFE-3.corp.brocade.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 26 Apr 2007 14:23:53 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 14:23:46 -0700 Message-ID: <39BA3BC178B4394DB184389E88A97F8C0233B17A@hq-exch-1.corp.brocade.com> In-Reply-To: <003f01c78848$c5bbcab0$05faa8c0@ivivity.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceISOhoW4cppxLJQ6uNV2xy6dd4cAAAC5AA From: "John Hufferd" To: "Eddy Quicksall" , "Silvano Gai" , X-OriginalArrivalTime: 26 Apr 2007 21:23:53.0207 (UTC) FILETIME=[308FD070:01C78849] X-Spam-Score: 0.0 (/) X-Scan-Signature: 082a9cbf4d599f360ac7f815372a6a15 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org And DCE and FCoE are Data Center Only things. NOT Cross country. . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =20 -----Original Message----- From: Eddy Quicksall [mailto:Quicksall_iSCSI@Bellsouth.net]=20 Sent: Thursday, April 26, 2007 2:21 PM To: Silvano Gai; ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI The statement was made with respect to two endpoints that may be across the=20 US. As I understand PAUSE is point-to-point. iSCSI is end-to-end. Eddy ----- Original Message -----=20 From: "Silvano Gai" To: Sent: Thursday, April 26, 2007 1:26 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote: > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 17:25:24 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBSt-0006i0-Q8; Thu, 26 Apr 2007 17:25:23 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhBSs-0006hK-Fe for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 17:25:22 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhBSs-0006h5-5z for ips@ietf.org; Thu, 26 Apr 2007 17:25:22 -0400 Received: from mx20.brocade.com ([66.243.153.19]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhBSr-0001VG-Ru for ips@ietf.org; Thu, 26 Apr 2007 17:25:22 -0400 Received: from discus.brocade.com ([192.168.126.240]) by mx20.brocade.com with ESMTP; 26 Apr 2007 14:25:21 -0700 X-IronPort-AV: i="4.14,457,1170662400"; d="scan'208"; a="8698343:sNHT20805176" Received: from HQ-EXCHFE-3.corp.brocade.com (unknown [192.168.126.212]) by discus.brocade.com (Postfix) with ESMTP id 9A5D22383AD; Thu, 26 Apr 2007 14:24:09 -0700 (PDT) Received: from hq-exch-1.corp.brocade.com ([10.3.8.21]) by HQ-EXCHFE-3.corp.brocade.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 26 Apr 2007 14:25:25 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 14:25:18 -0700 Message-ID: <39BA3BC178B4394DB184389E88A97F8C0233B17C@hq-exch-1.corp.brocade.com> In-Reply-To: <75852864BAD9684FBF5DCF4289DE4076078FFA97@CORPUSMX30B.corp.emc.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceHbhab+EXeFVfgQWiH7jzYWydYaAAKX7IQAAKlUiAABWKvIAAeuqUAAAVgg0A= From: "John Hufferd" To: , X-OriginalArrivalTime: 26 Apr 2007 21:25:25.0710 (UTC) FILETIME=[67B2A6E0:01C78849] X-Spam-Score: 0.0 (/) X-Scan-Signature: 287c806b254c6353fcb09ee0e53bbc5e Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org David, The first part of you question has perhaps a different answer than you "In other words" part. The first answer is it communicates the same way a FC adapter communicates with iSCSI based storage --- via FC to iSCSI gateways. The answer you maybe asking (in your Other Words part) is that it is possible for a smart NIC that supports maybe iSCSI, TOE, NIC, and even perhaps iWARP, to have another interface that supports FCoE. The FCoE part would send FC frames through the same NIC that was used by the iSCSI part, but the FCoE frames would be headed (without a gateway) to a different place then the iSCSI frames were headed.=20 . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =20 -----Original Message----- From: brown_David1@emc.com [mailto:brown_David1@emc.com]=20 Sent: Thursday, April 26, 2007 11:49 AM To: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI How would an FCoE-based initiator communicate with iSCSI-based storage?=20 In other words, can the same host adapter be used for FCoE connections and iSCSI sessions, and if so, would the performance be similar? thanks, dj ________________________________ From: John Hufferd [mailto:jhufferd@Brocade.COM]=20 Sent: Thursday, April 26, 2007 12:52 AM To: Sandars, Ken; Eddy Quicksall; Julian Satran Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =09 =09 Ken, The term FCoE has as its primary component FC. Consider the possibility that the DCE Link from the Host connects to a switch/device that is able to deal with the FC part of the FCoE. =20 . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com =20 Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 =09 ________________________________ From: Sandars, Ken [mailto:ken_sandars@adaptec.com]=20 Sent: Wednesday, April 25, 2007 6:41 PM To: John Hufferd; Eddy Quicksall; Julian Satran Cc: ips@ietf.org Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI =20 Hey John, =20 [Hufferd] Many servers are asking for an evolutionary way to combine their Networking connections from the Server. The customers I have dealt with do NOT want to rip out FC, they want to provide a single Link for transport of all networking needs, including storage, exiting their servers. =09 I'm not sure what you are saying here. Are you saying servers should have a single type of physical network connection, presumably ethernet? How does that align with not wanting to rip out FC? =20 Thanks Ken _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 18:08:39 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhC8l-0006Rx-Eb; Thu, 26 Apr 2007 18:08:39 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhC8k-0006Rs-SO for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 18:08:38 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhC8k-0006Rk-Ix for ips@ietf.org; Thu, 26 Apr 2007 18:08:38 -0400 Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhC8k-0000Qb-2X for ips@ietf.org; Thu, 26 Apr 2007 18:08:38 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.13.191.130]) by palrel10.hp.com (Postfix) with ESMTP id E79B0358F9 for ; Thu, 26 Apr 2007 15:08:36 -0700 (PDT) Received: from MK73191c.cup.hp.com ([15.244.201.133]) by esmail.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id PAA19896 for ; Thu, 26 Apr 2007 15:02:55 -0700 (PDT) Message-Id: <6.2.0.14.2.20070426145824.03223f58@esmail.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.0.14 Date: Thu, 26 Apr 2007 15:06:19 -0700 To: From: Michael Krause Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI In-Reply-To: <39BA3BC178B4394DB184389E88A97F8C0233B17C@hq-exch-1.corp.br ocade.com> References: <75852864BAD9684FBF5DCF4289DE4076078FFA97@CORPUSMX30B.corp.emc.com> <39BA3BC178B4394DB184389E88A97F8C0233B17C@hq-exch-1.corp.brocade.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: b1c41982e167b872076d0018e4e1dc3c X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Out of curiosity, unless someone is proposing a change in the iSCSI spec to accommodate something unique to FCoE, isn't this entire debate really not an IETF topic? Until the proponents of FCoE make their draft proposals available for public consumption and debate, this discussion is somewhat occurring in a vacuum. Same goes for comprehending all that is in DCE. It needs to be completely public to understand whether what is proposed makes sense to the broader implementation community w.r.t. to storage. Mike At 02:25 PM 4/26/2007, John Hufferd wrote: >David, >The first part of you question has perhaps a different answer than you >"In other words" part. > >The first answer is it communicates the same way a FC adapter >communicates with iSCSI based storage --- via FC to iSCSI gateways. > >The answer you maybe asking (in your Other Words part) is that it is >possible for a smart NIC that supports maybe iSCSI, TOE, NIC, and even >perhaps iWARP, to have another interface that supports FCoE. The FCoE >part would send FC frames through the same NIC that was used by the >iSCSI part, but the FCoE frames would be headed (without a gateway) to a >different place then the iSCSI frames were headed. > >. >. >. >John L Hufferd >Sr. Executive Director of Technology >jhufferd@brocade.com >Office Phone: (408) 333-5244; eFAX: (408) 904-4688 >Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > > >-----Original Message----- >From: brown_David1@emc.com [mailto:brown_David1@emc.com] >Sent: Thursday, April 26, 2007 11:49 AM >To: ips@ietf.org >Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > >How would an FCoE-based initiator communicate with iSCSI-based storage? > >In other words, can the same host adapter be used for FCoE connections >and iSCSI sessions, and if so, would the performance be similar? > >thanks, >dj >________________________________ > > From: John Hufferd [mailto:jhufferd@Brocade.COM] > Sent: Thursday, April 26, 2007 12:52 AM > To: Sandars, Ken; Eddy Quicksall; Julian Satran > Cc: ips@ietf.org > Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > > > > Ken, > > The term FCoE has as its primary component FC. Consider the >possibility that the DCE Link from the Host connects to a switch/device >that is able to deal with the FC part of the FCoE. > > > > . > > . > > . > > John L Hufferd > > Sr. Executive Director of Technology > > jhufferd@brocade.com > > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > > > > ________________________________ > > From: Sandars, Ken [mailto:ken_sandars@adaptec.com] > Sent: Wednesday, April 25, 2007 6:41 PM > To: John Hufferd; Eddy Quicksall; Julian Satran > Cc: ips@ietf.org > Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > > > > Hey John, > > > > [Hufferd] Many servers are asking for an evolutionary way to >combine their Networking connections from the Server. The customers I >have dealt with do NOT want to rip out FC, they want to provide a single >Link for transport of all networking needs, including storage, exiting >their servers. > > > I'm not sure what you are saying here. Are you saying servers >should have a single type of physical network connection, presumably >ethernet? How does that align with not wanting to rip out FC? > > > > Thanks > > Ken > > > >_______________________________________________ >Ips mailing list >Ips@ietf.org >https://www1.ietf.org/mailman/listinfo/ips > > >_______________________________________________ >Ips mailing list >Ips@ietf.org >https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 18:29:49 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhCTE-0002zN-6L; Thu, 26 Apr 2007 18:29:48 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhCTC-0002xR-5w for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 18:29:46 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhCTB-0002w0-Ra for ips@ietf.org; Thu, 26 Apr 2007 18:29:45 -0400 Received: from nuova-ex1.nuovasystems.com ([67.91.200.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhCTA-0004Dk-Ei for ips@ietf.org; Thu, 26 Apr 2007 18:29:45 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 15:29:32 -0700 Message-ID: <34BDD2A93E5FD84594BB230DD6C18EA2015E3C36@nuova-ex1.hq.nuovaimpresa.com> In-Reply-To: <6.2.0.14.2.20070426145824.03223f58@esmail.cup.hp.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: FW: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIT5ai7VbnkhCWTymH/1UVaqAlpQAAqUEg References: <75852864BAD9684FBF5DCF4289DE4076078FFA97@CORPUSMX30B.corp.emc.com><39BA3BC178B4394DB184389E88A97F8C0233B17C@hq-exch-1.corp.brocade.com> <6.2.0.14.2.20070426145824.03223f58@esmail.cup.hp.com> From: "Silvano Gai" To: "Michael Krause" , X-Spam-Score: 0.0 (/) X-Scan-Signature: 1449ead51a2ff026dcb23465f5379250 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org I agree, FCoE is being discussed in T11, in the FC-BB working group, as an additional transport for FC. -- Silvano > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > Sent: Thursday, April 26, 2007 3:06 PM > To: ips@ietf.org > Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 > Out of curiosity, unless someone is proposing a change in the iSCSI spec > to > accommodate something unique to FCoE, isn't this entire debate really not > an IETF topic? Until the proponents of FCoE make their draft proposals > available for public consumption and debate, this discussion is somewhat > occurring in a vacuum. Same goes for comprehending all that is in > DCE. It needs to be completely public to understand whether what is > proposed makes sense to the broader implementation community w.r.t. to > storage. >=20 > Mike >=20 >=20 > At 02:25 PM 4/26/2007, John Hufferd wrote: > >David, > >The first part of you question has perhaps a different answer than you > >"In other words" part. > > > >The first answer is it communicates the same way a FC adapter > >communicates with iSCSI based storage --- via FC to iSCSI gateways. > > > >The answer you maybe asking (in your Other Words part) is that it is > >possible for a smart NIC that supports maybe iSCSI, TOE, NIC, and even > >perhaps iWARP, to have another interface that supports FCoE. The FCoE > >part would send FC frames through the same NIC that was used by the > >iSCSI part, but the FCoE frames would be headed (without a gateway) to a > >different place then the iSCSI frames were headed. > > > >. > >. > >. > >John L Hufferd > >Sr. Executive Director of Technology > >jhufferd@brocade.com > >Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > >Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > > > > > >-----Original Message----- > >From: brown_David1@emc.com [mailto:brown_David1@emc.com] > >Sent: Thursday, April 26, 2007 11:49 AM > >To: ips@ietf.org > >Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > > > >How would an FCoE-based initiator communicate with iSCSI-based storage? > > > >In other words, can the same host adapter be used for FCoE connections > >and iSCSI sessions, and if so, would the performance be similar? > > > >thanks, > >dj > >________________________________ > > > > From: John Hufferd [mailto:jhufferd@Brocade.COM] > > Sent: Thursday, April 26, 2007 12:52 AM > > To: Sandars, Ken; Eddy Quicksall; Julian Satran > > Cc: ips@ietf.org > > Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > > > > > > > > Ken, > > > > The term FCoE has as its primary component FC. Consider the > >possibility that the DCE Link from the Host connects to a switch/device > >that is able to deal with the FC part of the FCoE. > > > > > > > > . > > > > . > > > > . > > > > John L Hufferd > > > > Sr. Executive Director of Technology > > > > jhufferd@brocade.com > > > > Office Phone: (408) 333-5244; eFAX: (408) 904-4688 > > > > Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 > > > > > > > > ________________________________ > > > > From: Sandars, Ken [mailto:ken_sandars@adaptec.com] > > Sent: Wednesday, April 25, 2007 6:41 PM > > To: John Hufferd; Eddy Quicksall; Julian Satran > > Cc: ips@ietf.org > > Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI > > > > > > > > Hey John, > > > > > > > > [Hufferd] Many servers are asking for an evolutionary way to > >combine their Networking connections from the Server. The customers I > >have dealt with do NOT want to rip out FC, they want to provide a single > >Link for transport of all networking needs, including storage, exiting > >their servers. > > > > > > I'm not sure what you are saying here. Are you saying servers > >should have a single type of physical network connection, presumably > >ethernet? How does that align with not wanting to rip out FC? > > > > > > > > Thanks > > > > Ken > > > > > > > >_______________________________________________ > >Ips mailing list > >Ips@ietf.org > >https://www1.ietf.org/mailman/listinfo/ips > > > > > >_______________________________________________ > >Ips mailing list > >Ips@ietf.org > >https://www1.ietf.org/mailman/listinfo/ips >=20 >=20 >=20 >=20 > _______________________________________________ > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 20:17:01 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhE8v-0006WU-2f; Thu, 26 Apr 2007 20:16:57 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhE8t-0006WP-OW for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 20:16:55 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhE8t-0006WH-Er for ips@ietf.org; Thu, 26 Apr 2007 20:16:55 -0400 Received: from imf22aec.mail.bellsouth.net ([205.152.59.70]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhE8s-0008Ce-4d for ips@ietf.org; Thu, 26 Apr 2007 20:16:55 -0400 Received: from ibm65aec.bellsouth.net ([74.245.52.54]) by imf22aec.mail.bellsouth.net with ESMTP id <20070427001506.CJMV11433.imf22aec.mail.bellsouth.net@ibm65aec.bellsouth.net> for ; Thu, 26 Apr 2007 20:15:06 -0400 Received: from IVVTDKV0981 ([74.245.52.54]) by ibm65aec.bellsouth.net with SMTP id <20070427001506.HUFG5647.ibm65aec.bellsouth.net@IVVTDKV0981>; Thu, 26 Apr 2007 20:15:06 -0400 Message-ID: <005901c78861$19f4d730$05faa8c0@ivivity.com> From: "Eddy Quicksall" To: "John Hufferd" , "Silvano Gai" , References: <39BA3BC178B4394DB184389E88A97F8C0233B17A@hq-exch-1.corp.brocade.com> Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 20:15:02 -0400 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3028 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 X-Spam-Score: 0.0 (/) X-Scan-Signature: d185fa790257f526fedfd5d01ed9c976 Cc: X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Yea, that's my point. Eddy ----- Original Message ----- From: "John Hufferd" To: "Eddy Quicksall" ; "Silvano Gai" ; Sent: Thursday, April 26, 2007 5:23 PM Subject: RE: FW: [Ips] Recent comments about FCoE and iSCSI And DCE and FCoE are Data Center Only things. NOT Cross country. . . . John L Hufferd Sr. Executive Director of Technology jhufferd@brocade.com Office Phone: (408) 333-5244; eFAX: (408) 904-4688 Alt Office Phone: (408) 997-6136; Cell: (408) 627-9606 -----Original Message----- From: Eddy Quicksall [mailto:Quicksall_iSCSI@Bellsouth.net] Sent: Thursday, April 26, 2007 2:21 PM To: Silvano Gai; ips@ietf.org Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI The statement was made with respect to two endpoints that may be across the US. As I understand PAUSE is point-to-point. iSCSI is end-to-end. Eddy ----- Original Message ----- From: "Silvano Gai" To: Sent: Thursday, April 26, 2007 1:26 PM Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * Subject: Re: FW: [Ips] Recent comments about FCoE and iSCSI * From: "Eddy Quicksall" * Date: Wed, 25 Apr 2007 15:15:22 -0400 Eddy Wrote: > Further, I think PAUSE is only for point-to-point and I think that is too > restrictive. Eddy, In Fibre Channel credits are only point-to-point and FC networks work well for datacenter environments. Where is the difference? -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Thu Apr 26 21:16:34 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhF4a-0002qk-4h; Thu, 26 Apr 2007 21:16:32 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhF4Y-0002U0-9r for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 21:16:30 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhF4X-0002O9-OC for ips@ietf.org; Thu, 26 Apr 2007 21:16:29 -0400 Received: from mtagate1.de.ibm.com ([195.212.29.150]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhF4T-0000RB-C6 for ips@ietf.org; Thu, 26 Apr 2007 21:16:29 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate1.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3R1GO2R110416 for ; Fri, 27 Apr 2007 01:16:24 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3R1GOT44096242 for ; Fri, 27 Apr 2007 03:16:24 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3R1GOmg006623 for ; Fri, 27 Apr 2007 03:16:24 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3R1GO8G006620; Fri, 27 Apr 2007 03:16:24 +0200 In-Reply-To: <179300.47262.qm@web36705.mail.mud.yahoo.com> To: Zack Best MIME-Version: 1.0 Subject: RE: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Thu, 26 Apr 2007 21:16:21 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 27/04/2007 04:16:23, Serialize complete at 27/04/2007 04:16:23 X-Spam-Score: 0.5 (/) X-Scan-Signature: c96e11e58076fc8e92061fb6cbdfae15 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1988015737==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============1988015737== Content-Type: multipart/alternative; boundary="=_alternative 0006FBDC852572CA_=" This is a multipart message in MIME format. --=_alternative 0006FBDC852572CA_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Excellent comments. My take (if not obvious from the previous text) is=20 that data centers will be very large and compute power (as evidenced by=20 the multicore) and advances in stack implementation are bound to improve=20 substantialy the performance of the protocol stacks (see Intel and our=20 work) and layer 3 switching. It is important also to point out that Ethernet has substantial latencies=20 if only bridging is using and replacement technologies (such as Rbridges=20 or others) may take some time to appear. Julo Zack Best =20 25/04/07 16:37 To ips@ietf.org cc Subject RE: [Ips] Recent comments about FCoE and iSCSI The real debate here is between two types of networks. The first is reliable at the link level and does not drop packets under congestion. The second is running a reliable transport protocol (i.e. TCP) over an unreliable link level network. I agree with the scaling argument. For sufficiently large networks, reliable link level doesn't work well because network component failure, or chronically congested links are not handled well. For sufficiently small networks, reliable link level has some significant advantages in simplicity, low hardware cost, performance, and worst case latency. My personal view is that the vast majority of enterprise storage networks fall in the "sufficiently small" category. This view has to some extent been vindicated by the continuing success of Fibre Channel in this space and the inability of iSCSI to displace FC in any significant way for enterprise storage. Of course, this may or may not change in the future. Whether FC is simpler than iSCSI depends largely on your definition of simplicity. If one defines simplicity/complexity as the number of gates or lines of code to reduce the protocol to hardware or firmware, then my experience is that iSCSI is 2X to 3X the complexity of FC. This has implications in cost and reliability. Particularly problematic with iSCSI is the unpredictability of the performance. Performance is great with no packet drop. However even a small amount of congestion can cause a sudden large drop and performance. This can be difficult to predict as a network that is almost but not quite congested can run great, but a small incremental change of any sort can cause the performance to become suddenly unacceptable. For FC, or other protocol using link level flow control, the reduction in performance is much more graceful and incremental when the level of congestion is small and intermittent. A second major problem with iSCSI is the unbounded nature of worst case latency. When a storage network fails, it is desirable to detect the failure in a fraction of a second and transition to a backup network. TCP, when implemented to the standards, can take many seconds or minutes to determine that a network has failed and close the connection. RFC 2988, for instance, requires that the minimum retransmission be one second. This means a single dropped packet may add one second to the latency of outstanding commands. This is a huge amount of time on a 10G link. No doubt this could be mitigated by drastically reducing the timeouts within TCP, but the market seems to be surprisingly resistant to tampering with accepted standards here. Overall, the FC and FCP protocol have a lot in common with the Intel i86 instruction set architecture. They are overly complex, and rather poorly designed by modern standards. But they are good enough, and there is a huge amount of value add that has been built on top of them, and therefore little incentive to change. FCoE is an interesting idea because it preserves 90% of the existing value add of FC, unifies the physical link with Ethernet, and uses the reliable link method of packet delivery. There are two significant possibilities for iSCSI to displace FC (or FCoE) in enterprise storage networks.=20 First is if the networks start to scale to large enough size that FC can't be made sufficiently reliable, and second if CPU compute cycles become sufficiently cheap that the iSCSI protocol can be run in host software with no negative performance impact.=20 Barring either of these, it seems that iSCSI will have an uphill battle, and FCoE may have a place. -----Original Message----- From: Julian Satran [mailto:Julian=5FSatran@il.ibm.com] Sent: Tuesday, April 24, 2007 3:10 PM To: ips@ietf.org Subject: [Ips] Recent comments about FCoE and iSCSI Dear All,=20 The trade press is lately full with comments about the latest and greatest reincarnation of Fiber Channel over ethernet.=20 It made me try and summarize all the long and hot debates that preceded the advent of iSCSI.=20 Although FCoE proponents make it look like no debate preceded iSCSI that was not so - FCoE was considered even then and was dropped as a dumb idea.=20 Here is a summary (as afar as I can remember) of the main arguments. They are not bad arguments even in retrospect and technically FCoE doesn't look better than it did then.=20 Feel free to use this material in a nay form. I expect this group to seriously expand my arguments and make them public - in personal or collective form.=20 And do not forget - it is a technical dispute - although we all must have some doubts about the way it is pursued.=20 Regards,=20 Julo=20 --------------------------------------------------------------------- What a piece of nostalgia :-)=20 Around 1997 when a team at IBM Research (Haifa and Almaden) started looking at connecting storage to servers using the "regular network" (the ubiquitous LAN) we considered many alternatives (another team even had a look at ATM - still a computer network candidate at the time). I won't get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple:=20 Fiber Channel Protocol (SCSI over Fiber Channel Link) is "mildly" effective because:=20 it implements endpoints in a dedicated engine (Offload)=20 it has no transport layer (recovery is done at the application layer under the assumption that the error rate will be very low)=20 the network is limited in physical span and logical span (number of switches)=20 flow-control/congestion control is achieved with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechanism)=20 However FCP endpoints are inherently costlier than simple NICs ? the cost argument (initiators are more expensive)=20 The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) ? the scaling argument=20 The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s ? the scaling argument=20 Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has always been the preferred choice of the networking community ? the community argument=20 The "performance penalty" of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computing scene have enough compute cycles available to make any "offloading" possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.)=20 Building on a complete stack makes available a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service location etc.) ? the community argument=20 Higher level storage access over an IP network is widely available and having both block and file served over the same connection with the same support and management structure is compelling ? the community argument=20 Highly efficient networks are easy to build over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don't know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited ? the scaling argument As a side argument ? a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI ? and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows).=20 The networking and storage community acknowledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc.=20 The community also acknowledged the need to support existing infrastructure and extend it in a reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been=20 implemented and their foundation is solid.=20 The current attempt of developing a "new-age" FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc.=20 It ignores the networking layering practice, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applications more expensive and leaves aside the whole "ecosystem" that accompanies TCP/IP (and not Ethernet).=20 In some related effort (and at a point also when developing iSCSI) we considered also moving away from SCSI (like some "no standardized" but popular in some circles software did ? e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time.=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around=20 http://mail.yahoo.com=20 =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --=_alternative 0006FBDC852572CA_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Excellent comments. My take (if not obvious from the previous text) is that data centers will be very large and compute power (as evidenced by the multicore) and advances in stack implementation are bound to improve substantialy the performance of the protocol stacks (see Intel and our work) and layer 3 switching.
It is important also to point out th= at Ethernet has substantial latencies if only bridging is using and replacement technologies (such as Rbridges or others) may take some time to appear.

Julo


Zack Best <zbest28= @yahoo.com>

25/04/07 16:37

To
ips@ietf.org
cc
Subject
RE: [Ips] Recent comments about FCoE and iSCSI





The real debate here is between two types of network= s.
The first is reliable at the link level and does not
drop packets under congestion.  The second is running
a reliable transport protocol (i.e. TCP) over an
unreliable link level network.

I agree with the scaling argument.  For sufficiently
large networks, reliable link level doesn't work well
because network component failure, or chronically
congested links are not handled well.  For
sufficiently small networks, reliable link level has
some significant advantages in simplicity, low
hardware cost, performance, and worst case latency.

My personal view is that the vast majority of
enterprise storage networks fall in the "sufficiently
small" category.  This view has to some extent been
vindicated by the continuing success of Fibre Channel
in this space and the inability of iSCSI to displace
FC in any significant way for enterprise storage.  Of
course, this may or may not change in the future.

Whether FC is simpler than iSCSI depends largely on
your definition of simplicity.  If one defines
simplicity/complexity as the number of gates or lines
of code to reduce the protocol to hardware or
firmware, then my experience is that iSCSI is 2X to 3X
the complexity of FC.  This has implications in cost
and reliability.

Particularly problematic with iSCSI is the
unpredictability of the performance.  Performance is
great with no packet drop.  However even a small
amount of congestion can cause a sudden large drop and
performance.  This can be difficult to predict as a
network that is almost but not quite congested can run
great, but a small incremental change of any sort can
cause the performance to become suddenly unacceptable.
For FC, or other protocol using link level flow
control, the reduction in performance is much more
graceful and incremental when the level of congestion
is small and intermittent.

A second major problem with iSCSI is the unbounded
nature of worst case latency.  When a storage network
fails, it is desirable to detect the failure in a
fraction of a second and transition to a backup
network.  TCP, when implemented to the standards, can
take many seconds or minutes to determine that a
network has failed and close the connection.  RFC
2988, for instance, requires that the minimum
retransmission be one second.  This means a single
dropped packet may add one second to the latency of
outstanding commands.  This is a huge amount of time
on a 10G link.  No doubt this could be mitigated by
drastically reducing the timeouts within TCP, but the
market seems to be surprisingly resistant to tampering
with accepted standards here.

Overall, the FC and FCP protocol have a lot in common
with the Intel i86 instruction set architecture.  They
are overly complex, and rather poorly designed by
modern standards.  But they are good enough, and there
is a huge amount of value add that has been built on
top of them, and therefore little incentive to change.
FCoE is an interesting idea because it preserves 90%
of the existing value add of FC, unifies the physical
link with Ethernet, and uses the reliable link method
of packet delivery.

There are two significant possibilities for iSCSI to
displace FC (or FCoE) in enterprise storage networks.
First is if the networks start to scale to large
enough size that FC can't be made sufficiently
reliable, and second if CPU compute cycles become
sufficiently cheap that the iSCSI protocol can be run
in host software with no negative performance impact.
Barring either of these, it seems that iSCSI will have
an uphill battle, and FCoE may have a place.

-----Original Message-----
From: Julian Satran [mailto:Julian=5FSatran@il.ibm.com]
Sent: Tuesday, April 24, 2007 3:10 PM
To: ips@ietf.org
Subject: [Ips] Recent comments about FCoE and iSCSI



Dear All,

The trade press is lately full with comments about the
latest and greatest reincarnation of Fiber Channel
over ethernet.
It made me try and summarize all the long and hot
debates that preceded the advent of iSCSI.
Although FCoE proponents make it look like no debate
preceded iSCSI that was not so - FCoE was considered
even then and was dropped as a dumb idea.

Here is a summary (as afar as I can remember) of the
main arguments. They are not bad arguments even in
retrospect and technically FCoE doesn't look better
than it did then.

Feel free to use this material in a nay form. I expect
this group to seriously  expand my arguments and make
them public - in personal or collective form.

And do not forget - it is a technical dispute -
although we all must have some doubts about the way it
is pursued.

Regards,
Julo

---------------------------------------------------------------------


What a piece of nostalgia :-)

Around 1997 when a team at IBM Research (Haifa and
Almaden) started looking at connecting storage to
servers using the "regular network" (the ubiquitous
LAN) we considered many alternatives (another team
even had a look at ATM - still a computer network
candidate at the time). I won't get you over all of
our rationale (and we went over some of them again at
the end of 1999 with a team from CISCO before we
convened the first IETF BOF in 2000 at Adelaide that
resulted in iSCSI and all the rest) but some of the
reasons we choose to drop Fiber Channel over raw
Ethernet where multiple:

Fiber Channel Protocol (SCSI over Fiber Channel Link)
is "mildly" effective because:
it implements endpoints in a dedicated engine
(Offload)
it has no transport layer (recovery is done at the
application layer under the assumption that the error
rate will be very low)
the network is limited in physical span and logical
span (number of switches)
flow-control/congestion control is achieved with a
mechanism adequate for a limited span network
(credits). The packet loss rate is almost nil and that
allows FCP to avoid using a transport (end-to-end)
layer
FCP she switches are simple (addresses are local and
the memory requirements cam be limited through the
credit mechanism)
However FCP endpoints are inherently costlier than
simple NICs – the cost argument (initiators are more
expensive)
The credit mechanisms is highly unstable for large
networks (check switch vendors planning docs for the
network diameter limits) – the scaling argument
The assumption of low losses due to errors might
radically change when moving from 1 to 10 Gb/s – the
scaling argument
Ethernet has no credit mechanism and any mechanism
with a similar effect increases the end point cost.
Building a transport layer in the protocol stack has
always been the preferred choice of the networking
community – the community argument
The "performance penalty" of a complete protocol stack
has always been overstated (and overrated). Advances
in protocol stack implementation and finer tuning of
the congestion control mechanisms make conventional
TCP/IP performing well even at 10 Gb/s and over.
Moreover the multicore processors that become dominant
on the computing scene have enough compute cycles
available to make any "offloading" possible as a mere
code restructuring exercise (see the stack reports
from Intel, IBM etc.)
Building on a complete stack makes available a wealth
of operational and management mechanisms built over
the years by the networking community (routing,
provisioning, security, service location etc.) – the
community argument
Higher level storage access over an IP network is
widely available and having both block and file served
over the same connection with the same support and
management structure is compelling – the community
argument
Highly efficient networks are easy to build over IP
with optimal (shortest path) routing while Layer 2
networks use bridging and are limited by the logical
tree structure that bridges must follow. The effort to
combine routers and bridges (rbridges) is promising to
change that but it will take some time to finalize
(and we don't know exactly how it will operate).
Untill then the scale of Layer 2 network is going to
seriously limited – the scaling argument


As a side argument – a performance comparison made in
1998 showed SCSI over TCP (a predecessor of the later
iSCSI) to perform better than FCP at 1Gbs for block
sizes typical for OLTP (4-8KB). That was what
convinced us to take the path that lead to iSCSI – and
we used plain vanilla x86 servers with plain-vanilla
NICs and Linux (with similar measurements conducted on
Windows).
The networking and storage community acknowledged
those arguments and developed iSCSI and the companion
protocols for service discovery, boot etc.

The community also acknowledged the need to support
existing infrastructure and extend it in a reasonable
fashion and developed 2 protocols iFCP (to support
hosts with FCP drivers and IP connections to connect
to storage by a simple conversion from FCP to TCP
packets) FCPIP to extend the reach of FCP through IP
(connects FCP islands through TCP links). Both have
been
implemented and their foundation is solid.

The current attempt of developing a "new-age" FCP over
an Ethernet link is going against most of the
arguments that have given us iSCSI etc.

It ignores the networking layering practice, build an
application protocol directly above a link and thus
limits scaling, mandates elements at the link layer
and application layer that make applications more
expensive and leaves aside the whole "ecosystem" that
accompanies TCP/IP (and not Ethernet).

In some related effort (and at a point also when
developing iSCSI) we considered also moving away from
SCSI (like some "no standardized" but popular in some
circles software did – e.g., NBP) but decided against.
SCSI is a mature and well understood access
architecture for block storage and is implemented by
many device vendors. Moving away from it would not
have been justified at the time.


=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com



=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=_alternative 0006FBDC852572CA_=-- --===============1988015737== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============1988015737==-- From ips-bounces@ietf.org Fri Apr 27 00:53:07 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhIS5-0007vw-A2; Fri, 27 Apr 2007 00:53:01 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhIS4-0007v5-13 for ips-confirm+ok@megatron.ietf.org; Fri, 27 Apr 2007 00:53:00 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhIS3-0007ux-NQ for ips@ietf.org; Fri, 27 Apr 2007 00:52:59 -0400 Received: from smtp103.sbc.mail.mud.yahoo.com ([68.142.198.202]) by ietf-mx.ietf.org with smtp (Exim 4.43) id 1HhIS3-0007hH-23 for ips@ietf.org; Fri, 27 Apr 2007 00:52:59 -0400 Received: (qmail 57996 invoked from network); 27 Apr 2007 04:52:58 -0000 Received: from unknown (HELO haakon2.local) (nicholas_bellinger@sbcglobal.net@70.231.231.166 with plain) by smtp103.sbc.mail.mud.yahoo.com with SMTP; 27 Apr 2007 04:52:57 -0000 X-YMail-OSG: RFjwy20VM1n.c8.TzgTHyLc6cHVWRSyJDzitYZSTCAk0DA5DpsNDlQeH0ZjE3YXUen16DqCpkw3XCBG9GkavN3x9i6MTSyMwmI_9K_HCy1GXeH0DBwgvhnJnQR44aw-- Subject: RE: [Ips] Recent comments about FCoE and iSCSI From: "Nicholas A. Bellinger" To: Julian Satran In-Reply-To: References: Content-Type: text/plain; charset=UTF-8 Date: Thu, 26 Apr 2007 21:52:56 -0700 Message-Id: <1177649576.5355.132.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 X-Mailer: Evolution 2.6.3 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.3 (/) X-Scan-Signature: 96e0f8497f38c15fbfc8f6f315bcdecb Cc: ips@ietf.org, Mike Mazarick , Eric Hall , Zack Best X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org (This is a resend, my original email bounced..) A quick comment in regards to the abundance large of computing resources available for initiator side software IP storage services.. Also Julo, many thanks for posting this great thread. :) --nab ------------------------------------------------------------------------ As the progress of the DDP TWG continues onward and 2nd generation hardware iWARP engines start to come online, the benefit of a hybrid software implementation with host OS software network stack modifications in kernel above TCP and SCTP starts to pose a question.. What real savings can a hyrbid iSER nodes using software DDP? What are those changes required to make high performance software DDP a reality..? As osc-iwarp has found out, there is a significant CPU overhead assoicated with sockets and software VERBS, but I think this can be minimized with the right set of changes. Those changes are moving away from receieve side sockets for software iSER mode. These changes will start to become attractive for new product designs as this will allow RNIC hardware engines to scale futher using a more sane method or less painfully (depending on who you ask, OFA uses a Hybrid IB-VERBs) than traditional TOEs with speciality engines. Really taking advantage of what metadata in DDP and iWARP metadata is telling about the framed network transport can help in RDMA WRITE scenarios because the software RNIC would already have Stagged memory ready to go in the iSER case. Espically when it comes to the API for the iSER stack, having a single codebase with vendors writing hardware drivers instead of re-inventing the wheel with sockets. I believe the smart software RNICs of the future will direct RDMA traffic directly into host OS SCSI memory buffers, and like today use something similar to sendpage() for TX. As multi-core microprocessor designs with large, intelligent shared caches, and CPU cache coherentecy and I/O interconnects that in the 90's where only available in the Alpha EV67 and highest of high end shared memory supercomputers and clusters are now starting to become the norm. Pushing software iSER to the next level and beyond is surely not going to happen with a 30 year old API (sockets). Also for the data center story with a traditional tiered SAN architecture and software case, the hyrbid iWARP software stack on the initiator will not get a whole lot of interest until it can show improved performance and overhead that is acceptable to traditional iSCSI today. For the 3rd generation IP storage stacks, typical multiport 1G workloads is what will really drive interest into areas where putting a hardware RNIC will not be cost feasable for some time. But just as with traditional iSCSI, we can also scale software iSER down towards towards platforms with more modest computing resources on low-power, wireless devices. Even in the type of mobile devices that IP storage services have been prototyped on today, the benefit of being able to scale server side hardware RNICs more efficently is not software iSER's only benefit. On a side note, I think the transparency that connection recovery in traditional iSCSI and iSER allows to internexus multiplexing, as well as end user requirements for configuration and management scenarios. Using a active-active recovery mechinism that is as close to completely transparent as possbile (which ERL=2 is IMHO) is I think what mobile IP storage services users need to be demanding from their transports. Thanks for listening! On Thu, 2007-04-26 at 21:16 -0400, Julian Satran wrote: > > Excellent comments. My take (if not obvious from the previous text) is > that data centers will be very large and compute power (as evidenced > by the multicore) and advances in stack implementation are bound to > improve substantialy the performance of the protocol stacks (see Intel > and our work) and layer 3 switching. > It is important also to point out that Ethernet has substantial > latencies if only bridging is using and replacement technologies (such > as Rbridges or others) may take some time to appear. > > Julo > > > Zack Best > > 25/04/07 16:37 > > > To > ips@ietf.org > cc > > Subject > RE: [Ips] Recent > comments about > FCoE and iSCSI > > > > > > > > > The real debate here is between two types of networks. > The first is reliable at the link level and does not > drop packets under congestion. The second is running > a reliable transport protocol (i.e. TCP) over an > unreliable link level network. > > I agree with the scaling argument. For sufficiently > large networks, reliable link level doesn't work well > because network component failure, or chronically > congested links are not handled well. For > sufficiently small networks, reliable link level has > some significant advantages in simplicity, low > hardware cost, performance, and worst case latency. > > My personal view is that the vast majority of > enterprise storage networks fall in the "sufficiently > small" category. This view has to some extent been > vindicated by the continuing success of Fibre Channel > in this space and the inability of iSCSI to displace > FC in any significant way for enterprise storage. Of > course, this may or may not change in the future. > > Whether FC is simpler than iSCSI depends largely on > your definition of simplicity. If one defines > simplicity/complexity as the number of gates or lines > of code to reduce the protocol to hardware or > firmware, then my experience is that iSCSI is 2X to 3X > the complexity of FC. This has implications in cost > and reliability. > > Particularly problematic with iSCSI is the > unpredictability of the performance. Performance is > great with no packet drop. However even a small > amount of congestion can cause a sudden large drop and > performance. This can be difficult to predict as a > network that is almost but not quite congested can run > great, but a small incremental change of any sort can > cause the performance to become suddenly unacceptable. > For FC, or other protocol using link level flow > control, the reduction in performance is much more > graceful and incremental when the level of congestion > is small and intermittent. > > A second major problem with iSCSI is the unbounded > nature of worst case latency. When a storage network > fails, it is desirable to detect the failure in a > fraction of a second and transition to a backup > network. TCP, when implemented to the standards, can > take many seconds or minutes to determine that a > network has failed and close the connection. RFC > 2988, for instance, requires that the minimum > retransmission be one second. This means a single > dropped packet may add one second to the latency of > outstanding commands. This is a huge amount of time > on a 10G link. No doubt this could be mitigated by > drastically reducing the timeouts within TCP, but the > market seems to be surprisingly resistant to tampering > with accepted standards here. > > Overall, the FC and FCP protocol have a lot in common > with the Intel i86 instruction set architecture. They > are overly complex, and rather poorly designed by > modern standards. But they are good enough, and there > is a huge amount of value add that has been built on > top of them, and therefore little incentive to change. > FCoE is an interesting idea because it preserves 90% > of the existing value add of FC, unifies the physical > link with Ethernet, and uses the reliable link method > of packet delivery. > > There are two significant possibilities for iSCSI to > displace FC (or FCoE) in enterprise storage networks. > First is if the networks start to scale to large > enough size that FC can't be made sufficiently > reliable, and second if CPU compute cycles become > sufficiently cheap that the iSCSI protocol can be run > in host software with no negative performance impact. > Barring either of these, it seems that iSCSI will have > an uphill battle, and FCoE may have a place. > > -----Original Message----- > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > Sent: Tuesday, April 24, 2007 3:10 PM > To: ips@ietf.org > Subject: [Ips] Recent comments about FCoE and iSCSI > > > > Dear All, > > The trade press is lately full with comments about the > latest and greatest reincarnation of Fiber Channel > over ethernet. > It made me try and summarize all the long and hot > debates that preceded the advent of iSCSI. > Although FCoE proponents make it look like no debate > preceded iSCSI that was not so - FCoE was considered > even then and was dropped as a dumb idea. > > Here is a summary (as afar as I can remember) of the > main arguments. They are not bad arguments even in > retrospect and technically FCoE doesn't look better > than it did then. > > Feel free to use this material in a nay form. I expect > this group to seriously expand my arguments and make > them public - in personal or collective form. > > And do not forget - it is a technical dispute - > although we all must have some doubts about the way it > is pursued. > > Regards, > Julo > > --------------------------------------------------------------------- > > > What a piece of nostalgia :-) > > Around 1997 when a team at IBM Research (Haifa and > Almaden) started looking at connecting storage to > servers using the "regular network" (the ubiquitous > LAN) we considered many alternatives (another team > even had a look at ATM - still a computer network > candidate at the time). I won't get you over all of > our rationale (and we went over some of them again at > the end of 1999 with a team from CISCO before we > convened the first IETF BOF in 2000 at Adelaide that > resulted in iSCSI and all the rest) but some of the > reasons we choose to drop Fiber Channel over raw > Ethernet where multiple: > > Fiber Channel Protocol (SCSI over Fiber Channel Link) > is "mildly" effective because: > it implements endpoints in a dedicated engine > (Offload) > it has no transport layer (recovery is done at the > application layer under the assumption that the error > rate will be very low) > the network is limited in physical span and logical > span (number of switches) > flow-control/congestion control is achieved with a > mechanism adequate for a limited span network > (credits). The packet loss rate is almost nil and that > allows FCP to avoid using a transport (end-to-end) > layer > FCP she switches are simple (addresses are local and > the memory requirements cam be limited through the > credit mechanism) > However FCP endpoints are inherently costlier than > simple NICs – the cost argument (initiators are more > expensive) > The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the > network diameter limits) – the scaling argument > The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s – the > scaling argument > Ethernet has no credit mechanism and any mechanism > with a similar effect increases the end point cost. > Building a transport layer in the protocol stack has > always been the preferred choice of the networking > community – the community argument > The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances > in protocol stack implementation and finer tuning of > the congestion control mechanisms make conventional > TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant > on the computing scene have enough compute cycles > available to make any "offloading" possible as a mere > code restructuring exercise (see the stack reports > from Intel, IBM etc.) > Building on a complete stack makes available a wealth > of operational and management mechanisms built over > the years by the networking community (routing, > provisioning, security, service location etc.) – the > community argument > Higher level storage access over an IP network is > widely available and having both block and file served > over the same connection with the same support and > management structure is compelling – the community > argument > Highly efficient networks are easy to build over IP > with optimal (shortest path) routing while Layer 2 > networks use bridging and are limited by the logical > tree structure that bridges must follow. The effort to > combine routers and bridges (rbridges) is promising to > change that but it will take some time to finalize > (and we don't know exactly how it will operate). > Untill then the scale of Layer 2 network is going to > seriously limited – the scaling argument > > > As a side argument – a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later > iSCSI) to perform better than FCP at 1Gbs for block > sizes typical for OLTP (4-8KB). That was what > convinced us to take the path that lead to iSCSI – and > we used plain vanilla x86 servers with plain-vanilla > NICs and Linux (with similar measurements conducted on > Windows). > The networking and storage community acknowledged > those arguments and developed iSCSI and the companion > protocols for service discovery, boot etc. > > The community also acknowledged the need to support > existing infrastructure and extend it in a reasonable > fashion and developed 2 protocols iFCP (to support > hosts with FCP drivers and IP connections to connect > to storage by a simple conversion from FCP to TCP > packets) FCPIP to extend the reach of FCP through IP > (connects FCP islands through TCP links). Both have > been > implemented and their foundation is solid. > > The current attempt of developing a "new-age" FCP over > an Ethernet link is going against most of the > arguments that have given us iSCSI etc. > > It ignores the networking layering practice, build an > application protocol directly above a link and thus > limits scaling, mandates elements at the link layer > and application layer that make applications more > expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet). > > In some related effort (and at a point also when > developing iSCSI) we considered also moving away from > SCSI (like some "no standardized" but popular in some > circles software did – e.g., NBP) but decided against. > SCSI is a mature and well understood access > architecture for block storage and is implemented by > many device vendors. Moving away from it would not > have been justified at the time. > > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > > > _______________________________________________ > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips > > _______________________________________________ > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Fri Apr 27 08:34:54 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhPf3-0003vZ-6y; Fri, 27 Apr 2007 08:34:53 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhPf2-0003vU-Fs for ips-confirm+ok@megatron.ietf.org; Fri, 27 Apr 2007 08:34:52 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhPf2-0003vM-64 for ips@ietf.org; Fri, 27 Apr 2007 08:34:52 -0400 Received: from mtagate8.de.ibm.com ([195.212.29.157]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhPey-0008AP-D4 for ips@ietf.org; Fri, 27 Apr 2007 08:34:52 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate8.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3RCYjFQ038282 for ; Fri, 27 Apr 2007 12:34:45 GMT Received: from d12av04.megacenter.de.ibm.com (d12av04.megacenter.de.ibm.com [9.149.165.229]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3RCYjbI3788942 for ; Fri, 27 Apr 2007 14:34:45 +0200 Received: from d12av04.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3RCYjQ3020808 for ; Fri, 27 Apr 2007 14:34:45 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av04.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3RCYjMu020801; Fri, 27 Apr 2007 14:34:45 +0200 In-Reply-To: <1177648868.5355.122.camel@haakon2.linux-iscsi.org> To: nab@kernel.org MIME-Version: 1.0 Subject: RE: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Fri, 27 Apr 2007 08:34:40 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 27/04/2007 15:34:44, Serialize complete at 27/04/2007 15:34:44 X-Spam-Score: 0.5 (/) X-Scan-Signature: 510197464235b7be9252a74662931ef9 Cc: ips@ietf.org, Mike Mazarick , Zack Best , nab@linux-iscsi.org, Eric Hall X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0789995034==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0789995034== Content-Type: multipart/alternative; boundary="=_alternative 004515A9852572CA_=" This is a multipart message in MIME format. --=_alternative 004515A9852572CA_= Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Great comments. You are all certainly aware that sockets are also=20 undergoing transformation (asynchronous sockets) but even with synchronous = sockets and some care not to break existing application that use=20 synchronous sockets a restructuring of the stack may enable (as shown by=20 the Intel and IBM-Haifa work) great increases in performance. Software RDMA for the new class of of multicore engines is definetly an=20 interesting proposition (on highly multithreaded engines it should come=20 with not cost associated with it - or almost no cost). I wish I knew more about the decrease in latencies in the switch fabric=20 (it would be interesting if somebody could comment) as large Layer-2=20 fabrics have some inherent latency issues. FCoE is asking us to forget all athis and go back and pay the hardware=20 price for several more years and ignore the IP-land and nothing that I=20 heard convinced me that we should do so. Regards, Julo "Nicholas A. Bellinger" =20 27/04/07 00:41 Please respond to nab@kernel.org To Julian Satran/Haifa/IBM@IBMIL cc Zack Best , ips@ietf.org, nab@linux-iscsi.org, Mike=20 Mazarick , Eric Hall Subject RE: [Ips] Recent comments about FCoE and iSCSI A quick comment in regards to the abundance large of computing resources available for initiator side software IP storage services.. Also Julo, many thanks for posting this great thread. :) --nab ------------------------------------------------------------------------ As the progress of the DDP TWG continues onward and 2nd generation hardware iWARP engines start to come online, the benefit of a hybrid software implementation with host OS software network stack modifications in kernel above TCP and SCTP starts to pose a question..=20 What real savings can a hyrbid iSER nodes using software DDP? What are those changes required to make high performance software DDP a reality..? As osc-iwarp has found out, there is a significant CPU overhead assoicated with sockets and software VERBS, but I think this can be minimized with the right set of changes. Those changes are moving away from receieve side sockets for software iSER mode. These changes will start to become attractive for new product designs as this will allow RNIC hardware engines to scale futher using a more sane method or less painfully (depending on who you ask, OFA uses a Hybrid IB-VERBs) than traditional TOEs with speciality engines. Really taking advantage of what metadata in DDP and iWARP metadata is telling about the framed network transport can help in RDMA WRITE scenarios because the software RNIC would already have Stagged memory ready to go in the iSER case. Espically when it comes to the API for the iSER stack, having a single codebase with vendors writing hardware drivers instead of re-inventing the wheel with sockets. I believe the smart software RNICs of the future will direct RDMA traffic directly into host OS SCSI memory buffers, and like today use something similar to sendpage() for TX. As multi-core microprocessor designs with large, intelligent shared caches, and CPU cache coherentecy and I/O interconnects that in the 90's where only available in the Alpha EV67 and highest of high end shared memory supercomputers and clusters are now starting to become the norm. Pushing software iSER to the next level and beyond is surely not going to happen with a 30 year old API (sockets). Also for the data center story with a traditional tiered SAN architecture and software case, the hyrbid iWARP software stack on the initiator will not get a whole lot of interest until it can show improved performance and overhead that is acceptable to traditional iSCSI today. For the 3rd generation IP storage stacks, typical multiport 1G workloads is what will really drive interest into areas where putting a hardware RNIC will not be cost feasable for some time. But just as with traditional iSCSI, we can also scale software iSER down towards towards platforms with more modest computing resources on low-power, wireless devices. Even in the type of mobile devices that IP storage services have been prototyped on today, the benefit of being able to scale server side hardware RNICs more efficently is not software iSER's only benefit. On a side note, I think the transparency that connection recovery in traditional iSCSI and iSER allows to internexus multiplexing, as well as end user requirements for configuration and management scenarios. Using a active-active recovery mechinism that is as close to completely transparent as possbile (which ERL=3D2 is IMHO) is I think what mobile IP storage services users need to be demanding from their transports.=20 Thanks for listening! On Thu, 2007-04-26 at 21:16 -0400, Julian Satran wrote: >=20 > Excellent comments. My take (if not obvious from the previous text) is > that data centers will be very large and compute power (as evidenced > by the multicore) and advances in stack implementation are bound to > improve substantialy the performance of the protocol stacks (see Intel > and our work) and layer 3 switching.=20 > It is important also to point out that Ethernet has substantial > latencies if only bridging is using and replacement technologies (such > as Rbridges or others) may take some time to appear.=20 >=20 > Julo=20 >=20 >=20 > Zack Best =20 >=20 > 25/04/07 16:37=20 >=20 >=20 > To > ips@ietf.org=20 > cc >=20 > Subject > RE: [Ips] Recent > comments about > FCoE and iSCSI >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > The real debate here is between two types of networks. > The first is reliable at the link level and does not > drop packets under congestion. The second is running > a reliable transport protocol (i.e. TCP) over an > unreliable link level network. >=20 > I agree with the scaling argument. For sufficiently > large networks, reliable link level doesn't work well > because network component failure, or chronically > congested links are not handled well. For > sufficiently small networks, reliable link level has > some significant advantages in simplicity, low > hardware cost, performance, and worst case latency. >=20 > My personal view is that the vast majority of > enterprise storage networks fall in the "sufficiently > small" category. This view has to some extent been > vindicated by the continuing success of Fibre Channel > in this space and the inability of iSCSI to displace > FC in any significant way for enterprise storage. Of > course, this may or may not change in the future. >=20 > Whether FC is simpler than iSCSI depends largely on > your definition of simplicity. If one defines > simplicity/complexity as the number of gates or lines > of code to reduce the protocol to hardware or > firmware, then my experience is that iSCSI is 2X to 3X > the complexity of FC. This has implications in cost > and reliability. >=20 > Particularly problematic with iSCSI is the > unpredictability of the performance. Performance is > great with no packet drop. However even a small > amount of congestion can cause a sudden large drop and > performance. This can be difficult to predict as a > network that is almost but not quite congested can run > great, but a small incremental change of any sort can > cause the performance to become suddenly unacceptable. > For FC, or other protocol using link level flow > control, the reduction in performance is much more > graceful and incremental when the level of congestion > is small and intermittent. >=20 > A second major problem with iSCSI is the unbounded > nature of worst case latency. When a storage network > fails, it is desirable to detect the failure in a > fraction of a second and transition to a backup > network. TCP, when implemented to the standards, can > take many seconds or minutes to determine that a > network has failed and close the connection. RFC > 2988, for instance, requires that the minimum > retransmission be one second. This means a single > dropped packet may add one second to the latency of > outstanding commands. This is a huge amount of time > on a 10G link. No doubt this could be mitigated by > drastically reducing the timeouts within TCP, but the > market seems to be surprisingly resistant to tampering > with accepted standards here. >=20 > Overall, the FC and FCP protocol have a lot in common > with the Intel i86 instruction set architecture. They > are overly complex, and rather poorly designed by > modern standards. But they are good enough, and there > is a huge amount of value add that has been built on > top of them, and therefore little incentive to change. > FCoE is an interesting idea because it preserves 90% > of the existing value add of FC, unifies the physical > link with Ethernet, and uses the reliable link method > of packet delivery. >=20 > There are two significant possibilities for iSCSI to > displace FC (or FCoE) in enterprise storage networks.=20 > First is if the networks start to scale to large > enough size that FC can't be made sufficiently > reliable, and second if CPU compute cycles become > sufficiently cheap that the iSCSI protocol can be run > in host software with no negative performance impact.=20 > Barring either of these, it seems that iSCSI will have > an uphill battle, and FCoE may have a place. >=20 > -----Original Message----- > From: Julian Satran [mailto:Julian=5FSatran@il.ibm.com] > Sent: Tuesday, April 24, 2007 3:10 PM > To: ips@ietf.org > Subject: [Ips] Recent comments about FCoE and iSCSI >=20 >=20 >=20 > Dear All,=20 >=20 > The trade press is lately full with comments about the > latest and greatest reincarnation of Fiber Channel > over ethernet.=20 > It made me try and summarize all the long and hot > debates that preceded the advent of iSCSI.=20 > Although FCoE proponents make it look like no debate > preceded iSCSI that was not so - FCoE was considered > even then and was dropped as a dumb idea.=20 >=20 > Here is a summary (as afar as I can remember) of the > main arguments. They are not bad arguments even in > retrospect and technically FCoE doesn't look better > than it did then.=20 >=20 > Feel free to use this material in a nay form. I expect > this group to seriously expand my arguments and make > them public - in personal or collective form.=20 >=20 > And do not forget - it is a technical dispute - > although we all must have some doubts about the way it > is pursued.=20 >=20 > Regards,=20 > Julo=20 >=20 > --------------------------------------------------------------------- >=20 >=20 > What a piece of nostalgia :-)=20 >=20 > Around 1997 when a team at IBM Research (Haifa and > Almaden) started looking at connecting storage to > servers using the "regular network" (the ubiquitous > LAN) we considered many alternatives (another team > even had a look at ATM - still a computer network > candidate at the time). I won't get you over all of > our rationale (and we went over some of them again at > the end of 1999 with a team from CISCO before we > convened the first IETF BOF in 2000 at Adelaide that > resulted in iSCSI and all the rest) but some of the > reasons we choose to drop Fiber Channel over raw > Ethernet where multiple:=20 >=20 > Fiber Channel Protocol (SCSI over Fiber Channel Link) > is "mildly" effective because:=20 > it implements endpoints in a dedicated engine > (Offload)=20 > it has no transport layer (recovery is done at the > application layer under the assumption that the error > rate will be very low)=20 > the network is limited in physical span and logical > span (number of switches)=20 > flow-control/congestion control is achieved with a > mechanism adequate for a limited span network > (credits). The packet loss rate is almost nil and that > allows FCP to avoid using a transport (end-to-end) > layer > FCP she switches are simple (addresses are local and > the memory requirements cam be limited through the > credit mechanism)=20 > However FCP endpoints are inherently costlier than > simple NICs ? the cost argument (initiators are more > expensive)=20 > The credit mechanisms is highly unstable for large > networks (check switch vendors planning docs for the > network diameter limits) ? the scaling argument=20 > The assumption of low losses due to errors might > radically change when moving from 1 to 10 Gb/s ? the > scaling argument=20 > Ethernet has no credit mechanism and any mechanism > with a similar effect increases the end point cost. > Building a transport layer in the protocol stack has > always been the preferred choice of the networking > community ? the community argument=20 > The "performance penalty" of a complete protocol stack > has always been overstated (and overrated). Advances > in protocol stack implementation and finer tuning of > the congestion control mechanisms make conventional > TCP/IP performing well even at 10 Gb/s and over. > Moreover the multicore processors that become dominant > on the computing scene have enough compute cycles > available to make any "offloading" possible as a mere > code restructuring exercise (see the stack reports > from Intel, IBM etc.)=20 > Building on a complete stack makes available a wealth > of operational and management mechanisms built over > the years by the networking community (routing, > provisioning, security, service location etc.) ? the > community argument=20 > Higher level storage access over an IP network is > widely available and having both block and file served > over the same connection with the same support and > management structure is compelling ? the community > argument=20 > Highly efficient networks are easy to build over IP > with optimal (shortest path) routing while Layer 2 > networks use bridging and are limited by the logical > tree structure that bridges must follow. The effort to > combine routers and bridges (rbridges) is promising to > change that but it will take some time to finalize > (and we don't know exactly how it will operate). > Untill then the scale of Layer 2 network is going to > seriously limited ? the scaling argument >=20 >=20 > As a side argument ? a performance comparison made in > 1998 showed SCSI over TCP (a predecessor of the later > iSCSI) to perform better than FCP at 1Gbs for block > sizes typical for OLTP (4-8KB). That was what > convinced us to take the path that lead to iSCSI ? and > we used plain vanilla x86 servers with plain-vanilla > NICs and Linux (with similar measurements conducted on > Windows).=20 > The networking and storage community acknowledged > those arguments and developed iSCSI and the companion > protocols for service discovery, boot etc.=20 >=20 > The community also acknowledged the need to support > existing infrastructure and extend it in a reasonable > fashion and developed 2 protocols iFCP (to support > hosts with FCP drivers and IP connections to connect > to storage by a simple conversion from FCP to TCP > packets) FCPIP to extend the reach of FCP through IP > (connects FCP islands through TCP links). Both have > been=20 > implemented and their foundation is solid.=20 >=20 > The current attempt of developing a "new-age" FCP over > an Ethernet link is going against most of the > arguments that have given us iSCSI etc.=20 >=20 > It ignores the networking layering practice, build an > application protocol directly above a link and thus > limits scaling, mandates elements at the link layer > and application layer that make applications more > expensive and leaves aside the whole "ecosystem" that > accompanies TCP/IP (and not Ethernet).=20 >=20 > In some related effort (and at a point also when > developing iSCSI) we considered also moving away from > SCSI (like some "no standardized" but popular in some > circles software did ? e.g., NBP) but decided against. > SCSI is a mature and well understood access > architecture for block storage and is implemented by > many device vendors. Moving away from it would not > have been justified at the time.=20 >=20 >=20 > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around=20 > http://mail.yahoo.com=20 >=20 >=20 >=20 > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips >=20 > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > Ips mailing list > Ips@ietf.org > https://www1.ietf.org/mailman/listinfo/ips --=_alternative 004515A9852572CA_= Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable
Great comments. You are all certainly aware that sockets are also undergoing transformation (asynchronous sockets) but even with synchronous sockets and some care not to break existing appli= cation that use synchronous sockets a restructuring of the stack may enable (as shown by the Intel and IBM-Haifa work) great increases in performance.
Software RDMA for the new class of of multicore engines is definetly an interesting proposition (on highly multit= hreaded engines it should come with not cost associated with it - or almost no cost).

I wish I knew more about the decrease in latencies in the switch fabric (it would be interesting if somebody could comment) as large Layer-2 fabrics have some inherent latency issues.<= /font>

FCoE is asking us to forget all athis and go back and pay the hardware price for several more years and ignore the IP-land and nothing that I heard convinced me that we should do so.

Regards,
Julo


"Nicholas A. Bel= linger" <nab@kernel.org>

27/04/07 00:41
Please respond to
nab@kernel.org
To
Julian Satran/Haifa/IBM@IBMIL
cc
Zack Best <zbest28@yahoo.com>, ips@ietf.org, nab@linux-iscsi.org, Mike Mazarick <mazarick@bellsouth.net= >, Eric Hall <ehall@ehsco.com>
Subject
RE: [Ips] Recent comments about FCoE and iSCSI





A quick comment in regards to the abundance large of computing resources
available for initiator side software IP storage services..  Also Julo,
many thanks for posting this great thread. :)

--nab

------------------------------------------------------------------------

As the progress of the DDP TWG continues onward and 2nd generation
hardware iWARP engines start to come online, the benefit of a hybrid
software implementation with host OS software network stack
modifications in kernel above TCP and SCTP starts to pose a question..
What real savings can a hyrbid iSER nodes using software DDP?  What are
those changes required to make high performance software DDP a
reality..?

As osc-iwarp has found out, there is a significant CPU overhead
assoicated with sockets and software VERBS, but I think this can be
minimized with the right set of changes.  Those changes are moving away
from receieve side sockets for software iSER mode.  These changes will
start to become attractive for new product designs as this will allow
RNIC hardware engines to scale futher using a more sane method or less
painfully (depending on who you ask, OFA uses a Hybrid IB-VERBs) than
traditional TOEs with speciality engines.  Really taking advantage of
what metadata in DDP and iWARP metadata is telling about the framed
network transport can help in RDMA WRITE scenarios because the software
RNIC would already have Stagged memory ready to go in the iSER case.
Espically when it comes to the API for the iSER stack, having a single
codebase with vendors writing hardware drivers instead of re-inventing
the wheel with sockets.  I believe the smart software RNICs of the
future will direct RDMA traffic directly into host OS SCSI memory
buffers, and like today use something similar to sendpage() for TX.

As multi-core microprocessor designs with large, intelligent shared
caches, and CPU cache coherentecy and I/O interconnects that in the 90's
where only available in the Alpha EV67 and highest of high end shared
memory supercomputers and clusters are now starting to become the norm.
Pushing software iSER to the next level and beyond is surely not going
to happen with a 30 year old API (sockets).  Also for the data center<= br> story with a traditional tiered SAN architecture and software case, the
hyrbid iWARP software stack on the initiator will not get a whole lot of
interest until it can show improved performance and overhead that is
acceptable to traditional iSCSI today.  For the 3rd generation IP
storage stacks, typical multiport 1G workloads is what will really drive
interest into areas where putting a hardware RNIC will not be cost
feasable for some time.

But just as with traditional iSCSI, we can also scale software iSER down
towards towards platforms with more modest computing resources on
low-power, wireless devices.  Even in the type of mobile devices that IP
storage services have been prototyped on today, the benefit of being
able to scale server side hardware RNICs more efficently is not software
iSER's only benefit. On a side note, I think the transparency that
connection recovery in traditional iSCSI and iSER allows to internexus
multiplexing, as well as end user requirements for configuration and
management scenarios.  Using a active-active recovery mechinism that is
as close to completely transparent as possbile (which ERL=3D2 is IMHO) is I think what mobile IP storage services users need to be demanding from
their transports.

Thanks for listening!

On Thu, 2007-04-26 at 21:16 -0400, Julian Satran wrote:
>
> Excellent comments. My take (if not obvious from the previous text) is
> that data centers will be very large and compute power (as evidenced > by the multicore) and advances in stack implementation are bound to
> improve substantialy the performance of the protocol stacks (see Intel=
> and our work) and layer 3 switching.
> It is important also to point out that Ethernet has substantial
> latencies if only bridging is using and replacement technologies (such=
> as Rbridges or others) may take some time to appear.
>
> Julo
>
>
> Zack Best <zbest28@yahoo.com>
>
> 25/04/07 16:37
>
>
>                To
> ips@ietf.org
>                cc
>
>           Subject
> RE: [Ips] Recent
> comments about
> FCoE and iSCSI
>
>
>
>
>
>
>
>
> The real debate here is between two types of networks.
> The first is reliable at the link level and does not
> drop packets under congestion.  The second is running
> a reliable transport protocol (i.e. TCP) over an
> unreliable link level network.
>
> I agree with the scaling argument.  For sufficiently
> large networks, reliable link level doesn't work well
> because network component failure, or chronically
> congested links are not handled well.  For
> sufficiently small networks, reliable link level has
> some significant advantages in simplicity, low
> hardware cost, performance, and worst case latency.
>
> My personal view is that the vast majority of
> enterprise storage networks fall in the "sufficiently
> small" category.  This view has to some extent been
> vindicated by the continuing success of Fibre Channel
> in this space and the inability of iSCSI to displace
> FC in any significant way for enterprise storage.  Of
> course, this may or may not change in the future.
>
> Whether FC is simpler than iSCSI depends largely on
> your definition of simplicity.  If one defines
> simplicity/complexity as the number of gates or lines
> of code to reduce the protocol to hardware or
> firmware, then my experience is that iSCSI is 2X to 3X
> the complexity of FC.  This has implications in cost
> and reliability.
>
> Particularly problematic with iSCSI is the
> unpredictability of the performance.  Performance is
> great with no packet drop.  However even a small
> amount of congestion can cause a sudden large drop and
> performance.  This can be difficult to predict as a
> network that is almost but not quite congested can run
> great, but a small incremental change of any sort can
> cause the performance to become suddenly unacceptable.
> For FC, or other protocol using link level flow
> control, the reduction in performance is much more
> graceful and incremental when the level of congestion
> is small and intermittent.
>
> A second major problem with iSCSI is the unbounded
> nature of worst case latency.  When a storage network
> fails, it is desirable to detect the failure in a
> fraction of a second and transition to a backup
> network.  TCP, when implemented to the standards, can
> take many seconds or minutes to determine that a
> network has failed and close the connection.  RFC
> 2988, for instance, requires that the minimum
> retransmission be one second.  This means a single
> dropped packet may add one second to the latency of
> outstanding commands.  This is a huge amount of time
> on a 10G link.  No doubt this could be mitigated by
> drastically reducing the timeouts within TCP, but the
> market seems to be surprisingly resistant to tampering
> with accepted standards here.
>
> Overall, the FC and FCP protocol have a lot in common
> with the Intel i86 instruction set architecture.  They
> are overly complex, and rather poorly designed by
> modern standards.  But they are good enough, and there
> is a huge amount of value add that has been built on
> top of them, and therefore little incentive to change.
> FCoE is an interesting idea because it preserves 90%
> of the existing value add of FC, unifies the physical
> link with Ethernet, and uses the reliable link method
> of packet delivery.
>
> There are two significant possibilities for iSCSI to
> displace FC (or FCoE) in enterprise storage networks.
> First is if the networks start to scale to large
> enough size that FC can't be made sufficiently
> reliable, and second if CPU compute cycles become
> sufficiently cheap that the iSCSI protocol can be run
> in host software with no negative performance impact.
> Barring either of these, it seems that iSCSI will have
> an uphill battle, and FCoE may have a place.
>
> -----Original Message-----
> From: Julian Satran [mailto:Julian=5FSatran@il.ibm.com]
> Sent: Tuesday, April 24, 2007 3:10 PM
> To: ips@ietf.org
> Subject: [Ips] Recent comments about FCoE and iSCSI
>
>
>
> Dear All,
>
> The trade press is lately full with comments about the
> latest and greatest reincarnation of Fiber Channel
> over ethernet.
> It made me try and summarize all the long and hot
> debates that preceded the advent of iSCSI.
> Although FCoE proponents make it look like no debate
> preceded iSCSI that was not so - FCoE was considered
> even then and was dropped as a dumb idea.
>
> Here is a summary (as afar as I can remember) of the
> main arguments. They are not bad arguments even in
> retrospect and technically FCoE doesn't look better
> than it did then.
>
> Feel free to use this material in a nay form. I expect
> this group to seriously  expand my arguments and make
> them public - in personal or collective form.
>
> And do not forget - it is a technical dispute -
> although we all must have some doubts about the way it
> is pursued.
>
> Regards,
> Julo
>
> ---------------------------------------------------------------------<= br> >
>
> What a piece of nostalgia :-)
>
> Around 1997 when a team at IBM Research (Haifa and
> Almaden) started looking at connecting storage to
> servers using the "regular network" (the ubiquitous
> LAN) we considered many alternatives (another team
> even had a look at ATM - still a computer network
> candidate at the time). I won't get you over all of
> our rationale (and we went over some of them again at
> the end of 1999 with a team from CISCO before we
> convened the first IETF BOF in 2000 at Adelaide that
> resulted in iSCSI and all the rest) but some of the
> reasons we choose to drop Fiber Channel over raw
> Ethernet where multiple:
>
> Fiber Channel Protocol (SCSI over Fiber Channel Link)
> is "mildly" effective because:
> it implements endpoints in a dedicated engine
> (Offload)
> it has no transport layer (recovery is done at the
> application layer under the assumption that the error
> rate will be very low)
> the network is limited in physical span and logical
> span (number of switches)
> flow-control/congestion control is achieved with a
> mechanism adequate for a limited span network
> (credits). The packet loss rate is almost nil and that
> allows FCP to avoid using a transport (end-to-end)
> layer
> FCP she switches are simple (addresses are local and
> the memory requirements cam be limited through the
> credit mechanism)
> However FCP endpoints are inherently costlier than
> simple NICs – the cost argument (initiators are more
> expensive)
> The credit mechanisms is highly unstable for large
> networks (check switch vendors planning docs for the
> network diameter limits) – the scaling argument
> The assumption of low losses due to errors might
> radically change when moving from 1 to 10 Gb/s – the
> scaling argument
> Ethernet has no credit mechanism and any mechanism
> with a similar effect increases the end point cost.
> Building a transport layer in the protocol stack has
> always been the preferred choice of the networking
> community – the community argument
> The "performance penalty" of a complete protocol stack
> has always been overstated (and overrated). Advances
> in protocol stack implementation and finer tuning of
> the congestion control mechanisms make conventional
> TCP/IP performing well even at 10 Gb/s and over.
> Moreover the multicore processors that become dominant
> on the computing scene have enough compute cycles
> available to make any "offloading" possible as a mere
> code restructuring exercise (see the stack reports
> from Intel, IBM etc.)
> Building on a complete stack makes available a wealth
> of operational and management mechanisms built over
> the years by the networking community (routing,
> provisioning, security, service location etc.) – the
> community argument
> Higher level storage access over an IP network is
> widely available and having both block and file served
> over the same connection with the same support and
> management structure is compelling – the community
> argument
> Highly efficient networks are easy to build over IP
> with optimal (shortest path) routing while Layer 2
> networks use bridging and are limited by the logical
> tree structure that bridges must follow. The effort to
> combine routers and bridges (rbridges) is promising to
> change that but it will take some time to finalize
> (and we don't know exactly how it will operate).
> Untill then the scale of Layer 2 network is going to
> seriously limited – the scaling argument
>
>
> As a side argument – a performance comparison made in
> 1998 showed SCSI over TCP (a predecessor of the later
> iSCSI) to perform better than FCP at 1Gbs for block
> sizes typical for OLTP (4-8KB). That was what
> convinced us to take the path that lead to iSCSI – and
> we used plain vanilla x86 servers with plain-vanilla
> NICs and Linux (with similar measurements conducted on
> Windows).
> The networking and storage community acknowledged
> those arguments and developed iSCSI and the companion
> protocols for service discovery, boot etc.
>
> The community also acknowledged the need to support
> existing infrastructure and extend it in a reasonable
> fashion and developed 2 protocols iFCP (to support
> hosts with FCP drivers and IP connections to connect
> to storage by a simple conversion from FCP to TCP
> packets) FCPIP to extend the reach of FCP through IP
> (connects FCP islands through TCP links). Both have
> been
> implemented and their foundation is solid.
>
> The current attempt of developing a "new-age" FCP over
> an Ethernet link is going against most of the
> arguments that have given us iSCSI etc.
>
> It ignores the networking layering practice, build an
> application protocol directly above a link and thus
> limits scaling, mandates elements at the link layer
> and application layer that make applications more
> expensive and leaves aside the whole "ecosystem" that
> accompanies TCP/IP (and not Ethernet).
>
> In some related effort (and at a point also when
> developing iSCSI) we considered also moving away from
> SCSI (like some "no standardized" but popular in some
> circles software did – e.g., NBP) but decided against.
> SCSI is a mature and well understood access
> architecture for block storage and is implemented by
> many device vendors. Moving away from it would not
> have been justified at the time.
>
>
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
> Ips mailing list
> Ips@ietf.org
> https://www1.ietf.org/mailman/listinfo/ips
>
> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
> Ips mailing list
> Ips@ietf.org
> https://www1.ietf.org/mailman/listinfo/ips
--=_alternative 004515A9852572CA_=-- --===============0789995034== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0789995034==-- From ips-bounces@ietf.org Fri Apr 27 11:00:00 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhRvR-0008Cs-QR; Fri, 27 Apr 2007 10:59:57 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhRvP-00089d-Aw for ips-confirm+ok@megatron.ietf.org; Fri, 27 Apr 2007 10:59:55 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhRvO-00089J-TX for ips@ietf.org; Fri, 27 Apr 2007 10:59:54 -0400 Received: from nuova-ex1.nuovasystems.com ([67.91.200.196]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhRvN-0005cP-LN for ips@ietf.org; Fri, 27 Apr 2007 10:59:54 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [Ips] Recent comments about FCoE and iSCSI Date: Fri, 27 Apr 2007 07:59:43 -0700 Message-ID: <34BDD2A93E5FD84594BB230DD6C18EA2015E3DBD@nuova-ex1.hq.nuovaimpresa.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Ips] Recent comments about FCoE and iSCSI Thread-Index: AceIyKNm9y94N5dUSG+8aeqmdhCm6AADgJ6w References: <1177648868.5355.122.camel@haakon2.linux-iscsi.org> From: "Silvano Gai" To: "Julian Satran" , X-Spam-Score: 0.0 (/) X-Scan-Signature: 798b2e660f1819ae38035ac1d8d5e3ab Cc: Eric Hall , ips@ietf.org, Mike Mazarick , nab@linux-iscsi.org, Zack Best X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org Julo, > FCoE is asking us to forget all athis and go back and pay the hardware > price for several more years and ignore the IP-land and nothing that I > heard convinced me that we should do so.=20 FCoE is not asking you (the ips WG) anything. FCoE is a proposed item for the FC-BB-5 WG of T11. If you have concern that T11 is making a mistake, I suggest you move this discussion to the T11.3 reflector. The FC-BB WG will meet the first time to discuss FCoE in Bloomington, MN Wednesday June 6th, 2007.=20 IMHO, it is a bit premature to discuss the limitations of a technology that is not yet public or defined. -- Silvano _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Fri Apr 27 11:25:06 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhSJl-0003kc-NP; Fri, 27 Apr 2007 11:25:05 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh6AI-0002wR-TR for ips-confirm+ok@megatron.ietf.org; Thu, 26 Apr 2007 11:45:50 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1Hh6AI-0002wJ-K3 for ips@ietf.org; Thu, 26 Apr 2007 11:45:50 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Hh64f-0003uH-Ur for ips@ietf.org; Thu, 26 Apr 2007 11:40:01 -0400 Received: from smtp155.iad.emailsrvr.com ([207.97.245.155]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Hh64f-0002Dd-Dr for ips@ietf.org; Thu, 26 Apr 2007 11:40:01 -0400 Received: from minuteman (unknown [12.44.168.99]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: larry@alacritech.com) by relay5.relay.iad.emailsrvr.com (SMTP Server) with ESMTP id A851E5EC4E6; Thu, 26 Apr 2007 11:40:00 -0400 (EDT) From: "Larry Boucher" To: Subject: RE: [Ips] Recent comments about FCoE and iSCSI Date: Thu, 26 Apr 2007 08:39:56 -0700 Organization: Alacritech, Inc. Message-ID: <05f101c78819$26c51a80$500a010a@alacritech.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.6822 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3028 Thread-Index: AceICjomfv0BkG35T9eK27tUkjQdqQACUqrw In-Reply-To: Importance: Normal X-Spam-Score: 0.3 (/) X-Scan-Signature: 90e8b0e368115979782f8b3d811b226b X-TMDA-Confirmed: Thu, 26 Apr 2007 11:45:50 -0400 X-Mailman-Approved-At: Fri, 27 Apr 2007 11:25:04 -0400 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: larry@alacritech.com List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org While it is hard to disagree with the accuracy of your observations, = there are a couple of points where I disagree, and an overall observation that = I would make. First, you suggest that iSCSI is a problem with dropped = packets in a congested environment. If you are comparing apples to apples, then = the same congested environment in a FC or FCoE environment will bring things = to a halt. It would seem that lower performance is better than no = performance. In the perfect world required by FC/FCoE iSCSI performs beautifully. = This would seem to give an advantage to iSCSI. Your other major complaint involves long latencies caused by network failure. If this is a common occurrence in either environment, there = are much worse problems than latency issues. While your point may be valid, = it would seem to me that covering five sigma events is just not a = performance issue. FCoE is most easily related to UDP. In similar network environments to those for which FCoE is considered useful UDP is still much more = efficient than TCP. However, despite this efficiency, over time UDP has fallen = into disuse. Part of this is, as you suggest, driven by the increasing performance of processors that allowed them to saturate the network = running TCP, but there are issues with this that I will try to address in the following paragraph. More importantly as networking and the internet = have grown, UDP has found fewer appropriate environments, and managing two protocols to achieve the same result has become more expensive. This is despite the fact that UDP is still much more efficient than TCP in a = noise free environment. As far as CPU performance and protocol processing are concerned, I = believe that theirs is a much misunderstood relationship. The general purpose = CPU is designed to do general purpose functions. Any system that is not = moving much data should certainly make use of the CPU, however inefficient, to = move the data, as garbage collection is one of the strengths of the CPU. For = any system that has been designed for the specific purpose of moving data, = the general purpose CPU (irrespective of size or clock rate) is a very inefficient mechanism. Since the number of machines purchased for the specific purpose of moving data (file service, web service, backup, streaming, etc.) is of sufficient size, it would seem that specialized hardware designed to do this efficiently would be of value. While = hardware designed to move data via TCP is as you point out not simple, it is much more cost effective, and uses much less power, than a general purpose = CPU, and if you want real complexity, take a look at that CPU. It seems like iSCSI has the best of both worlds--simplicity through the use of a NIC = and CPU in low demand environments, and speed and efficiency via TOE in high demand environments. And all without having to change protocols. Larry Boucher From: Zack Best [mailto:zbest28@yahoo.com] Sent: Wednesday, April 25, 2007 1:38 PM To: ips@ietf.org Subject: RE: [Ips] Recent comments about FCoE and iSCSI The real debate here is between two types of networks. The first is reliable at the link level and does not drop packets under congestion. The second is running a reliable transport protocol (i.e. TCP) over an unreliable link level network. I agree with the scaling argument. For sufficiently large networks, reliable link level doesn't work well because network component failure, or chronically congested links are not handled well. For sufficiently small networks, reliable link level has some significant advantages in simplicity, low hardware cost, performance, and worst case latency. My personal view is that the vast majority of enterprise storage networks fall in the "sufficiently small" category. This view has to some extent been vindicated by the continuing success of Fibre Channel in this space and the inability of iSCSI to displace FC in any significant way for enterprise storage. Of course, this may or may not change in the future. Whether FC is simpler than iSCSI depends largely on your definition of simplicity. If one defines simplicity/complexity as the number of gates or lines of code to reduce the protocol to hardware or firmware, then my experience is that iSCSI is 2X to 3X the complexity of FC. This has implications in cost and reliability. Particularly problematic with iSCSI is the unpredictability of the performance. Performance is great with no packet drop. However even a small amount of congestion can cause a sudden large drop and performance. This can be difficult to predict as a network that is almost but not quite congested can run great, but a small incremental change of any sort can cause the performance to become suddenly unacceptable. For FC, or other protocol using link level flow control, the reduction in performance is much more graceful and incremental when the level of congestion is small and intermittent. A second major problem with iSCSI is the unbounded nature of worst case latency. When a storage network fails, it is desirable to detect the failure in a fraction of a second and transition to a backup network. TCP, when implemented to the standards, can take many seconds or minutes to determine that a network has failed and close the connection. RFC 2988, for instance, requires that the minimum retransmission be one second. This means a single dropped packet may add one second to the latency of outstanding commands. This is a huge amount of time on a 10G link. No doubt this could be mitigated by drastically reducing the timeouts within TCP, but the market seems to be surprisingly resistant to tampering with accepted standards here. Overall, the FC and FCP protocol have a lot in common with the Intel i86 instruction set architecture. They are overly complex, and rather poorly designed by modern standards. But they are good enough, and there is a huge amount of value add that has been built on top of them, and therefore little incentive to change. FCoE is an interesting idea because it preserves 90% of the existing value add of FC, unifies the physical link with Ethernet, and uses the reliable link method of packet delivery. There are two significant possibilities for iSCSI to displace FC (or FCoE) in enterprise storage networks. First is if the networks start to scale to large enough size that FC can't be made sufficiently reliable, and second if CPU compute cycles become sufficiently cheap that the iSCSI protocol can be run in host software with no negative performance impact. Barring either of these, it seems that iSCSI will have an uphill battle, and FCoE may have a place. -----Original Message----- From: Julian Satran [mailto:Julian_Satran@il.ibm.com] Sent: Tuesday, April 24, 2007 3:10 PM To: ips@ietf.org Subject: [Ips] Recent comments about FCoE and iSCSI Dear All, The trade press is lately full with comments about the latest and greatest reincarnation of Fiber Channel over ethernet. It made me try and summarize all the long and hot debates that preceded the advent of iSCSI. Although FCoE proponents make it look like no debate preceded iSCSI that was not so - FCoE was considered even then and was dropped as a dumb idea. Here is a summary (as afar as I can remember) of the main arguments. They are not bad arguments even in retrospect and technically FCoE doesn't look better than it did then. Feel free to use this material in a nay form. I expect this group to seriously expand my arguments and make them public - in personal or collective form. And do not forget - it is a technical dispute - although we all must have some doubts about the way it is pursued. Regards, Julo --------------------------------------------------------------------- What a piece of nostalgia :-) Around 1997 when a team at IBM Research (Haifa and Almaden) started looking at connecting storage to servers using the "regular network" (the ubiquitous LAN) we considered many alternatives (another team even had a look at ATM - still a computer network candidate at the time). I won't get you over all of our rationale (and we went over some of them again at the end of 1999 with a team from CISCO before we convened the first IETF BOF in 2000 at Adelaide that resulted in iSCSI and all the rest) but some of the reasons we choose to drop Fiber Channel over raw Ethernet where multiple: Fiber Channel Protocol (SCSI over Fiber Channel Link) is "mildly" effective because: it implements endpoints in a dedicated engine (Offload) it has no transport layer (recovery is done at the application layer under the assumption that the error rate will be very low) the network is limited in physical span and logical span (number of switches) flow-control/congestion control is achieved with a mechanism adequate for a limited span network (credits). The packet loss rate is almost nil and that allows FCP to avoid using a transport (end-to-end) layer FCP she switches are simple (addresses are local and the memory requirements cam be limited through the credit mechanism) However FCP endpoints are inherently costlier than simple NICs - the cost argument (initiators are more expensive) The credit mechanisms is highly unstable for large networks (check switch vendors planning docs for the network diameter limits) - the scaling argument The assumption of low losses due to errors might radically change when moving from 1 to 10 Gb/s - the scaling argument Ethernet has no credit mechanism and any mechanism with a similar effect increases the end point cost. Building a transport layer in the protocol stack has always been the preferred choice of the networking community - the community argument The "performance penalty" of a complete protocol stack has always been overstated (and overrated). Advances in protocol stack implementation and finer tuning of the congestion control mechanisms make conventional TCP/IP performing well even at 10 Gb/s and over. Moreover the multicore processors that become dominant on the computing scene have enough compute cycles available to make any "offloading" possible as a mere code restructuring exercise (see the stack reports from Intel, IBM etc.) Building on a complete stack makes available a wealth of operational and management mechanisms built over the years by the networking community (routing, provisioning, security, service location etc.) - the community argument Higher level storage access over an IP network is widely available and having both block and file served over the same connection with the same support and management structure is compelling - the community argument Highly efficient networks are easy to build over IP with optimal (shortest path) routing while Layer 2 networks use bridging and are limited by the logical tree structure that bridges must follow. The effort to combine routers and bridges (rbridges) is promising to change that but it will take some time to finalize (and we don't know exactly how it will operate). Untill then the scale of Layer 2 network is going to seriously limited - the scaling argument As a side argument - a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI - and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows). The networking and storage community acknowledged those arguments and developed iSCSI and the companion protocols for service discovery, boot etc. The community also acknowledged the need to support existing infrastructure and extend it in a reasonable fashion and developed 2 protocols iFCP (to support hosts with FCP drivers and IP connections to connect to storage by a simple conversion from FCP to TCP packets) FCPIP to extend the reach of FCP through IP (connects FCP islands through TCP links). Both have been implemented and their foundation is solid. The current attempt of developing a "new-age" FCP over an Ethernet link is going against most of the arguments that have given us iSCSI etc. It ignores the networking layering practice, build an application protocol directly above a link and thus limits scaling, mandates elements at the link layer and application layer that make applications more expensive and leaves aside the whole "ecosystem" that accompanies TCP/IP (and not Ethernet). In some related effort (and at a point also when developing iSCSI) we considered also moving away from SCSI (like some "no standardized" but popular in some circles software did - e.g., NBP) but decided against. SCSI is a mature and well understood access architecture for block storage and is implemented by many device vendors. Moving away from it would not have been justified at the time. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Fri Apr 27 11:59:40 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhSrD-0002kA-C7; Fri, 27 Apr 2007 11:59:39 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhSrC-0002k5-GQ for ips-confirm+ok@megatron.ietf.org; Fri, 27 Apr 2007 11:59:38 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhSrC-0002jx-6m for ips@ietf.org; Fri, 27 Apr 2007 11:59:38 -0400 Received: from mtagate4.de.ibm.com ([195.212.29.153]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhSrB-0001qS-Md for ips@ietf.org; Fri, 27 Apr 2007 11:59:38 -0400 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49]) by mtagate4.de.ibm.com (8.13.8/8.13.8) with ESMTP id l3RFxbqP088042 for ; Fri, 27 Apr 2007 15:59:37 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [9.149.165.228]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l3RFxavS3985466 for ; Fri, 27 Apr 2007 17:59:36 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id l3RFxa3B009631 for ; Fri, 27 Apr 2007 17:59:36 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [9.149.167.114]) by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id l3RFxaUH009624; Fri, 27 Apr 2007 17:59:36 +0200 In-Reply-To: <34BDD2A93E5FD84594BB230DD6C18EA2015E3DBD@nuova-ex1.hq.nuovaimpresa.com> To: "Silvano Gai" MIME-Version: 1.0 Subject: RE: [Ips] Recent comments about FCoE and iSCSI X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 From: Julian Satran Message-ID: Date: Fri, 27 Apr 2007 11:59:34 -0400 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 27/04/2007 18:59:36, Serialize complete at 27/04/2007 18:59:36 X-Spam-Score: 0.1 (/) X-Scan-Signature: f49c97ce49302a02285a2d36a99eef8c Cc: Eric Hall , Mike Mazarick , nab@linux-iscsi.org, nab@kernel.org, ips@ietf.org, Zack Best X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0762186336==" Errors-To: ips-bounces@ietf.org This is a multipart message in MIME format. --===============0762186336== Content-Type: multipart/alternative; boundary="=_alternative 0057D826852572CA_=" This is a multipart message in MIME format. --=_alternative 0057D826852572CA_= Content-Type: text/plain; charset="US-ASCII" Silavo, A technical discussion is a technical discussion. It is for this community to decide if it want to discuss it or not. As for moving it - it is again for tis team to decide. I would point out that having an Ethernet application layer being discussed in T11 is a first and although T11s charter is "interfaces" it has in the past done FCp (a protocol).. FC however is the "product" of an industry consortium. T11 will have to "stretch" the one of it's project subgroups (recharter) or create a new one. I assume that with the economic might of your supporters you may do that. Convincing the technical community that this is the right thing to do is a different story. Julo "Silvano Gai" 27/04/07 10:59 To Julian Satran/Haifa/IBM@IBMIL, cc , "Mike Mazarick" , "Zack Best" , , "Eric Hall" Subject RE: [Ips] Recent comments about FCoE and iSCSI Julo, > FCoE is asking us to forget all athis and go back and pay the hardware > price for several more years and ignore the IP-land and nothing that I > heard convinced me that we should do so. FCoE is not asking you (the ips WG) anything. FCoE is a proposed item for the FC-BB-5 WG of T11. If you have concern that T11 is making a mistake, I suggest you move this discussion to the T11.3 reflector. The FC-BB WG will meet the first time to discuss FCoE in Bloomington, MN Wednesday June 6th, 2007. IMHO, it is a bit premature to discuss the limitations of a technology that is not yet public or defined. -- Silvano --=_alternative 0057D826852572CA_= Content-Type: text/html; charset="US-ASCII"
Silavo,

A technical discussion is a technical discussion. It is for this community to decide if it want to discuss it or not.
As for moving it - it is again for tis team to decide. I would point out that having an Ethernet application layer being discussed in T11 is a first and although T11s charter  is "interfaces" it has in the past done FCp (a protocol).. FC however is the "product" of an industry consortium. T11 will have to "stretch" the one of it's project subgroups (recharter) or create a new one. I assume that with the economic might of your supporters you may do that. Convincing the technical community that this is the right thing to do is a different story.

Julo


"Silvano Gai" <sgai@nuovasystems.com>

27/04/07 10:59

To
Julian Satran/Haifa/IBM@IBMIL, <nab@kernel.org>
cc
<ips@ietf.org>, "Mike Mazarick" <mazarick@bellsouth.net>, "Zack Best" <zbest28@yahoo.com>, <nab@linux-iscsi.org>, "Eric Hall" <ehall@ehsco.com>
Subject
RE: [Ips] Recent comments about FCoE and iSCSI






Julo,


> FCoE is asking us to forget all athis and go back and pay the hardware
> price for several more years and ignore the IP-land and nothing that I
> heard convinced me that we should do so.

FCoE is not asking you (the ips WG) anything.

FCoE is a proposed item for the FC-BB-5 WG of T11. If you have concern
that T11 is making a mistake, I suggest you move this discussion to the
T11.3 reflector.

The FC-BB WG will meet the first time to discuss FCoE in Bloomington, MN
Wednesday June 6th, 2007.

IMHO, it is a bit premature to discuss the limitations of a technology
that is not yet public or defined.

-- Silvano
--=_alternative 0057D826852572CA_=-- --===============0762186336== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0762186336==-- From ips-bounces@ietf.org Fri Apr 27 17:25:37 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhXuy-0005Kq-NY; Fri, 27 Apr 2007 17:23:52 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HhXux-0005Kk-6q for ips-confirm+ok@megatron.ietf.org; Fri, 27 Apr 2007 17:23:51 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HhXuw-0005KU-TR for ips@ietf.org; Fri, 27 Apr 2007 17:23:50 -0400 Received: from stork.istor.com ([12.170.66.34] helo=stork.inside.istor.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HhXuv-0001J2-JV for ips@ietf.org; Fri, 27 Apr 2007 17:23:50 -0400 Received: from mercury.inside.istor.com ([192.168.50.30]) by stork.inside.istor.com with ESMTP; 27 Apr 2007 14:23:49 -0700 X-IronPort-AV: i="4.14,463,1170662400"; d="scan'208"; a="3258457:sNHT16197950" X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 27 Apr 2007 14:23:48 -0700 Message-ID: <9F3F0A752CAEBE4FA7E906CC2FBFF57C06D444@MERCURY.inside.istor.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: StatSN question Thread-Index: AceJElgYCioIih3MQYOUhYbQpASkaQ== From: "Ken Craig" To: X-Spam-Score: 0.0 (/) X-Scan-Signature: 1ac7cc0a4cd376402b85bc1961a86ac2 Subject: [Ips] StatSN question X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: ips-bounces@ietf.org I have a question about what ExpStatSN means that I can't find an answer for in the reflector archives or in any of the RFCs. I, as an iSCSI Target running at ERL=3D0, send a DATA IN PDU with FINAL=3D1 and a StatSN of 0. I receive a SCSI CMD PDU with an ExpStatSN of 1. Does this mean that the Initiator has received the PDU BHS and all of the data or is it simply an acknowledgement that it has received the PDU BHS? Thanks, Ken Craig _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips From ips-bounces@ietf.org Mon Apr 30 15:21:28 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HibR5-0003ET-4I; Mon, 30 Apr 2007 15:21:23 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HibR3-0003EJ-T5 for ips-confirm+ok@megatron.ietf.org; Mon, 30 Apr 2007 15:21:21 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HibR3-0003EB-J0 for ips@ietf.org; Mon, 30 Apr 2007 15:21:21 -0400 Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HibR0-00073G-Tz for ips@ietf.org; Mon, 30 Apr 2007 15:21:21 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.13.191.130]) by palrel10.hp.com (Postfix) with ESMTP id 1C18534A22; Mon, 30 Apr 2007 12:21:14 -0700 (PDT) Received: from MK73191c.cup.hp.com (je061164.ssr.hp.com [15.47.61.164]) by esmail.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id MAA04887; Mon, 30 Apr 2007 12:14:58 -0700 (PDT) Message-Id: <6.2.0.14.2.20070430082328.02812c58@esmail.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.0.14 Date: Mon, 30 Apr 2007 08:26:56 -0700 To: Julian Satran , nab@kernel.org From: Michael Krause Subject: RE: [Ips] Recent comments about FCoE and iSCSI In-Reply-To: References: <1177648868.5355.122.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 X-Spam-Score: 0.5 (/) X-Scan-Signature: 16a1775db2061587296285ba70384116 Cc: Eric Hall , ips@ietf.org, Mike Mazarick , nab@linux-iscsi.org, Zack Best X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0827729375==" Errors-To: ips-bounces@ietf.org --===============0827729375== Content-Type: multipart/alternative; boundary="=====================_172001485==.ALT" --=====================_172001485==.ALT Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Just a data point: The OpenGroup did specify an async sockets API about a year or so ago. It= =20 was designed with RDMA interconnects / solutions in mind so provides=20 explicit memory management, completion events, etc. Unfortunately, while= =20 it was developed by a number of people with extensive Sockets development=20 and implementation experience, this specification has yet to be implemented= =20 not because of a lack of interest but more due to business priorities. In any case, there were also proposed extensions to eliminate the need for= =20 a ULP such as SDP as well through a few explicit RDMA calls. People might= =20 want to check out the work that has already occurred. There was a great=20 deal of thought put into that API to support both IPC, block and filer= storage. Mike At 05:34 AM 4/27/2007, Julian Satran wrote: >Great comments. You are all certainly aware that sockets are also=20 >undergoing transformation (asynchronous sockets) but even with synchronous= =20 >sockets and some care not to break existing application that use=20 >synchronous sockets a restructuring of the stack may enable (as shown by=20 >the Intel and IBM-Haifa work) great increases in performance. >Software RDMA for the new class of of multicore engines is definetly an=20 >interesting proposition (on highly multithreaded engines it should come=20 >with not cost associated with it - or almost no cost). > >I wish I knew more about the decrease in latencies in the switch fabric=20 >(it would be interesting if somebody could comment) as large Layer-2=20 >fabrics have some inherent latency issues. > >FCoE is asking us to forget all athis and go back and pay the hardware=20 >price for several more years and ignore the IP-land and nothing that I=20 >heard convinced me that we should do so. > >Regards, >Julo > > >"Nicholas A. Bellinger" > >27/04/07 00:41 >Please respond to >nab@kernel.org > >To >Julian Satran/Haifa/IBM@IBMIL >cc >Zack Best , ips@ietf.org, nab@linux-iscsi.org, Mike=20 >Mazarick , Eric Hall >Subject >RE: [Ips] Recent comments about FCoE and iSCSI > > > > >A quick comment in regards to the abundance large of computing resources >available for initiator side software IP storage services.. Also Julo, >many thanks for posting this great thread. :) > >--nab > >------------------------------------------------------------------------ > >As the progress of the DDP TWG continues onward and 2nd generation >hardware iWARP engines start to come online, the benefit of a hybrid >software implementation with host OS software network stack >modifications in kernel above TCP and SCTP starts to pose a question.. >What real savings can a hyrbid iSER nodes using software DDP? What are >those changes required to make high performance software DDP a >reality..? > >As osc-iwarp has found out, there is a significant CPU overhead >assoicated with sockets and software VERBS, but I think this can be >minimized with the right set of changes. Those changes are moving away >from receieve side sockets for software iSER mode. These changes will >start to become attractive for new product designs as this will allow >RNIC hardware engines to scale futher using a more sane method or less >painfully (depending on who you ask, OFA uses a Hybrid IB-VERBs) than >traditional TOEs with speciality engines. Really taking advantage of >what metadata in DDP and iWARP metadata is telling about the framed >network transport can help in RDMA WRITE scenarios because the software >RNIC would already have Stagged memory ready to go in the iSER case. >Espically when it comes to the API for the iSER stack, having a single >codebase with vendors writing hardware drivers instead of re-inventing >the wheel with sockets. I believe the smart software RNICs of the >future will direct RDMA traffic directly into host OS SCSI memory >buffers, and like today use something similar to sendpage() for TX. > >As multi-core microprocessor designs with large, intelligent shared >caches, and CPU cache coherentecy and I/O interconnects that in the 90's >where only available in the Alpha EV67 and highest of high end shared >memory supercomputers and clusters are now starting to become the norm. >Pushing software iSER to the next level and beyond is surely not going >to happen with a 30 year old API (sockets). Also for the data center >story with a traditional tiered SAN architecture and software case, the >hyrbid iWARP software stack on the initiator will not get a whole lot of >interest until it can show improved performance and overhead that is >acceptable to traditional iSCSI today. For the 3rd generation IP >storage stacks, typical multiport 1G workloads is what will really drive >interest into areas where putting a hardware RNIC will not be cost >feasable for some time. > >But just as with traditional iSCSI, we can also scale software iSER down >towards towards platforms with more modest computing resources on >low-power, wireless devices. Even in the type of mobile devices that IP >storage services have been prototyped on today, the benefit of being >able to scale server side hardware RNICs more efficently is not software >iSER's only benefit. On a side note, I think the transparency that >connection recovery in traditional iSCSI and iSER allows to internexus >multiplexing, as well as end user requirements for configuration and >management scenarios. Using a active-active recovery mechinism that is >as close to completely transparent as possbile (which ERL=3D2 is IMHO) is >I think what mobile IP storage services users need to be demanding from >their transports. > >Thanks for listening! > >On Thu, 2007-04-26 at 21:16 -0400, Julian Satran wrote: > > > > Excellent comments. My take (if not obvious from the previous text) is > > that data centers will be very large and compute power (as evidenced > > by the multicore) and advances in stack implementation are bound to > > improve substantialy the performance of the protocol stacks (see Intel > > and our work) and layer 3 switching. > > It is important also to point out that Ethernet has substantial > > latencies if only bridging is using and replacement technologies (such > > as Rbridges or others) may take some time to appear. > > > > Julo > > > > > > Zack Best > > > > 25/04/07 16:37 > > > > > > To > > ips@ietf.org > > cc > > > > Subject > > RE: [Ips] Recent > > comments about > > FCoE and iSCSI > > > > > > > > > > > > > > > > > > The real debate here is between two types of networks. > > The first is reliable at the link level and does not > > drop packets under congestion. The second is running > > a reliable transport protocol (i.e. TCP) over an > > unreliable link level network. > > > > I agree with the scaling argument. For sufficiently > > large networks, reliable link level doesn't work well > > because network component failure, or chronically > > congested links are not handled well. For > > sufficiently small networks, reliable link level has > > some significant advantages in simplicity, low > > hardware cost, performance, and worst case latency. > > > > My personal view is that the vast majority of > > enterprise storage networks fall in the "sufficiently > > small" category. This view has to some extent been > > vindicated by the continuing success of Fibre Channel > > in this space and the inability of iSCSI to displace > > FC in any significant way for enterprise storage. Of > > course, this may or may not change in the future. > > > > Whether FC is simpler than iSCSI depends largely on > > your definition of simplicity. If one defines > > simplicity/complexity as the number of gates or lines > > of code to reduce the protocol to hardware or > > firmware, then my experience is that iSCSI is 2X to 3X > > the complexity of FC. This has implications in cost > > and reliability. > > > > Particularly problematic with iSCSI is the > > unpredictability of the performance. Performance is > > great with no packet drop. However even a small > > amount of congestion can cause a sudden large drop and > > performance. This can be difficult to predict as a > > network that is almost but not quite congested can run > > great, but a small incremental change of any sort can > > cause the performance to become suddenly unacceptable. > > For FC, or other protocol using link level flow > > control, the reduction in performance is much more > > graceful and incremental when the level of congestion > > is small and intermittent. > > > > A second major problem with iSCSI is the unbounded > > nature of worst case latency. When a storage network > > fails, it is desirable to detect the failure in a > > fraction of a second and transition to a backup > > network. TCP, when implemented to the standards, can > > take many seconds or minutes to determine that a > > network has failed and close the connection. RFC > > 2988, for instance, requires that the minimum > > retransmission be one second. This means a single > > dropped packet may add one second to the latency of > > outstanding commands. This is a huge amount of time > > on a 10G link. No doubt this could be mitigated by > > drastically reducing the timeouts within TCP, but the > > market seems to be surprisingly resistant to tampering > > with accepted standards here. > > > > Overall, the FC and FCP protocol have a lot in common > > with the Intel i86 instruction set architecture. They > > are overly complex, and rather poorly designed by > > modern standards. But they are good enough, and there > > is a huge amount of value add that has been built on > > top of them, and therefore little incentive to change. > > FCoE is an interesting idea because it preserves 90% > > of the existing value add of FC, unifies the physical > > link with Ethernet, and uses the reliable link method > > of packet delivery. > > > > There are two significant possibilities for iSCSI to > > displace FC (or FCoE) in enterprise storage networks. > > First is if the networks start to scale to large > > enough size that FC can't be made sufficiently > > reliable, and second if CPU compute cycles become > > sufficiently cheap that the iSCSI protocol can be run > > in host software with no negative performance impact. > > Barring either of these, it seems that iSCSI will have > > an uphill battle, and FCoE may have a place. > > > > -----Original Message----- > > From: Julian Satran [mailto:Julian_Satran@il.ibm.com] > > Sent: Tuesday, April 24, 2007 3:10 PM > > To: ips@ietf.org > > Subject: [Ips] Recent comments about FCoE and iSCSI > > > > > > > > Dear All, > > > > The trade press is lately full with comments about the > > latest and greatest reincarnation of Fiber Channel > > over ethernet. > > It made me try and summarize all the long and hot > > debates that preceded the advent of iSCSI. > > Although FCoE proponents make it look like no debate > > preceded iSCSI that was not so - FCoE was considered > > even then and was dropped as a dumb idea. > > > > Here is a summary (as afar as I can remember) of the > > main arguments. They are not bad arguments even in > > retrospect and technically FCoE doesn't look better > > than it did then. > > > > Feel free to use this material in a nay form. I expect > > this group to seriously expand my arguments and make > > them public - in personal or collective form. > > > > And do not forget - it is a technical dispute - > > although we all must have some doubts about the way it > > is pursued. > > > > Regards, > > Julo > > > > --------------------------------------------------------------------- > > > > > > What a piece of nostalgia :-) > > > > Around 1997 when a team at IBM Research (Haifa and > > Almaden) started looking at connecting storage to > > servers using the "regular network" (the ubiquitous > > LAN) we considered many alternatives (another team > > even had a look at ATM - still a computer network > > candidate at the time). I won't get you over all of > > our rationale (and we went over some of them again at > > the end of 1999 with a team from CISCO before we > > convened the first IETF BOF in 2000 at Adelaide that > > resulted in iSCSI and all the rest) but some of the > > reasons we choose to drop Fiber Channel over raw > > Ethernet where multiple: > > > > Fiber Channel Protocol (SCSI over Fiber Channel Link) > > is "mildly" effective because: > > it implements endpoints in a dedicated engine > > (Offload) > > it has no transport layer (recovery is done at the > > application layer under the assumption that the error > > rate will be very low) > > the network is limited in physical span and logical > > span (number of switches) > > flow-control/congestion control is achieved with a > > mechanism adequate for a limited span network > > (credits). The packet loss rate is almost nil and that > > allows FCP to avoid using a transport (end-to-end) > > layer > > FCP she switches are simple (addresses are local and > > the memory requirements cam be limited through the > > credit mechanism) > > However FCP endpoints are inherently costlier than > > simple NICs =96 the cost argument (initiators are more > > expensive) > > The credit mechanisms is highly unstable for large > > networks (check switch vendors planning docs for the > > network diameter limits) =96 the scaling argument > > The assumption of low losses due to errors might > > radically change when moving from 1 to 10 Gb/s =96 the > > scaling argument > > Ethernet has no credit mechanism and any mechanism > > with a similar effect increases the end point cost. > > Building a transport layer in the protocol stack has > > always been the preferred choice of the networking > > community =96 the community argument > > The "performance penalty" of a complete protocol stack > > has always been overstated (and overrated). Advances > > in protocol stack implementation and finer tuning of > > the congestion control mechanisms make conventional > > TCP/IP performing well even at 10 Gb/s and over. > > Moreover the multicore processors that become dominant > > on the computing scene have enough compute cycles > > available to make any "offloading" possible as a mere > > code restructuring exercise (see the stack reports > > from Intel, IBM etc.) > > Building on a complete stack makes available a wealth > > of operational and management mechanisms built over > > the years by the networking community (routing, > > provisioning, security, service location etc.) =96 the > > community argument > > Higher level storage access over an IP network is > > widely available and having both block and file served > > over the same connection with the same support and > > management structure is compelling =96 the community > > argument > > Highly efficient networks are easy to build over IP > > with optimal (shortest path) routing while Layer 2 > > networks use bridging and are limited by the logical > > tree structure that bridges must follow. The effort to > > combine routers and bridges (rbridges) is promising to > > change that but it will take some time to finalize > > (and we don't know exactly how it will operate). > > Untill then the scale of Layer 2 network is going to > > seriously limited =96 the scaling argument > > > > > > As a side argument =96 a performance comparison made in > > 1998 showed SCSI over TCP (a predecessor of the later > > iSCSI) to perform better than FCP at 1Gbs for block > > sizes typical for OLTP (4-8KB). That was what > > convinced us to take the path that lead to iSCSI =96 and > > we used plain vanilla x86 servers with plain-vanilla > > NICs and Linux (with similar measurements conducted on > > Windows). > > The networking and storage community acknowledged > > those arguments and developed iSCSI and the companion > > protocols for service discovery, boot etc. > > > > The community also acknowledged the need to support > > existing infrastructure and extend it in a reasonable > > fashion and developed 2 protocols iFCP (to support > > hosts with FCP drivers and IP connections to connect > > to storage by a simple conversion from FCP to TCP > > packets) FCPIP to extend the reach of FCP through IP > > (connects FCP islands through TCP links). Both have > > been > > implemented and their foundation is solid. > > > > The current attempt of developing a "new-age" FCP over > > an Ethernet link is going against most of the > > arguments that have given us iSCSI etc. > > > > It ignores the networking layering practice, build an > > application protocol directly above a link and thus > > limits scaling, mandates elements at the link layer > > and application layer that make applications more > > expensive and leaves aside the whole "ecosystem" that > > accompanies TCP/IP (and not Ethernet). > > > > In some related effort (and at a point also when > > developing iSCSI) we considered also moving away from > > SCSI (like some "no standardized" but popular in some > > circles software did =96 e.g., NBP) but decided against. > > SCSI is a mature and well understood access > > architecture for block storage and is implemented by > > many device vendors. Moving away from it would not > > have been justified at the time. > > > > > > __________________________________________________ > > Do You Yahoo!? > > Tired of spam? Yahoo! Mail has the best spam protection around > > http://mail.yahoo.com > > > > > > > > _______________________________________________ > > Ips mailing list > > Ips@ietf.org > > https://www1.ietf.org/mailman/listinfo/ips > > > > _______________________________________________ > > Ips mailing list > > Ips@ietf.org > > https://www1.ietf.org/mailman/listinfo/ips > > >_______________________________________________ >Ips mailing list >Ips@ietf.org >https://www1.ietf.org/mailman/listinfo/ips --=====================_172001485==.ALT Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Just a data point:

The OpenGroup did specify an async sockets API about a year or so ago.   It was designed with RDMA interconnects / solutions in mind so provides explicit memory management, completion events, etc.   Unfortunately, while it was developed by a number of people with extensive Sockets development and implementation experience, this specification has yet to be implemented not because of a lack of interest but more due to business priorities.  

In any case, there were also proposed extensions to eliminate the need for a ULP such as SDP as well through a few explicit RDMA calls.   People might want to check out the work that has already occurred.  There was a great deal of thought put into that API to support both IPC, block and filer storage.

Mike


At 05:34 AM 4/27/2007, Julian Satran wrote:

Great= comments. You are all certainly aware that sockets are also undergoing transformation (asynchronous sockets) but even with synchronous sockets and some care not to break existing application that use synchronous sockets a restructuring of the stack may enable (as shown by the Intel and IBM-Haifa work) great increases in performance.
Software RDMA for the new class of of multicore engines is definetly an interesting proposition (on highly multithreaded engines it should come with not cost associated with it - or almost no cost).

I wish I knew more about the decrease in latencies in the switch fabric (it would be interesting if somebody could comment) as large Layer-2 fabrics have some inherent latency issues.

FCoE is asking us to forget all athis and go back and pay the hardware price for several more years and ignore the IP-land and nothing that I heard convinced me that we should do so.

Regards,
Julo


"Nicholas A. Bellinger" <nab@kernel.org>

27/04/07 00:41
Please respond to
nab@kernel.org

To
Julian Satran/Haifa/IBM@IBMIL
cc
Zack Best <zbest28@yahoo.com>, ips@ietf.org, nab@linux-iscsi.org, Mike Mazarick <mazarick@bellsouth.net>, Eric Hall <ehall@ehsco.com>
Subject
RE: [Ips] Recent comments about FCoE and iSCSI




A quick comment in regards to the abundance large of computing resources
available for initiator side software IP storage services..  Also Julo,
many thanks for posting this great thread. :)

--nab

------------------------------------------------------------------------

As the progress of the DDP TWG continues onward and 2nd generation
hardware iWARP engines start to come online, the benefit of a hybrid
software implementation with host OS software network stack
modifications in kernel above TCP and SCTP starts to pose a question..
What real savings can a hyrbid iSER nodes using software DDP?  What are
those changes required to make high performance software DDP a
reality..?

As osc-iwarp has found out, there is a significant CPU overhead
assoicated with sockets and software VERBS, but I think this can be
minimized with the right set of changes.  Those changes are moving away
from receieve side sockets for software iSER mode.  These changes will
start to become attractive for new product designs as this will allow
RNIC hardware engines to scale futher using a more sane method or less
painfully (depending on who you ask, OFA uses a Hybrid IB-VERBs) than
traditional TOEs with speciality engines.  Really taking advantage of
what metadata in DDP and iWARP metadata is telling about the framed
network transport can help in RDMA WRITE scenarios because the software
RNIC would already have Stagged memory ready to go in the iSER case.
Espically when it comes to the API for the iSER stack, having a single
codebase with vendors writing hardware drivers instead of re-inventing
the wheel with sockets.  I believe the smart software RNICs of the
future will direct RDMA traffic directly into host OS SCSI memory
buffers, and like today use something similar to sendpage() for TX.

As multi-core microprocessor designs with large, intelligent shared
caches, and CPU cache coherentecy and I/O interconnects that in the 90's
where only available in the Alpha EV67 and highest of high end shared
memory supercomputers and clusters are now starting to become the norm.
Pushing software iSER to the next level and beyond is surely not going
to happen with a 30 year old API (sockets).  Also for the data center
story with a traditional tiered SAN architecture and software case, the
hyrbid iWARP software stack on the initiator will not get a whole lot of
interest until it can show improved performance and overhead that is
acceptable to traditional iSCSI today.  For the 3rd generation IP
storage stacks, typical multiport 1G workloads is what will really drive
interest into areas where putting a hardware RNIC will not be cost
feasable for some time.

But just as with traditional iSCSI, we can also scale software iSER down
towards towards platforms with more modest computing resources on
low-power, wireless devices.  Even in the type of mobile devices that IP
storage services have been prototyped on today, the benefit of being
able to scale server side hardware RNICs more efficently is not software
iSER's only benefit. On a side note, I think the transparency that
connection recovery in traditional iSCSI and iSER allows to internexus
multiplexing, as well as end user requirements for configuration and
management scenarios.  Using a active-active recovery mechinism that is
as close to completely transparent as possbile (which ERL=3D2 is IMHO) is
I think what mobile IP storage services users need to be demanding from
their transports.

Thanks for listening!

On Thu, 2007-04-26 at 21:16 -0400, Julian Satran wrote:
>
> Excellent comments. My take (if not obvious from the previous text) is
> that data centers will be very large and compute power (as evidenced
> by the multicore) and advances in stack implementation are bound to
> improve substantialy the performance of the protocol stacks (see Intel
> and our work) and layer 3 switching.
> It is important also to point out that Ethernet has substantial
> latencies if only bridging is using and replacement technologies (such
> as Rbridges or others) may take some time to appear.
>
> Julo
>
>
> Zack Best <zbest28@yahoo.com>
>
> 25/04/07 16:37
>
>
>            =     To
> ips@ietf.org
>            =     cc
>
>           Subject
> RE: [Ips] Recent
> comments about
> FCoE and iSCSI
>
>
>
>
>
>
>
>
> The real debate here is between two types of networks.
> The first is reliable at the link level and does not
> drop packets under congestion.  The second is running
> a reliable transport protocol (i.e. TCP) over an
> unreliable link level network.
>
> I agree with the scaling argument.  For sufficiently
> large networks, reliable link level doesn't work well
> because network component failure, or chronically
> congested links are not handled well.  For
> sufficiently small networks, reliable link level has
> some significant advantages in simplicity, low
> hardware cost, performance, and worst case latency.
>
> My personal view is that the vast majority of
> enterprise storage networks fall in the "sufficiently
> small" category.  This view has to some extent been
> vindicated by the continuing success of Fibre Channel
> in this space and the inability of iSCSI to displace
> FC in any significant way for enterprise storage.  Of
> course, this may or may not change in the future.
>
> Whether FC is simpler than iSCSI depends largely on
> your definition of simplicity.  If one defines
> simplicity/complexity as the number of gates or lines
> of code to reduce the protocol to hardware or
> firmware, then my experience is that iSCSI is 2X to 3X
> the complexity of FC.  This has implications in cost
> and reliability.
>
> Particularly problematic with iSCSI is the
> unpredictability of the performance.  Performance is
> great with no packet drop.  However even a small
> amount of congestion can cause a sudden large drop and
> performance.  This can be difficult to predict as a
> network that is almost but not quite congested can run
> great, but a small incremental change of any sort can
> cause the performance to become suddenly unacceptable.
> For FC, or other protocol using link level flow
> control, the reduction in performance is much more
> graceful and incremental when the level of congestion
> is small and intermittent.
>
> A second major problem with iSCSI is the unbounded
> nature of worst case latency.  When a storage network
> fails, it is desirable to detect the failure in a
> fraction of a second and transition to a backup
> network.  TCP, when implemented to the standards, can
> take many seconds or minutes to determine that a
> network has failed and close the connection.  RFC
> 2988, for instance, requires that the minimum
> retransmission be one second.  This means a single
> dropped packet may add one second to the latency of
> outstanding commands.  This is a huge amount of time
> on a 10G link.  No doubt this could be mitigated by
> drastically reducing the timeouts within TCP, but the
> market seems to be surprisingly resistant to tampering
> with accepted standards here.
>
> Overall, the FC and FCP protocol have a lot in common
> with the Intel i86 instruction set architecture.  They
> are overly complex, and rather poorly designed by
> modern standards.  But they are good enough, and there
> is a huge amount of value add that has been built on
> top of them, and therefore little incentive to change.
> FCoE is an interesting idea because it preserves 90%
> of the existing value add of FC, unifies the physical
> link with Ethernet, and uses the reliable link method
> of packet delivery.
>
> There are two significant possibilities for iSCSI to
> displace FC (or FCoE) in enterprise storage networks.
> First is if the networks start to scale to large
> enough size that FC can't be made sufficiently
> reliable, and second if CPU compute cycles become
> sufficiently cheap that the iSCSI protocol can be run
> in host software with no negative performance impact.
> Barring either of these, it seems that iSCSI will have
> an uphill battle, and FCoE may have a place.
>
> -----Original Message-----
> From: Julian Satran [ mailto:Julian_Satran@il.ibm.com]
> Sent: Tuesday, April 24, 2007 3:10 PM
> To: ips@ietf.org
> Subject: [Ips] Recent comments about FCoE and iSCSI
>
>
>
> Dear All,
>
> The trade press is lately full with comments about the
> latest and greatest reincarnation of Fiber Channel
> over ethernet.
> It made me try and summarize all the long and hot
> debates that preceded the advent of iSCSI.
> Although FCoE proponents make it look like no debate
> preceded iSCSI that was not so - FCoE was considered
> even then and was dropped as a dumb idea.
>
> Here is a summary (as afar as I can remember) of the
> main arguments. They are not bad arguments even in
> retrospect and technically FCoE doesn't look better
> than it did then.
>
> Feel free to use this material in a nay form. I expect
> this group to seriously  expand my arguments and make
> them public - in personal or collective form.
>
> And do not forget - it is a technical dispute -
> although we all must have some doubts about the way it
> is pursued.
>
> Regards,
> Julo
>
> ---------------------------------------------------------------------
>
>
> What a piece of nostalgia :-)
>
> Around 1997 when a team at IBM Research (Haifa and
> Almaden) started looking at connecting storage to
> servers using the "regular network" (the ubiquitous
> LAN) we considered many alternatives (another team
> even had a look at ATM - still a computer network
> candidate at the time). I won't get you over all of
> our rationale (and we went over some of them again at
> the end of 1999 with a team from CISCO before we
> convened the first IETF BOF in 2000 at Adelaide that
> resulted in iSCSI and all the rest) but some of the
> reasons we choose to drop Fiber Channel over raw
> Ethernet where multiple:
>
> Fiber Channel Protocol (SCSI over Fiber Channel Link)
> is "mildly" effective because:
> it implements endpoints in a dedicated engine
> (Offload)
> it has no transport layer (recovery is done at the
> application layer under the assumption that the error
> rate will be very low)
> the network is limited in physical span and logical
> span (number of switches)
> flow-control/congestion control is achieved with a
> mechanism adequate for a limited span network
> (credits). The packet loss rate is almost nil and that
> allows FCP to avoid using a transport (end-to-end)
> layer
> FCP she switches are simple (addresses are local and
> the memory requirements cam be limited through the
> credit mechanism)
> However FCP endpoints are inherently costlier than
> simple NICs =96 the cost argument (initiators are more
> expensive)
> The credit mechanisms is highly unstable for large
> networks (check switch vendors planning docs for the
> network diameter limits) =96 the scaling argument
> The assumption of low losses due to errors might
> radically change when moving from 1 to 10 Gb/s =96 the
> scaling argument
> Ethernet has no credit mechanism and any mechanism
> with a similar effect increases the end point cost.
> Building a transport layer in the protocol stack has
> always been the preferred choice of the networking
> community =96 the community argument
> The "performance penalty" of a complete protocol stack
> has always been overstated (and overrated). Advances
> in protocol stack implementation and finer tuning of
> the congestion control mechanisms make conventional
> TCP/IP performing well even at 10 Gb/s and over.
> Moreover the multicore processors that become dominant
> on the computing scene have enough compute cycles
> available to make any "offloading" possible as a mere
> code restructuring exercise (see the stack reports
> from Intel, IBM etc.)
> Building on a complete stack makes available a wealth
> of operational and management mechanisms built over
> the years by the networking community (routing,
> provisioning, security, service location etc.) =96 the
> community argument
> Higher level storage access over an IP network is
> widely available and having both block and file served
> over the same connection with the same support and
> management structure is compelling =96 the community
> argument
> Highly efficient networks are easy to build over IP
> with optimal (shortest path) routing while Layer 2
> networks use bridging and are limited by the logical
> tree structure that bridges must follow. The effort to
> combine routers and bridges (rbridges) is promising to
> change that but it will take some time to finalize
> (and we don't know exactly how it will operate).
> Untill then the scale of Layer 2 network is going to
> seriously limited =96 the scaling argument
>
>
> As a side argument =96 a performance comparison made in
> 1998 showed SCSI over TCP (a predecessor of the later
> iSCSI) to perform better than FCP at 1Gbs for block
> sizes typical for OLTP (4-8KB). That was what
> convinced us to take the path that lead to iSCSI =96 and
> we used plain vanilla x86 servers with plain-vanilla
> NICs and Linux (with similar measurements conducted on
> Windows).
> The networking and storage community acknowledged
> those arguments and developed iSCSI and the companion
> protocols for service discovery, boot etc.
>
> The community also acknowledged the need to support
> existing infrastructure and extend it in a reasonable
> fashion and developed 2 protocols iFCP (to support
> hosts with FCP drivers and IP connections to connect
> to storage by a simple conversion from FCP to TCP
> packets) FCPIP to extend the reach of FCP through IP
> (connects FCP islands through TCP links). Both have
> been
> implemented and their foundation is solid.
>
> The current attempt of developing a "new-age" FCP over
> an Ethernet link is going against most of the
> arguments that have given us iSCSI etc.
>
> It ignores the networking layering practice, build an
> application protocol directly above a link and thus
> limits scaling, mandates elements at the link layer
> and application layer that make applications more
> expensive and leaves aside the whole "ecosystem" that
> accompanies TCP/IP (and not Ethernet).
>
> In some related effort (and at a point also when
> developing iSCSI) we considered also moving away from
> SCSI (like some "no standardized" but popular in some
> circles software did =96 e.g., NBP) but decided against.
> SCSI is a mature and well understood access
> architecture for block storage and is implemented by
> many device vendors. Moving away from it would not
> have been justified at the time.
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> _______________________________________________
> Ips mailing list
> Ips@ietf.org
> https://www1.ietf.org/mailman/listinfo/ips
>
> _______________________________________________
> Ips mailing list
> Ips@ietf.org
> https://www1.ietf.org/mailman/listinfo/ips


_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=====================_172001485==.ALT-- --===============0827729375== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============0827729375==-- From ips-bounces@ietf.org Mon Apr 30 15:21:33 2007 Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1HibRF-0003G9-Kq; Mon, 30 Apr 2007 15:21:33 -0400 Received: from ips by megatron.ietf.org with local (Exim 4.43) id 1HibRD-0003G3-Lc for ips-confirm+ok@megatron.ietf.org; Mon, 30 Apr 2007 15:21:31 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1HibRD-0003Fv-Bo for ips@ietf.org; Mon, 30 Apr 2007 15:21:31 -0400 Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1HibRC-00073i-0L for ips@ietf.org; Mon, 30 Apr 2007 15:21:31 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.13.191.130]) by palrel10.hp.com (Postfix) with ESMTP id 865A1349CD; Mon, 30 Apr 2007 12:21:29 -0700 (PDT) Received: from MK73191c.cup.hp.com (je061164.ssr.hp.com [15.47.61.164]) by esmail.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id MAA04908; Mon, 30 Apr 2007 12:15:34 -0700 (PDT) Message-Id: <6.2.0.14.2.20070430082925.01f02178@esmail.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.0.14 Date: Mon, 30 Apr 2007 08:38:58 -0700 To: Julian Satran , Zack Best From: Michael Krause Subject: RE: [Ips] Recent comments about FCoE and iSCSI In-Reply-To: References: <179300.47262.qm@web36705.mail.mud.yahoo.com> Mime-Version: 1.0 X-Spam-Score: 0.5 (/) X-Scan-Signature: 74c8c6a39062dbfd583931efcf641276 Cc: ips@ietf.org X-BeenThere: ips@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP Storage List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============2084374771==" Errors-To: ips-bounces@ietf.org --===============2084374771== Content-Type: multipart/alternative; boundary="=====================_172001505==.ALT" --=====================_172001505==.ALT Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Slight variation: As many-core technology becomes common place and compute power density is=20 combined with the rise of blades as the preferred mechanical packaging of=20 volume servers, the size of data center is irrelevant. What matters it the= =20 ability to co-locate or create locality associations that will enable=20 efficient., secure access and communication between all of these resources. A second order problem issue is how long or to what degree must legacy=20 components be accommodated? In the case of FCoE, the argument appears to= =20 center around if you replace all of your Ethernet equipment, then you can=20 continue to support your traditional SAN deployment. This is, in part,=20 predicated upon the need for DCE being paramount within the data center so= =20 given it is fait accompli, simply ride that train and add a few gateway=20 devices to bring your legacy SAN along for the ride. The counter argument= =20 would be to replace the slow but dependable diesel generator train with a=20 modern bullet train and cast off the legacy under the operating assumption= =20 that the overlap between the old vs. new will not be significant enough to= =20 warrant providing much more than a bridging solution for the glacial=20 customer. In either case, the functional changes in the fabric will take= =20 N years to occur making the how long question very relevant to what=20 technologies should be invested in and gather momentum. Both approaches=20 have benefits to various company players as either leads to a replacement=20 of equipment. One might argue though that the weight or cost of making all= =20 of the components - new, legacy, etc. - interoperate will be significant no= =20 matter what so nothing is a done deal or slam dunk no brainer to pursue. Mike At 06:16 PM 4/26/2007, Julian Satran wrote: >Excellent comments. My take (if not obvious from the previous text) is=20 >that data centers will be very large and compute power (as evidenced by=20 >the multicore) and advances in stack implementation are bound to improve=20 >substantialy the performance of the protocol stacks (see Intel and our=20 >work) and layer 3 switching. >It is important also to point out that Ethernet has substantial latencies= =20 >if only bridging is using and replacement technologies (such as Rbridges=20 >or others) may take some time to appear. > >Julo > > >Zack Best > >25/04/07 16:37 >To >ips@ietf.org >cc >Subject >RE: [Ips] Recent comments about FCoE and iSCSI > > > > >The real debate here is between two types of networks. >The first is reliable at the link level and does not >drop packets under congestion. The second is running >a reliable transport protocol (i.e. TCP) over an >unreliable link level network. > >I agree with the scaling argument. For sufficiently >large networks, reliable link level doesn't work well >because network component failure, or chronically >congested links are not handled well. For >sufficiently small networks, reliable link level has >some significant advantages in simplicity, low >hardware cost, performance, and worst case latency. > >My personal view is that the vast majority of >enterprise storage networks fall in the "sufficiently >small" category. This view has to some extent been >vindicated by the continuing success of Fibre Channel >in this space and the inability of iSCSI to displace >FC in any significant way for enterprise storage. Of >course, this may or may not change in the future. > >Whether FC is simpler than iSCSI depends largely on >your definition of simplicity. If one defines >simplicity/complexity as the number of gates or lines >of code to reduce the protocol to hardware or >firmware, then my experience is that iSCSI is 2X to 3X >the complexity of FC. This has implications in cost >and reliability. > >Particularly problematic with iSCSI is the >unpredictability of the performance. Performance is >great with no packet drop. However even a small >amount of congestion can cause a sudden large drop and >performance. This can be difficult to predict as a >network that is almost but not quite congested can run >great, but a small incremental change of any sort can >cause the performance to become suddenly unacceptable. >For FC, or other protocol using link level flow >control, the reduction in performance is much more >graceful and incremental when the level of congestion >is small and intermittent. > >A second major problem with iSCSI is the unbounded >nature of worst case latency. When a storage network >fails, it is desirable to detect the failure in a >fraction of a second and transition to a backup >network. TCP, when implemented to the standards, can >take many seconds or minutes to determine that a >network has failed and close the connection. RFC >2988, for instance, requires that the minimum >retransmission be one second. This means a single >dropped packet may add one second to the latency of >outstanding commands. This is a huge amount of time >on a 10G link. No doubt this could be mitigated by >drastically reducing the timeouts within TCP, but the >market seems to be surprisingly resistant to tampering >with accepted standards here. > >Overall, the FC and FCP protocol have a lot in common >with the Intel i86 instruction set architecture. They >are overly complex, and rather poorly designed by >modern standards. But they are good enough, and there >is a huge amount of value add that has been built on >top of them, and therefore little incentive to change. >FCoE is an interesting idea because it preserves 90% >of the existing value add of FC, unifies the physical >link with Ethernet, and uses the reliable link method >of packet delivery. > >There are two significant possibilities for iSCSI to >displace FC (or FCoE) in enterprise storage networks. >First is if the networks start to scale to large >enough size that FC can't be made sufficiently >reliable, and second if CPU compute cycles become >sufficiently cheap that the iSCSI protocol can be run >in host software with no negative performance impact. >Barring either of these, it seems that iSCSI will have >an uphill battle, and FCoE may have a place. > >-----Original Message----- >From: Julian Satran [mailto:Julian_Satran@il.ibm.com] >Sent: Tuesday, April 24, 2007 3:10 PM >To: ips@ietf.org >Subject: [Ips] Recent comments about FCoE and iSCSI > > > >Dear All, > >The trade press is lately full with comments about the >latest and greatest reincarnation of Fiber Channel >over ethernet. >It made me try and summarize all the long and hot >debates that preceded the advent of iSCSI. >Although FCoE proponents make it look like no debate >preceded iSCSI that was not so - FCoE was considered >even then and was dropped as a dumb idea. > >Here is a summary (as afar as I can remember) of the >main arguments. They are not bad arguments even in >retrospect and technically FCoE doesn't look better >than it did then. > >Feel free to use this material in a nay form. I expect >this group to seriously expand my arguments and make >them public - in personal or collective form. > >And do not forget - it is a technical dispute - >although we all must have some doubts about the way it >is pursued. > >Regards, >Julo > >--------------------------------------------------------------------- > > >What a piece of nostalgia :-) > >Around 1997 when a team at IBM Research (Haifa and >Almaden) started looking at connecting storage to >servers using the "regular network" (the ubiquitous >LAN) we considered many alternatives (another team >even had a look at ATM - still a computer network >candidate at the time). I won't get you over all of >our rationale (and we went over some of them again at >the end of 1999 with a team from CISCO before we >convened the first IETF BOF in 2000 at Adelaide that >resulted in iSCSI and all the rest) but some of the >reasons we choose to drop Fiber Channel over raw >Ethernet where multiple: > >Fiber Channel Protocol (SCSI over Fiber Channel Link) >is "mildly" effective because: >it implements endpoints in a dedicated engine >(Offload) >it has no transport layer (recovery is done at the >application layer under the assumption that the error >rate will be very low) >the network is limited in physical span and logical >span (number of switches) >flow-control/congestion control is achieved with a >mechanism adequate for a limited span network >(credits). The packet loss rate is almost nil and that >allows FCP to avoid using a transport (end-to-end) >layer >FCP she switches are simple (addresses are local and >the memory requirements cam be limited through the >credit mechanism) >However FCP endpoints are inherently costlier than >simple NICs =96 the cost argument (initiators are more >expensive) >The credit mechanisms is highly unstable for large >networks (check switch vendors planning docs for the >network diameter limits) =96 the scaling argument >The assumption of low losses due to errors might >radically change when moving from 1 to 10 Gb/s =96 the >scaling argument >Ethernet has no credit mechanism and any mechanism >with a similar effect increases the end point cost. >Building a transport layer in the protocol stack has >always been the preferred choice of the networking >community =96 the community argument >The "performance penalty" of a complete protocol stack >has always been overstated (and overrated). Advances >in protocol stack implementation and finer tuning of >the congestion control mechanisms make conventional >TCP/IP performing well even at 10 Gb/s and over. >Moreover the multicore processors that become dominant >on the computing scene have enough compute cycles >available to make any "offloading" possible as a mere >code restructuring exercise (see the stack reports >from Intel, IBM etc.) >Building on a complete stack makes available a wealth >of operational and management mechanisms built over >the years by the networking community (routing, >provisioning, security, service location etc.) =96 the >community argument >Higher level storage access over an IP network is >widely available and having both block and file served >over the same connection with the same support and >management structure is compelling =96 the community >argument >Highly efficient networks are easy to build over IP >with optimal (shortest path) routing while Layer 2 >networks use bridging and are limited by the logical >tree structure that bridges must follow. The effort to >combine routers and bridges (rbridges) is promising to >change that but it will take some time to finalize >(and we don't know exactly how it will operate). >Untill then the scale of Layer 2 network is going to >seriously limited =96 the scaling argument > > >As a side argument =96 a performance comparison made in >1998 showed SCSI over TCP (a predecessor of the later >iSCSI) to perform better than FCP at 1Gbs for block >sizes typical for OLTP (4-8KB). That was what >convinced us to take the path that lead to iSCSI =96 and >we used plain vanilla x86 servers with plain-vanilla >NICs and Linux (with similar measurements conducted on >Windows). >The networking and storage community acknowledged >those arguments and developed iSCSI and the companion >protocols for service discovery, boot etc. > >The community also acknowledged the need to support >existing infrastructure and extend it in a reasonable >fashion and developed 2 protocols iFCP (to support >hosts with FCP drivers and IP connections to connect >to storage by a simple conversion from FCP to TCP >packets) FCPIP to extend the reach of FCP through IP >(connects FCP islands through TCP links). Both have >been >implemented and their foundation is solid. > >The current attempt of developing a "new-age" FCP over >an Ethernet link is going against most of the >arguments that have given us iSCSI etc. > >It ignores the networking layering practice, build an >application protocol directly above a link and thus >limits scaling, mandates elements at the link layer >and application layer that make applications more >expensive and leaves aside the whole "ecosystem" that >accompanies TCP/IP (and not Ethernet). > >In some related effort (and at a point also when >developing iSCSI) we considered also moving away from >SCSI (like some "no standardized" but popular in some >circles software did =96 e.g., NBP) but decided against. >SCSI is a mature and well understood access >architecture for block storage and is implemented by >many device vendors. Moving away from it would not >have been justified at the time. > > >__________________________________________________ >Do You Yahoo!? >Tired of spam? Yahoo! Mail has the best spam protection around >http://mail.yahoo.com > > > >_______________________________________________ >Ips mailing list >Ips@ietf.org >https://www1.ietf.org/mailman/listinfo/ips > >_______________________________________________ >Ips mailing list >Ips@ietf.org >https://www1.ietf.org/mailman/listinfo/ips --=====================_172001505==.ALT Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Slight variation:

As many-core technology becomes common place and compute power density is combined with the rise of blades as the preferred mechanical packaging of volume servers, the size of data center is irrelevant.  What matters it the ability to co-locate or create locality associations that will enable efficient., secure access and communication between all of these resources.  

A second order problem issue is how long or to what degree must legacy components be accommodated?   In the case of FCoE, the argument appears to center around if you replace all of your Ethernet equipment, then you can continue to support your traditional SAN deployment.   This is, in part, predicated upon the need for DCE being paramount within the data center so given it is fait accompli, simply ride that train and add a few gateway devices to bring your legacy SAN along for the ride.   The counter argument would be to replace the slow but dependable diesel generator train with a modern bullet train and cast off the legacy under the operating assumption that the overlap between the old vs. new will not be significant enough to warrant providing much more than a bridging solution for the glacial customer.    In either case, the functional changes in the fabric will take N years to occur making the how long question very relevant to what technologies should be invested in and gather momentum.  Both approaches have benefits to various company players as either leads to a replacement of equipment.  One might argue though that the weight or cost of making all of the components - new, legacy, etc. - interoperate will be significant no matter what so nothing is a done deal or slam dunk no brainer to pursue.

Mike



At 06:16 PM 4/26/2007, Julian Satran wrote:

Excellent comm= ents. My take (if not obvious from the previous text) is that data centers will be very large and compute power (as evidenced by the multicore) and advances in stack implementation are bound to improve substantialy the performance of the protocol stacks (see Intel and our work) and layer 3 switching.
It is important also to point out that Ethernet has substantial latencies if only bridging is using and replacement technologies (such as Rbridges or others) may take some time to appear.

Julo


Zack Best <zbest28@yahoo.com>

25/04/07 16:37
To
ips@ietf.org
cc
Subject
RE: [Ips] Recent comments about FCoE and iSCSI




The real debate here is between two types of networks.
The first is reliable at the link level and does not
drop packets under congestion.  The second is running
a reliable transport protocol (i.e. TCP) over an
unreliable link level network.

I agree with the scaling argument.  For sufficiently
large networks, reliable link level doesn't work well
because network component failure, or chronically
congested links are not handled well.  For
sufficiently small networks, reliable link level has
some significant advantages in simplicity, low
hardware cost, performance, and worst case latency.

My personal view is that the vast majority of
enterprise storage networks fall in the "sufficiently
small" category.  This view has to some extent been
vindicated by the continuing success of Fibre Channel
in this space and the inability of iSCSI to displace
FC in any significant way for enterprise storage.  Of
course, this may or may not change in the future.

Whether FC is simpler than iSCSI depends largely on
your definition of simplicity.  If one defines
simplicity/complexity as the number of gates or lines
of code to reduce the protocol to hardware or
firmware, then my experience is that iSCSI is 2X to 3X
the complexity of FC.  This has implications in cost
and reliability.

Particularly problematic with iSCSI is the
unpredictability of the performance.  Performance is
great with no packet drop.  However even a small
amount of congestion can cause a sudden large drop and
performance.  This can be difficult to predict as a
network that is almost but not quite congested can run
great, but a small incremental change of any sort can
cause the performance to become suddenly unacceptable.
For FC, or other protocol using link level flow
control, the reduction in performance is much more
graceful and incremental when the level of congestion
is small and intermittent.

A second major problem with iSCSI is the unbounded
nature of worst case latency.  When a storage network
fails, it is desirable to detect the failure in a
fraction of a second and transition to a backup
network.  TCP, when implemented to the standards, can
take many seconds or minutes to determine that a
network has failed and close the connection.  RFC
2988, for instance, requires that the minimum
retransmission be one second.  This means a single
dropped packet may add one second to the latency of
outstanding commands.  This is a huge amount of time
on a 10G link.  No doubt this could be mitigated by
drastically reducing the timeouts within TCP, but the
market seems to be surprisingly resistant to tampering
with accepted standards here.

Overall, the FC and FCP protocol have a lot in common
with the Intel i86 instruction set architecture.  They
are overly complex, and rather poorly designed by
modern standards.  But they are good enough, and there
is a huge amount of value add that has been built on
top of them, and therefore little incentive to change.
FCoE is an interesting idea because it preserves 90%
of the existing value add of FC, unifies the physical
link with Ethernet, and uses the reliable link method
of packet delivery.

There are two significant possibilities for iSCSI to
displace FC (or FCoE) in enterprise storage networks.
First is if the networks start to scale to large
enough size that FC can't be made sufficiently
reliable, and second if CPU compute cycles become
sufficiently cheap that the iSCSI protocol can be run
in host software with no negative performance impact.
Barring either of these, it seems that iSCSI will have
an uphill battle, and FCoE may have a place.

-----Original Message-----
From: Julian Satran [ mailto:Julian_Satran@il.ibm.com]
Sent: Tuesday, April 24, 2007 3:10 PM
To: ips@ietf.org
Subject: [Ips] Recent comments about FCoE and iSCSI



Dear All,

The trade press is lately full with comments about the
latest and greatest reincarnation of Fiber Channel
over ethernet.
It made me try and summarize all the long and hot
debates that preceded the advent of iSCSI.
Although FCoE proponents make it look like no debate
preceded iSCSI that was not so - FCoE was considered
even then and was dropped as a dumb idea.

Here is a summary (as afar as I can remember) of the
main arguments. They are not bad arguments even in
retrospect and technically FCoE doesn't look better
than it did then.

Feel free to use this material in a nay form. I expect
this group to seriously  expand my arguments and make
them public - in personal or collective form.

And do not forget - it is a technical dispute -
although we all must have some doubts about the way it
is pursued.

Regards,
Julo

---------------------------------------------------------------------


What a piece of nostalgia :-)

Around 1997 when a team at IBM Research (Haifa and
Almaden) started looking at connecting storage to
servers using the "regular network" (the ubiquitous
LAN) we considered many alternatives (another team
even had a look at ATM - still a computer network
candidate at the time). I won't get you over all of
our rationale (and we went over some of them again at
the end of 1999 with a team from CISCO before we
convened the first IETF BOF in 2000 at Adelaide that
resulted in iSCSI and all the rest) but some of the
reasons we choose to drop Fiber Channel over raw
Ethernet where multiple:

Fiber Channel Protocol (SCSI over Fiber Channel Link)
is "mildly" effective because:
it implements endpoints in a dedicated engine
(Offload)
it has no transport layer (recovery is done at the
application layer under the assumption that the error
rate will be very low)
the network is limited in physical span and logical
span (number of switches)
flow-control/congestion control is achieved with a
mechanism adequate for a limited span network
(credits). The packet loss rate is almost nil and that
allows FCP to avoid using a transport (end-to-end)
layer
FCP she switches are simple (addresses are local and
the memory requirements cam be limited through the
credit mechanism)
However FCP endpoints are inherently costlier than
simple NICs =96 the cost argument (initiators are more
expensive)
The credit mechanisms is highly unstable for large
networks (check switch vendors planning docs for the
network diameter limits) =96 the scaling argument
The assumption of low losses due to errors might
radically change when moving from 1 to 10 Gb/s =96 the
scaling argument
Ethernet has no credit mechanism and any mechanism
with a similar effect increases the end point cost.
Building a transport layer in the protocol stack has
always been the preferred choice of the networking
community =96 the community argument
The "performance penalty" of a complete protocol stack
has always been overstated (and overrated). Advances
in protocol stack implementation and finer tuning of
the congestion control mechanisms make conventional
TCP/IP performing well even at 10 Gb/s and over.
Moreover the multicore processors that become dominant
on the computing scene have enough compute cycles
available to make any "offloading" possible as a mere
code restructuring exercise (see the stack reports
from Intel, IBM etc.)
Building on a complete stack makes available a wealth
of operational and management mechanisms built over
the years by the networking community (routing,
provisioning, security, service location etc.) =96 the
community argument
Higher level storage access over an IP network is
widely available and having both block and file served
over the same connection with the same support and
management structure is compelling =96 the community
argument
Highly efficient networks are easy to build over IP
with optimal (shortest path) routing while Layer 2
networks use bridging and are limited by the logical
tree structure that bridges must follow. The effort to
combine routers and bridges (rbridges) is promising to
change that but it will take some time to finalize
(and we don't know exactly how it will operate).
Untill then the scale of Layer 2 network is going to
seriously limited =96 the scaling argument


As a side argument =96 a performance comparison made in
1998 showed SCSI over TCP (a predecessor of the later
iSCSI) to perform better than FCP at 1Gbs for block
sizes typical for OLTP (4-8KB). That was what
convinced us to take the path that lead to iSCSI =96 and
we used plain vanilla x86 servers with plain-vanilla
NICs and Linux (with similar measurements conducted on
Windows).
The networking and storage community acknowledged
those arguments and developed iSCSI and the companion
protocols for service discovery, boot etc.

The community also acknowledged the need to support
existing infrastructure and extend it in a reasonable
fashion and developed 2 protocols iFCP (to support
hosts with FCP drivers and IP connections to connect
to storage by a simple conversion from FCP to TCP
packets) FCPIP to extend the reach of FCP through IP
(connects FCP islands through TCP links). Both have
been
implemented and their foundation is solid.

The current attempt of developing a "new-age" FCP over
an Ethernet link is going against most of the
arguments that have given us iSCSI etc.

It ignores the networking layering practice, build an
application protocol directly above a link and thus
limits scaling, mandates elements at the link layer
and application layer that make applications more
expensive and leaves aside the whole "ecosystem" that
accompanies TCP/IP (and not Ethernet).

In some related effort (and at a point also when
developing iSCSI) we considered also moving away from
SCSI (like some "no standardized" but popular in some
circles software did =96 e.g., NBP) but decided against.
SCSI is a mature and well understood access
architecture for block storage and is implemented by
many device vendors. Moving away from it would not
have been justified at the time.


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com



_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
Ips@ietf.org
https://www1.ietf.org/mailman/listinfo/ips
--=====================_172001505==.ALT-- --===============2084374771== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Ips mailing list Ips@ietf.org https://www1.ietf.org/mailman/listinfo/ips --===============2084374771==--