Subject: Re: [sami] First SAMI email in the new year, can we go further? Look forward to your opinions.

From: Thomas Narten <[email-masked]>
Date: Thu, 01 Mar 2012 08:29:36 -0500

Hi.

> According to previous discussion, one of the arguement is that we
>  don't need State Migration if there is no VM Migration, we can, for
>  example running active-active VMs mechanism to replace VM
>  Migration.

I think this is a bit of a strawman.

I would expect pretty much everyone on this list would agree that VM
migration takes place, under a wide range of conditions. So let's just
take it as a given that VM migration exists.

Personally, I can see the theoretical benefit of state migration
(e.g., of a firewall). But for me, the biggest reason I have been
skeptical of this effort so far is that I don't see real operators
stepping forwarding saying they need "state migration" in a real
concrete sense.

The IETF does much better when it has a real operational problem
driving a solution. We need such a problem in order to privide a
verification that should we attempt to develop a solution, the
solution actually solves a real world problem.

Thomas

---

From: Melinda Shore <[email-masked]>
Date: Thu, 01 Mar 2012 07:02:43 -0900

> take it as a given that VM migration exists.

Actually, it wasn't that long ago that that question was contested.
I was trying to get a handle on this last year and asked a few
hypervisor vendors, and the answers I got ranged from "no" to "we're
migrating VMs but they need to be quiesced before they can move" to
"we're doing it in real-time but aren't close to releasing a product."
So, if you're treating it as a given that's something of a shift right
there.

Clearly any use case needs to be rooted in actual use, but this is a
fast-moving area and while something that was true yesterday is probably
true today, something that was true six months ago could very well no
longer be.

Melinda

---

From: Thomas Narten <[email-masked]>
Date: Thu, 01 Mar 2012 11:13:10 -0500

Hmm. Maybe I misunderstand.

Live migration happens today.  You start transferring the VM, but it
continues running. At some point, you have to freeze the VM, finish
transferring state (and any "dirty" pages) and enable the transferred
VM at the new location and resume it.

That does mean that an active VM in one place is migrated "live" (with
a short downtime of sub seconds) to another place.

This is not a case where the VM shuts down and new one is instantiated
(i.e., started from scratch) elsewhere.

Isn't this what VM migration means?

Thomas

---

From: Melinda Shore <[email-masked]>
Date: Thu, 01 Mar 2012 07:24:42 -0900

> Isn't this what VM migration means?

I think so, and there's no doubt it would be helpful to be clearer about
that going forward.  However, my understanding that while the VM image
would be moved (craploads of kernel state - file tables, page tables,
all that crap), network activity would have to be quiesced first.  It
sounds to me like it's got much in common with old school supercomputer
checkpoint/restart.  For whatever it's worth we have to do something
similar with some mid-range "application availability" (you'll notice
the absence of the word "high") gizmos, and something along the lines of
a middlebox state migration tool would seem to have some applicability
there, too.

Melinda

---

From: Thomas Narten <[email-masked]>
Date: Thu, 01 Mar 2012 11:29:39 -0500

Melinda Shore <[email-masked]> writes:

> However, my understanding that while the VM image would be moved
> (craploads of kernel state - file tables, page tables, all that
> crap), network activity would have to be quiesced first.

Well, such traffic is dropped or held until it can be delivered to the
new location.

> It sounds to me like it's got much in common with old school
> supercomputer checkpoint/restart.

Of course! How else could this be done? :-)

In any case, it can be done today in the DC in under a second, which
is good enough for a lot of use cases.

Thomas

---

From: <[email-masked]>
Date: Thu, 1 Mar 2012 12:02:18 -0500

> all that crap), network activity would have to be quiesced first.  

That understanding is incorrect - network activity is not quiesced.  A
gratuitous ARP or RARP is issued after the migration, and some inbound
packets may get dropped because they are delivered only to the source
location of the migration.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Melinda Shore
> Sent: Thursday, March 01, 2012 11:25 AM
> To: Thomas Narten
> Cc: Yingjie Gu(yingjie); [email-masked]
> Subject: Re: [sami] First SAMI email in the new year, can we go further? Look forward to your
> opinions.
> 
> _______________________________________________
> sami mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sami

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 1 Mar 2012 18:28:41 +0100

On Thu, Mar 01, 2012 at 01:16:09AM +0000, Yingjie Gu(yingjie) wrote:
 
> Besides, VM migration has been a widely acknowledged technology and solutions (e.g. VMotion), so I would say we can reasonably assume VM Migration will happen and hence states need to be migrated with VM.
> 
> If we get a consensus that VM Migration is a reasonable assumption, I suggest we go to detail of what we can do in IETF to resolve State Migration and how.
> But I look forward to your opinions.

Lets assume VM migration happens today (which I am fine to assume - I
am however unsure how frequent live migration takes place and hence
whether its worth to optimize for it).

That said, this assumption alone does not necessarily imply there is a
state migration problem worth to be solved by the IETF. I am missing
statements from VMware engineers (since you mention VMotion) or the
like where they explain how IETF protocol work can help them to make
their migration solutions work even better as they do.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: zhangyunfei <[email-masked]>
Date: Fri, 2 Mar 2012 09:48:56 +0800

Totally agree. I do suggest to involve some Internet service operator, like Google, baidu (who may be the potential SAMI users) in the discussion if possible, which'll convince IETF that it's a real problem we encounter.

BR
Yunfei

zhangyunfei

From: Thomas Narten
Date: 2012-03-01 21:29
To: Yingjie Gu(yingjie)
CC: [email-masked]
Subject: Re: [sami] First SAMI email in the new year, can we go further? Look forward to your opinions.
Hi.

Thomas

---

From: Benson Schliesser <[email-masked]>
Date: Thu, 1 Mar 2012 18:30:45 -0800

> location of the migration.

Right. Minor packet loss is possible, but the VM migration is designed so that e.g. established TCP connections don't fail.

Like Thomas, I see that there is something to be done here. But I also agree that we need to figure out the use-cases first, driven by real operational requirements. It might help if we started talking about use-cases for VM migration - yes it's possible, but why do people use it? - so that we can figure out if any of them inform the discussion of state migration.

Further, are there any non-VM mobility use-cases for state migration? I can imagine a few, but again I want to make sure they're being driven by operational needs rather than mere possibility.

Cheers,
-Benson

---

From: "Dacheng Zhang(Dacheng)" <[email-masked]>
Date: Fri, 02 Mar 2012 02:31:53 +0000

>> whether its worth to optimize for it).
[Dacheng Zhang] I think this will happen quite frequently in some conditions. Assume for energy conversation purpose at night an operator of a DC can move active VMs to several physical servers and switch other physical servers off. In this case, migration will happen very day.

>> their migration solutions work even better as they do.
[Dacheng Zhang] 
Agree. If we all agree that VM migration is happening or will happen in DC, it is the right time to discuss where there are any important scenarios where state migration across network side devices is necessary.

By the way, there is a Cisco white paper " Virtual Machine Mobility with Vmware VMotion and Cisco Data Center Interconnect Technologies". I personally think it is quite interesting. Anybody having interests can download it and have a look. ^_^

Cheers

Dacheng

---

From: "Dacheng Zhang(Dacheng)" <[email-masked]>
Date: Fri, 02 Mar 2012 02:38:58 +0000

>> Right. Minor packet loss is possible, but the VM migration is designed so that
>> e.g. established TCP connections don't fail.
>> 
[Dacheng Zhang] I just checked with somebody working in this area, you are correct. So, +1

>> 
>> Further, are there any non-VM mobility use-cases for state migration? I can
>> imagine a few, but again I want to make sure they're being driven by
>> operational needs rather than mere possibility.
[Dacheng Zhang] 
In addition, even after we have found some useful use case, we still need to find out whether there are some other ways instead of state migration that we can go .

---

From: <[email-masked]>
Date: Thu, 1 Mar 2012 21:53:41 -0500

> Like Thomas, I see that there is something to be done here. But I also agree that we need
> to figure out the use-cases first, driven by real operational requirements. It might help
> if we started talking about use-cases for VM migration - yes it's possible, but why do
> people use it? - so that we can figure out if any of them inform the discussion of state migration.

Here are four use cases to start with (all are non-disruptive with VM migration):

(1) Server load balancing - migrate VMs for balanced server loading.
(2) Power savings - Consolidate VMs onto fewer servers when overall data center load
	is light (perhaps at night) so that unused servers can be powered down.
(3) Non-disruptive server maintenance - vacate a server that needs to be powered
	down so that it can be worked on.
(4) Non-disruptive server hardware refresh - add new servers and remove old servers.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Benson Schliesser
> Sent: Thursday, March 01, 2012 9:31 PM
> To: [email-masked]
> Like Thomas, I see that there is something to be done here. But I also agree that we need to figure
> out the use-cases first, driven by real operational requirements. It might help if we started talking
> about use-cases for VM migration - yes it's possible, but why do people use it? - so that we can
> figure out if any of them inform the discussion of state migration.
> 
> Further, are there any non-VM mobility use-cases for state migration? I can imagine a few, but again I
> want to make sure they're being driven by operational needs rather than mere possibility.
> 
> Cheers,
> -Benson
> 
>

---

From: Melinda Shore <[email-masked]>
Date: Thu, 01 Mar 2012 19:10:29 -0900

> (4) Non-disruptive server hardware refresh - add new servers and remove old servers.

Here's one that closely related to (3): there are load balancers that
work basically by presenting a NATted address and playing with the NAT
tables on the fly.  At the moment I find this case a little unclear but
I'm in the process of trying to get some feedback on it from people
who've got such a product.  This problem has also come up in the case
of multihomed sctp endpoings.  I think there are some use cases not
related to VM migration.

Melinda

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 2 Mar 2012 08:28:49 +0100

> [Dacheng Zhang] I think this will happen quite frequently in some conditions. Assume for energy conversation purpose at night an operator of a DC can move active VMs to several physical servers and…

Sure - but this still does not convince me there is a problem worth
solving. As long as there is a common layer two infrastructure,
migration seems to work today. If there is not a common layer two
infrastructure, you still need to have a common shared storage system.
I know, you will now draw a picture that VM will migrate across DCs
etc. We have discussed all these potential use cases. I am lacking
evidence that they (a) are real and (b) significant enough to be worth
solving. To me, this still all sounds more like a research group topic.

> By the way, there is a Cisco white paper " Virtual Machine Mobility with Vmware VMotion and Cisco Data Center Interconnect Technologies". I personally think it is quite interesting. Anybody having…

If I understand the paper correctly, then they assume an extended VLAN
with a common IP subnet is in place for cross DC migration to work and
they the leave migration across IP network boundaries as something to
be worked out in the future.

/js

---

From: "Yingjie Gu(yingjie)" <[email-masked]>
Date: Mon, 05 Mar 2012 07:25:47 +0000

Yes, I agree, and I am connecting with some ISPs to get their opinions.
They may not be so active in email discussion, but they can provide some use cases to compose a draft.

________________________________
Best Regards
Gu Yingjie

发件人: zhangyunfei [mailto:[email-masked]]
发送时间: 2012年3月2日 乐乐9:49
收件人: Thomas Narten; Yingjie Gu(yingjie)
抄送: [email-masked]
主题: Re: Re: [sami] First SAMI email in the new year, can we go further? Look forward to your opinions.

BR
Yunfei

________________________________
zhangyunfei

From: Thomas Narten<mailto:[email-masked]>
Date: 2012-03-01 21:29
To: Yingjie Gu(yingjie)<mailto:[email-masked]>
CC: [email-masked]<mailto:[email-masked]>
Subject: Re: [sami] First SAMI email in the new year, can we go further? Look forward to your opinions.
Hi.

Thomas