From glen@amsl.com Mon Sep 5 18:00:00 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 83E931B60ADF; Mon, 5 Sep 2011 18:00:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102 X-Spam-Level: X-Spam-Status: No, score=-102 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_21=0.6, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QElXJCcIqORG; Mon, 5 Sep 2011 18:00:00 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfc.amsl.com (Postfix) with ESMTP id 101B81B60ADD; Mon, 5 Sep 2011 17:59:51 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 99A4F1C380CE; Mon, 5 Sep 2011 17:59:50 -0700 (PDT) Date: Mon, 5 Sep 2011 17:59:50 -0700 From: Glen Barney To: iab@ietf.org, iaoc@ietf.org, iesg@ietf.org, wgchairs@ietf.org, codesprints@ietf.org Message-ID: <20110906005950.GA4153@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: rpelletier@isoc.org Subject: [codesprints] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 01:00:00 -0000 Dear IETF Leadership - At 3:14 PM Pacific Time today, Monday, September 5th, we received an alert from our monitoring systems that power at our primary facility had failed. Of course, we have battery backup here, and were still in operation. I responded to the facility, arriving approximately 20 minutes later, and confirmed that power was out. This was not difficult, as the traffic signals in the entire area were out as well. PG&E crews were arriving just as I did, and stated that they expected a restoration time no earlier than 5:30 PM. As a result, I executed our failover procedure, and transferred IETF operations to the Orlando data center. As with any such situation, some users may have encountered interruptions in service. However, our failover was completed very quickly, so any problems that were experienced were minimal. I will be remaining here for a bit longer until the PG&E crews leave, just to make sure that another power outage isn't happening. At 8PM Pacific time, I will initiate a planned outage, to do a controlled fail-back to Fremont. I will announce this planned outage on the IETF list momentarily. As always, if there are any questions, please feel free to let me know. Thanks, Glen From housley@vigilsec.com Mon Sep 5 18:06:57 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 5126D1B60B3D; Mon, 5 Sep 2011 18:06:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.299 X-Spam-Level: X-Spam-Status: No, score=-102.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V7OH4-TLD8GL; Mon, 5 Sep 2011 18:06:48 -0700 (PDT) Received: from odin.smetech.net (mail.smetech.net [208.254.26.82]) by ietfc.amsl.com (Postfix) with ESMTP id 8049F1B60B3C; Mon, 5 Sep 2011 18:06:48 -0700 (PDT) Received: from localhost (unknown [208.254.26.81]) by odin.smetech.net (Postfix) with ESMTP id 02FCCF240A6; Mon, 5 Sep 2011 21:07:13 -0400 (EDT) X-Virus-Scanned: amavisd-new at smetech.net Received: from odin.smetech.net ([208.254.26.82]) by localhost (ronin.smetech.net [208.254.26.81]) (amavisd-new, port 10024) with ESMTP id ET5rn27dMb4Y; Mon, 5 Sep 2011 21:06:45 -0400 (EDT) Received: from [192.168.2.107] (pool-96-231-29-247.washdc.fios.verizon.net [96.231.29.247]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by odin.smetech.net (Postfix) with ESMTP id F3FDDF24082; Mon, 5 Sep 2011 21:07:11 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Russ Housley In-Reply-To: <20110906005950.GA4153@amsl.com> Date: Mon, 5 Sep 2011 21:06:44 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> To: Glen Barney X-Mailer: Apple Mail (2.1084) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, rpelletier@isoc.org, iesg@ietf.org Subject: Re: [codesprints] [IAOC] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 01:06:57 -0000 Given this situation, it sounds like you made the right call. You might = want to leave things running in Orlando until tomorrow. Your call ... Russ On Sep 5, 2011, at 8:59 PM, Glen Barney wrote: > Dear IETF Leadership - >=20 > At 3:14 PM Pacific Time today, Monday, September 5th, we received an = alert > from our monitoring systems that power at our primary facility had = failed. > Of course, we have battery backup here, and were still in operation. >=20 > I responded to the facility, arriving approximately 20 minutes later, = and > confirmed that power was out. This was not difficult, as the traffic = signals > in the entire area were out as well. >=20 > PG&E crews were arriving just as I did, and stated that they expected = a > restoration time no earlier than 5:30 PM. >=20 > As a result, I executed our failover procedure, and transferred IETF = operations > to the Orlando data center. >=20 > As with any such situation, some users may have encountered = interruptions in > service. However, our failover was completed very quickly, so any = problems > that were experienced were minimal. >=20 > I will be remaining here for a bit longer until the PG&E crews leave, = just to > make sure that another power outage isn't happening. >=20 > At 8PM Pacific time, I will initiate a planned outage, to do a = controlled > fail-back to Fremont. I will announce this planned outage on the IETF = list > momentarily. >=20 > As always, if there are any questions, please feel free to let me = know. >=20 > Thanks, > Glen > _______________________________________________ > IAOC mailing list > IAOC@ietf.org > https://www.ietf.org/mailman/listinfo/iaoc From glen@amsl.com Mon Sep 5 18:40:10 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 52E3F1B60ADC; Mon, 5 Sep 2011 18:40:10 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.3 X-Spam-Level: X-Spam-Status: No, score=-102.3 tagged_above=-999 required=5 tests=[AWL=0.300, BAYES_00=-2.599, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qeQDwfhlkD+D; Mon, 5 Sep 2011 18:40:09 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfc.amsl.com (Postfix) with ESMTP id B81741B60B02; Mon, 5 Sep 2011 18:40:09 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 390611C380CE; Mon, 5 Sep 2011 18:40:09 -0700 (PDT) Date: Mon, 5 Sep 2011 18:40:09 -0700 From: Glen To: ietf-announce@ietf.org, ietf@ietf.org Message-ID: <20110906014009.GA9738@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, rpelletier@isoc.org, iesg@ietf.org Subject: [codesprints] Planned IETF outage Monday evening X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 01:40:10 -0000 Dear IETF Community - Earlier today a local-area power outage of several square miles occurred in the area of our Fremont data center. Since that left us running on backup, we executed an emergency failover to our Orlando data center, where we are currently running. Power has been restored, and I will be verifying full completion with the power crews shortly. Barring additional issues, I plan to transfer control back to our Fremont data center at 8:30 PM Pacific Time (4:30 AM GMT) this o evening. During the transfer, web and email services will be interrupted. We expect the interruption will take approximately 20 minutes. There is no need to panic, I will be handling the failback personally, and will have service restored as soon as possible. Thank you for your patience. Glen Glen Barney IT Director AMS (IETF Secretariat) From glen@amsl.com Mon Sep 5 20:00:36 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id B11571B60AE1; Mon, 5 Sep 2011 20:00:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.45 X-Spam-Level: X-Spam-Status: No, score=-102.45 tagged_above=-999 required=5 tests=[AWL=0.150, BAYES_00=-2.599, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BaD45pqplVlg; Mon, 5 Sep 2011 20:00:36 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfc.amsl.com (Postfix) with ESMTP id 4171A1B60ADD; Mon, 5 Sep 2011 20:00:36 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id CF1B21C380CE; Mon, 5 Sep 2011 20:00:35 -0700 (PDT) Date: Mon, 5 Sep 2011 20:00:35 -0700 From: Glen To: Pete Resnick Message-ID: <20110906030035.GA21428@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> <4E658924.1090103@qualcomm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E658924.1090103@qualcomm.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, iesg@ietf.org Subject: Re: [codesprints] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 03:00:36 -0000 On Mon, Sep 05, 2011 at 09:44:52PM -0500, Pete Resnick wrote: > Hi Glen, > The currently running datatracker has older data than the one I was using > earlier today. Is that is to be expected in an emergency failover? > Just checking. Certainly not. The MySQL database is synced on a transactional basis. Every transaction committed to the database is immediately committed on all three servers. And the process was running correctly the entire time. In the interest of haste, I am contacting you on IM right now, as you already know. Glen From glen@amsl.com Mon Sep 5 20:02:47 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id D4C751B60AE8; Mon, 5 Sep 2011 20:02:47 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.5 X-Spam-Level: X-Spam-Status: No, score=-102.5 tagged_above=-999 required=5 tests=[AWL=0.100, BAYES_00=-2.599, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g1bRUnwumprx; Mon, 5 Sep 2011 20:02:47 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfc.amsl.com (Postfix) with ESMTP id 915DF1B60AF2; Mon, 5 Sep 2011 20:02:46 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 28CEE1C380CE; Mon, 5 Sep 2011 20:02:46 -0700 (PDT) Date: Mon, 5 Sep 2011 20:02:46 -0700 From: Glen To: Pete Resnick Message-ID: <20110906030246.GB21428@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> <4E658924.1090103@qualcomm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E658924.1090103@qualcomm.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, iesg@ietf.org Subject: Re: [codesprints] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 03:02:48 -0000 On Mon, Sep 05, 2011 at 09:44:52PM -0500, Pete Resnick wrote: > Hi Glen, > The currently running datatracker has older data than the one I was using > earlier today. Is that is to be expected in an emergency failover? > Just checking. I have the information from Pete and am checking now. Glen From glen@amsl.com Mon Sep 5 20:20:06 2011 Return-Path: X-Original-To: codesprints@ietfc.amsl.com Delivered-To: codesprints@ietfc.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 770F41B60B02; Mon, 5 Sep 2011 20:20:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.525 X-Spam-Level: X-Spam-Status: No, score=-102.525 tagged_above=-999 required=5 tests=[AWL=0.075, BAYES_00=-2.599, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b8Gnqz9m7YNo; Mon, 5 Sep 2011 20:20:06 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfc.amsl.com (Postfix) with ESMTP id 086961B60B01; Mon, 5 Sep 2011 20:19:51 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 937E11C380CE; Mon, 5 Sep 2011 20:19:50 -0700 (PDT) Date: Mon, 5 Sep 2011 20:19:50 -0700 From: Glen To: Pete Resnick Message-ID: <20110906031950.GA22629@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> <4E658924.1090103@qualcomm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E658924.1090103@qualcomm.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, iesg@ietf.org Subject: Re: [codesprints] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 03:20:06 -0000 On Mon, Sep 05, 2011 at 09:44:52PM -0500, Pete Resnick wrote: > Hi Glen, > The currently running datatracker has older data than the one I was using > earlier today. Is that is to be expected in an emergency failover? > Just checking. I restarted the webserver on the backup machine and Pete is now seeing correct data. I'm now proceeding with the failback. Glen From glen@amsl.com Mon Sep 5 20:44:01 2011 Return-Path: X-Original-To: codesprints@ietfa.amsl.com Delivered-To: codesprints@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0FC6F21F844C; Mon, 5 Sep 2011 20:44:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.465 X-Spam-Level: X-Spam-Status: No, score=-101.465 tagged_above=-999 required=5 tests=[AWL=-0.724, BAYES_20=-0.74, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tanhb5700cYu; Mon, 5 Sep 2011 20:44:00 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfa.amsl.com (Postfix) with ESMTP id 2632721F8446; Mon, 5 Sep 2011 20:43:42 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 8112F1C380CE; Mon, 5 Sep 2011 20:45:27 -0700 (PDT) Date: Mon, 5 Sep 2011 20:45:27 -0700 From: Glen To: ietf-announce@ietf.org, ietf@ietf.org Message-ID: <20110906034527.GA27844@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, rpelletier@isoc.org, iesg@ietf.org Subject: [codesprints] Planned IETF outage Monday evening X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 03:44:01 -0000 Dear IETF Community - Previously, I mentioned: > Earlier today a local-area power outage of several square miles occurred in > the area of our Fremont data center. Since that left us running on backup, > we executed an emergency failover to our Orlando data center, where we are > currently running. > Power has been restored, and I will be verifying full completion with the > power crews shortly. Barring additional issues, I plan to transfer control > back to our Fremont data center at 8:30 PM Pacific Time (4:30 AM GMT) this o > evening. The failback procedure has been completed. Thank you for your patience. Glen Glen Barney IT Director AMS (IETF Secretariat) From glen@amsl.com Mon Sep 5 21:53:24 2011 Return-Path: X-Original-To: codesprints@ietfa.amsl.com Delivered-To: codesprints@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E5AD621F8486; Mon, 5 Sep 2011 21:53:24 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.073 X-Spam-Level: X-Spam-Status: No, score=-102.073 tagged_above=-999 required=5 tests=[AWL=-0.073, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id if0DczGZCMwQ; Mon, 5 Sep 2011 21:53:24 -0700 (PDT) Received: from mail.amsl.com (mail.amsl.com [IPv6:2001:1890:123a::1:14]) by ietfa.amsl.com (Postfix) with ESMTP id 8103E21F8479; Mon, 5 Sep 2011 21:53:24 -0700 (PDT) Received: by c1a.amsl.com (Postfix, from userid 1000) id 561281C380B4; Mon, 5 Sep 2011 21:55:10 -0700 (PDT) Date: Mon, 5 Sep 2011 21:55:10 -0700 From: Glen To: Russ Housley , rpelletier@isoc.org Message-ID: <20110906045510.GB11303@amsl.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, iesg@ietf.org Subject: Re: [codesprints] [IAOC] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 04:53:25 -0000 On Mon, Sep 05, 2011 at 09:06:44PM -0400, Russ Housley wrote: > Given this situation, it sounds like you made the right call. You > might want to leave things running in Orlando until tomorrow. Your call ... > Russ Russ - Thank you for the message and the support. I had already started the reconciliation and failback preparation, and then got caught up with doing a verification after Pete's trouble report, so I didn't reply to this earlier. I did speak with PG&E crews after the fact, and they indicated that the problem was a failed part "like a breaker" (apparently I don't get to hear their secret technical terms) and they fixed it and all was well. So, at the risk of jinxing everything, we do not expect any more issues, and I felt it would be best with North America "coming back online" tomorrow to have us back on the primary facility as quickly as possible. Judging that nobody was really doing anything (as confirmed by the marked lack of activity prior to the failover period) except for Pete (who is apparently, like me, someone who works 24/7/365), I felt now would be a more optimal time to fail back, and so proceeded. I hope this is sufficient. Please let me know if you have any questions. Thanks, Glen From housley@vigilsec.com Tue Sep 6 05:10:23 2011 Return-Path: X-Original-To: codesprints@ietfa.amsl.com Delivered-To: codesprints@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 614E621F86A5; Tue, 6 Sep 2011 05:10:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.205 X-Spam-Level: X-Spam-Status: No, score=-102.205 tagged_above=-999 required=5 tests=[AWL=-0.206, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7h85MJq1p+Iu; Tue, 6 Sep 2011 05:10:23 -0700 (PDT) Received: from odin.smetech.net (mail.smetech.net [208.254.26.82]) by ietfa.amsl.com (Postfix) with ESMTP id D09C821F86A4; Tue, 6 Sep 2011 05:10:22 -0700 (PDT) Received: from localhost (unknown [208.254.26.81]) by odin.smetech.net (Postfix) with ESMTP id 00F89F2416D; Tue, 6 Sep 2011 08:12:08 -0400 (EDT) X-Virus-Scanned: amavisd-new at smetech.net Received: from odin.smetech.net ([208.254.26.82]) by localhost (ronin.smetech.net [208.254.26.81]) (amavisd-new, port 10024) with ESMTP id vFEofbZnSLGN; Tue, 6 Sep 2011 08:12:08 -0400 (EDT) Received: from [192.168.2.101] (pool-96-231-29-247.washdc.fios.verizon.net [96.231.29.247]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by odin.smetech.net (Postfix) with ESMTP id 6A13EF2405A; Tue, 6 Sep 2011 08:12:08 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Russ Housley In-Reply-To: <20110906045510.GB11303@amsl.com> Date: Tue, 6 Sep 2011 08:12:07 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <6EAA1F1E-8D65-474E-80CB-5459F9EB6328@vigilsec.com> References: <84086D19-BD02-4EE8-ABC6-CD3F49196770@amsl.com> <20110906005950.GA4153@amsl.com> <20110906045510.GB11303@amsl.com> To: Glen X-Mailer: Apple Mail (2.1084) Cc: wgchairs@ietf.org, iab@ietf.org, codesprints@ietf.org, iaoc@ietf.org, rpelletier@isoc.org, iesg@ietf.org Subject: Re: [codesprints] [IAOC] Failover report X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Sep 2011 12:10:23 -0000 Seems like all is well. Russ On Sep 6, 2011, at 12:55 AM, Glen wrote: > On Mon, Sep 05, 2011 at 09:06:44PM -0400, Russ Housley wrote: >> Given this situation, it sounds like you made the right call. You >> might want to leave things running in Orlando until tomorrow. Your = call ... >> Russ >=20 > Russ - >=20 > Thank you for the message and the support. I had already started the > reconciliation and failback preparation, and then got caught up with = doing > a verification after Pete's trouble report, so I didn't reply to this > earlier. >=20 > I did speak with PG&E crews after the fact, and they indicated that = the > problem was a failed part "like a breaker" (apparently I don't get to = hear > their secret technical terms) and they fixed it and all was well. So, > at the risk of jinxing everything, we do not expect any more issues, = and I > felt it would be best with North America "coming back online" tomorrow = to > have us back on the primary facility as quickly as possible.=20 >=20 > Judging that nobody was really doing anything (as confirmed by the = marked > lack of activity prior to the failover period) except for Pete (who is > apparently, like me, someone who works 24/7/365), I felt now would be = a > more optimal time to fail back, and so proceeded. >=20 > I hope this is sufficient. Please let me know if you have any = questions. >=20 > Thanks, > Glen From rjsparks@nostrum.com Wed Sep 21 08:27:44 2011 Return-Path: X-Original-To: codesprints@ietfa.amsl.com Delivered-To: codesprints@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 21FA921F8459 for ; Wed, 21 Sep 2011 08:27:44 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.941 X-Spam-Level: X-Spam-Status: No, score=-101.941 tagged_above=-999 required=5 tests=[AWL=0.659, BAYES_00=-2.599, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ooZnbOXyI1na for ; Wed, 21 Sep 2011 08:27:43 -0700 (PDT) Received: from nostrum.com (nostrum-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:267::2]) by ietfa.amsl.com (Postfix) with ESMTP id A213C21F84D8 for ; Wed, 21 Sep 2011 08:27:43 -0700 (PDT) Received: from [192.168.2.105] (pool-173-71-46-224.dllstx.fios.verizon.net [173.71.46.224]) (authenticated bits=0) by nostrum.com (8.14.3/8.14.3) with ESMTP id p8LFUCT0049696 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Wed, 21 Sep 2011 10:30:12 -0500 (CDT) (envelope-from rjsparks@nostrum.com) From: Robert Sparks Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Wed, 21 Sep 2011 10:30:12 -0500 Message-Id: <53B3B3BE-276C-4E36-AD15-7D5CEB6CC195@nostrum.com> To: codesprints@ietf.org Mime-Version: 1.0 (Apple Message framework v1084) X-Mailer: Apple Mail (2.1084) Received-SPF: pass (nostrum.com: 173.71.46.224 is authenticated by a trusted mechanism) Subject: [codesprints] Please sign up for the sprint at IETF82 if you'll be attending. X-BeenThere: codesprints@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: "List for coordinating \(and following up on\) codesprint activities" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Sep 2011 15:27:44 -0000 If you'll be joining us in Taipei, please add yourself to = as = soon as you can. Thanks, RjS=