InCommon Operations Notifications

[Tom Scavo <>]

FYI

----- Forwarded Message -----
From: "Tom Scavo" 
<>
To: 

Sent: Monday, July 9, 2012 8:29:43 AM
Subject: Re: [InC] Re: power outage at Internet2 Ann Arbor offices

Hi,

Just to followup, commercial power was restored to the Internet2 offices in 
Ann Arbor at approximately 5:00pm on Friday, July 6. As of yesterday (July 
8), all InCommon/Internet2 online services and processes are fully 
operational.

The power outage lasted approximately 60 hours. AFAIK, InCommon online 
services suffered no significant downtime. We were not able to sign metadata 
on Thursday, July 5 as normal, but we did sign metadata twice on Friday, July 
6 to compensate.

A big THANK YOU to all the Internet2/InCommon staff who responded to this 
situation with dedication and resolve. In particular, the wonderful staff of 
the Internet2 Technical Services Group (led by Mike LaHaye) deserve a huge 
round of applause. Their efforts are MUCH APPRECIATED to be sure!

Tom

----- Original Message -----
> Good morning! We are still without power at the Internet2 offices in
> Ann Arbor. Although the heat in the building is rising due to lack
> of air conditioning, essential online services remain operational at
> this time. Please note:
> 
> WE WILL SIGN METADATA TODAY (Friday, July 6, 2012)
> 
> We intend to sign metadata at the regular time (~3:00pm ET) but may
> have to sign earlier if the temperature in the building continues to
> rise. In anticipation of this signing event, please update your
> metadata asap.
> 
> Once the metadata signing process is complete, we will power down
> service1.internet2.edu, which is the host for the Federation
> Manager. It will stay down until normal operating temperatures in
> the building are restored.
> 
> Host wayf.incommonfederation.org, which is now operating at our
> sister site in Indiana, will remain operational, indefinitely. In
> other words, the InCommon metadata service and the InCommon
> discovery service will continue to operate despite what happens here
> in Ann Arbor.
> 
> The National Weather Service predicts the temperature in Ann Arbor
> will reach 101 degrees (Fahrenheit) today. Tomorrow will be cooler
> with a high of 98 degrees. It is not known when public power will be
> restored to the Ann Arbor area.
> 
> Tom
> 
> ----- Original Message -----
> > A severe thunderstorm ripped through Ann Arbor early this morning,
> > causing widespread power outages, including the Internet2 offices.
> > Thanks to an automatic backup power system in our building, all
> > InCommon online services, including the InCommon metadata service
> > and the InCommon discovery service, are fully operational. At
> > approximately 5:15am EDT, these InCommon services were manually
> > failed over to a twin host in Indiana as a precaution. There is
> > nothing you need to do at this time.
> > 
> > Primary power to the Internet2 offices in Ann Arbor is still down.
> > Since the local area is in the midst of an excessive heat wave, all
> > Ann Arbor staff have been told to stay home. As a result,
> > 
> > THERE WILL BE NO METADATA SIGNING EVENT TODAY (Thursday, July 5,
> > 2012)
> > 
> > Metadata signing will resume tomorrow at the normal time (weather
> > permitting). If you have an emergency that needs immediate
> > attention, or just have questions or comments, please send them to
> > 
> > 
> > We appreciate your patience,
> > 
> > Tom Scavo
> > Operations Manager
> > InCommon.org
> > https://twitter.com/trscavo
> > 
> > 
>