InCommon Operations Notifications

["Nicole Roy" <>]

AWS has notified us that this issue has been resolved. We are closing the issue within InCommon Operations. We will do an after-action review after we are through the next couple weeks of eduroam-related activities that we are engaged in, and will prepare a report based on that AAR. We will also work with the InCommon Technical Advisory Committee on this.

Best Regards,

Nicole Roy
Pronouns: She/her/hers
Director of Technology and Strategy
InCommon / Internet2 Trust and Identity Services

On 8 Oct 2021, at 14:04, Nicole Roy wrote:

We are seeing signs that some of the CloudFront edge locations that were formerly failing may be starting to return correct MDQ results now. We have not received any confirmation from AWS that this is the case, but we are cautiously optimistic. I’ve updated the outage wiki page, linked below. We will communicate more as we receive more information from AWS.

Thank you,

Nicole

On 8 Oct 2021, at 12:23, Nicole Roy wrote:

Hello,

We continue to pursue this issue with AWS engineers. They have identified a problem with encoding certain characters from a request, which causes them not to be able to correctly resolve an entity descriptor within the S3 origin location that backs our MDQ service. They are working on a fix. As of this update, we do not have an ETA from them, but they assure us that this has the highest internal priority within AWS.

Updates will continue to be posted at: https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage

Best Regards,

Nicole

On 7 Oct 2021, at 14:34, Nicole Roy wrote:

Another update, we have identified a method of temporarily pinning MDQ requests to a specific, currently-known-good CloudFront edge location. Details at: https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage

Deployers are advised to test this approach locally, and ensure that whichever CloudFront edge location they pick, actually works before updating hosts files on their affected systems.

Best,

Nicole

On 7 Oct 2021, at 12:35, Nicole Roy wrote:

All future status updates will be posted in the Internet2 wiki, at: https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage

As of 12:30 MDT, AWS support is able to reproduce this issue and is escalating it within AWS.

Best,

Nicole

On 7 Oct 2021, at 10:06, Nicole Roy wrote:

Update 2020-10-07 10:00 MDT

We have opened a support case with Amazon Web Services with regard to the edge cache locations which appear to be failing:

13.32.208.20
13.32.208.84
13.32.208.58
13.32.208.38

On 7 Oct 2021, at 9:31, Nicole Roy wrote:

As of 9:15 a.m. MDT on October 7, we are able to reproduce the issue and it appears to be isolated to a number of CloudFront servers. We are working to further identify the problem with these distribution points.

Best Regards,

Nicole

On 7 Oct 2021, at 9:06, Nicole Roy wrote:

Hello,

This morning, InCommon operations was notified of some intermittent issues by some metadata clients, in retrieving SAML metadata from https://mdq.incommon.org. We are actively investigating this issue and will share more information when we know more. In the meantime, if you are experiencing issues with the MDQ service, please send email to , including (if possible) detailed information about the client you’re using (SAML software and version), IP address of the client, physical location (approximate), log data if share-able, etc.

Best Regards,

Nicole Roy
Pronouns: She/her/hers
Director of Technology and Strategy
InCommon / Internet2 Trust and Identity Services

Attachment: signature.asc
Description: OpenPGP digital signature