Per-Entity Metadata Working Group

["Domingues, Michael D" <>]

Your second point (load on federation infrastructure) shifting from a 
function of aggregate federation size to a function of aggregate federation 
login activity is what I was trying to hop in and make on the call last week. 
I think my microphone must have been on the fritz at the time.

That said, I'd argue that even though load is technically "unpredictable", it 
absolutely can be profiled, and infrastructure can be built out to support it 
accordingly. If we're introducing a caching / access layer of Apache or some 
other web server to be run by the Identity Federation instead of just 
pointing clients at an instance of MDQ server (a wise play), while not a walk 
in the park, this ought to be a fairly easy infrastructure to architect, 
build out, and operate sustainably.

From Apache's perspective, it'll be serving individually small, static files. 
It does this all day, every day, for hundreds of thousands of organizations, 
without fail. Add in cache lifetimes on these particular responses, and I 
think this is a non-issue. You allocate adequate resources to account for 
expected peak load, load balance things around, and call it good.

I'm incredibly wary of shifting the burden of running this infrastructure 
onto individual institutions. Donning the hat of a generic IdP operator for a 
moment, if I saw that there was a per-entity metadata service, but I was 
responsible for building out a caching layer myself, frankly, I wouldn't 
bother with using the service. And I'm at a large institution. Things would 
only be worse for smaller federation members without the technical capability 
(or human resources) to do so.

Michael
________________________________________
From: 

 
<>
 on behalf of David Walker 
<>
Sent: Monday, August 1, 2016 12:11:13 PM
To: Per-Entity Metadata Working Group
Subject: [Per-Entity] Some thoughts about availability and scalability

I've been thinking about a couple of things...

  *   Setting an expectation that MDQ client software should protect itself 
from failures of the server infrastructure.  We've talked a good amount about 
the pros and cons of this.
  *   Another risk to per-entity metadata distribution we haven't discussed, 
that the server infrastructure may not be able to handle peak loads.

A federation-provided MDQ service must be able to handle two types of load, 
1) metadata updates, and 2) queries from client IdPs and SPs.  The first of 
these is slow and fairly predictable at a federation level, but the latter is 
not.  Queries from IdPs and SPs will vary rapidly and unpredictably, based on 
the workload demands of individual federation members, but all federation 
members bear the impact.  The UK approach puts the unpredictable load on the 
web servers, which is better than putting it on the MDQ server, but it's 
still unpredictable.

The next thing I realized is that the UK's approach of creating an 
Apache-like web server layer between the MDQ server and the client IdPs and 
SPs doesn't require that the web servers be run by the federation.  They can 
be run by the member institutions:

[cid:]

Doing things this way lets each member institution decide what level of 
availability and scalability they want to provide to their community and 
deploy the necessary infrastructure without affecting the rest of the 
federation.  The federation is responsible only for the scalability and 
availability of the MDQ server behind the web servers.

Make sense?

David

Attachment: odlpjdcngjlbbngb.png
Description: odlpjdcngjlbbngb.png