Per-Entity Metadata Working Group

[David Walker <>]

Just thinking about a future where there are many more entities, probably with a skewed distribution of the number of entities per member organization...

I agree that it's possible to build a federation-wide infrastructure, but it means that everyone pays for that infrastructure and must agree on the level of service it provides.  Right now, such a strategy can be rolled into all the other effort required to consume per-entity metadata (which, presumably, is seen as a positive thing).  Later, it has the potential to be a negative thing that has to be done because the federation doesn't maintain a high enough level of service ("high enough" being measured by individual member organizations).

(By the way, even though my electronic mail address ends with "@internet2.edu," please don't interpret what I'm saying as an InCommon position.  I'm just exploring alternatives.)

David

On 08/01/2016 12:18 PM, Cantor, Scott wrote:

Or what he said. Should have waited like 30 seconds.

-- Scott

-----Original Message-----
From:  [-
] On Behalf Of Domingues, Michael D
Sent: Monday, August 1, 2016 3:12 PM
To: David Walker ; Per-Entity Metadata Working
Group 
Subject: Re: [Per-Entity] Some thoughts about availability and scalability

Your second point (load on federation infrastructure) shifting from a function
of aggregate federation size to a function of aggregate federation login
activity is what I was trying to hop in and make on the call last week. I think
my microphone must have been on the fritz at the time.

That said, I'd argue that even though load is technically "unpredictable", it
absolutely can be profiled, and infrastructure can be built out to support it
accordingly. If we're introducing a caching / access layer of Apache or some
other web server to be run by the Identity Federation instead of just
pointing clients at an instance of MDQ server (a wise play), while not a walk in
the park, this ought to be a fairly easy infrastructure to architect, build out,
and operate sustainably.

From Apache's perspective, it'll be serving individually small, static files. It
does this all day, every day, for hundreds of thousands of organizations,
without fail. Add in cache lifetimes on these particular responses, and I think
this is a non-issue. You allocate adequate resources to account for expected
peak load, load balance things around, and call it good.

I'm incredibly wary of shifting the burden of running this infrastructure onto
individual institutions. Donning the hat of a generic IdP operator for a
moment, if I saw that there was a per-entity metadata service, but I was
responsible for building out a caching layer myself, frankly, I wouldn't bother
with using the service. And I'm at a large institution. Things would only be
worse for smaller federation members without the technical capability (or
human resources) to do so.

Michael
________________________________________
From:  <per-entity-
> on behalf of David Walker

Sent: Monday, August 1, 2016 12:11:13 PM
To: Per-Entity Metadata Working Group
Subject: [Per-Entity] Some thoughts about availability and scalability

I've been thinking about a couple of things...

  *   Setting an expectation that MDQ client software should protect itself
from failures of the server infrastructure.  We've talked a good amount
about the pros and cons of this.
  *   Another risk to per-entity metadata distribution we haven't discussed,
that the server infrastructure may not be able to handle peak loads.

A federation-provided MDQ service must be able to handle two types of
load, 1) metadata updates, and 2) queries from client IdPs and SPs.  The first
of these is slow and fairly predictable at a federation level, but the latter is
not.  Queries from IdPs and SPs will vary rapidly and unpredictably, based on
the workload demands of individual federation members, but all federation
members bear the impact.  The UK approach puts the unpredictable load on
the web servers, which is better than putting it on the MDQ server, but it's
still unpredictable.

The next thing I realized is that the UK's approach of creating an Apache-like
web server layer between the MDQ server and the client IdPs and SPs
doesn't require that the web servers be run by the federation.  They can be
run by the member institutions:

[cid:]

Doing things this way lets each member institution decide what level of
availability and scalability they want to provide to their community and
deploy the necessary infrastructure without affecting the rest of the
federation.  The federation is responsible only for the scalability and
availability of the MDQ server behind the web servers.

Make sense?

David

Attachment: signature.asc
Description: OpenPGP digital signature