Per-Entity Metadata Working Group

["Cantor, Scott" <>]

Or what he said. Should have waited like 30 seconds.

-- Scott

> -----Original Message-----
> From: 
> 
>  [
> ]
>  On Behalf Of Domingues, Michael D
> Sent: Monday, August 1, 2016 3:12 PM
> To: David Walker 
> <>;
>  Per-Entity Metadata Working
> Group 
> <>
> Subject: Re: [Per-Entity] Some thoughts about availability and scalability
> 
> Your second point (load on federation infrastructure) shifting from a 
> function
> of aggregate federation size to a function of aggregate federation login
> activity is what I was trying to hop in and make on the call last week. I 
> think
> my microphone must have been on the fritz at the time.
> 
> That said, I'd argue that even though load is technically "unpredictable", 
> it
> absolutely can be profiled, and infrastructure can be built out to support 
> it
> accordingly. If we're introducing a caching / access layer of Apache or some
> other web server to be run by the Identity Federation instead of just
> pointing clients at an instance of MDQ server (a wise play), while not a 
> walk in
> the park, this ought to be a fairly easy infrastructure to architect, build 
> out,
> and operate sustainably.
> 
> From Apache's perspective, it'll be serving individually small, static 
> files. It
> does this all day, every day, for hundreds of thousands of organizations,
> without fail. Add in cache lifetimes on these particular responses, and I 
> think
> this is a non-issue. You allocate adequate resources to account for expected
> peak load, load balance things around, and call it good.
> 
> I'm incredibly wary of shifting the burden of running this infrastructure 
> onto
> individual institutions. Donning the hat of a generic IdP operator for a
> moment, if I saw that there was a per-entity metadata service, but I was
> responsible for building out a caching layer myself, frankly, I wouldn't 
> bother
> with using the service. And I'm at a large institution. Things would only be
> worse for smaller federation members without the technical capability (or
> human resources) to do so.
> 
> Michael
> ________________________________________
> From: 
> 
>  <per-entity-
> >
>  on behalf of David Walker
> <>
> Sent: Monday, August 1, 2016 12:11:13 PM
> To: Per-Entity Metadata Working Group
> Subject: [Per-Entity] Some thoughts about availability and scalability
> 
> I've been thinking about a couple of things...
> 
>   *   Setting an expectation that MDQ client software should protect itself
> from failures of the server infrastructure.  We've talked a good amount
> about the pros and cons of this.
>   *   Another risk to per-entity metadata distribution we haven't discussed,
> that the server infrastructure may not be able to handle peak loads.
> 
> A federation-provided MDQ service must be able to handle two types of
> load, 1) metadata updates, and 2) queries from client IdPs and SPs.  The 
> first
> of these is slow and fairly predictable at a federation level, but the 
> latter is
> not.  Queries from IdPs and SPs will vary rapidly and unpredictably, based 
> on
> the workload demands of individual federation members, but all federation
> members bear the impact.  The UK approach puts the unpredictable load on
> the web servers, which is better than putting it on the MDQ server, but it's
> still unpredictable.
> 
> The next thing I realized is that the UK's approach of creating an 
> Apache-like
> web server layer between the MDQ server and the client IdPs and SPs
> doesn't require that the web servers be run by the federation.  They can be
> run by the member institutions:
> 
> [cid:]
> 
> Doing things this way lets each member institution decide what level of
> availability and scalability they want to provide to their community and
> deploy the necessary infrastructure without affecting the rest of the
> federation.  The federation is responsible only for the scalability and
> availability of the MDQ server behind the web servers.
> 
> Make sense?
> 
> David