InCommon metadata support

[Jeffrey Eaton <>]

> On Feb 18, 2016, at 12:52 PM, Tom Scavo 
> <>
>  wrote:
> 
> On Thu, Feb 18, 2016 at 11:24 AM, Cantor, Scott 
> <>
>  wrote:
>>> I wrote some scripts to take the InCommon metadata file, split it into two
>>> files - one containing just SPs, the other containing IDPs, and then 
>>> signed
>>> them both with my own certificate (with the same sha256 signatures).
>>> Loading just the IDP metadata file (which is about 13MB) and starting 
>>> shibd
>>> takes about 20 seconds, compared to over a minute for loading the whole
>>> InCommon file.  Furthermore, the resident size of the shibd process drops
>>> from 256MB with the full InCommon file, down to about 120MB with just the
>>> IDP metadata loaded.
>> 
>> You can easily drop roles you don't want anyway, so I doubt that's a 
>> material difference, though obviously the verification time is.
> 
> +1
> 
> Jeffrey, can you add this filter and see if it makes a difference?
> 
>  <!-- Consume all IdP metadata in the aggregate -->
>  <MetadataFilter type="EntityRoleWhiteList">
>    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
>    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
>  </MetadataFilter>
> 
> See: https://spaces.internet2.edu/x/XAQjAQ
> 

If I put them in this order:
<MetadataProvider type="XML" file="incommon.xml" reloadInterval="7200">
  <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
  <!-- Consume all IdP metadata in the aggregate —>
  <MetadataFilter type="EntityRoleWhiteList">
    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
  </MetadataFilter>
  <MetadataFilter type="Signature" certificate="incommon.pem”/>
</MetadataProvider>

it fails with:

2016-02-18 18:43:33 INFO XMLTooling.SecurityHelper : loading certificate(s) 
from file (/etc/shibboleth/incommon.pem)
2016-02-18 18:43:33 INFO XMLTooling.CredentialResolver.File : no private key 
resolved, usable for verification/trust only
2016-02-18 18:43:33 INFO XMLTooling.StorageService : cleanup thread 
started...running every 900 seconds
2016-02-18 18:43:33 INFO Shibboleth.SecurityPolicyProvider.XML : reload 
thread started...running when signaled
2016-02-18 18:43:33 INFO OpenSAML.MetadataProvider.XML : loaded XML resource 
(/etc/shibboleth/incommon.xml)
2016-02-18 18:43:35 INFO OpenSAML.Metadata : applying metadata filter 
(RequireValidUntil)
2016-02-18 18:43:35 INFO OpenSAML.Metadata : applying metadata filter 
(EntityRoleWhiteList)
2016-02-18 18:43:36 INFO OpenSAML.Metadata : applying metadata filter 
(Signature)
2016-02-18 18:43:36 WARN OpenSAML.MetadataFilter.Signature : filtering out 
group at root of instance after failed signature check: Invalid signature 
profile for 
SAML object.
2016-02-18 18:43:36 CRIT Shibboleth.Application : error initializing 
MetadataProvider: SignatureMetadataFilter unable to verify signature at root 
of metadata ins
tance.

If I put them in this order:

<MetadataProvider type="XML" file="incommon.xml" reloadInterval="7200">
  <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
  <!-- Consume all IdP metadata in the aggregate —>
  <MetadataFilter type="EntityRoleWhiteList">
    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
  </MetadataFilter>
  <MetadataFilter type="Signature" certificate="incommon.pem”/>
</MetadataProvider>

The signature check passes (just as slowly as without the EntityRoleWhiteList 
filter), and the filtering completes but doesn’t seem to have much effect on 
the in-memory resident size (still taking ~250MB).

>>> Is there any particular reason why InCommon puts all of the SPs and IDPs
>>> together into one big metadata file?  As I understand it, the Service 
>>> Provider
>>> only needs to load the metadata for IDPs, and vice-versa, so there's a 
>>> lot of
>>> bloat in the file that doesn't really need to be there for normal 
>>> operations.
>> 
>> That's obviously up to InC Ops. I would imagine the answer is that it 
>> never mattered much before and now it might help, but only in a relatively 
>> small way since one doesn't restart shibd very often.
> 
> I'll deflect that question momentarily :-) and suggest that Jeffrey
> try the export aggregate. See: https://spaces.internet2.edu/x/vACVBQ
> 
> The export aggregate currently has 402 of the 424 total IdPs in the
> InCommon Federation. Not sure that meets your production needs but at
> least it might be useful for experimental purposes.

Loading that aggregate complete in 2 seconds, and the resulting shibd process 
only takes up about 40 MB of RAM.  That actually seems like a very reasonable 
compromise.  I suppose ideally, InCommon would produce something like this 
with all of the InCommon IDPs (I’m not sure which 22 are missing, but I’d 
imagine someone would find it useful to have the full set of InCommon only 
IDPs available.

I’m still evaluating what we’re going to recommend to our SP operators, but 
in the worst case, I can have them use the export aggregate, or my split and 
re-signed IDP-only aggregate, with no further changes needed from the 
InCommon side.

Thanks,

-jeaton