InCommon metadata support

[Jeffrey Eaton <>]

Sorry cut and past error.

This is what works (signature first, then filtering):

<MetadataProvider type="XML" file="incommon.xml" reloadInterval="7200">
  <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
  <MetadataFilter type="Signature" certificate="incommon.pem"/>
  <MetadataFilter type="EntityRoleWhiteList">
    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
  </MetadataFilter>
</MetadataProvider>

but as Scott pointed out, because on Linux the memory isn't really released, 
it still ends up chewing up a ton of memory.  I had a report from an SP 
operator indicating that it's even worse than just the ~250MB that it 
consumes just after startup, after the initial parsing.  When a new file is 
reloaded (in a background thread, so it doesn't block requests) the memory 
usage can grow to ~700MB.  Looking at my test SP, that does seem to be the 
case:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
shibd     3663  0.0 16.9 960920 684208 ?       Ssl  Jan16   7:29 
/usr/sbin/shibd -p /var/run/shibboleth/shibd.pid -f -w 30

684 MB is a lot of memory to chew up.  

The export aggregate is working well for the few places I've started using 
it, so that seems to be a reasonable enough solution for now.

-jeaton



> On Feb 19, 2016, at 3:08 PM, Tom Scavo 
> <>
>  wrote:
> 
> On Thu, Feb 18, 2016 at 7:06 PM, Jeffrey Eaton 
> <>
>  wrote:
>> 
>>> On Feb 18, 2016, at 12:52 PM, Tom Scavo 
>>> <>
>>>  wrote:
>>> 
>>> Jeffrey, can you add this filter and see if it makes a difference?
>>> 
>>> <!-- Consume all IdP metadata in the aggregate -->
>>> <MetadataFilter type="EntityRoleWhiteList">
>>>   <RetainedRole>md:IDPSSODescriptor</RetainedRole>
>>>   <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
>>> </MetadataFilter>
>>> 
>>> See: https://spaces.internet2.edu/x/XAQjAQ
>> 
>> If I put them in this order:
>> <MetadataProvider type="XML" file="incommon.xml" reloadInterval="7200">
>>  <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
>>  <!-- Consume all IdP metadata in the aggregate —>
>>  <MetadataFilter type="EntityRoleWhiteList">
>>    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
>>    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
>>  </MetadataFilter>
>>  <MetadataFilter type="Signature" certificate="incommon.pem”/>
>> </MetadataProvider>
>> 
>> it fails with:
>> 
>> 2016-02-18 18:43:33 INFO XMLTooling.SecurityHelper : loading 
>> certificate(s) from file (/etc/shibboleth/incommon.pem)
>> 2016-02-18 18:43:33 INFO XMLTooling.CredentialResolver.File : no private 
>> key resolved, usable for verification/trust only
>> 2016-02-18 18:43:33 INFO XMLTooling.StorageService : cleanup thread 
>> started...running every 900 seconds
>> 2016-02-18 18:43:33 INFO Shibboleth.SecurityPolicyProvider.XML : reload 
>> thread started...running when signaled
>> 2016-02-18 18:43:33 INFO OpenSAML.MetadataProvider.XML : loaded XML 
>> resource (/etc/shibboleth/incommon.xml)
>> 2016-02-18 18:43:35 INFO OpenSAML.Metadata : applying metadata filter 
>> (RequireValidUntil)
>> 2016-02-18 18:43:35 INFO OpenSAML.Metadata : applying metadata filter 
>> (EntityRoleWhiteList)
>> 2016-02-18 18:43:36 INFO OpenSAML.Metadata : applying metadata filter 
>> (Signature)
>> 2016-02-18 18:43:36 WARN OpenSAML.MetadataFilter.Signature : filtering out 
>> group at root of instance after failed signature check: Invalid signature 
>> profile for
>> SAML object.
>> 2016-02-18 18:43:36 CRIT Shibboleth.Application : error initializing 
>> MetadataProvider: SignatureMetadataFilter unable to verify signature at 
>> root of metadata ins
>> tance.
>> 
>> If I put them in this order:
>> 
>> <MetadataProvider type="XML" file="incommon.xml" reloadInterval="7200">
>>  <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
>>  <!-- Consume all IdP metadata in the aggregate —>
>>  <MetadataFilter type="EntityRoleWhiteList">
>>    <RetainedRole>md:IDPSSODescriptor</RetainedRole>
>>    <RetainedRole>md:AttributeAuthorityDescriptor</RetainedRole>
>>  </MetadataFilter>
>>  <MetadataFilter type="Signature" certificate="incommon.pem”/>
>> </MetadataProvider>
>> 
>> The signature check passes (just as slowly as without the 
>> EntityRoleWhiteList filter), and the filtering completes but doesn’t seem 
>> to have much effect on the in-memory resident size (still taking ~250MB).
> 
> I must be missing something since those two MetadataProviders look
> exactly the same. Jeffrey, can you double-check what you wrote?
> 
> Tom