InCommon Technical Discussions

[Albert Wu <>]

Hello,

Regarding the timing of MDQ meta refresh, it is a temporary side effect of as 
we transition from manual metadata signing to the fully automated one in AWS. 
This is why it's happening at 5PM ET right now:

The current MDQ refresh process is a scheduled job. For now, it uses the 
(legacy) InCommon metadata aggregate as input. The InCommon metadata 
signing/publishing is a manual procedure occurring daily at around 3PM ET. To 
guard against unforeseen delays in the aggregate signing, we scheduled the 
MDQ refresh job for 2 hours after the 3PM legacy aggregate publishing. 

As mentioned, this is temporary. We are completing a new, automated signing 
process built in AWS. To transition away from that manual signing, we need to 
move the entire metadata production pipeline, including Federation Manager, 
in to AWS. We still need to move Federation Manager (scheduled this winter). 
Once we complete the move, metadata signing/publishing becomes automated and 
therefore more flexible from a timing perspective.

In the interim, we are looking into possibly increasing the frequency of the 
MDQ refresh jobs to more than once daily to lessen the delays until  
(especially if there were emergency publishing outside the normal scheduled 
aggregate publishing).

albert


On 10/23/19, 7:49 AM, " on behalf of 
Basney, Jim" < on behalf of 
> wrote:

    Hi,
    
    If I understand correctly, metadata published by mdq.incommon.org is 
updated each business day at 5pm ET. If true, I request InCommon to 
reconsider this schedule.
    
    For comparison, I see updates from 
http://md.incommon.org/InCommon/InCommon-metadata.xml each business day 
around 3pm ET, which gives us a few hours left in the workday to identify any 
problems caused by the metadata update and contact  for 
assistance when needed. By contrast, since switching to mdq.incommon.org, 
problems with the metadata appear late in the evening, and we don't hear back 
from  until the next business day.
    
    For example, on Fri Oct 18 around 5pm ET 
https://idp.fnal.gov/idp/shibboleth was removed from InCommon metadata, 
causing problems for Fermilab users trying to log on to CILogon over the 
weekend. Since  was unable to assist until Monday morning, 
we configured a manual metadata update on the CILogon servers to continue 
providing service over the weekend. On Monday morning, InCommon acted quickly 
to add https://idp.fnal.gov/idp/shibboleth back in to 
http://md.incommon.org/InCommon/InCommon-metadata.xml at 9:26 ET, but since 
CILogon now uses mdq.incommon.org, we didn't see the update until 5pm ET on 
Monday. It took me a while on Monday afternoon to figure out why the CILogon 
servers weren't seeing the metadata update after  told us 
the problem was resolved.
    
    My expectation was that the move to MDQ would deliver more up-to-date 
metadata, not less.
    
    Sincerely,
    Jim