Per-Entity Metadata Working Group

[Tom Mitchell <>]

[I know the conversation has moved on, but I had already run some numbers and wanted to record it for later]

On Aug 5, 2016, at 9:22 AM, Cantor, Scott <> wrote:

The tricky part is that you need to know how many different SPs or IdPs
a peer is connecting with, because hitting the same one over and over
just hits the cache. For example, I can cull logs and find out how many
different IdPs log into my services, though of course it would be a hell
of a lot easier if I customize the SP logging format first.

The MDQ load isn't based on your load, but the heterogeneity of your
load. Does that make sense?

Yes, absolutely. And I guess it’s pretty hard to estimate that.

I did a little exercise with my own Shibboleth SP logs. I understand my SP is not representative, but it’s the type of exercise I like. After a little shell pipelining, I have 2,460 unique ‘New session’ lines from 60 unique IdPs in the transaction log for the month of July. Let’s assume all were InCommon IdPs (they weren’t). Without any caching that would amount to an average load of 54 MDQ hits per day. (You can see why I’d be tolerant of outages.)

What if we do a case study? Let’s assume that every InCommon SP (3,107) and every InCommon IdP (434) are fully connected. That would require ~2.7 million MDQ queries. That’s every SP asking about every IdP and vice versa.

Now comes the question of how long or short the endpoints can cache that data. I ran numbers for a few different cache lifetimes, from what I consider to be very short (10 minutes) up to the InCommon recommendation of at least daily (i.e. the minimum refresh).

10 minutes = 4,500 MDQ queries/second average

1 hour = 750 MDQ queries/second average

4 hours = 188 MDQ queries/second average

6 hours = 125 MDQ queries/second average

24 hours = 31 MDQ queries/second average

These numbers assume a fully connected InCommon federation, which is not the case in practice. These numbers also ignore the size of the eduGAIN data. But there again, in practice it is not a fully connected federation. These numbers also assume immediate refresh when the data expires, which is unlikely in practice.

A little more math, it looks like the metadata is around 10kb/entry. So for the 1 hour refresh rate above, that’s roughly 7.5 megabytes/second of network traffic outbound, not counting HTTP overhead.

Are these kinds of calculations useful to inform requirements related to potential load?

Taking Shibboleth as an exemplar, how often would the MDQ entries expire, causing a new query to go out? Or is that determined by the HTTP headers received from the MDQ server?

Does my math look correct?

Thanks,

Tom