Per-Entity Metadata Working Group

[Tom Scavo <>]

Patrick, thanks for setting this up! I've been running experiments for
the last three days :-)

Basic Method: Query each IdP registered by InCommon (440 in all) to
initialize the cache. Immediately query each IdP again to obtain
cached results.

All timings reported below are medians (not averages). This accounts
for any skewed data that might exist.

All timings reported below are curl total_time (for consistency with
other reports). However, see the discussion that follows.

WARNING: All timings (not just mine) are suspect since the metadata
refresh schedule on mdq-beta is opaque. Without inspecting the logs,
you don’t know when the next refresh is due, and therefore you don’t
know when to run an experiment. An intervening refresh potentially
impacts timing experiments since a refresh invalidates metadata in the
cache.

Experiment on mdq-beta directly:
On average, each query took 0.091 sec and 0.083 sec, respectively, for
each replication. That is, on average there is 8 ms of overhead on
mdq-beta.

I'll let Ian explain where that 8 ms comes from.

Experiment #1 via Cloudfront:
On average, each request took 0.082 sec and 0.049 sec, respectively,
for each replication. That is, there is 33 ms of time savings due to
caching on Cloudfront.

Experiment #2 via Cloudfront:
I repeated the above experiment on a different day. On average, each
request took 0.109 sec and 0.069 sec, respectively, for each
replication. That is, there is 40 ms of time savings due to caching on
Cloudfront on this particular day.

Discussion

Three events dominate the total time to complete a transaction:

1. How long does it take to resolve a DNS name?
2. How long does it take to establish a connection with the server?
3. How long does it take for the server to start sending the data?

The latter begins when the HTTP handshake is complete and ends once
the first byte is transmitted over the wire. It is the only time
interval under our control.

Curl reports the following timings (in order):

time_namelookup
time_connect
time_pretransfer
time_starttransfer
time_total

The most important timing is time_starttransfer (not time_total) since
that tells you how long it takes for the server to begin data
transfer. This suggests the following simple algorithm:

1) Record time_pretransfer and time_starttransfer for each query
2) Compute the difference (time_starttransfer - time_pretransfer) for each 
query
3) Analyze and compare the differences

The difference (time_starttransfer - time_pretransfer) tells you how
much time the server spends fetching and preparing the data to be
transferred, which is the only operation under our direct control.

Tom


On Thu, Sep 1, 2016 at 7:14 PM, Patrick Radtke
<>
 wrote:
> I realized I didn't reply to the list when I wrote this.
> I setup Cloudfront as a caching CDN for the beta mdq server. Details
> in the forwarded message below, along with my latency figures.
>
>
> ---------- Forwarded message ----------
> From: Patrick Radtke 
> <>
> Date: Fri, Aug 26, 2016 at 3:57 PM
> Subject: Re: [Per-Entity] Latency figures for CDNs
> To: David Walker 
> <>
>
>
> On Fri, Aug 26, 2016 at 1:56 PM, David Walker 
> <>
>  wrote:
>> Getting back to Nick's original question, I guess I am concerned about the
>> latency times for 12KB objects (~0.4 secs in the US), unless we think
>> InCommon sites have much better connectivity than Frost and Sullivan's
>> average (and that the predominant latency factor is the last-mile network).
>>
>> This, of course, brings us back to Chris's admonition that we focus on
>> business needs.  What delay is acceptable when an IdP or SP needs to
>> retrieve the metadata for some other SP or IdP?  0.4 seconds sounds pretty
>> long to me, but it is a rare event, we think.
>
> I setup Cloudfront (US, Canada and Europe locations) to be a caching
> proxy for the beta MDQ server.
> A cache miss in the edge location closest to you will result in a
> query going to the MDQ server. Cloud front will cache the result for
> an hour.
> You can test it by using http://drhqoesel6yr5.cloudfront.net/ in place
> of http://mdq-beta.incommon.org/
>
> Using curl to do the timing
>
> curl -o /dev/null -s -w \
> %{time_connect}:%{time_starttransfer}:%{time_total} \
> http://drhqoesel6yr5.cloudfront.net/global/entities/https%3A%2F%2Ffm.incommon.org%2Fsp
>
>
> I get, for a cache hit,
> 10-20 ms from an EC2 instance
> 33-50 ms from home
> cache misses seemed to add 1 second.
>
> If I hit the MDQ beta server directly I get
> 10-30 ms from EC2 instance (this surprised me)
> 175-200 ms from home.
>
> I'd be curious to see what the variance is for people on this list.
> Your first curl will be likely be a cache miss in the CDN.
>
> -Patrick
>
>
> --
> Patrick Radtke
> CTO
> Cirrus Identity