[Idpy-discuss] Questions about pyFF metadata downloads

Leif Johansson leifj at sunet.se
Fri Sep 13 13:08:30 UTC 2019


On 2019-09-13 11:51, Alex Stuart wrote:
> Hello folks,
> 
> The UK federation team have discovered that a pyFF deployment is making a large number of metadata aggregate downloads from our Metadata Publication Service. In August, 34.74.200.81 made over 3,000 gzipped downloads of our metadata, downloading 36GB of metadata. As we update metdadata once per day, this deployment is clearly downloading excessively.

So its a thumb drive worth of data :-)

> 
> The IP address 34.74.200.81 is somewhere in Google cloud (81.200.74.34.bc.googleusercontent.com.) and the user agent string is pyFF/1.0.1. Is this anyone's deployment?

I don't think thats anyone I own personally ... but it is a relatively
fresh version of pyFF tho.

> 
> We also recommend that metadata downloaders make use of the conditional GET. Is it possible to configure pyFF to do conditional GET for metadata downloads?

What is a conditional GET?

By default pyFF uses python-requests with the requests-caching on
by default which means that if the endpoint implements normal HTTP
cache controls the pyFF user-agent shouldn't download unless the
cache is expired (which is configurable of course).

	Cheers Leif

> 
> Thanks,
> Alex
> 
> 
>> Alex Stuart, Principal technical support specialist (UK federation)               
> alex.stuart at jisc.ac.uk
> UK federation helpdesk: service at ukfederation.org.uk
> 
> Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
> 
> Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.  
> _______________________________________________
> Idpy-discuss mailing list
> Idpy-discuss at lists.sunet.se
> https://lists.sunet.se/listinfo/idpy-discuss
> 



More information about the Idpy-discuss mailing list