8:59 a.m.
Hey,
Sorry for the crosspost...
After a few weeks of spending all of my available development bits on
the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
all nodejs!) I'm back to working on pyFF for a bit.
Here is what I have planned for in the quite near term:
1. merge the api-refactory branch which includes a pyramids-based API
2. merge documentation PR from Hannah Sebuliba (thx!)
3. tag and release the last monolothic version of pyFF
4. in HEAD which becomes the new 1.0.0 release:
- remove all frontend bits (old discovery, management web app)
- pyffd will now start pyramids-based API server
- wsgi will be available/recommended
- create a new "frontend app" as a separate webpack+nodejs project
- create docker-compose.yaml that starts pyffd (API) + frontend app
5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
After 4 it makes sense to talk about things like...
- new redis/#nosql backends
- work on reducing memory footprint
- pubsub for notifications between MDQ servers
- more instrumentation & monitoring
- adaptive aggregation for large-scale deployments
- elastic search
- management APIs for integrated editing of local metadata
- OIDC
- generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
Thoughts etc are as usual welcome.
Cheers Leif
12:52 p.m.
Hi,
Thanks for the note/update Leif.
Could this note, or some form of this note, become a roadmap document
that you keep and maintain somewhere in
https://github.com/IdentityPython/pyFF
?
I confess that I thought I understood where you were going with breaking
pyFF into backend and frontend pieces, but when I look at
https://github.com/TheIdentitySelector
I am confused about just what I would *do* to build a discovery service
that can leverage a pyFF backend server.
I think it would be helpful (for me at least) to have some higher level
documentation that showed the architecture, the pieces, and how you
expect them to fit together.
In short, I am trying to understand if I see a path for a small science
team with limited resources to use pyFF + (something) to get what they
get right now with pyFF, or will this really only be something
accessible to large teams going forward.
I am happy to create the architecture diagram(s) and associated
documentation with your help if you can dialogue with me. Please let me
know if you want me to contribute in that way.
Thanks,
Scott K
Hey,
Sorry for the crosspost...
After a few weeks of spending all of my available development bits on
the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
all nodejs!) I'm back to working on pyFF for a bit.
Here is what I have planned for in the quite near term:
1. merge the api-refactory branch which includes a pyramids-based API
2. merge documentation PR from Hannah Sebuliba (thx!)
3. tag and release the last monolothic version of pyFF
4. in HEAD which becomes the new 1.0.0 release:
- remove all frontend bits (old discovery, management web app)
- pyffd will now start pyramids-based API server
- wsgi will be available/recommended
- create a new "frontend app" as a separate webpack+nodejs project
- create docker-compose.yaml that starts pyffd (API) + frontend app
5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
After 4 it makes sense to talk about things like...
- new redis/#nosql backends
- work on reducing memory footprint
- pubsub for notifications between MDQ servers
- more instrumentation & monitoring
- adaptive aggregation for large-scale deployments
- elastic search
- management APIs for integrated editing of local metadata
- OIDC
- generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
Thoughts etc are as usual welcome.
Cheers Leif
2:42 p.m.
On 2019-04-18 12:52, Scott Koranda wrote:
Hi,
Thanks for the note/update Leif.
Could this note, or some form of this note, become a roadmap document
that you keep and maintain somewhere in
https://github.com/IdentityPython/pyFF
?
Yeah why not!
I confess that I thought I understood where you were going with breaking
pyFF into backend and frontend pieces, but when I look at
https://github.com/TheIdentitySelector
I am confused about just what I would *do* to build a discovery service
that can leverage a pyFF backend server.
Thats a great question. There are two options:
1. you deploy thiss-js on your own domain and use your pyFF instance
as backend MDQ. thiss-js includes a DS, a login button component and
a persistence service.
2. you deploy an html page that runs thiss-ds-js (the client parts
of thiss-js) but uses an existing deployment of thiss-js as persistence
service (eg the one on thiss.io or the future "ra21" one which will
probably live on seamlessaccess.org somwehre). You still use your
pyFF instance as backend MDQ in this case.
I plan to provide a simple docker container running (2) that can be
deployed along side a pyFF instance.
I think it would be helpful (for me at least) to have some higher level
documentation that showed the architecture, the pieces, and how you
expect them to fit together.
In short, I am trying to understand if I see a path for a small science
team with limited resources to use pyFF + (something) to get what they
get right now with pyFF, or will this really only be something
accessible to large teams going forward.
Yep, understood. The small team deployment pattern is important.
My idea is that the only difference between now and "1.0.0" is that
you'll have to spin up two or maybe three docker containers (if you
want both DS and mgmt app) to get what you today get in a single
container. Its more work but with docker compose its not that much
in practice and the result is much better separation of concerns.
I am happy to create the architecture diagram(s) and associated
documentation with your help if you can dialogue with me. Please let me
know if you want me to contribute in that way.
Sure!
Cheers Leif
Thanks,
Scott K
>
> Hey,
>
> Sorry for the crosspost...
>
> After a few weeks of spending all of my available development bits on
> the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
> all nodejs!) I'm back to working on pyFF for a bit.
>
> Here is what I have planned for in the quite near term:
>
> 1. merge the api-refactory branch which includes a pyramids-based API
> 2. merge documentation PR from Hannah Sebuliba (thx!)
> 3. tag and release the last monolothic version of pyFF
> 4. in HEAD which becomes the new 1.0.0 release:
> - remove all frontend bits (old discovery, management web app)
> - pyffd will now start pyramids-based API server
> - wsgi will be available/recommended
> - create a new "frontend app" as a separate webpack+nodejs project
> - create docker-compose.yaml that starts pyffd (API) + frontend app
> 5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
>
> After 4 it makes sense to talk about things like...
>
> - new redis/#nosql backends
> - work on reducing memory footprint
> - pubsub for notifications between MDQ servers
> - more instrumentation & monitoring
> - adaptive aggregation for large-scale deployments
> - elastic search
> - management APIs for integrated editing of local metadata
> - OIDC
> - generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
>
> Thoughts etc are as usual welcome.
>
> Cheers Leif
2:13 p.m.
Hi,
Just like Scott asked a while ago I'm struggling to understand where pyFF
stands at the moment and getting it to work "the right/new way".
My main incentive is to depart our current pyff deployment from the python2
dependancy, ultimate goal would be to have a future-proof pyff mdq + thiss DS.
We want to run the pyff service as a daemon and have thiss served by our own
nginx server. Shouldn't be too hard I guess?
Anyway, I moved our existing pyff deploy to the side and started by cloning
pyff from github, checking out master. Then I created a python3 virtualenv in
this directory and pip install'ed -e . (dot). This installs pyff 1.1.1 fine,
at least so it seems.
The bare pyff commandline, with edugain example pipeline runs fine, I see the
same output, but then things get weird. The gunicorn example does nothing but
start a process that listens at the bind parameters, but does nothing:
# gunicorn -b127.0.0.1:8083 -e PYFF_PIPELINE=/opt/pyFF/edugain.yaml
pyff.wsgi:app
[2019-08-26 10:57:32 +0000] [1744] [INFO] Starting gunicorn 19.9.0
[2019-08-26 10:57:32 +0000] [1744] [INFO] Listening at:
https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDg…
(1744)
[2019-08-26 10:57:32 +0000] [1744] [INFO] Using worker: sync
[2019-08-26 10:57:32 +0000] [1747] [INFO] Booting worker with pid: 1747
When I call the api/call/update hook using POST I get:
<html>
<head>
<title>404 Not Found</title>
</head>
<body>
<h1>404 Not Found</h1>
The resource could not be found.<br/><br/>
</body>
</html>
And no output in the gunicorn window.
When I specify -e PYFF_LOGLEVEL=DEBUG on the gunicorn commandline, nothing
more is shown than whats copied above, only INFO logging. Even specifying -e
PYFF_LOGLEVEL=FOOBAR shows nothing, which should at least trigger an unkown
loglevel exception as far as I understand the code? It looks like gunicorn
silently fails or refuses to boot the pyff wsgi app?
It turns out, pyffd still exists, so when I start pyffd just like we did in
the old days, things look better. And even our beloved mdq/ds interface still
available? Is this intended behaviour?
So, long story short: our python2/3 migration scenario seems secured as I
couldn't find any breaking issue using the latest and greatest pyff on python3
(thanks for that).
But: how about moving forward? What's up with the gunicorn output I see? How
to debug further so we can start testing pyff + thiss and raise issues, if
any?
@Scott: have you made any progress that we can make use of since last time you
asked?
Best regards,
Martin
On Thursday, April 18, 2019 8:59:04 AM CEST Leif Johansson wrote:
Hey,
Sorry for the crosspost...
After a few weeks of spending all of my available development bits on
the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
all nodejs!) I'm back to working on pyFF for a bit.
Here is what I have planned for in the quite near term:
1. merge the api-refactory branch which includes a pyramids-based API
2. merge documentation PR from Hannah Sebuliba (thx!)
3. tag and release the last monolothic version of pyFF
4. in HEAD which becomes the new 1.0.0 release:
- remove all frontend bits (old discovery, management web app)
- pyffd will now start pyramids-based API server
- wsgi will be available/recommended
- create a new "frontend app" as a separate webpack+nodejs project
- create docker-compose.yaml that starts pyffd (API) + frontend app
5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
After 4 it makes sense to talk about things like...
- new redis/#nosql backends
- work on reducing memory footprint
- pubsub for notifications between MDQ servers
- more instrumentation & monitoring
- adaptive aggregation for large-scale deployments
- elastic search
- management APIs for integrated editing of local metadata
- OIDC
- generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
Thoughts etc are as usual welcome.
Cheers Leif
_______________________________________________
Idpy-discuss mailing list
Idpy-discuss at lists.sunet.se
https://lists.sunet.se/listinfo/idpy-discuss
2:39 p.m.
Hi Martin,
Here is what I am doing right now with pyFF (though I just found a
somewhat critical issue that I am going to report on/ask for help soon):
(Apologies for the shorthand...)
docker pull redis:5.0.5
mkdir -p /tmp/redis-data
docker run \
-d \
--name redis \
-v /tmp/redis-data:/data \
-p 6379:6379 \
redis:5.0.5
mkdir /opt/pyff
pushd /opt/pyff
virtualenv --python=python3.5 .
/opt/pyff/bin/activate
pip install --upgrade pip
pip install git+https://github.com/IdentityPython/pyFF.git at
f74a2278be7a806fe4c0aae8e51c8e4f8708f26d
gunicorn \
--preload \
--log-config logger.ini \
--bind 0.0.0.0:8080 \
-t 600 \
-e PYFF_PIPELINE=test_mdx.yaml \
-e PYFF_UPDATE_FREQUENCY=3600 \
-e
PYFF_PUBLIC_URL=https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDgw&_s=ZGVmYXVsdA%3D%3D&_c=01104970&_r=c3VuZXQtc2U%3D
\
-e PYFF_STORE_CLASS=pyff.store:RedisWhooshStore \
--threads 4 \
--worker-tmp-dir=/dev/shm \
--worker-class=gthread \
pyff.wsgi:app
With this I can then do
curl
https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDg…
| xmllint --format -
to test MDQ.
To test the search API I can do
curl -H 'Accept: application/json'
'https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDgwL2FwaS9zZWFyY2g%2FcXVlcnk9d2lzY29uc2lu&_s=ZGVmYXVsdA%3D%3D&_c=92701b8b&_r=c3VuZXQtc2U%3D'
| python -m json.tool
The search API though does not run through a pyFF pipeline. If you want
that you need to do
curl -H 'Accept: application/json'
'https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDgwL2VudGl0aWVzLz9xdWVyeT13aXNjb25zaW4%3D&_s=ZGVmYXVsdA%3D%3D&_c=c3f4f7ba&_r=c3VuZXQtc2U%3D'
| python -m json.tool
If your plumbing includes something like this:
- select as special_discovery:
- some IdP entityID
- some other IdP entityID
- or some other selection pipe
then you can query like
curl -H 'Accept: application/json'
'https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDgwL2VudGl0aWVzL3NwZWNpYWxfZGlzY292ZXJ5P3F1ZXJ5PXdpc2NvbnNpbg%3D%3D&_s=ZGVmYXVsdA%3D%3D&_c=a9cfb8b0&_r=c3VuZXQtc2U%3D'
| python -m json.tool
and the results will be restricted to those IdPs you have included
for the "special" discovery.
Notes:
- You want to use the latest commit because of some issues found in the 1.1.1 release
that were later fixed.
- I am using the RedishWhooshStore backend not so much for Redis (though
that is nice) but for Whoosh. The indexing by Whoosh is better and provides
a better search experience for your users.
- I think I just found an issue, however, with how the Whoosh indexing
is generated (or not). I will be sending an email on it. Don't be surprised
if you find that the API search is not returning results if you follow
the recipe aboved. It was working for me previously, but I ran into trouble yesterday...
HTH,
Scott K
@Scott: have you made any progress that we can make
use of since last time you
asked?
3:56 p.m.
Hi Scott (and Giuseppe)
Here is what I am doing right now with pyFF (though I
just found a
somewhat critical issue that I am going to report on/ask for help soon):
Thx for the fast responses! I'm a quite bit further now!
I discovered three importand things:
1. PYFF_UPDATE_FREQUENCY defaults to 0 if not set. When set to anything other
than 0, it forces an internal request on PYFF_PUBLIC_URL/api/call/update to
bootstrap pipeline parsing. Without it, pyff won't give any decent answer.
2. Without PYFF_UPDATE_FREQUENCY one needs to call POST PYFF_PUBLIC_URL/api/
call/update oneself, which results in a "404 Not found". THIS IS A RED
HERRING, the update succeeded anyway (my oh my).
3. After 2, or when supplying PYFF_UPDATE_FREQUENCY other than 0, I can now
succesfully retreive correctly signed sane md for one of the loaded entityID's
in my test pipeline (hurray).
But: I still don't understand, when booted with gunicorn (is this the best way
forward?) pyFF logging is absent. I still only see the gunicorn boot sequence
and then silence, no matter what I set PYFF_LOGLEVEL to, unless pyFF generates
an exception, which are shown e.g. when accessing PYFF_PUBLIC_URL/api/call/
update at start fails. Where's my normal logs? Is your --log-config logger.ini
the secret to this Scott?
Best regards,
Martin
6:58 p.m.
On 2019-08-26 15:56, Martin van Es wrote:
Hi Scott (and Giuseppe)
The idea is that by default you use your own scheduler.
Here is what I am doing right now with pyFF
(though I just found a
somewhat critical issue that I am going to report on/ask for help soon):
Thx for the fast responses! I'm a quite bit further now!
I discovered three importand things:
1. PYFF_UPDATE_FREQUENCY defaults to 0 if not set. When set to anything other
than 0, it forces an internal request on PYFF_PUBLIC_URL/api/call/update to
bootstrap pipeline parsing. Without it, pyff won't give any decent answer.
2. Without PYFF_UPDATE_FREQUENCY one needs to call
POST PYFF_PUBLIC_URL/api/
call/update oneself, which results in a "404 Not found". THIS IS A RED
HERRING, the update succeeded anyway (my oh my).
Documentation is admittedly a bit behind... but you should get a 404
there if your pipeline is setup to have an 'update' entrypoint. What
does your pipeline look like?
3. After 2, or when supplying PYFF_UPDATE_FREQUENCY other than 0, I can now
succesfully retreive correctly signed sane md for one of the loaded entityID's
in my test pipeline (hurray).
But: I still don't understand, when booted with gunicorn (is this the best way
forward?) pyFF logging is absent. I still only see the gunicorn boot sequence
and then silence, no matter what I set PYFF_LOGLEVEL to, unless pyFF generates
an exception, which are shown e.g. when accessing PYFF_PUBLIC_URL/api/call/
update at start fails. Where's my normal logs? Is your --log-config logger.ini
the secret to this Scott?
Logging is a bit of a dark art in wsgi (at least to me) but if you set
PYFF_LOGGING to point to examples/debug.ini you'll see a lot more and
also hopefully understand what logging looks like.
Again docs is a bit behind...
Cheers Leif
Best regards,
Martin
_______________________________________________
Idpy-discuss mailing list
Idpy-discuss at lists.sunet.se
https://lists.sunet.se/listinfo/idpy-discuss
7:50 p.m.
Hi Leif,
On Monday, August 26, 2019 6:58:45 PM CEST Leif Johansson wrote:
It's the example pipeline from documentation, with the edugain md url replaced
by a local filesystem folder:
- when update:
- load:
- /opt/pyff.old/metadata
- when request:
- select:
- pipe:
- when accept application/xml:
- first
- finalize:
cacheDuration: PT12H
validUntil: P10D
- sign:
key: certs/signing.key
cert: certs/signing.crt
- emit application/xml
- break
- when accept application/json:
- discojson
- emit application/json
- break
2. Without
PYFF_UPDATE_FREQUENCY one needs to call POST
PYFF_PUBLIC_URL/api/ call/update oneself, which results in a "404 Not
found". THIS IS A RED HERRING, the update succeeded anyway (my oh my).
Documentation is admittedly a bit behind... but you should get a 404
there if your pipeline is setup to have an 'update' entrypoint. What
does your pipeline look like? Logging is a bit of a dark art in wsgi (at least to
me) but if you set
PYFF_LOGGING to point to examples/debug.ini you'll see a lot more and
also hopefully understand what logging looks like.
Scott's logging.ini helped a lot, I'm having loads of debug info now.
Which brings me to my next question: thiss.io
There are roughly three github repositories: thiss-js, thiss-ds-js and thiss-
jquery-plugin. It is my understanding thiss-ds-js is used when building one's
own DS. I'm simply looking for a replacement of the old pyff built-in DS
service. thiss-js seems to supply an index.html and /ds endpoint that look
promising, but when fed to a simplified nginx conf based on the one found in
the docker recipe I'm greeted with something that remotely looks like a broken
DS, with missing pictures and failing requests to URL's like
https://mdq.***.***/%3C%=%20require('../assets/RA21-Black.svg')%20%…
What am I missing here?
Also, I'm curious what mdq and ds endpoints I should configure in satosa to
make this work again once I get a decent DS interface up and running?
Best regards,
Martin
%20%25%3E){kind=link}
9:04 p.m.
Skickat från min iPhone
26 aug. 2019 kl. 19:50 skrev Martin van Es <martin
at surfnet.nl>:
Hi Leif,
On Monday, August 26, 2019 6:58:45 PM CEST Leif Johansson wrote:
It's the example pipeline from documentation, with the edugain md url replaced
by a local filesystem folder:
- when update:
- load:
- /opt/pyff.old/metadata
- when request:
- select:
- pipe:
- when accept application/xml:
- first
- finalize:
cacheDuration: PT12H
validUntil: P10D
- sign:
key: certs/signing.key
cert: certs/signing.crt
- emit application/xml
- break
- when accept application/json:
- discojson
- emit application/json
- break
Yes - look at thiss.io for links to docs!
2.
Without PYFF_UPDATE_FREQUENCY one needs to call POST
PYFF_PUBLIC_URL/api/ call/update oneself, which results in a "404 Not
found". THIS IS A RED HERRING, the update succeeded anyway (my oh my).
Documentation is admittedly a bit behind... but you should get a 404
there if your pipeline is setup to have an 'update' entrypoint. What
does your pipeline look like? Logging is a bit of a dark art in wsgi (at least
to me) but if you set
PYFF_LOGGING to point to examples/debug.ini you'll see a lot more and
also hopefully understand what logging looks like.
Scott's logging.ini helped a lot, I'm having loads of debug info now.
Which brings me to my next question: thiss.io
There are roughly three github repositories: thiss-js, thiss-ds-js and thiss-
jquery-plugin. It is my understanding thiss-ds-js is used when building one's
own DS. I'm simply looking for a replacement of the old pyff built-in DS
service. thiss-js seems to supply an - index.html and /ds endpoint that look
promising, but when fed to a simplified nginx conf based on the one found in
the docker recipe I'm greeted with something that remotely looks like a broken
DS, with missing pictures and failing requests to URL's like
https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9tZHE%3D&_s=b…
What am I missing here?
Some config probably!
Also, I'm curious what mdq and ds endpoints I
should configure in satosa to
make this work again once I get a decent DS interface up and running?
<base_url>/ds
Best regards,
Martin
10:44 p.m.
What am I missing here?
Also, I'm curious what mdq and ds endpoints I should configure in satosa to
make this work again once I get a decent DS interface up and running?
Best regards,
Martin
I suspect you may have missed setting environment variables pointing out
the mdq server and a few other things - look at the environment vars in
the Dockerfile. I just pushed a fix to make this even more clear so make
sure to do a git pull!
Cheers Leif
10:46 p.m.
What am I missing here?
One more thing - static one-page apps don't have configuration in the
normal sense since they are ... well ... static!
The way the docker image fixes this is by doing a envsubst in the html
and js based on those env variables.
If you are not using the docker image you need to do something similar.
The alternative would be for the js to load a config.json on each
request like mdq-browser does but that is too much of a network hit
for something like thiss-js where very ms counts.
Cheers Leif
8:32 a.m.
Hi Leif,
On Monday, August 26, 2019 10:46:56 PM CEST Leif Johansson wrote:
The way the docker image fixes this is by doing a
envsubst in the html
and js based on those env variables.
I'm not using the docker/Dockerfile as is, I use it as the recipe to create a
standalone installation of pyff + static thiss.io. I understand that the pyff
server is subsituted in the nginx file. As far as I understand thiss-js,
everything is served by the nginx server, so all code runs in the client
brower.
I found the env/lua substitution in the nginx conf file, but completely missed
the functionality of this line in start.sh:
envsubst '${BASE_URL} ${STORAGE_DOMAIN} ${MDQ_URL} ${SEARCH_URL}' < $f >
/usr/
share/nginx/html/$f
So, there's some substitution work keft to do now ;). Curious if this can be
simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making all
URL's relative when the thiss-js code resides in the root of the same server
as the pyff daemon and is served on the same URL?
Best regards,
Martin
8:55 a.m.
On 2019-08-27 08:32, Martin van Es wrote:
Hi Leif,
On Monday, August 26, 2019 10:46:56 PM CEST Leif Johansson wrote:
Yeah but the code includes static configuration.
The way the docker image fixes this is by doing a
envsubst in the html
and js based on those env variables.
I'm not using the docker/Dockerfile as is, I use it as the recipe to create a
standalone installation of pyff + static thiss.io. I understand that the pyff
server is subsituted in the nginx file. As far as I understand thiss-js,
everything is served by the nginx server, so all code runs in the client
brower.
I found the env/lua substitution in the nginx conf file, but completely missed
the functionality of this line in start.sh:
envsubst '${BASE_URL} ${STORAGE_DOMAIN} ${MDQ_URL} ${SEARCH_URL}' < $f >
/usr/
share/nginx/html/$f
So, there's some substitution work keft to do now ;). Curious if this can be
simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making all
URL's relative when the thiss-js code resides in the root of the same server
as the pyff daemon and is served on the same URL?
It probably can ... PR would be great!
MVH leifj
9:28 a.m.
Hi,
On Tuesday, August 27, 2019 8:55:14 AM CEST Leif Johansson wrote:
Ok I now understand what went wrong. I was mislead by the "Read the
documentation" link that points to the documentation on
https://thiss-ds.readthedocs.io/ which is a work-in-progress. Therefor I missed the /docs
directory and am now catching up on everything npm!
Have built the code using webpack, but not before I corrected two typos in the
src:
diff --git a/src/ds/index.html b/src/ds/index.html
index c03e4d1..1b40ee8 100644
--- a/src/ds/index.html
+++ b/src/ds/index.html
@@ -15,7 +15,7 @@
</h2>
</header>
<main class="content">
- <div class="container" id="dsclient"
+ <div class="container" id="dsclient">
<!-- choose -->
<div class="d-none" id="choose">
<h1>Choose Your Institution</h1>
diff --git a/src/ds/index.js b/src/ds/index.js
index e30e882..dbe961c 100644
--- a/src/ds/index.js
+++ b/src/ds/index.js
@@ -20,7 +20,7 @@ import '../assets/ds.css';
const Hogan = require("hogan.js");
-import '(a)theidentityselector/thiss-jquery-plugins/src/ds-widget.js'.js';
+import '(a)theidentityselector/thiss-jquery-plugin/src/ds-widget.js'.js';
const search = Hogan.compile(require('!raw-loader!./templates/search.html'));
const saved = Hogan.compile(require('!raw-loader!./templates/saved.html'));
Thx so far!
Best regards,
Martin
So,
there's some substitution work keft to do now ;). Curious if this can
be simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making
all URL's relative when the thiss-js code resides in the root of the same
server as the pyff daemon and is served on the same URL?
It probably can ... PR would be great!
9:39 a.m.
On 2019-08-27 09:28, Martin van Es wrote:
Hi,
On Tuesday, August 27, 2019 8:55:14 AM CEST Leif Johansson wrote:
Ok I now understand what went wrong. I was mislead by the "Read the
documentation" link that points to the documentation on
https://thiss-ds.readthedocs.io/ which is a work-in-progress. Therefor I missed the /docs
directory and am now catching up on everything npm!
you mean thiss-js.readthedocs.io right?
This one only talks about docker etc. For the guts readthedocs on
thiss-ds-js and thiss-jquery-plugin - all referenced from thiss.io
So,
there's some substitution work keft to do now ;). Curious if this can
be simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making
all URL's relative when the thiss-js code resides in the root of the same
server as the pyff daemon and is served on the same URL?
It probably can ... PR would be great!
Have built the code using webpack, but not before I corrected two typos in the
src:
Thx - pushed !
diff --git a/src/ds/index.html b/src/ds/index.html
index c03e4d1..1b40ee8 100644
--- a/src/ds/index.html
+++ b/src/ds/index.html
@@ -15,7 +15,7 @@
</h2>
</header>
<main class="content">
- <div class="container" id="dsclient"
+ <div class="container" id="dsclient">
<!-- choose -->
<div class="d-none" id="choose">
<h1>Choose Your Institution</h1>
diff --git a/src/ds/index.js b/src/ds/index.js
index e30e882..dbe961c 100644
--- a/src/ds/index.js
+++ b/src/ds/index.js
@@ -20,7 +20,7 @@ import '../assets/ds.css';
const Hogan = require("hogan.js");
-import '(a)theidentityselector/thiss-jquery-plugins/src/ds-widget.js'.js';
+import '(a)theidentityselector/thiss-jquery-plugin/src/ds-widget.js'.js';
const search = Hogan.compile(require('!raw-loader!./templates/search.html'));
const saved = Hogan.compile(require('!raw-loader!./templates/saved.html'));
Thx so far!
Best regards,
Martin
9:57 a.m.
On Tuesday, August 27, 2019 9:39:31 AM CEST Leif Johansson wrote:
No, there is a typo on the thiss-js github README.md that points to an empty
readthedocs page, hence my confusion.
I'm trying to get satosa to interact with the new pyFF, but pyFF now complains
on the /entities request satosa fires:
2019-08-27 07:48:23,560 ERROR [pyff.api:31][MainThread] "application/
samlmetadata+xml" is not a valid mime type
So either we need to fix satosa or have pyFF accept satosa's "weird" Accept
header?
2019-08-27 07:48:02,760 DEBUG [pyff.api:31][MainThread] GET /entities/
%7Bsha1%7D9cf868c63facb578874f88ec55eb097ee2bbb2bb HTTP/1.0
Accept: application/samlmetadata+xml
Accept-Encoding: gzip, deflate
Connection: close
Host: 127.0.0.1:8083
User-Agent: python-requests/2.20.1
I changed the pipeline to have a "when accept application/samlmetadata+xml"
line, but that doesn't seem to do the trick?
2019-08-27 07:48:23,560 DEBUG [pyff.api:31][MainThread] Traceback (most recent
call last):
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 152, in
process_handler
accepter = MediaAccept(accept)
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 54, in
__init__
self._type = AcceptableType(accept)
File "/opt/pyFF/lib/python3.7/site-packages/accept_types/accept_types.py",
line 20, in __init__
raise ValueError('"%s" is not a valid mime type' % mime_type)
ValueError: "application/samlmetadata+xml" is not a valid mime type
M.
Ok I now
understand what went wrong. I was mislead by the "Read the
documentation" link that points to the documentation on
https://thiss-ds.readthedocs.io/ which is a work-in-progress. Therefor I
missed the /docs directory and am now catching up on everything npm!
you mean thiss-js.readthedocs.io right?
10:10 a.m.
On 2019-08-27 09:57, Martin van Es wrote:
On Tuesday, August 27, 2019 9:39:31 AM CEST Leif
Johansson wrote:
No, there is a typo on the thiss-js github README.md that points to an empty
readthedocs page, hence my confusion.
found and fixed - thx!
Ok I now
understand what went wrong. I was mislead by the "Read the
documentation" link that points to the documentation on
https://thiss-ds.readthedocs.io/ which is a work-in-progress. Therefor I
missed the /docs directory and am now catching up on everything npm!
you mean thiss-js.readthedocs.io right?
I'm trying to get satosa to interact with the new pyFF, but pyFF now complains
on the /entities request satosa fires:
2019-08-27 07:48:23,560 ERROR [pyff.api:31][MainThread] "application/
samlmetadata+xml" is not a valid mime type
So either we need to fix satosa or have pyFF accept satosa's "weird" Accept
header?
pyFF is at fault - I'll fix.
2019-08-27 07:48:02,760 DEBUG [pyff.api:31][MainThread] GET /entities/
%7Bsha1%7D9cf868c63facb578874f88ec55eb097ee2bbb2bb HTTP/1.0
Accept: application/samlmetadata+xml
Accept-Encoding: gzip, deflate
Connection: close
Host: 127.0.0.1:8083
User-Agent: python-requests/2.20.1
I changed the pipeline to have a "when accept application/samlmetadata+xml"
line, but that doesn't seem to do the trick?
2019-08-27 07:48:23,560 DEBUG [pyff.api:31][MainThread] Traceback (most recent
call last):
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 152, in
process_handler
accepter = MediaAccept(accept)
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 54, in
__init__
self._type = AcceptableType(accept)
File "/opt/pyFF/lib/python3.7/site-packages/accept_types/accept_types.py",
line 20, in __init__
raise ValueError('"%s" is not a valid mime type' % mime_type)
ValueError: "application/samlmetadata+xml" is not a valid mime type
M.
10:16 a.m.
I changed the pipeline to have a "when accept application/samlmetadata+xml"
line, but that doesn't seem to do the trick?
The problem is most likely in https://pypi.org/project/accept-types/
Cheers Leif
10:49 a.m.
On Tuesday, August 27, 2019 10:16:07 AM CEST Leif Johansson wrote:
Yes, + is not allowed in
valid_mime_type = re.compile(r'^(\*|[a-zA-Z0-9._-]+)(/(\*|[+a-zA-Z0-9._-]+))?
$')
(Temporarily) fixed it by adding a + in the regex in the second characterset
match.
And now pyff complains about the request being HTTPNotAcceptable
2019-08-27 08:36:05,992 DEBUG [pyff.api:31][MainThread] Traceback (most recent
call last):
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 172, in
process_handler
r, t = _fmt(r, accepter)
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 75, in _fmt
raise exc.exception_response(406)
pyramid.httpexceptions.HTTPNotAcceptable
Which seems to have it's roots deep in the application/xml acceptable
headers... in _fmt()
On the other hand, I found that it's not satosa but pysaml2 requesting the
application/samlmetadata+xml Accept header. It's a nasty mess :(
Martin
I changed the
pipeline to have a "when accept
application/samlmetadata+xml"
line, but that doesn't seem to do the trick?
The problem is most likely in https://pypi.org/project/accept-types/
11:01 a.m.
On 2019-08-27 10:49, Martin van Es wrote:
On Tuesday, August 27, 2019 10:16:07 AM CEST Leif
Johansson wrote:
Yes, + is not allowed in
valid_mime_type = re.compile(r'^(\*|[a-zA-Z0-9._-]+)(/(\*|[+a-zA-Z0-9._-]+))?
$')
(Temporarily) fixed it by adding a + in the regex in the second characterset
match.
https://bitbucket.org/leifj/python-accept-types/src/master/ has
a similar fix
I changed
the pipeline to have a "when accept
application/samlmetadata+xml"
line, but that doesn't seem to do the trick?
The problem is most likely in https://pypi.org/project/accept-types/
And now pyff complains about the request being HTTPNotAcceptable
2019-08-27 08:36:05,992 DEBUG [pyff.api:31][MainThread] Traceback (most recent
call last):
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 172, in
process_handler
r, t = _fmt(r, accepter)
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 75, in _fmt
raise exc.exception_response(406)
pyramid.httpexceptions.HTTPNotAcceptable
OK I pushed a fix for that.
Which seems to have it's roots deep in the application/xml acceptable
headers... in _fmt()
On the other hand, I found that it's not satosa but pysaml2 requesting the
application/samlmetadata+xml Accept header. It's a nasty mess :(
Martin
11:15 a.m.
On Tuesday, August 27, 2019 11:01:41 AM CEST Leif Johansson wrote:
Well, you should at least try to run your code before you push, because you
missed a closing bracket and it's not helping. I tried that as well, so there
is a really messy interaction between pyff and accept-types going on...
Going to add some debug statements in _fmt to see what's happening.
And I would suggest not to return application/samlmetadata+xml for everyone,
just because pysaml2 asks for it? Unless I don't understand the implications
of the return value of _fmt correctly?
Martin
pyramid.httpexceptions.HTTPNotAcceptable
OK I pushed a fix for that.
11:19 a.m.
On 2019-08-27 11:15, Martin van Es wrote:
On Tuesday, August 27, 2019 11:01:41 AM CEST Leif
Johansson wrote:
Well, you should at least try to run your code before you push, because you
missed a closing bracket and it's not helping. I tried that as well, so there
is a really messy interaction between pyff and accept-types going on...
Going to add some debug statements in _fmt to see what's happening.
And I would suggest not to return application/samlmetadata+xml for everyone,
just because pysaml2 asks for it? Unless I don't understand the implications
of the return value of _fmt correctly?
Martin
So _fmt is just a default fallback that actually shouldn't get called -
if you run emit in your pipeline the mime-type should be set there.
I'll debug some too. Sorry about the confusion!
Cheers Leif
pyramid.httpexceptions.HTTPNotAcceptable
OK I pushed a fix for that.
11:49 a.m.
On Tuesday, August 27, 2019 11:19:25 AM CEST Leif Johansson wrote:
so there is a really messy interaction between pyff
and accept-types going
> on...
Found the problem.
accept-types does a sloppy substitution when pattern contains a + in
get_pattern. It only replaces a * with [a-zA-Z0-9_.$#!%^*-]+ but omits
replacing a + with \+, probabaly because the developer never thought about x/
Y+Z acceptable types.
^application/samlmetadata+xml$ will never match application/samlmetadata+xml
^application/samlmetadata\+xml$ will.
Martin
12:19 p.m.
On 2019-08-27 11:49, Martin van Es wrote:
On Tuesday, August 27, 2019 11:19:25 AM CEST Leif
Johansson wrote:
Did you see my PR against the main branch?
Cheers Leif
so there is a really messy interaction between
pyff and accept-types going
> on...
Found the problem.
accept-types does a sloppy substitution when pattern contains a + in
get_pattern. It only replaces a * with [a-zA-Z0-9_.$#!%^*-]+ but omits
replacing a + with \+, probabaly because the developer never thought about x/
Y+Z acceptable types.
^application/samlmetadata+xml$ will never match application/samlmetadata+xml
^application/samlmetadata\+xml$ will.
Martin
12:52 p.m.
On Tuesday, August 27, 2019 11:49:20 AM CEST Martin van Es wrote:
On Tuesday, August 27, 2019 11:19:25 AM CEST Leif
Johansson wrote:
Nearly there.
/ds endpoint renders a correct DS form and search returns a correct json
string containing the valid IdP's.
One of them has id {sha1}42295b198743c7d33973a004f5560e843eb49bb0 but the next
DS request on
/%7Bsha1%7D42295b198743c7d33973a004f5560e843eb49bb0.json fails with a 404 (The
resource could not be found) and so I'm not redirected to the IdP.
Is that request fired at the correct pyff endpoint? How can pyFF return an id,
but not have a json reply for the same id?
Martin
so there is a really messy interaction between
pyff and accept-types going
> on...
Found the problem.
1:59 p.m.
On Tuesday, August 27, 2019 12:52:32 PM CEST Martin van Es wrote:
/%7Bsha1%7D42295b198743c7d33973a004f5560e843eb49bb0.json fails with a 404
(The resource could not be found) and so I'm not redirected to the IdP.
Fixed by correctly setting MDQ url in webpack json configuration file.
Just finished a succesful complete satosa end-to-end test!
Thx!
Martin
10:50 a.m.
2019-08-27 07:48:23,560 DEBUG
[pyff.api:31][MainThread] Traceback (most recent
call last):
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 152, in
process_handler
accepter = MediaAccept(accept)
File "/opt/pyFF/lib/python3.7/site-packages/pyff/api.py", line 54, in
__init__
self._type = AcceptableType(accept)
File "/opt/pyFF/lib/python3.7/site-packages/accept_types/accept_types.py",
line 20, in __init__
raise ValueError('"%s" is not a valid mime type' % mime_type)
ValueError: "application/samlmetadata+xml" is not a valid mime type
I found it and send a PR to the maintainer. Hopefully this will be fixed
soon.
9:40 a.m.
BTW - are you aware there is a hackathon on thiss-* on the Monday before
the nordunet technical workshop? Same location.
2:30 p.m.
On Tuesday, August 27, 2019 8:32:52 AM CEST Martin van Es wrote:
Curious if this can be
simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making all
URL's relative when the thiss-js code resides in the root of the same
server as the pyff daemon and is served on the same URL?
Managed to webpack a working dist root using the following HOST agnostic
webpack config:
module.exports = merge(common, {
mode: 'production',
plugins: [
new webpack.EnvironmentPlugin({
BASE_URL: '/',
COMPONENT_URL: '/cta/',
MDQ_URL: "/entities/",
PERSISTENCE_URL: '/ps/',
SEARCH_URL: "api/search",
STORAGE_DOMAIN: "/",
LOGLEVEL: 'error',
DEFAULT_CONTEXT: "/"
})
]
});
I don't know if "/" is a valid STORAGE_DOMAIN or CONTEXT, but it seems to
work.
Grtz
Martin
2:34 p.m.
Skickat från min iPhone
27 aug. 2019 kl. 14:30 skrev Martin van Es <martin
at surfnet.nl>:
loopback?
I’m pretty sure it won’t work in prod bc of how cross-origin postMessage works
On Tuesday, August 27, 2019 8:32:52 AM CEST
Martin van Es wrote:
Curious if this can be
simplified for the HTTP_HOST=BASE_URL=STORAGE_DOMAIN=MDQ_URL by making all
URL's relative when the thiss-js code resides in the root of the same
server as the pyff daemon and is served on the same URL?
Managed to webpack a working dist root using the following HOST agnostic
webpack config:
module.exports = merge(common, {
mode: 'production',
plugins: [
new webpack.EnvironmentPlugin({
BASE_URL: '/',
COMPONENT_URL: '/cta/',
MDQ_URL: "/entities/",
PERSISTENCE_URL: '/ps/',
SEARCH_URL: "api/search",
STORAGE_DOMAIN: "/",
LOGLEVEL: 'error',
DEFAULT_CONTEXT: "/"
})
]
});
I don't know if "/" is a valid STORAGE_DOMAIN or CONTEXT, but it seems to
work.
Grtz
Martin
2:49 p.m.
Hi Martin and everybody,
my setup example is here:
https://github.com/peppelinux/Satosa-saml2saml/tree/master/pyFF_example/pro…
You will find also a base nginx setup.
In production I use pyff as a metadata downloader and validator, as MDQ I
use some nginx rewrite rules[1] on top of pyff's md folder, as DS I'm using
something coded by myself for specific needs.
I also staged pyff MDQ (it takes much time than httpd rewrite rules [1]...)
and DS but this latter with some html patches [2][3] but I think to
continue with my setup until I'll have some more time to get latests pyff
updates and then move to API/Thiss.io updates. I'll try to follow Leifj and
Scott suggestions and examples when available.
regards
[1] https://github.com/IdentityPython/pyFF/issues/172
[2] https://github.com/IdentityPython/pyFF/pull/166
[3] https://github.com/IdentityPython/pyFF/pull/165
Il giorno lun 26 ago 2019 alle ore 14:14 Martin van Es <martin at surfnet.nl>
ha scritto:
Hi,
Just like Scott asked a while ago I'm struggling to understand where pyFF
stands at the moment and getting it to work "the right/new way".
My main incentive is to depart our current pyff deployment from the
python2
dependancy, ultimate goal would be to have a future-proof pyff mdq + thiss
DS.
We want to run the pyff service as a daemon and have thiss served by our
own
nginx server. Shouldn't be too hard I guess?
Anyway, I moved our existing pyff deploy to the side and started by
cloning
pyff from github, checking out master. Then I created a python3 virtualenv
in
this directory and pip install'ed -e . (dot). This installs pyff 1.1.1
fine,
at least so it seems.
The bare pyff commandline, with edugain example pipeline runs fine, I see
the
same output, but then things get weird. The gunicorn example does nothing
but
start a process that listens at the bind parameters, but does nothing:
# gunicorn -b127.0.0.1:8083 -e PYFF_PIPELINE=/opt/pyFF/edugain.yaml
pyff.wsgi:app
[2019-08-26 10:57:32 +0000] [1744] [INFO] Starting gunicorn 19.9.0
[2019-08-26 10:57:32 +0000] [1744] [INFO] Listening at:
https://urlproxy.sunet.se/canit/urlproxy.php?_q=aHR0cDovLzEyNy4wLjAuMTo4MDg…
(1744)
[2019-08-26 10:57:32 +0000] [1744] [INFO] Using worker: sync
[2019-08-26 10:57:32 +0000] [1747] [INFO] Booting worker with pid: 1747
When I call the api/call/update hook using POST I get:
<html>
<head>
<title>404 Not Found</title>
</head>
<body>
<h1>404 Not Found</h1>
The resource could not be found.<br/><br/>
</body>
</html>
And no output in the gunicorn window.
When I specify -e PYFF_LOGLEVEL=DEBUG on the gunicorn commandline, nothing
more is shown than whats copied above, only INFO logging. Even specifying
-e
PYFF_LOGLEVEL=FOOBAR shows nothing, which should at least trigger an
unkown
loglevel exception as far as I understand the code? It looks like gunicorn
silently fails or refuses to boot the pyff wsgi app?
It turns out, pyffd still exists, so when I start pyffd just like we did
in
the old days, things look better. And even our beloved mdq/ds interface
still
available? Is this intended behaviour?
So, long story short: our python2/3 migration scenario seems secured as I
couldn't find any breaking issue using the latest and greatest pyff on
python3
(thanks for that).
But: how about moving forward? What's up with the gunicorn output I see?
How
to debug further so we can start testing pyff + thiss and raise issues, if
any?
@Scott: have you made any progress that we can make use of since last time
you
asked?
Best regards,
Martin
On Thursday, April 18, 2019 8:59:04 AM CEST Leif Johansson wrote:
--
____________________
Dott. Giuseppe De Marco
CENTRO ICT DI ATENEO
University of Calabria
87036 Rende (CS) - Italy
Phone: +39 0984 496961
e-mail: giuseppe.demarco at unical.it
Hey,
Sorry for the crosspost...
After a few weeks of spending all of my available development bits on
the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
all nodejs!) I'm back to working on pyFF for a bit.
Here is what I have planned for in the quite near term:
1. merge the api-refactory branch which includes a pyramids-based API
2. merge documentation PR from Hannah Sebuliba (thx!)
3. tag and release the last monolothic version of pyFF
4. in HEAD which becomes the new 1.0.0 release:
- remove all frontend bits (old discovery, management web app)
- pyffd will now start pyramids-based API server
- wsgi will be available/recommended
- create a new "frontend app" as a separate webpack+nodejs project
- create docker-compose.yaml that starts pyffd (API) + frontend app
5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
After 4 it makes sense to talk about things like...
- new redis/#nosql backends
- work on reducing memory footprint
- pubsub for notifications between MDQ servers
- more instrumentation & monitoring
- adaptive aggregation for large-scale deployments
- elastic search
- management APIs for integrated editing of local metadata
- OIDC
- generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
Thoughts etc are as usual welcome.
Cheers Leif
_______________________________________________
Idpy-discuss mailing list
Idpy-discuss at lists.sunet.se
https://lists.sunet.se/listinfo/idpy-discuss
_______________________________________________
Idpy-discuss mailing list
Idpy-discuss at lists.sunet.se
https://lists.sunet.se/listinfo/idpy-discuss
1697
days inactive
1828
days old
32 comments
5 participants
participants (5)
-
giuseppe.demarco＠unical.it -
leifj＠sunet.se -
martin＠surfnet.nl -
skoranda＠gmail.com -
skoranda＠sphericalcowgroup.com