Hi Leif,
I added 2 pipes to buildin.py:
- publish_html creates static HTML views of IDPs and SPs, using XSLT based on Peter Schober’s alternative to MET;
- publish_split: similar to store, but added validUntil and creates signed XML-file per EntityDescriptor. This can be consumed dynamically by ADFS in an IDP role.
I put it directly into buildin.py because it shares some code with the sign pipe. Is this viable from your PoV - if yes, I would make an PR.
Cheers, Rainer
Hi all,
being part of Commons Conservancy brought up yet another subject,
which is whether we should add a header with license information in
every file in the projects under idpy. This is not something done in
an abstract way, there is a specific format modelling this information
(see https://spdx.org/ and https://reuse.software/ - more specifically
https://reuse.software/practices/2.0/) Still, I find it problematic.
We want to open up the question to the wider community and consider
their thoughts on this. The forwarded message below is discussing this
subject. You can see the question we posed, the answer we got and my
comments. Feel free to tell us what you think on this.
---------- Forwarded message ---------
Date: Thu, 16 May 2019 at 09:56
> ---------- Forwarded message ----------
> Date: May 8, 2019, 8:15 AM -0700
>
> > Why does CC think having a single license file per project is
> > insufficient? Our thought is that if we can avoid adding a header to
> > every single file, that would be nice, esp. given we already have this
> > info in the license file and we have the Note Well.
>
>
> this is not just our opinion, but something that is an industry and
> community standard for legal compliance these days. When companies like
> Siemens, Samsung or Honeywell use some code in one of the hundreds or
> thousands of devices and systems in their product line, they need to be
> able to provide the correct license and a download of the exact version.
> This means machine readability too.
>
I've actually observed the opposite of that. Communities abandon the
"license in every file" model, and just use a single LICENSE file in
the root of the project. The LICENSE file contains license
information, that is, it is not a single license but it has exception
sections and so on.
> To quote from https://reuse.software/practices/2.0/ :
>
> Scroll to the section "2. Include a copyright notice and license in each
> file"...
>
> "Source code files are often reused across multiple projects, taken from
> their origin and repurposed, or otherwise end up in repositories where
> they are separate from its origin. You should therefore ensure that all
> files in your project have a comment header that convey that file’s
> copyright and license information: Who are the copyright holders and
> under which license(s) do they release the file?
>
Continuing from above, the standardization of package-management
formats and tools has helped exactly with that: to avoid distribution
of single files, and instead provide packages and modules. It is bad
practice and considered a hack to copy files. Nobody liked that model
and everyone is moving away; it is unstructured, it becomes
unmanageable and it will cause problems.
> It is highly recommended that you keep the format of these headers
> consistent across your files. It is important, however, that you do not
> remove any information from headers in files of which you are not the
> sole author.
>
> You must convey the license information of your source code file in a
> standardised way, so that computers can interpret it. You can do this
> with an SPDX-License-Identifier tag followed by an SPDX expression
> defined by the SPDX specifications."
>
> (the text goes on for a while after this, to clarify the point but this
> is the basic gist of it)
>
> There is a nice Python tool to check:
>
> https://github.com/fsfe/reuse-tool
>
> I hope this makes sense
>
Well, it does not make complete sense. We're talking about licensing a
project. A project is not just code; there are data files (html, xml,
yaml, json files), binary files (archives/zip, images, audio, video,
etc), text files (configs, ini-files, etc) all "not-code". How do you
mark those files? Does the LICENSE file need a license-header? The
json format does not define comments, how do you add a header there?
If a binary file does not get a license header, why should a file with
code get one?
I would expect there to be a way to have the needed information
unified. If the files themselves cannot provide this information it
has to be external; thus the LICENSE file. If someone is worried about
somebody else re-using single files that do not have license
information (a python file, a png image, etc) there is really nothing
you can do (the DRM industry has been trying to solve for a long time;
and still your best bet is "social DRM").
Since, we're developing on open source with a permissive license, even
if someone does that, should we be happy that someone is actually
using what we built or sad that the files they copied did not have a
license header? And if they include the license information of that
copied file in their project's LICENSE file, is this solved?
Having pointed these contradictions, I am thinking that the "license
in every file" model seems to be a step backwards. It is introducing
overhead and does not really solve the problem, while at the same time
it enables a culture of bad practice (copying files around).
Cheers,
--
Ivan c00kiemon5ter Kanakarakis >:3
*Idpy meeting 23 January 2024*
Attendees: Johan W, Shayna, Ivan, Matthew E.
0 - Agenda bash
1 - Project review
a. General -
We would like to get Roland to do a demo on his wallet system. There is
great interest about the trust within the system, the issuers, the
credentials, etc. Shayna will message him about scheduling it.
b. OIDC libraries - https://github.com/IdentityPython (idpy-oidc,
JWTConnect-Python-CryptoJWT, etc)
- Focus is currently on projects around oidc that implement the wallets.
- version 3.0.0 of idpy-oidc was released 17 December.
- side news - Christos is creating a repo to host GÉANT Core AAI
frontend, which is based on idpy-oidc. The main component is
SATOSA - this
will happen soon.
c. Satosa - https://github.com/IdentityPython/SATOSA
- Repo needs to be cleaned up. Some things have been merged but no
release. After cleanup, Ivan needs to add dependencies for oidc
backend as
an extra requirement (optional dependency).
- Other PRs that are pending to go in:
- Roland's work on removing SAML parts -
https://github.com/IdentityPython/SATOSA/pull/442 -
- Pavel's PR - https://github.com/IdentityPython/SATOSA/pull/419
- Johan's PRs - https://github.com/IdentityPython/SATOSA/pull/449
and https://github.com/IdentityPython/SATOSA/pull/450
- Ivan is looking at these Metadata PRs -
https://github.com/IdentityPython/SATOSA/pull/438 and
https://github.com/IdentityPython/SATOSA/pull/448 - Ivan has some
issues with these - there is a better way, using this (metaform) instead
as a base : https://github.com/SUNET/swamid-satosa/tree/main/src
- 438 is getting info from oidc services to be accessed by user -
denotes different metadata in terms of language in a specific
way, which is
also something that the oidc standard specifies. It's a nice
notation that
gets us away from having to nest things based on a language signifier.
Instead, this uses display name of a service with a language
tag - default
(no language tag) is english. Only does this for some
information, but more
would be useful and there will be some work to accommodate that.
- 448 - work is from Vladimir in NZ- mixing attibutes for person
with the metadata, because there is no other place for this
information. A
new slot should be dedicated for this information. Also there is some
information such as display name, scopes, etc., that is
implied to be about
the IdP, but it could be about the SP - there is no
indication where it
came from. The idea is to have a dedicated slot - not mix
things with the
attributes of the subject, and then under its own key also
have info for
the service AND for IdP. Then everything is separated and
made clear. And
then we can denote the languages with the notation from 438
(inherited from
oidc spec). Can be based on metainfo (see above) which has
collators that
work with both SAML and OIDC. For OIDC it is implied there is a data
structure - someone who has statically registered the client
would need to
add this data - it is not part of the official spec.
- Discussion of using Scott K's attribute store plugin response
microservice for IdP metadata attributes. It pulls from
metadata store in
the context. Be careful if you redirect - the object with the
information
will go away. Part of the reason Ivan would like proper
server side storage
for the proxy is that some federations may have many scopes
for their IdP
which creates a huge payload, and if you grab the metadata
from this object
and try to put it in a cookie, it is too large.
- Ivan would like to make this part of SATOSA, not just a
microservice, so not just attributes of user are available,
but also of the
service and the issuer. This will require proper APIs for
both the backends
and frontends, to get the data in a specific, defined format.
d. pySAML2 - https://github.com/IdentityPython/pysaml2
- Some updates need to be done for dependencies - there will be a new
release to accommodate these things
- cryptography has a new version
- cryptojwt
- xmlschema - new version/release is breaking APIs that are
currently being used.
- Fixing datetime() calls for python 3.12 as they are deprecated
- Get back to configurable encrpytion algorithms / signing
e. Any other project (pyFF, djangosaml2, pyMDOC-CBOR, etc)
- djangosaml2 - possibly looking to add new logging calls to do
tracebacks?
- thiss - working on container to get readymade setup
- Matthew E. is also trying to get an all-inclusive seamless access
deployment working, and once they do they will contribute their
containers/changes back to the community.
- His goal is to create a docker compose file that others can use,
using as a default eduGain metadata, to get their own seamless access
working. Then put pyff, thiss-mdq and thiss-js in the docker official
images library to get regular rebuilds for security updates
and follow good
current practices for containers.
- Johan may be able to share something with Matthew from his own
deployment, for smaller federations. They restart pyff
service every 15
minutes to reload metadata.
- pyff can be used two different ways: as a service, or as a
process that runs once and processes metadata, spitting out
xml files for
you, along with a json feed which can be used by
thiss-mdq/thiss-js. There
may still be a memory leak in the service variant?
- SWAMID is wanting to get a new release of pyff - there are some
small commits to HEAD that they need.
2 - AOB
Attendees: Johan Wassberg, Shayna, Ivan, Roland, Matthew E., Hannah
0 - Agenda bash
1 - Project review
a. General - Welcome to Johan Wassberg - working with SWAMID
b. OIDC libraries - https://github.com/IdentityPython (idpy-oidc,
JWTConnect-Python-CryptoJWT, etc)
- Roland has been working on a front end to SATOSA to be a credential
issuer for a digital wallet system - it is built and running!
- It has a number of endpoints - an authentication endpoint which can be
mapped to a number of backends, but also a credential
endpoint - to receive
a credential. Initially this used a SATOSA backend connected
to credential
constructor. This could be considered "twisting the arm" of
SATOSA , so
instead there is now an API with a built-in credential
constructor. Using
the API you could also call a service outside SATOSA for an
authenticated
user with an id token. The credential issuer uses a trust
pattern , which
is the OpenID federation. Roland says he should probably make
a OP frontend
without the credential issuer stuff in it , but with the
federation stuff
in it. If you want to test, you need a number of entities : SATOSA, a
wallet, and a federation with at least a trust anchor, wallet
provider and
a PID issuer (or some other credential issuer).
- Roland is writing documentation from the digital wallet point of view.
He wants to write it so that different audiences have their
own targeted
documentation; for example one section for IT architects or
CIOs, another
for deployers, another for developers who wish to extend the
capabilities.
- Discussion on how to register an OIDC RP with satosa-oidcop
- using *https://openidconnect.net/* <https://openidconnect.net/> (test
RP) and entering
*https://federation.scienceforum.sc/.well-known/openid-configuration*
<https://federation.scienceforum.sc/.well-known/openid-configuration>
for configuration, Matthew E receives "response type is not
registered"
errors.
- client needs to define property which indicates what response types
are allowed - by default it seems nothing is set. Matthew E
also reported
looking in database and saw no clients. May need to
statically register a
client - add an entry with Client id, client secret, client uri, and
probably some structure that defines response types that
client should be
using. May need to get Giuseppe involved. There doesn't seem to be an
example with the repo, and Matthew has looked through the
code and can't
seem to get it working correctly.
- Roland says there are a number of classes you can use to implement the
client database - it behaves like a dictionary. Roland could possibly
provide a configuration file that provides hints how to implement. For
instance, with the credential issuer frontend he didn't want to use
mongoDb, so he used a simple persistent layer with a
key-value database
behind it - implemented where the file name is the attribute
name, values
are json within the file. He can provide an example of this.
It would be
easy to add a client in this case - just drop a new file in a specific
directory, with a name that follows a convention and the
content as json
with the appropriate attributes (client id, client secret,
redirect uri)
c. Satosa - https://github.com/IdentityPython/SATOSA
- Many things are pending/need review and Ivan will look at them soon.
- New PRs from Johan:
- https://github.com/IdentityPython/SATOSA/pull/449 - current
implementation only generates parts of the metadata - doesn't
include all
backends/frontends
- https://github.com/IdentityPython/SATOSA/pull/450 - want to be able to
set a preferred backend when generating metadata where
multiple backends
are available - problematic because routing is based on which
endpoints the
request comes through. In other words: One of the proxies SUNET is
operating is set up such that there are multiple back ends
and these are
combined into one front end. There are multiple entity
descriptors with no
root. Also, the endpoints within the descriptors contain all
the backends.
It would be better to be able to select which endpoints of
which back end
are included in the metadata of the front end - a new option for a
preferred backend. Changes current behavior of the proxy, but
this problem
needs to be addressed. Ivan needs to experiment and recreate
this SUNET
environment. This opens up a new concept for SATOSA -
multi-tenancy. You
would be able to specify which back ends can talk to which
front ends -
multiple groups - where policies are applied. This is a
complex concept to
implement.
- https://github.com/IdentityPython/SATOSA/pull/451 - extend SATOSA
support paths beyond the root of the domain - Kristof created a new PR
after closing old one, with a clearer path on how to move forward
- https://github.com/IdentityPython/SATOSA/pull/438 - enriching data
structure of SATOSA with metadata information about oidc services. Adds
some things using conventions from oidc, but it is missing some useful
information that should be there - it's not taking into account how those
things are named/treated in the SAML world. You need to know
first whether
this is a SAML service or OIDC service to determine how to represent
things. Ivan will write a review. Hoping to get Pavel to do some of the
work.
- https://github.com/IdentityPython/SATOSA/pull/419 - involves
IsPassive/ForceAuthn in SAML, prompt in OIDC. Prompt is usually mandatory
when offline access is requested. Pavel wants to proxy this
concept through
ForceAuthn and IsPassive - if set, then prompt is skipped; or
reverse - if
prompt is no, IsPassive is set in SAML request to IdP. And when
ForceAuthn
is there - Internally store the association , then set with
ForceAuthn. The
work to make it configurable is done. Needs review to make sure it done
correctly. This is important for legal reasons - prompt is used
as consent.
Want to make sure that the new capability is available, but that the
defaults don't interfere with what is already in existence.
- https://github.com/IdentityPython/SATOSA/pull/452 - moving away from
SHA1 , removed from xmlsec. Needs to be removed from places it is used in
SATOSA as default. The PR allows new parameters to specify which
algorithms
to use. Needs some review and guidance, but should go in soon. Ivan also
needs to see what xmlsec is doing.
d. pySAML2 - https://github.com/IdentityPython/pysaml2
- Not much activity, but there are two issues that are annoying.
- https://github.com/IdentityPython/pysaml2/issues/944 - problems with
Windows temporary files . There is a template that acts as a
configuration
as to how a request or response should be signed. A temporary file is
written to (the temp file is an actual class within Python
ecosystem / core
Python libraries). Then xmlsec reads from this file to know
what to do.
Once a temporary file (that was created through the core
Python library) is
opened, Windows won't let other processes open the file and
read it. Python
should be addressing this rather than the project, but it's
not happening.
Ivan has a workaround that was never pushed - taking
advantage of Python
garbagecollector. Ivan will push and have the user test.
- https://github.com/IdentityPython/pysaml2/issues/946 - race condition.
Pysaml2 keeps state internally - in either an in-memory
dictionary or a
shelve file (Python-related) - a binary file that can be shared across
processes. User is using the shelve module for state in a
multi-processing
service. Processes are trying to access the file at the same time. One
process errors out and then the shelve module is corrupted.
Solution would
be to expose the state and manage it through a database or
some other way.
An interface is needed for this state, with proper adapters
to be able to
write or read from the storage back ends. Then using the
shelve module or
the in-memory dictionary (mySQL, Rides. etc) is a
configurable option. This
PR needs work to build that interface, carefully. Ivan will write a
response to this, but there are other priorities right now.
- There are some other pending PRs.
- There are some commits on the master branch and they will be creating
a new release.
e. Any other project (pyFF, djangosaml2, pyMDOC-CBOR, etc)
- Ivan knows there was discussion on djangosaml2 - adding support for
the content security policy headers. Not sure if there was a new release.
- pyff - there was a patch release previously.
- no activity on pyMDOC-CBOR - not clear if CBOR will be used as one of
the formats.
2 - AOB
- Matthew E has been using github actions -for a personal project, he
has a working github actions workflow that does linting and formatting
checks (purely style and syntax), testing (using pytest), a semantic
release build, and PyPI /github packages publishing.
- not sure how to do branch protection rules - Matthew will get this
figured out
- He will come back with some proposed configs ( aiming for 2 weeks)
- can use github oidc connector to authorize PyPI to get credentials
- doesn't need PyPI login. Ivan would just need to authorize the git
repositories, github action workflows and environment within the repo
- idpy not using travis CI anymore - it wasn't working
- Matthew is willing to offer his work in this area - can set up
something using TestPyPI. These are his pre-commit and ci yml files.
- https://github.com/irtnog/lethbridge/blob/main/.pre-commit-config.yaml
-
https://github.com/irtnog/lethbridge/blob/main/.github/workflows/ci.yml
- pysaml2 - Ivan says there are 2 PRs to introduce using githubactions
and pre-commit configuration. He has not had time to look into them. To run
tests on pysaml2, run "poetry pytest" and they also use tox.
- https://github.com/IdentityPython/pysaml2/pull/816
- https://github.com/IdentityPython/pysaml2/pull/882
- SATOSA - Ivan thinks we should do the same here. Decide on what the
configuration should look like. SATOSA also uses pytest with the options
you want. Has not been converted to use poetry. SATOSA would need more work
- linting etc has not been applied.
- pysaml2 had all this stuff incorporated in one big commit.
Sorry, I left the wrong date at the top of the agenda for the idpy
developers meeting. It should say "Tuesday, 9 January 2024, 14:00 UTC"
Thank you,
Shayna Atkinson
SCG Consulting Group
