sharing state between satosa microservices
by io.paraskev＠gmail.com
On 8/2/19 8:24 AM, Rainer Hoerbe wrote:> Hi John
>
>> Am 2019-08-01 um 18:52 schrieb John Paraskevopoulos
>> <io.paraskev at gmail.com>:
>>
>>
>> I'm not sure I understand the use case where a shared config is
>> needed.
>
> The ADFS-IDP Use Case is the requirement to redirect to some service
> to complete role selection or any other interaction that would
> change the set of attributes in the assertion. The solution is to
> replay the AuthnRequest to the IDP after the user selected the role.
>
> I implemented this by 2 cooperative microservices, one Request-MS
> saving the AuthnRequest and one Reponse-MS replaying it. There are a
> few of shared parameters fron the config, and you need to know the
> own backend entityID as well.
>
>From what I understand and looked briefly about the ADFS-IDP use case,
this sound more like a need for a custom saml2 backend/frontend than
microservices since we're talking about a need to establish a custom
re-routing before completing the assertion. It might be better since
we're talking about a specific use case that is the exception to the
classic workflow.
>>
>> I also believe that Singleton is an anti-pattern and should be
>> avoided unless absolutely needed (i.e mostly logging). Let's not
>> forget that this pattern was invented ~25 years ago where things
>> and resources were really different. In the case we're discussing
>> here, I think that the configuration is not a cross cutting
>> concern, and even if it is, we should try to fix this instead of
>> accepting it and trying to come up with a singleton. Making the
>> config globally available to everyone would hide the exact
>> dependencies of the components who call it (because the would
>> consume the whole config instead of some specific data) breaking
>> the separation of concerns concept. This would mean that we won't
>> be sure which config options we could change without breaking
>> things. Just because Django uses a single settings file doesn't
>> mean we should as well (firstly because Django is a different kind
>> of framework and secondly because Django doesn't mean perfect
>> design. An object should have all the data it needs, no more no
>> less and these should be clearly declared as dependency. This can
>> be achieved either as data parameter during initialization or
>> through a config Factory that will take care of the data processing
>> and return the needed information for each component (although this
>> in a way hides the config dependency it is still more preferable
>> than the singleton in my opinion).
>>
>> Currently from what I've seen and with my little experience on the
>> project's internals, there is some separation of concerns by having
>> different config files for different components. If we believe some
>> options from different components should be used by other
>> components as well we should first examine why this is happening
>> (maybe the use case scenario is wrong or maybe we need to redesign
>> things) and then see what needs fixing and how.
>
> These are valid arguments. But how would you avoid duplication of
> config elements? Adding some reference mechanism in yaml seems too
> complex to me. Explicitly passing additional config files where the
> ms is configured seem possible may be possible, but is extra
> overhead. It is considered quit pythonic to access variables in
> objects without strict scope definitions, just using the underscore
> prefix by convention. In that sense my opinion is that dependencies
> may be less explicit to trade off for convenience.
>
Although as I've said, I don't like the term "pythonic", it is not
considered pythonic to access variables/methods of objects that have an
underscore prepended. Even though Python doesn't have a strict enforcing
rule about private variables and methods, the current convention is to
use a single underscore (double is used for name mangling, and double
prepended and appended for system defined magic) to define class
internal stuff. Internal stuff mean that they are not part of the public
interface (that's why help(<class name>) doesn't show them) and by not
being part of the public interface means that you are not supposed to
rely on them (because they can freely change without notice unlike
public methods and variables). So it's not pythonic but even if it was,
it would be against the OO convention about public/private
methods/variables.
On 8/2/19 12:39 AM, Giuseppe De Marco wrote:
> Hi John, What you wrote make sense.
>
> What do you think about the possibility to move in
> satosa.satosa_config those lines of code [1]?
>
> This way we would import satosa_config where needed and nothing
> else.
This would probably create a racing condition type. During the
initialization, a specific config file is loaded and passed to the
creation of the plugins etc. If we did something like this, then if a
microservice decides to load on its own the config file, how would you
be sure that the config file hasn't changed? This would mean that in the
core app there would be static loading (once during initialization), but
for specific plugins/microservices, there would be dynamic loading when
they would run. There is more danger than value in having multiple
different instances of a config file. Plus it doesn't change the fact,
that we would still be getting more information in our objects than we
need (since we obviously don't need the whole config info)
>
> What do you think also about the shared session? During the meeting
> on july 9th Ivan proposed an abstract class to handle shared
> sessions, he proposed many usable storage backends, like cookies,
> RDBMS, nosql and possibly other.
I'm not sure about what kind of shared session (and for what) you are
talking. Unfortunately I couldn't join the meeting on July 9th so I
can't express any opinion about that.
>
>
> [1]
> https://github.com/IdentityPython/SATOSA/blob/master/src/satosa/wsgi.py#L12
>
>
>
>
>
>
>
> Il gio 1 ago 2019, 18:52 John Paraskevopoulos <io.paraskev at gmail.com>
> ha scritto:
>
>> Hello,
>>
>> My two cents regarding this discussion (I didn't have time to
>> check the discussion before the previous meeting so I thought I put
>> my two cents on email).
>>
>> I'm not sure I understand the use case where a shared config is
>> needed.
>>
>> I also believe that Singleton is an anti-pattern and should be
>> avoided unless absolutely needed (i.e mostly logging). Let's not
>> forget that this pattern was invented ~25 years ago where things
>> and resources were really different. In the case we're discussing
>> here, I think that the configuration is not a cross cutting
>> concern, and even if it is, we should try to fix this instead of
>> accepting it and trying to come up with a singleton. Making the
>> config globally available to everyone would hide the exact
>> dependencies of the components who call it (because the would
>> consume the whole config instead of some specific data) breaking
>> the separation of concerns concept. This would mean that we won't
>> be sure which config options we could change without breaking
>> things. Just because Django uses a single settings file doesn't
>> mean we should as well (firstly because Django is a different kind
>> of framework and secondly because Django doesn't mean perfect
>> design. An object should have all the data it needs, no more no
>> less and these should be clearly declared as dependency. This can
>> be achieved either as data parameter during initialization or
>> through a config Factory that will take care of the data processing
>> and return the needed information for each component (although this
>> in a way hides the config dependency it is still more preferable
>> than the singleton in my opinion).
>>
>> Currently from what I've seen and with my little experience on the
>> project's internals, there is some separation of concerns by having
>> different config files for different components. If we believe some
>> options from different components should be used by other
>> components as well we should first examine why this is happening
>> (maybe the use case scenario is wrong or maybe we need to redesign
>> things) and then see what needs fixing and how.
>>
>> On Thu, 1 Aug 2019, 00:26 Giuseppe De Marco,
>> <giuseppe.demarco at unical.it> wrote:
>>
>>> Hi All,
>>>
>>> this is a preliminar analisys about the strategy to obtain in MS
>>> constructors (__init__) the proxy configuration and also an
>>> abstract object where to store the shared state between MS (as we
>>> talked July the 9nth).
>>>
>>> In this place:
>>>
>>> https://github.com/IdentityPython/SATOSA/blob/master/src/satosa/base.py#L89
>>>
>>>
>>>
>>>
>>>
>>>
we could pass proxy config as argument or calling a global function that
>>> does the following before returning satosa_config
>>>
>>> config_file = os.environ.get("SATOSA_CONFIG", "proxy_conf.yaml")
>>> satosa_config = SATOSAConfig(config_file)
>>>
>>> This latter idea could introduce in addition also a refactor of
>>> this:
>>>
>>> https://github.com/IdentityPython/SATOSA/blob/master/src/satosa/wsgi.py#L12
>>>
>>>
>>>
>>>
>>>
>>>
>>>
moving it to a specialized function, importable everywhere, to get proxy
>>> configuration where needed. I think this is a better strategy to
>>> avoid passing configuration as argument in MS __init__.
>>>
>>> Regards
>>>
>>>
>>>
>>> Il giorno ven 12 lug 2019 alle ore 17:52 Rainer Hoerbe
>>> <rainer at hoerbe.at> ha scritto:
>>>
>>>> I am open to different solutions, with the preference to do it
>>>> consistently within SATOSA. So I am happy to accept decisions
>>>> from the group or our BDFL.
>>>>
>>>> However, I am a bit opinionated wrt passing the config via
>>>> function args. (1) By making an explicit argument an implicit
>>>> kwarg you would remove the possibility of static type checking.
>>>> And (2) in general, configuration is rather a cross-cutting
>>>> concern and should therefore not be part of a method signature.
>>>> Also, when tracing the code (-> import autologging), the config
>>>> data clutters the trace output.
>>>>
>>>> - Rainer
>>>>
>>>> Am 2019-07-12 um 15:41 schrieb Giuseppe De Marco <
>>>> giuseppe.demarco at unical.it>:
>>>>
>>>> Il giorno ven 12 lug 2019 alle ore 07:56 Rainer Hoerbe
>>>> <rainer at hoerbe.at> ha scritto:
>>>>
>>>>>
>>>>> right, with the exception that django hides some flexibility
>>>>> in configuring the settings file behind the import
>>>>> statement.
>>>>>
>>>>
>>>> settings = LazySettings() where LazySettings is at:
>>>> https://github.com/django/django/blob/master/django/conf/__init__.py#L42
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
If I'm not wrong you told me that in the django approach the filename
>>>> "settings" is a constant and this is a limit. This default
>>>> behaviour can be overriden:
>>>> os.environ.setdefault("DJANGO_SETTINGS_MODULE",
>>>> "django_peo.settings") so it just rely to a environment
>>>> variable called DJANGO_SETTINGS_MODULE.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Not exactly. I have to 3 use cases:
>>>>>
>>>>> 1) obtain the satosa configuration in a microservice
>>>>> (currently only the ms configuration is available) 2) a
>>>>> potential refactoring that where modules would import
>>>>> configuration like in django, instead of passing the config
>>>>> as constructor argument. 3) pass the request state from
>>>>> request ms to the response ms
>>>>>
>>>>> The „singleton" pattern applies to 1 + 2, whereas I think the
>>>>> 3 is a separated concern. But it would be nice to have both
>>>>> types of state sharing as a defined infrastructure in satosa,
>>>>> not as a ms-specific hack.
>>>>>
>>>>>
>>>> Point 1) Got it! We could in addition let the "config" argument
>>>> to be handled in **kwargs. For example now we have:
>>>> RequestOrResponseMicroService.__init__(self, config, *args,
>>>> **kwargs):
>>>>
>>>> we could refactor in this way:
>>>> RequestOrResponseMicroService.__init__(self, *args, **kwargs):
>>>>
>>>> and handle the config with a overridable approach, like this
>>>> PoC: self.config = kwarg.get('config',
>>>> lazy_approach_to_get_global_config()) if theresn't an
>>>> overloaded config argument it will always get the global one.
>>>> This refactor will only impact on MS classes and the
>>>> contruction calls in the code to these, where the first
>>>> arguments "config" should be named, if needs to be overloaded
>>>> just in that MS runtime, or not -> for default behaviour.
>>>>
>>>> Point 2) Completely agree with you. I want to be honest, every
>>>> time I'll hear "Django" I'll say: Yes, it is, Yes, it does,
>>>> Yes, let's do it. I apologize and confess for it in advance! If
>>>> we agree we could spend some thoughs about an abstraction layer
>>>> to get and set configurations attributes runtime. We should
>>>> have a class to handle get/set for lazy_conf. In it,for every
>>>> plugins in the dictionary (key: name_of_plugin) for
>>>> INTERNAL_ATTRIBUTES, BACKEND_MODULES, FRONTEND_MODULES and
>>>> MICRO_SERVICES, we should have the same type with the same
>>>> methods... Or a multilevel nested dictionary for all. You'll
>>>> know which is better starting from to what we have at the
>>>> moment.
>>>>
>>>> Point 3) I'll listen to this topic the better I can to
>>>> undertand in the deep every related critical issues. From my
>>>> point of view, that ignorant one, if we would configure SATOSA
>>>> in a loadbalancer where the same user-agent session should be
>>>> shared by different workers/agents/servers we need to make some
>>>> shared storage persistence between these (Ivan told us about
>>>> that layer, starting from sqlalchemy that sounds quite good). I
>>>> confess in advance that I dislike cookies. From this stored
>>>> "session" we could get and set information runtime from
>>>> everywhere in the code.
>>>>
>>>> If these words were be code we would have just finished the
>>>> refactor! Too verbose!!
>>>>
>>>>
>>>>> - Rainer
>>>>>
>>>>
>>>> Thank you R
>>>>
>>>> ____________________ Dott. Giuseppe De Marco CENTRO ICT DI
>>>> ATENEO University of Calabria 87036 Rende (CS) - Italy Phone:
>>>> +39 0984 496961 e-mail: giuseppe.demarco at unical.it
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
Il banner è generato automaticamente dal servizio di posta elettronica
>>>> dell'Università della Calabria <http://www.unical.it/5x1000>
>>>>
>>>>
>>>>
>>>
>>> -- ____________________ Dott. Giuseppe De Marco CENTRO ICT DI
>>> ATENEO University of Calabria 87036 Rende (CS) - Italy Phone: +39
>>> 0984 496961 e-mail: giuseppe.demarco at unical.it
>>> _______________________________________________ Satosa-dev
>>> mailing list Satosa-dev at lists.sunet.se
>>> https://lists.sunet.se/listinfo/satosa-dev
>>>
>>
>
5 years, 9 months
- 2
- 1
sharing state between satosa microservices
by rainer＠hoerbe.at
We did not have time in the last call to discuss this:
There are use cases where we need to share state between a request and a response microservice. Now we have already 2 cases, so I suggest to define a common method to achieve this. The same approach could also be used to access the common config, e.g. if you need to know the proxy configuration (such as a backend entityid) in a microservice.
A simple mechanism would be to use a module-level variable as singleton:
=====================
shared_state.py
state = {}
———
plugins/microservices/a.py
from shared_state import state # import executes only once
…
state[’a’] = 'foo'
———
plugins/microservices/b.py
from shared_state import state
whatever(state['a‘])
=====================
I thing that for just passing request status to response microservices and passing config data around this should be good enough. There are several alternatives, like the Borg pattern, which I find harder to read.
- Rainer
5 years, 9 months
- 3
- 8